Configuration

Configure Voithos for your applications
This document provides a walk through of how to configure Voithos to autoscale your applications. If you’re looking a quick and easy tl;dr, see this Tutorial for common usage patterns.

The VoithosAutoscalingGroup custom resource allows you to group your workloads according to a common set of autoscaling configuration rules. This allows you to tell Voithos to do things like:

  1. Set the target memory utilization equal to 72% on all workloads with kind=Deployment and name=foo across all namespaces.
  2. Do the same as (i) and enable automatic patching of memory, but not cpu.
  3. Set the cpu requests on all workloads in namespace=foo based on the 95th percentile of usage with a 15% bufffer, and adjust the cpu target utilization of any associated HPAs accordingly, with automatic patching of both cpu and memory.
  4. Do the same as (iii) but only for containers with name=bar and HPAs with name=baz.
  5. Do the same as (iv) and perform updates every 48 hrs while allowing Voithos to make intermediate updates as necessary based on load fluctuations.

VoithosAutoscalingGroup

There are six logical components to the VoithosAutoscalingGroup abstraction:

  1. Workload selection: select which workloads a configuration applies to.
  2. Container selection: select which containers in the selected workload(s) a configuration applies to.
  3. Container resources configuration: configure how container resource requests and limits get set.
  4. HPA selection: select which HPAs a configuration applies to.
  5. HPA configuration: control whether the existing HPA target utilization should be honored or overwritten according to its associated container resources configuration.
  6. Patch schedule: configure the timing of updates.

These components each relate to a top level field in the VoithosAutoscalingGroup custom resource definition, and are described in further detail below.

# Top level schema of VoithosAutoscalingGroup
apiVersion: virtex.ai/v1
kind: VoithosAutoscalingGroup
metadata:
  name: foobar
spec:
  selector: object
  configuration:
    patching: object
    resources:
      containers: array
      horizontalPodAutoscalers: array
If a workload has an owner reference to a builtin workload type, Voithos will evaluate the selection rules on the parent object. If a workoad has an owner reference to a non-builtin workload type, for example an Argo Rollout, Voithos will evaluate the selection rule on the child object, but will never patch the child object. Workload patches can be accessed through Voithos’ internal API.

Workload Selection

Refers to the selector section of the CRD API reference docs.

The spec.selector field gives you the ability to select workloads by kind, namespace, workload name, and label selector.

Kinds

Array (Optional)

You can select workloads by kind using an array of valid built-in workload types (Deployment, ReplicaSet, DaemonSet, StatefulSet, Job, CronJob). This field is optional; if left blank, all builtin kinds will be selected. For example, the expression:

# Example kind selection rule
apiVersion: virtex.ai/v1
kind: VoithosAutoscalingGroup
metadata:
  name: foobar
spec:
  selector:
    kinds:
    - Deployment
    - ReplicaSet

will filter the search to only deployments and replicasets.

Make this field granular to avoid inadvertantly selecting unwanted workloads. This can happen when, for example, you’re trying to select a workload with a generic name (e.g., proxy) across multiple namespaces.
Namespaces

Object (Optional)

To select resources by namespace (i.e., the metadata.namespace field), you can use a match expression with operators of “In” or “NotIn”. For example, the selection rule:

# Example namespace selection rule
apiVersion: virtex.ai/v1
kind: VoithosAutoscalingGroup
metadata:
  name: foobar
spec:
  selector:
    namespaces:
      operator: In
      values:
      - foo
      - bar

will include only workloads in the foo and bar namespaces. This field is optional; if left blank, all non Kubernetes control plane namespaces will be selected.

Workloads

Object (Optional)

To select resources by workload name (i.e., the metadata.name field), you can use a match expression with operators of “In” or “NotIn”. For example, the selection rule:

# Example workload name selection rule
apiVersion: virtex.ai/v1
kind: VoithosAutoscalingGroup
metadata:
  name: foobar
spec:
  selector:
    workloads:
      operator: NotIn
      values:
      - foo
      - bar

will exclude workloads that are named foo and bar. This field is optional; if left blank, the selection will not be filtered by workload name.

Label Selector

Array[Object] (Optional)

You can also use label selectors in your selection rule. This field accepts a list of match expressions. You can use a match expression with operators of “In” or “NotIn”. Only workloads matching all of the expressions are selected. For example, the label selector rule:

# Example label selector selection rule
apiVersion: virtex.ai/v1
kind: VoithosAutoscalingGroup
metadata:
  name: foobar
spec:
  selector:
    labelSelectors:
    - key: foo
      operator: In
      values:
      - bar
      - baz
    - key: qux
      operator: NotIn
      values:
      - quux

matches workloads containing the label foo=“bar|baz” AND NOT qux=“quux”.

Container Selection

Refers to containers section of the CRD API reference docs.

In this section we’ll walk through how configurations get associated with resources. The spec.configuration.resources.containers schema gives you flexibility to associate a configuration with a resource-type or a unique {container-name, resource-type} combination. As a guide, let’s assume that our workload selector was used to exclusively select the following workloads:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: foo
spec:
  template:
    spec:
      containers:
      - name: baz
        ...
      - name: qux
        ...
      - name: quux
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bar
spec:
  template:
    spec:
      containers:
      - name: baz
        ...
      - name: garply
        ...

Let’s now pretend we want to do the following:

  • (a) Specify a cpu configuration common to all containers with name baz
  • (b) Specify a cpu configuration common to all other containers
  • (c) Specify a memory configuration common to the baz and garply containers

We can express this as follows:

apiVersion: virtex.ai/v1
kind: VoithosAutoscalingGroup
metadata:
  name: foobar
spec:
  configuration:
    resources:
      containers:
      - container: baz
        resources:
          requests:
            cpu: ...    # (a)
            memory: ... # (c)
          limits:
            cpu: ...    # (a)
            memory: ... # (c)
      - resources:
          requests:
            cpu: ...    # (b)
          limits:
            cpu: ...    # (b)
      - container: garply
        resources:
          requests:
            memory: ... # (c)
          limits:
            memory: ... # (c)
Note: Omiting the container name serves as a wildcard capture. If multiple selection rules match to a container resource, priority will be based on rule specificity (i.e., selector that specifies a container name is chosen) followed by list order (i.e., first item is chosen).

Container Configuration

Refers to containers section of the CRD API reference docs.

Voithos allows you to autoscale container resources two different ways:

  • By specifying a target resource utilization ($u_{targ}$)
  • By specifying a target usage percentile ($q_{targ}$) and buffer ($S$)

When you specify a target utilization, you’re instructing Voithos to set resource requests, $x_{requests}$, according to:

$$ x_{requests} = E\Big[ \frac{x_{usage}}{u_{targ}}\Big] $$

Or in words, the expected value (mean) of the resource usage weighted by the inverse target utilization. When you specify a percentile and buffer value, you’re instructing Voithos to set resource requests according to:

$$ x_{requests} = Q_{x_{usage}}^{q_{targ}} + S $$

where $Q_{x_{usage}}^{q_{targ}}$ is the $q_{targ}^{th}$ percentile value of the resource usage distribution $p(x_{usage})$. The difference between these approaches can be minimal, or severe, depending on the distribution of a workload’s resource consumption. The former option allows you to control resource utilization/slack quasi deterministically, which is the option to take for non mission critical workloads for which moderate amounts of throttling or an occasional OOM is not the end of the world. At the other end of the spectrum are mission critical workloads, for which you must to mainain performance, availability, SLAs etc.., regardless of what the load distribution looks like. The latter option (requests = percentile + buffer) gives you knobs to express this desire without having to think too much.

Using the example from the preceeding section, lets specify configurations a, b, and c, plus one additional setting:

  • (a) All containers with name baz should have a target cpu percentile of 96 with a 15% buffer, a minimum cpu requests range of 20-1000 millicore, and a 1.2 limits-to-requests ratio. Patching should be enabled.
  • (b) All other containers should have a target cpu utilization of 70%. Patching should be enabled.
  • (c) The baz and garply containers should have a target memory percentile of 99 with a 25% buffer. Patching should be enabled for requests, but not limits.
  • (d) All other containers should use the default setting for memory, with patching enabled.

You can express this as follows:

apiVersion: virtex.ai/v1
kind: VoithosAutoscalingGroup
metadata:
  name: foobar
spec:
  configuration:
    resources:
      containers:
      - container: baz
        resources:
          requests:
            cpu:
              enablePatching: true
              min: 20m
              max: 1.0
              target:
                percentile: 96
                bufferPercentage: 15
            memory:
              enablePatching: true
              target:
                pecentile: 99
                bufferPercentage: 25
          limits:
            cpu:
              enablePatching: true
              target:
                requestsRatio: 1.2
            memory:
              enablePatching: true
      - container: garply
        resources:
          requests:
            memory:
              enablePatching: true
              target:
                pecentile: 99
                bufferPercentage: 25
      - resources:
          requests:
            cpu:
              enablePatching: true
              target:
                utilization: 70
            memory:
              enablePatching: true
          limits:
            cpu:
              enablePatching: true
            memory:
              enablePatching: true
Note: Patching is opt-in and must be enabled granularly for each nested subfield within the containers[].resources section. Refer to the API reference. You also must ensure that Voithos has the required permissions to perform patching.

HPA Selection

Refers to horizontalPodAutoscalers section of the CRD API reference docs.
Note: Voithos requires the /v2/autoscaling API which is present in v1.23+.

In this section we’ll walk through how HPA configurations get associated with HPA resources. Using the example above, lets attach HPAs to workloads foo and bar:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: foo
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: foo
  metrics:
  - type: Resource
    resource:
      name: cpu
      target: ...
  - type: Resource
    resource:
      name: memory
      target: ...
  - type: ContainerResource
    containerResource:
      container: baz
      name: cpu
      target: ...
...
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: bar
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: bar
  metrics:
  - type: Resource
    resource:
      name: cpu
      target: ...
  - type: External
    external: ...

Notice that we’ve specified three different types of scaling metrics: Resource, ContainerResource, and External. Refer to the K8s API Reference Docs to learn more about the five different HPA scaling metrics. Voithos works seamlessly with the full autoscaling/v2 API.

Let’s now pretend we want to do the following:

  • (e) Specify a configuration for the ContainerResource metric in HPA foo, which is pointed at the baz container cpu utilization.
  • (f) Specify a separate configuration for all cpu Resource HPA metrics.
  • (g) Specify a configuration for the Resource metric in HPA bar, which is pointed at the Pod memory utilization.

This can be expressed as follows:

apiVersion: virtex.ai/v1
kind: VoithosAutoscalingGroup
metadata:
  name: foobar
spec:
  configuration:
    resources:
      containers:
      - ... # (a-d)
      horizontalPodAutoscalers:
      - metrics:
        - name: foo
          type: ContainerResource
          containerResource:
            name: "cpu"
            target: ...   # (e)
        - type: Resource
          resource:
            name: "cpu"
            target: ...   # (f)
        - name: bar
          type: Resource
          resource:
            name: "memory"
            target: ...   # (g)

As shown above, Voithos gives you the flexibility to configure individual metric targets within an HPA object. Similar to the container resources configuration schema, leaving the name field blank serves as a wildcard capture, which allows you to also configure HPA metric targets across multiple HPAs using a single declaration.

HPA Configuration

Refers to horizontalPodAutoscalers section of the CRD API reference docs.

Voithos allows you to control whether an HPA target utilization will be honored or patched. When HPA target utilization patching is not enabled, Voithos will set the requests according to the current HPA target utilization value and the current usage distribution. If HPA target utilization patching is enabled, Voithos will set resource requests according to what is declared in the container resources configuration, and then patch the HPA target utilization to a value derived from the utilization corresponding to that configuration and the current usage distribution.

Using the example from the preceeding sections, let’s pretend we want to do the following:

  • (e) Enable patching of the ContainerResource target utilization in HPA foo, which is pointed at the baz container cpu utilization.
  • (f) Enable patching for cpu Resource metric targets across all HPAs, but specify bounds of 40-80%.
  • (g) Keep patching for the Resource metric in HPA bar disabled.

This can be expressed as follows:

apiVersion: virtex.ai/v1
kind: VoithosAutoscalingGroup
metadata:
  name: foobar
spec:
  configuration:
    resources:
      containers:
      - ... # (a-d)
      horizontalPodAutoscalers:
      - metrics:
        - name: foo
          type: ContainerResource
          containerResource:
            name: "cpu"
            target:
              utilization:
                patchingEnabled: true
        - type: Resource
          resource:
            name: "cpu"
            target:
              utilization:
                patchingEnabled: true
Note: Because HPA metric target utilization patching is disabled by default, configuration (f) is equivalent to the fallback/default behavior, and thus it’s not necessary to declare it.
Note: Configuration (g) is shown solely to demonstrate Voithos’ behavior, it wouldn’t make sense to declare a container configuration for a resource that is scaled by an non-custom HPA metric target without enabling patching on that HPA metric target because Voithos will ignore the container resource configuration in that case.

Patch Schedule

Refers to patching section of the CRD API reference docs.

To control how frequently updates will be made to the workload’s pod spec (i.e. container resources), you can use the schedule field. The implied units are seconds unless otherwise specified with a qualified suffix in the set (s=seconds, m=minutes, h=hours, d=days, w=weeks). Additionally, you can allow Voithos to make off-cycle updates when load fluctuations warrant them using the boolean allowReactiveUpdates flag. The default setting is:

configuration:
  patching:
    schedule: "1d"
    allowReactiveUpdates: true
Voithos purposfully does not support crontabs in the schedule field. Crontabs encourage you to apply bulk changes all that the same time during predefined maintenance periods. Realtime provisioning is the way of the future (in-place pod spec updates are supported as of v1.27!), and Voithos is here to promote force them.
Note: Setting allowReactiveUpdates=true does not trigger patching; it only allows voithos to make off-cycle updates to resources with patching enabled. Refer to the container resources and HPA sections for details about how to enable patching.

Default Settings

Voithos does not require you to configure all fields of the workload object when you create a VoithosAutoscalingGroup custom resource. For example, if you don’t provide a memory limits configuration, Voithos will use a fallback setting contained in Voithos’ default autoscaling group, which gets created during the helm installation. These fallback settings are shown below.

# Default autoscaling group fallbacks
configuration:
  patching:
    schedule: "1d"
    allowReactiveUpdates: true
  resources:
    containers:
    - resources:
        requests:
          cpu:
            enablePatching: false
            min: "5m"
            target:
              percentile: 97
              bufferPercentage: 15
          memory:
            enablePatching: false
            min: "25Mi"
            target:
              percentile: 99
              bufferPercentage: 20
        limits:
          cpu: 
            enablePatching: false
            min: "50m"
            target:
              requestsRatio: 2.0
          memory:
            enablePatching: false
            min: "100Mi"
            target:
              requestsRatio: 2.0
    horizontalPodAutoscalers:
    - metrics:
      - type: Resource
        resource:
          name: "cpu"
          target:
            type: Utilization
            averageUtilization:
              enablePatching: false
    - metrics:
      - type: Resource
        resource:
          name: "cpu"
          target:
            type: AverageValue
            averageValue:
              enablePatching: false
    - metrics:
      - type: Resource
        resource:
          name: "memory"
          target:
            type: Utilization
            averageUtilization:
              enablePatching: false
    - metrics:
      - type: Resource
        resource:
          name: "memory"
          target:
            type: AverageValue
            averageValue:
              enablePatching: false
    - metrics:
      - type: ContainerResource
        containerResource:
          name: "cpu"
          target:
            type: Utilization
            averageUtilization:
              enablePatching: false
    - metrics:
      - type: ContainerResource
        containerResource:
          name: "cpu"
          target:
            type: AverageValue
            averageValue:
              enablePatching: false
    - metrics:
      - type: ContainerResource
        containerResource:
          name: "memory"
          target:
            type: Utilization
            averageUtilization:
              enablePatching: false
    - metrics:
      - type: ContainerResource
        containerResource:
          name: "memory"
          target:
            type: AverageValue
            averageValue:
              enablePatching: false

Evaluate Selection Rules

Note: Refers to the status field in the VoithosAutoscalingGroup.

The spec.selector field allows you to group resources and configure them using a common set of autoscaling rules. This means that autoscaling groups can have overlapping workload selection rules. Voithos implements a simple rule to resolve these collisions: workloads can only be assigned to a non-default autoscaling group if they are currently assigned to the default autoscaling group. Voithos does not use an admission controller to enforce this logic, but will ignore any autoscaling group that captures workloads that are already assigned to a non-default autoscaling group. We provide an internal API for you to easily check the status of a newly created autoscaling group.

Note: if the creation of a voithosautoscalinggroup precedes the creation of a workload that the autoscalinggroup captures with its workload selector, the workload will be assigned to that group.
Note: The Voithos controller reads autoscaling groups every 5 minutes by default. The initial status of the autoscaling group will be set to pending until it’s picked up by the controller.

A valid autoscaling group will have no collisions; we can determine wehther or not an autoscaling group is valid by inspecting the .status.numCollisions field of the object:

curl -s http://localhost:8081/autoscalinggroups/<name> | jq `.status.numCollisions`

which returns the number of collisions that need to be resolved. If there are collisions, you’ll want to know what they are. You can retrieve a map of autoscaling group-to-workload object references using this command:

curl -s http://localhost:8081/autoscalinggroups/<name> | jq `.status.collisions`

The absence of collisions doesn’t confirm that your workload selection rule is correct. You might be unintentionally capturing unwanted workloads that haven’t been assigned to another autoscaling group (and are thus being autoscaled according to the default autoscaling group). To confirm that your selection rule captures the right set of workloads, you can fetch both the number of matched workloads as well as a list of matched workload names using these commands:

curl -s http://localhost:8081/autoscalinggroups/<name> | jq `.status.numMatched`

curl -s http://localhost:8081/autoscalinggroups/<name> | jq `.status.matches`
Tip: It’s generally a good idea to deploy an autoscaling group with patching disabled to validate the selection rule before allowing Voithos to patch resources.

initContainers

When the k8s scheduler schedules a Pod, it uses the effective requests, which is defined as:

$$ \max\Big( \sum_{c \ \in \ containers} x_{c,req} \ , \ \max_{c’ \ \in \ initContainers}( x_{c’,req} ) \Big) $$

In words, the kubernetes scheduler uses the maximum requests that a Pod requires to run at any point in time over its lifetime, which equates to either the largest init container or the sum of the container requests, depending on which is larger. If any container resource has patching enabled in a Pod, Voithos reserves the right to patch that resource in each of the Pod’s init containers. It will only do this for init containers that have resource requests and/or limts set. It does this to ensure that init containers don’t exceed the size of the Pod. Limits are subsequently patched as needed to maintain the validity of the Pod spec, and then any contraints imposed by LimitRanges are be enforced.

Note: Voithos does not observe ResourceQuotas