Workload Autoscaling

Workload Autoscaling (workload-autoscaling) scales apps on demand, down to zero. It runs KEDA: a workload's replicas follow real demand, such as a queue's depth, a request rate, a schedule, or plain CPU and memory, and go to zero when there is nothing to do. Node Autoscaling scales the nodes those replicas land on; the two are often installed together. It scales nothing until you ask: a model application scales through its Scaling settings, and any other workload through your own ScaledObject or ScaledJob.

When to use it

• A generative model's traffic swings. Without it, a GPU model runs at a fixed replica count, because CPU does not follow a GPU's load. With it, and a metrics source that reads the model's queue, the model scales on requestsWaiting, the requests waiting in its runtime, which rises as soon as requests queue.
• Your app should scale on its own metric. With a ScaledObject and a Prometheus trigger, replicas follow a metric your app exports, such as request rate or jobs in flight.
• Workers drain a queue or a topic. With a ScaledObject or ScaledJob on the queue, workers grow with the backlog and go to zero when it is empty.
• The Gateway proxy should scale on request rate. That needs this add-on with a metrics source: Gateway.

You don't need it when CPU tracks your load: a model can scale on CPU, and other workloads on a plain HorizontalPodAutoscaler, without it.

Before you start

• A cluster on Kubernetes 1.34 or newer.
• To scale on runtime metrics: a Prometheus reachable from the cluster that already collects the metrics of the workloads being scaled. AstroPulse installs none for this add-on. With Cost Visibility installed, the console suggests its Prometheus.

In the console

After Version, its steps are Metrics source, then Advanced, Resources and Components. Review runs the dry run, and Ask Nova at the top of the steps answers about the open step.

1. Open the cluster from the cluster list, select the Add-ons tab, then Install Add-on, and pick Workload Autoscaling. It says what it unlocks: Unlocks model scaling on queue depth.

2. Under Prometheus, enter the Endpoint. With Cost Visibility on the cluster, the field suggests its Prometheus. Leave it empty if nothing will scale on runtime metrics.

   Open zoom view

   Configure Workload Autoscaling with the Prometheus endpoint Cost Visibility installed

   

3. On Review, the dry run runs by itself; select Continue to deploy once it passes, then Deploy workload-autoscaling.

To change it later, open the add-on, select Update, then Change configuration, and change the endpoint. Go to Review, select Continue to apply once the dry run passes, type workload-autoscaling, then select Update workload-autoscaling.

You should see

The add-on reads Deployed. Without a metrics source, astroctl infra k8s addons get <cluster> workload-autoscaling says so: "installed, but no metrics source is configured: set configuration.metricsSource.prometheus.endpoint to a Prometheus that scrapes the workloads being scaled".

Open zoom view

Workload Autoscaling on its Overview: Deployed, with 1/1 check passing

How the three fit together

Each add-on answers one question:

Add-on	Decides
Workload Autoscaling	How many replicas a workload should have
Workload Queueing	Which replicas may run now, within each queue's quota, and in what order the rest wait
Node Autoscaling	Which machines to add so admitted replicas have somewhere to run, and which to remove once they are empty

A burst of requests to a model goes through all three: requests wait in the model's runtime, so Workload Autoscaling raises its replica count; Workload Queueing admits the new replicas while their queue has GPU quota left; the admitted replicas don't fit on the current nodes, so Node Autoscaling launches GPU nodes for them. Each new replica loads its model, then takes traffic. After the burst, the replica count falls back, and Node Autoscaling removes each GPU node once it has stayed empty for its consolidation delay.

How a model picks its signal

A model that names a scaling metric scales on it. One that names none:

• scales on requestsWaiting when it is generative and this add-on has a metrics source;
• runs at a fixed replica count when it uses GPUs and this add-on can't read its queue;
• scales on CPU otherwise.

Install the add-on before you deploy GPU models, so they scale with demand. A model that names requestsWaiting on a cluster without this add-on is refused (below). A trigger that reads a cloud queue signs in with your own TriggerAuthentication, or the cluster's workload identity.

Run it in production

• Treat it as critical. Its metrics API is part of the Kubernetes API. While it is down, API discovery and namespace deletion stall across the whole cluster, not only for scaled workloads. Set Availability to High on a production cluster.
• What it reserves: the operator 100m CPU and 512Mi memory, the metrics API 100m and 256Mi, and its webhook 50m and 128Mi.
• Uninstalling it is refused while the Gateway scales on requests per second. Switch the Gateway to CPU first.

Other ways in

• Nova: ask it, for example "install workload autoscaling on eks-prod using the Prometheus at http://prometheus.monitoring.svc:9090". Nova asks for the endpoint rather than guessing one.
• SDK and API: the same operations, in the API reference.

With astroctl

workload-autoscaling.yaml

apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: workload-autoscaling
spec:
  clusterName: eks-prod
  configuration:
    workloadAutoscaling:
      metricsSource:
        prometheus:
          endpoint: http://prometheus.monitoring.svc:9090

Terminal

astroctl infra k8s addons apply -f workload-autoscaling.yaml --dry-run
astroctl infra k8s addons apply -f workload-autoscaling.yaml

astroctl infra k8s addons template eks-prod workload-autoscaling writes the template, with the suggested endpoint when the cluster has one.

Settings

Its own setting, plus availability and resources:

EndpointworkloadAutoscaling.metricsSource.prometheus.endpoint

The address of a Prometheus, reachable from the cluster, that already collects the metrics of the workloads being scaled. Scaling on request rate finds a workload's metrics by their namespace label.

Default

Empty: no runtime metrics source. The add-on is installed and runs, and a model that names no scaling metric falls back as above.

If it's refused

"metricsSource.prometheus.endpoint is required: AstroPulse does not install a Prometheus for autoscaling, it queries one you already run"

Cause

The manifest has metricsSource with no Prometheus endpoint in it.

Fix

Set the Endpoint, or remove metricsSource to install with no runtime metrics source.

"Gateway scales its proxy on requests per second through Workload Autoscaling. Apply Gateway with autoScaling.metric CPU first, or remove Workload Autoscaling with force, which skips this check."

Cause

An uninstall while the Gateway scales on requests per second.

Fix

Change the Gateway's Auto scaling metric to CPU, then uninstall.

"Queue depth scaling needs the workload-autoscaling add-on"

Cause

A model names model.scaling.metric.requestsWaiting, and the cluster has no Workload Autoscaling with a metrics source to read its queue.

Fix

Install Workload Autoscaling with a Prometheus Endpoint that scrapes your models, or scale the model on cpu instead.

"Queue-depth scaling can't read this model's queue"

Cause

The metrics source is the Prometheus Cost Visibility installed, and it does not collect model metrics.

Fix

Do what the message says: point the Endpoint at a Prometheus that scrapes your models, or scale the model on cpu.

Next

• Node Autoscaling
• Workload Queueing
• Model Serving