Skip to main content

Scale the model with demand

Alice, the platform engineer, updates the model qwen-chat (Qwen/Qwen2.5-0.5B-Instruct) on eks-prod to one L4 per replica and 1 to 2 replicas. She sets the knobs; AstroPulse adds a replica when requests wait, up to her maximum, and Node Autoscaling launches its GPU node.

Before you start​

  • model-serving and cert-manager on eks-prod (your cluster's name).
  • workload-autoscaling, and Node Autoscaling with its GPU pool enabled.
  • A GPU node group on eks-prod.
  • cost-visibility, for cost per model.

1. Alice checks workload autoscaling​

Path: Infrastructure > Kubernetes > eks-prod > Add-ons

  • Confirm workload-autoscaling, installed in step 5 of Deploy your first model, is healthy with its Endpoint set to a Prometheus that collects the model's metrics. Without it, a model on a GPU runs at a fixed replica count, so qwen-chat (your model deployment's name) cannot grow.
Open zoom view

workload-autoscaling settings: Endpoint

workload-autoscaling settings: Endpointworkload-autoscaling settings: Endpoint

2. Alice picks the GPU​

Path: Workloads > Models > qwen-chat > Update > Compute

  • GPU accelerator l4 (the GPU model you need), 1 per replica. A 0.5B model fits on one L4 with room to spare; a second GPU per replica only adds cost.
  • Optional: capacityType OnDemand, because Spot capacity can be reclaimed mid-request.
  • Optional: Cache weights on the node, so a new replica starts from the node's disk. Memory and CPU are under Resources and tuning; left empty, the platform sizes them.
Open zoom view

Update, Compute step: GPU accelerator with l4 selected, GPUs per replica, capacityType, Cache weights on the node, Resources and tuning (Memory, CPU, Context length)

Update, Compute step: GPU accelerator with l4 selected, GPUs per replica, capacityType, Cache weights on the node, Resources and tuning (Memory, CPU, Context length)Update, Compute step: GPU accelerator with l4 selected, GPUs per replica, capacityType, Cache weights on the node, Resources and tuning (Memory, CPU, Context length)

3. Alice sets the scaling​

Path: Workloads > Models > qwen-chat > Update > Scaling

  • Replicas min 1, max 2 (the range your demand needs).
  • Scale on stays Automatic, which means requests waiting for a generative model whose queue workload autoscaling can read.
Open zoom view

Update, Scaling step: Replicas min 1 and max 2, Scale on Automatic selected, with Queue depth and CPU utilisation as the other choices

Update, Scaling step: Replicas min 1 and max 2, Scale on Automatic selected, with Queue depth and CPU utilisation as the other choicesUpdate, Scaling step: Replicas min 1 and max 2, Scale on Automatic selected, with Queue depth and CPU utilisation as the other choices
The knobs and what they trade
  • Replicas min: how many stay warm, at least 1 (a model does not scale to zero). More means no wait for a cold start and more GPUs billed while idle; 2 lets one restart during a release while the other serves.
  • Replicas max: your cost ceiling for this model, one L4 per extra replica. It defaults to the minimum, so set it above.
  • Scale on: what scaling follows, requests waiting (2 per replica on Automatic), Queue depth or CPU utilisation (80% by default). A Scaling target chosen explicitly at deploy can change here; a lower target adds replicas sooner and costs more. A model without a GPU scales on CPU.
  • GPUs per replica and Context length: more fits a larger model or longer prompts, at more GPU per replica.

4. Alice checks the API format​

Path: Workloads > Models > qwen-chat > Update > API

  • API format OpenAI-compatible, so Bob's existing OpenAI clients and SDKs call it unchanged.
Open zoom view

Update, API step: API format OpenAI-compatible, Media per request

Update, API step: API format OpenAI-compatible, Media per requestUpdate, API step: API format OpenAI-compatible, Media per request

5. Alice reads the result​

Path: Workloads > Models > qwen-chat > Overview

  • Under Serving, read Replicas (ready of desired, then the range), Scales on and Accelerator.
Open zoom view

Overview, Serving section: Replicas 1 of 1 (range 1 – 2), Scales on More than 2 requests waiting (default), Accelerator 1x nvidia l4

Overview, Serving section: Replicas 1 of 1 (range 1 – 2), Scales on More than 2 requests waiting (default), Accelerator 1x nvidia l4Overview, Serving section: Replicas 1 of 1 (range 1 – 2), Scales on More than 2 requests waiting (default), Accelerator 1x nvidia l4
Changing the signal on a running model

Moving from a fixed count is allowed. Moving from CPU to requests waiting takes two applies: first remove the signal and set the minimum equal to the maximum, then set the range and the signal; with one replica, the model is down during each apply. Moving off requests waiting is refused; delete and redeploy to change it.

When requests wait, the second replica appears and Replicas reads 2 of 2 (range 1 – 2).

NovaNova
Update qwen-chat to one L4 per replica, with 1 to 2 replicas.
Ask Nova

It shows the plan and applies it when you approve.