Sizing and scaling
Set what one replica of a model runs on, how many replicas run, and the signal that adds or removes them. Every field has a default, so most models need nothing on this page.
In Deploy model, the Compute step sets the GPU and Resources and tuning, and the Scaling step sets Replicas and Scale on. Leave Scale on at Automatic. On a running model, the same steps are under Update.
Before you start
- A cluster with the Model Serving add-on installed and healthy. See Model Serving.
- For a GPU: a GPU node group, or a GPU pool on the Node Autoscaling add-on.
- To scale a generative model on waiting requests: the Workload Autoscaling add-on, pointed at a Prometheus that collects the model's metrics.
- To share a quota between models: a queue on the Workload Queueing add-on.
In the console
Choose the hardware
-
Open Workloads > Models > Deploy model and go to the Compute step. For a running model, open the model and select Update.
-
Under GPU accelerator, select a GPU, or None, CPU model. Each GPU shows the pool or node group that launches it, its limit, its capacity type and its instance types. A GPU that cannot launch shows the reason in red, for example a pool limit or a cloud quota that is too low.
Open zoom view -
Open Resources and tuning to set Memory, CPU, Context length and KV cache. Left empty, AstroPulse picks the defaults.
Open zoom view -
Optional: select Cache weights on the node (see Cut model download cost), and name a Workload queue (see Queueing).
Set replicas and scaling
-
On the Scaling step, set Replicas as a min and a max. Leave both empty for one replica. Set the max above the min to autoscale.
Open zoom view -
Under Scale on, keep Automatic: AstroPulse picks the signal for the model and the cluster. To choose, select Queue depth (
requestsWaiting) and set requests per replica, or CPU utilisation (cpu). See Scaling.Open zoom view -
Continue to Review, where the dry run checks every value, then Deploy (or Apply for an update).
You should see
The model's Overview has a Serving card with what runs: Accelerator, Memory budget, CPU, Replicas (the current count and the range), Scales on and Context length. A value AstroPulse chose is marked default.
While a model starts, its status shows the stage it has reached. See Startup stages.
Runtimes
A Hugging Face model is served as a generative model by default, with vLLM, on the API most AI clients already speak. For any other source, set Format:
| Format | Generative |
|---|---|
vLLM | yes |
SKLearn, XGBoost, LightGBM | no |
TensorFlow, PyTorch | no |
ONNX, TensorRT, Triton | no |
MLflow, PMML, Paddle | no |
huggingface is not a format and is refused. To use a serving runtime already installed on the cluster, name it in Runtime engine; AstroPulse does not check it against the format.
Sizing a replica
A replica is one copy of the model. Its memory and CPU are both the request and the limit, so one model's spike never evicts another. A replica runs on one node, so all its GPUs are on that node.
- Capacity type.
OnDemandsuits serving;Spotcosts less and can be reclaimed at two minutes' notice. Reserved capacity counts asOnDemand. ASpotmodel with no Spot capacity gets no node and does not fall back to On-Demand; anAnymodel runs on On-Demand when Spot is unavailable. - GPU vendor. The vLLM runtime AstroPulse installs runs on NVIDIA only, so
amdis refused for it. - Registered clusters. AstroPulse does not know a registered cluster's node groups, so it skips the GPU model and count checks there.
The GPUs offered per cloud and region are under Accelerators in the console.
Memory on a GPU instance
On a GPU, Memory is the machine's memory, not the GPU's. The weights end in GPU memory but pass through the machine's memory, and the runtime needs some of its own. A replica asks for the GPU and for CPU and Memory from the machine that carries it.
The request must fit what the instance has left after its system pods, which is less than its listed size. The dry run does not check this yet. A request that does not fit deploys, and the model waits for a node that never comes, with the status "The model is waiting for a node it can run on: a GPU node the cluster couldn't launch, or no room on its nodes. Its events say which." The model's Events tab then shows "no instance type has enough resources" or "Insufficient memory".
To choose memory for a GPU instance:
- Look up the instance your GPU node group or GPU pool launches. The Compute step lists it on each GPU.
- Start at the default, 8Gi. Raise it only if a replica runs out of memory while it starts.
- Keep it a few GiB under the instance's memory. On a 16 GiB instance, 12Gi fits and 14Gi does not.
- Keep CPU under the instance's vCPUs. The default of 3 per GPU fits the smallest single-GPU instances, which have 4.
If the model needs more, use a node group or GPU pool with a larger instance.
For example, on g6.xlarge (4 vCPU, 16 GiB, one L4), a 15 GB model runs at the default 8Gi and at 12Gi, and waits for a node at 14Gi.
Context length
Context length (serving.maxContextTokens) is the longest conversation, prompt and answer together, in tokens, that the runtime reserves room for. Set it to reserve less memory, or to cap what callers may send. Left empty, the runtime picks its own.
It must not exceed the model's own limit: the model card's context length, or max_position_embeddings in the model's config.json on Hugging Face. When AstroPulse can read the model's config.json, the dry run warns about a value above it:
Warning: Context is longer than the model's
source.model.serving.maxContextTokens is 2048, above this model's context of 512 tokens (its config.json), so the model won't start. Set it to 512 or less, or remove it to use the model's own.
It is a warning, not a refusal: the apply still goes through. Deployed, the model stops within a minute of loading and reads Can't start: "The model can't start: source.model.serving.maxContextTokens is longer than the model's context of 512 tokens. Set source.model.serving.maxContextTokens to 512 or less, or remove it to use the model's own, and apply the application again."
KV cache
KV cache (serving.kvCacheSize) is the memory a generative model keeps for the conversations it is working on. Most models need it left empty. An embedding model keeps no such cache and ignores the field.
| Where the model runs | Default | Taken from |
|---|---|---|
| CPU | about a quarter of Memory | Memory. A 1Gi cache with 6Gi of memory leaves 5Gi for the weights and the runtime |
| GPU | the runtime sizes it from the GPU's free memory | GPU memory, per GPU. Memory does not size it |
Set it on CPU to give a model more room for long or many conversations. Set it on a GPU only to hold the cache to a fixed size: the runtime then skips its own check of the GPU's memory, and every dry run warns Check the KV cache size. On an L4, 4Gi held about 350,000 tokens for Qwen2.5-0.5B-Instruct; a larger model holds fewer.
Images, video and audio
For a model that reads images, video or audio on the vLLM runtime, serving.multimodal sets how many of each one request may carry. Left out, the runtime's own limit stands, often one image. The runtime reserves memory for these limits on every replica, so a high limit can need more memory, and a video clip costs far more than an image. Above 64 the dry run warns.
Scaling
Replicas sets the fewest and the most replicas. Scale to zero is not available. Scale on picks the signal between them. There are four setups. With more than one replica, the replicas spread across nodes, and across zones on a multi-zone tier: see Availability tiers.
A fixed number of replicas
Set min equal to max, or leave both empty for one replica. There is no autoscaler. The console and the dry run say Model runs at a fixed replica count: "Set maxReplicas above minReplicas to scale this model." Choosing Queue depth or CPU utilisation with equal bounds is refused.
A range on Automatic
Set max above min and keep Automatic. AstroPulse picks the signal:
- a generative model on a cluster whose Workload Autoscaling add-on can read the model's metrics: Queue depth, shown as "More than 2 requests waiting";
- a model without a GPU: CPU utilisation;
- otherwise a GPU model runs at the min, and the dry run says what would let it scale, for example "Install Workload Autoscaling to scale this model on waiting requests."
A model on Automatic is never refused for its scaling.
Scale on waiting requests: requestsWaiting
Queue depth (requestsWaiting) is the number of requests waiting at the model, per replica, with a default target of 2. It is the signal for a generative model, on a GPU or not. It needs the Workload Autoscaling add-on; a predictive model publishes no such queue and is told to scale on CPU.
How scale-out behaves for a small chat model on L4 GPUs:
- A replica is added only when requests wait. The model answered 128 requests at once with nothing waiting; about 400 at once added a second replica.
- When the new replica needs a new GPU node, it serves about 8 to 9 minutes after it is added: the node starts, the model downloads, the image is pulled and the model loads.
- Once the load stops, the extra replica is removed after 3 to 5 minutes, and its empty GPU node about 10 minutes later. The 10 minutes is the GPU pool's setting for keeping an empty node.
- Each replica needs its own GPUs. If the pool's GPU limit is lower than max replicas times GPUs per replica, the dry run says so, for example "Autoscaling to 2 replicas needs 2 GPUs; the pool allows 1.", and the extra replicas wait.
To check the signal is read, send the model traffic and open its Metrics tab: requests waiting should rise above zero. If it stays at zero under load, the add-on cannot see the model's metrics, and the model stays at the min.
Scale on CPU: cpu
CPU utilisation (cpu) is CPU use as a whole percentage of the replica's CPU, from 1 to 100, default 80. It works on every cluster with a working Kubernetes metrics API. Use it for a model without a GPU.
A GPU model does its work on the GPU, so its CPU barely moves. CPU scaling on one is accepted with the warning CPU scaling doesn't follow GPU load. For example, a GPU model under full GPU load can sit at 27% CPU: a target of 30 adds no replica, and a target of 20 adds one.
On a cluster without a working metrics API, the model still serves, with the notice Application cannot scale: install metrics-server and the autoscaler starts with no redeploy, or set a fixed count.
Changing replicas or the metric on a running model
A change that replaces a model's replicas asks for no extra capacity. A model with a max of 1 stops and starts again, so it is down until the new replica is ready, about a minute for a small model on a warm node. With two or more replicas, they are replaced one at a time while the rest serve.
The target on the same signal can always change. Moving between signals:
| From | To | Result |
|---|---|---|
| CPU | a fixed count | Allowed |
| CPU | queue depth | Refused in one update. Update twice: first to a fixed count, and let it finish; then to the range and queue depth |
| a fixed count | CPU | Allowed |
| a fixed count | queue depth | Allowed once the earlier autoscaler is gone; until then refused with The earlier autoscaler hasn't gone yet. A range on Automatic lands here too on a generative model, with the warning Model moves to scaling on waiting requests, because it cannot be undone |
| queue depth | CPU | Refused: AstroPulse cannot take that autoscaler off a running model. Delete and redeploy the model, or keep queue depth |
| queue depth | equal bounds on Automatic | Accepted, but it keeps scaling on queue depth, with the warning Model keeps scaling on waiting requests. To run with no autoscaler, delete and redeploy it |
Queueing
A queue lets models share a quota and wait their turn for it, instead of piling up on a full cluster. Your platform team declares the queues on the Workload Queueing add-on, with deployment in its integrations, because a model's replicas run as a Deployment. The model then names a queue in Workload queue on the Compute step. The queue is fixed once the model exists.
A replica runs only once the queue admits it, when its GPU, CPU and memory fit the quota left. Until then it has no node, the notice is The application isn't serving, and the status reads "The model is waiting for its workload queue to admit it. Its metrics show how the queue stands." When quota frees up, for example when another model on the queue is deleted, the replica is admitted and starts. A small CPU model can start about two minutes after the model ahead of it is deleted.
The model's Metrics tab, under Scaling, has a Queue summary: the queue's name, "Replicas are waiting for quota." or "Nothing is waiting for quota.", Replicas admitted with how many wait, and the admission wait. See A queued model.
Startup stages
While a model starts, the console shows how far it has come. Not every model passes through every stage, and a short stage can be missed.
| Stage | Reason shown | Meaning |
|---|---|---|
| Provisioning | none, or a reason | The deploy is accepted. A reason here, such as waiting for a node or a queue, is what holds it back |
| Getting a node | "The model is waiting for a node to run on." | A node is being found or started. A new GPU node takes a few minutes |
| Preparing | "The model is being prepared on its node." | The replica is set up on its node |
| Downloading model | "The model is downloading to its node." | The weights are downloading. Skipped when they are on a volume or in the weight cache |
| Pulling image | "The model's serving image is being pulled onto its node." | The runtime's image is pulled. It is large, so this can take minutes on a new node |
| Loading model | "The model is loading; it serves once it has started." | The runtime loads the weights. On CPU, the first request after this can take a few minutes more |
| Can't start | why, in the notice | A container keeps failing, for example a context length above the model's limit, or running out of memory |
As a guide, a small CPU model starts in about 5 minutes on a running node and 8 to 9 minutes with a new one; a small model on a new GPU node in about 9 minutes; a 15 GB model in about 13 minutes. If a model stays in a stage, see How to debug a model.
Other ways in
- Nova: ask it in plain language, for example "run qwen-chat on one L4 and scale it to 2 replicas on waiting requests".
- SDK and API: the Python SDK and the API reference use the same fields.
With astroctl
resources:
cpu: "3"
memory: "8Gi"
accelerator:
gpu:
model: "l4"
countPerReplica: 1
capacityType: "OnDemand"
serving:
maxContextTokens: 4096
scaling:
minReplicas: 1
maxReplicas: 2
metric:
requestsWaiting:
target: "2"
queue:
name: research
astroctl application template model # every accepted value, with a starter spec
astroctl app apply -f model.yaml --dry-run # checks every field, deploys nothing
astroctl app events <app> --k8s # why a replica waits for a node
astroctl app metrics <app> # inference.requestsWaiting and the queue figures
astroctl cloud <provider> machines # instance types, vCPUs and memory
The queue is declared on the Workload Queueing add-on:
configuration:
workloadQueueing:
integrations: [batch/job, deployment]
clusterQueues:
- name: research
quota: { gpu: 2 }
See the astroctl reference.
Settings
All fields are under source.model.
formatThe model format, which picks the serving runtime. In the console, Advanced > Format on the Model step.
vLLM, SKLearn, XGBoost, LightGBM, TensorFlow, PyTorch, ONNX, TensorRT, Triton, MLflow, PMML, PaddlevLLM for a Hugging Face model; required otherwiseruntime.engineA serving runtime to use instead of the one the format picks. In the console, Advanced > Runtime engine.
resources.memoryMemory for one replica: weights, runtime and, on CPU, the KV cache. In the console, Memory. A value below the default is a dry-run warning, not a refusal.
"8Gi"resources.cpuCPU for one replica. In the console, CPU. On CPU inference, cores set the throughput.
"2" or "500m"resources.accelerator.gpu.modelThe GPU. In the console, the card selected under GPU accelerator. Without resources.accelerator.gpu, the model runs on CPU.
l4resources.accelerator.gpu.countPerReplicaresources.accelerator.gpu.capacityTypeHow the replica's GPU node is bought.
OnDemandOn-Demand Never runs the replica on Spot capacity; reserved capacity counts as on-demand.SpotSpot Runs the replica on Spot capacity, which costs less but can be reclaimed at short notice.AnyAny Runs the replica on whatever GPU capacity the cluster offers.
Any
resources.accelerator.gpu.vendorThe GPU vendor. amd is refused on the vLLM runtime.
nvidia, amdnvidiaserving.maxContextTokensThe longest conversation, in tokens, the runtime reserves room for. In the console, Context length. Keep it at or below the model's own limit.
serving.kvCacheSizeThe memory the runtime keeps for its KV cache. In the console, KV cache. On CPU it comes out of resources.memory and must be smaller than it; on a GPU it is device memory per GPU.
"2Gi"resources.memory. GPU: sized by the runtimeserving.multimodal.maxImagesPerPrompt, maxVideosPerPrompt, maxAudioClipsPerPromptHow many images, video clips or audio clips one request may carry, for a generative model on the vLLM runtime. Set at least one.
scaling.minReplicasscaling.maxReplicasThe most replicas. Above the min, the model autoscales; equal, it runs at that count.
scaling.metricThe signal replicas follow, with a target. In the console, Scale on. Leave it out for Automatic. Needs a max above the min.
requestsWaiting (a positive number, generative models only, needs the Workload Autoscaling add-on) or cpu (a whole percentage, 1 to 100, no leading zero)queue.nameA queue the cluster's Workload Queueing add-on declares. In the console, Workload queue. Fixed once the model exists.
If it's refused
These come back from Check model, Review or a dry run, before anything is deployed, unless the card says otherwise.
source.model.resources.memory, with the example "8Gi", for memory.2, 500m or 8Gi.nvidia or amd.multimodal block is empty.multimodal.configuration.workloadQueueing.clusterQueues.deployment.Spot model with no Spot capacity, or memory or CPU larger than the instance, both read like this.Any or OnDemand.kvCacheSize and maxContextTokens.max_model_lenNext
- Expose a model: give it a public HTTPS address for chosen callers.
- Operate a model: watch its metrics, update it and roll it back.
- Cut model download cost: what a new GPU node downloads, and what removes it.