Operating a model

Read what a running model is doing, change it, roll it back, and fix it when it does not serve. Everything is on the model's page under Workloads > Models.

Open the model from Workloads > Models. Overview shows its status and why, Metrics its latency and tokens, Events and Logs what happened, History its versions with Roll back, and Update changes it with a dry run first. Ask Nova on the same page investigates a model that does not serve.

Load, metrics, queues and rollback

Before you start

• A model deployed with Serve a model.
• For latency, token and queue metrics: the Model Serving add-on; for GPU utilization, Accelerator Metrics.

In the console

Read its status, events and logs

1. Open the model from Workloads > Models. The header shows its status, for example Running, the model URI, the runtime and the GPU.

2. On Overview, a model that is not serving shows its reason in the header and in a notice with See events and See logs. The Serving card shows what runs, and Last exit the last container exit, for example "code 137 (OOMKilled)", which means the replica ran out of memory.

3. Open Events for the deployment, Kubernetes and platform events: scheduling, image pulls and probes.

   Open zoom view

   The Events tab: Deployment Events with Application is ready, Application is starting, and FailedScheduling warnings while a GPU node starts

   

4. Open Logs and select Start streaming for the runtime's output and the weight download's output.

   Open zoom view

   The Logs tab: Application logs for qwen-chat with Connect, Search logs and Start streaming

   

Read its metrics

Open Metrics. Inference metrics shows request rate, latency, time to first token, inter-token latency, token rate, requests running and waiting, and KV-cache utilization, reported by the runtime in windows of about two minutes. The Speed, Load, Accelerator and Resources tabs chart them over Last Hour, Last 24 Hours or Last 7 Days. A window that served nothing has no latency, so the line breaks instead of reading zero.

Open zoom view

The Metrics tab: inference metric cards, then the Speed tab with request latency and time to first token charts

What each one tells you:

A model reports what its serving runtime measured, alongside CPU, memory and network:

• Latency (p50, p95, p99): end-to-end time per request.
• Time to first token: how long before a caller sees output.
• Inter-token latency: how fast the stream runs once it starts.
• Token rate: prompt and generated tokens per second; generated tokens are what a GPU-hour buys.
• Requests running and waiting: waiting rising while running stays flat means too few replicas.
• KV-cache utilization: accelerator memory pressure, which comes before queueing.

A predictive model (sklearn, XGBoost) reports latency and request counts only. Percentiles are computed from the merged histograms of every replica, never averaged. GPU utilization, memory, power and faults need the Accelerator Metrics add-on. A model in a queue also shows its replicas admitted and waiting, and how long admission took, under Workload queue.

Metrics are kept about 24 hours. Without the Model Serving add-on there are no inference metrics; it reads the model's own metrics endpoint, with no Prometheus needed.

Model metrics: latency, time to first token, token rate

Load arriving live (sped up 8x)

Update it

1. On the model's page, select Update. The steps are Model, Compute, Scaling, Access, API and Notifications.

2. Change what you need, for example the GPU, Memory or Replicas. Changing compute replaces the replicas with ones of the new size; the model and its runtime stay the same.

   Open zoom view

   Update application, Compute step: GPU l4, 1 per replica, ondemand, nvidia, and Resources and tuning

   

3. On Review, the dry run checks the change. Select Apply.

The model, its format and its runtime are fixed once the model has served. To serve a newer model or revision, deploy it as a new model, move callers to it, then delete the old one. Sizing and scaling changes: see Changing replicas or the metric on a running model.

Roll it back

1. On the model's page, open History. Each version shows its result, for example Successful, Failed or Superseded, who deployed it, and when.

   Open zoom view

   Version history: v4 Current and Successful, v3 Failed with its reason, v2 and v1 Superseded with Roll back

   

2. Select Roll back on the version you want. The dialog shows the move, for example v4 to v2, and what the rollback does and does not change.

   Open zoom view

   Roll back application: v4 to v2, the endpoint keeps its access policy, type qwen-chat to confirm

   

3. Type the model's name and select Confirm rollback. The rollback deploys that version's specification as a new version.

A rollback returns the same bytes only when the version named one exact artifact: a Hugging Face commit, an oci:// digest, an s3:// object with versionId or a gs:// object with generation. A version that pointed at anything else (a Hugging Face branch or tag, an image tag, an s3:// or gs:// path with no version, an Azure Blob path, pvc://) redeploys its specification, but serves whatever is at that location now. A rollback never happens by itself. See Rollback and deploy history.

You should see

A healthy model reads Running in its header and on Workloads > Models, with its endpoint, and its metrics fill in once it serves requests. A first deploy reads Provisioning until a replica serves. While it starts, the status names the stage; see Startup stages.

Open zoom view

Workloads > Models listing a model as Running, with its cluster, GPU and endpoint

Call the model from code

The endpoint speaks the OpenAI API, so any OpenAI client works: point its base URL at the endpoint and send a key the access policy accepts. Call it from code on the model's Overview gives the code for that model.

Open zoom view

Call it from code: find the endpoint with the AstroPulse SDK, then call it with an OpenAI client

The model id a request sends is the one AstroPulse reports as the model's servedModelName; the runtime answers 404 to the application name and to the repository id. The model's task says what it serves: Generate for chat and completions, Embed for embeddings. A chat model answers 404 on /v1/embeddings, and an embedding model on /v1/chat/completions.

Python

from openai import OpenAI

chat = OpenAI(base_url="https://qwen-chat.acme.ai/v1", api_key="<key the policy accepts>")

chat.chat.completions.create(
    model="qwen-chat-a1b2c3",                   # model.servedModelName
    messages=[{"role": "user", "content": "Hello"}],
)

Drive the platform from Python

The AstroPulse Python SDK is the whole API: applications, clusters, add-ons, access policies, domains, profiles, integrations, audit and API keys. It uses your platform key, not the model's key. For automation, use a key from Service Accounts in the sidebar; see service accounts.

Terminal

pip install "git+https://github.com/astropulseinc/astropulse-python"
export ASTROPULSE_API_KEY=...

Python

from astropulse import Client

client = Client()                                   # ASTROPULSE_API_KEY from the environment
series = client.applications.get_application_metrics_series(
    app_name="qwen-chat", window="15m",
).data

s = series.inference_summary
print(s.runtime_name, s.request_count)              # vLLM 18
print(s.request_latency_p50_seconds)                # 17.5
print(s.time_to_first_token_p50_seconds)            # 1.84

series.points has the same figures per window, as the console charts them. A window that served nothing reports its latency percentiles as absent, not zero. Runnable examples are in examples/.

GPU quota

Before a GPU model deploys, AstroPulse reads the cloud account's GPU quota in the cluster's region. A quota with room adds a line to the dry run, for example "Quota Running On-Demand G and VT instances (L-DB2E81BA) has room: 4 of 8 vCPU in use; one L4 replica needs 4 on g6.xlarge." The Compute step shows the same per GPU. A short quota refuses a new model; it never refuses a re-apply, an update, a rollback, or a GPU pool that launches on capacity reservations first, which deploy with the warning GPU quota may be too low. See If it's refused.

Other ways in

• Nova: select Ask Nova on the model's page, or ask in plain language, for example "why is qwen-chat not serving?".
• SDK and API: the Python SDK and the API reference.

With astroctl

Terminal

astroctl app get <app> -o yaml           # status, statusReason, model.servedModelName
astroctl app events <app>                # deployment events
astroctl app events <app> --k8s          # the cluster's events: scheduling, image pulls, probes
astroctl app logs <app>                  # the runtime's and the weight download's output
astroctl app metrics <app>               # inference metrics and queue figures
astroctl app apply -f model.yaml --dry-run   # check a change; deploys nothing
astroctl app apply -f model.yaml         # update
astroctl app rollback <app>              # roll back to a version from history

See the application CLI reference. Deploying any application, by every path: Deploy an application.

If it's refused

A model has the statuses every application has: Provisioning until a replica serves, then Running, or Failed. Start with the status and its reason on Overview, then Events and Logs.

Refused before deploy

Check model, Review and a dry run refuse these with the field to change. Nothing is deployed.

A refusal that starts "Hugging Face model <repo>" or "Hugging Face has no public model"

Cause

Hugging Face says the model, its revision or its gated files cannot be read.

Fix

Each refusal and its fix is on Hugging Face.

"the cluster's model-serving credential cannot read <uri>: grant it read access, or change the model-serving add-on's secretRef or identity"

Cause

The bucket or registry refuses the identity of the cluster's Model Serving add-on.

Fix

Grant that identity read access, or change it on the add-on. See Serve a model from your own storage.

"the Secret <name> that the cluster's model-serving add-on names (secretRef) does not exist in the <namespace> namespace: create it, or change secretRef" or "the storage for <uri> rejects the credential in the Secret ...: check its keys"

Cause

The add-on's secretRef points at a Secret that is missing or holds the wrong keys.

Fix

Create or correct the Secret, then deploy the model again.

"nothing exists at <uri>: check the path, and the pinned version if there is one"

Cause

The path or version in the model's URI does not exist.

Fix

Correct the URI.

"the cluster has no serving runtime named <runtime>, ..." or "the serving runtime <runtime> is disabled on the cluster: enable it, or pin another"

Cause

Runtime engine names a runtime the cluster does not provide.

Fix

Pick a runtime the message lists, or set Runtime engine back to Derive from format.

"Cluster has no capacity for that GPU", "The GPU node pool cannot launch that GPU", or "No node group offers that many accelerators on one node"

Cause

No node group or GPU pool on the cluster can hold the GPU the model asks for.

Fix

Pick another GPU on the Compute step, or add a node group that carries it. See Sizing a replica.

"The cloud account's GPU quota cannot launch one replica"

Cause

The account's quota for that GPU family is below one node.

Fix

Request the quota the message names, in the cluster's region.

GPU quota too low: "Your Running On-Demand G and VT instances quota in us-west-2 is 8 vCPUs with 8 in use; this model needs 4 vCPUs. Request a quota increase." With a quota of 0: "... is 0 vCPUs; this model needs 4 vCPUs."

Cause

A new model needs more of the cloud account's GPU quota than is left in the cluster's region. AstroPulse reads the live quota, so the deploy is refused instead of waiting for a node that never launches.

Fix

Select Request a quota increase on the notice, which opens your cloud's quota page, then deploy again.

GPU quota may be too low: "... is 8 vCPUs with 8 in use; this model needs 4 vCPUs. A node of the type this model takes is already running, so this goes ahead; with no room there, it waits for quota."

Cause

A warning: the quota is short, and a running node may hold the model.

Fix

Nothing if the model fits on the running node. Otherwise request the increase.

GPU quota not checked: "The cloud account's GPU quota could not be checked: the read timed out. The replica still deploys; if the quota is too low its node never launches." On a registered cluster: "The cloud account's GPU quota is not checked on a registered cluster."

Cause

A warning: AstroPulse could not read the quota.

Fix

Check the quota in your cloud's console.

"source.model.resources.accelerator.gpu.capacityType asks for spot, and the Karpenter GPU node pool astro-gpu-nodepool on cluster <cluster> buys ondemand. ..."

Cause

The GPU pool does not buy the capacity type the model asks for.

Fix

Pick what the pool buys or Any on the Compute step, or add the type to the pool. See Sizing a replica.

"model.resources.accelerator.gpu.vendor is "amd", and the kserve-vllmserver runtime runs on nvidia GPUs only. ..."

Cause

The vLLM runtime runs on NVIDIA GPUs only.

Fix

Pick an NVIDIA GPU, or None, CPU model.

A weight-cache refusal, such as "source.model.cache is enabled, but this model runs on cpu-work, platform, system on cluster <cluster>, which the model-serving add-on's modelCache.nodeGroups (astro-gpu-nodepool) does not list. ..."

Cause

The weight cache cannot be kept where this model runs.

Fix

See Caching a model's weights.

"Model-serving add-on is not installed" or "Model-serving add-on is not running yet"

Cause

Cache weights on the node is on, and it needs the Model Serving add-on running.

Fix

Install the add-on from the cluster's Add-ons tab, or wait until it reads running, then deploy again.

"source.model.serving.kvCacheSize <value> fills source.model.resources.memory <value>, leaving no room for the runtime or the model: raise the memory or lower the cache"

Cause

On CPU the KV cache comes out of Memory, and this one takes all of it.

Fix

Raise Memory or lower KV cache. See KV cache.

"model.scaling.minReplicas must be at least 1: scaling to zero needs a request-driven wake-up path AstroPulse does not install yet, ..."

Cause

A model always keeps at least one replica.

Fix

Set the min under Replicas to 1 or more.

"Queue depth scaling needs the workload-autoscaling add-on"

Cause

Scale on is Queue depth, and the cluster has no Workload Autoscaling add-on.

Fix

Install it from the cluster's Add-ons tab, or choose Automatic or CPU utilisation.

"uri is fixed once the application has served: its weights may be cached and shared with other applications, and a running model is not moved to another artifact or runtime. Create a new application to serve it differently"

Cause

An update changes the model, its format or its runtime.

Fix

Deploy the new model as a new application, move callers, then delete the old one.

Refused at deploy

The status becomes Failed, and the reason gives the title, the problem and the fix.

"Authenticated endpoints need the gateway add-on"

Cause

The model names an access policy, and the cluster exposes through NGINX Ingress, which cannot enforce one.

Fix

Install the Gateway add-on. See Exposing a model.

"Ingress exposure needs the nginx-ingress add-on", or "No usable way to expose this model on the cluster"

Cause

The model has external access, and the cluster has nothing to serve it through.

Fix

Install the add-on the fix names, then deploy the model again.

"No certificate issuer for this cluster", "Certificate issuer is not ready", or "Certificate issuer has not been observed yet"

Cause

An https:// endpoint needs a certificate issuer the cluster reports ready.

Fix

Fix the issuer on the Certificates add-on. See Exposing a model.

"Hugging Face token can't be read"

Cause

The organization's stored token cannot be read.

Fix

Revoke the token and connect it again on Integrations > Hugging Face.

"Cluster agent is too old to read this artifact"

Cause

The artifact needs a credential, and the cluster's agent cannot deliver one.

Fix

Update the cluster's agent: astroctl infra k8s register agent update --cluster-name <cluster>.

Not serving after deploy

The status carries the notice The application isn't serving, with the cause and a next step. A model that cannot start reads Can't start. If no replica becomes ready within 10 minutes, the status turns Failed with the notice Deployment failed.

Waiting for a cluster operation: "AstroPulse is updating node group cpu-work on this cluster. Your application is accepted and deploys when that update finishes."

Cause

A cluster update is running, and the model deploys after it.

Fix

Nothing: the model deploys when the cluster update finishes.

"The model is waiting for its workload queue to admit it. Its metrics show how the queue stands."

Cause

The model names a queue, and other workloads use its quota.

Fix

Wait, or free quota in the queue. Metrics shows the queue, for example "Replicas admitted 0 / 1 waiting for quota". The model starts within minutes of quota freeing. See Queueing.

"The model is waiting for a node it can run on: a GPU node the cluster couldn't launch, or no room on its nodes. Its events say which." On EKS and GKE it adds "If the cloud keeps refusing the GPU, add a node group with that GPU on reserved capacity (instanceType reserved or capacityBlock) with a cluster update; the model runs there first." (on GKE naming only instanceType reserved)

Cause

No node has room, or the cloud would not launch one. A Spot model with no Spot capacity, or memory or CPU larger than the GPU pool's instances, both read like this. A Spot model does not fall back to On-Demand.

Fix

Open Events. "InsufficientCapacityError" means the cloud had no capacity: wait, or pick Any or OnDemand. "no instance type has enough resources" means the replica asks for more than a node has: lower Memory or CPU. See Memory on a GPU instance.

Can't start with "Container storage-initializer, which prepares the model before it serves, keeps running out of memory at its memory limit of 1Gi; the application's events say why."

Cause

The weight download ran out of memory.

Fix

The download retries by itself. If the status stays, read Events and Logs.

Can't start, then Deployment failed, with "The model artifact could not be downloaded: Cannot access gated repo for url https://huggingface.co/..."

Cause

The model is gated on Hugging Face, and the account behind your organization's token has not accepted its terms. Check model catches this unless Hugging Face does not answer in time.

Fix

Accept the terms with that account (see Gated models), then select Apply the application again.

Can't start with "The model keeps exiting with code 1 and restarting; its logs say why." and a notice "Nothing serves yet because container kserve-container keeps exiting and restarting (CrashLoopBackOff): its last run exited with code 1 (Error); it printed: ..."

Cause

The runtime started and stopped with an error. The notice quotes the runtime's last line.

Fix

Select Read the crashing container's logs, or open Logs, and fix what it names. A line naming max_model_len is the context length: see Context length.

"The model's serving runtime does not exist on the cluster: No runtime found to support specified framework/version. Use a runtime the cluster's model-serving add-on provides."

Cause

The model was deployed while the Model Serving add-on was still installing, or its runtime was removed.

Fix

Wait until the add-on reads running on the cluster's Add-ons tab, then deploy the model again.

"The model can't start: its KV cache is too small for its context. ..."

Cause

The KV cache cannot hold the model's context.

Fix

Do what the message says, and update the model. See KV cache.

"... ran out of memory at its memory limit of ..." or "... keeps running out of memory at its memory limit of ..."

Cause

The replica used all its memory.

Fix

Raise Memory, or lower KV cache or Context length.

"The model artifact could not be downloaded: <cause>" followed by "Correct the uri and apply the application again."

Cause

The artifact is not at the model's URI.

Fix

Correct the URI, then deploy again.

"The model artifact could not be downloaded: <cause>" followed by "If the model is gated, accept its terms on Hugging Face ..."

Cause

A Hugging Face download was refused.

Fix

See Gated models.

"The model artifact could not be downloaded: <cause>" followed by "Check the model-serving add-on's identity or secretRef, then apply the application again."

Cause

A download from storage (s3://, gs://, Azure Blob, oci://) was refused.

Fix

Fix the identity on the Model Serving add-on, then select Apply the application again. See Serve a model from your own storage.

"The model artifact could not be downloaded, and the downloader gave no cause. Its output is in the application's logs."

Cause

The download failed without a reason AstroPulse could read.

Fix

Open Logs.

"A container image of the model cannot be pulled: <reason>"

Cause

The runtime's image cannot be pulled.

Fix

Open Events, Kubernetes Events, for the registry's answer.

"The model failed to load: <message>"

Cause

The runtime started, but could not load the weights.

Fix

Open Logs for the runtime's error.

Failed: "The application has no ready replica and is not serving: <workload> has had no ready replica for longer than its <duration> progress window", or "The application's workloads are degraded and it is not serving"

Cause

The model has had no ready replica for longer than 600 seconds (10 minutes), the Kubernetes default progress window. There is no model-specific setting. The status returns to Running by itself once a replica is ready again.

Fix

Read the reason in the header and Last exit on Overview, then Logs and Events.

For a replica that is pending, crashing, or running but not ready, the Kubernetes guide Debug Pods explains each state. Kubernetes Events on the Events tab shows the same events kubectl describe pod does.

Next

• Sizing and scaling: change the GPU, memory or replicas.
• Exposing a model: endpoint states and their fixes.
• Cost: what a model costs, per hour.