Model Serving

Model Serving (model-serving) serves AI models on your cluster. It runs KServe, and it is what the Model application source deploys onto: name a model artifact on an application and AstroPulse renders the InferenceService, picks a serving runtime from the model's format, and gives a generative model an OpenAI-compatible endpoint. Everything about one model lives on its application (Serve a model); this add-on holds what the whole cluster shares: how models read private artifacts, and where their weights may be cached. It needs Certificates.

When to use it

• You want to serve a model on your own GPUs. Without it, a Model application is refused, and you would write the runtime, its resources, health checks and endpoint yourself. With it, an application names the artifact, such as hf://Qwen/Qwen2.5-0.5B-Instruct, and gets all of that.
• The weights are private, in your bucket or registry. Set once here how the cluster's models read them, and no model carries a key: Letting models read private artifacts.
• Large weights make every restart slow and costly. Models can keep their weights on a node group's disks, so a restart on those nodes reads them locally: Weight cache settings.

You don't need it to run your own serving stack from an image or a Helm chart; only the Model source needs it.

Before you start

• A cluster on Kubernetes 1.32 or newer.
• Certificates installed.
• For private artifacts: workload identity on the cluster (Enable workload identity on your cluster) and a cloud identity that can read them, or a Secret you create.

In the console

After Version, its steps are Settings and Model cache, then Advanced, Resources and Components. Identity joins them once Cloud access is Workload identity. Review runs the dry run, and Ask Nova at the top of the steps answers about the open step.

1. Open the cluster from the cluster list, select the Add-ons tab, then Install Add-on, and pick Model Serving. Requires shows whether Certificates is installed.

2. Optional, under Model cache: pick Where model weights are cached. Not set is the default, and every replica downloads its own weights; Node disk asks for the Node groups, such as cpu-work, astro-gpu-nodepool.

3. Optional: pick Cloud access. Workload identity asks, under Identity, for the Cloud and its identity: AWS role ARN, GCP service account or Azure client ID. Secret asks for the Secret ref. None is for public models.

   Open zoom view

   Configure Model Serving: Cloud access with How it authenticates set to Workload identity (recommended), Amazon Web Services (IRSA) and the IAM role ARN

   

4. On Review, the dry run runs by itself; select Continue to deploy once it passes, then Deploy model-serving. On Change configuration, a change to cloud access redeploys the cluster's models, and Review says which.

Install Model Serving: where model weights are cached, how it authenticates

You should see

The add-on reads Deployed with its availability, model metrics and model credential delivery checks passing. With a cache or cloud access set, the Weight cache readiness panel on its Overview shows the node groups that hold weights, the Models trust subject and Cache trust subject to grant in your cloud, or the Secret it reads.

Open zoom view

Model Serving on its Overview: Deployed, with 3/3 checks passing

Letting models read private artifacts

Set once on this add-on and shared by every model on the cluster: identity binds a cloud workload identity, secretRef names a Secret you create. One or the other, never both; neither when every model is public or readable some other way. An application never sets either itself.

identity is the way: no key to store or rotate, and, on every cloud but Azure, it also covers the weight cache's own downloads. Use secretRef only where a workload identity can't be used: the artifact is in another cloud than the cluster, the cluster has no workload identity, or the store is S3-compatible and not a cloud account.

Follow the recipe for your cloud. Each covers the grant, the trust, a check and how to undo it:

• EKS: Let models read a private S3 bucket
• GKE: Let models read a private GCS bucket
• AKS: Let models read a private Blob container

More on what a model sees: Private artifacts.

Security: with identity, no key is stored on the cluster or in AstroPulse. With secretRef, AstroPulse stores only the Secret's name, never its contents; the Secret stays in your cluster. Give the identity read-only access to the model buckets and nothing else.

Trust subjects

Every model runs as a ServiceAccount named astro-model-<application>, so one trust covers the cluster's models:

• Models: system:serviceaccount:*:astro-model-*
• The weight cache's downloads: system:serviceaccount:kserve-localmodel-jobs:default

On GKE and AKS, which trust exact subjects only, bind each model's own subject. On AKS, identity also needs the label azure.workload.identity/use: "true" in serviceAccountLabels; the console adds it for you.

Choosing the AWS trust policy

The wildcard lets any ServiceAccount whose name starts with astro-model-, in any namespace, assume the role. Pick the narrowest that fits:

• StringLike on system:serviceaccount:*:astro-model-*: the cluster serves models only, and only AstroPulse creates ServiceAccounts on it. Every namespace is inside the trust.
• StringLike on system:serviceaccount:<namespace>:astro-model-*, one per namespace that serves models: a new namespace needs one policy edit.
• StringEquals on each system:serviceaccount:<namespace>:astro-model-<application>: the role reads sensitive artifacts, or other teams share the cluster. Each new model needs one policy edit. astroctl app get <application> shows its namespace.

A Secret instead

secretRef names a Secret you create in this add-on's namespace, kserve:

Terminal

kubectl -n kserve create secret generic model-store-credentials \
  --from-literal=AWS_ACCESS_KEY_ID=<key> \
  --from-literal=AWS_SECRET_ACCESS_KEY=<secret>

The keys the serving runtime reads:

• S3 or S3-compatible: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. For a custom endpoint, annotate the Secret with serving.kserve.io/s3-endpoint (and s3-region, s3-usehttps or s3-verifyssl as needed).
• Google Cloud Storage: gcloud-application-credentials.json, a service-account key file.
• Azure Blob: AZURE_STORAGE_ACCESS_KEY, or AZURE_SUBSCRIPTION_ID, AZURE_TENANT_ID, AZURE_CLIENT_ID and AZURE_CLIENT_SECRET.
• Hugging Face, for the whole cluster: HF_TOKEN, used only by an organization that has not connected its own Hugging Face integration.

Applying the add-on before the Secret exists does not fail: models wait for it, and the add-on's health says the Secret was not found. Changing the Secret's contents needs no apply: the cluster's agent keeps each model's copy in step.

KServe documents an IAM-role-bound ServiceAccount for Amazon S3, and Secrets only for Google Cloud Storage and Azure Blob. If a model on GKE or AKS can't read a private artifact with identity and the cloud binding in place, use secretRef.

Changing this on a running cluster

A cloud binds an identity when a pod is created, so a model keeps the access it started with until its pod is replaced. When an apply changes identity or secretRef, the cluster's model applications are redeployed, each restarting and reloading its model; the dry run names them first. Applying the same access again redeploys nothing. A model in the middle of its own deploy is redeployed once that finishes.

Weight cache settings

The cache keeps model weights on nodes, so a replica starting there reads them from disk instead of downloading them. Off, every replica downloads its model's weights when it starts.

• Node disk (backing: Node): each node in the listed groups keeps its own copy, in the space its disk leaves after the operating system, images and pods. No extra storage cost, but every new node downloads again, and the copy goes when the node is replaced. Not on a registered cluster, where AstroPulse doesn't know the nodes' disks.
• Shared volume (backing: Storage): refused. AstroPulse cannot bound what a download takes from a shared volume. Use Node, or leave the cache off.
• Node groups (nodeGroups): required with Node. List a GPU group or the GPU node pool (astro-gpu-nodepool) for GPU models, and a CPU group for CPU models; a CPU model can't be cached on the GPU node pool. Leave out the groups that run the cluster's own components.

The dry run says how much room the cache has on each group, for example "the cache takes what each node's own disk leaves after its operating system, images and pods on astro-gpu-nodepool (100 GB disk), at /mnt/models".

On a cluster AstroPulse has just created, a CPU node group may warn that it does not carry AstroPulse's node-group label yet, so nothing is cached on it. Serve CPU models uncached there; GPU models cache on astro-gpu-nodepool as usual.

Caching a model's weights

The add-on decides where a cache may live; each application turns it on for its own model:

YAML

source:
  model:
    uri: hf://Qwen/Qwen2.5-0.5B-Instruct:7ae557604adf67be50417f59c2c2f167def9a775
    cache:
      enabled: true
      size: 2Gi
      nodeGroup: astro-gpu-nodepool

Set size to the model's real size on disk, rounded up: the cache holds the whole model whatever size says. For a Hugging Face model, the dry run adds up the model's files and warns when size is smaller. Every field and default: Weight cache. When it pays off: What the weight cache does, and doesn't.

The cache's downloads run under a fixed account that can't carry Azure's identity label, so a cached model whose artifact is in Azure Blob needs secretRef.

Model Serving: cache weights on the GPU nodes

Upgrade and roll back

KServe builds each model's pod from the runtime images its own release ships. A Model Serving version that moves the cluster to another KServe release replaces every model's pod, and each model reloads its weights before it serves again. Plan it like a restart of every model on the cluster. The dry run and the apply say so first, naming the KServe release and the models, for example "Changing the model-serving version from <old> to <new> moves KServe to <release>, which restarts the 3 model application(s) on this cluster (bge-embed, llama, qwen-chat): ...". astroctl infra k8s addons versions <cluster> model-serving lists each version with its KServe release.

A rollback restores the add-on's earlier release without redeploying the models, so a rollback to a version with different identity or secretRef is refused (below). To change access, apply the add-on with the access you want; to return to an older version with the same access, apply with spec.version set to it.

Run it in production

• One controller. It always runs one copy, whatever Availability says. While it is down, running replicas keep serving, but a model can't start, restart or add a replica.
• The weight cache uses the node's disk. Cached weights sit on each GPU node's own disk. The space left for them is the disk minus the larger of 30 GiB or 15%: on the GPU pool's 100 GiB root volume, about 70 GiB. A replaced node starts with an empty cache.
• Upgrades restart every model. Plan them like a release: Upgrade and roll back.

Other ways in

• Nova: ask it, for example "install Model Serving on eks-prod"; Nova proposes Certificates first if it is missing.
• SDK and API: the same operations, in the API reference.

With astroctl

model-serving.yaml

apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: model-serving
spec:
  clusterName: eks-prod
  configuration:
    modelServing:
      identity:
        serviceAccountAnnotations:
          eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/model-reader
      modelCache:
        backing: Node
        nodeGroups: [cpu-work, astro-gpu-nodepool]

Terminal

astroctl infra k8s addons apply -f model-serving.yaml --dry-run
astroctl infra k8s addons apply -f model-serving.yaml
astroctl infra k8s addons get eks-prod model-serving

Settings

Its own settings, under modelServing, plus availability and resources. Every one is optional.

Where model weights are cachedmodelServing.modelCache.backing

Accepts

Node (Node disk). Storage (Shared volume) is refused.

Default

Required when the cache is set. Without a cache (Not set), every replica downloads its own weights.

See

Weight cache settings

Node groupsmodelServing.modelCache.nodeGroups

The node groups, and the GPU node pool, whose nodes hold the weights. Required with Node.

IdentitymodelServing.identity

The cloud workload identity every model, and the cache's downloads, read private artifacts as: serviceAccountAnnotations (one of eks.amazonaws.com/role-arn, iam.gke.io/gcp-service-account or azure.workload.identity/client-id) and serviceAccountLabels. In the console: Cloud access set to Workload identity, then Cloud and AWS role ARN, GCP service account or Azure client ID, or Other for Service account annotations and Service account labels. Not with secretRef.

See

Letting models read private artifacts

Secret refmodelServing.secretRef

A Secret you created in the kserve namespace (Cloud access: Secret). Not with identity.

See

A Secret instead

If it's refused

"identity and secretRef cannot be combined: identity binds <add-on> to a cloud workload identity, secretRef names a Secret it reads instead. Set one"

Cause

Both are set.

Fix

Keep identity where the cluster has a workload identity for the artifacts' cloud, secretRef otherwise.

"identity.serviceAccountAnnotations must carry a cloud workload-identity annotation: eks.amazonaws.com/role-arn (IRSA), iam.gke.io/gcp-service-account (GKE Workload Identity), or azure.workload.identity/client-id (AKS Workload Identity)"

Cause

identity has no annotation a cloud reads.

Fix

Add exactly one of them; other annotations may sit beside it.

"an AKS workload identity needs both halves: identity.serviceAccountAnnotations[azure.workload.identity/client-id] and identity.serviceAccountLabels[azure.workload.identity/use]: "true" -- the annotation alone never takes effect"

Cause

The Azure annotation without its label.

Fix

Add the label, or pick Microsoft Azure (Workload Identity) in the console, which adds it.

"identity carries a <cloud> workload-identity annotation, but cluster <cluster> is <cloud>, where it binds nothing. ..."

Cause

The identity is for another cloud than the cluster's.

Fix

Use the annotation for the cluster's cloud, or, for artifacts in another cloud, name a Secret with secretRef.

"secretRef <name> is not a valid Secret name: use lowercase letters, digits, '-' and '.', starting and ending with a letter or digit, at most 253 characters"

Cause

The Secret's name is malformed. Whether it exists is checked on the cluster.

Fix

Use the Secret's real name.

"modelCache.nodeGroups is required with backing node: list the node groups, and the GPU node pool, whose nodes hold the cache on cluster <cluster>; ..."

Cause

The cache is on with no node groups.

Fix

Enter the Node groups.

"modelCache.backing is node, but cluster <cluster> was registered, not provisioned: AstroPulse does not know its nodes' disks, so it keeps no weight cache there and an application that asks for the cache is refused. ..."

Cause

The cache on a registered cluster.

Fix

Leave the cache off and serve models uncached.

"modelCache.backing storage is not available in this AstroPulse version: ..."

Cause

A shared volume as the cache.

Fix

Use Node, or leave the cache off.

"modelCache.nodeGroups names <group>, which cluster <cluster> does not have; ..."

Cause

A group the cluster doesn't have, or doesn't have yet.

Fix

Fix the name. For a group just added, apply the cluster and try again once it is ready.

"Model Serving still serves <n> application(s) on this cluster (<applications>). Delete those applications first, or remove Model Serving with force, which skips this check; AstroPulse will preserve the CustomResourceDefinitions."

Cause

An uninstall while model applications run on it.

Fix

Delete those applications first, or uninstall with force.

"Applications using Model Serving could not be listed. Retry when the application inventory is available, or remove Model Serving with force, which skips this check. CustomResourceDefinitions remain installed."

Cause

An uninstall while AstroPulse cannot read which applications use it.

Fix

Try again shortly, or uninstall with force.

Rollback would change how models read artifacts, "Version <n> of the model-serving add-on on cluster <cluster> uses different cloud access (identity or secretRef) than the current one, and a rollback does not redeploy the models that depend on it."

Cause

A rollback to a version with other access.

Fix

"Apply the add-on with the access you want ('astroctl infra k8s addons apply'); that redeploys the cluster's models. To go back to an older add-on version without changing access, apply with spec.version set to it and the current identity or secretRef."

These come on a model application's dry run and apply, or in the add-on's health:

Credentials are not part of an application

Cause

The application sets source.model.identity or source.model.secretRef.

Fix

Remove it; access is set on this add-on.

Cluster agent is too old to read this artifact

Cause

The artifact is read with a credential the cluster's agent can't deliver.

Fix

Update the agent (Astro Agent), then apply the application again.

Weight cache cannot use this cluster's identity

Cause

A cached model in Azure Blob, read with an Azure workload identity.

Fix

Name a Secret with secretRef on this add-on, or remove cache from the application.

"Secret <name> was not found in namespace kserve"

Cause

secretRef names a Secret that doesn't exist.

Fix

Create it in kserve with the keys above. Nothing needs applying again: the waiting model restarts.

"<n> model(s) are waiting for a credential this cluster has not been given ..."

Cause

A model needs a credential and none applies: no Hugging Face integration for an hf:// model, and no secretRef.

Fix

Connect Hugging Face (astroctl integrations huggingface connect), or set secretRef.

A model whose download is denied with no message above has access that is accepted but wrong: check that the identity can read the bucket and that its trust names the trust subjects. After fixing it in your cloud, change nothing here: the download retries on its own.

Next

• Serve a model
• Gateway
• Accelerator Metrics