Skip to main content

Deploy your first model

Alice, the platform engineer, installs Certificates and Model Serving on the EKS cluster eks-prod (your cluster's name), then deploys the model qwen-chat internal-only.

Before you start​

  • eks-prod with nodes, on a Kubernetes version the console marks compatible with Model Serving and the selected GPU node configuration.
  • To cache weights on nodes: a node group for your models, or Node Autoscaling with its GPU pool.
  • Private bucket: an IAM role that can read it.
  • Gated model: a read-only Hugging Face token.

1. Alice installs Certificates​

Path: Infrastructure > Kubernetes > eks-prod > Add-ons > Install Add-on

  1. Select Certificates.
  2. Install CRDs: on.
  3. On Review, select Continue to deploy once the dry run passes, then Deploy cert-manager.
Model Serving requires Certificates.Open zoom view

Model Serving requires Certificates.

Installing Certificates on eks-prod with the Install CRDs switch onInstalling Certificates on eks-prod with the Install CRDs switch on

2. Alice installs Model Serving with the cache​

Path: Infrastructure > Kubernetes > eks-prod > Add-ons > Install Add-on

  1. Select Model Serving.
  2. Where model weights are cached: Node disk.
  3. Node groups: a CPU node group for a CPU model; a GPU node group, or astro-gpu-nodepool with Node Autoscaling's GPU pool, for a GPU model. A cached model runs on one of these, so leave out system and platform, which run the cluster's own components.
  4. Cloud access: None.
  5. On Review, select Continue to deploy once the dry run passes, then Deploy model-serving.
One setting turns on the cache.Open zoom view

One setting turns on the cache.

Installing Model Serving, showing Requires Certificates installed, Where model weights are cached set to The local disk of the listed node groups, and Cloud access with How it authenticates set to Nothing extraInstalling Model Serving, showing Requires Certificates installed, Where model weights are cached set to The local disk of the listed node groups, and Cloud access with How it authenticates set to Nothing extra
Cache or download

The cache lets a new replica start from the node's disk. Left Not set, every replica downloads the weights.

3. Alice lets models read a private bucket​

Path: Infrastructure > Kubernetes > eks-prod > Add-ons > Model Serving > Configure

  1. Cloud access: Workload identity.
  2. On the Identity step, Cloud: Amazon Web Services.
  3. AWS role ARN: arn:aws:iam::123456789012:role/model-reader (the ARN of your own IAM role, with your AWS account ID).
  4. In your cloud IAM, let the role be assumed by the two trust subjects the console names, system:serviceaccount:*:astro-model-* (every model on this cluster) and system:serviceaccount:kserve-localmodel-jobs:default (the weight cache).
  5. On Review, select Continue to apply once the dry run passes, type model-serving, then select Update model-serving.
Identity keeps any key off the cluster.Open zoom view

Identity keeps any key off the cluster.

Configuring Model Serving at Cloud access: How it authenticates set to Workload identity (recommended), Cloud set to Amazon Web Services (IRSA), and the IAM role ARN filled inConfiguring Model Serving at Cloud access: How it authenticates set to Workload identity (recommended), Cloud set to Amazon Web Services (IRSA), and the IAM role ARN filled in
Applying restarts models

Applying it redeploys every running model on the cluster.

4. Alice connects a Hugging Face token​

Path: Integrations > Hugging Face

  1. Select Connect token.
  2. User access token: paste the hf_ token.
  3. Select Connect.
One token for the organization.Open zoom view

One token for the organization.

The Hugging Face integration page with the User access token field, the Connect button and a token row showing ConnectedThe Hugging Face integration page with the User access token field, the Connect button and a token row showing Connected

Revoke withdraws the token from clusters within about ten minutes.

5. Alice installs the scaling add-ons​

Install these before the first deploy

A model's queue cannot change once it runs, and moving its scaling signal later means extra applies or a redeploy.

Path: Infrastructure > Kubernetes > eks-prod > Add-ons > Install Add-on

  • cost-visibility, for cost per model.
  • workload-autoscaling (KEDA). Endpoint: use the Prometheus the console suggests. Without it, a model on a GPU stays at a fixed replica count.
  • Optional: workload-queueing, with a queue and a GPU quota, so models or teams sharing GPUs get a quota instead of competing for them. See Workload Queueing.

6. Alice deploys qwen-chat, internal only​

Anyone with the Editor role can deploy a model too, with no AWS credentials and no cluster-admin access.

Path: Workloads > Models > Deploy model

  1. Name qwen-chat (a name for your model deployment), cluster eks-prod.
  2. Model step: Hugging Face, Qwen/Qwen2.5-0.5B-Instruct (the Hugging Face model you want to run), Check model. Weights can also come from s3://, gs://, an Azure Blob address, pvc:// or oci://.
  3. Compute step: CPU for this first deploy, or the GPU.
  4. Optional: Cache weights on the node, Artifact size 2Gi (the size of your model's weights); Workload queue, if you installed one.
  5. Scaling step: Replicas 1 to 2, Scale on left on Automatic. For a generative model whose queue Workload Autoscaling can read, that means requests waiting, 2 per replica: AstroPulse adds a replica when more wait, up to your maximum.
  6. Access step: leave Internal only (default).
  7. Review, then Deploy.
Check the model before it runs.Open zoom view

Check the model before it runs.

The Model step of Deploy model with Hugging Face chosen, Qwen/Qwen2.5-0.5B-Instruct entered and Check model showing the platform can serve this modelThe Model step of Deploy model with Hugging Face chosen, Qwen/Qwen2.5-0.5B-Instruct entered and Check model showing the platform can serve this model
Compute and cache.Open zoom view

Compute and cache.

The Compute step with None, CPU model chosen and Cache weights on the node ticked with Artifact size 2GiThe Compute step with None, CPU model chosen and Cache weights on the node ticked with Artifact size 2Gi

qwen-chat turns Running in the Models list, reachable only inside the cluster. Cache weights on the node is disabled, with a reason, when Model Serving has no cache; a weight cache is refused on a registered cluster, because AstroPulse does not know its nodes' disks. A model that fails reports one of ArtifactNotFound, ArtifactAccessDenied, ArtifactDownloadFailed, ImagePullFailed, Unschedulable, RuntimeNotFound or ModelLoadFailed.

Pin the model address

Use a Hugging Face commit, an S3 version or a GCS generation. The cache is shared and keyed by it, and a rollback returns the same bytes.

What one replica is

Generative models run on one engine (vLLM), and format and runtime are fixed once a model has served. Each replica runs on one node's GPUs, at most 8; there is no multi-node serving. Engine settings beyond context length and GPUs per replica go in a serving runtime of your own on the cluster, named under Advanced > Runtime engine.

NovaNova
Install Certificates and Model Serving on eks-prod.
Ask Nova
Deploy Qwen/Qwen2.5-0.5B-Instruct from Hugging Face as qwen-chat on eks-prod, internal only, with 1 to 2 replicas.
Ask Nova

Ask them in this order. Each shows the plan and runs when you approve.

Save download cost

Every new node downloads the model again, and on a private cluster those bytes go through the NAT gateway, which the cloud bills per GB. On an EKS cluster AstroPulse creates, a model at an s3:// address or an oci:// image in your ECR, in the cluster's region, skips that charge: see Cut model download cost.

Clean up​

Delete qwen-chat first: on its page select Delete, type the name, then Delete Permanently. The add-ons stay; select Uninstall on the Add-ons tab for any you no longer need. For the full order, see Clean up.

Next: Put the model on your domain