Skip to main content

Cut model download cost

Every new node downloads the model's weights and the runtime's image before it serves. On a cluster with private nodes, those bytes cross the cluster's NAT gateway, which the cloud bills per GB. Keeping the weights in the cluster's own region removes most of that charge.

All prices on this page are estimates for AWS us-west-2 at list price. Check your own region's prices.

Copy the model once into an S3 bucket in the cluster's region, then deploy it from Deploy model with Object storage as the model source. On an EKS cluster AstroPulse creates, S3 traffic in the region skips the NAT gateway at no charge.

Before you start​

In the console​

  1. Copy the model into the bucket once, pinned to one commit, with the commit in the path so the address names one set of files:

    Terminal
    hf download Qwen/Qwen2.5-7B-Instruct --revision <40-character commit sha> --local-dir ./qwen
    aws s3 sync ./qwen s3://acme-models-usw2/qwen2.5-7b-instruct/<40-character commit sha>/

  2. Open Workloads > Models > Deploy model. On the Model step, choose Object storage under Model source, and paste the bucket path into Location.

  3. Open Advanced and set Format to vLLM. Select Check model.

  4. Optional, for a model that restarts often on long-lived nodes: on the Compute step, select Cache weights on the node and set Artifact size to the model's size on disk. See What the weight cache does, and doesn't.

    Open zoom view

    The Compute step with Cache weights on the node selected and Artifact size 2Gi

    The Compute step with Cache weights on the node selected and Artifact size 2GiThe Compute step with Cache weights on the node selected and Artifact size 2Gi

  5. Continue to Review and Deploy.

The same works for a model packaged as an OCI image in ECR in the cluster's region: choose Registry and pin the image by digest (oci://…@sha256:…). See Serve a model from your own storage.

You should see​

The model's Serving card shows the s3:// address as its Model URI. A new node now downloads the weights through the S3 gateway endpoint, not the NAT gateway. To confirm, see Check where the bytes went.

Where the cost comes from​

Each new node that runs a model downloads, through the NAT gateway:

WhatSizeNAT processing (est., $0.045/GB)
The GPU runtime image (vLLM)about 10.5 GBabout $0.47
The cluster's own images on that nodeabout 1 to 2 GBabout $0.05 to $0.09
The model's files, the whole Hugging Face repositoryabout 1 GB for Qwen2.5-0.5B, about 15.2 GB for Qwen2.5-7B$0.045 per GB

A replica that restarts on a node that already has the image pulls nothing again. A new node always pulls everything. With Node Autoscaling, every scale-out and every replacement of a GPU node is a new node, and on a one-GPU instance every extra replica is one.

The NAT gateways also cost by the hour, whatever they carry: a cluster AstroPulse creates on EKS has one in each zone, three by default, about $98 a month for three (est.).

The S3 gateway endpoint​

On an EKS cluster AstroPulse creates, the cluster's VPC gets an S3 gateway endpoint. It costs nothing, and S3 traffic in the cluster's region goes through it instead of the NAT gateway. That covers:

  • a model at an s3:// address in the cluster's region;
  • an image in ECR in the cluster's region, including an oci:// model image, because ECR serves image layers from S3.

It does not cover Hugging Face, Docker Hub or any registry outside your account: those still go through the NAT gateway.

The endpoint is created with the cluster and deleted with it. The cluster's create progress says when it was added, or warns, with the reason, when it could not be; the cluster works either way.

A VPC you bring is never changed. If it has no S3 gateway endpoint, the create says so. Add one in the VPC console on the route tables of the cluster's subnets, or with the CLI under With astroctl.

Bucket policies keyed on an IP address

Through the endpoint, S3 sees requests from the VPC, not from the NAT gateway's public IP. A bucket policy that allows aws:SourceIp of the NAT gateway stops matching; use aws:SourceVpc or aws:SourceVpce instead.

Worked example (est.)​

Qwen2.5-7B, 15.2 GB, started on a new node 10 times in a month:

SourceNAT bytesCost
Hugging Face152 GB$6.84
s3:// in the cluster's region0 GB after the one copythe copy, plus about $0.35 a month of S3 storage

Making the copy downloads the model from Hugging Face once, about $0.68 of NAT if you run it on a node of the cluster; the upload to the bucket goes through the endpoint. It pays off from the second start on a new node. It does not pay off for a model you start once, a model under 1 GB, or nodes that live for weeks.

What the weight cache does, and doesn't​

Cache weights on the node keeps a copy of the weights on the disks of a node group, so a replica that restarts or redeploys on a node that already holds the copy reads it from disk instead of downloading it.

  • It does not stop a new node from downloading: every node that joins the group downloads its own copy.
  • Every node of the group downloads every model cached on that group, whether a replica of that model lands there or not. Three cached models on a GPU pool means three downloads per new GPU node.
  • It does not cache the runtime image.

The cache saves on restarts and redeploys on long-lived nodes. On a pool that scales up and down, keep the weights in your region instead. See Weight cache settings.

How long an idle GPU node stays​

Node Autoscaling removes an empty GPU node after gpuDisruption.consolidateAfter, 10m by default; see Node Autoscaling.

An empty g6.xlarge costs about $0.805 an hour, about $0.13 for each 10 idle minutes (est.). A new GPU node costs about $0.50 to $1.20 of NAT for its images and weights and about 9 minutes before it serves (est.). If traffic comes in bursts with gaps of under an hour, a longer consolidateAfter can cost less than relaunching. If the model is idle for hours, keep the default.

Pin a revision​

Pin the model to one Hugging Face commit (see Pin a revision), a commit in your s3:// path, or an oci:// digest. A pinned address names the same files on every node, so a copy you made stays valid and a rollback returns the same bytes.

Check where the bytes went​

  • NAT bytes: in CloudWatch, AWS/NATGateway BytesInFromDestination for each of the cluster's NAT gateways, before and after a deploy.
  • The endpoint: in the VPC console, a Gateway endpoint for com.amazonaws.<region>.s3 on the cluster's VPC.
  • The model address in use: Model URI on the model's Serving card.

Other ways in​

  • Nova: ask it in plain language, for example "serve qwen-chat from s3://acme-models-usw2/qwen2.5-7b-instruct/<commit>".
  • SDK and API: the Python SDK and the API reference use the same manifest.

With astroctl​

model.yaml
source:
  type: "Model"
  model:
    uri: "s3://acme-models-usw2/qwen2.5-7b-instruct/<40-character commit sha>"
    format: "vLLM"
    cache:
      enabled: true
      size: "16Gi"
Terminal
astroctl app apply -f model.yaml --dry-run
astroctl app apply -f model.yaml
astroctl app get <app>                          # the model address in use
kubectl get pod <pod> -o jsonpath='{..image}'   # the image a pod pulled
# a VPC you bring: add the S3 gateway endpoint on the cluster subnets' route tables
aws ec2 create-vpc-endpoint \
  --vpc-id vpc-0123456789abcdef0 \
  --vpc-endpoint-type Gateway \
  --service-name com.amazonaws.us-west-2.s3 \
  --route-table-ids rtb-0123456789abcdef0 rtb-0123456789abcdef1
aws ec2 describe-vpc-endpoints --filters Name=vpc-id,Values=<vpc id> Name=vpc-endpoint-type,Values=Gateway

See the astroctl reference.

Settings​

All fields are under source.model.

uriRequired

Where the weights are. Put the commit or version in the path.

Accepts
for this page, s3://<bucket>/<path> in the cluster's region, or oci://<registry>/<repo>@sha256:<digest> in ECR in the cluster's region
See
Serve a model from your own storage
cache.enabled

Keeps the weights on the node's disk so a restart reads them instead of downloading. In the console, Cache weights on the node. Needs the weight cache on the cluster's Model Serving add-on. Not for pvc:// or oci://.

Accepts
true, false
Default
false
See
What the weight cache does, and doesn't
cache.size

The artifact's size on disk, rounded up. In the console, Artifact size. A size larger than the node group's cache is refused.

Accepts
a quantity above zero, for example "30Gi"
Default
required when the cache is enabled
cache.nodeGroup

The node group that holds the cached weights; every replica then runs there. Set it only when more than one cached node group could run the model.

Accepts
a node group the add-on caches on
Default
the one cached node group that can run the model

If it's refused​

The weight cache's refusals are listed under Caching a model's weights. Storage refusals are on Serve a model from your own storage.

Next​