Add GPU nodes only when needed
Alice, the platform engineer, installs Node Autoscaling on eks-prod (your cluster's name) and sets the GPU limit. She chooses the limits; AstroPulse launches GPU nodes for pending model replicas, up to her limit, and removes them when empty.
On GKE and AKS
Node Autoscaling launches GPU nodes on EKS. On GKE and AKS the cloud's own node pools scale, so you set the pool's limits there. On EKS Auto Mode, the cluster's own GPU pool settings do this.
Before you start
eks-prodon Managed node groups, with a Kubernetes version the console marks compatible with Node Autoscaling and GPU pools.- GPU quota in the cluster's region, with enough vCPU for your GPU limit. A new AWS account has none; see If your GPU quota is 0.
1. Alice opens Add-ons
Path: Infrastructure > Kubernetes > eks-prod > Add-ons
- Select Install Add-on, then Node Autoscaling.
2. Alice turns on the GPU pool
Path: eks-prod > Add-ons > Install Add-on > Node Autoscaling
- Interruption queue: keep the default. A cluster AstroPulse created already has the queue; on a registered cluster, turn it on only after deploying the
generate-karpenterstack. - Leave every cloud field empty: Service account name, IAM role ARN, Instance profile, Subnet tags and Security group tags. AstroPulse supplies them on a cluster it created; on a registered cluster, create the roles first, then enter the ARN, and set Subnet tags only for subnets other than those tagged
karpenter.sh/discovery=<cluster>. - Enable GPU node pool: on. GPU limit:
4(the most GPUs you want to pay for at once; it takes 1 to 1000). - Leave GPU capacity types, GPU node expiry and GPU node drain timeout empty: a new pool buys On-Demand (
gpuCapacityTypes: ["OnDemand"]), never expires its nodes (gpuNodeExpiry: Never), and bounds a drain at 1 hour (gpuNodeDrainTimeout: 1h). It launches NVIDIAgandpinstances of generation 4 or later. Set Allowed accelerators (such asl4, the GPU model you need) to keep larger GPUs out of reach. - AMI alias: keep the default (empty). AstroPulse pins a reviewed AMI for the cluster's Kubernetes version (
al2023@v20260818on our cluster) and moves it at each upgrade. Set your ownfamily@versiononly for another family or to hold one release, and then move it yourself. - Leave GPU capacity reservations (IDs such as
cr-0123456789abcdef0, Tags such asteam=ml) empty unless you own EC2 capacity reservations or Capacity Blocks for GPU instances (see Check GPU prices and quota). The pool fills them before On-Demand. For a recurring block, prefer tags: they keep matching a renewed reservation. - On Review, select Continue to deploy once the dry run passes, then Deploy karpenter.
The knobs and what they trade
- GPU limit: the most GPUs billed at once, 10 by default; a node counts all its GPUs. Higher absorbs bigger bursts; lower caps spend.
- Allowed accelerators: which GPU models the pool may launch. Unpinned, a model gets the cheapest GPU that fits, which may be a 16 GB T4, unless it names
resources.accelerator.gpu.model. - GPU capacity types: On-Demand by default. Adding
Spotis cheaper but nodes can be reclaimed; then setcapacityType: OnDemandon serving models. - How long an empty GPU node stays: 10 minutes by default (
gpuDisruption.consolidateAfter). Longer keeps a warm node for faster restarts; the dry run states what the wait bills. - AMI alias: a new value replaces nodes. A later apply keeps the GPU settings you left out, and its dry run says how many GPU nodes a change replaces.
3. Alice checks it is healthy
Path: eks-prod > Add-ons
- Wait for Node Autoscaling to show Deployed, with its health checks passing (for example
1/1 passing). No GPU node runs yet.
When no node appears later, see How to debug Node Autoscaling.
Install Node Autoscaling on eks-prod with the GPU pool on and a GPU limit of 4.
It shows the plan and installs it when you approve.
Clean up
To stop GPU nodes launching, select Uninstall on Node Autoscaling in the Add-ons tab. Deleting the cluster uninstalls it too. For the full order, see Clean up.