GPU node groups
A GPU node group is a node group whose nodes carry the GPUs you ask for, in zones that sell them, ready for pods that request nvidia.com/gpu. Ask for it in one of two ways: name the hardware, the GPU model and how many per node, and AstroPulse finds the machines in your region that carry exactly that; or pin machine types you already know. Both work on EKS, GKE and AKS, when you create a cluster or on a running one. On EKS, the Node Autoscaling GPU pool is the other way to get GPUs: see Node groups or the GPU pool.
Before you start
- A cluster on EKS, GKE or AKS, or one you are creating. Self-hosted clusters take no GPU node groups.
- GPU quota for the GPU family in your cloud account. It starts at zero on every cloud: request it before the first real workload.
- A Kubernetes version at or above the GPU minimum, 1.36 or newer. The editor says which version the GPU needs.
Find the GPU you want
Infrastructure › Accelerators lists every GPU in a region: the vendor, model, memory per device and the per-node counts it comes in; the machine types that carry it; and the lowest Spot and On-Demand price per hour. Expand a row for each machine type's zones, host CPU and memory, and price. A dash in a price column means the catalog has no public price for it in that region, as for hardware sold through capacity reservations.
In the console
When you create a cluster, on the wizard's Node groups step (New cluster):
-
Select Add Node Group. Under Compute, pick Request GPU hardware: "Name the GPU you need; AstroPulse resolves the machines."
-
Pick the GPU model and the Devices per node. Only counts the region sells are offered. The editor shows the price per node and the cheapest machine that carries the GPU.
-
Optional: under Pinned machines, select Pin machines to limit the group to machines you choose. Each one must carry the GPU above.
-
Set Nodes, Pricing (On-Demand or Spot) and Name. Under Availability Zones, pick the zones that sell the GPU.
Open zoom view -
Fix anything the editor flags in place: a zone that does not sell the hardware, a pinned machine that does not carry the GPU, or a Kubernetes version below the GPU minimum.
To pin machines without naming a GPU, pick Pick machine types and Choose machines; each machine's GPU shows in the picker's GPU column.
On a running cluster, open the cluster's Update tab, select Change node groups, then Add node group. The same editor and checks apply.
You should see
The dry run on Review lists the machines the request resolved to under Machines AstroPulse chose. Once the cluster or the update finishes, the cluster's Node groups tab lists the group with its nodes, and kubectl get nodes shows them with nvidia.com/gpu capacity.
Zones
GPU machines are often sold in only some zones of a region. A node group places nodes in every zone it occupies, so a GPU group in zones that do not sell the hardware is refused before anything is created, and the refusal names the zones that do. Set the group's Availability Zones (availabilityZones) to those; the rest of the cluster keeps its zones.
Run a workload on the GPUs
Nothing needs installing by hand: EKS boots accelerated images, and GKE and AKS install the NVIDIA driver and device plugin, so the GPUs are schedulable when the node joins. A pod asks for GPUs the standard way, and tolerates the group's taint if it has one:
resources:
limits:
nvidia.com/gpu: 1
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
A model application asks with resources.accelerator.gpu instead. Its capacityType (OnDemand, Spot or Any, default Any) keeps replicas on nodes bought that way; OnDemand never runs on Spot and counts reserved capacity and Capacity Blocks.
Cost
- Scale to zero: a group with
minNode: 0holds no nodes while idle where an autoscaler watches the group: GKE, and AKS except its first node group. EKS managed node groups are fixed capacity, andminNode: 0is refused there. For elastic GPUs on EKS, use the Node Autoscaling GPU pool orautoMode.enableGpuNodePoolon EKS Auto Mode. - Spot: costs less, but the cloud can take a node back at short notice. Use it for work that can restart, and keep serving on On-Demand in its own group.
- Spend per cluster and application is under Cost and metrics.
Node groups or the Node Autoscaling GPU pool (EKS)
| GPU node group | Node Autoscaling GPU pool | |
|---|---|---|
| Set with | nodeGroups[].accelerator or machineTypes | enableGpuNodePool and gpuLimit on the Node Autoscaling add-on |
| Size | Fixed at the number of nodes you set | 0 nodes while idle, up to gpuLimit GPUs |
| Instance | The machine types you pinned, or those the model resolves to | Any NVIDIA g or p instance of generation 4 or later that fits, limited by allowedAccelerators |
| Bought as | instanceType: OnDemand, Spot, Reserved or CapacityBlock | On-Demand by default, Spot when gpuCapacityTypes lists it |
| Best for | Serving that must stay up | Bursty serving, batch and experiments that should cost nothing idle |
How the pool picks an instance, and how to debug it: Node Autoscaling.
Other ways in
- Nova: ask, for example, "add a node group with one L4 per node to eks-prod". Nova shows the change and applies it after you approve it.
- SDK and API: the same
acceleratorfield on a node group in the cluster manifest. See the API reference.
With astroctl
List a region's GPUs. The MODEL column is what gpu.model takes, and PER NODE lists the valid countPerNode values:
astroctl cloud aws accelerators list -r us-west-2
astroctl cloud gcp accelerators list -r us-central1
astroctl cloud azure accelerators list -r eastus
A node group under spec.clusterSpec.dataPlane.nodeGroups, by GPU model:
apiVersion: platform.astropulse.io/v1
kind: K8sCluster
spec:
clusterName: eks-gpu
provider: aws
region: us-west-2
clusterSpec:
dataPlane:
nodeGroups:
- name: system
minNode: 1
maxNode: 1
machineTypes: [m6i.large]
instanceType: OnDemand
- name: gpu-pool
minNode: 1
maxNode: 1
instanceType: OnDemand
accelerator:
gpu:
model: l4 # the catalog's model name
countPerNode: 1 # a count the catalog lists for it
taints:
- key: nvidia.com/gpu
value: "present"
effect: NoSchedule
provisioner:
type: eks
eks:
accountId: "123456789012"
credentials:
type: Dynamic
123456789012 stands for your AWS account ID. Dry-run it, then apply it. The dry run reports which machine types the request selects; the same fields in a cluster update add the group to a running cluster, with the same checks.
astroctl infra k8s apply -f eks-gpu.yaml --dry-run
astroctl infra k8s apply -f eks-gpu.yaml
Settings
accelerator.gpu.modelRequiredThe GPU each node carries.
l4, a100, h200 or b200.astroctl cloud <aws|gcp|azure> accelerators list -r <region>accelerator.gpu.countPerNodeRequiredHow many of that GPU each node carries.
accelerator.gpu.vendorThe GPU's vendor, when the model name alone is ambiguous.
nvidia or amd.machineTypesMachine types the group may use. With accelerator, every pinned machine must carry exactly the requested GPU. On eks a group draws from every listed type; on gke and aks a group runs exactly one.
A fractional GPU, a slice of a device such as the AWS g6f sizes, shows as fractional in the catalog and cannot be requested by model: pin the machine type, for example machineTypes: [g6f.large].
availabilityZonesThe zones the group places nodes in. Set it to zones that sell the GPU.
instanceTypeHow the nodes are bought.
OnDemandOn-Demand Uses capacity at the standard price, for as long as the nodes run.SpotSpot Uses discounted capacity that the cloud can reclaim at short notice.ReservedReserved Uses capacity reservations your account holds, then on-demand capacity.CapacityBlockCapacity Blocks for ML Uses only an EC2 Capacity Block for ML that your account holds.
OnDemand