Skip to main content

Accelerator Metrics

Accelerator Metrics (accelerator-metrics) tracks GPU usage per application. It runs the NVIDIA DCGM exporter as a DaemonSet on the cluster's NVIDIA GPU nodes, and the cluster's Astro Agent reads it directly and attributes each device to the application whose pod holds it, so no Prometheus is needed. It measures NVIDIA devices only.

When to use it​

  • You pay for GPUs and need to know they are busy. Without it, an application's metrics show CPU, memory and network only. With it, each application holding a GPU reports utilization and memory used, so an idle or oversized GPU shows.
  • A model is slower than it should be. It shows whether the GPU was throttled and why (power limit, too hot, or a hardware slowdown), and SM occupancy: high utilization with low occupancy usually means batches too small to fill the GPU.
  • A model fails or restarts on some nodes. XID faults, the GPU's own hardware and driver errors, are reported per application.
  • You are choosing a GPU size. Memory used against memory total shows whether the model fits a smaller, cheaper GPU.

You don't need it on a cluster with no NVIDIA GPU nodes, or to measure AMD, Google TPU or other accelerators, which it doesn't read.

Before you start​

  • NVIDIA GPU nodes: from an NVIDIA accelerator node group, the Node Autoscaling GPU node pool, or the EKS Auto Mode GPU node pool. Without any, the add-on installs and reports nothing.
  • The NVIDIA driver on those nodes. The GPU node images of EKS, GKE and AKS ship one; on a registered cluster, install it yourself.
  • On a registered cluster: the node label the exporter is placed by (Label GPU nodes on a registered cluster). Nodes AstroPulse creates carry it.
  • Workloads that request nvidia.com/gpu. A device is attributed only while a pod holds it that way.

In the console​

After Version, Settings says it installs with the platform's defaults, then Advanced and Resources size it. Review runs the dry run, and Ask Nova at the top of the steps answers about the open step.

  1. Open the cluster from the cluster list, select the Add-ons tab, then Install Add-on, and pick Accelerator Metrics. It has no settings of its own; Accelerator readiness shows what the cluster has for it to measure.

  2. On Review, the dry run runs by itself; select Continue to deploy once it passes, then Deploy accelerator-metrics.

You should see​

The add-on reads Deployed with its availability and accelerator metrics checks passing. On a cluster with no GPU node running, accelerator metrics passes as idle with "Nothing to check yet: no GPU nodes" and reports as soon as a GPU node starts.

Open zoom view

The cluster's Add-ons tab with Accelerator Metrics and Cost Visibility deployed, 2/2 checks passing

The cluster's Add-ons tab with Accelerator Metrics and Cost Visibility deployed, 2/2 checks passingThe cluster's Add-ons tab with Accelerator Metrics and Cost Visibility deployed, 2/2 checks passing

An application whose pods hold a device gains an Accelerator group on its Metrics tab: Devices, Utilization (with SM occupancy), Memory used, Power draw, Temperature and Throttled, charts for GPU utilization, Memory and Power draw, and a banner when XID faults were reported in the range.

Accelerator Metrics, installed: waiting for GPU nodes

What it measures​

For each application, the agent reduces the exporter's fields to one figure per window: a mean for gauges, a count for faults. An application holding no device has no accelerator figures at all: absent means no device, never zero use.

  • Devices (devices): the mean count of devices its pods held. A fraction means the replica count changed mid-window.
  • Utilization (utilization, DCGM_FI_DEV_GPU_UTIL): the share of the window the devices ran any kernel.
  • SM occupancy (smOccupancy, DCGM_FI_PROF_SM_OCCUPANCY): the share of warp slots in use. Absent on GPUs or drivers without profiling counters, or while another process holds profiling.
  • Memory used and total (memoryUsedBytes, memoryTotalBytes, DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_FB_TOTAL).
  • Power draw (powerWatts, DCGM_FI_DEV_POWER_USAGE): mean board power, summed across devices.
  • Temperature (temperatureCelsius, DCGM_FI_DEV_GPU_TEMP): mean, across devices.
  • Throttled (throttledFraction, with powerCapFraction, thermalFraction, hardwareSlowdownFraction and throttleCause): the share of the window a device was slowed by a power cap, heat, a hardware slowdown or a power brake. The power limit is normal under sustained full load.
  • XID faults (xidErrors): faults reported in the window. Any above zero is a hardware or driver fault; look the code up in NVIDIA's XID reference.

What it installs​

  • A DaemonSet in namespace astro-metrics, one pod per NVIDIA GPU node, with pod label app.kubernetes.io/name: dcgm-exporter, serving /metrics on port 9400.
  • Placement by node selector node.cluster.platform.astropulse.io/accelerator: nvidia, tolerating nvidia.com/gpu:NoSchedule and the taints declared on the cluster's accelerator node groups.
  • No Prometheus, no ServiceMonitor, no driver and no CRDs.

Security: the exporter runs privileged, as root, which is how it reaches the GPUs on every node image. It needs no cloud credential, and only per-application figures leave the cluster.

Label GPU nodes on a registered cluster​

AstroPulse didn't create a registered cluster's nodes, so they carry no AstroPulse label. Label each node that has an NVIDIA GPU and a driver:

Terminal
kubectl label node <node-name> node.cluster.platform.astropulse.io/accelerator=nvidia

The exporter starts on it within a minute or two. A label on a node without a working driver gives an exporter pod that finds no devices.

Run it in production​

  • It runs on GPU nodes only: one pod per NVIDIA GPU node, reserving 100m CPU and 128Mi memory, with a limit of 200m CPU and 512Mi. It keeps no volume and needs no Prometheus, so its cost is that small slice of each GPU node.
  • A failed upgrade rolls back on its own.

Other ways in​

  • Nova: ask it, for example "install Accelerator Metrics on eks-prod and tell me whether my GPUs are busy".
  • SDK and API: the same operations, in the API reference.

With astroctl​

accelerator-metrics.yaml
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: accelerator-metrics
spec:
  clusterName: eks-prod
Terminal
astroctl infra k8s addons apply -f accelerator-metrics.yaml --dry-run
astroctl infra k8s addons apply -f accelerator-metrics.yaml
astroctl infra k8s addons get eks-prod accelerator-metrics -o yaml
astroctl app metrics <application> --window 1h -o yaml

addons get returns the readiness in its accelerator block (measurable, declaredAccelerators, karpenterGpuPool, vendors, warning, gpuNodes). app metrics returns the range's figures in acceleratorSummary, and each point's in its accelerator block.

Settings​

No settings of its own: only availability and resources, which size each exporter pod. Availability does not change the copy count: the exporter always runs one pod per NVIDIA GPU node, and its CPU and memory come out of each of them.

If it's refused​

The install is never stopped for what the cluster lacks: the dry run and the apply warn instead.

"this cluster declares no accelerator capacity; the exporter will install and report nothing until an accelerator node group or a GPU node pool (Node Autoscaling's, or EKS Auto Mode's) exists"
Cause
No accelerator node group and no GPU node pool.
Fix
Add an NVIDIA node group, or turn on the Node Autoscaling GPU node pool.
"this cluster's only accelerator capacity is a GPU node pool; the exporter reports only while the pool has a GPU node running"
Cause
Figures appear only while the pool has a node, which it adds when a pod requests nvidia.com/gpu.
Fix
Nothing to set.
"this cluster declares <vendor> accelerators and no NVIDIA ones; the exporter reads NVIDIA devices only, so it will install and report nothing"
Cause
Only non-NVIDIA accelerators.
Fix
Nothing to measure: no add-on reads other vendors yet.
"configuration does not match add-on "accelerator-metrics""
Cause
The manifest carries another add-on's settings block.
Fix
Remove it: this add-on takes only availability and resources.

The accelerator metrics check, when GPU figures are not reaching AstroPulse. Each failing message starts with "Accelerator metrics are not reaching AstroPulse; the accelerators may be working normally, but their utilization, memory and power figures are not being collected", and names up to five nodes:

"... 1 NVIDIA GPU node does not carry the label node.cluster.platform.astropulse.io/accelerator=nvidia the GPU metrics exporter is placed by (<nodes>), ... Add that label to those nodes"
Cause
A GPU node, usually on a registered cluster, lacks the placement label.
Fix
Label it.
"... NVIDIA accelerator nodes are ready, but the GPU metrics exporter is running on none of them, ..." or "... ready but not running the GPU metrics exporter (<nodes>), ..."
Cause
The nodes carry a taint the exporter doesn't tolerate, or the pods can't start.
Fix
Check the pods: kubectl get pods -n astro-metrics -l app.kubernetes.io/name=dcgm-exporter -o wide. Declare a hand-added taint on the node group in the cluster spec and apply the add-on again, or remove it.
"... no accelerator exporter on this cluster could be read; GPU utilization and memory are not being collected" or "<n> accelerator exporter(s) could not be read; their nodes' devices are missing from this window"
Cause
Exporter pods run but don't answer, usually because the NVIDIA driver is missing or not loaded.
Fix
Read the pod's logs (kubectl logs -n astro-metrics <pod> --previous). Fix the driver, then delete the pod so the DaemonSet starts a fresh one.

Next​