Skip to main content

Watch metrics and cost

Alice, the platform engineer, turns on cost-visibility and accelerator-metrics on eks-prod once, then reads the Metrics, Cost and Security tabs of the model qwen-chat.

Before you start​

  • qwen-chat (your model deployment's name) runs on a GPU and has some traffic.

1. Alice installs the add-ons​

Path: Infrastructure > Kubernetes > eks-prod > Add-ons (eks-prod is your cluster's name)

  1. Install cost-visibility, unless it is already healthy. Pricing: keep Published prices, or pick Your own rates for figures closer to your invoice.
  2. Install accelerator-metrics.
  3. Install vulnerability-scanning, so the Security tab shows a scan result.
Open zoom view

eks-prod Add-ons tab: cost-visibility and accelerator-metrics healthy

eks-prod Add-ons tab: cost-visibility and accelerator-metrics healthyeks-prod Add-ons tab: cost-visibility and accelerator-metrics healthy
Published prices on EKS

The default prices eks-prod from AWS published prices, plus the control plane and the NAT gateways AstroPulse created with their public IPv4 addresses. They leave out your discounts, data transfer and NAT data processing. Cost is per model per closed hour, so the first figure appears after the first full hour; it is not per token or per customer.

2. Alice checks the scaling Prometheus​

Path: Infrastructure > Kubernetes > eks-prod > Add-ons > workload-autoscaling

  • Endpoint is the Prometheus that feeds scaling. Metrics and Cost do not need it.
Open zoom view

workload-autoscaling: Endpoint

workload-autoscaling: Endpointworkload-autoscaling: Endpoint

3. Alice reads Metrics​

Path: Workloads > Models > qwen-chat > Metrics

The Load and Accelerator screenshots show oct11-cpumetric, a second model the size of qwen-chat, under a load test.

  • Speed: Request latency, Time to first token.
  • Load: Token throughput, KV-cache utilization, Concurrency.
  • Accelerator: GPU utilization, Memory, Power draw.
Open zoom view

Metrics, Speed group: Request latency, Time to first token

Metrics, Speed group: Request latency, Time to first tokenMetrics, Speed group: Request latency, Time to first token
Open zoom view

Metrics, Load group: Token throughput, KV-cache utilization, Concurrency

Metrics, Load group: Token throughput, KV-cache utilization, ConcurrencyMetrics, Load group: Token throughput, KV-cache utilization, Concurrency
Open zoom view

Metrics, Accelerator group: GPU utilization, Memory, Power draw

Metrics, Accelerator group: GPU utilization, Memory, Power drawMetrics, Accelerator group: GPU utilization, Memory, Power draw

4. Alice reads Cost and Security​

Path: Workloads > Models > qwen-chat > Cost, Security

  • Open Cost, then Security.
Open zoom view

Cost tab: what the model costs to run

Cost tab: what the model costs to runCost tab: what the model costs to run
Open zoom view

Security tab: scan result

Security tab: scan resultSecurity tab: scan result

Security reads Vulnerability scanner isn't installed until vulnerability-scanning is installed, then no scan result yet until a workload has been scanned.

A clean Security tab

It does not prove every workload was scanned.

A model's spend and traffic: $2.47 in the last 24 hours, 81 requests a second

5. Alice pins the numbers to watch​

  1. Open Nova (in the console's Ask Nova, select Open Nova) and ask: qwen-chat running and waiting requests, last hour.
  2. Select Pin on the card in the answer.
  3. Start a New chat: the card is under Pinned, with its reading time and a trend line. It reads again on a new chat once its last reading is more than 5 minutes old; only the first answer needs a question to Nova.
NovaNova
What did qwen-chat cost in the last 24 hours, and how many requests waited in the last hour?
Ask Nova

Nova reads the model's cost and metrics and answers.

Clean up​

The add-ons stay on the cluster; select Uninstall on the Add-ons tab for any you no longer need. For the full order, see Clean up.