Skip to main content

Troubleshoot Cost Visibility

When a Cost Visibility figure looks wrong, the cluster says why: every unpriced or partly priced figure carries a notice with a code, a title, a message and, where there is one, a next step. This page is what each one means. How to read coverage and missing hours is on Cost.

When to use it​

  • Observed spend reads Not priced, or a figure followed by "+ Not priced".
  • The total is lower than your cloud bill.
  • The add-on is healthy and there is no cost yet.
  • A configuration change doesn't take effect.

Before you start​

In the console​

  1. Open the cluster from the cluster list and select Cost. Read the notices under the cards: each names what is not priced and its next step.

    Open zoom view

    A cluster's Cost tab: observed spend, coverage, utilization, the spend trend and the cost allocation by namespace

    A cluster's Cost tab: observed spend, coverage, utilization, the spend trend and the cost allocation by namespaceA cluster's Cost tab: observed spend, coverage, utilization, the spend trend and the cost allocation by namespace

  2. Select the Add-ons tab and open Cost Visibility. The Health card shows the availability and cost reporting checks; Events shows why a change did not apply.

  3. Follow the notice's next step, such as Set a price sheet or Update the cluster.

You should see​

Once a notice's cause is fixed, the next closed hour is priced and the notice goes. Closed hours keep the price, or lack of one, they were reported with: rates added now apply from the next hour, not to history.

Cost reads Not priced​

Read the reasons from the API:

Terminal
astroctl infra k8s cost summary <cluster> --window 24h -o yaml

pricingNotes say how the cluster is priced, and an unpriced entry's unpricedReasons say what has no price. Each code is under If it's refused. A registered cluster outside AWS and Azure, or a local one, has no published prices and no notice: give it a price sheet.

The total is lower than your cloud bill​

Terminal
astroctl infra k8s cost allocation <cluster> --group-by Namespace --window 24h -o yaml
  • A notice in pricingNotes or unpricedReasons: that part is left out. Follow it.
  • complete: false: some hours were not measured. See Cost.
  • The Cluster infrastructure rows are the management fee and what AstroPulse provisioned around the cluster. No NAT gateway rows on an EKS cluster AstroPulse created means it runs in a VPC you supplied, whose gateways are yours, or its NAT gateway count is not recorded yet (CostCountNotRecorded): update the cluster.
  • Some charges are never in the total, such as data transfer, support, taxes, credits and discounts: What the total includes.
  • Published prices are list prices. With discounts, a price sheet at your real rates comes closer.

No cost yet, add-on healthy​

Expected for one to two hours. Spend is reported one closed UTC hour at a time, starting with the first hour the add-on watched from its start: installing at 10:05 or at 10:50 makes 11:00 to 12:00 the first hour, reported a few minutes after 12:00. Usage is measured from the start; only the money waits. The add-on's cost reporting check says which hour it is waiting for.

A change doesn't take effect​

Before switching to a backend of yours, the add-on checks it, and refuses rather than replace a working source with a broken one. Read why:

Terminal
kubectl -n astro-cost get jobs
kubectl -n astro-cost logs job/<the -source-preflight job>

A failed check is kept for a day; a passed one is removed. The usual cause is a missing series: What the backend must carry.

Every node reports no cost​

On a backend of yours, kube_node_labels is missing: kube-state-metrics needs node labels enabled.

Spot rates seem ignored​

Spot rates in a price sheet apply only to nodes carrying the Spot label you declared. Check what your nodes carry:

Terminal
kubectl get nodes -o json | jq -r '.items[].metadata.labels | keys[]' \
  | sort -u | grep -iE 'spot|preempt|capacity'

Other ways in​

  • Nova: ask it, for example "why is eks-prod's cost lower than my AWS bill?"
  • SDK and API: the same operations, in the API reference.

With astroctl​

Terminal
astroctl infra k8s addons get <cluster> cost-visibility -o yaml
astroctl infra k8s cost summary <cluster> --window 24h -o yaml
astroctl infra k8s cost trend <cluster> --window 7d --interval 1h

The add-on's pricing.notices carry the same notices before any hour is reported.

Settings​

This page has no settings of its own: Cost Visibility settings and price sheet settings.

If it's refused​

Each notice by its code, with the title and message the console shows.

CostListPriceUnavailable: This cluster's costs aren't priced, "The metering engine can't read Google Cloud list prices on this cluster, so its costs aren't priced. Set a price sheet on the cost-visibility add-on to price it at your rates."
Cause
Google Cloud publishes no list prices the add-on can read.
Fix
Set a price sheet: Price sheets.
CostSpotPriceUnavailable: Spot nodes are priced at On-Demand rates
Cause
The add-on can't read Spot prices on this AWS cluster.
Fix
On an EKS cluster AstroPulse created, Update the cluster, then apply the add-on again. On a registered cluster, Set the Spot pricing role: Spot prices on AWS.
CostGPUNotPriced: GPU usage isn't priced, "Usage on GPU nodes isn't priced: the price sheet sets no rate for them."
Cause
The price sheet has no gpuHour.
Fix
Set a GPU rate: GPU hour under Price sheet.
CostSpotGPUNotPriced: Spot GPU usage isn't priced, "Usage on Spot GPU nodes isn't priced: the price sheet has no Spot GPU rate."
Cause
Spot rates with no spotGpuHour.
Fix
Set a Spot GPU rate: Spot GPU hour under Price sheet (GPU rates).
CostManagementNoRate: Cluster management isn't priced, "Cluster management isn't priced: the price sheet sets no cluster management rate."
Cause
The price sheet has no clusterManagementHour.
Fix
Set a cluster management rate, or "0" if the cluster has no management fee.
CostAgentUpgradeRequired: Cluster management isn't priced, "Cluster management isn't priced: the astro-agent on this cluster is older than the price sheet's cluster management rate needs."
Cause
The Astro Agent is too old for clusterManagementHour.
Fix
Upgrade the astro-agent: Astro Agent.
CostAttributionInconsistent: Some hours can't be split by application
Cause
Applications were charged more than their namespaces in some hours, so workloads outside applications have no price for those hours.
Fix
Nothing to set. Read the namespace grouping for those hours.
CostNotMetered: Some charges aren't measured
Cause
A charge billed by a volume nobody measures, such as the data a NAT gateway processes.
Fix
Nothing to set. Add it from your cloud bill.
CostNoPriceSource: No published price for some charges
Cause
No published price for that charge in the cluster's region.
Fix
Nothing to set. Add it from your cloud bill.
CostCountNotRecorded: Some charges aren't counted
Cause
How many of the item the cluster has is not recorded, such as the public IPv4 addresses of load balancers.
Fix
On an EKS cluster AstroPulse created, update the cluster: an update records its NAT gateway count. Otherwise add it from your cloud bill.
CostPriceUnavailable: Some charges couldn't be priced
Cause
This region's published prices couldn't be read yet for those hours.
Fix
Nothing to set. Those hours are priced again on later reads.
CostPriceStale: Some charges use the last prices read
Cause
The region's latest published prices couldn't be read when the hour was stored, so charges around the cluster use the last prices read. The amount is still included.
Fix
Nothing to set.

Next​