Skip to main content

Cost Visibility Metrics Backend

By default Cost Visibility installs its own metrics backend next to its metering stack, and there is nothing to configure. This page is for the other case: you already run a Prometheus-compatible backend (Prometheus, Thanos, Cortex or Mimir, VictoriaMetrics) on the cluster and want Cost Visibility to read from it instead of running a second one.

When to use it​

  • You already run Prometheus on the cluster and don't want a second copy of the same series, or its volume.
  • Your metrics live in a query frontend (Thanos, Mimir) scoped to this cluster.

You don't need it when you have no backend of your own, or when the one you have holds several clusters behind one endpoint with no way to scope it to this one: keep the backend Cost Visibility installs.

Before you start​

  • Cost Visibility installed, or about to be: Cost Visibility.
  • A backend reachable from inside the cluster, with no credentials in its URL.
  • The series it must carry, including node labels from kube-state-metrics.

In the console​

  1. Open the cluster from the cluster list, select the Add-ons tab, open Cost Visibility, open Update and select Change configuration (or pick Cost Visibility under Install Add-on).

  2. Set Metrics source to Your Prometheus.

  3. Enter the Prometheus endpoint: the backend's full in-cluster URL, such as http://prometheus-operated.monitoring.svc:9090 (the Service port, not the container's). A query frontend, a proxy, a TLS endpoint or a path prefix goes in the same URL.

  4. Go to Review, select Continue to apply once the dry run passes, type cost-visibility, then select Update cost-visibility (on a new install: Continue to deploy, then Deploy cost-visibility).

You should see​

The add-on reads Deployed with its cost reporting check passing, and the cluster's Cost tab keeps filling hour by hour. Before it switches, the add-on checks the new backend; if the check fails, the previous source keeps running and nothing is lost.

Open zoom view

Cost Visibility on its Overview: Deployed, with 2/2 checks passing

Cost Visibility on its Overview: Deployed, with 2/2 checks passingCost Visibility on its Overview: Deployed, with 2/2 checks passing

What the backend must carry​

The check before the switch asks the backend for these series, and refuses it if any is missing:

  • container_cpu_usage_seconds_total and container_memory_working_set_bytes (the kubelet's cAdvisor)
  • kube_pod_owner (kube-state-metrics)
  • node_cpu_seconds_total (node-exporter)

It also refuses an endpoint whose results come from more than one cluster, rather than charging another cluster's usage to this one. A backend that holds several clusters needs an endpoint already scoped to this one.

kube-state-metrics needs node labels enabled​

Cost is attributed to nodes through kube_node_labels, and the check does not ask for it. Without it every node fails to resolve: the metering stack logs failed to find node for each one and reports no cost, although every other series is there. kube-state-metrics exports no node labels unless you allow them. The backend Cost Visibility installs allows them; a backend of yours has to.

Check it, where 0 or an empty result means it is missing:

count(kube_node_labels)

Allow node labels on the kube-state-metrics that feeds your backend, with its --metric-labels-allowlist flag:

--metric-labels-allowlist=nodes=[*]

With the community Prometheus chart, that is kube-state-metrics.metricLabelsAllowlist="{nodes=[*]}". Cardinality grows with node count, not pod count. To keep it lower, list only the region, zone and instance-type labels:

--metric-labels-allowlist=nodes=[topology.kubernetes.io/region,topology.kubernetes.io/zone,node.kubernetes.io/instance-type]

Switch back or move​

To rotate the endpoint, or to go back to the backend Cost Visibility installs, change the setting and apply again; setting Metrics source back to Managed Prometheus returns to the installed one. Any closed hour that can't be collected while neither source answers shows as missing on the Cost tab, never as $0.

Security: the endpoint takes no credentials, so none is stored in AstroPulse or on the add-on. Put an authentication proxy inside the cluster in front of a backend that needs one, and point the Prometheus endpoint at it.

Other ways in​

  • Nova: ask it, for example "point Cost Visibility on eks-prod at the Prometheus in the monitoring namespace".
  • SDK and API: the same operations, in the API reference.

With astroctl​

cost-visibility.yaml
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: cost-visibility
spec:
  clusterName: eks-prod
  configuration:
    costVisibility:
      prometheusEndpoint: http://prometheus-operated.monitoring.svc:9090
Terminal
astroctl infra k8s addons apply -f cost-visibility.yaml --dry-run
astroctl infra k8s addons apply -f cost-visibility.yaml

Remove prometheusEndpoint and apply again to return to the installed backend.

Settings​

Prometheus endpointcostVisibility.prometheusEndpoint

The in-cluster URL of your backend, shown when Metrics source is Your Prometheus. Can't be combined with the installed backend's settings (prometheus).

Accepts
A URL reachable from workloads in the cluster, with no credentials in it, such as http://prometheus.monitoring.svc:9090.
Default
Not set: Cost Visibility installs its own backend.

The installed backend's own settings (Retention, and Name and Size under Storage) are on Cost Visibility.

If it's refused​

"Configured cost source did not pass connectivity and data validation before the deadline; verify its namespace, Service name, Service port, and required metrics"
Cause
The backend was not reachable from the cluster, or one of the required series is missing.
Fix
Check the Service, port and protocol, and that the four series are there. node_cpu_seconds_total missing means no node-exporter; kube_pod_owner missing means no kube-state-metrics. Then apply again.
"Prometheus endpoint contains multiple <label> values; provide a cluster-scoped endpoint"
Cause
The endpoint returns series from more than one cluster.
Fix
Point it at a query endpoint scoped to this cluster, or keep the installed backend.
"<field> must not contain credentials or a fragment"
Cause
The URL carries a user name, password or #.
Fix
Remove them. Put an authentication proxy inside the cluster in front of the backend if it needs credentials.
"<field> must be an absolute HTTP(S) URL"
Cause
The Prometheus endpoint is not a full http:// or https:// URL.
Fix
Enter the whole URL, such as http://prometheus.monitoring.svc:9090.

Next​