Skip to main content

Vulnerability Scanning

Vulnerability Scanning (vulnerability-scanning) scans the image behind every running workload in scope for known vulnerabilities (CVEs), rescans it as the vulnerability database changes, and reports findings by severity on the cluster's and each application's Security tab, in astroctl and in the API. It runs Trivy Operator inside your cluster; your applications don't change, and nothing leaves the cluster but the results. How to read the findings is on Vulnerability Findings.

When to use it​

  • You run images you did not build, such as public base images or a vendor's. Without it, nothing tells you which running workloads carry known vulnerabilities. With it, each workload's findings show by severity.
  • A vulnerability is published for an image already running. Running images are rescanned on a schedule, so the finding appears without a redeploy.
  • You need a report for an audit or a customer. astroctl infra k8s vulnerabilities and astroctl app vulnerabilities give the same counts as the console.

You don't need it when you only need findings for images in your registry or CI pipeline, not for what is running on the cluster.

Before you start​

  • A cluster with a connected Astro Agent.
  • Images in a private registry in your cloud: read access for the scanner (Give it access to your cloud). Public images, and images your workloads pull with their own imagePullSecrets, need nothing.

In the console​

After Version, its steps are Settings, Identity, Scan resources and Scan scope, then Advanced, Resources and Components. Review runs the dry run, and Ask Nova at the top of the steps answers about the open step.

  1. Open the cluster from the cluster list, select the Add-ons tab, then Install Add-on, and pick Vulnerability Scanning.

  2. Optional: pick the Lowest severity to report (from Unknown, every severity, to Critical), and turn on Ignore unfixed.

  3. Optional: under Scan scope, the Mode (Every namespace, Only these namespaces or All but these namespaces, with the list in Include namespaces or Exclude namespaces); Include kernel header findings, Max concurrent scans, Scan frequency; and under Scan resources, each scan's Requests CPU, Requests memory, Limits CPU and Limits memory.

  4. On Review, the dry run runs by itself; select Continue to deploy once it passes, then Deploy vulnerability-scanning.

You should see​

The add-on reads Deployed with its availability, vulnerability scanning and vulnerability reporting checks passing; its Findings card says which findings it leaves out. The first scan of an image takes a few minutes, then findings appear on the Security tabs.

Open zoom view

A model's Security tab: findings by severity, and each package with its CVE, installed and fixed version

A model's Security tab: findings by severity, and each package with its CVE, installed and fixed versionA model's Security tab: findings by severity, and each package with its CVE, installed and fixed version

How scanning works​

  • When: when a workload with an in-scope image first appears, and again every Scan frequency (24h by default). Your images don't change; the vulnerability database does, so this is how quickly a new CVE shows up.
  • How: the scanner downloads each image from its registry and reads the packages inside, so it needs read access to that registry. The node's own pull permission does not reach it.
  • What: by default every namespace except kube-system and the namespaces of AstroPulse's own add-ons, including the Astro Agent's astroplatform-system.
  • A refused image is tried again the next time its workload changes, and at least every 10 hours, so access you add later is picked up with no reinstall. To scan at once, restart the workload (kubectl rollout restart deployment/<name> -n <namespace>).
  • Its cost: scanning is bursty, so it is always bounded: at most Max concurrent scans at once, each within its own CPU and memory. Lower Max concurrent scans on a small cluster. A scan killed for exceeding its memory limit reports nothing: raise the limit rather than lower it.

Choose the namespaces​

Setting Only these namespaces (scanScope.includeNamespaces) or All but these namespaces (scanScope.excludeNamespaces) replaces the default skips entirely; it does not add to them. An empty list counts as unset.

  • To scan everything, including kube-system and AstroPulse's own namespaces, skip only kube-public.
  • To keep skipping kube-system or the add-on namespaces while adding your own skips, list them too.
  • Exclude namespaces takes glob patterns, such as load-test-*.

To scan AstroPulse's own namespaces with their private images, include them, then give the scanner access to the registry that holds those images (Give it access to your cloud). Reports appear after each workload's next scan: kubectl get vulnerabilityreports -n astroplatform-system.

Give it access to your cloud​

The scanner runs as service account trivy-operator in namespace astro-security. Every grant below is read access for that one account.

  • EKS, created by AstroPulse: a role that can only pull from ECR is created with the cluster, or at its next cluster update, and used when the add-on is applied. On an older cluster, update the cluster with any change (astroctl infra k8s update <cluster> -f update.yaml), then apply the add-on again; until then the apply warns that the scanner has no registry role yet.
  • GKE, created by AstroPulse: granted when the add-on is applied: the scanner can read Artifact Registry in the cluster's project.
  • AKS, created by AstroPulse: granted when the add-on is applied: a managed identity with AcrPull on the registries the cluster already pulls from. It needs the Managed Identity Contributor role on the resource group, which the current Azure connect setup grants; run astroctl cloud azure connect again for an older connection.
  • Registered, any cloud, or a registry in another account, project or subscription: set it up yourself: EKS, GKE or AKS. The cluster needs workload identity first: Enable workload identity on your cluster. On EKS and AKS you then name the identity in the add-on's Identity (identity): pick the Cloud, then its AWS role ARN or Azure client ID. On GKE the grant names the service account directly.

Other registries (Docker Hub, GHCR, a registry you run): give the workloads that use them an imagePullSecrets entry, or put the Secret on the namespace's service account (Registry access). The scanner reads each workload's pull secrets as the kubelet does, so nothing is set on the add-on, which takes no secretRef.

An identity you write always wins over an automatic one. The rules every add-on shares: Cloud access for add-ons.

Security: the scanner reads images with a workload identity, so no registry key sits on the cluster or in AstroPulse, and your cloud can revoke it in one place. Every grant is pull access only; on EKS and GKE you can narrow it to the repositories you scan, and on AKS it is the one registry.

Images not scannable​

When a registry refuses the scanner, the add-on's health says so instead of reporting a clean result, naming up to three images. An image that could not be scanned is not an image with no findings: give the scanner read access to that registry as above, then restart the workload. Without a restart, it is retried within 10 hours.

Run it in production​

  • It keeps a volume. The shared scanner stores its vulnerability database on a 5Gi volume from the cluster's default storage class. The cluster needs a default storage class, and your cloud bills the volume while the add-on is installed.
  • Scans come in bursts. Up to Max concurrent scans run at once (10 by default, at most 20), each using up to 1 CPU and 2Gi memory: up to 10 CPU and 20Gi at the defaults. On a small cluster, lower Max concurrent scans, or the scan's CPU and memory limits.
  • Each scan downloads the image again, and the scanner downloads its database. On a cluster with private nodes that traffic goes through the NAT gateway, which your cloud bills per GB.
  • Findings live in the cluster as Kubernetes objects and are sent to AstroPulse for the Security tab.

Other ways in​

  • Nova: ask it, for example "install Vulnerability Scanning on eks-prod, scanning only the production namespace, high and critical only".
  • SDK and API: the same operations, in the API reference.

With astroctl​

vulnerability-scanning.yaml
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: vulnerability-scanning
spec:
  clusterName: eks-prod
  configuration:
    vulnerabilityScanning:
      minimumSeverity: High
      scanFrequency: "24h"
      scanScope:
        includeNamespaces:
          - production
Terminal
astroctl infra k8s addons apply -f vulnerability-scanning.yaml --dry-run
astroctl infra k8s addons apply -f vulnerability-scanning.yaml
astroctl infra k8s addons get eks-prod vulnerability-scanning
astroctl infra k8s vulnerabilities -c eks-prod

Settings​

Its own settings, under vulnerabilityScanning, plus availability and resources. Every one is optional.

Lowest severity to reportvulnerabilityScanning.minimumSeverity

The lowest severity reported: that one and every one above it. High reports High and Critical, and Unknown, the lowest, reports every severity.

Accepts
  • Unknown Unknown The scanner has no severity rating for it.
  • Low Low Rated low severity.
  • Medium Medium Rated medium severity.
  • High High Rated high severity.
  • Critical Critical Rated critical severity.
Default

Unknown

Ignore unfixedvulnerabilityScanning.ignoreUnfixed

Leaves out findings with no available fix.

Default
Off.
Include namespacesvulnerabilityScanning.scanScope.includeNamespaces

Scans exactly these namespaces. Not with excludeNamespaces.

See
Choose the namespaces
Exclude namespacesvulnerabilityScanning.scanScope.excludeNamespaces

Skips exactly these namespaces and scans the rest; replaces the default skips.

Accepts
Names or glob patterns.
Default
kube-system and AstroPulse's add-on namespaces.
Include kernel header findingsvulnerabilityScanning.includeKernelHeaderFindings

Also reports findings in the kernel headers package (linux-libc-dev) that have no fix and are below High. It can leave a very large image with no stored scan result at all.

Default
Off.
Max concurrent scansvulnerabilityScanning.maxConcurrentScans

How many scans may run at once.

Accepts
1 to 20.
Default
10.
Scan frequencyvulnerabilityScanning.scanFrequency

How often running images are scanned again.

Accepts
A duration from 1h to 720h.
Default
24h.
Scan resources: Requests CPU, Requests memory, Limits CPU, Limits memoryvulnerabilityScanning.scanResources

What each scan reserves (requestsCpu, requestsMemory) and may use (limitsCpu, limitsMemory), as Kubernetes quantities. Applying a change restarts the scanner once, so images whose scan timed out are scanned again.

Default
AstroPulse's bounded default.
IdentityvulnerabilityScanning.identity

The cloud workload identity the scanner reads private registries as: serviceAccountAnnotations and serviceAccountLabels.

See
Give it access to your cloud

If it's refused​

"minimumSeverity <value> is not a severity: use one of <severities>; that severity and every one above it are reported"
Cause
A list of severities, or a value that is not one.
Fix
Set one severity, such as High.
"scanScope.includeNamespaces and scanScope.excludeNamespaces cannot be used together"
Cause
Both namespace lists are set.
Fix
Keep one (Choose the namespaces).
"scanScope.excludeNamespaces entry <entry> is not a valid namespace name or glob pattern"
Cause
A skipped namespace is not a valid name or pattern.
Fix
Use lowercase names or patterns such as load-test-*.
"maxConcurrentScans must be from 1 through 20: scanning is bursty and must stay bounded"
Cause
Max concurrent scans is out of range.
Fix
Enter 1 to 20.
"scanFrequency must be between 1h and 720h: a shorter value risks effectively continuous rescanning, and a longer one stops being a meaningful description of continuous scanning"
Cause
Scan frequency is out of range.
Fix
Enter 1h to 720h.
"<add-on> cannot read a cloud credential from a Secret: bind a cloud workload identity with identity instead"
Cause
The manifest sets secretRef.
Fix
Remove it. Use identity, or imagePullSecrets on the workloads (Give it access to your cloud).
"the scanner has no registry role on cluster <cluster> yet, so private ECR images will be reported as not scannable; the next cluster update that changes the cluster creates it, then apply the add-on again"
Cause
A warning on an older EKS cluster AstroPulse created.
Fix
Update the cluster with a change, then apply the add-on again.
"<n> images not scannable: registry denied (<images>)"
Cause
The add-on's health: a registry refused the scanner.
Fix
Give the scanner read access to that registry, then restart the workloads (Images not scannable).
"<n> images not scannable: scan finished but its result was not stored (<images>); the usual cause is a result too large to store, which a higher minimumSeverity or ignoreUnfixed on the vulnerability-scanning add-on makes smaller"
Cause
The add-on's health: a result was too large to store.
Fix
Raise Lowest severity to report, or turn on Ignore unfixed.
"Nothing to scan: no workloads run in the namespaces the vulnerability scanner covers; deploy a workload to one of them"
Cause
The add-on's health: nothing runs in scope.
Fix
Deploy a workload there, or widen the namespaces.

Next​