Manage add-ons

Once an add-on is installed, its page on the cluster's Add-ons tab is where you run it: its health, its events and logs, its settings, its upgrades, rollbacks and removal. Every change goes through a dry run first, and the console, astroctl, Nova and the API make the same request, so they are checked and refused the same way. To install one: Install add-ons.

Before you start

• An installed add-on on a cluster that reads Ready. One operation runs on a cluster at a time, so a change waits while the cluster is being created, updated or deleted.

In the console

1. Open the cluster from the cluster list and select the Add-ons tab. The list shows each add-on's Version, Health, Availability and Status; Needs attention and Update available narrow it. Select a row to open the add-on.

   Open zoom view

   The Add-ons tab of eks-prod with Node Autoscaling deployed

   

2. The add-on opens as its own page, on Overview: its version, health checks, availability, resources and the configuration as applied. Ask Nova and Uninstall sit at the top, and All add-ons or the cluster's name in the path goes back to the list. Events, Logs, History, Metrics, Cost, Security and Update are beside it. Update lists Change configuration, Resync, Upgrade (when a newer release is offered) and Roll back, each with its astroctl command.

   Open zoom view

   Certificates on its Overview, with each health check passing

   

   Open zoom view

   The Update tab of Ingress: Change configuration, Resync and Roll back, each with its astroctl command

   

3. To change its settings, open Update and select Change configuration. The same steps as the install open, without Add-on and Version; change what you need, then go to Review. The dry run runs by itself and answers with the configuration the add-on would run, every setting you left out filled in as the platform reads it. Review compares that with what runs now: each change under its heading as "Setting: before → after", the diff, and the Manifest disclosure with the astroctl lines. When nothing would change, it says so and there is nothing to apply. While you fill in the steps, Your changes so far shows your own edits. Select Continue to apply, type the add-on's name on Apply (such as nginx-ingress), then select Update nginx-ingress.

   Open zoom view

   The review of a change to Ingress: the setting that changes, and the dry run that passed

   

4. To move to a newer release, select Upgrade on Update. The confirm names the release it moves to, any releases that follow one at a time, and any breaking changes; select Upgrade.

5. To build it again from its saved settings, select Resync on Update, then Resync in the confirm. While a change is being applied, Update offers only Roll back until it finishes.

6. To go back to an earlier revision, open History and select Roll back on a revision marked eligible.

   Open zoom view

   Ingress History: three versions, the current one and two superseded, each with Roll back

   

7. To remove it, select Uninstall at the top of its page, type the add-on's name, then Uninstall Permanently. An add-on marked Managed with cluster has no Uninstall: it leaves with the cluster.

You should see

An add-on's status reads the way an application's does: Running once its workload is up, and Deploying or Updating before it has one. Its lifecycle status (Deploying, Updating, Deployed) sits beside it on its page, and Events shows each step. A failing health check reads Health check failed and shows on Overview under a notice headed What is wrong; the list's Needs attention filter shows it. A version leaving support shows as its own mark beside the status.

Cloud access for add-ons

Three add-ons call your cloud from inside the cluster:

Add-on	Namespace	Service account	Needs
DNS Automation	external-dns	external-dns	Change records in your DNS zone
Certificates	cert-manager	cert-manager	Add and remove challenge TXT records in your zone, only with the DNS check (dnsSolver)
Vulnerability Scanning	astro-security	trivy-operator	Read images in your private registries

Each runs as one Kubernetes service account. Your cloud trusts that account and allows it what the add-on does, and nothing else. Other add-ons call no cloud API, or say what they reach on their own page, as Node Autoscaling and Model Serving do.

Cluster	DNS Automation	Certificates (DNS check)	Vulnerability Scanning
EKS, created by AstroPulse	Automatic. A Route 53 role is created with the cluster	Automatic for a Route 53 zone	Automatic. ECR pull access
GKE, created by AstroPulse	You	You	Automatic. Artifact Registry read in the cluster's project
AKS, created by AstroPulse	You	You	Automatic. AcrPull on the registries the cluster already pulls from
Registered, any cloud	You	You	You, for registries the workloads' own pull secrets do not cover

Where it is yours to set up, DNS Automation and Certificates' DNS check are refused until you name an identity (DNS Automation also takes a Secret), except DNS Automation on a registered cluster outside AWS, which installs and publishes nothing until it has access. Vulnerability Scanning installs and scans public images; images its registry refuses are reported as not scannable in its health.

identity binds a cloud workload identity to the add-on's service account with the annotations (and on Azure, labels) your cloud reads:

Cloud	identity
EKS	serviceAccountAnnotations: eks.amazonaws.com/role-arn: ROLE_ARN
GKE	serviceAccountAnnotations: iam.gke.io/gcp-service-account: GSA_EMAIL
AKS	serviceAccountAnnotations: azure.workload.identity/client-id: CLIENT_ID, and serviceAccountLabels: azure.workload.identity/use: "true". Both are required

secretRef, on DNS Automation only, names a Secret you create in the add-on's namespace, for a zone an identity can't reach. Set one or the other, never both. An identity you write always wins over an automatic one. On GKE the cluster needs Workload Identity, and its node pools the GKE_METADATA workload metadata mode; clusters AstroPulse created have both. Each add-on's page has the steps: DNS Automation, Certificates, Vulnerability Scanning.

Security: with identity no key exists anywhere, so there is nothing to store, leak or rotate. AstroPulse stores only the identity's annotations or the Secret's name, never reads the Secret, and never lends a credential of its own to a cluster it did not create.

Availability and resources

Every add-on takes two settings beyond its own, shown in its steps as Availability (on Advanced) and Resources:

• availability: Standard or High, how many copies of its components run. Under Standard, the components whose outage stops traffic or blocks changes already run 2 copies: the ingress controller, the Certificates webhook, Workload Autoscaling's metrics API, the gateway proxy and the Workload Queueing controller. High adds a second copy of the rest. Two copies run only when the cluster has 2 or more fixed nodes they can be scheduled on; nodes an autoscaler adds don't count. A registered cluster runs one copy, and two with High, without a disruption budget.
• resources: the CPU and memory of the add-on's main component, as requests and limits the way Kubernetes sizes any workload. components sizes the add-on's other components by name. See Advanced: sizing.

A disruption budget and a spread across hosts are applied only when two copies run, so a budget never blocks a node drain.

Clusters without a cloud load balancer

The whole flow works on a local cluster such as kind, or on bare metal:

• Ingress: turn on Enable node port. addons get then reports the node ports and node addresses to dial.
• Gateway: set gateway.enableNodePort: true.
• Certificates: use the self-signed issuer type. It needs no ACME account or public DNS.
• DNS Automation: skip it. It needs a real cloud DNS zone.

Upgrade and roll back

Upgrade appears when a newer release is offered. It upgrades in place with the current settings, one release at a time. Read the breaking changes the confirm lists before moving an add-on that carries live traffic or CRDs. With astroctl, list the versions, then set spec.version in the manifest and apply it again:

Terminal

astroctl infra k8s addons versions <cluster> cert-manager
astroctl infra k8s addons history <cluster> cert-manager
astroctl infra k8s addons rollback <cluster> cert-manager --to <revision> --dry-run
astroctl infra k8s addons rollback <cluster> cert-manager --to <revision>

--to takes a revision from history, not a version. Without --to, rollback goes to the previous eligible revision.

Version support

• The two newest minor versions of each add-on are supported, and within a minor only its newest patch.
• A version leaves support with notice: it is marked deprecated with an end-of-support date at least 30 days later for a patch and 90 days later for a minor.
• AstroPulse never upgrades or removes a running add-on for you. Retiring a version only stops new installs and changes at it. Rolling back to an earlier revision is never refused, so you can always recover.

Version is	An apply	Offered as an upgrade
Recommended	Installs	Yes
Supported	Installs	Yes
Deprecated	Installs, with a warning naming the end-of-support date and the version to move to	No
Retired	Refused, naming the version to move to	No

The console shows a deprecated or retired version with its date and the version to move to. addons versions lists every version with its support status, and the API reports it as support on each version and on the installed add-on.

Resync

Resync builds the add-on again from its saved settings, at the version it runs, and deploys it even when nothing changed. Use it to take AstroPulse's latest fix for a running add-on without changing its version or settings, or when it reads Deployed but is not behaving. A resync follows an apply's rules: it waits for another operation on the cluster, cancels a removal still waiting, and is refused for a version that is retired or doesn't run on the cluster's Kubernetes version.

Terminal

astroctl infra k8s addons resync <cluster> cert-manager

--force also restarts a stuck sync and reapplies every resource; the console never sends it.

Uninstall

The console asks you to type the add-on's name. Uninstall is refused while another add-on depends on this one, for example Certificates while Model Serving or the Gateway is installed. Nova never uninstalls: it reports the impact and gives you the command.

Terminal

astroctl infra k8s addons delete <cluster> cert-manager

With -o json or -o yaml, delete needs --yes, because standard output is the document and nothing is left to ask on.

How to debug add-ons

Start with the add-on's status and health checks, then its events and logs:

Terminal

astroctl infra k8s addons get <cluster>
astroctl infra k8s addons get <cluster> <addon> -o yaml
astroctl infra k8s addons events <cluster> <addon> --k8s
astroctl infra k8s addons logs <cluster> <addon> --level Error --include-previous

status is the add-on's lifecycle: one of Pending, Deploying, Deployed, Updating, Failed, Deleting or Deleted. phase is what its workload is doing now, in the same words as an application's (Running, InProgress, Failed and so on), and is absent before the add-on has a workload. displayStatus is set only when a notice changes what to show, such as Health check failed. Health is separate: health.overallStatus, and health.checks[], each with a name, passing, pending and a message that says what to do. With -o json, match on status or phase in a script, never on displayStatus.

Other ways in

• Nova: ask it, for example "is cert-manager healthy on eks-prod?" or "upgrade Ingress on eks-prod". It reads the same health, events and logs, shows a change before it makes it, and applies it after you approve.
• SDK and API: the same operations, in the API reference.

With astroctl

Terminal

astroctl infra k8s addons get <cluster> nginx-ingress --manifest > nginx-ingress.yaml
astroctl infra k8s addons apply -f nginx-ingress.yaml --dry-run
astroctl infra k8s addons apply -f nginx-ingress.yaml
astroctl infra k8s addons status <cluster> nginx-ingress

apply is declarative: edit spec.version or spec.configuration and apply the same file again to upgrade or reconfigure. events, logs, metrics, cost and history read the same as the console's tabs. Every command: astroctl infra k8s addons.

Settings

The two settings every add-on takes. Each add-on's own are on its page.

Availabilityavailability

How many copies of the add-on's components run.

Accepts

Standard or High.

Default

An update keeps the current one; a new add-on runs Standard.

Advanced: sizing

AstroPulse sizes every add-on for production. Change it only when a component runs out of memory or the cluster is too small for it. In the console it is on the Resources step (the main component) and the Components step (the others): requests can be changed, and limits are shown for reference but cannot be changed there. To read an add-on's current sizing, print its manifest with astroctl infra k8s addons template <cluster> <addon> (reference) rather than copying numbers from here.

In the manifest, sizing sits under spec.configuration:

YAML

spec:
  configuration:
    resources:
      requests:
        cpu: 250m
        memory: 512Mi

CPUresources.requests.cpu

The CPU reserved for the add-on's main component.

Accepts

From 10m to 16 cores, such as 250m.

Default

An update keeps the current value; a new add-on gets AstroPulse's sizing.

Memoryresources.requests.memory

The memory reserved for the add-on's main component.

Accepts

From 64Mi, or the component's default where that is lower, to 64Gi, such as 512Mi.

Default

An update keeps the current value; a new add-on gets AstroPulse's sizing.

Limitsresources.limits

The most CPU and memory the component may use, at least its requests. Where AstroPulse sets a memory limit and you set none, the memory request is also the limit. Set it in the manifest; the console shows it for reference.

Componentscomponents.<name>.resources

The same requests and limits for one of the add-on's other components, keyed by its name. A component left out keeps its sizing; a component given as {} returns to AstroPulse's sizing.

If it's refused

A refusal changes nothing, and the dry run refuses the same way. The console's Deploy and Apply steps send the same request as astroctl infra k8s addons apply.

Apply and configure

"resources.cpu is no longer accepted; set resources.requests.cpu (and resources.limits.cpu for a limit)"

Cause

The manifest uses the old flat sizing fields. astroctl refuses it before sending anything.

Fix

Move the values under resources.requests, as in Advanced: sizing.

"resources.requests.memory for <component> of <addon> must be from 64Mi (64 MiB) to 64Gi (64 GiB), got <value>"

Cause

A size outside what the component runs with. HTTP 400 InvalidArgument; details.fields names the field.

Fix

Set a value in the range the message gives.

"These add-on settings cannot be changed after installation: <settings>. Keep the installed values, or delete and reinstall the add-on to change them"

Cause

Those settings are fixed once the add-on is installed. HTTP 409 CONFLICT.

Fix

Keep the installed values, or uninstall and install again with the new ones.

"Unsupported <addon> version; supported versions are <versions> (astroctl infra k8s addons versions <cluster> <addon>)"

Cause

spec.version is not offered for this cluster.

Fix

Pick a version from astroctl infra k8s addons versions <cluster> <addon>.

"<Add-on> <version> reached end of support on <date> and can no longer be installed or reconfigured; apply <version>"

Cause

The version is retired. HTTP 400 InvalidArgument.

Fix

Apply the version the message names.

"<Add-on> needs Kubernetes <version> or newer; this cluster runs <version>"

Cause

The cluster's Kubernetes version is below what the add-on runs on. The catalog shows the same reason before you install.

Fix

Upgrade the cluster first.

"<Add-on> is not installed on this cluster because its cloud already manages that for it"

Cause

The cluster's cloud already does what this add-on does.

Fix

Leave the add-on off.

"model-serving needs cert-manager to issue the certificates of its admission webhooks, and cert-manager is not installed on cluster <cluster>. Install it first."

Cause

An add-on that needs another is installed before it. The Gateway names cert-manager the same way.

Fix

Install the add-on it names first.

"The cluster must be ready before this add-on operation can start"

Cause

The cluster is still being created, or its create failed. HTTP 409 CONFLICT.

Fix

See How to debug cluster create, update and delete.

HTTP 409 ClusterOperationInProgress

Cause

The cluster is being created, updated or deleted.

Fix

Wait for it with astroctl infra k8s progress stream <cluster>, then apply again.

"<addon> needs an Astro agent on the cluster. Register the cluster with 'astroctl infra k8s register --cluster-name <cluster>', then retry."

Cause

The add-on reports through the cluster's agent, and there is none.

Fix

Register the cluster as the message says.

"... needs the cluster's agent to be connected, and it is not connected right now. Wait for the agent to reconnect, then retry."

Cause

The agent is installed but disconnected.

Fix

Read the Astro Agent's health, then retry.

"... needs a newer Astro agent. Run 'astroctl infra k8s register agent update --cluster-name <cluster>', wait for the agent to reconnect, then retry."

Cause

The agent is too old for this add-on.

Fix

Run the command it names.

"<addon> is already running on this cluster outside AstroPulse (<namespace>/<name>); AstroPulse installs one owner per brick. ..."

Cause

You already run cert-manager or ExternalDNS yourself. HTTP 409 CONFLICT.

Fix

Keep yours and leave the add-on off, or remove yours first.

"An application with this name already exists on the cluster and is not this managed add-on"

Cause

An application you deployed uses the add-on's name. HTTP 409 CONFLICT.

Fix

Rename or delete that application.

"The add-on operation is still running; check its status before retrying"

Cause

An earlier apply is still rolling out. HTTP 409 CONFLICT.

Fix

Follow it with astroctl infra k8s addons status <cluster> <addon>.

"Another change to this cluster's add-ons is in progress; retry when it finishes."

Cause

Another change to an add-on on the same cluster is being applied. HTTP 409, with a Retry-After header in seconds.

Fix

Wait the Retry-After time, then send the same request again.

"add-on '<addon>' is currently being deleted on cluster '<cluster>'; wait for deletion to complete before applying again"

Cause

An apply, update, resync or rollback of an add-on that is being uninstalled. HTTP 409, with a Retry-After header in seconds.

Fix

Follow the removal with astroctl infra k8s addons status <cluster> <addon>, then apply again once it is gone.

"<Dependent> depends on <Add-on>. Remove <Dependent> first, or remove <Add-on> with force, which skips this check."

Cause

Uninstall of an add-on another one needs. HTTP 409 CONFLICT.

Fix

Select Remove <Dependent> in the console, or uninstall the dependent add-on first. For example: "Gateway depends on Certificates. Remove Gateway first, or remove Certificates with force, which skips this check."

"-o json cannot ask for confirmation; pass --yes to uninstall"

Cause

delete with -o json or -o yaml.

Fix

Add --yes.

"Version <n> of the <addon> add-on on cluster <cluster> does not exist. ..." or "Version <n> is the version of the <addon> add-on already running on cluster <cluster>, so there is nothing to roll back to."

Cause

The revision is not in the add-on's history, or is the one running.

Fix

List revisions with astroctl infra k8s addons history <cluster> <addon> and pick one marked eligible.

Health and rollout

"Add-on deployment has been in progress since <time>; ..."

Cause

The new version was applied and the cluster has not reported it healthy yet.

Fix

Read astroctl infra k8s addons events <cluster> <addon> --k8s for image pulls, scheduling and probe failures.

"add-on <addon> failed to become ready; run 'astroctl infra k8s addons events <cluster> <addon>' for details"

Cause

The rollout ended without the add-on becoming healthy. Its availability check reads "Add-on availability is not confirmed healthy"; the cause is in the events.

Fix

Run the events command, then logs --level Error --include-previous. Fix the cause, then apply again.

"timed out after <duration> waiting for add-on <addon>; the deployment is still running"

Cause

astroctl stopped waiting; the rollout goes on.

Fix

Follow it with astroctl infra k8s addons status <cluster> <addon>.

"The saved add-on configuration could not be read; re-apply the add-on to restore it"

Cause

AstroPulse cannot read the configuration it stored for this add-on.

Fix

Apply the add-on's manifest again: addons get <cluster> <addon> --manifest, review it, then addons apply -f.

"<addon> is being removed. It is waiting for <object> to finish deleting; ..."

Cause

The uninstall waits for Kubernetes to release an object.

Fix

Remove what the message names, or apply the add-on again to cancel the removal.

"... this cluster's agent is too old for <capability>; update the agent to turn it on"

Cause

The cluster's agent predates what this add-on reports through it.

Fix

Run astroctl infra k8s register agent update --cluster-name <cluster>.

"... is turned off for this cluster; ..."

Cause

That reporting is switched off in the cluster's agent.

Fix

Do what the message says; it names the add-on to apply.

"... stopped with an error on this cluster; restarting the agent starts it again"

Cause

The agent's reporting for this add-on stopped.

Fix

Restart the agent's workload on the cluster, then read the check again.

"... is running on this cluster, but its recent reports did not get through; the agent keeps retrying"

Cause

Reports are not reaching AstroPulse right now; the agent retries on its own.

Fix

Check the cluster's connection. If it reads Disconnected, read the Astro Agent's health.

"... has waited longer than expected for its first data on this cluster; the agent keeps retrying"

Cause

The add-on runs, but its first report has not arrived.

Fix

Read the add-on's events and logs for why it produces no data.

"no status arrived within 1 minute; run the same command again"

Cause

A one-time status read got no answer in its minute. events says "no events arrived" the same way.

Fix

Run the same command again.

Next

• Install add-ons
• Add-ons: which one does what
• Update and upgrade a cluster