Update and upgrade
Change a cluster AstroPulse created from its Update tab: upgrade Kubernetes, scale or change node groups, add or remove a node group, or change the settings your provider exposes and the cluster's tags. Every change goes through the same steps: you configure it, AstroPulse reads the cluster and writes a readiness report, you tick the checks it leaves to you, a dry run shows the plan, and only then is it applied. Kubernetes upgrades go one minor version at a time, the control plane first and then each node group in turn, and a control plane is never downgraded. A cluster you registered is upgraded with the tools that run it: AstroPulse updates only its agent.
Before you start
- An org admin role. Deleting a cluster needs the owner role: Delete a cluster.
- A cluster that is ready, with no other operation running on it. One operation runs on a cluster at a time.
- For an upgrade or a change that replaces nodes: room in your cloud account's quotas for the extra nodes a rolling update adds.
In the console
- Open the cluster from the cluster list and select
the Update tab. It has three cards: Upgrade Kubernetes, Node groups and Provider
settings, each with the
astroctlcommand that does the same. - Pick the change:
- Start upgrade on Upgrade Kubernetes, offered when an upgrade is available. The cluster page's header and the Overview's Review upgrade open the same flow.
- Change node groups on Node groups: "Scale a group, change its machines, or add a group." Add node group opens the same editor and checks as cluster create. On the Node groups tab, Modify on a group opens it here too.
- Edit settings on Provider settings: "The settings your provider exposes for this cluster, and its tags."
- Configure. Pick the version, the node group sizes and machines, or the settings. An upgrade also asks how node groups roll: Max Surge and Max Unavailable. Select Continue to readiness report.
- Readiness report. AstroPulse reads the cluster's health, the version change, the workload impact and the capacity a rolling update needs, and lists critical issues, warnings and recommendations. A blocking issue stops here.
- Checklist. "The checks the platform leaves to you. Tick each one to continue."
- Review plan. The dry run shows what each node group does and the rolling strategy. Nothing changes.
- Apply. Type the cluster name, then select Upgrade to v<version> or Apply changes. The upgrade warns at its "Point of no return": the control plane cannot be downgraded once upgraded.
You should see
The cluster's header shows the running operation, and the Events tab streams each step: the control plane, then each node group. Once it ends, the cluster reads Ready with the new version, and the Node groups tab shows the new sizes.
Is an upgrade available?
AstroPulse works out the next upgrade for each cluster it can read and puts it on the cluster in the API, as upgrade. The console and astroctl show it in the API's own words.
- Cluster page. When an upgrade is offered, the header shows its label, and the Overview shows the message with Review upgrade. When standard support is ending or has ended, the Overview also shows the support warning.
- Update tab. Upgrade Kubernetes shows the label and the message, with Start upgrade when an upgrade is offered.
- Cluster list. The Version column shows the label, and Upgrades available narrows the list to clusters with one offered.
The upgrade field:
availabletrue when the cluster can be upgraded to nextVersion.
nextVersionThe version the next upgrade moves to, one minor version at a time.
latestVersionThe newest version that upgrades of one minor version each reach from here.
stepsToLatestHow many upgrades, one minor version each, reach latestVersion.
labelA few words: "<next-minor> available", "Up to date", "No upgrade path" or "Upgraded outside AstroPulse". A newer patch of the cluster's own minor version is named in full.
messageOne or two sentences: "Kubernetes <next-minor> is available, the newest supported version.", "Kubernetes <next-minor> is available; <latest-minor> is 2 upgrades away.", "This cluster runs <current-version>. No newer Kubernetes version is supported yet." or, for No upgrade path, "Kubernetes <latest-minor> is supported, but not <next-minor>, the next minor version, so this cluster cannot be upgraded one minor version at a time."
standardSupportEndsAtEKS only: the date AWS ends standard support of the cluster's version.
supportEndingEKS only: true when that date is within 90 days or has passed.
inExtendedSupportEKS only: true when the version is past standard support and in AWS extended support.
supportMessageEKS only, when supportEnding is true: "Standard support ends <date>; extended support after that has an extra cost.", "In extended support since <date>; extra cost applies." or "Support for Kubernetes <version> ended <date>." --list-versions prints it as a warning.
upgrade is left out while an operation runs on the cluster, when no cluster exists behind the record, before the cluster reports a version, and when the provider's version list cannot be read right now. A missing upgrade never means up to date. On the clusters list it can be missing until the provider's version list has been read once; astroctl infra k8s get -c <cluster-name> reads it. GKE, AKS and self-hosted clusters get no support dates, because their providers publish none.
On a registered cluster available is false, the label is "Upgraded outside AstroPulse", and the message is "AstroPulse does not upgrade registered clusters. Upgrade this cluster with the tools that manage it." --list-versions prints that message and lists no versions.
astroctl infra k8s upgrade <cluster-name> --list-versions prints the offer, the supported versions with the next one marked (next), and the commands to run. Versions your cloud offers: astroctl cloud <aws|gcp|azure> k8s-versions, with --show-upgrade-paths for what a cluster can upgrade to.
How an upgrade runs
- Preflight: the version path, the cluster's health, add-on compatibility and capacity. A blocking item in the readiness report stops the upgrade before anything changes.
- Control plane: the cloud's managed upgrade on EKS, GKE and AKS, or the self-hosted control plane. On EKS, add-ons that cannot run on the target version move first.
- Node groups: a rolling update, one group after another. On EKS,
maxUnavailableapplies and AWS adds the replacement nodes itself;maxSurgeapplies on GKE and AKS. A PodDisruptionBudget that blocks an eviction stops that group with the reason; pods are never forced off. - Add-ons: EKS only, the EKS add-ons move to the target's version.
- Verification: on EKS, AstroPulse checks the control plane reports the target version.
Rolling updates
--max-surge is how many extra nodes a group adds during the update, and needs spare capacity; --max-unavailable is how many can be down at once, and lowers capacity for a while. Both apply per node group, and groups go one after another.
The capacity a rolling update needs at its peak is the largest group plus its surge: a group of 3 nodes with --max-surge 1 needs 4 for a while. Common choices:
--max-surge 1 --max-unavailable 0: one extra node per group, no capacity lost. The usual choice for production.--max-surge 0 --max-unavailable 1: no extra nodes, one node down at a time. Slower, and no extra cost.--max-surge 25% --max-unavailable 0: more nodes at once, faster, and needs more capacity.
On EKS, only --max-unavailable applies: AWS adds the replacement nodes itself, and 0 keeps the group's own setting.
EKS add-ons during an upgrade
AstroPulse checks every EKS add-on installed on the cluster (such as vpc-cni, kube-proxy, coredns, eks-pod-identity-agent and aws-ebs-csi-driver) against the target version, and moves each one in one of two places:
- Before the control plane: only an add-on whose version does not run on the target, and only to a version that runs on both the current and the target Kubernetes version. If there is no such version, the add-on waits for the control plane instead. If a move before the control plane fails, the control-plane step fails with "EKS add-on update before the control plane upgrade failed:
<reason>" and is retried, as described below. kube-proxy always waits, because each build matches one Kubernetes version. The progress says so, for example "Add-on kube-proxy stays at<version>until the control plane is upgraded, then moves to<version>". - After the control plane and the node groups: every add-on moves to the target's default version, in the order vpc-cni, kube-proxy, coredns, eks-pod-identity-agent, aws-ebs-csi-driver, then the rest by name. An add-on already on that version or newer is left alone; nothing is downgraded. Each one reports "Updating addon
<name>to<version>", then "Add-on<name>updated from<version>to<version>" or "Add-on<name>stays at<version>on Kubernetes<version>".
An add-on that fails to update after the control plane is reported as "Add-on <name> was not updated: <reason>" and does not fail the upgrade; the other add-ons still move. If none can be updated, the progress says "Add-ons were not updated: <reason>", or "Add-ons were not updated: their versions could not be read: <reason>", and the upgrade still completes. The readiness report (--generate-report) lists each move under Add-on Compatibility as "Before the control plane: ..." or "After the control plane: ...".
When a step fails, it is retried
The control-plane and node-group steps of an upgrade are each retried on their own, up to 20 attempts within 6 hours, whichever comes first. The wait between attempts starts at a second and doubles, up to 15 minutes. Each retried attempt appears on the Events tab and in astroctl infra k8s progress stream <cluster-name> as:
Updating control plane failed (attempt 5 of 20): <reason>; retrying
On EKS, a retry picks up where the upgrade stands: if the cloud finished the control plane before an attempt was interrupted, the next attempt sees the control plane already on the target version and moves on to the node groups. An upgrade can also be cancelled while it runs (see Cancel a running operation). When a step fails its last attempt, or hits an error retrying cannot fix, the upgrade stops and the cluster shows a Failed upgrade status (hover it for the reason); see How to debug cluster create, update and delete. Fix the cause and run the upgrade again.
Rollback
A control plane cannot be rolled back on EKS, GKE or AKS: only node groups go back. On EKS they are replaced by new node groups on the target version; on GKE and AKS the pools are downgraded. A self-hosted cluster can roll back its control plane too.
astroctl infra k8s rollback <cluster-name> --list-versions
astroctl infra k8s rollback <cluster-name> <version> --dry-run
Change node groups
To check a change to an existing cluster first, add --dry-run to astroctl infra k8s update or astroctl infra k8s scale. Nothing changes. The dry run plans the target: each node group is shown with its current size beside the size it will have, for example platform: 3-3 → 1-1 nodes. A new node group has no current size, and a removed one goes to 0-0.
This dry run checks the change against the cluster's settings and your cloud account, not the cluster's nodes or workloads, so it gives no readiness score. Its report reads Readiness: not assessed, riskLevel is Unknown, notEvaluated names the sections it did not read, and the notice Readiness not assessed says why: AstroPulse checked this change against the cluster's settings and your cloud account, not the cluster's nodes or workloads, so there is no readiness score.
A node group that gives up nodes or is removed raises Pods move off the nodes given up, for example Node group platform goes from 3-3 to 1-1 nodes, so the pods on the nodes it gives up move to the other nodes. AstroPulse did not check that they fit. Its next step is Check the pods fit on fewer nodes. The checklist follows the kind of change: upgrade checks appear only for a version change, Pod Disruption Budgets only when pods move, and Node Capacity only when nodes are replaced.
Remove a node group
On the cluster's Node groups tab, Remove is on each group the API allows; on any other group it is off, and the info button beside it shows the API's reason and next step. Remove opens Remove node group: "Its nodes are drained, then the node group is deleted." The dry run runs first, and Remove in the dialog turns on once it passes.
Removing a node group is a cluster update. Each named group's nodes are drained, then the group is deleted. List one or more names in the update's removeNodeGroups:
# Check the removal first: nothing changes
astroctl infra k8s update <cluster-name> --remove-node-group <node-group> --dry-run
# Remove it (repeat the flag to remove several)
astroctl infra k8s update <cluster-name> --remove-node-group <node-group>
The same in a manifest:
apiVersion: platform.astropulse.io/v1
kind: K8sClusterUpdate
spec:
removeNodeGroups:
- <node-group>
Every worker node group the API returns for a cluster says whether it can be removed now, under removal. allowed is true when an update can remove it. When it is false, notice carries a code, a title, a message with the reason and, when there is one, a nextStep:
# workerNodes[].removal for every worker node group
astroctl infra k8s get -c <cluster-name> -o yaml
NodeGroupRemovalLastSystemGroupminNode is at least 1 and it has no taint other than CriticalAddonsOnly or a PreferNoSchedule oneupdateCluster), then remove this one. A caller who may not update the cluster is told to ask an organization adminNodeGroupRemovalProviderComputeNodeGroupRemovalNotSupportedEach group is judged on its own. Removing two groups that can run system pods in one update can still be refused even when each reads allowed: true; the dry run says so before anything changes. A refused update answers with the same code (as the error's reason), title and nextStep.
NodeGroupRemovalLastSystemGroupminNode is at least 1 and it has no taint other than CriticalAddonsOnly or a PreferNoSchedule oneupdateCluster), then remove this one. A caller who may not update the cluster is told to ask an organization adminNodeGroupRemovalProviderComputeNodeGroupRemovalNotSupportedEach group is judged on its own. Removing two groups that can run system pods in one update can still be refused even when each reads allowed: true; the dry run says so before anything changes. A refused update answers with the same code (as the error's reason), title and nextStep.
Cancel a running operation
A cancel stops the running update, scale, upgrade or rollback of a cluster (admin role). It stops the current step, then removes only the node groups that update was adding. Node groups the cluster already had are kept as they are.
astroctl infra k8s cancel <cluster-name>
Before anything is sent, astroctl asks the API what the cancel would do and shows it, for example Cancelling stops the running update on cluster my-cluster, removing node group gpu-a, which it was adding. Existing node groups are kept as they are. It then asks Cancel the running update on cluster my-cluster? (yes/no): and cancels only on yes. --yes skips the prompt; without a terminal and without --yes, astroctl cancels nothing. The API offers the same check as a dry run.
While the cleanup runs, the cluster's lastCancellation reads Cancelling, then Cancelled with one line per node group the update was adding: removed, not created, or left in the cluster with why and the step to remove it. Follow it with astroctl infra k8s get <cluster-name>.
The cluster page shows Cancel operation at the top while an operation runs. Its dialog runs the same dry run as it opens and shows the API's answer, including the node groups the cancel removes, before you confirm. Cancel operation in the dialog turns on once the dry run passes and sends the same cancel; if the dry run is refused, the dialog shows why and sends nothing.
A create, a delete and a registration can't be stopped part-way. A cancel of one answers 409 with the notice This operation can't be cancelled, for example Cluster my-cluster is being created, and that can't be stopped part-way. Once it finishes, you can delete the cluster.
Delete a cluster
Select Delete at the top of the cluster page. The dialog runs a dry run of the delete first and lists what it removes and what stays in your cloud account. A registered cluster has Unregister instead. What a delete removes and keeps: Delete a cluster.
Cluster details and connectivity
Opening a cluster shows its overview: provider and region, Kubernetes version, node groups and application counts, spend over the last 24 hours, capacity, security findings, when it was created and last updated, and its provisioner configuration. Next to its name, the status and the connectivity: Connected when AstroPulse can reach the cluster now, Disconnected when it cannot, Unknown when that could not be checked, with when it was last seen. astroctl infra k8s get <cluster-name> shows the same.
How to debug cluster create, update and delete
Read the cluster's state first, then the operation's own events. In the console, open Infrastructure › Kubernetes › your cluster: the status pill sits next to the name, a red notice under it carries the platform's reason, and See events opens the Events tab.
# state, stateTone, statusReason, connectivity, upgradeStatus
astroctl infra k8s get -c <cluster> -o yaml
# every step of the current and past operations, with their errors
astroctl infra k8s progress history <cluster>
# follow a running operation live
astroctl infra k8s progress stream <cluster>
# the same checks a create runs, without creating anything
astroctl infra k8s apply -f cluster.yaml --dry-run
The API reports state (for example ProvisionFailed), which the console shows
by its display name (for example "Provision failed"). The failed states are Failed,
ProvisionFailed, DeleteFailed, ApprovalFailed and RegisterFailed;
the console's status pill reads Failed for each of them. statusReason
has three lines: what happened, the problem, and the fix.
The provider keeps its own record of what failed. For a failed EKS create or node join, see Troubleshoot problems with Amazon EKS clusters and nodes.
Other ways in
- Nova: ask "what versions can prod-eks upgrade to?" or "is prod-eks ready for the next version?". Nova lists the versions, writes the readiness and capacity report, and returns the upgrade command for you to run: it never runs an upgrade, rollback or delete itself. Ask "scale the ng-app node group to 12 nodes" and Nova previews the change and applies it after you confirm.
- SDK and API: the same update request the console sends, with
dryRunfor the plan. See the API reference.
With astroctl
astroctl infra k8s upgrade <cluster-name> --list-versions
astroctl infra k8s upgrade <cluster-name> <next-version> --generate-report
astroctl infra k8s upgrade <cluster-name> <next-version> --dry-run
astroctl infra k8s upgrade <cluster-name> <next-version> --max-surge 1 --max-unavailable 0
astroctl infra k8s scale <cluster-name> --list
astroctl infra k8s scale <cluster-name> <node-group> --min 3 --max 3 --dry-run
astroctl infra k8s update <cluster-name> --example
astroctl infra k8s update <cluster-name> -f update.yaml --dry-run
astroctl infra k8s progress stream <cluster-name>
<next-version> is the version --list-versions marks (next). A real upgrade always runs the readiness report first and does not start when it blocks; --yes skips only the confirmation. Several changes at once go in an update file:
kubernetesVersion: "<next-version>"
updateConfig:
maxUnavailable: "1"
maxSurge: "1"
nodeGroups:
- name: "default-pool"
minNode: 2
maxNode: 8
machineTypes: ["t3.medium"] # or ["e2-medium"] for GKE, ["Standard_D2s_v3"] for AKS
labels:
workload: "general"
Every flag: astroctl infra k8s upgrade, scale, update.
If it's refused
PreflightFailed: "the cluster cannot be created: <check>: <problem> <fix>"details.checks with its name (for example "Account Connection", "IAM Permissions", "Subnet IP Capacity", "Zone Offering", "Quota Headroom", "Instance Type Availability") and its remediationastroctl infra k8s apply -f cluster.yaml --dry-run until every check passes. In the console, the create wizard runs the same dry run before Create<checks>. Nothing was changed; run the dry run again."<kind> cluster <name> in <where>"clusterName, or register the existing cluster with astroctl infra k8s register --cluster-name <name>ClusterAlreadyExists (Cluster already exists): "Cluster <name> already exists. Update it to change it."clusterName, or change the existing cluster with astroctl infra k8s update (the next step Update the cluster)<id> and name <name> is currently being deleted. Please wait before deploying a new cluster with the same name."astroctl infra k8s progress stream <cluster>), then apply again404 with the notice Not found in this organization: "Cluster <name> was not found in this organization." and the next step "Check the name and try again"astroctl infra k8s get, and the organization with astroctl whoami. Switch with astroctl org use <org> if the cluster is in another oneApprovalPending ("Pending approval"), or ApprovalFailed ("Approval failed")ApprovalFailed means the owner rejected it and nothing was createdProvisionFailed ("Provision failed"). statusReason starts with one of "Cloud account refused access", "Not enough capacity or quota", "Network settings rejected", "Cluster settings rejected", "Resource not found" or "Cluster creation failed", then "Creating cluster <name> stopped: <detail>."statusReason says (for example "Raise the quota, or choose other machine types or zones, then create the cluster again."), then run astroctl infra k8s apply -f cluster.yaml again with the same name. A failed create can be re-appliedastro-agent add-on (see Manage add-ons), then check the cluster reads ConnectedClusterOperationInProgress: "Cluster <name> is still being <created, updated or deleted>. A new change can start once it ends." The sentence can add when it started and the retry attemptastroctl infra k8s progress stream <cluster>, then send the change againNodeGroupRemovalLastSystemGroup, NodeGroupRemovalProviderCompute or NodeGroupRemovalNotSupportedremoveNodeGroups cannot be removed. The error's title and nextStep match the group's removal.noticeworkerNodes[].removal with astroctl infra k8s get -c <cluster> -o yaml and do what its nextStep says. See Remove a node group<state>"statusReason with astroctl infra k8s get -c <cluster> -o yaml. Fix a failed create by applying the create spec again<current> to <target> not allowed. Please upgrade to <next> first", or "cannot downgrade from <current> to <target>"astroctl infra k8s upgrade <cluster> --list-versions and upgrade one step at a time<reasons>. Resolve these, then upgrade again; a dry run or readiness report shows the same checks."astroctl infra k8s upgrade <cluster> <version> --generate-report, resolve each blocking item, then upgrade againPrecheckFailed stopped before anything changed. Failed stopped partway. The cluster reads Failed only when it cannot be reached afterwards, and then its statusReason reads "The last update of cluster <name> failed: <detail>."upgradeStatus.reason (or rollbackStatus.reason) with astroctl infra k8s get -c <cluster> -o yaml, and the step that failed with astroctl infra k8s progress history <cluster>. Fix it and run the upgrade againprogress history shows "Cluster update failed (<state>): <error>"AccessDenied: "Insufficient organization permissions"astroctl infra k8s delete <cluster> or select Delete in the console<cluster> is registered, not provisioned. AstroPulse does not delete registered clusters: it did not create their infrastructure. To stop managing the cluster from AstroPulse, unregister it; the cluster and its applications keep running, and its AstroPulse cost history is removed."astroctl infra k8s unregister --cluster-name <cluster>SelfHostedKeyMissing ("No stored key for this cluster") or SelfHostedKeyUnreadable ("The stored key can't be read") (self-hosted clusters)astroctl infra k8s delete <cluster> --record-only<name> is already being deleted. Follow its progress in the console or with astroctl infra k8s progress stream <name>."astroctl infra k8s progress stream <cluster>DeleteFailed ("Delete failed"). The event on the Events tab reads "Cluster deletion failed: <error>" or "Removing the cluster's add-ons failed, so the cluster was not deleted: <error>"statusReason is not set for a failed delete; the reason is only in the eventsastroctl infra k8s progress history <cluster>, fix what it names, then run astroctl infra k8s delete <cluster> again. See What deleting a cluster removes, and what it keeps for what a delete leaves behindlastSeen is the agent's last report. The API may give a more specific reason in place of this sentence.astroctl infra k8s addons get <cluster> astro-agent -o yaml.