Skip to main content

Self-hosted Kubernetes

AstroPulse builds Kubernetes on virtual machines in your AWS account or Google Cloud project. The control plane runs on your machines next to the workers, so you size both, and you pay your cloud for every machine. Choose it over Amazon EKS or Google Kubernetes Engine when you need the control plane itself in your account; otherwise a managed service is the simpler choice. Self-hosted clusters take no GPU node groups.

When you need it​

  • Your policy requires the Kubernetes control plane and etcd to run on machines in your own account.
  • You want to choose the control plane's machines and how many there are.
  • You need to roll a cluster back to an earlier Kubernetes version, control plane included.
  • You run on AWS or Google Cloud. Self-hosted Kubernetes is not offered on Azure.

Before you start​

  • An AWS account or a Google Cloud project you can create IAM resources in: a CloudFormation stack on AWS, a service account and role bindings on Google Cloud.
  • An organization owner or admin in AstroPulse. Only they can start the setup and store the cloud key: "Only an organization admin can manage the organization's credentials."
  • astroctl on your computer, signed in to your organization. The key goes from your terminal to AstroPulse, never through the browser.
  • Room in the account's quotas for the control-plane machines as well as the workers.

In the console​

Open Infrastructure › Kubernetes › New cluster and choose Bring your own account. The steps are those of Create a cluster; what differs for self-hosted:

  1. Name and cloud. Pick Amazon Web Services or Google Cloud under Cloud, and a Region.

  2. Account. Under How Kubernetes runs, pick Self-hosted Kubernetes. Enter the Account ID on AWS or the Project ID on Google Cloud.

    Open zoom view

    The Account step offering Amazon EKS and Self-hosted Kubernetes on Amazon Web Services

    The Account step offering Amazon EKS and Self-hosted Kubernetes on Amazon Web ServicesThe Account step offering Amazon EKS and Self-hosted Kubernetes on Amazon Web Services

  3. Connect. Select Start the setup. The step lists what the setup creates in your account, then gives you the command that sends the key to AstroPulse under Hand the credentials to the vault. It moves on by itself once AstroPulse has checked and stored the key. What each cloud creates: AWS, Google Cloud.

  4. Kubernetes. Pick the Kubernetes Version, or keep AstroPulse default.

  5. Network. Leave Use Existing VPC (AWS) or Use Existing Network (Google Cloud) off for a new network, or turn it on and enter yours: Networking.

  6. Node groups. The Control plane card comes first: under Machine type select Choose machine, and set Name, Minimum nodes and Maximum nodes (Control plane). Then add the worker groups with Add Node Group.

  7. Notifications. Optional.

  8. Review. The dry run checks the manifest before anything is created, then Create.

You should see​

The cluster's page reads Ready once the control plane and the workers are up, its Node groups list the control-plane and worker machines, and Events shows each stage of the create.

The cloud key and how it is kept​

A managed cluster needs no key: AstroPulse uses short-lived access you approve. A self-hosted cluster needs a long-lived cloud key, because AstroPulse runs the control plane's machines itself: an IAM user's access key on AWS, a service account key on Google Cloud.

  • What you give. The key that the setup creates for this one cluster, or, with --without-setup, the key of an IAM user or service account you made yourself.
  • How it reaches AstroPulse. astroctl ... selfHosted connect sends it from your terminal. It takes the key from the setup stack, an AWS CLI profile, its own flags or a key file, never from the browser and never from environment variables.
  • What AstroPulse checks first. That the key reaches the account you named, who it belongs to, and that it can use the state bucket. A key made by setup is accepted only by the organization that started that setup, and only for its cluster. A key whose owner AstroPulse cannot confirm is refused.
  • How it is kept. AstroPulse stores it encrypted and never shows it back, in the console, astroctl or the API. Your manifest says only credentials: {type: Vault}.
  • Where you see it. Integrations › Vault lists it at cluster scope, with when it was stored and last checked. An owner or admin can Revoke or Replace it there; Replace gives the command to run in your terminal.
  • Rotate it by making a new key in your cloud and sending it with connect --replace. AstroPulse checks the new key before it replaces the stored one.

The key does not expire. Revoking it in AstroPulse, or selfHosted disconnect, deletes only AstroPulse's copy: the key keeps working in your cloud until you delete it there, as the AWS and Google Cloud pages show. Without a stored key the cluster keeps running, but AstroPulse cannot change, upgrade, scale or delete it.

To keep the key to least privilege, give it an account or project of its own. The permissions the setup grants cover the whole account or project, not one cluster: each cloud page lists them.

The control plane​

AstroPulse runs one control-plane group in each of the cluster's zones, the region's first three, and etcd on the same machines, on encrypted volumes with backups kept for 90 days. Minimum nodes and Maximum nodes apply to each group, so the default of 1 runs three control-plane machines in a region with three zones. Each group runs one machine type. Every detail of the card: Control plane.

Node groups​

Each group runs one machine type, chosen under Pick machine types. An autoscaler on the cluster resizes each group between its Minimum nodes and Maximum nodes; a minimum of 0 lets a group scale to zero while idle. Pricing is On-Demand or Spot. Reserved capacity and GPU node groups are not offered on self-hosted clusters: for GPUs, use EKS or GKE.

Kubernetes versions​

A self-hosted cluster takes an exact major.minor.patch version from the list AstroPulse supports; major.minor is refused. AstroPulse default sends no version, and the platform picks its default. To list the versions and where each can upgrade to:

Terminal
astroctl cloud aws k8s-versions --self-hosted
astroctl cloud gcp k8s-versions --self-hosted --show-upgrade-paths

Updates and upgrades​

The cluster's Update tab upgrades Kubernetes and changes node groups; Node groups › Modify changes one group. The upgrade wizard can also resize the Control Plane. Each change runs a dry run first and shows on Events. Update and upgrade covers every surface; what is particular to self-hosted:

  • The control plane upgrades first, then the workers, by rolling replacement: each node is replaced by a new one. Leave room in your quotas for the extra machines.
  • A control plane you resize must have a fixed size, the same minimum and maximum, of 1, 3, 5 or 7.
  • A rollback moves the control plane back as well as the workers, within the same major version and at most two minor versions back.
Terminal
astroctl infra k8s upgrade <cluster> <next-version> --dry-run
astroctl infra k8s rollback <cluster> --list-versions
astroctl infra k8s scale <cluster> <node-group> --min 2 --max 5 --dry-run

<next-version> stands for a version from --show-upgrade-paths. For kubectl access: Access with kubectl.

Other ways in​

  • Nova: ask it, for example "plan a self-hosted cluster in my AWS account with three control-plane machines". Nova shows the plan; the key still goes from your terminal with astroctl.
  • SDK and API: the same operations, in the API reference.

With astroctl​

Terminal
astroctl cloud aws selfHosted setup --account-id 123456789012 --region us-west-2 --cluster-name prod
astroctl cloud aws selfHosted connect --account-id 123456789012 --region us-west-2 \
  --from-stack <setup-stack> --cluster-name prod
astroctl infra k8s template selfHosted > prod.yaml
astroctl infra k8s apply -f prod.yaml --dry-run
astroctl infra k8s apply -f prod.yaml

On Google Cloud, astroctl cloud gcp selfHosted setup and connect take --project-id instead. status shows each recorded setup step, cleanup removes what the setup created, and forget drops only AstroPulse's record of the setup. The cloud pages have each command in full: AWS, Google Cloud.

Settings​

The settings particular to self-hosted. Each cloud's spec.provisioner.selfHosted fields are on its page; the settings every cloud shares are on Create a cluster.

How Kubernetes runsspec.provisioner.typeRequired
Accepts
selfHosted, on aws or gcp.
Control planespec.clusterSpec.controlPlane.nodeGroupRequired

The control plane's machines: name, one entry in machineTypes, minNode and maxNode, per zone.

Default
minNode: 1, maxNode: 1.
See
Control plane
Node groupsspec.clusterSpec.dataPlane.nodeGroups

The worker groups: one machine type each, a minimum and maximum, OnDemand or Spot.

If it's refused​

"must configure clusterSpec.controlPlane field for provisioner type selfHosted"
Cause
The manifest has no control-plane node group.
Fix
Add clusterSpec.controlPlane.nodeGroup, as astroctl infra k8s template selfHosted shows.
"credentials.type "static" is no longer supported for selfHosted clusters. …"
Cause
The manifest carries the cloud key itself.
Fix
Store the key with astroctl cloud <aws|gcp> selfHosted connect, take it out of the file, and set credentials: {type: Vault}.
"credentials.type "dynamic" is not supported for selfHosted clusters. Use "vault" instead"
Cause
A managed cluster's credentials type was used.
Fix
Set credentials: {type: Vault}.
"unsupported Kubernetes version <version>. Supported versions are: […]"
Cause
The version is not one AstroPulse supports for self-hosted clusters, or it is major.minor only.
Fix
Pick a full version from the list, or leave kubernetesVersion out for the default.
"DriverOwnerMissing: node group gpu: accelerator node groups are not supported for self-hosted clusters; use a managed provisioner (eks, gke, aks)"
Cause
A node group requests a GPU.
Fix
Remove accelerator from the group, or run GPU work on EKS or GKE.
"node group workers: selfHosted node groups run a single machine type, got 2 (t3.medium, t3.large); keep exactly one (multiple machine types are supported on eks)"
Cause
A node group lists more than one machine type.
Fix
Keep one machine type per group, and add a group for each other type.
"node group workers: instanceType reserved is eks and gke only for now; selfHosted node groups do not support it"
Cause
A node group asks for reserved capacity.
Fix
Use OnDemand or Spot.
"Control plane size must be 1, 3, 5, or 7 (got 2)"
Cause
An update resizes the control plane to a count etcd cannot keep a quorum with.
Fix
Use 1, 3, 5 or 7, with the same minimum and maximum.
"Only an organization admin can manage the organization's credentials."
Cause
Starting the setup, sending a key or reading its status needs an organization owner or admin.
Fix
Ask an owner or admin to run the Connect step or selfHosted connect.

Next​