Self-hosted Kubernetes on Google Cloud
AstroPulse builds Kubernetes on Compute Engine instances in your Google Cloud project: the control plane and the workers both run on your machines, and you size both. AstroPulse works in the project with a service account key that a Cloud Shell setup creates for this one cluster and that AstroPulse stores encrypted. Choose it over Google Kubernetes Engine when you need the control plane itself in your project; GKE needs no stored key and takes GPU node pools, which self-hosted clusters do not. What every self-hosted cluster shares is on Self-hosted Kubernetes.
When you need it
- The control plane and etcd must run on Compute Engine instances in your own project.
- You want to pick the control plane's machine type and count.
- You need rollbacks that take the control plane back too.
Before you start
- Your Google Cloud project ID, and a Google account that can create service accounts, grant project roles, enable APIs and create buckets in it.
- An organization owner or admin in AstroPulse.
astroctlv0.0.179 or later on your computer, to send the key after you rotate it.- Room in the project's Compute Engine quotas for the control plane, one group in each of the region's first three zones, and the workers.
In the console
On New cluster, choose Bring your own account.
-
Name and cloud. Enter the Cluster Name, pick Google Cloud under Cloud, and pick the Region.
Open zoom view -
Account. Under How Kubernetes runs, pick Self-hosted Kubernetes. Enter the Project ID, a name such as
my-project, not the number (gcloud config get-value projectprints it).Open zoom view -
Connect. Check the Google account that owns this project. It starts as your sign-in email; Cloud Shell opens as this account, and the setup stops if it runs as another. Select Start the setup.
Open zoom view -
What the setup creates in your account names the service account and bucket for this cluster (what they are). Select Open Cloud Shell and run the commands shown under Cloud Shell. The first one checks which account it runs as. They create the service account, grant its roles, enable the APIs, create the bucket and write a key to
~/.astropulse/credentials/<project-id>.json, which your browser then downloads. -
Move the downloaded key to the same path on your computer and run the command under Hand the credentials to the vault, labelled astroctl, on your computer. AstroPulse checks the key against your project before it stores it. Then delete the key file from your computer and from Cloud Shell; AstroPulse keeps its own copy. The step reads "The vault holds working credentials for <cluster>." and moves on by itself.
-
Kubernetes. Pick the Kubernetes Version, or keep AstroPulse default.
-
Network. Under Network Configuration, leave Use Existing Network off: "A new Network will be created for your cluster with default settings." Or turn it on and enter the Network ID (Your own network).
-
Node groups. On the Control plane card, select Choose machine under Machine type and set Minimum nodes and Maximum nodes. Then Add Node Group for the workers: one machine type each from Choose machines, Minimum nodes, Maximum nodes, Pricing (On-Demand or Spot) and Name.
-
Notifications. Optional.
-
Review. The dry run checks the manifest, then Create.
You should see
The cluster's page reads Ready, Node groups lists the control-plane groups and your worker groups, and Events shows each stage of the create without a failure. Integrations › Vault lists the cluster's Google Cloud key at cluster scope.
What the setup creates
The Cloud Shell commands create, in your project:
- A service account named for the cluster, with one key. This is the key AstroPulse stores.
- Eight project roles for it:
roles/viewer,roles/storage.admin,roles/compute.admin,roles/container.admin,roles/iam.serviceAccountUser,roles/iam.serviceAccountAdmin,roles/resourcemanager.projectIamAdminandroles/artifactregistry.admin. - Five APIs enabled:
compute.googleapis.com,container.googleapis.com,cloudresourcemanager.googleapis.com,containerregistry.googleapis.comandartifactregistry.googleapis.com. - A Cloud Storage bucket in the cluster's region that holds the cluster's state, labelled
astropulse-managed,astropulse-organdastropulse-cluster. If a bucket by that name exists and the setup did not create it, the setup stops rather than reuse it.
Least privilege
The roles apply to the whole project, and two of them let the key change IAM: roles/iam.serviceAccountAdmin and roles/resourcemanager.projectIamAdmin. To keep that scope to one cluster:
- Use a Google Cloud project for self-hosted clusters only.
- Delete the key file once
connectsucceeds, from your computer and from Cloud Shell. - Limit who can create keys for the service account and read the state bucket.
- To use a service account you made yourself, grant it the same roles, enable the same APIs and connect with
--without-setup. AstroPulse refuses a key that lacks any of them.
The service account key
AstroPulse checks the key reaches the project, that the service account is the one the setup made for this cluster in your organization, that the APIs are enabled, that the account holds every role and that the state bucket is reachable. It stores the key only when every check passes, encrypted, and never shows it back. connect reads the key from the file --credentials names, never from environment variables or Application Default Credentials.
The key does not expire. To rotate it, create a new key for the service account and send it in place of the stored one, either with Replace on Integrations › Vault or:
astroctl cloud gcp selfHosted connect --cluster-name prod --project-id my-project --region us-central1 \
--credentials <new-key.json> --replace
AstroPulse checks the new key before it replaces the stored one. The old key stays valid in Google Cloud until you delete it there.
If a key leaks
Revoking the key in AstroPulse, or selfHosted disconnect, deletes only AstroPulse's copy. The key stays valid in Google Cloud until you delete it there:
gcloud iam service-accounts keys list --iam-account <service-account-email> --project my-project
gcloud iam service-accounts keys delete <key-id> --iam-account <service-account-email> --project my-project
Then send a new key with connect --replace, or run cleanup to remove the service account and everything the setup created. Until a new key is stored, the cluster keeps running but AstroPulse cannot change, upgrade, scale or delete it. If the cluster's machines are already gone and you ran cleanup, astroctl infra k8s delete prod --record-only removes AstroPulse's record and lists anything that may remain in the project.
Networking
The cluster runs in one regional subnet that spans its zones. The Kubernetes API is reached through a public load balancer. Networking has the details.
Your own network
Turn on Use Existing Network and enter its name, or set networkId in the manifest. The network must be in the cluster's project and must not use 100.64.0.0/10, which the cluster keeps for pods and services. The dry run checks the name's format, not the network itself.
Kubernetes versions
A self-hosted cluster takes an exact major.minor.patch version. AstroPulse default sends none, and the platform picks its default. To list what is supported and where each version can upgrade to:
astroctl cloud gcp k8s-versions --self-hosted
astroctl cloud gcp k8s-versions --self-hosted --show-upgrade-paths
Updates and upgrades
The cluster's Update tab upgrades Kubernetes and changes node groups, and the upgrade wizard can resize the Control Plane. The control plane upgrades first, then the workers by rolling replacement, and a rollback moves the control plane back too: Updates and upgrades.
astroctl infra k8s upgrade prod <next-version> --dry-run
astroctl infra k8s rollback prod --list-versions
astroctl infra k8s update prod --example
Other ways in
- Nova: ask it, for example "plan a self-hosted cluster called prod in my Google Cloud project in us-central1". Nova shows the plan; the key still goes from your terminal with
astroctl. - SDK and API: the same operations, in the API reference.
With astroctl
astroctl cloud gcp selfHosted setup --project-id my-project --region us-central1 \
--cluster-name prod --account you@company.com
astroctl cloud gcp selfHosted connect --project-id my-project --region us-central1 \
--credentials ~/.astropulse/credentials/my-project.json --cluster-name prod
astroctl cloud gcp selfHosted status --cluster-name prod
astroctl infra k8s apply -f prod.yaml --dry-run
astroctl infra k8s apply -f prod.yaml
--account is the Google account that owns the project: Cloud Shell opens as it, and the commands stop before creating anything when they run as another. status shows each recorded setup step. The manifest:
apiVersion: platform.astropulse.io/v1
kind: K8sCluster
spec:
clusterName: prod
provider: gcp
region: us-central1
clusterSpec:
controlPlane:
nodeGroup:
name: control-plane
machineTypes: [n1-standard-2]
minNode: 1
maxNode: 1
dataPlane:
nodeGroups:
- name: workers
minNode: 2
maxNode: 5
machineTypes: [n1-standard-2]
instanceType: OnDemand
provisioner:
type: selfHosted
selfHosted:
accountId: my-project
credentials:
type: Vault
astroctl infra k8s template selfHosted prints the AWS form of this manifest; change provider, region and machineTypes.
To remove what the setup created, delete the cluster first, then:
astroctl cloud gcp selfHosted cleanup --cluster-name prod
It removes exactly what the setup made, by the names it recorded: the role bindings, the service account and its keys, the bucket and its contents, the key file, and the key stored in AstroPulse. Anything else by those names is kept and reported. --show-commands prints the commands without running them; the Connect step shows them under Remove what the setup created, to run in Cloud Shell. Every flag: astroctl cloud gcp selfHosted.
Settings
The fields under spec.provisioner.selfHosted on Google Cloud. The control plane and node groups: Self-hosted Kubernetes.
accountIdRequiredThe Google Cloud project the cluster is created in.
credentialsRequiredtype: Vault, the key stored with selfHosted connect.kubernetesVersionmajor.minor.patch from the supported list.astroctl cloud gcp k8s-versions --self-hostednetworkIdAn existing VPC network to create the cluster in. Left out, a new network is created.
bucketNameThe Cloud Storage bucket that holds the cluster's state, in the cluster's project and region.
If it's refused
gcloud config get-value project.networkId is not a network name.--credentials.astroctl is signed in to another organization than the one that started the setup, or no setup was started.--without-setup for a key you made yourself.connect again in a minute.