Skip to main content

Self-hosted Kubernetes on Google Cloud

AstroPulse builds Kubernetes on Compute Engine instances in your Google Cloud project: the control plane and the workers both run on your machines, and you size both. AstroPulse works in the project with a service account key that a Cloud Shell setup creates for this one cluster and that AstroPulse stores encrypted. Choose it over Google Kubernetes Engine when you need the control plane itself in your project; GKE needs no stored key and takes GPU node pools, which self-hosted clusters do not. What every self-hosted cluster shares is on Self-hosted Kubernetes.

When you need it​

  • The control plane and etcd must run on Compute Engine instances in your own project.
  • You want to pick the control plane's machine type and count.
  • You need rollbacks that take the control plane back too.

Before you start​

  • Your Google Cloud project ID, and a Google account that can create service accounts, grant project roles, enable APIs and create buckets in it.
  • An organization owner or admin in AstroPulse.
  • astroctl v0.0.179 or later on your computer, to send the key after you rotate it.
  • Room in the project's Compute Engine quotas for the control plane, one group in each of the region's first three zones, and the workers.

In the console​

On New cluster, choose Bring your own account.

  1. Name and cloud. Enter the Cluster Name, pick Google Cloud under Cloud, and pick the Region.

    Open zoom view

    The Name and cloud step with a cluster on Google Cloud

    The Name and cloud step with a cluster on Google CloudThe Name and cloud step with a cluster on Google Cloud
  2. Account. Under How Kubernetes runs, pick Self-hosted Kubernetes. Enter the Project ID, a name such as my-project, not the number (gcloud config get-value project prints it).

    Open zoom view

    The Account step with Self-hosted Kubernetes picked and a Google Cloud project ID

    The Account step with Self-hosted Kubernetes picked and a Google Cloud project IDThe Account step with Self-hosted Kubernetes picked and a Google Cloud project ID
  3. Connect. Check the Google account that owns this project. It starts as your sign-in email; Cloud Shell opens as this account, and the setup stops if it runs as another. Select Start the setup.

    Open zoom view

    The Connect step with the Google account that owns this project and Start the setup

    The Connect step with the Google account that owns this project and Start the setupThe Connect step with the Google account that owns this project and Start the setup
  4. What the setup creates in your account names the service account and bucket for this cluster (what they are). Select Open Cloud Shell and run the commands shown under Cloud Shell. The first one checks which account it runs as. They create the service account, grant its roles, enable the APIs, create the bucket and write a key to ~/.astropulse/credentials/<project-id>.json, which your browser then downloads.

  5. Move the downloaded key to the same path on your computer and run the command under Hand the credentials to the vault, labelled astroctl, on your computer. AstroPulse checks the key against your project before it stores it. Then delete the key file from your computer and from Cloud Shell; AstroPulse keeps its own copy. The step reads "The vault holds working credentials for <cluster>." and moves on by itself.

  6. Kubernetes. Pick the Kubernetes Version, or keep AstroPulse default.

  7. Network. Under Network Configuration, leave Use Existing Network off: "A new Network will be created for your cluster with default settings." Or turn it on and enter the Network ID (Your own network).

  8. Node groups. On the Control plane card, select Choose machine under Machine type and set Minimum nodes and Maximum nodes. Then Add Node Group for the workers: one machine type each from Choose machines, Minimum nodes, Maximum nodes, Pricing (On-Demand or Spot) and Name.

  9. Notifications. Optional.

  10. Review. The dry run checks the manifest, then Create.

You should see​

The cluster's page reads Ready, Node groups lists the control-plane groups and your worker groups, and Events shows each stage of the create without a failure. Integrations › Vault lists the cluster's Google Cloud key at cluster scope.

What the setup creates​

The Cloud Shell commands create, in your project:

  • A service account named for the cluster, with one key. This is the key AstroPulse stores.
  • Eight project roles for it: roles/viewer, roles/storage.admin, roles/compute.admin, roles/container.admin, roles/iam.serviceAccountUser, roles/iam.serviceAccountAdmin, roles/resourcemanager.projectIamAdmin and roles/artifactregistry.admin.
  • Five APIs enabled: compute.googleapis.com, container.googleapis.com, cloudresourcemanager.googleapis.com, containerregistry.googleapis.com and artifactregistry.googleapis.com.
  • A Cloud Storage bucket in the cluster's region that holds the cluster's state, labelled astropulse-managed, astropulse-org and astropulse-cluster. If a bucket by that name exists and the setup did not create it, the setup stops rather than reuse it.

Least privilege​

The roles apply to the whole project, and two of them let the key change IAM: roles/iam.serviceAccountAdmin and roles/resourcemanager.projectIamAdmin. To keep that scope to one cluster:

  • Use a Google Cloud project for self-hosted clusters only.
  • Delete the key file once connect succeeds, from your computer and from Cloud Shell.
  • Limit who can create keys for the service account and read the state bucket.
  • To use a service account you made yourself, grant it the same roles, enable the same APIs and connect with --without-setup. AstroPulse refuses a key that lacks any of them.

The service account key​

AstroPulse checks the key reaches the project, that the service account is the one the setup made for this cluster in your organization, that the APIs are enabled, that the account holds every role and that the state bucket is reachable. It stores the key only when every check passes, encrypted, and never shows it back. connect reads the key from the file --credentials names, never from environment variables or Application Default Credentials.

The key does not expire. To rotate it, create a new key for the service account and send it in place of the stored one, either with Replace on Integrations › Vault or:

Terminal
astroctl cloud gcp selfHosted connect --cluster-name prod --project-id my-project --region us-central1 \
--credentials <new-key.json> --replace

AstroPulse checks the new key before it replaces the stored one. The old key stays valid in Google Cloud until you delete it there.

If a key leaks​

Revoking the key in AstroPulse, or selfHosted disconnect, deletes only AstroPulse's copy. The key stays valid in Google Cloud until you delete it there:

Terminal
gcloud iam service-accounts keys list --iam-account <service-account-email> --project my-project
gcloud iam service-accounts keys delete <key-id> --iam-account <service-account-email> --project my-project

Then send a new key with connect --replace, or run cleanup to remove the service account and everything the setup created. Until a new key is stored, the cluster keeps running but AstroPulse cannot change, upgrade, scale or delete it. If the cluster's machines are already gone and you ran cleanup, astroctl infra k8s delete prod --record-only removes AstroPulse's record and lists anything that may remain in the project.

Networking​

The cluster runs in one regional subnet that spans its zones. The Kubernetes API is reached through a public load balancer. Networking has the details.

Your own network​

Turn on Use Existing Network and enter its name, or set networkId in the manifest. The network must be in the cluster's project and must not use 100.64.0.0/10, which the cluster keeps for pods and services. The dry run checks the name's format, not the network itself.

Kubernetes versions​

A self-hosted cluster takes an exact major.minor.patch version. AstroPulse default sends none, and the platform picks its default. To list what is supported and where each version can upgrade to:

Terminal
astroctl cloud gcp k8s-versions --self-hosted
astroctl cloud gcp k8s-versions --self-hosted --show-upgrade-paths

Updates and upgrades​

The cluster's Update tab upgrades Kubernetes and changes node groups, and the upgrade wizard can resize the Control Plane. The control plane upgrades first, then the workers by rolling replacement, and a rollback moves the control plane back too: Updates and upgrades.

Terminal
astroctl infra k8s upgrade prod <next-version> --dry-run
astroctl infra k8s rollback prod --list-versions
astroctl infra k8s update prod --example

Other ways in​

  • Nova: ask it, for example "plan a self-hosted cluster called prod in my Google Cloud project in us-central1". Nova shows the plan; the key still goes from your terminal with astroctl.
  • SDK and API: the same operations, in the API reference.

With astroctl​

Terminal
astroctl cloud gcp selfHosted setup --project-id my-project --region us-central1 \
--cluster-name prod --account you@company.com
astroctl cloud gcp selfHosted connect --project-id my-project --region us-central1 \
--credentials ~/.astropulse/credentials/my-project.json --cluster-name prod
astroctl cloud gcp selfHosted status --cluster-name prod
astroctl infra k8s apply -f prod.yaml --dry-run
astroctl infra k8s apply -f prod.yaml

--account is the Google account that owns the project: Cloud Shell opens as it, and the commands stop before creating anything when they run as another. status shows each recorded setup step. The manifest:

prod.yaml
apiVersion: platform.astropulse.io/v1
kind: K8sCluster
spec:
clusterName: prod
provider: gcp
region: us-central1
clusterSpec:
controlPlane:
nodeGroup:
name: control-plane
machineTypes: [n1-standard-2]
minNode: 1
maxNode: 1
dataPlane:
nodeGroups:
- name: workers
minNode: 2
maxNode: 5
machineTypes: [n1-standard-2]
instanceType: OnDemand
provisioner:
type: selfHosted
selfHosted:
accountId: my-project
credentials:
type: Vault

astroctl infra k8s template selfHosted prints the AWS form of this manifest; change provider, region and machineTypes.

To remove what the setup created, delete the cluster first, then:

Terminal
astroctl cloud gcp selfHosted cleanup --cluster-name prod

It removes exactly what the setup made, by the names it recorded: the role bindings, the service account and its keys, the bucket and its contents, the key file, and the key stored in AstroPulse. Anything else by those names is kept and reported. --show-commands prints the commands without running them; the Connect step shows them under Remove what the setup created, to run in Cloud Shell. Every flag: astroctl cloud gcp selfHosted.

Settings​

The fields under spec.provisioner.selfHosted on Google Cloud. The control plane and node groups: Self-hosted Kubernetes.

Project IDaccountIdRequired

The Google Cloud project the cluster is created in.

Accepts
A project ID: 6 to 30 lowercase letters, digits or hyphens, starting with a letter.
credentialsRequired
Accepts
type: Vault, the key stored with selfHosted connect.
Kubernetes VersionkubernetesVersion
Accepts
An exact major.minor.patch from the supported list.
Default
AstroPulse's default version.
See
astroctl cloud gcp k8s-versions --self-hosted
Network IDnetworkId

An existing VPC network to create the cluster in. Left out, a new network is created.

Accepts
The network's name: 1 to 63 lowercase letters, digits or hyphens.
bucketName

The Cloud Storage bucket that holds the cluster's state, in the cluster's project and region.

Default
The bucket the setup created.

If it's refused​

"invalid accountId for gcp: a GCP project ID is 6-30 lowercase letters, digits or hyphens, starting with a letter"
Cause
The project ID has another character, or is the project number.
Fix
Copy it from gcloud config get-value project.
"invalid networkId for GCP: must be 1-63 lowercase letters, digits, or hyphens (e.g., my-vpc-network)"
Cause
networkId is not a network name.
Fix
Use the network's name, not its ID or URL.
"The key file isn't a valid service account key (it may be truncated or edited): download it again from the setup, then connect again."
Cause
The file is not a whole service account key.
Fix
Download the key again and pass its path to --credentials.
"This key is for project other-project, not my-project: send a key for a service account in project my-project."
Cause
The key belongs to a service account in another project.
Fix
Send a key for a service account in the cluster's project.
"No self-hosted setup for cluster prod in organization …: this key is for …. Sign in to the organization that started the setup, or start the setup here first; to store a key you made yourself, connect with --without-setup."
Cause
astroctl is signed in to another organization than the one that started the setup, or no setup was started.
Fix
Sign in to the organization that started the setup, or add --without-setup for a key you made yourself.
"AstroPulse couldn't confirm which account this key belongs to; try again in a minute."
Cause
Google Cloud did not answer who the key belongs to. Nothing was stored.
Fix
Run connect again in a minute.

Next​