GCP

GCP Self-Hosted (Vanilla) K8s Cluster Provisioner

The Astro platform deploys a Kubernetes cluster on GCP instances on your behalf, allowing you to select the instance type, size, zone, region, and other configurations. This approach is beneficial when you require a cluster that is not managed by the cloud provider, providing you with full control over the cluster, including the control plane.

Prerequisites

To create a self-hosted cluster on GCP, ensure the following CLI tools are installed on your local machine:

• GCP CLI for GCP cluster communication
• kubectl (optional) for GCP cluster interaction
• astroctl for cluster management

There are three pre-requisites to create a self-hosted cluster on GCP

Setup Service Account & Permissions

To configure the service account, follow these steps:

1. Open the Google Cloud Console.
2. Navigate to "IAM & Admin" > "Service Accounts".
3. Click "Create Service Account".
4. Enter a name and description, then click "Create".
5. Add the following roles:

Role	Description
Viewer	roles/viewer
Storage Admin	roles/storage.admin
Compute Admin	roles/compute.admin
Kubernetes Engine Admin	roles/container.admin
Service Account User	roles/iam.serviceAccountUser
Service Account Admin	roles/iam.serviceAccountAdmin
Project IAM Admin	roles/resourcemanager.projectIamAdmin
Artifact Registry Admin	roles/artifactregistry.admin

6. Click "Continue" and then "Done".
7. Click on the new service account name.
8. Go to the "Keys" tab and click "Add Key" > "Create New Key".
9. Select "JSON" and click "Create" to download the key file.

Keep this JSON key file secure for authenticating the Astro platform and managing GCP resources.

GCP Permissions

The astro platform internally use tools like cluster-api, kops or kubeadm to provision the kubernetes cluster. These tools require additional permissions to create and manage resources like GCP resources. Also, the astro platform will store the json file in encryted format in the filesystem to prevent unauthorized access.

Enable Required APIs

To enable the required APIs, follow these steps:

1. Open the Google Cloud Console

2. Select your project from the project selector

3. Navigate to "APIs & Services" > "Library"

4. Search for and enable the following APIs:

   API Name	API ID
   Compute Engine	compute.googleapis.com
   Kubernetes Engine	container.googleapis.com
   Cloud Resource Manager	cloudresourcemanager.googleapis.com
   Container Registry	containerregistry.googleapis.com
   Artifact Registry	artifactregistry.googleapis.com

5. For each API:

   • Type the API name in the search bar
   • Click on the API from the results
   • Click the "Enable" button

Alternatively, you can enable these APIs using the gcloud command-line tool with the following commands:

gcloud services enable compute.googleapis.com
gcloud services enable container.googleapis.com
gcloud services enable cloudresourcemanager.googleapis.com
gcloud services enable containerregistry.googleapis.com
gcloud services enable artifactregistry.googleapis.com

Setup GCS Bucket

To create a Google Cloud Storage (GCS) bucket for your self-hosted GCP cluster:

1. Navigate to the Google Cloud Console (https://console.cloud.google.com/).
2. In the left sidebar, expand "Storage" and click on "Buckets".
3. Click the "+ Create" button at the top of the page.
4. Choose a globally unique name for your bucket.
5. Select the same region as your planned cluster deployment for optimal performance.
6. Configure other settings as needed (e.g., storage class, access control).
7. Review your settings and click "Create" to provision the bucket.

Remember to note down the bucket name, as you'll need it when configuring your cluster provisioner.

Pre-requisite Validation

The astroctl CLI will automatically validate all pre-requisites before creating the cluster, ensuring that your environment is properly configured.

API Specification

The selfHosted type represents the spec for the GCP self-hosted provisioner.

Field	Description	Required
accountId	The account ID in the cloud provider. For GCP, it is the project ID.	Yes
networkId	The network ID (VPC) in the cloud provider.	No
credentials	The cloud provider credentials. Only static credentials are supported and must provide the path to the JSON key file.	Yes
bucketName	The bucket name in the cloud provider. Make sure the bucket is in the same region as the cluster.	Yes

The credentails will be used to authenticate the astro platform and other tools when managing your GCP resources.

BYOA (Bring Your Own Account)

provider: gcp
region: us-west2
provisioner:
  type: selfHosted
  selfHosted:
    accountId: <project-id>
    bucketName: <bucket-name> # this bucket should be created in the same region as the cluster
    credentials:
      type: static
      data:
        GOOGLE_APPLICATION_CREDENTIALS: <path-to-key-file> # path to the JSON key file

Static Credentials support only

Make sure to pass the file path to the JSON key file in the GOOGLE_APPLICATION_CREDENTIALS environment variable.

Bring Your Own VPC (BYOVPC)

provider: gcp
region: us-west2
provisioner:
  type: selfHosted
  selfHosted:
    accountId: <project-id>
    networkId: <vpc-id>
    credentials:
      type: static
      data:
        GOOGLE_APPLICATION_CREDENTIALS: <path-to-key-file> # path to the JSON key file
    bucketName: <bucket-name> # this bucket should be created in the same region as the cluster

Static Credentials support only

Make sure to pass the file path to the JSON key file in the GOOGLE_APPLICATION_CREDENTIALS environment variable.

KubeConfig

With simple command like below, you can set the context to the cluster and have the access to the cluster. You need to make sure to install the kubectl CLI as required. The kubeconfig will be valid for 1 hour by default and you need to generate the kubeconfig again with the new duration.

astroctl clusters generate-kubeconfig <cluster-name> --duration <duration>

and you can set the context to the cluster with the following command:

astroctl clusters set-context <cluster-name>

The cluster uses the client authentication type mutual TLS authentication to authenticate the kubeconfig. The platform lets you to configure the duration for the kubeconfig to be valid. Only the platform admin or admin of the organization can export the kubeconfig.