Give a workload access to Google Cloud

What do I need to connect my app api in namespace shop on my-cluster to the GCS bucket acme-data?

Ask Nova

Give DNS Automation on my-cluster access to the Cloud DNS zone example-com.

Ask Nova

Turn on workload identity for my registered GKE cluster my-cluster and its node pools.

Ask Nova

This is the hard part, and Nova does it with you. Name your app or add-on, the cluster and the resource: Nova reads the cluster and the workload's settings, and writes the identity, the grant, the trust and the values for your app, ready to run.

Each recipe gives one workload on a GKE cluster, an add-on or one of your applications, access to one Google Cloud resource: a GCS bucket, an Artifact Registry repository, a Cloud DNS zone or a Pub/Sub subscription. A pod signs in as a Google principal trusted only by its own namespace and ServiceAccount on this cluster, so no Google Cloud key is stored on the cluster. Each grant reaches one resource, and every recipe has a check you can run and an undo that removes what it created.

Before you start

• The Google Cloud CLI signed in with rights to create identities and grant access, kubectl pointed at the cluster, and astroctl signed in.
• Workload identity on the cluster. Clusters AstroPulse creates have it; a registered cluster turns it on once.

In the console

1. Open the cluster from the cluster list. On its Overview, Cloud identity shows who the cluster is in Google Cloud: under Per application (recommended), Workload identity and the Workload identity pool your trust policy names, and under All nodes, the identity the nodes run as.
2. Create the identity and its grant with the recipe below for your add-on or application.
3. Open the cluster's Add-ons tab and open the add-on, then Update and Change configuration. For DNS Automation and Model Serving, set Cloud access to Workload identity; Certificates and Vulnerability Scanning take Identity directly. On the Identity step, pick the Cloud and enter the identity the recipe printed. Go to Review, select Continue to apply once the dry run passes, type the add-on's name (such as external-dns), then select Update external-dns.

You should see

Each recipe's check prints the identity the workload signs in as, then reads the resource. The add-on's health shows no access warning.

Recipes

Your own applications:

• Give my application access to a GCS bucket
• Give my application access to a Pub/Sub subscription
• Let GKE nodes pull from an Artifact Registry repository in another project
• Pull from any registry with a pull Secret, on any cloud

Platform add-ons:

• Let vulnerability scanning read private images
• Let DNS Automation change records in Cloud DNS
• Let Certificates add DNS challenge records in Cloud DNS
• Let models read a private GCS bucket

Turn on workload identity on a registered cluster

Clusters AstroPulse created have it: skip this. A registered cluster does not report it in the console, so check it with the command in step 2. On GKE this is a workload identity pool on the cluster, and GKE_METADATA on every node pool your workloads run on. Background: Use Workload Identity Federation for GKE.

1. Set your values

Terminal

export CLUSTER=my-cluster
export LOCATION=us-central1
export PROJECT_ID=my-project

2. Check

Terminal

gcloud container clusters describe "${CLUSTER}" --location "${LOCATION}" --project "${PROJECT_ID}" \
  --format='value(workloadIdentityConfig.workloadPool)'
gcloud container node-pools list --cluster "${CLUSTER}" --location "${LOCATION}" --project "${PROJECT_ID}" \
  --format='table(name,config.workloadMetadataConfig.mode)'

Expected when on: my-project.svc.id.goog, then GKE_METADATA for every pool. Nothing printed for the cluster, or GCE_METADATA or a blank mode for a pool, means it is off there.

3. Turn it on

Replace POOL_NAME with each pool that is off. Updating a pool recreates its nodes one at a time, so its pods move: do it in a quiet window.

Terminal

gcloud container clusters update "${CLUSTER}" --location "${LOCATION}" --project "${PROJECT_ID}" \
  --workload-pool="${PROJECT_ID}.svc.id.goog"
gcloud container node-pools update POOL_NAME --cluster "${CLUSTER}" --location "${LOCATION}" \
  --project "${PROJECT_ID}" --workload-metadata=GKE_METADATA

Run step 2 again to confirm.

Undo. Every workload that reaches Google Cloud this way stops working, and the pool's nodes are recreated.

Terminal

gcloud container node-pools update POOL_NAME --cluster "${CLUSTER}" --location "${LOCATION}" \
  --project "${PROJECT_ID}" --workload-metadata=GCE_METADATA
gcloud container clusters update "${CLUSTER}" --location "${LOCATION}" --project "${PROJECT_ID}" \
  --disable-workload-identity

Placeholders used on this page. Replace each with your own value.

Placeholder	Meaning
CLUSTER	Cluster name
LOCATION	Google Cloud location of the cluster or repository
PROJECT_ID	Google Cloud project ID of the cluster
PROJECT_NUMBER	Google Cloud project number of the cluster (gcloud projects describe PROJECT_ID --format='value(projectNumber)')
NAMESPACE	Kubernetes namespace of the workload
SERVICE_ACCOUNT	Kubernetes ServiceAccount name of the workload
BUCKET	Bucket name (S3 or GCS)
REPOSITORY, REGISTRY	Image repository (ECR, Artifact Registry) or Azure registry name
APPLICATION, APP, POD, SERVICE	Name of the Astro application, its Deployment, a pod, and a Kubernetes Service in the recipe's namespace
SUBSCRIPTION, TOPIC, QUEUE_NAME, QUEUE_URL, VAULT, SECRET_NAME	Names of the Pub/Sub subscription and topic, SQS queue, Key Vault and secret you are granting access to
ROLE_ARN, CLIENT_ID, PRINCIPAL_ID, TENANT_ID, SUBSCRIPTION_ID	Values printed by earlier commands in the same recipe

The recipes bind a Google principal directly to the ServiceAccount where Google supports it ("direct" access, no Google service account). The principal looks like:

Output

principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/NAMESPACE/sa/SERVICE_ACCOUNT

Your own applications

Bind the identity on the ServiceAccount your workload already runs as. Where that is defined depends on the application source: see How your app gets its identity.

Reading a bucket

Give my application access to a GCS bucket

Do I need to do anything?

Your situation	What to do
Any cluster	Manual: the steps below

Before you start. Workload identity must be on for the cluster and its node pools: Enable workload identity on GKE. One-line check: gcloud container clusters describe CLUSTER --location LOCATION --format='value(workloadIdentityConfig.workloadPool)' prints PROJECT_ID.svc.id.goog.

Values you need. Find each one with the command shown.

Placeholder	What it is	Command that finds it
CLUSTER	Your cluster name	gcloud container clusters list
LOCATION	Location (region or zone) of the cluster, and of an Artifact Registry repository	gcloud container clusters list --format="table(name,location)"
PROJECT_ID	Project ID of the cluster	gcloud config get-value project
PROJECT_NUMBER	Project number of the cluster	gcloud projects describe PROJECT_ID --format='value(projectNumber)'
BUCKET	GCS bucket	gcloud storage buckets list --format='value(name)'
APPNS	Your workload's namespace and ServiceAccount (empty means the app runs as default: give it its own ServiceAccount first, see how)	kubectl get deploy -A kubectl -n NAMESPACE get deploy APP -o jsonpath='{.spec.template.spec.serviceAccountName}'

Steps.

1. Look up the cluster's project number. This changes nothing; it prints PROJECT_NUMBER for the next step.

Terminal

gcloud projects describe PROJECT_ID --format='value(projectNumber)'

2. Grant read on one bucket. This lets the workload's service account read objects in that bucket and nowhere else. No annotation is needed. To cover every bucket in a project instead, use gcloud projects add-iam-policy-binding PROJECT_ID with the same --role and --member.

Terminal

gcloud storage buckets add-iam-policy-binding gs://BUCKET \
  --role roles/storage.objectViewer \
  --member "principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/NAMESPACE/sa/SERVICE_ACCOUNT"

Validate. Three checks, in order. Stop at the first one that fails.

Check 1: the cluster is enabled (Enable workload identity on GKE).

Terminal

gcloud container clusters describe CLUSTER --location LOCATION \
  --format='value(workloadIdentityConfig.workloadPool)'
gcloud container node-pools describe POOL --cluster CLUSTER --location LOCATION \
  --format='value(config.workloadMetadataConfig.mode)'

Expected: PROJECT_ID.svc.id.goog, and GKE_METADATA for every pool the workload can run on. If either is empty or different, stop and go to Enable workload identity on GKE.

Check 2: the identity works for this ServiceAccount. The ServiceAccount must already exist in the namespace. Run a throwaway pod as the ServiceAccount and ask the metadata server who it is.

Terminal

kubectl run wi-test -n NAMESPACE --rm -it --restart=Never --image=google/cloud-sdk:slim \
  --override-type=strategic --overrides='{"spec":{"serviceAccountName":"SERVICE_ACCOUNT"}}' \
  -- curl -s -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email

Expected: PROJECT_ID.svc.id.goog, the workload identity pool. If it prints the email of a service account instead (ending .iam.gserviceaccount.com), the pod is running as the node's identity, not as the ServiceAccount's; a token request would still succeed, so do not use one as the check.

Check 2b (optional): a registry accepts the identity. Request a token as the same pod (replace the last line of the command above with -- gcloud auth print-access-token) and test a registry with it: LOCATION-docker.pkg.dev/v2/PROJECT_ID/REPOSITORY/IMAGE/tags/list with the header Authorization: Bearer TOKEN returns HTTP 200.

If the namespace enforces the Pod Security "restricted" level and rejects the pod, use the override below that adds a securityContext, or run the test in a namespace that allows it with a ServiceAccount bound identically (same annotation, and a trust or grant that names that namespace).

Restricted-namespace override (also add --env HOME=/tmp, because gcloud writes its config under HOME): {"spec":{"serviceAccountName":"SERVICE_ACCOUNT","securityContext":{"runAsNonRoot":true,"runAsUser":65534,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"wi-test","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]}}

An email ending .iam.gserviceaccount.com, or default

Cause

No workload identity reached the pod: it is running as the node's identity. Workload identity is off on the cluster or node pool (go to Check 1), or the pod is not using the ServiceAccount you grant.

Permission '...' denied on resource in Check 2b or the workload

Cause

The identity is working; the role or scope on the resource is wrong (the grant step).

If this does not print PROJECT_ID.svc.id.goog, the binding is wrong; fix it before looking at the workload.

Check 3: the workload does its job.

Run the same "who am I" check from your application's own pod, then the call your application makes.

Terminal

kubectl -n NAMESPACE run cloud-check --rm -it --restart=Never \
  --image=google/cloud-sdk:slim \
  --override-type=strategic --overrides='{"spec":{"serviceAccountName":"SERVICE_ACCOUNT"}}' \
  -- gcloud storage ls gs://BUCKET/

Expected: the call succeeds.

Then confirm your application's own call succeeds (its logs show no access error).

TROUBLESHOOT.

403 ... does not have storage.objects.list access to the Google Cloud Storage bucket

Fix

The member names the wrong project number, namespace or ServiceAccount (step 2). IAM changes can take a few minutes.

DENIED: Unauthenticated request or Your default credentials were not found

Fix

Workload identity is not enabled on the cluster or node pool. See Enable workload identity on GKE.

Undo. To remove this access:

Terminal

gcloud storage buckets remove-iam-policy-binding gs://BUCKET \
  --role roles/storage.objectViewer \
  --member "principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/NAMESPACE/sa/SERVICE_ACCOUNT"

This removes only that grant.

Calling a queue, topic or vault

The steps are the same as the bucket recipe above; only the grant changes: the role, scoped to one resource.

Give my application access to a Pub/Sub subscription

Do I need to do anything?

Your situation	What to do
Any cluster	Manual: the steps below

Before you start. Workload identity must be on for the cluster and its node pools: Enable workload identity on GKE. One-line check: gcloud container clusters describe CLUSTER --location LOCATION --format='value(workloadIdentityConfig.workloadPool)' prints PROJECT_ID.svc.id.goog.

Values you need. Find each one with the command shown.

Placeholder	What it is	Command that finds it
CLUSTER	Your cluster name	gcloud container clusters list
LOCATION	Location (region or zone) of the cluster, and of an Artifact Registry repository	gcloud container clusters list --format="table(name,location)"
PROJECT_ID	Project ID of the cluster	gcloud config get-value project
PROJECT_NUMBER	Project number of the cluster	gcloud projects describe PROJECT_ID --format='value(projectNumber)'
SUB	Pub/Sub subscription	gcloud pubsub subscriptions list --project PROJECT_ID
APPNS	Your workload's namespace and ServiceAccount (empty means the app runs as default: give it its own ServiceAccount first, see how)	kubectl get deploy -A kubectl -n NAMESPACE get deploy APP -o jsonpath='{.spec.template.spec.serviceAccountName}'

Steps.

1. Look up the cluster's project number. This changes nothing; it prints PROJECT_NUMBER for the next step.

Terminal

gcloud projects describe PROJECT_ID --format='value(projectNumber)'

2. Grant subscribe on one subscription. This lets the workload's service account pull and acknowledge messages from that subscription only.

Terminal

gcloud pubsub subscriptions add-iam-policy-binding SUBSCRIPTION \
  --project PROJECT_ID \
  --role roles/pubsub.subscriber \
  --member "principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/NAMESPACE/sa/SERVICE_ACCOUNT"

3. To publish, grant publish on one topic. This lets the same service account publish to that topic only. Skip it if the workload only reads.

Terminal

gcloud pubsub topics add-iam-policy-binding TOPIC \
  --project PROJECT_ID \
  --role roles/pubsub.publisher \
  --member "principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/NAMESPACE/sa/SERVICE_ACCOUNT"

Validate. Three checks, in order. Stop at the first one that fails.

Check 1: the cluster is enabled (Enable workload identity on GKE).

Terminal

gcloud container clusters describe CLUSTER --location LOCATION \
  --format='value(workloadIdentityConfig.workloadPool)'
gcloud container node-pools describe POOL --cluster CLUSTER --location LOCATION \
  --format='value(config.workloadMetadataConfig.mode)'

Expected: PROJECT_ID.svc.id.goog, and GKE_METADATA for every pool the workload can run on. If either is empty or different, stop and go to Enable workload identity on GKE.

Check 2: the identity works for this ServiceAccount. The ServiceAccount must already exist in the namespace. Run a throwaway pod as the ServiceAccount and ask the metadata server who it is.

Terminal

kubectl run wi-test -n NAMESPACE --rm -it --restart=Never --image=google/cloud-sdk:slim \
  --override-type=strategic --overrides='{"spec":{"serviceAccountName":"SERVICE_ACCOUNT"}}' \
  -- curl -s -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email

Expected: PROJECT_ID.svc.id.goog, the workload identity pool. If it prints the email of a service account instead (ending .iam.gserviceaccount.com), the pod is running as the node's identity, not as the ServiceAccount's; a token request would still succeed, so do not use one as the check.

Check 2b (optional): a registry accepts the identity. Request a token as the same pod (replace the last line of the command above with -- gcloud auth print-access-token) and test a registry with it: LOCATION-docker.pkg.dev/v2/PROJECT_ID/REPOSITORY/IMAGE/tags/list with the header Authorization: Bearer TOKEN returns HTTP 200.

If the namespace enforces the Pod Security "restricted" level and rejects the pod, use the override below that adds a securityContext, or run the test in a namespace that allows it with a ServiceAccount bound identically (same annotation, and a trust or grant that names that namespace).

Restricted-namespace override (also add --env HOME=/tmp, because gcloud writes its config under HOME): {"spec":{"serviceAccountName":"SERVICE_ACCOUNT","securityContext":{"runAsNonRoot":true,"runAsUser":65534,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"wi-test","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]}}

An email ending .iam.gserviceaccount.com, or default

Cause

No workload identity reached the pod: it is running as the node's identity. Workload identity is off on the cluster or node pool (go to Check 1), or the pod is not using the ServiceAccount you grant.

Permission '...' denied on resource in Check 2b or the workload

Cause

The identity is working; the role or scope on the resource is wrong (the grant step).

If this does not print PROJECT_ID.svc.id.goog, the binding is wrong; fix it before looking at the workload.

Check 3: the workload does its job.

Run the same "who am I" check from your application's own pod, then the call your application makes.

Terminal

kubectl -n NAMESPACE run cloud-check --rm -it --restart=Never \
  --image=google/cloud-sdk:slim \
  --override-type=strategic --overrides='{"spec":{"serviceAccountName":"SERVICE_ACCOUNT"}}' \
  -- gcloud pubsub subscriptions pull SUBSCRIPTION --project PROJECT_ID --limit 1

Expected: the call succeeds. pull without --auto-ack does not remove messages.

Then confirm your application's own call succeeds (its logs show no access error).

TROUBLESHOOT.

PERMISSION_DENIED: User not authorized to perform this action. or Permission 'pubsub.subscriptions.consume' denied on resource

Fix

The grant is missing, or the member names the wrong project number, namespace or ServiceAccount (step 2). IAM changes can take a few minutes.

NOT_FOUND: Resource not found

Fix

SUBSCRIPTION or PROJECT_ID is wrong.

DENIED: Unauthenticated request or default credentials not found

Fix

Workload identity is off on the cluster or node pool. See Enable workload identity on GKE.

Undo. To remove this access:

Terminal

gcloud pubsub subscriptions remove-iam-policy-binding SUBSCRIPTION --project PROJECT_ID \
  --role roles/pubsub.subscriber \
  --member "principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/NAMESPACE/sa/SERVICE_ACCOUNT"

This removes only that grant.

Pulling images from a private registry

Image pulls are done by the node, not by the pod's ServiceAccount (see why). Grant the node identity access, or use an pull Secret.

Let GKE nodes pull from an Artifact Registry repository in another project

Nodes pull images with the node pool's service account, not with workload identity. For a repository in another project, grant that service account read on the repository. Background: Artifact Registry access control.

1. Set your values

Terminal

export REPOSITORY=apps
export REGISTRY_PROJECT_ID=acme-images
export LOCATION=us-central1
export NODE_SERVICE_ACCOUNT=123456789012-compute@developer.gserviceaccount.com

NODE_SERVICE_ACCOUNT: on the cluster's Overview (https://astropulse.io/console?type=clusters&cluster=my-cluster), Cloud identity › All nodes shows it (All node pools, or one row per node pool). From a terminal: gcloud container node-pools describe POOL --cluster CLUSTER --location CLUSTER_LOCATION --format='value(config.serviceAccount)' (default is PROJECT_NUMBER-compute@developer.gserviceaccount.com).

2. Grant read on the one repository

Terminal

gcloud artifacts repositories add-iam-policy-binding "${REPOSITORY}" \
  --project "${REGISTRY_PROJECT_ID}" --location "${LOCATION}" \
  --role roles/artifactregistry.reader --member "serviceAccount:${NODE_SERVICE_ACCOUNT}"

The node pool also needs an access scope that reads Google Cloud APIs (cloud-platform, or at least devstorage.read_only).

3. Check

Deploy a workload with an image from the repository, then:

Terminal

kubectl get events -A --field-selector reason=Pulled

Expected: a Successfully pulled image "LOCATION-docker.pkg.dev/REGISTRY_PROJECT_ID/REPOSITORY/IMAGE:TAG" event.

If it fails

• ImagePullBackOff with 403 Forbidden in kubectl describe pod: the node service account has no reader role on that repository (step 2), or the pool's access scopes are too narrow.

Undo. Removes only this grant.

Terminal

gcloud artifacts repositories remove-iam-policy-binding "${REPOSITORY}" \
  --project "${REGISTRY_PROJECT_ID}" --location "${LOCATION}" \
  --role roles/artifactregistry.reader --member "serviceAccount:${NODE_SERVICE_ACCOUNT}"

Platform add-ons

Each add-on runs as one fixed ServiceAccount. You give that account access, then name the identity on the add-on's identity field. The field has this shape on every add-on that takes it:

values.yaml

identity:
  serviceAccountAnnotations: {}   # the annotation the recipe prints
  serviceAccountLabels: {}        # Azure only: azure.workload.identity/use: "true"

Add-on	Namespace	ServiceAccount	Needs
Vulnerability Scanning (vulnerability-scanning)	astro-security	trivy-operator	Read images in one registry
DNS Automation (external-dns)	external-dns	external-dns	Change records in one DNS zone
Certificates (cert-manager)	cert-manager	cert-manager	Add and remove TXT records in one DNS zone (DNS-01 check only)
Model Serving (model-serving)	each model's own namespace	astro-model-APPLICATION for each model	Read the model bucket

Applying an add-on's identity is covered in Managing cluster add-ons.

Vulnerability Scanning

The scanner reads each image from its registry through the registry's API, so it needs read access to that registry. It runs as trivy-operator in astro-security. Clusters AstroPulse created get this access when the add-on is applied. Registered clusters, and registries in another account, project or subscription, need the steps below.

Let vulnerability scanning read private images

Do I need to do anything?

Your situation	What to do
Registry in the cluster's own project, cluster created by AstroPulse	Automatic, when you apply the add-on
Registered cluster, or a repository in another project	Manual: the steps below

Clusters created by AstroPulse: nothing to do for registries in the cluster's own project. Workload identity is enabled, and the add-on's read access to Artifact Registry is granted when you apply it. The steps below are for clusters you registered, and for a repository in another project (always manual). A cluster created earlier gets access the next time you apply the add-on, and that re-apply is not automatic.

Before you start. Workload identity must be on for the cluster and its node pools: Enable workload identity on GKE. One-line check: gcloud container clusters describe CLUSTER --location LOCATION --format='value(workloadIdentityConfig.workloadPool)' prints PROJECT_ID.svc.id.goog.

Values you need. Find each one with the command shown.

Placeholder	What it is	Command that finds it
CLUSTER	Your cluster name	gcloud container clusters list
LOCATION	Location (region or zone) of the cluster, and of an Artifact Registry repository	gcloud container clusters list --format="table(name,location)"
PROJECT_ID	Project ID of the cluster	gcloud config get-value project
PROJECT_NUMBER	Project number of the cluster	gcloud projects describe PROJECT_ID --format='value(projectNumber)'
REGISTRY_PROJECT_ID	Project that holds the repository (in the image reference LOCATION-docker.pkg.dev/PROJECT/REPOSITORY/IMAGE)	gcloud artifacts repositories list --project REGISTRY_PROJECT_ID
REPOSITORY	Artifact Registry repository, from the image reference LOCATION-docker.pkg.dev/PROJECT/REPOSITORY/IMAGE	gcloud artifacts repositories list --project REGISTRY_PROJECT_ID
IMAGES	Images the cluster runs	kubectl get pods -A -o jsonpath='{..image}' | tr ' ' '\n' | sort -u
NAMESPACE, SERVICE_ACCOUNT	Fixed for this add-on: astro-security and trivy-operator	kubectl -n astro-security get sa trivy-operator
Current add-on configuration and health	The add-on as applied	astroctl infra k8s addons get CLUSTER vulnerability-scanning astroctl infra k8s addons get CLUSTER vulnerability-scanning --manifest

On clusters created by AstroPulse, skip to check 3 (the workload does its job).

Steps.

1. Look up the cluster's project number. This changes nothing; it prints PROJECT_NUMBER for the next step.

Terminal

gcloud projects describe PROJECT_ID --format='value(projectNumber)'

2. Grant read on one repository. This lets the scanner's service account read images in that repository and nowhere else. PROJECT_ID and PROJECT_NUMBER name the cluster's project, and --project is where the images live, which is how the grant reaches a registry in another project. To cover every repository in a project instead, use gcloud projects add-iam-policy-binding PROJECT with the same --role and --member.

Terminal

gcloud artifacts repositories add-iam-policy-binding REPOSITORY \
  --project REGISTRY_PROJECT_ID --location LOCATION \
  --role roles/artifactregistry.reader \
  --member "principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/astro-security/sa/trivy-operator"

3. Write the add-on manifest. This is only a file on your machine. No identity is needed in the K8sClusterAddon manifest on this path.

vulnerability-scanning.yaml

apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: vulnerability-scanning
spec:
  clusterName: CLUSTER

4. Apply the add-on. This installs or updates the add-on with the identity and settings in the file.

Terminal

astroctl infra k8s addons apply -f vulnerability-scanning.yaml

Validate. Three checks, in order. On clusters created by AstroPulse, skip to check 3 (the workload does its job). Stop at the first one that fails.

Check 1: the cluster is enabled (Enable workload identity on GKE).

Terminal

gcloud container clusters describe CLUSTER --location LOCATION \
  --format='value(workloadIdentityConfig.workloadPool)'
gcloud container node-pools describe POOL --cluster CLUSTER --location LOCATION \
  --format='value(config.workloadMetadataConfig.mode)'

Expected: PROJECT_ID.svc.id.goog, and GKE_METADATA for every pool the workload can run on. If either is empty or different, stop and go to Enable workload identity on GKE.

Check 2: the identity works for this ServiceAccount. For an add-on, run this after you apply it: the add-on creates the ServiceAccount. Run a throwaway pod as the ServiceAccount and ask the metadata server who it is.

Terminal

kubectl run wi-test -n astro-security --rm -it --restart=Never --image=google/cloud-sdk:slim \
  --override-type=strategic --overrides='{"spec":{"serviceAccountName":"trivy-operator"}}' \
  -- curl -s -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email

Expected: PROJECT_ID.svc.id.goog, the workload identity pool. If it prints the email of a service account instead (ending .iam.gserviceaccount.com), the pod is running as the node's identity, not as the ServiceAccount's; a token request would still succeed, so do not use one as the check.

Check 2b (optional): a registry accepts the identity. Request a token as the same pod (replace the last line of the command above with -- gcloud auth print-access-token) and test a registry with it: LOCATION-docker.pkg.dev/v2/PROJECT_ID/REPOSITORY/IMAGE/tags/list with the header Authorization: Bearer TOKEN returns HTTP 200.

If the namespace enforces the Pod Security "restricted" level and rejects the pod, use the override below that adds a securityContext, or run the test in a namespace that allows it with a ServiceAccount bound identically (same annotation, and a trust or grant that names that namespace).

Restricted-namespace override (also add --env HOME=/tmp, because gcloud writes its config under HOME): {"spec":{"serviceAccountName":"trivy-operator","securityContext":{"runAsNonRoot":true,"runAsUser":65534,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"wi-test","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]}}

An email ending .iam.gserviceaccount.com, or default

Cause

No workload identity reached the pod: it is running as the node's identity. Workload identity is off on the cluster or node pool (go to Check 1), or the pod is not using the ServiceAccount you grant.

Permission '...' denied on resource in Check 2b or the workload

Cause

The identity is working; the role or scope on the resource is wrong (the grant step).

If this does not print PROJECT_ID.svc.id.goog, the binding is wrong; fix it before looking at the workload.

Check 3: the workload does its job.

Terminal

kubectl get vulnerabilityreports -A
astroctl infra k8s addons get CLUSTER vulnerability-scanning

Expected: a report for the private image, and no "images not scannable" note in the add-on's health. The first scan of an image takes a few minutes. After a new grant allow up to about 5 minutes for the cloud to apply it, then restart the workload (kubectl rollout restart deployment/NAME -n NAMESPACE) to scan it again now; otherwise a refused image is retried within 10 hours.

TROUBLESHOOT.

DENIED: Unauthenticated request in the scan Job

Fix

The grant is missing, or workload identity is not enabled on the cluster or node pool. Check the member string in step 2, then see Enable workload identity on GKE.

Permission 'artifactregistry.repositories.downloadArtifacts' denied

Fix

The identity works; the role is on a different repository or project than the image. Grant it on the right one.

N images not scannable: registry denied (...) in the add-on's health

Fix

The scanner cannot read that repository. Grant it (the grant step above), then restart the workload to scan it again.

Undo. To remove this access:

Terminal

gcloud artifacts repositories remove-iam-policy-binding REPOSITORY --project REGISTRY_PROJECT_ID --location LOCATION \
  --role roles/artifactregistry.reader \
  --member "principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/astro-security/sa/trivy-operator"

This removes only that grant.

DNS Automation (external-dns)

DNS Automation runs as external-dns in external-dns. Without cloud access it installs, runs healthy, and publishes nothing.

Let DNS Automation change records in Cloud DNS

Do I need to do anything?

Your situation	What to do
Any GKE cluster	Manual: the steps below

Needed on every GKE cluster, including ones AstroPulse created: the platform does not set up DNS access for this add-on on GKE. Workload identity itself is already enabled on clusters AstroPulse created.

Before you start. Workload identity must be on for the cluster and its node pools: Enable workload identity on GKE. One-line check: gcloud container clusters describe CLUSTER --location LOCATION --format='value(workloadIdentityConfig.workloadPool)' prints PROJECT_ID.svc.id.goog.

Values you need. Find each one with the command shown.

Placeholder	What it is	Command that finds it
CLUSTER	Your cluster name	gcloud container clusters list
LOCATION	Location (region or zone) of the cluster, and of an Artifact Registry repository	gcloud container clusters list --format="table(name,location)"
PROJECT_ID	Project ID of the cluster	gcloud config get-value project
PROJECT_NUMBER	Project number of the cluster	gcloud projects describe PROJECT_ID --format='value(projectNumber)'
NAMESPACE, SERVICE_ACCOUNT	Fixed for this add-on: external-dns and external-dns	kubectl -n external-dns get sa external-dns
Current add-on configuration and health	The add-on as applied	astroctl infra k8s addons get CLUSTER external-dns astroctl infra k8s addons get CLUSTER external-dns --manifest

Steps.

1. Look up the cluster's project number. This changes nothing; it prints PROJECT_NUMBER for the next step.

Terminal

gcloud projects describe PROJECT_ID --format='value(projectNumber)'

2. Grant Cloud DNS access. This lets the add-on's service account change records in the zones of that project. gcloud has no per-zone binding, so keep the zones you publish to in a project of their own. PROJECT_ID is the project that holds the zone, which is the cluster's own project.

Terminal

gcloud projects add-iam-policy-binding PROJECT_ID \
  --role roles/dns.admin \
  --member "principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/external-dns/sa/external-dns"

3. Write the add-on manifest. This is only a file on your machine. No identity is needed on this path.

external-dns.yaml

apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: external-dns
spec:
  clusterName: CLUSTER
  configuration:
    externalDns:
      provider: gcp
      domainFilter: example.com

4. Apply the add-on. This installs or updates the add-on with the identity and settings in the file.

Terminal

astroctl infra k8s addons apply -f external-dns.yaml

Validate. Three checks, in order. Stop at the first one that fails.

Check 1: the cluster is enabled (Enable workload identity on GKE).

Terminal

gcloud container clusters describe CLUSTER --location LOCATION \
  --format='value(workloadIdentityConfig.workloadPool)'
gcloud container node-pools describe POOL --cluster CLUSTER --location LOCATION \
  --format='value(config.workloadMetadataConfig.mode)'

Expected: PROJECT_ID.svc.id.goog, and GKE_METADATA for every pool the workload can run on. If either is empty or different, stop and go to Enable workload identity on GKE.

Check 2: the identity works for this ServiceAccount. For an add-on, run this after you apply it: the add-on creates the ServiceAccount. Run a throwaway pod as the ServiceAccount and ask the metadata server who it is.

Terminal

kubectl run wi-test -n external-dns --rm -it --restart=Never --image=google/cloud-sdk:slim \
  --override-type=strategic --overrides='{"spec":{"serviceAccountName":"external-dns"}}' \
  -- curl -s -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email

Expected: PROJECT_ID.svc.id.goog, the workload identity pool. If it prints the email of a service account instead (ending .iam.gserviceaccount.com), the pod is running as the node's identity, not as the ServiceAccount's; a token request would still succeed, so do not use one as the check.

Check 2b (optional): a registry accepts the identity. Request a token as the same pod (replace the last line of the command above with -- gcloud auth print-access-token) and test a registry with it: LOCATION-docker.pkg.dev/v2/PROJECT_ID/REPOSITORY/IMAGE/tags/list with the header Authorization: Bearer TOKEN returns HTTP 200.

If the namespace enforces the Pod Security "restricted" level and rejects the pod, use the override below that adds a securityContext, or run the test in a namespace that allows it with a ServiceAccount bound identically (same annotation, and a trust or grant that names that namespace).

Restricted-namespace override (also add --env HOME=/tmp, because gcloud writes its config under HOME): {"spec":{"serviceAccountName":"external-dns","securityContext":{"runAsNonRoot":true,"runAsUser":65534,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"wi-test","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]}}

An email ending .iam.gserviceaccount.com, or default

Cause

No workload identity reached the pod: it is running as the node's identity. Workload identity is off on the cluster or node pool (go to Check 1), or the pod is not using the ServiceAccount you grant.

Permission '...' denied on resource in Check 2b or the workload

Cause

The identity is working; the role or scope on the resource is wrong (the grant step).

If this does not print PROJECT_ID.svc.id.goog, the binding is wrong; fix it before looking at the workload.

Check 3: the workload does its job.

Terminal

astroctl infra k8s addons get CLUSTER external-dns
astroctl infra k8s addons logs CLUSTER external-dns --level Error --follow=false --interactive=false
kubectl annotate service SERVICE external-dns.alpha.kubernetes.io/hostname=test.example.com
dig +short test.example.com

Expected: the add-on is healthy, the error log is empty (no 403), and within a minute or two the name resolves to the service's address.

TROUBLESHOOT.

Healthy, but no record appears

Fix

The member string names the wrong project number, namespace or ServiceAccount. It must end /subject/ns/external-dns/sa/external-dns.

403 from Cloud DNS in the error log

Fix

The grant is missing in the zone's project. Repeat step 2 there.

DENIED: Unauthenticated request or default credentials not found

Fix

Workload identity is off on the cluster or node pool. See Enable workload identity on GKE.

Undo. To remove this access:

Terminal

gcloud projects remove-iam-policy-binding PROJECT_ID \
  --role roles/dns.admin \
  --member "principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/external-dns/sa/external-dns"

This removes only that grant.

Certificates (cert-manager), DNS-01 check

Only the DNS-01 check (dnsSolver) calls your cloud. It runs as cert-manager in cert-manager, and needs enableClusterIssuer: true and issuerType: LetsEncrypt (or LetsEncryptStaging). A dnsSolver with no identity is refused when you apply.

Let Certificates add DNS challenge records in Cloud DNS

Do I need to do anything?

Your situation	What to do
Any GKE cluster	Manual: the steps below

Needed on every GKE cluster, including ones AstroPulse created: the platform does not set up DNS access for this add-on on GKE. Workload identity itself is already enabled on clusters AstroPulse created.

Before you start. Workload identity must be on for the cluster and its node pools: Enable workload identity on GKE. One-line check: gcloud container clusters describe CLUSTER --location LOCATION --format='value(workloadIdentityConfig.workloadPool)' prints PROJECT_ID.svc.id.goog.

Values you need. Find each one with the command shown.

Placeholder	What it is	Command that finds it
CLUSTER	Your cluster name	gcloud container clusters list
LOCATION	Location (region or zone) of the cluster, and of an Artifact Registry repository	gcloud container clusters list --format="table(name,location)"
PROJECT_ID	Project ID of the cluster	gcloud config get-value project
NAMESPACE, SERVICE_ACCOUNT	Fixed for this add-on: cert-manager and cert-manager	kubectl -n cert-manager get sa cert-manager
Current add-on configuration and health	The add-on as applied	astroctl infra k8s addons get CLUSTER cert-manager astroctl infra k8s addons get CLUSTER cert-manager --manifest

Steps.

1. Create a Google service account. This creates an account with no permissions yet.

Terminal

gcloud iam service-accounts create cert-manager-dns --project PROJECT_ID

2. Let it manage DNS. This lets that account change records in the zones of the project. gcloud has no per-zone binding, so keep the zones you validate in a project of their own.

Terminal

gcloud projects add-iam-policy-binding PROJECT_ID \
  --role roles/dns.admin \
  --member "serviceAccount:cert-manager-dns@PROJECT_ID.iam.gserviceaccount.com"

3. Let the add-on act as that account. This lets only the cert-manager service account in cert-manager use it.

Terminal

gcloud iam service-accounts add-iam-policy-binding \
  cert-manager-dns@PROJECT_ID.iam.gserviceaccount.com \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:PROJECT_ID.svc.id.goog[cert-manager/cert-manager]"

4. Write the add-on manifest. This is only a file on your machine. The DNS check is configured only when identity carries the annotation.

cert-manager.yaml

apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: cert-manager
spec:
  clusterName: CLUSTER
  configuration:
    certManager:
      enableClusterIssuer: true
      issuerType: LetsEncrypt
      email: ops@example.com
      identity:
        serviceAccountAnnotations:
          iam.gke.io/gcp-service-account: cert-manager-dns@PROJECT_ID.iam.gserviceaccount.com
      dnsSolver:
        provider: gcp
        project: PROJECT_ID

5. Apply the add-on. This installs or updates the add-on with the identity and settings in the file, after a dry run that reports refusals without changing anything.

Terminal

astroctl infra k8s addons apply -f cert-manager.yaml --dry-run
astroctl infra k8s addons apply -f cert-manager.yaml

Validate. Three checks, in order. Stop at the first one that fails.

Check 1: the cluster is enabled (Enable workload identity on GKE).

Terminal

gcloud container clusters describe CLUSTER --location LOCATION \
  --format='value(workloadIdentityConfig.workloadPool)'
gcloud container node-pools describe POOL --cluster CLUSTER --location LOCATION \
  --format='value(config.workloadMetadataConfig.mode)'

Expected: PROJECT_ID.svc.id.goog, and GKE_METADATA for every pool the workload can run on. If either is empty or different, stop and go to Enable workload identity on GKE.

Check 2: the identity works for this ServiceAccount. For an add-on, run this after you apply it: the add-on creates the ServiceAccount. Run a throwaway pod as the ServiceAccount and ask the metadata server who it is.

Terminal

kubectl run wi-test -n cert-manager --rm -it --restart=Never --image=google/cloud-sdk:slim \
  --override-type=strategic --overrides='{"spec":{"serviceAccountName":"cert-manager"}}' \
  -- curl -s -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email

Expected: PROJECT_ID.svc.id.goog, the workload identity pool. If it prints the email of a service account instead (ending .iam.gserviceaccount.com), the pod is running as the node's identity, not as the ServiceAccount's; a token request would still succeed, so do not use one as the check.

Check 2b (optional): a registry accepts the identity. Request a token as the same pod (replace the last line of the command above with -- gcloud auth print-access-token) and test a registry with it: LOCATION-docker.pkg.dev/v2/PROJECT_ID/REPOSITORY/IMAGE/tags/list with the header Authorization: Bearer TOKEN returns HTTP 200.

If the namespace enforces the Pod Security "restricted" level and rejects the pod, use the override below that adds a securityContext, or run the test in a namespace that allows it with a ServiceAccount bound identically (same annotation, and a trust or grant that names that namespace).

Restricted-namespace override (also add --env HOME=/tmp, because gcloud writes its config under HOME): {"spec":{"serviceAccountName":"cert-manager","securityContext":{"runAsNonRoot":true,"runAsUser":65534,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"wi-test","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]}}

An email ending .iam.gserviceaccount.com, or default

Cause

No workload identity reached the pod: it is running as the node's identity. Workload identity is off on the cluster or node pool (go to Check 1), or the pod is not using the ServiceAccount you grant.

Permission '...' denied on resource in Check 2b or the workload

Cause

The identity is working; the role or scope on the resource is wrong (the grant step).

If this does not print PROJECT_ID.svc.id.goog, the binding is wrong; fix it before looking at the workload.

Check 3: the workload does its job.

Terminal

astroctl infra k8s addons get CLUSTER cert-manager
kubectl apply -f dns-check.yaml
kubectl get certificate dns-check --watch

dns-check.yaml is a test certificate (below). Expected: challenges.dns01 in the tls block of get says the DNS check can issue, and the certificate becomes READY: True within a few minutes. It issues a real Let's Encrypt certificate and counts against its rate limits; delete it with kubectl delete certificate dns-check when you are done.

dns-check.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: dns-check
  namespace: default
  labels:
    platform.astropulse.io/challenge: dns01
spec:
  secretName: dns-check-tls
  dnsNames:
    - test.example.com
  issuerRef:
    name: astro-letsencrypt
    kind: ClusterIssuer

TROUBLESHOOT. On failure run kubectl describe challenge -n default.

Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist)

Fix

The roles/iam.workloadIdentityUser binding (step 3) is missing or names the wrong member. It must be PROJECT_ID.svc.id.goog[cert-manager/cert-manager].

403 from Cloud DNS

Fix

The Google service account lacks roles/dns.admin in the zone's project (step 2).

Credentials not found

Fix

Workload identity is off. See Enable workload identity on GKE.

Undo. To remove this access:

Terminal

gcloud iam service-accounts remove-iam-policy-binding \
  cert-manager-dns@PROJECT_ID.iam.gserviceaccount.com \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:PROJECT_ID.svc.id.goog[cert-manager/cert-manager]"
gcloud projects remove-iam-policy-binding PROJECT_ID \
  --role roles/dns.admin \
  --member "serviceAccount:cert-manager-dns@PROJECT_ID.iam.gserviceaccount.com"
gcloud iam service-accounts delete cert-manager-dns@PROJECT_ID.iam.gserviceaccount.com

Then remove identity from the manifest and apply the add-on again.

Model Serving (model-serving)

Model Serving reads private model artifacts for every model on the cluster with one identity, set on the add-on, never on an application. Each model runs as ServiceAccount astro-model-APPLICATION in its application's namespace NAMESPACE, so your cloud grants must cover those names. (The add-on's own controllers live in the kserve namespace.) Applications cannot set credentials.

Let models read a private GCS bucket

Do I need to do anything?

Your situation	What to do
Any cluster	Manual: the steps below

Before you start. Workload identity must be on for the cluster and its node pools: Enable workload identity on GKE. One-line check: gcloud container clusters describe CLUSTER --location LOCATION --format='value(workloadIdentityConfig.workloadPool)' prints PROJECT_ID.svc.id.goog.

Values you need. Find each one with the command shown.

Placeholder	What it is	Command that finds it
CLUSTER	Your cluster name	gcloud container clusters list
LOCATION	Location (region or zone) of the cluster, and of an Artifact Registry repository	gcloud container clusters list --format="table(name,location)"
PROJECT_ID	Project ID of the cluster	gcloud config get-value project
PROJECT_NUMBER	Project number of the cluster	gcloud projects describe PROJECT_ID --format='value(projectNumber)'
BUCKET	GCS bucket	gcloud storage buckets list --format='value(name)'
NAMESPACE, APPLICATION	The model application's namespace and name	astroctl app get APPLICATION kubectl -n NAMESPACE get sa astro-model-APPLICATION
Current add-on configuration and health	The add-on as applied	astroctl infra k8s addons get CLUSTER model-serving astroctl infra k8s addons get CLUSTER model-serving --manifest

Steps.

1. Look up the cluster's project number. This changes nothing; it prints PROJECT_NUMBER for the next step.

Terminal

gcloud projects describe PROJECT_ID --format='value(projectNumber)'

2. Grant read on one bucket, for one model. This lets that model's service account read objects in that bucket only. Workload Identity has no wildcard over ServiceAccount names, so repeat it for each model application.

Terminal

gcloud storage buckets add-iam-policy-binding gs://BUCKET \
  --role roles/storage.objectViewer \
  --member "principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/NAMESPACE/sa/astro-model-APPLICATION"

If the cluster also caches model weights, the cache downloads them as ServiceAccount default in namespace kserve-localmodel-jobs. Grant it the same read once:

Terminal

gcloud storage buckets add-iam-policy-binding gs://BUCKET \
  --role roles/storage.objectViewer \
  --member "principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/kserve-localmodel-jobs/sa/default"

3. Write the add-on manifest. This is only a file on your machine. Add an identity only if you bind through a Google service account instead.

model-serving.yaml

apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: model-serving
spec:
  clusterName: CLUSTER

4. Apply the add-on. This installs or updates the add-on with the identity and settings in the file.

Terminal

astroctl infra k8s addons apply -f model-serving.yaml

Validate. Three checks, in order. Stop at the first one that fails.

Check 1: the cluster is enabled (Enable workload identity on GKE).

Terminal

gcloud container clusters describe CLUSTER --location LOCATION \
  --format='value(workloadIdentityConfig.workloadPool)'
gcloud container node-pools describe POOL --cluster CLUSTER --location LOCATION \
  --format='value(config.workloadMetadataConfig.mode)'

Expected: PROJECT_ID.svc.id.goog, and GKE_METADATA for every pool the workload can run on. If either is empty or different, stop and go to Enable workload identity on GKE.

Check 2: the identity works for this ServiceAccount. For an add-on, run this after you apply it: the add-on creates the ServiceAccount. Run a throwaway pod as the ServiceAccount and ask the metadata server who it is.

Terminal

kubectl run wi-test -n NAMESPACE --rm -it --restart=Never --image=google/cloud-sdk:slim \
  --override-type=strategic --overrides='{"spec":{"serviceAccountName":"astro-model-APPLICATION"}}' \
  -- curl -s -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email

Expected: PROJECT_ID.svc.id.goog, the workload identity pool. If it prints the email of a service account instead (ending .iam.gserviceaccount.com), the pod is running as the node's identity, not as the ServiceAccount's; a token request would still succeed, so do not use one as the check.

Check 2b (optional): a registry accepts the identity. Request a token as the same pod (replace the last line of the command above with -- gcloud auth print-access-token) and test a registry with it: LOCATION-docker.pkg.dev/v2/PROJECT_ID/REPOSITORY/IMAGE/tags/list with the header Authorization: Bearer TOKEN returns HTTP 200.

If the namespace enforces the Pod Security "restricted" level and rejects the pod, use the override below that adds a securityContext, or run the test in a namespace that allows it with a ServiceAccount bound identically (same annotation, and a trust or grant that names that namespace).

Restricted-namespace override (also add --env HOME=/tmp, because gcloud writes its config under HOME): {"spec":{"serviceAccountName":"astro-model-APPLICATION","securityContext":{"runAsNonRoot":true,"runAsUser":65534,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"wi-test","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]}}

An email ending .iam.gserviceaccount.com, or default

Cause

No workload identity reached the pod: it is running as the node's identity. Workload identity is off on the cluster or node pool (go to Check 1), or the pod is not using the ServiceAccount you grant.

Permission '...' denied on resource in Check 2b or the workload

Cause

The identity is working; the role or scope on the resource is wrong (the grant step).

If this does not print PROJECT_ID.svc.id.goog, the binding is wrong; fix it before looking at the workload.

Check 3: the workload does its job.

Terminal

astroctl infra k8s addons get CLUSTER model-serving
astroctl app status APPLICATION
kubectl -n NAMESPACE logs POD -c storage-initializer

Expected: the add-on is healthy, the model application leaves its download step and serves, and the storage-initializer log has no 403 or AccessDenied. A pod binds its identity when it is created; applying a changed identity redeploys the cluster's models, so wait for the restart.

addons get prints no trust subject here, because this recipe sets no identity on the add-on: the trust subjects are listed only for an add-on that carries identity annotations. Its health should read healthy.

TROUBLESHOOT.

403 ... does not have storage.objects.get access to the Google Cloud Storage object

Fix

The grant names the wrong namespace, application or bucket. The member must end /subject/ns/NAMESPACE/sa/astro-model-APPLICATION (step 2).

DENIED: Unauthenticated request or credentials not found

Fix

Workload identity is off on the cluster or node pool. See Enable workload identity on GKE.

Still denied with the grant in place

Fix

KServe documents Secrets, not workload identity, for GCS. Use the add-on's secretRef instead.

Undo. To remove this access:

Terminal

gcloud storage buckets remove-iam-policy-binding gs://BUCKET \
  --role roles/storage.objectViewer \
  --member "principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/NAMESPACE/sa/astro-model-APPLICATION"

If you also granted the weight cache, remove that member the same way (.../subject/ns/kserve-localmodel-jobs/sa/default). This removes only those grants.

If it's refused

My app api in namespace shop on my-cluster gets 403 from GCS. Which part is missing, and how do I fix it?

Ask Nova

Nova reads the app's logs, events and settings and the cluster, names the missing part (the identity, the trust or the grant), and writes the fix.

When your app's call to your cloud is refused:

DENIED: Unauthenticated request or Your default credentials were not found

Cause

Workload identity is off on the cluster or on the node pool the pod runs on.

Fix

Turn on workload identity.

403 ... does not have storage.objects.list access

Cause

The grant names another project number, namespace or ServiceAccount.

Fix

Grant the role to the exact principal:// of the app's NAMESPACE and SERVICE_ACCOUNT. IAM changes can take a few minutes.

Permission '...' denied on resource

Cause

The pod signs in, but the role or its scope on the resource is wrong.

Fix

Grant the role on that exact resource.

When an add-on's identity is refused at apply:

"identity.serviceAccountAnnotations["iam.gke.io/gcp-service-account"] must be a service account email (<name>@<project>.iam.gserviceaccount.com)"

Cause

The value on the add-on's identity is not the kind of identity GKE expects.

Fix

Paste the value the recipe printed, unchanged.

"identity.serviceAccountAnnotations must carry a cloud workload-identity annotation: eks.amazonaws.com/role-arn (IRSA), iam.gke.io/gcp-service-account (GKE Workload Identity), or azure.workload.identity/client-id (AKS Workload Identity)"

Cause

identity was set with no workload-identity annotation.

Fix

Add the iam.gke.io/gcp-service-account annotation from the recipe.

Other ways in

• Nova: ask, for example, "why can't DNS Automation on my-cluster change records?". Nova reads the add-on's health and the cluster's cloud identity.
• SDK and API: an add-on's identity is part of its configuration. See the API reference.

With astroctl

Each recipe ends by applying the add-on with its identity:

Terminal

astroctl infra k8s addons apply -f ADDON.yaml
astroctl infra k8s addons get my-cluster ADDON

Next

• Keyless cloud access: how keyless works, and how your app gets its identity.
• Registry access: pull your images, and push them from a build.
• Deploy an application.