Access a cloud bucket

Let one application read one bucket in your cloud: Amazon S3, Google Cloud Storage or Azure Blob Storage. No cloud key is stored on the cluster. The application's pods sign in as a cloud identity that trusts only that application's ServiceAccount, and that identity can read only that bucket. You create the identity with your cloud's CLI, then put it on the application in the console.

The example is the same on every cloud: cluster prod-01, a Helm application shop-api in namespace shop running as ServiceAccount api, and bucket (or container) acme-data.

Before you start

• The cluster has workload identity. Clusters AstroPulse created have it. Open the cluster from Infrastructure > Kubernetes: the Cloud identity section on its Overview shows Workload identity: Available, or Missing when it is off. On Google Cloud, every node pool the application runs on must also use GKE_METADATA. A registered cluster turns it on once: AWS, Google Cloud, Azure.
• The application's ServiceAccount is one you control. A Helm chart sets it in its values; your own YAML manifests set it in a ServiceAccount and the pod's serviceAccountName. Nova can write either for you. An app from the Container image source runs as its namespace's default ServiceAccount, which you don't set from the app. See How your app gets its identity.
• Your cloud's CLI, signed in with permission to create roles or grants: aws, gcloud or az. Creating the cloud identity is the one step only your cloud can do. You also need kubectl for the check.
• A namespace that allows the test pod. A namespace enforcing Pod Security "restricted" refuses the check's test pod. Each cloud page gives the test with a securityContext: AWS, Google Cloud, Azure.

In the console

1. Create the cloud identity with your cloud's CLI: AWS, Google Cloud or Azure. On Google Cloud, that is all: skip to the check.
2. Open the application from Workloads > Applications, select its Update tab, and select Change configuration.
3. On the Values step, select Customize values (or Edit), add your chart's ServiceAccount keys for your cloud, then select Apply Overrides.
4. Select Continue to review, then Run dry run, then Continue to apply and Update.
5. On AWS, restart the pods: a pod gets the role only when it starts, and a change to the ServiceAccount alone does not restart them. On Azure, the pod label changes the pod template, so the pods restart on their own.

You should see

A test pod running as the application's ServiceAccount lists the bucket. Each cloud's section gives the check and its expected output.

Amazon S3 on AWS

The application's ServiceAccount carries an IAM role (IAM roles for service accounts).

1. Set your values.

Terminal

export REGION=us-west-2
export CLUSTER=prod-01
export NAMESPACE=shop
export DEPLOYMENT=shop-api
export SERVICE_ACCOUNT=api
export BUCKET=acme-data
export ROLE_NAME=app-s3-${NAMESPACE}-${SERVICE_ACCOUNT}
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export POLICY_ARN=arn:aws:iam::${ACCOUNT_ID}:policy/${ROLE_NAME}
export OIDC_HOST=$(aws eks describe-cluster --name "${CLUSTER}" --region "${REGION}" \
  --query cluster.identity.oidc.issuer --output text | sed 's#^https://##')

OIDC_HOST is the cluster's OIDC issuer from the Cloud identity section, without https://.

2. Create the IAM role. A policy that reads only ${BUCKET}, and a role that only system:serviceaccount:${NAMESPACE}:${SERVICE_ACCOUNT} on this cluster can assume.

Terminal

cat > policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::${BUCKET}" },
    { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::${BUCKET}/*" }
  ]
}
EOF
cat > trust.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_HOST}" },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": { "StringEquals": {
      "${OIDC_HOST}:sub": "system:serviceaccount:${NAMESPACE}:${SERVICE_ACCOUNT}",
      "${OIDC_HOST}:aud": "sts.amazonaws.com"
    } }
  }]
}
EOF
aws iam create-policy --policy-name "${ROLE_NAME}" --policy-document file://policy.json \
  --query Policy.Arn --output text
export ROLE_ARN=$(aws iam create-role --role-name "${ROLE_NAME}" \
  --assume-role-policy-document file://trust.json --query Role.Arn --output text)
aws iam attach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"
echo "${ROLE_ARN}"

Expected: the policy ARN, then the role ARN, for example arn:aws:iam::123456789012:role/app-s3-shop-api.

3. Put the role on the application. On the Values step (In the console), add the ARN from step 2. The usual keys:

values.yaml

serviceAccount:
  create: true
  name: api
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/app-s3-shop-api

Then restart the pods:

Terminal

kubectl -n "${NAMESPACE}" rollout restart deployment "${DEPLOYMENT}"

Your own ServiceAccount manifest instead of a chart:

Terminal

kubectl -n "${NAMESPACE}" annotate serviceaccount "${SERVICE_ACCOUNT}" "eks.amazonaws.com/role-arn=${ROLE_ARN}"
kubectl -n "${NAMESPACE}" rollout restart deployment "${DEPLOYMENT}"

4. Check that the application can read the bucket.

Terminal

kubectl -n "${NAMESPACE}" run cloud-check --rm -i --restart=Never --image=amazon/aws-cli:2.39.3 \
  --override-type=strategic --overrides="{\"spec\":{\"serviceAccountName\":\"${SERVICE_ACCOUNT}\"}}" \
  -- s3 ls "s3://${BUCKET}/"

Expected:

Output

All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
                           PRE data/
pod "cloud-check" deleted from shop namespace

Undo. In a new terminal, run step 1 first. Remove the annotation (console: the Values step; own manifest: kubectl -n "${NAMESPACE}" annotate serviceaccount "${SERVICE_ACCOUNT}" eks.amazonaws.com/role-arn-), restart the Deployment, then delete the role:

Terminal

aws iam detach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"
aws iam delete-role --role-name "${ROLE_NAME}"
aws iam delete-policy --policy-arn "${POLICY_ARN}"

Cloud Storage on Google Cloud

You grant the bucket directly to the application's Kubernetes ServiceAccount (Workload Identity Federation for GKE). Nothing changes on the application.

1. Set your values.

Terminal

export PROJECT_ID=my-project
export NAMESPACE=shop
export SERVICE_ACCOUNT=api
export BUCKET=acme-data
export PROJECT_NUMBER=$(gcloud projects describe "${PROJECT_ID}" --format='value(projectNumber)')
export MEMBER="principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${PROJECT_ID}.svc.id.goog/subject/ns/${NAMESPACE}/sa/${SERVICE_ACCOUNT}"

The Cloud identity section shows the cluster's Workload identity pool (my-project.svc.id.goog).

2. Grant read on the bucket.

Terminal

gcloud storage buckets add-iam-policy-binding "gs://${BUCKET}" \
  --role roles/storage.objectViewer --member "${MEMBER}"

3. Nothing to change on the application. The grant names the ServiceAccount itself.

4. Check that the application can read the bucket. First, that the pod runs as the ServiceAccount, not as the node:

Terminal

kubectl -n "${NAMESPACE}" run cloud-check --rm -i --restart=Never --image=google/cloud-sdk:slim \
  --override-type=strategic --overrides="{\"spec\":{\"serviceAccountName\":\"${SERVICE_ACCOUNT}\"}}" \
  -- curl -s -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email

Expected: my-project.svc.id.goog. An email ending .iam.gserviceaccount.com means the pod runs as the node: a node pool is not on GKE_METADATA, and the next check could pass for the wrong reason. Then the read:

Terminal

kubectl -n "${NAMESPACE}" run cloud-check --rm -i --restart=Never --image=google/cloud-sdk:slim \
  --override-type=strategic --overrides="{\"spec\":{\"serviceAccountName\":\"${SERVICE_ACCOUNT}\"}}" \
  -- gcloud storage ls "gs://${BUCKET}/"

Expected: the bucket's folders and objects, for example gs://acme-data/data/.

Undo. Removes only this grant.

Terminal

gcloud storage buckets remove-iam-policy-binding "gs://${BUCKET}" \
  --role roles/storage.objectViewer --member "${MEMBER}"

Blob Storage on Azure

A managed identity trusts the application's ServiceAccount, and the application's pods carry its client ID and a label (Microsoft Entra Workload ID on AKS).

1. Set your values.

Terminal

export RESOURCE_GROUP=my-group
export CLUSTER=prod-01
export NAMESPACE=shop
export DEPLOYMENT=shop-api
export SERVICE_ACCOUNT=api
export STORAGE_ACCOUNT=acmestorage
export CONTAINER=acme-data
export IDENTITY_NAME=app-blob-${NAMESPACE}-${SERVICE_ACCOUNT}
export ISSUER=$(az aks show --resource-group "${RESOURCE_GROUP}" --name "${CLUSTER}" \
  --query oidcIssuerProfile.issuerUrl --output tsv)

ISSUER is the cluster's OIDC issuer from the Cloud identity section.

2. Create the managed identity and grant read on the container.

Terminal

az identity create --resource-group "${RESOURCE_GROUP}" --name "${IDENTITY_NAME}"
export CLIENT_ID=$(az identity show --resource-group "${RESOURCE_GROUP}" --name "${IDENTITY_NAME}" --query clientId --output tsv)
export PRINCIPAL_ID=$(az identity show --resource-group "${RESOURCE_GROUP}" --name "${IDENTITY_NAME}" --query principalId --output tsv)
az identity federated-credential create --name "${SERVICE_ACCOUNT}" \
  --identity-name "${IDENTITY_NAME}" --resource-group "${RESOURCE_GROUP}" --issuer "${ISSUER}" \
  --subject "system:serviceaccount:${NAMESPACE}:${SERVICE_ACCOUNT}" --audiences api://AzureADTokenExchange
az role assignment create --assignee-object-id "${PRINCIPAL_ID}" --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Reader" \
  --scope "$(az storage account show --name "${STORAGE_ACCOUNT}" --query id --output tsv)/blobServices/default/containers/${CONTAINER}"
echo "${CLIENT_ID}"

Expected: the identity's client ID, a UUID.

3. Put the identity on the application. On the Values step (In the console), add the client ID from step 2 and the pod label. Azure needs both; the annotation alone does nothing. The usual keys:

values.yaml

serviceAccount:
  create: true
  name: api
  annotations:
    azure.workload.identity/client-id: 00000000-0000-0000-0000-000000000000
podLabels:
  azure.workload.identity/use: "true"

Your own manifests instead: annotate the ServiceAccount with azure.workload.identity/client-id, add the label azure.workload.identity/use: "true" to the pod template, and restart the Deployment.

4. Check that the application can read the container.

Terminal

kubectl -n "${NAMESPACE}" run cloud-check --rm -i --restart=Never --image=mcr.microsoft.com/azure-cli \
  --labels=azure.workload.identity/use=true \
  --override-type=strategic --overrides="{\"spec\":{\"serviceAccountName\":\"${SERVICE_ACCOUNT}\"}}" \
  -- sh -c 'az login --federated-token "$(cat $AZURE_FEDERATED_TOKEN_FILE)" --service-principal -u $AZURE_CLIENT_ID -t $AZURE_TENANT_ID --allow-no-subscriptions >/dev/null && az storage blob list --account-name '"${STORAGE_ACCOUNT}"' --container-name '"${CONTAINER}"' --auth-mode login --query "[].name" -o tsv'

Expected: the container's blob names, for example data/part-0001.csv.

Undo. Delete the identity last. Then remove the annotation and the label (console: the Values step) and deploy again.

Terminal

az role assignment delete --assignee "${PRINCIPAL_ID}" --role "Storage Blob Data Reader" \
  --scope "$(az storage account show --name "${STORAGE_ACCOUNT}" --query id --output tsv)/blobServices/default/containers/${CONTAINER}"
az identity federated-credential delete --name "${SERVICE_ACCOUNT}" --identity-name "${IDENTITY_NAME}" \
  --resource-group "${RESOURCE_GROUP}" --yes
az identity delete --resource-group "${RESOURCE_GROUP}" --name "${IDENTITY_NAME}"

Other ways in

• Nova: ask it to write the chart values or the ServiceAccount manifest for your cloud. Nova does not create cloud roles or grants: those are made in your cloud account, with your cloud's CLI.
• SDK and API: the same operations, in the API reference.

With astroctl

Add your cloud's keys under source.helm.values.object in the application's manifest and apply it. On AWS, restart the pods afterwards:

Terminal

astroctl app spec shop-api -o yaml > shop-api.yaml
astroctl app apply -f shop-api.yaml --dry-run
astroctl app apply -f shop-api.yaml
kubectl -n "${NAMESPACE}" rollout restart deployment "${DEPLOYMENT}"

If it's refused

Each message is what the test pod prints in the check.

aws: [ERROR]: An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity

Cause

The pod runs as a ServiceAccount the role does not trust, or the annotation names another role.

Fix

Compare the annotation with the ARN create-role printed. If they match, run the AWS Undo, fix NAMESPACE or SERVICE_ACCOUNT, and create the role again.

aws: [ERROR]: An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: No OpenIDConnect provider found in your account

Cause

The cluster is registered and workload identity is not on yet.

Fix

Turn it on, then run the check again.

aws: [ERROR]: An error occurred (NoCredentials): Unable to locate credentials.

Cause

The ServiceAccount has no eks.amazonaws.com/role-arn annotation, or the pod started before it. On clusters AstroPulse created, such a pod gets no AWS identity at all.

Fix

Add the annotation and restart the Deployment.

aws: [ERROR]: An error occurred (AccessDenied) when calling the ListObjectsV2 operation

Cause

The role works, but its policy names another bucket.

Fix

Run the AWS Undo, fix BUCKET, and create the role again.

aws: [ERROR]: An error occurred (NoSuchBucket) when calling the ListObjectsV2 operation: The specified bucket does not exist

Cause

BUCKET is misspelled.

Fix

Run the AWS Undo, fix BUCKET, and create the role again.

Next

• Other access from the same pattern (queues, registries, add-ons): AWS, Google Cloud, Azure.
• Custom domain
• Releases and rollback