Give a workload access to AWS

What do I need to connect my app api in namespace shop on my-cluster to the S3 bucket acme-data?

Give DNS Automation on my-cluster access to the Route 53 zone example.com.

Turn on workload identity for my registered EKS cluster my-cluster.

This is the hard part, and Nova does it with you. Name your app or add-on, the cluster and the resource: Nova reads the cluster and the workload's settings, and writes the identity, the grant, the trust and the values for your app, ready to run.

Each recipe gives one workload on an EKS cluster, an add-on or one of your applications, access to one AWS resource: an S3 bucket, an ECR repository, a Route 53 zone or an SQS queue. A pod signs in as an IAM role trusted only by its own namespace and ServiceAccount on this cluster, so no AWS key is stored on the cluster. Each grant reaches one resource, and every recipe has a check you can run and an undo that removes what it created.

Before you start

The AWS CLI signed in with rights to create identities and grant access, kubectl pointed at the cluster, and astroctl signed in.
Workload identity on the cluster. Clusters AstroPulse creates have it; a registered cluster turns it on once.

In the console

Open the cluster from the cluster list. On its Overview, Cloud identity shows who the cluster is in AWS: under Per application (recommended), Workload identity and the OIDC issuer your trust policy names, and under All nodes, the identity the nodes run as.
Create the identity and its grant with the recipe below for your add-on or application.
Open the cluster's Add-ons tab and open the add-on, then Update and Change configuration. For DNS Automation and Model Serving, set Cloud access to Workload identity; Certificates and Vulnerability Scanning take Identity directly. On the Identity step, pick the Cloud and enter the identity the recipe printed. Go to Review, select Continue to apply once the dry run passes, type the add-on's name (such as external-dns), then select Update external-dns.

Open zoom view
Configure Model Serving with Workload identity, Amazon Web Services (IRSA) and an IAM role ARN under Cloud access

You should see

Each recipe's check prints the identity the workload signs in as, then reads the resource. The add-on's health shows no access warning.

Recipes

Your own applications:

Give my application access to an S3 bucket
Give my application access to an SQS queue
Let EKS nodes pull from an ECR repository in another AWS account
Pull from any registry with a pull Secret, on any cloud

Platform add-ons:

Let vulnerability scanning read private images
Let DNS Automation change records in one Route 53 zone
Let Certificates add DNS challenge records in one Route 53 zone
Let models read a private S3 bucket

Turn on workload identity on a registered cluster

Clusters AstroPulse created have it: skip this. A registered cluster does not report it in the console, so check it with the command in step 2. On EKS this is the cluster's IAM OIDC provider. Background: IAM OIDC provider for your cluster.

1. Set your values

Terminal
export REGION=us-west-2
export CLUSTER=my-cluster
export ISSUER=$(aws eks describe-cluster --name "${CLUSTER}" --region "${REGION}" \
  --query cluster.identity.oidc.issuer --output text)
export ISSUER_ID=${ISSUER##*/}

2. Check

Terminal
aws iam list-open-id-connect-providers \
  --query "OpenIDConnectProviderList[?ends_with(Arn, '/id/${ISSUER_ID}')].Arn" --output text

Expected when on: arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE. Nothing printed means it is off.

3. Turn it on

Terminal
aws iam create-open-id-connect-provider --url "${ISSUER}" --client-id-list sts.amazonaws.com

No thumbprint is needed: without --thumbprint-list, IAM reads it from the issuer's certificate.

Expected: an OpenIDConnectProviderArn. An error occurred (EntityAlreadyExists) when calling the CreateOpenIDConnectProvider operation means it was already on. Running pods are not affected. Run step 2 again to confirm.

Undo. Every role that trusts this cluster stops working, so remove it only when nothing uses one.

Terminal
aws iam delete-open-id-connect-provider --open-id-connect-provider-arn \
  "$(aws iam list-open-id-connect-providers --query "OpenIDConnectProviderList[?ends_with(Arn, '/id/${ISSUER_ID}')].Arn" --output text)"

Your own applications

Bind the identity on the ServiceAccount your workload already runs as. Where that is defined depends on the application source: see How your app gets its identity.

Reading a bucket

Give my application access to an S3 bucket

Read access to one S3 bucket for one ServiceAccount. Background: IAM roles for service accounts.

Needs the AWS CLI and kubectl. Registered cluster: turn on workload identity first.

1. Set your values

Terminal
export REGION=us-west-2
export CLUSTER=my-cluster
export NAMESPACE=shop
export DEPLOYMENT=api
export SERVICE_ACCOUNT=api
export BUCKET=acme-data
export ROLE_NAME=app-s3-${NAMESPACE}-${SERVICE_ACCOUNT}
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export POLICY_ARN=arn:aws:iam::${ACCOUNT_ID}:policy/${ROLE_NAME}
export OIDC_HOST=$(aws eks describe-cluster --name "${CLUSTER}" --region "${REGION}" \
  --query cluster.identity.oidc.issuer --output text | sed 's#^https://##')

SERVICE_ACCOUNT is the Deployment's serviceAccountName (kubectl -n "${NAMESPACE}" get deploy "${DEPLOYMENT}" -o jsonpath='{.spec.template.spec.serviceAccountName}'); if it prints nothing, the app runs as default: give it its own ServiceAccount first (how).

2. Create the role

Terminal
cat > policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::${BUCKET}" },
    { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::${BUCKET}/*" }
  ]
}
EOF
cat > trust.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_HOST}" },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": { "StringEquals": {
      "${OIDC_HOST}:sub": "system:serviceaccount:${NAMESPACE}:${SERVICE_ACCOUNT}",
      "${OIDC_HOST}:aud": "sts.amazonaws.com"
    } }
  }]
}
EOF
aws iam create-policy --policy-name "${ROLE_NAME}" --policy-document file://policy.json \
  --query Policy.Arn --output text
export ROLE_ARN=$(aws iam create-role --role-name "${ROLE_NAME}" \
  --assume-role-policy-document file://trust.json --query Role.Arn --output text)
aws iam attach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"
echo "${ROLE_ARN}"

3. Put the role on your app's ServiceAccount

Set it on the application, so AstroPulse deploys it with the app (how): in the console, the app's Update › Values; in a manifest, under source.helm.values.object, then astroctl app apply -f application.yaml. With your own manifests, put the annotation on the ServiceAccount in your repository and commit. Use the ARN from step 2:

values.yaml
serviceAccount:
  create: true
  name: SERVICE_ACCOUNT
  annotations:
    eks.amazonaws.com/role-arn: ROLE_ARN

4. Check

Terminal
kubectl -n "${NAMESPACE}" run cloud-check --rm -i --restart=Never --image=amazon/aws-cli:2.39.3 \
  --override-type=strategic --overrides="{\"spec\":{\"serviceAccountName\":\"${SERVICE_ACCOUNT}\"}}" \
  -- s3 ls "s3://${BUCKET}/"

Expected: the bucket's top-level folders as PRE name/, and its objects. An empty bucket prints none.

Output
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
                           PRE data/
pod "cloud-check" deleted from shop namespace

Namespace refuses the test pod (Pod Security "restricted")

Terminal
kubectl -n "${NAMESPACE}" run cloud-check --rm -i --restart=Never --image=amazon/aws-cli:2.39.3 \
  --override-type=strategic \
  --overrides='{"spec":{"serviceAccountName":"'"${SERVICE_ACCOUNT}"'","securityContext":{"runAsNonRoot":true,"runAsUser":65534,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"cloud-check","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]}}' \
  -- s3 ls "s3://${BUCKET}/"

To see the identity itself, run the same pod with -- sts get-caller-identity --query Arn --output text: arn:aws:sts::123456789012:assumed-role/ROLE_NAME/botocore-session-<generated>.

If it fails

Unless a bullet says where, the test pod in step 4 prints it.

aws: [ERROR]: An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity: the role ARN on the ServiceAccount is not the one step 2 printed (fix step 3), or step 1 named the wrong namespace or ServiceAccount (run Undo, fix step 1, run step 2 again).
aws: [ERROR]: An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: No OpenIDConnect provider found in your account for https://...: the cluster is registered and has no workload identity yet; turn on workload identity.
aws: [ERROR]: An error occurred (NoCredentials): Unable to locate credentials. You can configure credentials by running "aws login".: the ServiceAccount has no role annotation, or its pods started before it (step 3). On clusters AstroPulse created, a pod without the annotation gets no AWS identity at all. On nodes that let pods reach instance metadata it gets the node role instead, and the Arn shows the node role.
aws: [ERROR]: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: User: arn:aws:sts::...:assumed-role/... is not authorized to perform: s3:ListBucket ...: the policy names another bucket. Run Undo, fix step 1, then run step 2 again.
aws: [ERROR]: An error occurred (NoSuchBucket) when calling the ListObjectsV2 operation: The specified bucket does not exist: BUCKET is misspelled. Run Undo, fix step 1, then run step 2 again.

Undo. In a new terminal, run step 1 first.

Terminal
aws iam detach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"
aws iam delete-role --role-name "${ROLE_NAME}"
aws iam delete-policy --policy-arn "${POLICY_ARN}"

Expected after: aws iam get-role --role-name "${ROLE_NAME}" prints aws: [ERROR]: An error occurred (NoSuchEntity) when calling the GetRole operation: The role with name app-s3-shop-api cannot be found.

Then remove the eks.amazonaws.com/role-arn annotation from the app's values (or your ServiceAccount manifest) and apply the application again.

Calling a queue, topic or vault

Give my application access to an SQS queue

Send, receive and delete on one SQS queue for one ServiceAccount. Drop any action you do not use. Background: IAM roles for service accounts.

Needs the AWS CLI and kubectl. Registered cluster: turn on workload identity first.

1. Set your values

Terminal
export REGION=us-west-2
export CLUSTER=my-cluster
export NAMESPACE=shop
export DEPLOYMENT=api
export SERVICE_ACCOUNT=api
export QUEUE_NAME=orders
export ROLE_NAME=app-sqs-${NAMESPACE}-${SERVICE_ACCOUNT}
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export POLICY_ARN=arn:aws:iam::${ACCOUNT_ID}:policy/${ROLE_NAME}
export OIDC_HOST=$(aws eks describe-cluster --name "${CLUSTER}" --region "${REGION}" \
  --query cluster.identity.oidc.issuer --output text | sed 's#^https://##')

2. Create the role

Terminal
cat > policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes", "sqs:SendMessage"],
    "Resource": "arn:aws:sqs:${REGION}:${ACCOUNT_ID}:${QUEUE_NAME}"
  }]
}
EOF
cat > trust.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_HOST}" },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": { "StringEquals": {
      "${OIDC_HOST}:sub": "system:serviceaccount:${NAMESPACE}:${SERVICE_ACCOUNT}",
      "${OIDC_HOST}:aud": "sts.amazonaws.com"
    } }
  }]
}
EOF
aws iam create-policy --policy-name "${ROLE_NAME}" --policy-document file://policy.json \
  --query Policy.Arn --output text
export ROLE_ARN=$(aws iam create-role --role-name "${ROLE_NAME}" \
  --assume-role-policy-document file://trust.json --query Role.Arn --output text)
aws iam attach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"
echo "${ROLE_ARN}"

3. Put the role on your app's ServiceAccount

values.yaml
serviceAccount:
  create: true
  name: SERVICE_ACCOUNT
  annotations:
    eks.amazonaws.com/role-arn: ROLE_ARN

4. Check

Terminal
kubectl -n "${NAMESPACE}" run cloud-check --rm -i --restart=Never --image=amazon/aws-cli:2.39.3 \
  --override-type=strategic --overrides="{\"spec\":{\"serviceAccountName\":\"${SERVICE_ACCOUNT}\"}}" \
  -- sqs get-queue-attributes --region "${REGION}" --attribute-names ApproximateNumberOfMessages \
     --queue-url "https://sqs.${REGION}.amazonaws.com/${ACCOUNT_ID}/${QUEUE_NAME}"

Expected: the queue's message count.

Output
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
{
    "Attributes": {
        "ApproximateNumberOfMessages": "0"
    }
}
pod "cloud-check" deleted from shop namespace

Namespace refuses the test pod (Pod Security "restricted")

Terminal
kubectl -n "${NAMESPACE}" run cloud-check --rm -i --restart=Never --image=amazon/aws-cli:2.39.3 \
  --override-type=strategic \
  --overrides='{"spec":{"serviceAccountName":"'"${SERVICE_ACCOUNT}"'","securityContext":{"runAsNonRoot":true,"runAsUser":65534,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"cloud-check","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]}}' \
  -- sqs get-queue-attributes --region "${REGION}" --attribute-names ApproximateNumberOfMessages \
     --queue-url "https://sqs.${REGION}.amazonaws.com/${ACCOUNT_ID}/${QUEUE_NAME}"

To see the identity itself, run the same pod with -- sts get-caller-identity --query Arn --output text: arn:aws:sts::123456789012:assumed-role/ROLE_NAME/botocore-session-<generated>.

If it fails

Unless a bullet says where, the test pod in step 4 prints it.

aws: [ERROR]: An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity: the role ARN on the ServiceAccount is not the one step 2 printed (fix step 3), or step 1 named the wrong namespace or ServiceAccount (run Undo, fix step 1, run step 2 again).
aws: [ERROR]: An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: No OpenIDConnect provider found in your account for https://...: the cluster is registered and has no workload identity yet; turn on workload identity.
aws: [ERROR]: An error occurred (NoCredentials): Unable to locate credentials. You can configure credentials by running "aws login".: the ServiceAccount has no role annotation, or its pods started before it (step 3). On clusters AstroPulse created, a pod without the annotation gets no AWS identity at all. On nodes that let pods reach instance metadata it gets the node role instead, and the Arn shows the node role.
aws: [ERROR]: An error occurred (AccessDenied) when calling the GetQueueAttributes operation: User: arn:aws:sts::...:assumed-role/... is not authorized to perform: sqs:getqueueattributes on resource: arn:aws:sqs:REGION:ACCOUNT:QUEUE because no identity-based policy allows the sqs:getqueueattributes action: the policy names another queue, region or account. Run Undo, fix step 1, then run step 2 again.
aws: [ERROR]: An error occurred (AWS.SimpleQueueService.NonExistentQueue) when calling the GetQueueAttributes operation: ...: no queue named QUEUE_NAME in REGION, whatever the policy says.

Undo. In a new terminal, run step 1 first.

Terminal
aws iam detach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"
aws iam delete-role --role-name "${ROLE_NAME}"
aws iam delete-policy --policy-arn "${POLICY_ARN}"

Then remove the eks.amazonaws.com/role-arn annotation from the app's values (or your ServiceAccount manifest) and apply the application again.

Pulling images from a private registry

Image pulls are done by the node, not by the pod's ServiceAccount (see why). Grant the node identity access, or use an pull Secret.

Let EKS nodes pull from an ECR repository in another AWS account

Nodes pull images with the cluster's node roles, not with workload identity. Private images in ECR in the cluster's own account pull with no secret. For a repository in another account, that account allows the node roles in its repository policy. Background: ECR repository policies.

Clusters AstroPulse created: the node roles can already read ECR in any account; only the other account's repository policy is missing. Listing the account in the cluster's ecrAccountIds (step 3) makes the cluster's Cloud identity › Image pulls say those accounts pull once their repository policies allow the node roles. Registered cluster: the node roles also need ECR read in their own account (AmazonEC2ContainerRegistryReadOnly).

1. Set your values

Terminal
export CLUSTER=my-cluster
export REPOSITORY=apps/api
export REGISTRY_ACCOUNT_ID=210987654321
export REGISTRY_REGION=us-west-2

The node roles: on the cluster's Overview (https://astropulse.io/console?type=clusters&cluster=my-cluster), Cloud identity › All nodes › Copy all as policy principals copies them as a JSON list (NODE_ROLES below); Roles beside it lists each one.

2. Allow the node roles (in the account that owns the repository)

This replaces the repository's policy; if it has one, add this statement to it instead (aws ecr get-repository-policy shows it).

Terminal
cat > ecr-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowClusterNodesPull",
    "Effect": "Allow",
    "Principal": { "AWS": NODE_ROLES },
    "Action": ["ecr:BatchGetImage", "ecr:GetDownloadUrlForLayer", "ecr:BatchCheckLayerAvailability"]
  }]
}
EOF
aws ecr set-repository-policy --registry-id "${REGISTRY_ACCOUNT_ID}" --region "${REGISTRY_REGION}" \
  --repository-name "${REPOSITORY}" --policy-text file://ecr-policy.json

Replace NODE_ROLES with the copied list before you run it.

3. List the account on the cluster (clusters AstroPulse created, optional)

At create, the wizard's Network step has it in the collapsed section at its foot: Image registries in other AWS accounts (ECR). On a running cluster:

Terminal
cat > ecr-accounts.yaml <<EOF
apiVersion: platform.astropulse.io/v1
kind: K8sClusterUpdate
spec:
  eks:
    ecrAccountIds: ["${REGISTRY_ACCOUNT_ID}"]
EOF
astroctl infra k8s update "${CLUSTER}" -f ecr-accounts.yaml --dry-run
astroctl infra k8s update "${CLUSTER}" -f ecr-accounts.yaml

ecrAccountIds replaces the whole list: include every account you want kept.

4. Check

Deploy a workload with an image from the repository, then:

Terminal
kubectl get events -A --field-selector reason=Pulled

Expected: a Successfully pulled image "REGISTRY_ACCOUNT_ID.dkr.ecr.REGISTRY_REGION.amazonaws.com/REPOSITORY:TAG" event.

If it fails

ImagePullBackOff with 403 Forbidden or pull access denied in kubectl describe pod: the repository policy does not name every node role (step 2), or, on a registered cluster, the node role has no ECR read in its own account.

Undo. Removes the repository's whole policy; if it held other statements, set it back to those instead. On the cluster, send ecrAccountIds without the account.

Terminal
aws ecr delete-repository-policy --registry-id "${REGISTRY_ACCOUNT_ID}" --region "${REGISTRY_REGION}" \
  --repository-name "${REPOSITORY}"

Platform add-ons

Each add-on runs as one fixed ServiceAccount. You give that account access, then name the identity on the add-on's identity field. The field has this shape on every add-on that takes it:

values.yaml
identity:
  serviceAccountAnnotations: {}   # for example eks.amazonaws.com/role-arn: ROLE_ARN
  serviceAccountLabels: {}        # Azure only: azure.workload.identity/use: "true"

Add-on	Namespace	ServiceAccount	Needs
Vulnerability Scanning (`vulnerability-scanning`)	`astro-security`	`trivy-operator`	Read images in one registry
DNS Automation (`external-dns`)	`external-dns`	`external-dns`	Change records in one DNS zone
Certificates (`cert-manager`)	`cert-manager`	`cert-manager`	Add and remove TXT records in one DNS zone (DNS-01 check only)
Model Serving (`model-serving`)	each model's own namespace	`astro-model-APPLICATION` for each model	Read the model bucket

Applying an add-on's identity is covered in Managing cluster add-ons.

Vulnerability Scanning

Let vulnerability scanning read private images

Pull access to one ECR repository for the scanner (astro-security/trivy-operator). Background: IAM roles for service accounts.

Clusters AstroPulse created: nothing to do for registries in the cluster's own account. If the add-on warns that the role is missing (a cluster created before the role existed), update the cluster once with any change (astroctl infra k8s update "${CLUSTER}" -f update.yaml), then apply the add-on again. The steps below are for a registered cluster, or a registry in another account.

Needs the AWS CLI, kubectl and astroctl. Registered cluster: turn on workload identity first.

1. Set your values

Terminal
export REGION=us-west-2
export CLUSTER=my-cluster
export REPOSITORY=apps/api
export REGISTRY_REGION=us-west-2
export ROLE_NAME=vulnerability-scanning-${CLUSTER}
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export POLICY_ARN=arn:aws:iam::${ACCOUNT_ID}:policy/${ROLE_NAME}
export OIDC_HOST=$(aws eks describe-cluster --name "${CLUSTER}" --region "${REGION}" \
  --query cluster.identity.oidc.issuer --output text | sed 's#^https://##')
export REGISTRY_ACCOUNT_ID=${ACCOUNT_ID}

For a registry in another account, set REGISTRY_ACCOUNT_ID to that account. Both values come from the image reference: ACCOUNT.dkr.ecr.REGION.amazonaws.com/REPOSITORY:TAG.

2. Create the role

Terminal
cat > policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": "ecr:GetAuthorizationToken", "Resource": "*" },
    {
      "Effect": "Allow",
      "Action": ["ecr:BatchGetImage", "ecr:GetDownloadUrlForLayer", "ecr:BatchCheckLayerAvailability"],
      "Resource": "arn:aws:ecr:${REGISTRY_REGION}:${REGISTRY_ACCOUNT_ID}:repository/${REPOSITORY}"
    }
  ]
}
EOF
cat > trust.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_HOST}" },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": { "StringEquals": {
      "${OIDC_HOST}:sub": "system:serviceaccount:astro-security:trivy-operator",
      "${OIDC_HOST}:aud": "sts.amazonaws.com"
    } }
  }]
}
EOF
aws iam create-policy --policy-name "${ROLE_NAME}" --policy-document file://policy.json \
  --query Policy.Arn --output text
export ROLE_ARN=$(aws iam create-role --role-name "${ROLE_NAME}" \
  --assume-role-policy-document file://trust.json --query Role.Arn --output text)
aws iam attach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"
echo "${ROLE_ARN}"

Registry in another account: signed in to that account, also allow the role. This replaces the repository's policy; if it has one, merge this statement into aws ecr get-repository-policy's output.

Terminal
cat > repository-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowScanner",
    "Effect": "Allow",
    "Principal": { "AWS": "${ROLE_ARN}" },
    "Action": ["ecr:BatchGetImage", "ecr:GetDownloadUrlForLayer", "ecr:BatchCheckLayerAvailability"]
  }]
}
EOF
aws ecr set-repository-policy --registry-id "${REGISTRY_ACCOUNT_ID}" --region "${REGISTRY_REGION}" \
  --repository-name "${REPOSITORY}" --policy-text file://repository-policy.json

3. Put the role on the add-on

In the console: Update, then Change configuration, on Vulnerability Scanning; on the Identity step set Cloud to Amazon Web Services, and paste the ARN into AWS role ARN. With astroctl:

Terminal
cat > vulnerability-scanning.yaml <<EOF
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: vulnerability-scanning
spec:
  clusterName: ${CLUSTER}
  configuration:
    vulnerabilityScanning:
      identity:
        serviceAccountAnnotations:
          eks.amazonaws.com/role-arn: ${ROLE_ARN}
EOF
astroctl infra k8s addons apply -f vulnerability-scanning.yaml

4. Check

The scanner's identity:

Terminal
kubectl -n "astro-security" run cloud-check --rm -i --restart=Never --image=amazon/aws-cli:2.39.3 \
  --override-type=strategic --overrides="{\"spec\":{\"serviceAccountName\":\"trivy-operator\"}}" \
  -- sts get-caller-identity --query Arn --output text

Expected:

Output
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
arn:aws:sts::123456789012:assumed-role/vulnerability-scanning-my-cluster/botocore-session-<generated>
pod "cloud-check" deleted from astro-security namespace

The account ID is a placeholder for yours. <generated> stands for the session number AWS creates on every call, so yours will differ.

Namespace refuses the test pod (Pod Security "restricted")

Terminal
kubectl -n "astro-security" run cloud-check --rm -i --restart=Never --image=amazon/aws-cli:2.39.3 \
  --override-type=strategic \
  --overrides='{"spec":{"serviceAccountName":"trivy-operator","securityContext":{"runAsNonRoot":true,"runAsUser":65534,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"cloud-check","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]}}' \
  -- sts get-caller-identity --query Arn --output text

Then the role's access to the repository (replace TAG with an image tag in it):

Terminal
kubectl -n "astro-security" run cloud-check --rm -i --restart=Never --image=amazon/aws-cli:2.39.3 \
  --override-type=strategic --overrides="{\"spec\":{\"serviceAccountName\":\"trivy-operator\"}}" \
  -- ecr batch-get-image --repository-name "${REPOSITORY}" --registry-id "${REGISTRY_ACCOUNT_ID}" --region "${REGISTRY_REGION}" --image-ids imageTag=TAG --query "images[].imageId"

Expected: the image's imageDigest and imageTag. [] means no image has that tag.

Namespace refuses the test pod (Pod Security "restricted")

Terminal
kubectl -n "astro-security" run cloud-check --rm -i --restart=Never --image=amazon/aws-cli:2.39.3 \
  --override-type=strategic \
  --overrides='{"spec":{"serviceAccountName":"trivy-operator","securityContext":{"runAsNonRoot":true,"runAsUser":65534,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"cloud-check","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]}}' \
  -- ecr batch-get-image --repository-name "${REPOSITORY}" --registry-id "${REGISTRY_ACCOUNT_ID}" --region "${REGISTRY_REGION}" --image-ids imageTag=TAG --query "images[].imageId"

Then the scan:

Terminal
kubectl get vulnerabilityreports -A
astroctl infra k8s addons get "${CLUSTER}" vulnerability-scanning

Expected: a report for the private image, and no images not scannable note. A new grant takes up to about 5 minutes; to rescan a refused image now, restart its workload.

If it fails

Unless a bullet says where, the test pod in step 4 prints it.

aws: [ERROR]: An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity: the role ARN on the add-on is not the one step 2 printed (fix step 3), or the pod does not run as astro-security/trivy-operator, which the role trusts.
aws: [ERROR]: An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: No OpenIDConnect provider found in your account for https://...: the cluster is registered and has no workload identity yet; turn on workload identity.
aws: [ERROR]: An error occurred (NoCredentials): Unable to locate credentials. You can configure credentials by running "aws login".: the add-on has no role annotation, or its pods started before it (step 3). On clusters AstroPulse created, a pod without the annotation gets no AWS identity at all. On nodes that let pods reach instance metadata it gets the node role instead, and the Arn shows the node role.
aws: [ERROR]: An error occurred (AccessDeniedException) when calling the BatchGetImage operation: User: ... is not authorized to perform: ecr:BatchGetImage on resource: ...repository/REPOSITORY because no identity-based policy allows ...: the policy names another repository. Run Undo, fix step 1, then run step 2 again. The same error with "no resource-based policy allows" means the other account's AllowScanner repository policy is missing.
aws: [ERROR]: An error occurred (RepositoryNotFoundException) when calling the BatchGetImage operation: ...: wrong REPOSITORY, REGISTRY_ACCOUNT_ID or REGISTRY_REGION in step 1.
N images not scannable: registry denied (...) in the add-on's health: the role cannot read that repository, or the other account's repository policy is missing. Fix step 2, then restart the workload.

Undo. In a new terminal, run step 1 first.

Terminal
aws iam detach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"
aws iam delete-role --role-name "${ROLE_NAME}"
aws iam delete-policy --policy-arn "${POLICY_ARN}"

Expected after: aws iam get-role --role-name "${ROLE_NAME}" prints aws: [ERROR]: An error occurred (NoSuchEntity) when calling the GetRole operation: The role with name vulnerability-scanning-my-cluster cannot be found.

Then remove identity from the manifest and apply it again. Another account: remove the AllowScanner statement there, or, if it was the only one, run aws ecr delete-repository-policy --registry-id "${REGISTRY_ACCOUNT_ID}" --region "${REGISTRY_REGION}" --repository-name "${REPOSITORY}".

DNS Automation (`external-dns`)

Let DNS Automation change records in one Route 53 zone

Record changes in one hosted zone for DNS Automation (external-dns/external-dns). Without it, the add-on runs but publishes nothing. Background: IAM roles for service accounts.

Clusters AstroPulse created: nothing to do for a Route 53 zone. The steps below are for a registered cluster, or a zone the built-in role does not cover.

Needs the AWS CLI, kubectl and astroctl. Registered cluster: turn on workload identity first.

1. Set your values

Terminal
export REGION=us-west-2
export CLUSTER=my-cluster
export ZONE=example.com
export ZONE_ID=$(aws route53 list-hosted-zones-by-name --dns-name "${ZONE}" \
  --query "HostedZones[?Name=='${ZONE}.'].Id" --output text | sed 's#/hostedzone/##')
export ROLE_NAME=external-dns-${CLUSTER}
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export POLICY_ARN=arn:aws:iam::${ACCOUNT_ID}:policy/${ROLE_NAME}
export OIDC_HOST=$(aws eks describe-cluster --name "${CLUSTER}" --region "${REGION}" \
  --query cluster.identity.oidc.issuer --output text | sed 's#^https://##')

Check with echo "${ZONE_ID}". Empty means no hosted zone is named exactly ${ZONE}: do not continue. Two IDs mean a public and a private zone share the name: set ZONE_ID to the one you want.

2. Create the role

Terminal
cat > policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["route53:ChangeResourceRecordSets", "route53:ListResourceRecordSets", "route53:ListTagsForResources"],
      "Resource": "arn:aws:route53:::hostedzone/${ZONE_ID}"
    },
    { "Effect": "Allow", "Action": "route53:ListHostedZones", "Resource": "*" }
  ]
}
EOF
cat > trust.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_HOST}" },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": { "StringEquals": {
      "${OIDC_HOST}:sub": "system:serviceaccount:external-dns:external-dns",
      "${OIDC_HOST}:aud": "sts.amazonaws.com"
    } }
  }]
}
EOF
aws iam create-policy --policy-name "${ROLE_NAME}" --policy-document file://policy.json \
  --query Policy.Arn --output text
export ROLE_ARN=$(aws iam create-role --role-name "${ROLE_NAME}" \
  --assume-role-policy-document file://trust.json --query Role.Arn --output text)
aws iam attach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"
echo "${ROLE_ARN}"

3. Put the role on the add-on

Console: the cluster's Add-ons (https://astropulse.io/console?type=clusters&cluster=my-cluster&tab=add-ons), Update, then Change configuration, on DNS Automation (or Install Add-on), then set Cloud access to Workload identity, on the Identity step set Cloud to Amazon Web Services, and paste the ARN into AWS role ARN. Set Domain filter to your zone, for example example.com.

Or:

Terminal
cat > external-dns.yaml <<EOF
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: external-dns
spec:
  clusterName: ${CLUSTER}
  configuration:
    externalDns:
      provider: aws
      domainFilter: ${ZONE}
      identity:
        serviceAccountAnnotations:
          eks.amazonaws.com/role-arn: ${ROLE_ARN}
EOF
astroctl infra k8s addons apply -f external-dns.yaml

4. Check

The add-on's identity:

Terminal
kubectl -n "external-dns" run cloud-check --rm -i --restart=Never --image=amazon/aws-cli:2.39.3 \
  --override-type=strategic --overrides="{\"spec\":{\"serviceAccountName\":\"external-dns\"}}" \
  -- sts get-caller-identity --query Arn --output text

Expected:

Output
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
arn:aws:sts::123456789012:assumed-role/external-dns-my-cluster/botocore-session-<generated>
pod "cloud-check" deleted from external-dns namespace

Namespace refuses the test pod (Pod Security "restricted")

Terminal
kubectl -n "external-dns" run cloud-check --rm -i --restart=Never --image=amazon/aws-cli:2.39.3 \
  --override-type=strategic \
  --overrides='{"spec":{"serviceAccountName":"external-dns","securityContext":{"runAsNonRoot":true,"runAsUser":65534,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"cloud-check","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]}}' \
  -- sts get-caller-identity --query Arn --output text

Then a test name on a LoadBalancer Service:

Terminal
export APP_NAMESPACE=shop
export SERVICE=api
kubectl -n "${APP_NAMESPACE}" annotate service "${SERVICE}" external-dns.alpha.kubernetes.io/hostname=test.${ZONE}
dig +short "test.${ZONE}"

Expected: the Service's address within a minute or two, for example 203.0.113.10. Remove the test with kubectl -n "${APP_NAMESPACE}" annotate service "${SERVICE}" external-dns.alpha.kubernetes.io/hostname-.

If it fails

Unless a bullet says where, the test pod in step 4 prints it.

aws: [ERROR]: An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity: the role ARN on the add-on is not the one step 2 printed (fix step 3), or the pod does not run as external-dns/external-dns, which the role trusts.
aws: [ERROR]: An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: No OpenIDConnect provider found in your account for https://...: the cluster is registered and has no workload identity yet; turn on workload identity.
aws: [ERROR]: An error occurred (NoCredentials): Unable to locate credentials. You can configure credentials by running "aws login".: the add-on has no role annotation, or its pods started before it (step 3). On clusters AstroPulse created, a pod without the annotation gets no AWS identity at all. On nodes that let pods reach instance metadata it gets the node role instead, and the Arn shows the node role.
Healthy, but no record appears: Domain filter does not match the zone. Fix step 3.
An AccessDenied error naming route53:ListResourceRecordSets or route53:ChangeResourceRecordSets and another hostedzone/ in the add-on's error log (astroctl infra k8s addons logs "${CLUSTER}" external-dns --level Error --follow=false --interactive=false): the policy names another zone. Run Undo, fix step 1, then run step 2 again.

Undo. In a new terminal, run step 1 first.

Terminal
aws iam detach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"
aws iam delete-role --role-name "${ROLE_NAME}"
aws iam delete-policy --policy-arn "${POLICY_ARN}"

Expected after: aws iam get-role --role-name "${ROLE_NAME}" prints aws: [ERROR]: An error occurred (NoSuchEntity) when calling the GetRole operation: The role with name external-dns-my-cluster cannot be found.

Then set Cloud access back to None, or remove identity from the manifest and apply it again.

Certificates (`cert-manager`), DNS-01 check

Let Certificates add DNS challenge records in one Route 53 zone

TXT-record access in one hosted zone for the DNS-01 check (cert-manager/cert-manager). Needs enableClusterIssuer: true and issuerType: LetsEncrypt or LetsEncryptStaging. Background: IAM roles for service accounts.

Clusters AstroPulse created: nothing to do for a Route 53 zone beyond setting dnsSolver. The steps below are for a registered cluster.

Needs the AWS CLI, kubectl and astroctl. Registered cluster: turn on workload identity first.

1. Set your values

Terminal
export REGION=us-west-2
export CLUSTER=my-cluster
export ZONE=example.com
export ZONE_ID=$(aws route53 list-hosted-zones-by-name --dns-name "${ZONE}" \
  --query "HostedZones[?Name=='${ZONE}.'].Id" --output text | sed 's#/hostedzone/##')
export ROLE_NAME=cert-manager-${CLUSTER}
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export POLICY_ARN=arn:aws:iam::${ACCOUNT_ID}:policy/${ROLE_NAME}
export OIDC_HOST=$(aws eks describe-cluster --name "${CLUSTER}" --region "${REGION}" \
  --query cluster.identity.oidc.issuer --output text | sed 's#^https://##')

Check with echo "${ZONE_ID}". Empty means no hosted zone is named exactly ${ZONE}: do not continue. Two IDs mean a public and a private zone share the name: set ZONE_ID to the one you want.

2. Create the role

Terminal
cat > policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": "route53:GetChange", "Resource": "arn:aws:route53:::change/*" },
    {
      "Effect": "Allow",
      "Action": ["route53:ChangeResourceRecordSets", "route53:ListResourceRecordSets"],
      "Resource": "arn:aws:route53:::hostedzone/${ZONE_ID}"
    },
    { "Effect": "Allow", "Action": "route53:ListHostedZonesByName", "Resource": "*" }
  ]
}
EOF
cat > trust.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_HOST}" },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": { "StringEquals": {
      "${OIDC_HOST}:sub": "system:serviceaccount:cert-manager:cert-manager",
      "${OIDC_HOST}:aud": "sts.amazonaws.com"
    } }
  }]
}
EOF
aws iam create-policy --policy-name "${ROLE_NAME}" --policy-document file://policy.json \
  --query Policy.Arn --output text
export ROLE_ARN=$(aws iam create-role --role-name "${ROLE_NAME}" \
  --assume-role-policy-document file://trust.json --query Role.Arn --output text)
aws iam attach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"
echo "${ROLE_ARN}"

3. Put the role on the add-on

In the console: Update, then Change configuration, on Certificates; on the Identity step set Cloud to Amazon Web Services, and paste the ARN into AWS role ARN. Use this role, not an EKS Pod Identity association: the DNS check looks for the annotation.

Terminal
cat > cert-manager.yaml <<EOF
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: cert-manager
spec:
  clusterName: ${CLUSTER}
  configuration:
    certManager:
      enableClusterIssuer: true
      issuerType: LetsEncrypt
      email: ops@example.com
      identity:
        serviceAccountAnnotations:
          eks.amazonaws.com/role-arn: ${ROLE_ARN}
      dnsSolver:
        provider: aws
        hostedZoneId: ${ZONE_ID}
EOF
astroctl infra k8s addons apply -f cert-manager.yaml --dry-run
astroctl infra k8s addons apply -f cert-manager.yaml

4. Check

The add-on's identity:

Terminal
kubectl -n "cert-manager" run cloud-check --rm -i --restart=Never --image=amazon/aws-cli:2.39.3 \
  --override-type=strategic --overrides="{\"spec\":{\"serviceAccountName\":\"cert-manager\"}}" \
  -- sts get-caller-identity --query Arn --output text

Expected:

Output
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
arn:aws:sts::123456789012:assumed-role/cert-manager-my-cluster/botocore-session-<generated>
pod "cloud-check" deleted from cert-manager namespace

Namespace refuses the test pod (Pod Security "restricted")

Terminal
kubectl -n "cert-manager" run cloud-check --rm -i --restart=Never --image=amazon/aws-cli:2.39.3 \
  --override-type=strategic \
  --overrides='{"spec":{"serviceAccountName":"cert-manager","securityContext":{"runAsNonRoot":true,"runAsUser":65534,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"cloud-check","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]}}' \
  -- sts get-caller-identity --query Arn --output text

Then a test certificate for test.${ZONE} (a real one, counted against Let's Encrypt limits).

Terminal
cat > dns-check.yaml <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: dns-check
  namespace: default
  labels:
    platform.astropulse.io/challenge: dns01
spec:
  secretName: dns-check-tls
  dnsNames:
    - test.${ZONE}
  issuerRef:
    name: astro-letsencrypt
    kind: ClusterIssuer
EOF
kubectl apply -f dns-check.yaml
kubectl get certificate dns-check --watch

Expected: READY turns True within a few minutes. Remove it with kubectl delete certificate dns-check.

If it fails

Unless a bullet says where, the test pod in step 4 prints it.

aws: [ERROR]: An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity: the role ARN on the add-on is not the one step 2 printed (fix step 3), or the pod does not run as cert-manager/cert-manager, which the role trusts.
aws: [ERROR]: An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: No OpenIDConnect provider found in your account for https://...: the cluster is registered and has no workload identity yet; turn on workload identity.
aws: [ERROR]: An error occurred (NoCredentials): Unable to locate credentials. You can configure credentials by running "aws login".: the add-on has no role annotation, or its pods started before it (step 3). On clusters AstroPulse created, a pod without the annotation gets no AWS identity at all. On nodes that let pods reach instance metadata it gets the node role instead, and the Arn shows the node role.
An AccessDenied error naming route53:ListResourceRecordSets or route53:ChangeResourceRecordSets and another hostedzone/ in kubectl describe challenge -n default: hostedZoneId and the policy's zone differ. Run Undo, fix step 1, then run step 2 again.
Apply refused: dnsSolver needs identity.serviceAccountAnnotations on cluster "CLUSTER": ...: the identity block is missing. Run step 3 again.

Undo. In a new terminal, run step 1 first.

Terminal
aws iam detach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"
aws iam delete-role --role-name "${ROLE_NAME}"
aws iam delete-policy --policy-arn "${POLICY_ARN}"

Expected after: aws iam get-role --role-name "${ROLE_NAME}" prints aws: [ERROR]: An error occurred (NoSuchEntity) when calling the GetRole operation: The role with name cert-manager-my-cluster cannot be found.

Then remove identity and dnsSolver from the manifest and apply it again.

Model Serving (`model-serving`)

Let models read a private S3 bucket

Read access to one bucket for the models in one namespace (NAMESPACE/astro-model-*) and the weight cache (kserve-localmodel-jobs/default). Set once on the add-on, never on an application. Background: IAM roles for service accounts.

Needs the AWS CLI, kubectl and astroctl. Registered cluster: turn on workload identity first.

1. Set your values

Terminal
export REGION=us-west-2
export CLUSTER=my-cluster
export NAMESPACE=models
export APPLICATION=my-model
export BUCKET=acme-models
export ROLE_NAME=model-reader-${CLUSTER}
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export POLICY_ARN=arn:aws:iam::${ACCOUNT_ID}:policy/${ROLE_NAME}
export OIDC_HOST=$(aws eks describe-cluster --name "${CLUSTER}" --region "${REGION}" \
  --query cluster.identity.oidc.issuer --output text | sed 's#^https://##')

No bucket yet? Create it and block public access. In us-east-1, leave out --create-bucket-configuration.

Terminal
aws s3api create-bucket --bucket "${BUCKET}" --region "${REGION}" \
  --create-bucket-configuration LocationConstraint="${REGION}"
aws s3api put-public-access-block --bucket "${BUCKET}" --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

APPLICATION is one model application in NAMESPACE, used by the check.

2. Create the role

Terminal
cat > policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::${BUCKET}" },
    { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::${BUCKET}/*" }
  ]
}
EOF
cat > trust.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_HOST}" },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": { "StringLike": {
      "${OIDC_HOST}:sub": ["system:serviceaccount:${NAMESPACE}:astro-model-*",
                    "system:serviceaccount:kserve-localmodel-jobs:default"],
      "${OIDC_HOST}:aud": "sts.amazonaws.com"
    } }
  }]
}
EOF
aws iam create-policy --policy-name "${ROLE_NAME}" --policy-document file://policy.json \
  --query Policy.Arn --output text
export ROLE_ARN=$(aws iam create-role --role-name "${ROLE_NAME}" \
  --assume-role-policy-document file://trust.json --query Role.Arn --output text)
aws iam attach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"
echo "${ROLE_ARN}"

Models in several namespaces: * in place of ${NAMESPACE}, as the console suggests; this lets any namespace with an astro-model-* ServiceAccount use the role. One model only: astro-model-${APPLICATION} in place of astro-model-*. No weight cache: drop the kserve-localmodel-jobs line.

3. Put the role on the add-on

Console: the cluster's Add-ons (https://astropulse.io/console?type=clusters&cluster=my-cluster&tab=add-ons), Update, then Change configuration, on Model Serving (or Install Add-on), then set Cloud access to Workload identity, on the Identity step set Cloud to Amazon Web Services, and paste the ARN into AWS role ARN.

Or:

Terminal
cat > model-serving.yaml <<EOF
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: model-serving
spec:
  clusterName: ${CLUSTER}
  configuration:
    modelServing:
      identity:
        serviceAccountAnnotations:
          eks.amazonaws.com/role-arn: ${ROLE_ARN}
EOF
astroctl infra k8s addons apply -f model-serving.yaml

The cluster's models restart to pick up the role.

4. Check

A model's identity:

Terminal
kubectl -n "${NAMESPACE}" run cloud-check --rm -i --restart=Never --image=amazon/aws-cli:2.39.3 \
  --override-type=strategic --overrides="{\"spec\":{\"serviceAccountName\":\"astro-model-${APPLICATION}\"}}" \
  -- sts get-caller-identity --query Arn --output text

Expected:

Output
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
arn:aws:sts::123456789012:assumed-role/model-reader-my-cluster/botocore-session-<generated>
pod "cloud-check" deleted from models namespace

Namespace refuses the test pod (Pod Security "restricted")

Terminal
kubectl -n "${NAMESPACE}" run cloud-check --rm -i --restart=Never --image=amazon/aws-cli:2.39.3 \
  --override-type=strategic \
  --overrides='{"spec":{"serviceAccountName":"'"astro-model-${APPLICATION}"'","securityContext":{"runAsNonRoot":true,"runAsUser":65534,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"cloud-check","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]}}' \
  -- sts get-caller-identity --query Arn --output text

Then a read of the bucket as that model:

Terminal
kubectl -n "${NAMESPACE}" run cloud-check --rm -i --restart=Never --image=amazon/aws-cli:2.39.3 \
  --override-type=strategic --overrides="{\"spec\":{\"serviceAccountName\":\"astro-model-${APPLICATION}\"}}" \
  -- s3 ls "s3://${BUCKET}/"

Expected: the bucket's top-level folders as PRE name/.

Namespace refuses the test pod (Pod Security "restricted")

Terminal
kubectl -n "${NAMESPACE}" run cloud-check --rm -i --restart=Never --image=amazon/aws-cli:2.39.3 \
  --override-type=strategic \
  --overrides='{"spec":{"serviceAccountName":"'"astro-model-${APPLICATION}"'","securityContext":{"runAsNonRoot":true,"runAsUser":65534,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"cloud-check","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]}}' \
  -- s3 ls "s3://${BUCKET}/"

Then the add-on and the model:

Terminal
astroctl infra k8s addons get "${CLUSTER}" model-serving
astroctl app status "${APPLICATION}"

Expected: the add-on shows the subjects it expects (the cache line only with the weight cache on), and the model leaves its download step and serves.

Output
models trust subject: system:serviceaccount:*:astro-model-*
cache trust subject: system:serviceaccount:kserve-localmodel-jobs:default

If it fails

Unless a bullet says where, the test pod in step 4 prints it.

aws: [ERROR]: An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity: the role ARN on the add-on is not the one step 2 printed (fix step 3), or the pod does not run as an astro-model-* ServiceAccount in NAMESPACE, which the role trusts.
aws: [ERROR]: An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: No OpenIDConnect provider found in your account for https://...: the cluster is registered and has no workload identity yet; turn on workload identity.
aws: [ERROR]: An error occurred (NoCredentials): Unable to locate credentials. You can configure credentials by running "aws login".: the add-on has no role annotation, or its pods started before it (step 3). On clusters AstroPulse created, a pod without the annotation gets no AWS identity at all. On nodes that let pods reach instance metadata it gets the node role instead, and the Arn shows the node role.
aws: [ERROR]: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: User: ... is not authorized to perform: s3:ListBucket on resource: "arn:aws:s3:::BUCKET" because no identity-based policy allows ...: the step 2 policy names another bucket than ${BUCKET}. Run Undo, fix step 1, then run step 2 again.
An AccessDenied error for s3:GetObject in the model pod's storage-initializer log (kubectl -n "${NAMESPACE}" get pods lists the pod): the policy names another bucket than the model's uri. Run Undo, fix step 1, then run step 2 again.

Undo. In a new terminal, run step 1 first.

Terminal
aws iam detach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"
aws iam delete-role --role-name "${ROLE_NAME}"
aws iam delete-policy --policy-arn "${POLICY_ARN}"

Expected after: aws iam get-role --role-name "${ROLE_NAME}" prints aws: [ERROR]: An error occurred (NoSuchEntity) when calling the GetRole operation: The role with name model-reader-my-cluster cannot be found.

Then set Cloud access back to None, or remove identity from the manifest and apply it again.

If it's refused

My app api in namespace shop on my-cluster gets AccessDenied from S3. Which part is missing, and how do I fix it?

Ask Nova

Nova reads the app's logs, events and settings and the cluster, names the missing part (the identity, the trust or the grant), and writes the fix.

When your app's call to your cloud is refused:

InvalidIdentityToken ... No OpenIDConnect provider found

Cause

The cluster gives pods no identity: a registered cluster without workload identity.

Fix

Turn on workload identity.

AccessDenied ... AssumeRoleWithWebIdentity

Cause

The role's trust policy names another namespace or ServiceAccount, or the ServiceAccount carries another role's ARN.

Fix

Make the trust name exactly the app's NAMESPACE and SERVICE_ACCOUNT, and put the right ARN on the app.

NoCredentials: Unable to locate credentials

Cause

The ServiceAccount has no role annotation, or the pods started before it was added.

Fix

Put the role on the app, then restart its pods.

AccessDenied ... is not authorized to perform: s3:ListBucket

Cause

The pod signs in, but the role's policy names another bucket or action.

Fix

Grant the action on that exact resource. IAM changes can take a few minutes.

When an add-on's identity is refused at apply:

"identity.serviceAccountAnnotations["eks.amazonaws.com/role-arn"] must be an IAM role ARN (arn:aws:iam::<account-id>:role/<name>)"

Cause

The value on the add-on's identity is not the kind of identity EKS expects.

Fix

Paste the value the recipe printed, unchanged.

"identity.serviceAccountAnnotations must carry a cloud workload-identity annotation: eks.amazonaws.com/role-arn (IRSA), iam.gke.io/gcp-service-account (GKE Workload Identity), or azure.workload.identity/client-id (AKS Workload Identity)"

Cause

identity was set with no workload-identity annotation.

Fix

Add the eks.amazonaws.com/role-arn annotation from the recipe.

Other ways in

Nova: ask, for example, "why can't DNS Automation on my-cluster change records?". Nova reads the add-on's health and the cluster's cloud identity.
SDK and API: an add-on's identity is part of its configuration. See the API reference.

With astroctl

Each recipe ends by applying the add-on with its identity:

Terminal
astroctl infra k8s addons apply -f ADDON.yaml
astroctl infra k8s addons get my-cluster ADDON

Keyless cloud access: how keyless works, and how your app gets its identity.
Registry access: pull your images, and push them from a build.
Deploy an application.

Before you start​

In the console​

You should see​

Recipes​

Turn on workload identity on a registered cluster​

Your own applications​

Reading a bucket​

Give my application access to an S3 bucket​

Calling a queue, topic or vault​

Give my application access to an SQS queue​

Pulling images from a private registry​

Let EKS nodes pull from an ECR repository in another AWS account​

Platform add-ons​

Vulnerability Scanning​

Let vulnerability scanning read private images​

DNS Automation (external-dns)​

Let DNS Automation change records in one Route 53 zone​

Certificates (cert-manager), DNS-01 check​

Let Certificates add DNS challenge records in one Route 53 zone​

Model Serving (model-serving)​

Let models read a private S3 bucket​

If it's refused​

Other ways in​

With astroctl​

Next​

Before you start

In the console

You should see

Recipes

Turn on workload identity on a registered cluster

Your own applications

Reading a bucket

Give my application access to an S3 bucket

Calling a queue, topic or vault

Give my application access to an SQS queue

Pulling images from a private registry

Let EKS nodes pull from an ECR repository in another AWS account

Platform add-ons

Vulnerability Scanning

Let vulnerability scanning read private images

DNS Automation (`external-dns`)

Let DNS Automation change records in one Route 53 zone

Certificates (`cert-manager`), DNS-01 check

Let Certificates add DNS challenge records in one Route 53 zone

Model Serving (`model-serving`)

Let models read a private S3 bucket

If it's refused

Other ways in

With astroctl

Next