Give a workload access to Azure

What do I need to connect my app api in namespace shop on my-cluster to the Blob container acme-data?

Give DNS Automation on my-cluster access to the Azure DNS zone example.com.

Turn on workload identity for my registered AKS cluster my-cluster.

This is the hard part, and Nova does it with you. Name your app or add-on, the cluster and the resource: Nova reads the cluster and the workload's settings, and writes the identity, the grant, the trust and the values for your app, ready to run.

Each recipe gives one workload on an AKS cluster, an add-on or one of your applications, access to one Azure resource: a Blob container, an Azure Container Registry, an Azure DNS zone or a Key Vault secret. A pod signs in as a managed identity trusted only by its own namespace and ServiceAccount on this cluster, so no Azure key is stored on the cluster. Each grant reaches one resource, and every recipe has a check you can run and an undo that removes what it created.

Before you start

The Azure CLI signed in with rights to create identities and grant access, kubectl pointed at the cluster, and astroctl signed in.
Workload identity on the cluster. Clusters AstroPulse creates have it; a registered cluster turns it on once.

In the console

Open the cluster from the cluster list. On its Overview, Cloud identity shows who the cluster is in Azure: under Per application (recommended), Workload identity and the OIDC issuer your trust policy names, and under All nodes, the identity the nodes run as.
Create the identity and its grant with the recipe below for your add-on or application.
Open the cluster's Add-ons tab and open the add-on, then Update and Change configuration. For DNS Automation and Model Serving, set Cloud access to Workload identity; Certificates and Vulnerability Scanning take Identity directly. On the Identity step, pick the Cloud and enter the identity the recipe printed. Go to Review, select Continue to apply once the dry run passes, type the add-on's name (such as external-dns), then select Update external-dns.

You should see

Each recipe's check prints the identity the workload signs in as, then reads the resource. The add-on's health shows no access warning.

Recipes

Your own applications:

Give my application access to an Azure Blob container
Give my application access to a Key Vault secret
Let AKS nodes pull from an Azure Container Registry
Pull from any registry with a pull Secret, on any cloud

Platform add-ons:

Let vulnerability scanning read private images
Let DNS Automation change records in one Azure DNS zone
Let Certificates add DNS challenge records in one Azure DNS zone
Let models read a private Blob container

Turn on workload identity on a registered cluster

Clusters AstroPulse created have it: skip this. A registered cluster does not report it in the console, so check it with the command in step 2. On AKS this is the cluster's OIDC issuer and the workload identity feature. Background: Use Microsoft Entra Workload ID with AKS.

1. Set your values

Terminal
export CLUSTER=my-cluster
export RESOURCE_GROUP=my-group

2. Check

Terminal
az aks show --resource-group "${RESOURCE_GROUP}" --name "${CLUSTER}" \
  --query oidcIssuerProfile.issuerUrl --output tsv
az aks show --resource-group "${RESOURCE_GROUP}" --name "${CLUSTER}" \
  --query securityProfile.workloadIdentity.enabled --output tsv

Expected when on: an issuer URL such as https://eastus.oic.prod-aks.azure.com/00000000-0000-0000-0000-000000000000/11111111-1111-1111-1111-111111111111/, then true. Nothing printed, or false, means it is off.

3. Turn it on

Terminal
az aks update --resource-group "${RESOURCE_GROUP}" --name "${CLUSTER}" \
  --enable-oidc-issuer --enable-workload-identity

Run step 2 again to confirm. Restart pods created before the change so they get the federated token.

Undo. Every workload that uses an Azure workload identity stops working. The OIDC issuer stays on.

Terminal
az aks update --resource-group "${RESOURCE_GROUP}" --name "${CLUSTER}" --disable-workload-identity

Placeholders used on this page. Replace each with your own value.

Placeholder	Meaning
`CLUSTER`	Cluster name
`RESOURCE_GROUP`	Azure resource group of the cluster and identity
`NAMESPACE`	Kubernetes namespace of the workload
`SERVICE_ACCOUNT`	Kubernetes ServiceAccount name of the workload
`STORAGE_ACCOUNT`, `CONTAINER`	Azure storage account and blob container
`REPOSITORY`, `REGISTRY`	Image repository (ECR, Artifact Registry) or Azure registry name
`IDENTITY_NAME`	Name of the Azure managed identity you create
`ZONE`, `DNS_RESOURCE_GROUP`	DNS zone name and its resource group (Azure)
`APPLICATION`, `APP`, `POD`, `SERVICE`	Name of the Astro application, its Deployment, a pod, and a Kubernetes Service in the recipe's namespace
`SUBSCRIPTION`, `TOPIC`, `QUEUE_NAME`, `QUEUE_URL`, `VAULT`, `SECRET_NAME`	Names of the Pub/Sub subscription and topic, SQS queue, Key Vault and secret you are granting access to
`ROLE_ARN`, `CLIENT_ID`, `PRINCIPAL_ID`, `TENANT_ID`, `SUBSCRIPTION_ID`	Values printed by earlier commands in the same recipe

Your own applications

Bind the identity on the ServiceAccount your workload already runs as. Where that is defined depends on the application source: see How your app gets its identity.

Reading a bucket

Give my application access to an Azure Blob container

Do I need to do anything?

Your situation	What to do
Any cluster	Manual: the steps below

Before you start. Workload identity must be on for the cluster: Enable workload identity on AKS. One-line check: az aks show -g RESOURCE_GROUP -n CLUSTER --query securityProfile.workloadIdentity.enabled -o tsv prints true.

Values you need. Find each one with the command shown.

Placeholder	What it is	Command that finds it
`CLUSTER`	Your cluster name	`az aks list -o table`
`RESOURCE_GROUP`	Resource group of the cluster	`az aks list --query "[].{name:name,resourceGroup:resourceGroup}" -o table`
`TENANT_ID`	Azure tenant ID	`az account show --query tenantId -o tsv`
`CLIENT_ID`	Client ID of the managed identity you create	`az identity show -g RESOURCE_GROUP -n IDENTITY_NAME --query clientId -o tsv`
`PRINCIPAL_ID`	Principal (object) ID of that identity	`az identity show -g RESOURCE_GROUP -n IDENTITY_NAME --query principalId -o tsv`
`STORAGE`	Storage account and container	`az storage account list -o table` `az storage container list --account-name STORAGE_ACCOUNT --auth-mode login -o table`
`APPNS`	Your workload's namespace and ServiceAccount (empty means the app runs as `default`: give it its own ServiceAccount first, see how)	`kubectl get deploy -A` `kubectl -n NAMESPACE get deploy APP -o jsonpath='{.spec.template.spec.serviceAccountName}'`

Steps.

Create a managed identity. This creates an empty identity. It can do nothing until you grant it a role.

Terminal
az identity create --resource-group RESOURCE_GROUP --name IDENTITY_NAME
az identity show --resource-group RESOURCE_GROUP --name IDENTITY_NAME \
  --query "{clientId:clientId, principalId:principalId}"

Trust one ServiceAccount. This lets only system:serviceaccount:NAMESPACE:SERVICE_ACCOUNT on this cluster sign in as the identity.

Terminal
az identity federated-credential create --name SERVICE_ACCOUNT \
  --identity-name IDENTITY_NAME --resource-group RESOURCE_GROUP \
  --issuer "$(az aks show --resource-group RESOURCE_GROUP --name CLUSTER --query oidcIssuerProfile.issuerUrl --output tsv)" \
  --subject system:serviceaccount:NAMESPACE:SERVICE_ACCOUNT \
  --audiences api://AzureADTokenExchange

Grant read on one container, on one resource only. This lets the identity act as Storage Blob Data Reader on that resource and nowhere else.

Terminal
az role assignment create --assignee-object-id PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal --role "Storage Blob Data Reader" \
  --scope "$(az storage account show --name STORAGE_ACCOUNT --query id --output tsv)/blobServices/default/containers/CONTAINER"

Put the identity on your app's ServiceAccount. Set it on the application, so AstroPulse deploys it with the app (how): in the console, the app's Update › Values; in a manifest, under source.helm.values.object, then astroctl app apply -f application.yaml. With your own manifests, put the annotation on the ServiceAccount and the label on the pod template in your repository, and commit. These are the usual chart keys; check your chart's values.yaml.

values.yaml
serviceAccount:
  create: true
  name: SERVICE_ACCOUNT
  annotations:
    azure.workload.identity/client-id: CLIENT_ID
podLabels:
  azure.workload.identity/use: "true"

Validate. Three checks, in order. Stop at the first one that fails.

Check 1: the cluster is enabled (Enable workload identity on AKS).

Terminal
az aks show -g RESOURCE_GROUP -n CLUSTER --query oidcIssuerProfile.issuerUrl -o tsv
az aks show -g RESOURCE_GROUP -n CLUSTER --query securityProfile.workloadIdentity.enabled -o tsv

Expected: an issuer URL, then true. If not, stop and go to Enable workload identity on AKS.

Check 2: the identity works for this ServiceAccount. The ServiceAccount must already exist in the namespace. Run a throwaway pod as the ServiceAccount, with the workload identity label, and sign in with its federated token.

Terminal
kubectl run wi-test -n NAMESPACE --rm -it --restart=Never \
  --labels=azure.workload.identity/use=true \
  --image=mcr.microsoft.com/azure-cli \
  --override-type=strategic --overrides='{"spec":{"serviceAccountName":"SERVICE_ACCOUNT"}}' \
  -- sh -c 'az login --federated-token "$(cat $AZURE_FEDERATED_TOKEN_FILE)" --service-principal -u $AZURE_CLIENT_ID -t $AZURE_TENANT_ID --allow-no-subscriptions'

Expected: the command succeeds and prints the signed-in service principal.

If the namespace enforces the Pod Security "restricted" level and rejects the pod, use the override below that adds a securityContext, or run the test in a namespace that allows it with a ServiceAccount bound identically (same annotation, and a trust or grant that names that namespace).

Restricted-namespace override (also add --env HOME=/tmp): {"spec":{"serviceAccountName":"SERVICE_ACCOUNT","securityContext":{"runAsNonRoot":true,"runAsUser":65534,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"wi-test","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]}}

AADSTS70021: No matching federated identity record found for presented assertion (or AADSTS700213, the same message)

Cause

The federated credential's subject is not exactly system:serviceaccount:NAMESPACE:SERVICE_ACCOUNT, or its issuer is not this cluster's.

AZURE_FEDERATED_TOKEN_FILE is empty

Cause

The pod lacks the label azure.workload.identity/use: "true", or workload identity is off (Check 1).

If this fails, the binding is wrong; fix it before looking at the workload.

Check 3: the workload does its job.

Run the same "who am I" check from your application's own pod, then the call your application makes.

Terminal
kubectl -n NAMESPACE exec deploy/APP -- env | grep AZURE_

Terminal
kubectl -n NAMESPACE run cloud-check --rm -it --restart=Never \
  --labels=azure.workload.identity/use=true \
  --image=mcr.microsoft.com/azure-cli \
  --override-type=strategic --overrides='{"spec":{"serviceAccountName":"SERVICE_ACCOUNT"}}' \
  -- sh -c 'az login --federated-token "$(cat $AZURE_FEDERATED_TOKEN_FILE)" --service-principal -u $AZURE_CLIENT_ID -t $AZURE_TENANT_ID --allow-no-subscriptions >/dev/null && az storage blob list --account-name STORAGE_ACCOUNT --container-name CONTAINER --auth-mode login --query "[].name" -o tsv'

Expected: the call succeeds.

Then confirm your application's own call succeeds (its logs show no access error).

TROUBLESHOOT.

AADSTS70021: No matching federated identity record found for presented assertion (or AADSTS700213)

Fix

The federated credential's subject is not exactly system:serviceaccount:NAMESPACE:SERVICE_ACCOUNT, or its issuer is not this cluster's (step 2).

AuthorizationPermissionMismatch

Fix

The identity lacks Storage Blob Data Reader on that container (step 3).

The AZURE_* variables are missing

Fix

The pod lacks the azure.workload.identity/use: "true" label, was created before the change, or workload identity is off. See Enable workload identity on AKS.

Undo. To remove this access:

Terminal
az role assignment delete --assignee PRINCIPAL_ID --role "Storage Blob Data Reader" --scope "$(az storage account show --name STORAGE_ACCOUNT --query id --output tsv)/blobServices/default/containers/CONTAINER"
az identity federated-credential delete --name SERVICE_ACCOUNT --identity-name IDENTITY_NAME --resource-group RESOURCE_GROUP --yes
az identity delete --resource-group RESOURCE_GROUP --name IDENTITY_NAME

This removes the role assignment, the federated credential and the identity. Delete the identity last. Then remove the annotation and pod label.

Calling a queue, topic or vault

The steps are the same as the bucket recipe above; only the grant changes: the role, scoped to one resource.

Give my application access to a Key Vault secret

Do I need to do anything?

Your situation	What to do
Any cluster	Manual: the steps below

Values you need. Find each one with the command shown.

Placeholder	What it is	Command that finds it
`CLUSTER`	Your cluster name	`az aks list -o table`
`RESOURCE_GROUP`	Resource group of the cluster	`az aks list --query "[].{name:name,resourceGroup:resourceGroup}" -o table`
`TENANT_ID`	Azure tenant ID	`az account show --query tenantId -o tsv`
`CLIENT_ID`	Client ID of the managed identity you create	`az identity show -g RESOURCE_GROUP -n IDENTITY_NAME --query clientId -o tsv`
`PRINCIPAL_ID`	Principal (object) ID of that identity	`az identity show -g RESOURCE_GROUP -n IDENTITY_NAME --query principalId -o tsv`
`VAULT`	Key Vault and secret	`az keyvault list -o table` `az keyvault secret list --vault-name VAULT -o table`
`APPNS`	Your workload's namespace and ServiceAccount (empty means the app runs as `default`: give it its own ServiceAccount first, see how)	`kubectl get deploy -A` `kubectl -n NAMESPACE get deploy APP -o jsonpath='{.spec.template.spec.serviceAccountName}'`

Steps.

Create a managed identity. This creates an empty identity. It can do nothing until you grant it a role.

Terminal
az identity create --resource-group RESOURCE_GROUP --name IDENTITY_NAME
az identity show --resource-group RESOURCE_GROUP --name IDENTITY_NAME \
  --query "{clientId:clientId, principalId:principalId}"

Trust one ServiceAccount. This lets only system:serviceaccount:NAMESPACE:SERVICE_ACCOUNT on this cluster sign in as the identity.

Terminal
az identity federated-credential create --name SERVICE_ACCOUNT \
  --identity-name IDENTITY_NAME --resource-group RESOURCE_GROUP \
  --issuer "$(az aks show --resource-group RESOURCE_GROUP --name CLUSTER --query oidcIssuerProfile.issuerUrl --output tsv)" \
  --subject system:serviceaccount:NAMESPACE:SERVICE_ACCOUNT \
  --audiences api://AzureADTokenExchange

Grant read on one secret (the vault must use Azure RBAC), on one resource only. This lets the identity act as Key Vault Secrets User on that resource and nowhere else.

Terminal
az role assignment create --assignee-object-id PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal --role "Key Vault Secrets User" \
  --scope "$(az keyvault show --name VAULT --query id --output tsv)/secrets/SECRET_NAME"

Put the identity on your app's ServiceAccount. Set it on the application, so AstroPulse deploys it with the app (how): in the console, the app's Update › Values; in a manifest, under source.helm.values.object, then astroctl app apply -f application.yaml. With your own manifests, put the annotation on the ServiceAccount and the label on the pod template in your repository, and commit. These are the usual chart keys; check your chart's values.yaml.

values.yaml
serviceAccount:
  create: true
  name: SERVICE_ACCOUNT
  annotations:
    azure.workload.identity/client-id: CLIENT_ID
podLabels:
  azure.workload.identity/use: "true"

Validate. Three checks, in order. Stop at the first one that fails.

Check 1: the cluster is enabled (Enable workload identity on AKS).

Terminal
az aks show -g RESOURCE_GROUP -n CLUSTER --query oidcIssuerProfile.issuerUrl -o tsv
az aks show -g RESOURCE_GROUP -n CLUSTER --query securityProfile.workloadIdentity.enabled -o tsv

Expected: an issuer URL, then true. If not, stop and go to Enable workload identity on AKS.

Terminal
kubectl run wi-test -n NAMESPACE --rm -it --restart=Never \
  --labels=azure.workload.identity/use=true \
  --image=mcr.microsoft.com/azure-cli \
  --override-type=strategic --overrides='{"spec":{"serviceAccountName":"SERVICE_ACCOUNT"}}' \
  -- sh -c 'az login --federated-token "$(cat $AZURE_FEDERATED_TOKEN_FILE)" --service-principal -u $AZURE_CLIENT_ID -t $AZURE_TENANT_ID --allow-no-subscriptions'

Expected: the command succeeds and prints the signed-in service principal.

AADSTS70021: No matching federated identity record found for presented assertion (or AADSTS700213, the same message)

Cause

The federated credential's subject is not exactly system:serviceaccount:NAMESPACE:SERVICE_ACCOUNT, or its issuer is not this cluster's.

AZURE_FEDERATED_TOKEN_FILE is empty

Cause

The pod lacks the label azure.workload.identity/use: "true", or workload identity is off (Check 1).

If this fails, the binding is wrong; fix it before looking at the workload.

Check 3: the workload does its job.

Run the same "who am I" check from your application's own pod, then the call your application makes.

Terminal
kubectl -n NAMESPACE exec deploy/APP -- env | grep AZURE_

Terminal
kubectl -n NAMESPACE run cloud-check --rm -it --restart=Never \
  --labels=azure.workload.identity/use=true \
  --image=mcr.microsoft.com/azure-cli \
  --override-type=strategic --overrides='{"spec":{"serviceAccountName":"SERVICE_ACCOUNT"}}' \
  -- sh -c 'az login --federated-token "$(cat $AZURE_FEDERATED_TOKEN_FILE)" --service-principal -u $AZURE_CLIENT_ID -t $AZURE_TENANT_ID --allow-no-subscriptions >/dev/null && az keyvault secret show --vault-name VAULT --name SECRET_NAME --query name -o tsv'

Expected: the call succeeds.

Then confirm your application's own call succeeds (its logs show no access error).

TROUBLESHOOT.

AADSTS70021: No matching federated identity record found for presented assertion (or AADSTS700213)

Fix

The federated credential's subject is not exactly system:serviceaccount:NAMESPACE:SERVICE_ACCOUNT, or its issuer is not this cluster's (step 2).

(Forbidden) Caller is not authorized to perform action on resource with ForbiddenByRbac

Fix

The role assignment is missing (step 3), or the vault uses access policies instead of Azure RBAC.

SecretNotFound

Fix

SECRET_NAME or VAULT is wrong.

The AZURE_* variables are missing

Fix

The pod lacks the azure.workload.identity/use: "true" label, was created before the change, or workload identity is off. See Enable workload identity on AKS.

Undo. To remove this access:

Terminal
az role assignment delete --assignee PRINCIPAL_ID --role "Key Vault Secrets User" --scope "$(az keyvault show --name VAULT --query id --output tsv)/secrets/SECRET_NAME"
az identity federated-credential delete --name SERVICE_ACCOUNT --identity-name IDENTITY_NAME --resource-group RESOURCE_GROUP --yes
az identity delete --resource-group RESOURCE_GROUP --name IDENTITY_NAME

This removes the role assignment, the federated credential and the identity. Delete the identity last. Then remove the annotation and pod label.

Pulling images from a private registry

Image pulls are done by the node, not by the pod's ServiceAccount (see why). Grant the node identity access, or use an pull Secret.

Let AKS nodes pull from an Azure Container Registry

Nodes pull images with the cluster's kubelet identity, not with workload identity. The registry gives that identity AcrPull, and images then pull with no secret. Background: Authenticate with ACR from AKS.

Clusters AstroPulse created: list the registry on the cluster, and AstroPulse grants AcrPull for you (step 2). AstroPulse's connection to Azure needs permission to assign roles on the registry, for example Owner or User Access Administrator on it. Registered cluster: attach it yourself (step 3).

1. Set your values

Terminal
export CLUSTER=my-cluster
export RESOURCE_GROUP=my-group
export REGISTRY=acmeregistry
export REGISTRY_ID=$(az acr show --name "${REGISTRY}" --query id --output tsv)

For a registry in another subscription, add --subscription REGISTRY_SUBSCRIPTION to az acr show.

2. List the registry on the cluster (clusters AstroPulse created)

At create, the wizard's Network step has it in the collapsed section at its foot: Container registries (ACR). On a running cluster:

Terminal
cat > acr.yaml <<EOF
apiVersion: platform.astropulse.io/v1
kind: K8sClusterUpdate
spec:
  aks:
    acrResourceIds: ["${REGISTRY_ID}"]
EOF
astroctl infra k8s update "${CLUSTER}" -f acr.yaml --dry-run
astroctl infra k8s update "${CLUSTER}" -f acr.yaml

acrResourceIds replaces the whole list: added registries get AcrPull, removed ones lose it.

3. Attach the registry yourself (registered cluster)

Terminal
az aks update --resource-group "${RESOURCE_GROUP}" --name "${CLUSTER}" --attach-acr "${REGISTRY_ID}"

4. Check

Deploy a workload with an image from the registry, then:

Terminal
kubectl get events -A --field-selector reason=Pulled

Expected: a Successfully pulled image "REGISTRY.azurecr.io/IMAGE:TAG" event.

If it fails

ImagePullBackOff with 401 Unauthorized in kubectl describe pod: the kubelet identity has no AcrPull on that registry. The cluster's Cloud identity shows the Kubelet identity to check the role assignment against.

Undo. Clusters AstroPulse created: send acrResourceIds without the registry. Registered cluster:

Terminal
az aks update --resource-group "${RESOURCE_GROUP}" --name "${CLUSTER}" --detach-acr "${REGISTRY_ID}"

Platform add-ons

Each add-on runs as one fixed ServiceAccount. You give that account access, then name the identity on the add-on's identity field. The field has this shape on every add-on that takes it:

values.yaml
identity:
  serviceAccountAnnotations: {}   # for example azure.workload.identity/client-id: CLIENT_ID
  serviceAccountLabels: {}        # Azure only: azure.workload.identity/use: "true"

Add-on	Namespace	ServiceAccount	Needs
Vulnerability Scanning (`vulnerability-scanning`)	`astro-security`	`trivy-operator`	Read images in one registry
DNS Automation (`external-dns`)	`external-dns`	`external-dns`	Change records in one DNS zone
Certificates (`cert-manager`)	`cert-manager`	`cert-manager`	Add and remove TXT records in one DNS zone (DNS-01 check only)
Model Serving (`model-serving`)	each model's own namespace	`astro-model-APPLICATION` for each model	Read the model bucket

Applying an add-on's identity is covered in Managing cluster add-ons.

Vulnerability Scanning

The scanner reads each image from its registry through the registry's API, so it needs read access to that registry. It runs as trivy-operator in astro-security. Clusters AstroPulse created get this access when the add-on is applied. Registered clusters, and registries in another account, project or subscription, need the steps below.

Let vulnerability scanning read private images

Do I need to do anything?

Your situation	What to do
Registry the cluster already pulls from, cluster created by AstroPulse	Automatic, when you apply the add-on
Registered cluster, or a registry in another subscription	Manual: the steps below

Clusters created by AstroPulse: nothing to do for registries the cluster already pulls from in its own subscription. Workload identity is enabled, and a managed identity with AcrPull is set up when you apply this add-on. The steps below are for clusters you registered, and for a registry in another subscription (always manual). A cluster created earlier gets access the next time you apply the add-on, and that re-apply is not automatic.

Values you need. Find each one with the command shown.

Placeholder	What it is	Command that finds it
`CLUSTER`	Your cluster name	`az aks list -o table`
`RESOURCE_GROUP`	Resource group of the cluster	`az aks list --query "[].{name:name,resourceGroup:resourceGroup}" -o table`
`CLIENT_ID`	Client ID of the managed identity you create	`az identity show -g RESOURCE_GROUP -n IDENTITY_NAME --query clientId -o tsv`
`PRINCIPAL_ID`	Principal (object) ID of that identity	`az identity show -g RESOURCE_GROUP -n IDENTITY_NAME --query principalId -o tsv`
`REGISTRY`	Azure Container Registry name	`az acr list -o table` `az acr show -n REGISTRY --query id -o tsv # resource ID`
`IMAGES`	Images the cluster runs	`kubectl get pods -A -o jsonpath='{..image}' \| tr ' ' '\n' \| sort -u`
`NAMESPACE, SERVICE_ACCOUNT`	Fixed for this add-on: `astro-security` and `trivy-operator`	`kubectl -n astro-security get sa trivy-operator`
`Current add-on configuration and health`	The add-on as applied	`astroctl infra k8s addons get CLUSTER vulnerability-scanning` `astroctl infra k8s addons get CLUSTER vulnerability-scanning --manifest`

On clusters created by AstroPulse, skip to check 3 (the workload does its job).

Steps.

Create a managed identity. This creates an empty identity. It can do nothing until you grant it a role.

Terminal
az identity create --resource-group RESOURCE_GROUP --name IDENTITY_NAME
az identity show --resource-group RESOURCE_GROUP --name IDENTITY_NAME \
  --query "{clientId:clientId, principalId:principalId}"

Trust one ServiceAccount. This lets only system:serviceaccount:astro-security:trivy-operator on this cluster sign in as the identity.

Terminal
az identity federated-credential create --name vulnerability-scanning \
  --identity-name IDENTITY_NAME --resource-group RESOURCE_GROUP \
  --issuer "$(az aks show --resource-group RESOURCE_GROUP --name CLUSTER --query oidcIssuerProfile.issuerUrl --output tsv)" \
  --subject system:serviceaccount:astro-security:trivy-operator \
  --audiences api://AzureADTokenExchange

Grant pull on one registry, on one resource only. This lets the identity act as AcrPull on that resource and nowhere else.

Terminal
az role assignment create --assignee-object-id PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal --role "AcrPull" \
  --scope "$(az acr show --name REGISTRY --query id --output tsv)"

Write the add-on manifest. This is only a file on your machine. Azure needs both the annotation and the label; the annotation alone never takes effect. Add azure.workload.identity/tenant-id as well if the identity lives in a different tenant than the cluster.

vulnerability-scanning.yaml
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: vulnerability-scanning
spec:
  clusterName: CLUSTER
  configuration:
    vulnerabilityScanning:
      identity:
        serviceAccountAnnotations:
          azure.workload.identity/client-id: CLIENT_ID
        serviceAccountLabels:
          azure.workload.identity/use: "true"

Apply the add-on. This installs or updates the add-on with the identity and settings in the file.

Terminal
astroctl infra k8s addons apply -f vulnerability-scanning.yaml

Validate. Three checks, in order. On clusters created by AstroPulse, skip to check 3 (the workload does its job). Stop at the first one that fails.

Check 1: the cluster is enabled (Enable workload identity on AKS).

Terminal
az aks show -g RESOURCE_GROUP -n CLUSTER --query oidcIssuerProfile.issuerUrl -o tsv
az aks show -g RESOURCE_GROUP -n CLUSTER --query securityProfile.workloadIdentity.enabled -o tsv

Expected: an issuer URL, then true. If not, stop and go to Enable workload identity on AKS.

Check 2: the identity works for this ServiceAccount. For an add-on, run this after you apply it: the add-on creates the ServiceAccount. Run a throwaway pod as the ServiceAccount, with the workload identity label, and sign in with its federated token.

Terminal
kubectl run wi-test -n astro-security --rm -it --restart=Never \
  --labels=azure.workload.identity/use=true \
  --image=mcr.microsoft.com/azure-cli \
  --override-type=strategic --overrides='{"spec":{"serviceAccountName":"trivy-operator"}}' \
  -- sh -c 'az login --federated-token "$(cat $AZURE_FEDERATED_TOKEN_FILE)" --service-principal -u $AZURE_CLIENT_ID -t $AZURE_TENANT_ID --allow-no-subscriptions'

Expected: the command succeeds and prints the signed-in service principal.

Restricted-namespace override (also add --env HOME=/tmp): {"spec":{"serviceAccountName":"trivy-operator","securityContext":{"runAsNonRoot":true,"runAsUser":65534,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"wi-test","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]}}

AADSTS70021: No matching federated identity record found for presented assertion (or AADSTS700213, the same message)

Cause

The federated credential's subject is not exactly system:serviceaccount:astro-security:trivy-operator, or its issuer is not this cluster's.

AZURE_FEDERATED_TOKEN_FILE is empty

Cause

The pod lacks the label azure.workload.identity/use: "true", or workload identity is off (Check 1).

If this fails, the binding is wrong; fix it before looking at the workload.

Check 3: the workload does its job.

Terminal
kubectl get vulnerabilityreports -A
astroctl infra k8s addons get CLUSTER vulnerability-scanning

Expected: a report for the private image, and no "images not scannable" note in the add-on's health. The first scan of an image takes a few minutes. After a new grant allow up to about 5 minutes for the cloud to apply it, then restart the workload (kubectl rollout restart deployment/NAME -n NAMESPACE) to scan it again now; otherwise a refused image is retried within 10 hours.

TROUBLESHOOT.

AADSTS70021: No matching federated identity record found for presented assertion (or AADSTS700213)

Fix

The federated credential's subject is not exactly system:serviceaccount:astro-security:trivy-operator, or its issuer is not this cluster's. Recreate it with step 2.

The pod has no AZURE_FEDERATED_TOKEN_FILE

Fix

The azure.workload.identity/use: "true" label is missing from serviceAccountLabels, or workload identity is off. See Enable workload identity on AKS.

N images not scannable: registry denied (...) in the add-on's health

Fix

The scanner cannot read that repository. Grant it (the grant step above), then restart the workload to scan it again.

Undo. To remove this access:

Terminal
az role assignment delete --assignee PRINCIPAL_ID --role "AcrPull" --scope "$(az acr show --name REGISTRY --query id --output tsv)"
az identity federated-credential delete --name vulnerability-scanning --identity-name IDENTITY_NAME --resource-group RESOURCE_GROUP --yes
az identity delete --resource-group RESOURCE_GROUP --name IDENTITY_NAME

This removes the role assignment, the federated credential and the identity. Delete the identity last. Then remove identity from the manifest and apply the add-on again.

DNS Automation (`external-dns`)

DNS Automation runs as external-dns in external-dns. Without cloud access it installs, runs healthy, and publishes nothing.

Let DNS Automation change records in one Azure DNS zone

Do I need to do anything?

Your situation	What to do
Any AKS cluster	Manual: the steps below

Needed on every AKS cluster, including ones AstroPulse created: the platform does not set up DNS access for this add-on on AKS. Workload identity itself is already enabled on clusters AstroPulse created.

Before you start. This path uses a Secret, not workload identity.

Values you need. Find each one with the command shown.

Placeholder	What it is	Command that finds it
`CLUSTER`	Your cluster name	`az aks list -o table`
`RESOURCE_GROUP`	Resource group of the cluster	`az aks list --query "[].{name:name,resourceGroup:resourceGroup}" -o table`
`SUBSCRIPTION_ID`	Azure subscription ID	`az account show --query id -o tsv`
`ZONE`	Azure DNS zone and its resource group (DNS_RESOURCE_GROUP)	`az network dns zone list --query "[].{zone:name,resourceGroup:resourceGroup}" -o table`
`NAMESPACE, SERVICE_ACCOUNT`	Fixed for this add-on: `external-dns` and `external-dns`	`kubectl -n external-dns get sa external-dns`
`Current add-on configuration and health`	The add-on as applied	`astroctl infra k8s addons get CLUSTER external-dns` `astroctl infra k8s addons get CLUSTER external-dns --manifest`

Steps.

Create a service principal. This creates an Azure application with a secret. It has no role yet. Note appId (APP_ID), password and tenant.

Terminal
az ad sp create-for-rbac --name external-dns

Let it change records in one zone. This gives the principal DNS Zone Contributor on that one zone.

Terminal
az role assignment create --assignee APP_ID --role "DNS Zone Contributor" \
  --scope "$(az network dns zone show --resource-group DNS_RESOURCE_GROUP --name ZONE --query id --output tsv)"

Let it read the zone's resource group. This gives the principal Reader on the resource group that holds the zone.

Terminal
az role assignment create --assignee APP_ID --role Reader \
  --scope "$(az group show --name DNS_RESOURCE_GROUP --query id --output tsv)"

Write the credentials file. This is only a file on your machine. Keep it out of version control.

azure.json
{
  "tenantId": "TENANT_ID",
  "subscriptionId": "SUBSCRIPTION_ID",
  "resourceGroup": "DNS_RESOURCE_GROUP",
  "aadClientId": "APP_ID",
  "aadClientSecret": "PASSWORD"
}

Create the Secret in the add-on's namespace. This creates the external-dns namespace and a Secret in it holding the file. Do it before you apply the add-on.

Terminal
kubectl create namespace external-dns
kubectl create secret generic azure-dns-credentials \
  --namespace external-dns --from-file=azure.json

Write the add-on manifest. This is only a file on your machine; nothing changes until you apply it.

external-dns.yaml
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: external-dns
spec:
  clusterName: CLUSTER
  configuration:
    externalDns:
      provider: azure
      domainFilter: example.com
      secretRef: azure-dns-credentials

Apply the add-on. This installs or updates the add-on with the identity and settings in the file.

Terminal
astroctl infra k8s addons apply -f external-dns.yaml

Validate. This path uses a Secret, not workload identity, so Checks 1 and 2 do not apply.

Check 3: the workload does its job.

Terminal
astroctl infra k8s addons get CLUSTER external-dns
astroctl infra k8s addons logs CLUSTER external-dns --level Error --follow=false --interactive=false
kubectl annotate service SERVICE external-dns.alpha.kubernetes.io/hostname=test.example.com
dig +short test.example.com

Expected: the add-on is healthy, the error log is empty (no 403), and within a minute or two the name resolves to the service's address.

TROUBLESHOOT.

AuthorizationFailed in the error log

Fix

The principal lacks DNS Zone Contributor on the zone or Reader on its resource group (steps 2 and 3).

The pod cannot read its config

Fix

An identity was set on AKS. Remove it and use secretRef.

Healthy, but no record appears

Fix

The Secret was created after the add-on started, or its key is not azure.json. Recreate it and apply again.

The Secret is a long-lived credential on the cluster: rotate it and keep the principal to this one zone.

Undo. To remove this access:

Terminal
kubectl -n external-dns delete secret azure-dns-credentials
az role assignment delete --assignee APP_ID
az ad sp delete --id APP_ID
az ad app delete --id APP_ID

Then remove secretRef from the manifest and apply the add-on again. Deleting the service principal does not remove its role assignments, and create-for-rbac also leaves an app registration behind, so the commands above remove the role assignments first, then the service principal, then the app registration.

Certificates (`cert-manager`), DNS-01 check

Only the DNS-01 check (dnsSolver) calls your cloud. It runs as cert-manager in cert-manager, and needs enableClusterIssuer: true and issuerType: LetsEncrypt (or LetsEncryptStaging). A dnsSolver with no identity is refused when you apply.

Let Certificates add DNS challenge records in one Azure DNS zone

Do I need to do anything?

Your situation	What to do
Any AKS cluster	Manual: the steps below

Needed on every AKS cluster, including ones AstroPulse created: the platform does not set up DNS access for this add-on on AKS. Workload identity itself is already enabled on clusters AstroPulse created.

Values you need. Find each one with the command shown.

Placeholder	What it is	Command that finds it
`CLUSTER`	Your cluster name	`az aks list -o table`
`RESOURCE_GROUP`	Resource group of the cluster	`az aks list --query "[].{name:name,resourceGroup:resourceGroup}" -o table`
`SUBSCRIPTION_ID`	Azure subscription ID	`az account show --query id -o tsv`
`CLIENT_ID`	Client ID of the managed identity you create	`az identity show -g RESOURCE_GROUP -n IDENTITY_NAME --query clientId -o tsv`
`PRINCIPAL_ID`	Principal (object) ID of that identity	`az identity show -g RESOURCE_GROUP -n IDENTITY_NAME --query principalId -o tsv`
`ZONE`	Azure DNS zone and its resource group (DNS_RESOURCE_GROUP)	`az network dns zone list --query "[].{zone:name,resourceGroup:resourceGroup}" -o table`
`NAMESPACE, SERVICE_ACCOUNT`	Fixed for this add-on: `cert-manager` and `cert-manager`	`kubectl -n cert-manager get sa cert-manager`
`Current add-on configuration and health`	The add-on as applied	`astroctl infra k8s addons get CLUSTER cert-manager` `astroctl infra k8s addons get CLUSTER cert-manager --manifest`

Steps.

Create a managed identity. This creates an empty identity. It can do nothing until you grant it a role.

Terminal
az identity create --resource-group RESOURCE_GROUP --name IDENTITY_NAME
az identity show --resource-group RESOURCE_GROUP --name IDENTITY_NAME \
  --query "{clientId:clientId, principalId:principalId}"

Trust one ServiceAccount. This lets only system:serviceaccount:cert-manager:cert-manager on this cluster sign in as the identity.

Terminal
az identity federated-credential create --name cert-manager \
  --identity-name IDENTITY_NAME --resource-group RESOURCE_GROUP \
  --issuer "$(az aks show --resource-group RESOURCE_GROUP --name CLUSTER --query oidcIssuerProfile.issuerUrl --output tsv)" \
  --subject system:serviceaccount:cert-manager:cert-manager \
  --audiences api://AzureADTokenExchange

Grant DNS record changes, on one resource only. This lets the identity act as DNS Zone Contributor on that resource and nowhere else.

Terminal
az role assignment create --assignee-object-id PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal --role "DNS Zone Contributor" \
  --scope "$(az network dns zone show --resource-group DNS_RESOURCE_GROUP --name ZONE --query id --output tsv)"

Look up the subscription ID. This changes nothing; it prints SUBSCRIPTION_ID for the manifest.

Terminal
az account show --query id --output tsv

Write the add-on manifest. This is only a file on your machine. Azure needs both the annotation and the label.

cert-manager.yaml
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: cert-manager
spec:
  clusterName: CLUSTER
  configuration:
    certManager:
      enableClusterIssuer: true
      issuerType: LetsEncrypt
      email: ops@example.com
      identity:
        serviceAccountAnnotations:
          azure.workload.identity/client-id: CLIENT_ID
        serviceAccountLabels:
          azure.workload.identity/use: "true"
      dnsSolver:
        provider: azure
        resourceGroup: DNS_RESOURCE_GROUP
        subscriptionId: SUBSCRIPTION_ID

Apply the add-on. This installs or updates the add-on with the identity and settings in the file, after a dry run that reports refusals without changing anything.

Terminal
astroctl infra k8s addons apply -f cert-manager.yaml --dry-run
astroctl infra k8s addons apply -f cert-manager.yaml

Validate. Three checks, in order. Stop at the first one that fails.

Check 1: the cluster is enabled (Enable workload identity on AKS).

Terminal
az aks show -g RESOURCE_GROUP -n CLUSTER --query oidcIssuerProfile.issuerUrl -o tsv
az aks show -g RESOURCE_GROUP -n CLUSTER --query securityProfile.workloadIdentity.enabled -o tsv

Expected: an issuer URL, then true. If not, stop and go to Enable workload identity on AKS.

Terminal
kubectl run wi-test -n cert-manager --rm -it --restart=Never \
  --labels=azure.workload.identity/use=true \
  --image=mcr.microsoft.com/azure-cli \
  --override-type=strategic --overrides='{"spec":{"serviceAccountName":"cert-manager"}}' \
  -- sh -c 'az login --federated-token "$(cat $AZURE_FEDERATED_TOKEN_FILE)" --service-principal -u $AZURE_CLIENT_ID -t $AZURE_TENANT_ID --allow-no-subscriptions'

Expected: the command succeeds and prints the signed-in service principal.

Restricted-namespace override (also add --env HOME=/tmp): {"spec":{"serviceAccountName":"cert-manager","securityContext":{"runAsNonRoot":true,"runAsUser":65534,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"wi-test","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]}}

AADSTS70021: No matching federated identity record found for presented assertion (or AADSTS700213, the same message)

Cause

The federated credential's subject is not exactly system:serviceaccount:cert-manager:cert-manager, or its issuer is not this cluster's.

AZURE_FEDERATED_TOKEN_FILE is empty

Cause

The pod lacks the label azure.workload.identity/use: "true", or workload identity is off (Check 1).

If this fails, the binding is wrong; fix it before looking at the workload.

Check 3: the workload does its job.

Terminal
astroctl infra k8s addons get CLUSTER cert-manager
kubectl apply -f dns-check.yaml
kubectl get certificate dns-check --watch

dns-check.yaml is a test certificate (below). Expected: challenges.dns01 in the tls block of get says the DNS check can issue, and the certificate becomes READY: True within a few minutes. It issues a real Let's Encrypt certificate and counts against its rate limits; delete it with kubectl delete certificate dns-check when you are done.

dns-check.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: dns-check
  namespace: default
  labels:
    platform.astropulse.io/challenge: dns01
spec:
  secretName: dns-check-tls
  dnsNames:
    - test.example.com
  issuerRef:
    name: astro-letsencrypt
    kind: ClusterIssuer

TROUBLESHOOT. On failure run kubectl describe challenge -n default.

AADSTS70021: No matching federated identity record found for presented assertion (or AADSTS700213)

Fix

The federated credential's subject is not exactly system:serviceaccount:cert-manager:cert-manager, or its issuer is not this cluster's (step 2).

AuthorizationFailed

Fix

The identity lacks DNS Zone Contributor on that zone (step 3).

No federated token in the pod

Fix

The label is missing from serviceAccountLabels, or workload identity is off. See Enable workload identity on AKS.

Undo. To remove this access:

Terminal
az role assignment delete --assignee PRINCIPAL_ID --role "DNS Zone Contributor" --scope "$(az network dns zone show --resource-group DNS_RESOURCE_GROUP --name ZONE --query id --output tsv)"
az identity federated-credential delete --name cert-manager --identity-name IDENTITY_NAME --resource-group RESOURCE_GROUP --yes
az identity delete --resource-group RESOURCE_GROUP --name IDENTITY_NAME

This removes the role assignment, the federated credential and the identity. Delete the identity last. Then remove identity from the manifest and apply the add-on again.

Model Serving (`model-serving`)

Model Serving reads private model artifacts for every model on the cluster with one identity, set on the add-on, never on an application. Each model runs as ServiceAccount astro-model-APPLICATION in its application's namespace NAMESPACE, so your cloud grants must cover those names. (The add-on's own controllers live in the kserve namespace.) Applications cannot set credentials.

Let models read a private Blob container

Do I need to do anything?

Your situation	What to do
Any cluster	Manual: the steps below

Values you need. Find each one with the command shown.

Placeholder	What it is	Command that finds it
`CLUSTER`	Your cluster name	`az aks list -o table`
`RESOURCE_GROUP`	Resource group of the cluster	`az aks list --query "[].{name:name,resourceGroup:resourceGroup}" -o table`
`CLIENT_ID`	Client ID of the managed identity you create	`az identity show -g RESOURCE_GROUP -n IDENTITY_NAME --query clientId -o tsv`
`PRINCIPAL_ID`	Principal (object) ID of that identity	`az identity show -g RESOURCE_GROUP -n IDENTITY_NAME --query principalId -o tsv`
`STORAGE`	Storage account and container	`az storage account list -o table` `az storage container list --account-name STORAGE_ACCOUNT --auth-mode login -o table`
`NAMESPACE, APPLICATION`	The model application's namespace and name	`astroctl app get APPLICATION` `kubectl -n NAMESPACE get sa astro-model-APPLICATION`
`Current add-on configuration and health`	The add-on as applied	`astroctl infra k8s addons get CLUSTER model-serving` `astroctl infra k8s addons get CLUSTER model-serving --manifest`

Steps.

Create a managed identity. This creates an empty identity. It can do nothing until you grant it a role.

Terminal
az identity create --resource-group RESOURCE_GROUP --name IDENTITY_NAME
az identity show --resource-group RESOURCE_GROUP --name IDENTITY_NAME \
  --query "{clientId:clientId, principalId:principalId}"

Trust one ServiceAccount. This lets only system:serviceaccount:NAMESPACE:astro-model-APPLICATION on this cluster sign in as the identity.

Terminal
az identity federated-credential create --name model-APPLICATION \
  --identity-name IDENTITY_NAME --resource-group RESOURCE_GROUP \
  --issuer "$(az aks show --resource-group RESOURCE_GROUP --name CLUSTER --query oidcIssuerProfile.issuerUrl --output tsv)" \
  --subject system:serviceaccount:NAMESPACE:astro-model-APPLICATION \
  --audiences api://AzureADTokenExchange

Grant read on one container, on one resource only. This lets the identity act as Storage Blob Data Reader on that resource and nowhere else.

Terminal
az role assignment create --assignee-object-id PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal --role "Storage Blob Data Reader" \
  --scope "$(az storage account show --name STORAGE_ACCOUNT --query id --output tsv)/blobServices/default/containers/CONTAINER"

Write the add-on manifest. This is only a file on your machine. Repeat step 2 for each model application. A cluster that caches model weights cannot use an Azure identity for the cached download; use secretRef for that artifact.

model-serving.yaml
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: model-serving
spec:
  clusterName: CLUSTER
  configuration:
    modelServing:
      identity:
        serviceAccountAnnotations:
          azure.workload.identity/client-id: CLIENT_ID
        serviceAccountLabels:
          azure.workload.identity/use: "true"

Apply the add-on. This installs or updates the add-on with the identity and settings in the file.

Terminal
astroctl infra k8s addons apply -f model-serving.yaml

Validate. Three checks, in order. Stop at the first one that fails.

Check 1: the cluster is enabled (Enable workload identity on AKS).

Terminal
az aks show -g RESOURCE_GROUP -n CLUSTER --query oidcIssuerProfile.issuerUrl -o tsv
az aks show -g RESOURCE_GROUP -n CLUSTER --query securityProfile.workloadIdentity.enabled -o tsv

Expected: an issuer URL, then true. If not, stop and go to Enable workload identity on AKS.

Terminal
kubectl run wi-test -n NAMESPACE --rm -it --restart=Never \
  --labels=azure.workload.identity/use=true \
  --image=mcr.microsoft.com/azure-cli \
  --override-type=strategic --overrides='{"spec":{"serviceAccountName":"astro-model-APPLICATION"}}' \
  -- sh -c 'az login --federated-token "$(cat $AZURE_FEDERATED_TOKEN_FILE)" --service-principal -u $AZURE_CLIENT_ID -t $AZURE_TENANT_ID --allow-no-subscriptions'

Expected: the command succeeds and prints the signed-in service principal.

Restricted-namespace override (also add --env HOME=/tmp): {"spec":{"serviceAccountName":"astro-model-APPLICATION","securityContext":{"runAsNonRoot":true,"runAsUser":65534,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"wi-test","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]}}

AADSTS70021: No matching federated identity record found for presented assertion (or AADSTS700213, the same message)

Cause

The federated credential's subject is not exactly system:serviceaccount:NAMESPACE:astro-model-APPLICATION, or its issuer is not this cluster's.

AZURE_FEDERATED_TOKEN_FILE is empty

Cause

The pod lacks the label azure.workload.identity/use: "true", or workload identity is off (Check 1).

If this fails, the binding is wrong; fix it before looking at the workload.

Check 3: the workload does its job.

Terminal
astroctl infra k8s addons get CLUSTER model-serving
astroctl app status APPLICATION
kubectl -n NAMESPACE logs POD -c storage-initializer

Expected: the add-on shows the trust subject it expects, the model application leaves its download step and serves, and the storage-initializer log has no 403 or AccessDenied. A pod binds its identity when it is created; applying a changed identity redeploys the cluster's models, so wait for the restart.

TROUBLESHOOT.

AADSTS70021: No matching federated identity record found for presented assertion (or AADSTS700213)

Fix

There is no federated credential for this exact subject system:serviceaccount:NAMESPACE:astro-model-APPLICATION (step 2).

AuthorizationPermissionMismatch

Fix

The identity lacks Storage Blob Data Reader on that container (step 3).

No federated token in the pod

Fix

The azure.workload.identity/use: "true" label is missing, or workload identity is off. See Enable workload identity on AKS.

Undo. To remove this access:

Terminal
az role assignment delete --assignee PRINCIPAL_ID --role "Storage Blob Data Reader" --scope "$(az storage account show --name STORAGE_ACCOUNT --query id --output tsv)/blobServices/default/containers/CONTAINER"
az identity federated-credential delete --name model-APPLICATION --identity-name IDENTITY_NAME --resource-group RESOURCE_GROUP --yes
az identity delete --resource-group RESOURCE_GROUP --name IDENTITY_NAME

This removes the role assignment, the federated credential and the identity. Delete the identity last. Then remove identity from the manifest and apply the add-on again.

If it's refused

My app api in namespace shop on my-cluster gets AuthorizationPermissionMismatch from Blob storage. Which part is missing, and how do I fix it?

Ask Nova

Nova reads the app's logs, events and settings and the cluster, names the missing part (the identity, the trust or the grant), and writes the fix.

When your app's call to your cloud is refused:

AZURE_FEDERATED_TOKEN_FILE is empty, or the AZURE_* variables are missing

Cause

The pod lacks the label azure.workload.identity/use: "true", started before it was added, or workload identity is off.

Fix

Put the label on the app and restart its pods, or turn on workload identity.

AADSTS70021: No matching federated identity record found

Cause

The federated credential's subject is not exactly system:serviceaccount:NAMESPACE:SERVICE_ACCOUNT, or its issuer is not this cluster's.

Fix

Recreate the federated credential with the app's namespace and ServiceAccount.

AuthorizationPermissionMismatch

Cause

The pod signs in, but the identity has no role on that resource.

Fix

Assign the role on that exact resource.

When an add-on's identity is refused at apply:

"identity.serviceAccountAnnotations["azure.workload.identity/client-id"] must be a client ID (a UUID)"

Cause

The value on the add-on's identity is not the kind of identity AKS expects.

Fix

Paste the value the recipe printed, unchanged.

"identity.serviceAccountAnnotations must carry a cloud workload-identity annotation: eks.amazonaws.com/role-arn (IRSA), iam.gke.io/gcp-service-account (GKE Workload Identity), or azure.workload.identity/client-id (AKS Workload Identity)"

Cause

identity was set with no workload-identity annotation.

Fix

Add the azure.workload.identity/client-id annotation from the recipe.

Other ways in

Nova: ask, for example, "why can't DNS Automation on my-cluster change records?". Nova reads the add-on's health and the cluster's cloud identity.
SDK and API: an add-on's identity is part of its configuration. See the API reference.

With astroctl

Each recipe ends by applying the add-on with its identity:

Terminal
astroctl infra k8s addons apply -f ADDON.yaml
astroctl infra k8s addons get my-cluster ADDON

Keyless cloud access: how keyless works, and how your app gets its identity.
Registry access: pull your images, and push them from a build.
Deploy an application.

Before you start​

In the console​

You should see​

Recipes​

Turn on workload identity on a registered cluster​

Your own applications​

Reading a bucket​

Give my application access to an Azure Blob container​

Calling a queue, topic or vault​

Give my application access to a Key Vault secret​

Pulling images from a private registry​

Let AKS nodes pull from an Azure Container Registry​

Platform add-ons​

Vulnerability Scanning​

Let vulnerability scanning read private images​

DNS Automation (external-dns)​

Let DNS Automation change records in one Azure DNS zone​

Certificates (cert-manager), DNS-01 check​

Let Certificates add DNS challenge records in one Azure DNS zone​

Model Serving (model-serving)​

Let models read a private Blob container​

If it's refused​

Other ways in​

With astroctl​

Next​

Before you start

In the console

You should see

Recipes

Turn on workload identity on a registered cluster

Your own applications

Reading a bucket

Give my application access to an Azure Blob container

Calling a queue, topic or vault

Give my application access to a Key Vault secret

Pulling images from a private registry

Let AKS nodes pull from an Azure Container Registry

Platform add-ons

Vulnerability Scanning

Let vulnerability scanning read private images

DNS Automation (`external-dns`)

Let DNS Automation change records in one Azure DNS zone

Certificates (`cert-manager`), DNS-01 check

Let Certificates add DNS challenge records in one Azure DNS zone

Model Serving (`model-serving`)

Let models read a private Blob container

If it's refused

Other ways in

With astroctl

Next