Self-hosted Kubernetes on AWS
AstroPulse builds Kubernetes on EC2 instances in your AWS account: the control plane and the workers both run on your machines, and you size both. AstroPulse works in the account with an IAM user's access key that a CloudFormation stack creates for this one cluster and that AstroPulse stores encrypted. Choose it over Amazon EKS when you need the control plane itself in your account; EKS needs no stored key and takes GPU node groups, which self-hosted clusters do not. What every self-hosted cluster shares is on Self-hosted Kubernetes.
When you need it
- The control plane and etcd must run on EC2 instances in your own account.
- You want to pick the control plane's instance type and count.
- You need rollbacks that take the control plane back too.
Before you start
- Your 12-digit AWS account ID, and permission to create a CloudFormation stack with IAM resources (
CAPABILITY_NAMED_IAM) in it. - An organization owner or admin in AstroPulse.
astroctlv0.0.179 or later and the AWS CLI on your computer, the AWS CLI signed in to that account.connectreads the stack's keys with it.- Room in the account's EC2 quotas for the control plane, one group in each of the region's first three zones, and the workers.
- The Elastic Load Balancing service-linked role in the account. AWS creates it the first time you make a load balancer; otherwise
aws iam create-service-linked-role --aws-service-name elasticloadbalancing.amazonaws.com.
In the console
On New cluster, choose Bring your own account.
-
Name and cloud. Enter the Cluster Name, pick Amazon Web Services under Cloud, and pick the Region.
Open zoom view -
Account. Under How Kubernetes runs, pick Self-hosted Kubernetes. Enter the 12-digit Account ID (
aws sts get-caller-identityprints it).Open zoom view -
Connect. Select Start the setup. What the setup creates in your account names the stack, IAM user and group, bucket and queue for this cluster (what they are), and the steps follow: Plan the resources, Prepare the setup template, Create the resources in your account, Check the credentials reach your account and Store the credentials in AstroPulse.
Open zoom view -
Select Create the stack in AWS: the CloudFormation console opens with the stack ready to create. Or run the
aws cloudformation create-stackcommand shown under it with a profile for that account. -
When the stack is created, run the command under Hand the credentials to the vault in a terminal where
astroctlis signed in to your organization and the AWS CLI to the account. It reads the stack's keys and sends them; AstroPulse checks them before it stores them. The step reads "The vault holds working credentials for <cluster>." and moves on by itself. -
Kubernetes. Pick the Kubernetes Version, or keep AstroPulse default.
-
Network. Under VPC Configuration, leave Use Existing VPC off: "A new VPC will be created for your cluster with default settings." Or turn it on and enter the VPC ID (Your own VPC).
-
Node groups. On the Control plane card, select Choose machine under Machine type and set Minimum nodes and Maximum nodes. Then Add Node Group for the workers: one machine type each from Choose machines, Minimum nodes, Maximum nodes, Pricing (On-Demand or Spot) and Name.
-
Notifications. Optional.
-
Review. The dry run checks the manifest, then Create.
You should see
The cluster's page reads Ready, Node groups lists the control-plane groups and your worker groups, and Events shows each stage of the create without a failure. Integrations › Vault lists the cluster's AWS credentials at cluster scope.
What the setup creates
One CloudFormation stack per cluster, named for the cluster, creates:
- An IAM user with one access key. This is the key AstroPulse stores.
- An IAM group the user belongs to, with eight managed policies: S3, EC2 (with Elastic Load Balancing, CloudWatch and Auto Scaling), EventBridge, Route 53, SQS, VPC (with ECR image pulls), IAM, and read-only Service Quotas.
- A private, versioned S3 bucket that holds the cluster's state.
- An SQS queue.
The stack's outputs are the bucket name and the access key and secret, which is how connect --from-stack reads them.
Least privilege
The policies grant their services on every resource in the account, IAM included, because the cluster creates its own instance roles, security groups, load balancers and DNS records. To keep that scope to one cluster:
- Use an AWS account for self-hosted clusters only.
- Limit who can read the stack (
cloudformation:DescribeStacks): its outputs hold the secret key. - Limit who can read the state bucket.
- To use an IAM user you made yourself, give it the same policies and connect with
--without-setup.astroctl cloud aws selfHosted setup ... --output-template ./selfhosted-cf.yamlwrites the template to read them from and creates nothing.
The access key
AstroPulse checks the key reaches the account you named, that the IAM user is the one the setup made for this cluster in your organization, that the state bucket is reachable and that the Elastic Load Balancing service-linked role exists. It stores the key only when every check passes, encrypted, and never shows it back. Only long-lived IAM user keys are accepted; temporary credentials with a session token are refused. connect reads the key from --from-stack, --profile, or --access-key and --secret-key, never from environment variables or the default AWS credential chain.
The access key does not expire. To rotate it, make a new access key for the IAM user and send it in place of the stored one, either with Replace on Integrations › Vault or:
astroctl cloud aws selfHosted connect --account-id 123456789012 --region us-west-2 \
--from-stack <setup-stack> --cluster-name prod --replace
AstroPulse checks the new key before it replaces the stored one. The old key stays active in AWS until you delete it there.
If a key leaks
Revoking the key in AstroPulse, or selfHosted disconnect, deletes only AstroPulse's copy. The access key stays active in your AWS account until you delete it in IAM:
aws iam list-access-keys --user-name <iam-user>
aws iam delete-access-key --user-name <iam-user> --access-key-id <access-key-id>
Then send a new key with connect --replace, or run cleanup to remove the stack. Until a new key is stored, the cluster keeps running but AstroPulse cannot change, upgrade, scale or delete it. If the cluster's machines are already gone and you ran cleanup, astroctl infra k8s delete prod --record-only removes AstroPulse's record and lists anything that may remain in the account.
Networking
The cluster runs in a public and a private subnet in each of its zones. The control plane and the workers run in the private subnets; load balancers go in the public ones. The Kubernetes API is reached through a public Network Load Balancer. Networking has the details.
Your own VPC
Turn on Use Existing VPC and enter the VPC ID, or set networkId in the manifest. The VPC must be in the cluster's account and region, and must not use 100.64.0.0/10, which the cluster keeps for pods and services. The dry run checks the ID's format, not the VPC itself.
Kubernetes versions
A self-hosted cluster takes an exact major.minor.patch version. AstroPulse default sends none, and the platform picks its default. To list what is supported and where each version can upgrade to:
astroctl cloud aws k8s-versions --self-hosted
astroctl cloud aws k8s-versions --self-hosted --show-upgrade-paths
Updates and upgrades
The cluster's Update tab upgrades Kubernetes and changes node groups, and the upgrade wizard can resize the Control Plane. The control plane upgrades first, then the workers by rolling replacement, and a rollback moves the control plane back too: Updates and upgrades.
astroctl infra k8s upgrade prod <next-version> --dry-run
astroctl infra k8s rollback prod --list-versions
astroctl infra k8s update prod --example
Other ways in
- Nova: ask it, for example "plan a self-hosted cluster called prod in my AWS account in us-west-2". Nova shows the plan; the key still goes from your terminal with
astroctl. - SDK and API: the same operations, in the API reference.
With astroctl
astroctl cloud aws selfHosted setup --account-id 123456789012 --region us-west-2 --cluster-name prod
astroctl cloud aws selfHosted connect --account-id 123456789012 --region us-west-2 \
--from-stack <setup-stack> --cluster-name prod
astroctl cloud aws selfHosted status --cluster-name prod
astroctl infra k8s apply -f prod.yaml --dry-run
astroctl infra k8s apply -f prod.yaml
setup lists what it will create and asks how to create the stack: with your AWS CLI, which also sends the keys when the stack is ready, or in the AWS console, after which you run connect. status shows each recorded setup step. 123456789012 stands for your account ID. The manifest:
apiVersion: platform.astropulse.io/v1
kind: K8sCluster
spec:
clusterName: prod
provider: aws
region: us-west-2
clusterSpec:
controlPlane:
nodeGroup:
name: control-plane
machineTypes: [t3.medium]
minNode: 1
maxNode: 1
dataPlane:
nodeGroups:
- name: workers
minNode: 2
maxNode: 5
machineTypes: [t3.medium]
instanceType: OnDemand
provisioner:
type: selfHosted
selfHosted:
accountId: "123456789012"
credentials:
type: Vault
To remove what the setup created, delete the cluster first, then:
astroctl cloud aws selfHosted cleanup --account-id 123456789012 --region us-west-2 --cluster-name prod --profile <aws-profile>
It empties and deletes the bucket, deletes the stack and the key stored in AstroPulse. The Connect step shows the same commands under Remove what the setup created. Every flag: astroctl cloud aws selfHosted.
Settings
The fields under spec.provisioner.selfHosted on AWS. The control plane and node groups: Self-hosted Kubernetes.
accountIdRequiredThe AWS account the cluster is created in.
"123456789012".credentialsRequiredtype: Vault, the key stored with selfHosted connect.kubernetesVersionmajor.minor.patch from the supported list.astroctl cloud aws k8s-versions --self-hostednetworkIdAn existing VPC to create the cluster in. Left out, a new VPC is created.
vpc- followed by its hexadecimal ID.bucketNameThe S3 bucket that holds the cluster's state, in the cluster's account and region.
If it's refused
aws sts get-caller-identity.networkId is not a VPC ID.vpc-.--from-stack or --profile instead of pasting.astroctl is signed in to another organization than the one that started the setup, or no setup was started.--without-setup for a key you made yourself.connect again in a minute.aws iam create-service-linked-role --aws-service-name elasticloadbalancing.amazonaws.com, then connect again.