Serve models behind an API

Serve open models, such as one from Hugging Face, on GPUs in your own AWS account, as an OpenAI-compatible API on your own domain. Only callers holding a key you issued reach it, capacity grows with demand, and each model's GPU use and cost is visible.

This is the serving use case of the AI infrastructure blueprint: which AstroPulse features to combine, in what order, and why. Each feature's own page holds its steps and settings.

AI inference end to end: build what matters

What it is made of

Building block	Its job in this design	Set it up
EKS cluster with two On-Demand node groups	Runs the add-ons; GPUs come from the GPU pool	Create a cluster
Certificates	Issues the TLS certificate for each model's hostname	Certificates
Gateway	Terminates TLS, checks the caller's key, routes to a ready replica	Gateway
DNS Automation, with `gatewayRoutes` on	Publishes each model's DNS record	DNS Automation
Model Serving	Runs the model's replicas	Model Serving
Cost Visibility	Prometheus, and spend per model	Cost Visibility
Workload Autoscaling	Adds replicas when requests wait	Workload Autoscaling
Accelerator Metrics	GPU utilization, memory, throttling and hardware faults per model, so an idle or oversized GPU shows	Accelerator Metrics
Vulnerability Scanning	Known CVEs in each model's images, on its Security tab	Vulnerability Scanning
Node Autoscaling, with a GPU pool	Adds a GPU node when a replica is pending, removes it when empty	Node Autoscaling
Workload Queueing (several teams)	A GPU quota per team	Workload Queueing
A verified domain	The hostnames models are served on	Domains
An access policy with API keys	Who may call each model	Access policies
The model	What callers reach	Serve a model

How a request reaches the model

The caller resolves the model's hostname, such as qwen-chat.apps.example.com. DNS Automation published that record.
The Gateway presents the hostname's certificate, issued by Certificates.
The Gateway checks the caller's key against the model's access policy. A missing or wrong key gets HTTP 401. Only the Gateway enforces an access policy.
The Gateway sends the request to a ready replica, which Model Serving runs.

Build it

In the console:

Connect your AWS account under Infrastructure > Cloud accounts, and check you have On-Demand GPU quota in the region: Reserve GPU capacity.
Create the cluster with two On-Demand node groups: Infrastructure > Kubernetes > New cluster. See Create a cluster.
On the cluster's Add-ons tab, install in this order: Certificates, Gateway, DNS Automation (with gatewayRoutes on), Model Serving, Cost Visibility, Workload Autoscaling, Accelerator Metrics, Vulnerability Scanning, Node Autoscaling with a GPU pool, then Workload Queueing if teams share GPUs. Model Serving and the Gateway need Certificates; Workload Autoscaling reads the Prometheus that Cost Visibility installs; Accelerator Metrics needs no Prometheus and reports once a GPU node runs; Workload Queueing needs the GPU pool. See Install add-ons.
Verify your domain under Integrations > Domains. See Domains.
Create an access policy with managed API keys under Integrations > Access Policies, and one key per caller. See API keys.
Deploy the model from Workloads > Models > Deploy model: on Compute pick On-Demand, on Scaling set a minimum of 2 replicas, on Access pick your domain and the access policy. See Serve a model.
Call it with a key, as Authorization: Bearer <key>: Call a model with an API key.

The example Run production AI inference walks these steps for one company, with screenshots. To apply the same setup as files, see With astroctl.

Production choices

GPU capacity. Use the Node Autoscaling GPU pool when traffic varies: no GPU node while idle, one per pending replica up to gpuLimit. A replica on a new node takes about 8 to 9 minutes to serve. Use a fixed GPU node group when a replica must start in seconds: GPU node groups or the GPU pool. Node Autoscaling runs on EKS only.
On-Demand or Spot. Serve on On-Demand (capacityType: OnDemand, and gpuCapacityTypes: [OnDemand] on the GPU pool). Spot GPU capacity can be unavailable for long periods and a Spot node can be reclaimed; use it only for work that can restart: Sizing a replica.
Replicas. Run minReplicas of 2 or more. With one replica, a change stops it before the new one is ready. Replicas spread across nodes, and across zones on a multi-zone availability tier, so a second replica may add a GPU node.
GPU quota per team. With Workload Queueing, each queue gets a GPU quota and replicas wait for quota instead of taking another team's GPUs. Quota lent within a cohort is not taken back from running work, so leave cohort off the serving queue unless you accept that: Queueing a model.
Failure alerts. Add an email or Slack channel under Organization settings › Notifications: Notifications. A model that stops serving after it reached Running is not a notification topic; send your Prometheus alerts to Nova: Nova alerts.

How it scales

Requests wait in the runtime when callers send more than the replicas batch at once.
Workload Autoscaling sees more waiting requests per replica than the target, and adds a replica, up to maxReplicas.
With no free GPU, the replica is pending, and Node Autoscaling launches a GPU node, within gpuLimit.
The replica downloads the weights, starts the runtime and loads the model, then the Gateway sends it traffic.
When load drops, the extra replica is removed, and the empty GPU node after the GPU pool's consolidateAfter.

Scale-out stops at the first of: maxReplicas; gpuLimit divided by countPerReplica; the queue's GPU quota; your cloud account's GPU quota; available cloud capacity. Use requestsWaiting as the signal for a generative model on a GPU, whose CPU barely moves: Scaling. A model keeps at least one replica.

Size minReplicas for normal traffic with nothing waiting, plus headroom for growth faster than a new GPU node can serve. Size maxReplicas for the peak you will pay for. As a reference, one replica of Qwen/Qwen2.5-0.5B-Instruct on an L4 GPU served 16 concurrent streams at a p95 latency of 1.45 seconds. Read your own numbers on the model's Metrics tab: Model inference metrics.

Run it

Job	Where	How
Release a new model revision	Deploy it under a new name on the same access policy, move callers, delete the old one	Upgrade a model
Undo a change to scaling or resources	The model's History tab	Releases and rollback
Watch latency, GPU use and cost	The model's Metrics and Cost tabs	Cost and metrics
Rotate or revoke a caller's key	Integrations › Access Policies › Keys	API keys
Upgrade Kubernetes and add-ons	The cluster's upgrade offer; each add-on's Upgrade	Upgrade a cluster, Upgrade add-ons
Fix a model that is not serving	The model's Overview, Events and Logs	Debug a model
Review security findings	The model's Security tab	Vulnerability findings
See who changed what	Organization settings › Audit Logs	Audit logs
Lower idle GPU cost	The cluster's Cost tab, Idle capacity	Spend outside any application

With astroctl

The manifests for this design, applied in file order

The AI inference blueprint as manifests, for AWS us-west-2, with the names the example Run production AI inference uses (eks-prod, qwen-chat, inference-callers). The example serves on its own domain, apps.example.com, and adds its second GPU node through the GPU pool as these manifests do.

Before applying them, have:

your AWS account connected with astroctl cloud aws connect: EKS: connect the AWS account;
On-Demand GPU quota in us-west-2 for the GPUs in gpuLimit, 4 vCPU per g6.xlarge: Run production AI inference: reserve GPU capacity;
a public Route 53 hosted zone for your domain: Run production AI inference: give your domain a Route 53 hosted zone;
your domain claimed and verified with astroctl domain add and astroctl domain verify: Custom domains: claim and verify a domain.

Replace 123456789012 with your AWS account ID, apps.example.com with your verified domain, and platform@example.com with the address Let's Encrypt writes to.

AI inference cluster manifest

Two On-Demand node groups for the add-ons and no GPU node group: GPUs come from the GPU pool in 10-node-autoscaling.yaml. kubernetesVersion is left out; its default is in EKS: core fields.

00-cluster.yaml
apiVersion: platform.astropulse.io/v1
kind: K8sCluster
spec:
  clusterName: eks-prod
  provider: aws
  region: us-west-2
  clusterSpec:
    dataPlane:
      nodeGroups:
        - name: system
          minNode: 1
          maxNode: 1
          machineTypes: [m6i.large]
          instanceType: OnDemand
        - name: platform
          minNode: 2
          maxNode: 2
          machineTypes: [m6i.large]
          instanceType: OnDemand
  provisioner:
    type: eks
    eks:
      accountId: "123456789012"
      credentials:
        type: Dynamic

01-profile.yaml
apiVersion: platform.astropulse.io/v1
kind: ApplicationProfile
spec:
  profileName: prod-inference
  clusterName: eks-prod

AI inference add-on manifests

Apply the add-ons in file order. Model Serving and the Gateway require Certificates; Workload Autoscaling reads the Prometheus Cost Visibility installs, so Cost Visibility comes first; Workload Queueing refuses a GPU quota until the GPU pool exists.

02-cert-manager.yaml
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: cert-manager
spec:
  clusterName: eks-prod
  configuration:
    certManager:
      enableClusterIssuer: true
      issuerType: LetsEncrypt
      email: platform@example.com

03-gateway.yaml
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: gateway
spec:
  clusterName: eks-prod
  configuration:
    gateway:
      autoScaling:
        enabled: true
        minReplicas: 2
        maxReplicas: 10
        metric: CPU

04-external-dns.yaml
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: external-dns
spec:
  clusterName: eks-prod
  configuration:
    externalDns:
      domainFilter: apps.example.com
      gatewayRoutes: true

05-model-serving.yaml
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: model-serving
spec:
  clusterName: eks-prod

06-cost-visibility.yaml
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: cost-visibility
spec:
  clusterName: eks-prod
  configuration:
    costVisibility: {}

07-workload-autoscaling.yaml
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: workload-autoscaling
spec:
  clusterName: eks-prod
  configuration:
    workloadAutoscaling:
      metricsSource:
        prometheus:
          # the Prometheus Cost Visibility installs, which the console suggests
          endpoint: http://cost-visibility-prometheus-server.astro-cost.svc.cluster.local:80

08-accelerator-metrics.yaml
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: accelerator-metrics
spec:
  clusterName: eks-prod

09-vulnerability-scanning.yaml
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: vulnerability-scanning
spec:
  clusterName: eks-prod

10-node-autoscaling.yaml
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: karpenter
spec:
  clusterName: eks-prod
  configuration:
    karpenter:
      enableGpuNodePool: true
      gpuLimit: 4
      allowedAccelerators: [l4]
      gpuCapacityTypes: [OnDemand]

11-workload-queueing.yaml
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: workload-queueing
spec:
  clusterName: eks-prod
  configuration:
    workloadQueueing:
      integrations: [batch/job, deployment]
      clusterQueues:
        - name: serving
          quota: { gpu: 3 }
          preemption: Never
        - name: research
          quota: { gpu: 1 }

AI inference access policy and model manifests

12-access-policy.yaml
apiVersion: platform.astropulse.io/v1
kind: AccessPolicy
metadata:
  name: inference-callers
spec:
  apiKeys:
    managed: true
    header: Authorization

qwen-chat serves a small open model pinned to one revision, one L4 per replica on On-Demand capacity, two to three replicas in the serving queue. With a range and no scaling.metric, the signal is chosen when the model is applied: applied after Workload Autoscaling can read its queue, it scales on waiting requests; applied before, it runs at a fixed count until it is applied again.

13-model.yaml
apiVersion: platform.astropulse.io/v1
kind: Application
metadata:
  name: qwen-chat
spec:
  profileName: prod-inference
  externalAccess:
    domain: apps.example.com
    accessPolicyRef:
      name: inference-callers
  source:
    type: Model
    model:
      uri: hf://Qwen/Qwen2.5-0.5B-Instruct:7ae557604adf67be50417f59c2c2f167def9a775
      queue:
        name: serving
      resources:
        accelerator:
          gpu:
            model: l4
            countPerReplica: 1
            capacityType: OnDemand
      scaling:
        minReplicas: 2
        maxReplicas: 3

Apply the AI inference manifests

astroctl infra k8s apply, astroctl infra k8s addons apply and astroctl app apply take --dry-run, which changes nothing. astroctl app profile apply and astroctl access-policy apply have no dry run.

Terminal
astroctl infra k8s apply -f 00-cluster.yaml --dry-run
astroctl infra k8s apply -f 00-cluster.yaml
astroctl app profile apply -f 01-profile.yaml

# 02 to 11 in order; wait for each to report running before the next that needs it
astroctl infra k8s addons apply -f 02-cert-manager.yaml --dry-run
astroctl infra k8s addons apply -f 02-cert-manager.yaml

astroctl access-policy apply -f 12-access-policy.yaml
astroctl access-policy key create inference-callers --label first-caller

astroctl app apply -f 13-model.yaml --dry-run
astroctl app apply -f 13-model.yaml

A key's value is shown once, when it is created. Callers send it as Authorization: Bearer <key>: Give callers API keys: call a model with an API key.

Smallest AI inference setup to try it

To try the AI inference blueprint at the lowest cost, set gpuLimit: 1 in 10-node-autoscaling.yaml, leave out 11-workload-queueing.yaml and the queue block, and set minReplicas: 1 and maxReplicas: 1 in 13-model.yaml. The model then runs on one L4 with no autoscaling, and every change replaces its only replica.

What it is made of​

How a request reaches the model​

Build it​

Production choices​

How it scales​

Run it​

With astroctl​

AI inference cluster manifest​

AI inference add-on manifests​

AI inference access policy and model manifests​

Apply the AI inference manifests​

Smallest AI inference setup to try it​

Next​