Skip to main content

Run production AI inference

NovaNova
Set up eks-prod to serve qwen-chat to your customers, the way this example does.
Ask Nova

Rather not read it all? Ask Nova this. It walks you through each step and does the ones it can when you approve.

Every AI company starts with a choice: buy tokens from a frontier lab, or run models in your own cloud. You chose to run your own, for weights you own, data that stays in your account and cost you control at volume, and now you are building an inference company: developers pay to call your models. That choice brings seven layers of machinery the lab's API used to hide, from GPUs and serving to access, scaling, cost and releases. AstroPulse gives you those layers as a foundation in your cloud, with the knobs to tune them. In this example you end up with qwen-chat on your domain behind a JWT policy, scaling within the limits you set, with its cost visible and every release reversible.

What you're getting into​

AstroPulse runs the machinery and gives you the knobs. The models and the product are yours. Operating it is the hard part, and Nova, your AI platform engineer, runs it with you: it finds what broke and fixes it when you approve.

Open zoom view

An inference company in four layers, top to bottom: your inference product, your models, AstroPulse and your cloud. Nova runs on a rail outside them, working across every layer.

your AI platform engineerYOUR INFERENCE PRODUCTThe API your customers pay for: keys, limits, billing. That stays yours.YOUR MODELSqwen-chat and every model you serve, from the console, Nova, astroctl or the SDK.Clusters and identityGPU capacityModel servingDomain, TLS and DNSAccess policiesReplica scalingGPU metrics, cost and releasesYOUR CLOUDAWS here, in your account. Google Cloud and Azure work too.

In words: your cloud account is the ground. AstroPulse runs on it as the machinery: clusters and identity, GPU capacity, model serving, your domain with TLS and DNS, access policies, replica scaling, and GPU metrics, cost and releases. Your models run on AstroPulse, served from the console, Nova, astroctl or the SDK, and your inference product sits on top: the API your customers pay for, with its keys, limits and billing. That product stays yours. Nova, your AI platform engineer, works across every layer with your approval.

When you're done, qwen-chat looks like this:

One model, running in your cloud and ready for authenticated callers.Open zoom view

One model, running in your cloud and ready for authenticated callers.

The qwen-chat model overview showing Running, its public endpoint on apps.example.com, Ready with what callers must send, serving facts and code to call itThe qwen-chat model overview showing Running, its public endpoint on apps.example.com, Ready with what callers must send, serving facts and code to call it
Models: deploy, then run

Who does what​

  • Alice, the platform engineer, builds and runs the platform and the models on it. Anyone with the Editor role can deploy a model too, with no AWS credentials.
  • Bob, a developer at your customer, calls qwen-chat with his key. A request without one is refused.

How the platform fits together​

AstroPulse runs layers 1 to 7 in your cloud and you set their knobs; layer 8 is the product you sell.

Serves the layer above ↑
8Your productThe customer experience and business
7OperationsGPU metrics, cost, alerts and releases6ElasticityReplicas that follow demand5AccessWho may call the model4NetworkYour domain, TLS and DNS3ServingModel weights, cache and runtime2CapacityGPU supply and node scaling1FoundationAWS, EKS and cloud identitiesRelies on the layer below ↓
  1. 8
    8. Your product

    AstroPulsedoes not provide this layer. It supplies the platform beneath it.

    Youcustomer limits, usage metering, billing, routing and the product experience.

  2. 7
    7. Operations

    AstroPulseconnects model health, GPU use, spend, findings and releases; alerts you route to Nova start an investigation.

    Youthe alert rules, release decisions and service levels.

    Watch metrics and costStart an investigationRelease and roll back

  3. 6
    6. Elasticity

    AstroPulseadds model replicas when requests wait; Node Autoscaling supplies their GPU nodes.

    Youthe replica floor, ceiling and scaling target.

  4. 5
    5. Access

    AstroPulseenforces a JWT or API-key policy at the gateway before traffic reaches the model.

    Youthe identity provider or test keys.

  5. 4
    4. Network

    AstroPulseverifies your domain and manages the certificate, gateway route and DNS record.

    Youthe domain and its DNS zone.

  6. 3
    3. Serving

    AstroPulsechecks the model, caches its weights and serves it through an OpenAI-compatible API.

    Youthe model, source, serving settings and optional gated-model token.

  7. 2
    2. Capacity

    AstroPulseshows prices and zones, checks capacity and launches GPU nodes for pending replicas.

    Youthe GPU type, quota, purchase model and maximum spend.

    Check GPU capacityScale GPU nodes

  8. 1
    1. Foundation

    AstroPulseprovisions EKS and gives each cloud-facing add-on its own workload identity.

    Youthe AWS account, region and least-privilege policies.

    Create the clusterConnect cloud identities

Follow the build​

Each step lists what it needs, from GPU quota to a domain and an identity provider, and names the add-ons it installs; Certificates goes before Model Serving and Gateway. The design behind it, with the manifests: Serve models behind an API.

  1. Alice checks GPU prices and quota
  2. Alice creates the EKS cluster
  3. Alice connects add-ons to AWS without keys
  4. Alice adds GPU nodes only when needed
  5. Alice deploys the first model
  6. Alice puts the model on your domain
  7. Alice lets only customers like Bob call it
  8. Alice scales the model with demand
  9. Alice watches metrics and cost
  10. Alice gets alerted when something breaks
  11. Alice releases a new version, and rolls back
  • Workload Autoscaling: replicas follow requests waiting. Without it, a model on a GPU stays at a fixed replica count.
  • Workload Queueing, where models or teams share GPUs: each gets a GPU quota instead of competing for the same GPUs.
  • GPU limit: caps the GPUs billed at once. Unset, the pool may launch up to 10.
  • An access policy on every public endpoint: with No authentication, anyone who finds the URL runs inference on your GPUs.
  • Let's Encrypt production for real hostnames: clients don't trust Let's Encrypt staging.
  • On-Demand for serving: Spot is cheaper but can be reclaimed mid-request.
  • Fixed costs run before any model does: the managed control plane, three platform nodes, the NAT gateways and a load balancer.
  • A model keeps at least one replica, so it does not scale to zero. A request times out after 300 seconds, and there is no durable request queue.
  • AstroPulse provides no per-customer limits, usage metering, billing, routing across models or cross-region failover: those are your product.

Mechanics: Model Serving, Node Autoscaling, Access Policies, Alerts that start a Nova investigation. Nova on another app: Investigate a failing app.

Clean up​

GPU nodes and the cluster keep costing until you delete them. Remove them in this order, because a policy an application references cannot be deleted:

  1. Optional: close the endpoint (Let only your customers call it).
  2. Delete the model: on the qwen-chat page, Delete, type the name, Delete Permanently. This frees its policy and hostname.
  3. Delete the access policy under Integrations > Access Policies.
  4. Delete the domain if you claimed it only for this example (Put the model on your domain).
  5. Delete the cluster; only an org Owner can (Create the EKS cluster). This uninstalls every add-on first, so Node Autoscaling's GPU nodes go too, and deletes the cluster's cost history.

The AWS connect stack stays, because other clusters in the account reuse it. Remove it in the AWS CloudFormation console only when you are done with AstroPulse in this account.