Serve a model
Pick a model and AstroPulse serves it on your cluster: it sizes it, starts it, and tells you when it is ready to call. A generative model answers on the API most AI clients already speak. Every field has a default, and Check model says whether the model will serve before anything runs.
Before you start
What a model needs depends on who calls it. Install add-ons from the cluster's Add-ons tab.
| To | The cluster needs | You also need |
|---|---|---|
| Serve it inside the cluster | Certificates, then Model Serving | Nothing else for a public Hugging Face model |
| Call it from outside, on your domain | Also the Gateway. DNS Automation to publish its DNS record, or you create the record | A verified domain and an access policy, created under Integrations > Access Policies |
| Serve a gated or private model | The same as above | Your organization's Hugging Face token (Hugging Face), or read access to your bucket or registry (Private artifacts) |
How these add-ons connect, and the settings for production: the AI inference blueprint.
In the console
-
Open Workloads > Models > Deploy model. Name the model and pick its cluster and profile.
-
On the Model step, type part of a model's name and pick it from the Hugging Face list, or paste a URI from your own storage.
Open zoom view -
Select Check model. AstroPulse reads the model and answers whether it can serve it ("The platform can serve this model."), or says why not.
Open zoom view -
On Compute, keep the size AstroPulse chose, or pick a GPU and set memory. See Sizing and scaling.
-
On Access, choose who can call it:
- Internal only: workloads in the cluster call it. Nothing else to set.
- On your domain: pick a verified domain and an access policy. The policy decides which callers the Gateway lets through, for example one API key per caller. No policy yet: create one under Integrations > Access Policies and return to this step. See Access policies and Expose a model.
Open zoom view -
Optional: on API, keep OpenAI-compatible as the API format, or pick another; it sets the endpoint's path (Expose a model). For vision, video and audio models, Media per request caps how many of each one request may carry.
Open zoom view -
On Review, the dry run checks the manifest and says which runtime serves the model and how it scales. Then select Deploy.
Open zoom view
You should see
The model's page shows Running, its endpoint, live request numbers and the serving facts. Call it from code gives you the Python to call it.
If the model does not start, its page says why and what to do next. Every refusal and its fix: Operate a model.
Every new node downloads the model again, and on a private cluster those bytes go through the NAT gateway, which the cloud bills per GB. On an EKS cluster AstroPulse creates, a model at an s3:// address or an oci:// image in your ECR, in the cluster's region, skips that charge: see Cut model download cost.
Add what you need
Each is optional, in any order.
| If you want | Read |
|---|---|
| A model that is not public on Hugging Face | Hugging Face |
| A model from your own bucket, registry or volume | Private artifacts |
| It to run on a GPU, or hold more load | Sizing and scaling |
| A public HTTPS endpoint, for chosen callers only | Expose a model |
| To watch latency, tokens and queue depth | Operate a model |
Other ways in
- Nova: ask it in plain language, for example "serve BAAI/bge-small-en-v1.5 on prod-01".
- SDK and API: the Python SDK and the API reference use the same manifest.
With astroctl
The shortest manifest that serves a public Hugging Face model. Every other setting below has a default:
apiVersion: platform.astropulse.io/v1
kind: Application
spec:
name: "bge-embed"
profileName: "prod-inference"
source:
type: "Model"
model:
uri: "hf://BAAI/bge-m3"
astroctl app apply -f model.yaml --dry-run # check it; deploys nothing
astroctl app apply -f model.yaml
astroctl app get bge-embed # what runs, defaults filled in
See the astroctl reference.
What it doesn't do
- Quotas, rate limits, metering and billing for your own customers are yours. The gateway checks who is calling; see Access policies.
- Each model has its own hostname. Routing several models behind one
/v1, or batching requests, belongs in your API layer. - Canary traffic, multi-node inference and custom serving runtimes: apply a raw
InferenceServicethrough the YAML source. - A release never rolls itself back. You choose a version in deploy history and apply it.
- Cost is reported per model, not per token or per customer. See Cost and Metrics.
Settings
name, profileName, namespace, and syncType behave exactly as they do
for every other source, see the manifest overview. Below is everything under source.model.### Artifact and runtime
uriRequiredThe model artifact. Always set. Pin it to fixed bytes so a rollback serves the same model. Can be corrected until the model has served, then fixed.
hf://<org>/<repo>[:<revision>], s3://, gs://, pvc://<claim>/<path>, oci://, or an Azure Blob URL https://<account>.blob.core.windows.net/<container>/<path>. No credentials or query strings, except ?versionId= on s3:// and ?generation= on gs://.formatThe model format, which picks the serving runtime. Set it for any artifact that is not on Hugging Face. Fixed once the model has served.
vLLM, SKLearn, XGBoost, LightGBM, TensorFlow, PyTorch, ONNX, TensorRT, Triton, MLflow, PMML, Paddle.vLLM for hf://; required otherwiseruntime.enginePins a serving runtime instead of letting format choose. Set it only to use a runtime of your own. Fixed once the model has served.
vllm, triton, torchserve, tensorflow, sklearn, xgboost, lightgbm, mlserver, or the name of a serving runtime on the clusterformatServing
serving.apiFormatThe API the endpoint speaks. Set V2 when callers use the open inference protocol.
OpenAI (served under /v1), V2 (/v2/models/<name>/infer)OpenAI for a generative model; a predictive model answers on /v1/models/<name>:predictserving.maxContextTokensThe context window the runtime allocates for. Set it to save memory, and keep it at or below the model's own limit: above it, the model will not start.
serving.kvCacheSizeThe memory the runtime reserves for its KV cache. Set it when a replica runs out of cache or memory. On CPU it comes out of resources.memory and must be smaller than it; on a GPU it is device memory per GPU.
"2Gi"resources.memory. GPU: the runtime sizes it from the deviceserving.maxConcurrentRequestsThe most requests one replica serves at once; further requests wait, and scaling on waiting requests counts them. Generative models only.
serving.multimodal.maxImagesPerPrompt, maxVideosPerPrompt, maxAudioClipsPerPromptHow many images, video clips or audio clips one request may carry. Set it for a vision, video or audio model on the vLLM runtime. Refused for a predictive model or a runtime AstroPulse did not install; a text-only model is not checked. Each limit costs memory on every replica; above 64 the dry run warns.
Resources
resources.cpuCPU for one replica. Set it for CPU inference, where cores set the throughput.
"2" or "500m"resources.memoryMemory for one replica: weights, runtime and KV cache. Raise it when a replica runs out of memory. A value below the default is a dry-run warning, not a refusal. It must also fit the node the replica runs on.
"8Gi"4Gi on CPU, 8Gi on a GPU. Predictive model: the runtime's own defaultresources.accelerator.gpu.vendorThe GPU vendor. Set amd only for a runtime that runs on AMD GPUs; the vLLM runtime is NVIDIA only.
nvidia, amdnvidiaresources.accelerator.gpu.modelThe GPU model, in the same words a GPU node group uses. Set it to target one kind of GPU.
l4, a10g, a100, h100, t4, v100 and the rest of the machine catalogresources.accelerator.gpu.countPerReplicaGPUs one replica needs. Setting it is what puts the model on a GPU. A replica runs on one node.
gpu; without accelerator the model runs on CPUresources.accelerator.gpu.capacityTypeHow the replica's GPU node is bought. Set OnDemand for a model that must not restart, Spot to pay less and accept that the node can be reclaimed.
OnDemandOn-Demand Never runs the replica on Spot capacity; reserved capacity counts as on-demand.SpotSpot Runs the replica on Spot capacity, which costs less but can be reclaimed at short notice.AnyAny Runs the replica on whatever GPU capacity the cluster offers.
Any
Scaling and queueing
scaling.minReplicasThe fewest replicas. Raise it to keep spare capacity warm.
1scaling.maxReplicasThe most replicas. Set it above minReplicas to autoscale; equal, the model runs at that fixed count.
minReplicasminReplicasscaling.metricThe signal replicas follow, with a target. Leave it out unless you need a different signal; it needs maxReplicas above minReplicas.
cpu (a whole percent, 1 to 100) or requestsWaiting (a positive number, generative models only, needs the workload-autoscaling add-on)requestsWaiting at 2 for a generative model on a cluster with the workload-autoscaling add-on; otherwise cpu at 80 on CPU, and no autoscaler on a GPUqueue.nameA ClusterQueue the cluster's workload-queueing add-on declares; replicas wait for its quota instead of pending. Set it when models share a quota. Fixed once the application exists.
Weight cache
cache.enabledKeeps the weights on the node's disk so a restart reads them instead of downloading. Needs the weight cache on the cluster's model-serving add-on. Not for pvc:// or oci://.
true, falsefalsecache.sizeThe artifact's size on disk, rounded up. A size larger than the node group's cache is refused.
"30Gi"cache.nodeGroupThe node group that holds the cached weights; every replica then runs there. Set it only when more than one cached node group could run the model.
There is no field here for how a private artifact is read: an application carries no credential at all. See Private artifacts.