Deploy an open model
Rather not read it all? Ask Nova this. It walks you through each step and does the ones it can when you approve.
Your team needs embeddings for search and RAG, and you'd rather not send every document to an outside API and pay for it by the token. So you run an open model, BAAI/bge-small-en-v1.5, on your own cluster, where only your own services can reach it, and call it from Python. Alice, the platform engineer, sets up the cluster once; after that Bob, a developer, deploys and calls the model and never needs cluster-admin.
Before you start
- A cluster,
eks-prodhere, with a profile,prod-inferencehere (the settings the apps on a cluster share). - For the Python call: an API key in
ASTROPULSE_API_KEY, and a shell inside the cluster.
1. Alice prepares the cluster
Path: Infrastructure > Kubernetes > eks-prod > Add-ons > Install Add-on
- Select Certificates and turn Install CRDs on. On Review, select Continue to deploy once the dry run passes, then Deploy cert-manager.
- Select Install Add-on again, pick Model Serving, then on Review select Continue to deploy and Deploy model-serving.
It shows the plan and installs them when you approve.
2. Bob picks the model
Path: Workloads > Models > Deploy model
- Name and cluster:
open-model-demo(its name in the console and in code),eks-prod. - Profile:
prod-inference. - Model: Hugging Face, type
bge smalland pickBAAI/bge-small-en-v1.5.
It searches the public Hub and reads the model card.
3. Bob checks the model before it runs
Path: Deploy model > Model
- Select Check model.
A model it cannot serve, or a missing Revision, is refused with the reason:
Set Revision to a model commit, so a rollback returns the same weights.
It runs the check without deploying and gives the platform's reason if the model is refused.
4. Bob sizes it and keeps it internal
Path: Deploy model > Compute, then Access
- Compute > Resources and tuning: Memory
3Gi, enough for this small model. - Access: leave External access on Internal only (default), so only workloads in
eks-prodcan call it.
Verify a domain and put an access policy on it: Run production AI inference.
It runs the same dry run as Review and reports any warning, without deploying.
5. Bob deploys it
Path: Deploy model > Review
- Select Review, then Deploy.
The dry run warns Memory is below the default. That is fine here: the default is sized for larger models.
Every new node downloads the model again, and on a private cluster those bytes go through the NAT gateway, which the cloud bills per GB. On an EKS cluster AstroPulse creates, a model at an s3:// address or an oci:// image in your ECR, in the cluster's region, skips that charge: see Cut model download cost.
It shows the plan and deploys when you approve.
6. Bob sees it running
Path: Workloads > Models > open-model-demo
You should see: Running, the In-cluster endpoint and Memory budget 3Gi.
It reads the app's status and health.
7. Bob calls the model
Path: Workloads > Models > open-model-demo > Call it from code
- Install the SDKs:
pip install "git+https://github.com/astropulseinc/astropulse-python" openai. - Copy the snippet, with the model name filled in, and run it inside the cluster.
from astropulse import Client
from openai import OpenAI
ap = Client() # reads ASTROPULSE_API_KEY
application = ap.applications.get_application("open-model-demo")
client = OpenAI(base_url=application.data.model.internal_endpoint,
api_key="unused") # nothing checks a key in the cluster; the client needs one
response = client.embeddings.create(model="<model-name>", input=["Hello"])
print(len(response.data[0].embedding))
It prints the embedding's length.
It answers from the AstroPulse docs.
Clean up
On the open-model-demo page select Delete, type the name, then Delete Permanently. The add-ons stay.
It explains what goes and gives you the command to run. Nova never deletes an app itself.