Skip to main content

Gateway

Gateway (gateway) exposes model endpoints securely. It installs Gateway API with a GatewayClass named astro, and Envoy Gateway behind it. With it installed, an exposed Model application is served through the gateway over HTTPS instead of an Ingress, with nothing changed on the application. It is the only path that enforces a model's access policy: API keys or tokens are checked here, before a request reaches the model. It needs Certificates.

When to use it​

  • A model endpoint must answer only callers you allow. Without the gateway, a model with an access policy is refused with Authenticated endpoints need the gateway add-on. With it, every request is checked against the policy before it reaches the model.
  • Some models are public and some private, on one cluster. With externalAccess.exposure on each model, public endpoints (the internet) and private ones (only the cluster's network) are served side by side.
  • Many model endpoints share one entry point that has to keep up. Every model routes through one proxy, which can scale on CPU or on request rate.

You don't need it for image applications, for a model with no access policy, which Ingress can expose, or for a model reached only inside the cluster.

Before you start​

  • A cluster on Kubernetes 1.30 or newer.
  • Certificates with an issuer. For Let's Encrypt, Certificates needs its DNS check: Let's Encrypt can't check names over HTTP behind the gateway (How a certificate is checked). A self-signed issuer works with no extra step.
  • On a cluster with no cloud load balancer, such as a local or bare-metal one, plan to turn on Enable node port.

In the console​

After Version, its steps are Settings and Auto scaling, then Advanced, Resources and Components. Review runs the dry run, and Ask Nova at the top of the steps answers about the open step.

  1. Open the cluster from the cluster list, select the Add-ons tab, then Install Add-on, and pick Gateway. Requires shows whether Certificates is installed.

  2. Under Auto scaling, turn on Enabled for production. With it off, the proxy runs 2 copies on a cluster AstroPulse created with 2 or more fixed nodes, and 1 otherwise. Then pick the Metric: CPU, or Requests per second (needs Workload Autoscaling), and set Min replicas, Max replicas and Target CPU percent or Target requests per second.

    Open zoom view

    Install Gateway with Requires Certificates, the version, Scale the gateway proxy, availability and the dry run

    Install Gateway with Requires Certificates, the version, Scale the gateway proxy, availability and the dry runInstall Gateway with Requires Certificates, the version, Scale the gateway proxy, availability and the dry run

  3. Optional: Exposure, and Enable node port for a cluster with no cloud load balancer.

  4. On Review, the dry run runs by itself; select Continue to deploy once it passes, then Deploy gateway.

You should see​

The add-on reads Deployed, and its Overview shows the address to point clients at under Access, or the node ports and node addresses with Enable node port on. astroctl infra k8s addons get <cluster> gateway reports the same, with endpointAuthentication, the methods this version enforces (jwt, apiKeys).

Open zoom view

Gateway on its Overview: Deployed, 1/1 check passing, and Access with NodePort, the http and https ports, the node address and Ready for traffic

Gateway on its Overview: Deployed, 1/1 check passing, and Access with NodePort, the http and https ports, the node address and Ready for trafficGateway on its Overview: Deployed, 1/1 check passing, and Access with NodePort, the http and https ports, the node address and Ready for traffic

How it serves models​

  • Exposure. A model's externalAccess.exposure picks Public (the internet) or Private (only the cluster's network). A model that names none gets the add-on's exposure, Public by default.
  • DNS. Turn on Gateway routes (gatewayRoutes) on DNS Automation to publish each route's hostname.
  • Request duration. Each model route times out a request after 300 seconds, while capacity scales too. The gateway is not a request queue: size the model's maximum replicas for its longest request.
  • Scaling is separate. Scaling the proxy adds gateway replicas to accept more calls; it adds no model replicas or GPU nodes. How they fit together: Workload Autoscaling.

Security: with the default exposure, the gateway is on an internet-facing cloud load balancer, billed by your cloud. Set exposure: Private when no model on the cluster should be reachable from the internet; only the gateway checks access policies, so a model that needs one is never served through Ingress.

Run it in production​

Every exposed model's traffic passes through its proxy, so treat it as critical.

  • Scale the proxy. Under Auto scaling, turn on Enabled with Min replicas of 2 or more. With it off, the proxy runs 2 copies on a cluster AstroPulse created with 2 or more fixed nodes, and 1 otherwise.
  • Load balancers are billed by your cloud. The public one exists while the add-on is installed. The first model served privately adds a second, internal load balancer.
  • It grows with traffic. The proxy reserves 100m CPU and 256Mi memory, with no memory limit. Raise Resources as traffic grows: Availability and resources.
  • Scaling on requests per second needs Workload Autoscaling. While the gateway scales that way, uninstalling Workload Autoscaling is refused. Switch the gateway to CPU first.

Other ways in​

  • Nova: ask it, for example "install the Gateway on eks-prod and scale the proxy between 2 and 10 replicas on CPU".
  • SDK and API: the same operations, in the API reference.

With astroctl​

gateway.yaml
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: gateway
spec:
  clusterName: eks-prod
  configuration:
    gateway:
      autoScaling:
        enabled: true
        metric: CPU
        minReplicas: 2
        maxReplicas: 10
Terminal
astroctl infra k8s addons apply -f gateway.yaml --dry-run
astroctl infra k8s addons apply -f gateway.yaml

astroctl infra k8s addons template <cluster> gateway writes every setting with what it does.

Settings​

Its own settings, under gateway, plus availability and resources. Every one is optional.

Enabledgateway.autoScaling.enabled

Scales the proxy every exposed model's traffic goes through.

Default
Off: 2 proxy copies on a cluster AstroPulse created with 2 or more fixed nodes, 1 otherwise.
Metricgateway.autoScaling.metric

The one signal replicas follow. RequestsPerSecond needs Workload Autoscaling with a Prometheus that scrapes the proxy; without it the apply is refused, never moved to CPU.

Accepts
  • CPU CPU Replicas follow the gateway's CPU use.
  • RequestsPerSecond Requests per second Replicas follow the requests the gateway serves each second; needs the Workload Autoscaling add-on with a metrics source.
Default

CPU

Min replicasgateway.autoScaling.minReplicas
Accepts
1 or more.
Default
2.
Max replicasgateway.autoScaling.maxReplicas
Accepts
At least Min replicas.
Default
10.
Target CPU percentgateway.autoScaling.targetCPUPercent

The CPU use to hold, with CPU.

Default
70.
Target requests per secondgateway.autoScaling.targetRequestsPerSecond

The request rate per replica to hold, with RequestsPerSecond.

Default
500.
Enable node portgateway.enableNodePort

Publishes the gateway on a NodePort on every node instead of a cloud load balancer, for clusters that have none. The HTTP listener is on node port 30089.

Default
Off.
Exposuregateway.exposure

The exposure of the shared gateway, and the default for model endpoints that name none. Private needs a cloud load balancer.

Accepts
  • Public Public Reachable from the internet.
  • Private Private Reachable only from inside the cluster's network.
Default

Public

If it's refused​

"gateway autoScaling.metric requestsPerSecond needs the workload-autoscaling add-on installed with a metricsSource.prometheus.endpoint on this cluster, so the proxy's request rate can be read from somewhere; it is not, so this apply is refused rather than scaling on CPU instead. Install and configure that add-on first, or use metric cpu"
Cause
Requests per second with no Workload Autoscaling, or one with no Prometheus endpoint.
Fix
Install Workload Autoscaling with its Prometheus endpoint, or scale on CPU.
"autoScaling.maxReplicas (<max>) must be at least minReplicas (<min>)"
Cause
Max replicas is below Min replicas.
Fix
Raise Max replicas or lower Min replicas.
"autoScaling.minReplicas must be at least 1: the proxy is the only path to every exposed application"
Cause
Min replicas is 0.
Fix
Set 1 or more.
"exposure private needs a cloud load balancer, and enableNodePort publishes on a node port instead; turn enableNodePort off or use exposure public"
Cause
exposure: Private with Enable node port on.
Fix
Turn Enable node port off, or use Public.
"gateway needs cert-manager to issue the TLS certificate of every route it serves, and cert-manager is not installed on cluster <cluster>. Install it first."
Cause
Certificates is not installed.
Fix
Install Certificates, then the Gateway.

These come on a model application's dry run and apply:

Authenticated endpoints need the gateway add-on, "externalAccess.accessPolicyRef names a policy, and only the gateway add-on enforces one; its exposure path is not ready (<reason>)."
Cause
The model names an access policy, and the gateway is not installed or not ready.
Fix
"Install the gateway add-on on this cluster, then apply the application again.", or finish setting it up as the reason says.
No usable way to expose this model on the cluster, "This model cannot be exposed yet: the gateway add-on's exposure path is not ready (<reason>), and the Ingress path is not available either (<reason>)."
Cause
The gateway is installed but not ready, and Ingress can't serve the model either.
Fix
Finish setting up the gateway as the reason says, usually a Certificates issuer.

Next​