Skip to main content

Workload Queueing

Workload Queueing (workload-queueing) queues jobs fairly within quotas. It runs Kueue: the platform team declares the queues and their quota here, and work is admitted against that quota instead of piling up Pending. A Model application names a queue in source.model.queue.name; any other workload queues the way it always does with Kueue, through a LocalQueue in its namespace and the queue-name label on the Job.

When to use it​

  • Teams share a fixed set of GPUs. Without it, whoever submits first takes every GPU and the rest sits Pending. With it, each team's queue has a GPU quota, and work beyond it waits its turn.
  • A training or batch job needs all its pods at once. Without it, half a job's pods can start, hold their GPUs and wait forever for the rest. With Gang scheduling, a job is admitted only with room for all its pods, and one whose pods are not all ready in time is evicted and requeued, which frees its GPUs.
  • A model should wait for capacity instead of failing. A model that names a queue has its replicas admitted against that queue's quota, and its Metrics tab shows how many wait and for how long.
  • One team's quota sits idle while another's work waits. Queues in one Cohort borrow each other's unused quota. Borrowed quota returns as the borrowing work finishes.
  • Growth should stop at a team's quota. Node Autoscaling adds machines only for admitted work, so work beyond the quota waits instead of growing the cluster.

You don't need it when one team runs the cluster and every workload can start as soon as it is deployed. How it works with the two autoscaling add-ons: How the three fit together.

Before you start​

  • A cluster on Kubernetes 1.30 or newer.
  • For GPU quota: an accelerator node group, or a GPU node pool from Node Autoscaling or EKS Auto Mode.
  • For each LocalQueue: its namespace, already created.

In the console​

After Version, its steps are Settings and Gang scheduling, then Advanced and Resources. Review runs the dry run, and Ask Nova at the top of the steps answers about the open step.

  1. Open the cluster from the cluster list, select the Add-ons tab, then Install Add-on, and pick Workload Queueing.

  2. Under Cluster queues, with none added, the platform's default queue applies, created automatically. Select Add Cluster queues for each queue you want, and set its Name, an optional Cohort, Preemption (Lower priority or Never), and its Quota: CPU, GPU and Memory.

  3. Optional: under Gang scheduling, turn on Enabled and set the Timeout. Under Integrations, pick the workload kinds the queues admit; add deployment to queue models. Under Local queues, select Add Local queues to bind a Namespace of yours to a Cluster queue.

  4. On Review, the dry run runs by itself; select Continue to deploy once it passes, then Deploy workload-queueing.

To change it later, open Workload Queueing on the Add-ons tab, open Update and select Change configuration. Go to Review, select Continue to apply once the dry run passes, type workload-queueing, then select Update workload-queueing.

You should see​

The add-on reads Deployed. A model that names a queue shows, on its Metrics tab under Scaling, its replicas admitted and waiting and how long admission took; astroctl app metrics <application> reports the same in its queue section.

Open zoom view

Workload Queueing on its Overview: Deployed, with 2/2 checks passing

Workload Queueing on its Overview: Deployed, with 2/2 checks passingWorkload Queueing on its Overview: Deployed, with 2/2 checks passing

How queues admit work​

  • The default queue. With no queues declared, one queue named default takes the cluster's whole GPU capacity, and only batch/job workloads are admitted.
  • GPU quota is checked against the cluster. One queue above what the accelerator node groups and GPU node pools can provision is refused; quota declared across queues above it is accepted with a warning, and queues in one cohort share the difference. On a cluster with no GPU capacity, any GPU quota is refused.
  • Unset and 0 differ. Unset gpu on the only queue means the whole capacity; 0 means none. A queue among several with gpu unset gets no GPU quota.
  • Preemption. LowerPriority (the default) evicts lower-priority work to make room in a full queue; Never makes new work wait.
  • Admission, not a request queue. Quota decides which workloads may run; it does not reserve or buy GPUs, and it does not buffer inference calls.
Replicas waiting in a workload queue

Queue a model​

deployment must be in Integrations, because a model's replicas run as a Deployment. Adding it puts Kueue's admission check in front of every Deployment in your workload namespaces, so treat the add-on as critical to the cluster: while Kueue is unavailable, Deployment updates there fail. AstroPulse's own workloads are never queued.

A model's queue is fixed once the application exists: delete and recreate the application to move it. A queue still named by an application can't be removed.

Run it in production​

  • Two copies. On a cluster AstroPulse created with 2 or more fixed nodes, the controller runs 2 copies with a disruption budget. It reserves 500m CPU and 512Mi memory.
  • While it is down, workloads in namespaces that have a queue can't be created or changed.
  • Uninstalling deletes every queue and every queued workload's record, and a Job it suspended stays suspended. Move work off the queues first.

Other ways in​

  • Nova: ask it, for example "install Workload Queueing on eks-prod with a serving queue of 4 GPUs and a research queue of 2, sharing a cohort".
  • SDK and API: the same operations, in the API reference.

With astroctl​

Two queues sharing CPU and memory through a cohort, and a LocalQueue for the namespace team-a. It runs on any cluster, with or without GPUs:

workload-queueing.yaml
apiVersion: platform.astropulse.io/v1
kind: K8sClusterAddon
metadata:
  name: workload-queueing
spec:
  clusterName: eks-prod
  configuration:
    workloadQueueing:
      integrations: [batch/job, deployment]
      gangScheduling:
        enabled: true
        timeout: 10m
      clusterQueues:
        - name: serving
          cohort: pool
          quota: { cpu: "8", memory: 16Gi }
          preemption: Never
        - name: research
          cohort: pool
          quota: { cpu: "4", memory: 8Gi }
      localQueues:
        - namespace: team-a
          clusterQueue: research

On a cluster with GPU capacity, add gpu to each quota, such as quota: { gpu: 4, cpu: "64", memory: 256Gi }.

Terminal
astroctl infra k8s addons apply -f workload-queueing.yaml --dry-run
astroctl infra k8s addons apply -f workload-queueing.yaml

To read a queue's Active condition directly: kubectl get localqueue -n <namespace> -o yaml.

Settings​

Its own settings, under workloadQueueing, plus availability and resources. Every one is optional.

NameworkloadQueueing.clusterQueues[].name

The name a model application sets in source.model.queue.name. default is reserved.

CohortworkloadQueueing.clusterQueues[].cohort

A name shared by queues that lend each other unused quota.

Default
Empty: the queue keeps its quota to itself.
PreemptionworkloadQueueing.clusterQueues[].preemption
Accepts
  • LowerPriority Lower priority Evicts a lower-priority workload to make room for the new work.
  • Never Never Leaves running work alone, and the new work waits.
Default

LowerPriority

Quota: CPU, GPU, MemoryworkloadQueueing.clusterQueues[].quota

How much the queue may hold at once. cpu and memory are Kubernetes quantities, such as 64 and 256Gi; gpu counts accelerators.

Default
CPU and memory: bounded only by the cluster. GPU: the whole capacity on the only queue, none on a queue among several.
Gang scheduling: Enabled, TimeoutworkloadQueueing.gangScheduling

All-or-nothing admission: every pod of an admitted workload must be ready within the timeout, or the workload is evicted and requeued.

Accepts
A timeout from 1m to 24h.
Default
Off; timeout 10m.
IntegrationsworkloadQueueing.integrations

The workload kinds the queues admit. Any kind but batch/job needs that kind's operator installed.

Accepts
batch/job, deployment, statefulset, pod, jobset.x-k8s.io/jobset, ray.io/rayjob, ray.io/raycluster, ray.io/rayservice, kubeflow.org/mpijob, trainer.kubeflow.org/trainjob, workload.codeflare.dev/appwrapper, leaderworkerset.x-k8s.io/leaderworkerset.
Default
batch/job alone. A list you write replaces it.
Local queues: Name, Cluster queue, NamespaceworkloadQueueing.localQueues[]

A LocalQueue in a namespace of yours, bound to a ClusterQueue. An application that names a queue gets its own LocalQueue without one here.

Default
Name: the ClusterQueue's. Cluster queue: default.

If it's refused​

"clusterQueues[&lt;queue&gt;].quota.gpu is <n>, but cluster <cluster> has no accelerator node group and no GPU node pool yet; a GPU quota here can't be admitted until it has one. ..."
Cause
GPU quota on a cluster with no GPU capacity.
Fix
Add an accelerator node group, or turn on a GPU node pool and apply again once it is ready, or drop the GPU quota.
"clusterQueues[&lt;queue&gt;].quota.gpu is <n>, more than the <m> accelerators cluster <cluster>'s node groups and GPU node pools can provision; a single queue above the ceiling can never be fully admitted"
Cause
One queue's GPU quota is above the cluster's capacity.
Fix
Lower it, or add GPU capacity.
"clusterQueues[<i>].name "default" is reserved for the AstroPulse default; omit clusterQueues to use it"
Cause
A queue named default.
Fix
Pick another name, or declare no queues.
"this apply drops ClusterQueue <queue>, but it is still named by an application on this cluster; move those applications to another queue first, or keep the queue declared. ..."
Cause
A queue removed while an application uses it.
Fix
Recreate those applications on another queue first, or keep the queue.
"gangScheduling.timeout must be between 1m and 24h"
Cause
The Timeout is out of range.
Fix
Enter 1m to 24h.
"localQueues[<i>] binds to ClusterQueue <queue>, which clusterQueues does not declare (declared: <queues>)"
Cause
A LocalQueue names a queue that doesn't exist.
Fix
Name a declared Cluster queue.
"localQueues: <namespace> is an AstroPulse namespace; AstroPulse's own workloads are never queued"
Cause
A LocalQueue in one of AstroPulse's namespaces.
Fix
Use a namespace of yours.
"Workload Queueing is still used by applications: <applications>. Move or delete those applications before removing Workload Queueing."
Cause
An uninstall while applications use a queue.
Fix
Move or delete those applications first.

These come on a model application's dry run and apply:

Workload-queueing add-on does not watch Deployments, "source.model.queue names <queue>, but the workload-queueing add-on on cluster <cluster> does not watch Deployments, and a model's replicas run as one."
Cause
deployment is not in Integrations.
Fix
Add deployment to Integrations and apply the add-on again.
Workload queue is not declared on the cluster
Cause
The model names a queue the add-on doesn't declare.
Fix
Name a declared queue, or add it under Cluster queues.
Workload-queueing add-on is not installed
Cause
The model names a queue on a cluster without this add-on.
Fix
Install Workload Queueing, or remove queue from the model.

Next​