Priority, Preemption, and the Workloads That Should Wait

At 10:14 a CI job scaled from 20 to 200 pods. It had been given a PriorityClass copied from a platform component, because "CI is important". The scheduler did what priority means: it preempted lower-priority pods to make room.

Forty interactive API pods were killed. The API degraded for twelve minutes while the CI job ran at full parallelism, and the job would have finished fine at half the size, an hour later.

The incident was not a scheduler bug. It was a missing design.

Priority is a contract, not a knob

Every workload falls into one of four classes, and the class should be decided by the service owner and reviewed like an API change:

Class	Priority	Examples	Preemption
platform	100000	CNI, DNS, ingress, autoscaler	may preempt
interactive	10000	user-facing APIs, queues	may preempt batch
batch	1000	CI, rendering, backfills	Never
best-effort	0	experiments, dev tools	Never

The important column is the last one. preemptionPolicy: Never means "can be preempted, but will not preempt". Batch work should almost always be in this category: it is patient, it is retryable, and it should not evict a user request to start sooner.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: batch
value: 1000
preemptionPolicy: Never
globalDefault: false
description: "Interruptible work. Never preempts anything."

PDBs do not protect against preemption

This is the detail that surprises people: a PodDisruptionBudget governs voluntary disruptions such as drains and evictions from the eviction API. Preemption is not eviction; the scheduler deletes pods directly, and a PDB will not stop it.

So "we have a PDB" is not a defense against a badly designed priority class. The defenses are:

• preemptionPolicy: Never on anything that should not preempt;
• a reserved buffer of capacity for interactive work;
• priority values that are far apart, so accidental collisions are obvious.

Preemption also needs a victim policy

When preemption happens, the scheduler picks victims by lowest priority, preferring pods whose PDB would not be violated — but if no such victim exists, it preempts anyway. That "anyway" is why the buffer matters more than the algorithm.

We reserve 10% of each interactive pool as unschedulable headroom by keeping a small pause deployment at priority: 10000 that holds the space. It is ugly, it is cheap, and it means a batch surge finds nothing to eat.

Gang scheduling and queues

Some workloads are all-or-nothing: a job with 20 workers is useless if 19 start and one waits. Default Kubernetes will happily schedule 19 and starve the twentieth.

Two mechanisms:

• Gang scheduling (Volcano, scheduler plugins) admits the whole group or none of it.
• Queueing with quotas (Kueue) holds jobs until the quota allows the full group, rather than letting them occupy the cluster piecemeal.

We use Kueue for CI: a ClusterQueue with a nominal quota and borrowing from a shared pool. Jobs wait in a queue, which is exactly what batch work should do. The cluster stops being a race and becomes a schedule.

What changed

Metric	Before	After
Preemption events/day	310	12
Interactive pods preempted/week	180	0
Batch median queue time	0s (all at once)	4m 20s
API error minutes from batch	12	0

The batch median queue time went up, and that is the intended trade: batch work is patient, user requests are not.

The design rules I would keep

1. Four priority classes, no more; values far apart.
2. preemptionPolicy: Never for batch and best-effort.
3. A capacity buffer for interactive pools, not just a PDB.
4. Queue batch work; do not let it race.
5. Review PriorityClass changes like API changes, because they are.

---

Priority is the part of scheduling that can delete someone else's work. Treat it with the seriousness of a production change, because that is what it is.