Kubernetes Scheduling Is a Policy, Not a YAML

Every cluster has a scheduling policy. Most teams have never read theirs; it is the sum of defaults, a few nodeSelectors copied from a tutorial, and whatever the Helm chart set.

We audited ours after an incident where all six replicas of a service landed in one availability zone. The cause was a node affinity that expressed "prefer this instance family" and nothing that expressed "spread". The scheduler did exactly what it was told.

This is the design I use now: four layers, each answering a different question.

Layer 1: node pools are the coarse policy

Pools express the shape of the hardware and the price of the workload:

Pool	Instance	Taint	Workload
general	c6i.2xlarge	none	APIs, services
memory	r6i.4xlarge	dedicated=memory:NoSchedule	caches, analytics
spot	mixed	lifecycle=spot:NoSchedule	batch, CI

Taints keep accidental placement out; tolerations make the exception explicit and reviewable. A pod that tolerates spot is declaring that it can be interrupted — a product statement, not a scheduling detail.

Layer 2: requests are the real scheduler input

The scheduler does not see actual usage. It sees requests. That makes requests a policy instrument:

• Requests too high: nodes look full while CPU is idle. Our average utilization was 38%.
• Requests too low: noisy neighbours, evictions, and an HPA reacting to the wrong signal.

We treat requests as a contract reviewed with the service owner, and we measure p95 usage divided by requests monthly. Below 0.3 for two weeks, the requests come down. Above 0.9, the owner gets a warning and a capacity plan.

Layer 3: spread and affinity describe failure domains

The default scheduler spreads pods only when it has to. If zone or node spread matters, say it explicitly:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels: { app: checkout }

ScheduleAnyway is soft; DoNotSchedule is hard. Hard constraints can leave pods unschedulable during a zone failure, which is the opposite of resilience. We use hard constraints only when a workload genuinely cannot run twice in one zone, and soft constraints everywhere else.

A PodDisruptionBudget is not a scheduling constraint, but it belongs to this layer: it governs voluntary evictions. A PDB of minAvailable: 100% blocks every node drain. We cap PDBs at 80% and alert when a drain takes longer than 15 minutes.

Layer 4: priority and preemption

Priority decides who wins when the cluster is full, and it is the only layer that can remove running work. It gets its own review process and, in practice, its own article. The short version: every workload has a PriorityClass, and preemptionPolicy: Never is the default for anything that must not displace others.

What changed after the audit

Metric	Before	After
Zone skew across 3 AZs	8%	2%
Batch minutes on spot	0%	62%
Average node utilization	38%	52%
Drain time during upgrades	26 min	9 min

The utilization change came from fixing requests, not from packing harder. The drain time came from realistic PDBs — four services had been blocking evictions silently for months.

The audit questions

1. Which pools exist, and what keeps the wrong workload out?
2. Are requests within 2× of p95 usage?
3. Which failure domain matters, and is it a soft constraint?
4. Which PDBs block drains today?
5. Which workloads can preempt, and which can be preempted?

---

Scheduling is not a YAML field you set once. It is a set of answers about cost, resilience, and fairness — and the defaults have answers too.