Bin Packing Is a Budget: Scheduler Scores in Practice

Average node utilization is a misleading number. Ours was 38%, which sounded like plenty of room, until we looked at the distribution: the 90th percentile node was at 71% and the 10th was at 12%. The cluster was not uniformly idle; it was unevenly packed, and the scheduler was doing that on purpose.

The default scoring plugins spread pods across nodes. NodeResourcesFit defaults to LeastAllocated, which prefers the emptiest node. That is a good default for resilience and a bad one for cost.

The knobs that matter

Plugin	Effect	When to change it
NodeResourcesFit (LeastAllocated)	spreads load	default; good for latency
NodeResourcesFit (MostAllocated)	packs tightly	batch and CI
NodeResourcesBalancedAllocation	prefers balanced CPU/memory	mixed workloads
TopologySpread	spreads by zone/node	HA
ImageLocality	prefers nodes with the image	large images
InterPodAffinity	co-location rules	caches, sidecars

Packing and spreading are not opposites you pick globally. They are per-workload policies, and scheduler profiles let you have both:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: default-scheduler
    plugins:
      score:
        enabled:
          - name: NodeResourcesBalancedAllocation
            weight: 2
  - schedulerName: pack-scheduler
    pluginConfig:
      - name: NodeResourcesFit
        args:
          scoringStrategy:
            type: MostAllocated

Batch pods set schedulerName: pack-scheduler and land on the fullest node that fits. Latency-sensitive pods keep the default profile.

What packing bought, and what it cost

Configuration	Avg utilization	API p99	Evictions/day
LeastAllocated (default)	38%	210ms	3
MostAllocated, all workloads	61%	248ms	11
Pack for batch, default for services	54%	212ms	4
Above + descheduler	58%	214ms	6

The second row is the tempting one and the wrong one: 18% worse p99 for 7 points of utilization. Packing batch separately got most of the savings without touching the latency-sensitive pool.

The descheduler is a garbage collector, not a scheduler

The scheduler only acts at pod creation. A cluster that changes shape — scale-downs, node failures, deployments — drifts away from its policy. The descheduler rebalances it by evicting pods, which means it is also a source of churn.

Our configuration:

apiVersion: descheduler/v1alpha2
kind: DeschedulerPolicy
profiles:
  - name: default
    pluginConfig:
      - name: LowNodeUtilization
        args:
          thresholds: { cpu: 20, memory: 20, pods: 20 }
          targetThresholds: { cpu: 50, memory: 50, pods: 50 }
      - name: RemoveDuplicates
    plugins:
      balance:
        enabled: [LowNodeUtilization, RemoveDuplicates]

Two lessons:

• Run it often and evict little. Every 15 minutes with a small batch is gentler than hourly with a large one.
• Exclude stateful and singleton pods from eviction, or the descheduler will move your database for fun.

We hold evictions under 20 per day per pool. When the number rises, it means the scheduling policy and the workload shape have diverged, and that is worth fixing rather than papering over.

Where the headroom goes

Packing to 80% looks great on a cost dashboard and terrible during a deployment, when every rolling update needs room for a surge pod. We keep:

• 65% target for latency-sensitive pools, because bursts and rolling updates need somewhere to land;
• 80% for batch pools, where the work is interruptible;
• one node of slack per zone for failure and drain.

The metrics that tell the truth

1. Node utilization distribution, not average: p10, p50, p90.
2. Scheduler queue latency (scheduler_pending_pods), which spikes before users notice.
3. Preemption count per minute.
4. Descheduler evictions per day, by pool.
5. Pending pods older than 5 minutes.

---

Bin packing is a budget. Spend it on batch first, keep some for latency, and measure the p99 you are buying with it.