How Kubernetes Topology Spread Constraints Create Scheduling Hotspots

Lucas: So you set topology spread constraints on a deployment, you see pods land across three zones, everything looks balanced on paper - and then your cluster starts throwing pod pending alerts. What's actually going on? Luna: I've seen that. It's usually something subtle about how the scheduler interprets those constraints.

But let's start with the basics - topology spread constraints were supposed to give us fine-grained control over pod distribution, right? Lucas: Exactly. They were introduced as a more flexible alternative to pod anti-affinity. You define a topology key - like 'topology.

kubernetes.io/zone' - and a maxSkew. The scheduler then tries to ensure that the difference in the number of pods across each topology domain never exceeds that maxSkew. Simple in theory.

Luna: So what breaks? I've heard people complain that even with maxSkew set to 1, they still get unbalanced pods. Lucas: A few things. First, the maxSkew constraint only applies to pods that match the selector - and it only counts pods that are scheduled, not pending.

So if you have a rolling update, old pods that are still running skew the count. New pods get scheduled to domains with fewer matching pods, but the actual distribution might look uneven during the rollout. Luna: That's a timing issue, though. It resolves once the old pods terminate.

What about persistent hotspots? I've debugged a deployment where one node kept getting overloaded despite the constraint. Lucas: That's a classic case of node label mismatch. The topology key must be present on every node.

If some nodes don't have the label, the scheduler treats them as 'unknown domain' and applies a separate spread constraint. But by default, pods can still be scheduled to unlabeled nodes, which can accumulate more pods than intended. Luna: Right. I've seen that with custom topology keys - like if you use 'failure-domain.

beta.kubernetes.io/zone' instead of the newer 'topology.kubernetes.

io/zone'. Some nodes might only have the old label. Then you get a split domain. Lucas: Another common culprit: resource requests.

The scheduler only considers the count of pods, not their resource consumption. So if you have two pods in zone A each requesting 4 CPU cores, and one pod in zone B requesting 0.5 cores, zone A is far more loaded, but the skew is still 1 - because it's 2 vs 1 pods. Luna: Wait, so the constraint is purely count-based?

That sounds like a design choice that could easily mislead people who think it's about resource balance. Lucas: It is count-based. The docs are clear about that, but it's easy to overlook. There's a long-standing issue on the Kubernetes GitHub - issue number 107985 - where users argue for a resource-weighted spread.

But as of Kubernetes 1.30, it's still count-only. Luna: So you combine that with a pod that has high resource requests, and you get a scheduling hotspot. I've seen a case where a deployment with 10 replicas, each requesting 8 CPU, hit a three-node cluster.

The spread constraint made sure 4, 3, 3 pods across nodes - but the node with 4 pods was completely saturated, and the other two had room. The scheduler didn't care. Lucas: Exactly. And that's where the 'silent skew' gets dangerous.

Your pod distribution looks balanced in the scheduler's eyes, but your actual cluster utilization is lopsided. The fix isn't always straightforward - you might need to combine topology spread with taints and tolerations, or use pod anti-affinity with topologyKey and a labelSelector that includes resource requirements. Luna: Speaking of combining constraints, there's another subtlety: if you specify multiple topology spread constraints, the scheduler must satisfy all of them simultaneously.

That can cause scheduling failures even when each constraint individually seems fine. Lucas: Yes, that's a known gotcha. For example, if you set one constraint on zone and another on hostname, the scheduler has to find a node where both constraints are met - meaning the pod count per zone and per hostname both stay within maxSkew. If you have uneven hostname counts per zone, some nodes become effectively unschedulable.

Luna: So you might see pods stuck in pending with no clear reason. I've debugged that - the scheduler logs just say '0/10 nodes available: 2 node didn't match topology constraint'. Not super helpful if you don't know which constraint. Lucas: You can increase the log verbosity on the scheduler - set - v=6 - and you'll see which constraint failed and the domain counts.

But in production, you don't want to restart the scheduler to debug. Better to use the scheduling framework's metrics. There's a metric called 'scheduler_pod_scheduling_duration_seconds' that can help you correlate pending pods with constraint evaluations. Luna: Let's talk about a real-world example.

I had a deployment with 20 replicas across 3 zones, each with 2 nodes. I set two constraints: one on zone with maxSkew 1, and one on hostname with maxSkew 2. The scheduler consistently failed to schedule the last few pods. Lucas: Classic.

Because the zone constraint forces an even distribution - ideally 7, 7, 6 - but the hostname constraint wants no more than 2 pods per hostname. In a two-node zone, that means at most 4 pods per zone. So the zone constraint expects 7, but the hostname constraint caps it at 4. The scheduler can't reconcile that, and pods stay pending.

Luna: Exactly. We had to relax the hostname maxSkew to 4, or remove it entirely and rely on pod anti-affinity for host-level spread. Lucas: That's the right call. The general rule: if you use multiple topology keys, make sure the domains are nested consistently.

Each node belongs to exactly one domain per key. And the product of the maxSkews across keys shouldn't exceed the total number of nodes, otherwise you're inviting scheduling deadlock. Luna: There's also a less common but interesting case: when you use 'whenUnsatisfiable: DoNotSchedule' with a maxSkew of 0. That forces strict even distribution.

But if you have more replicas than nodes, it becomes impossible - you're requiring that no node has more than any other, but with more pods than nodes, some nodes will have more. That causes scheduling failures for the extra pods. Lucas: Right. You need to understand that maxSkew of 0 is essentially 'exactly equal distribution'.

That's fine if the number of replicas is a multiple of the number of topology domains. Otherwise, you need maxSkew at least 1 to allow a difference of 1. Luna: So what's the best practice? Say you want to balance pods across zones but also avoid stacking too many on one node.

Lucas: I'd recommend using one topology spread constraint per deployment - usually on zone - and combine it with pod anti-affinity for host-level spread. That gives you zone balance via spread constraint, and host-level anti-affinity via a separate rule. You get the same effect as multiple spread constraints but with more predictable behavior. Luna: And be mindful of the pod count.

If you have, say, 3 nodes per zone and 9 replicas, spread constraint on zone with maxSkew 1 will put 3 pods per zone. That's fine. But if you have 4 replicas, one zone gets 2 and the others get 1 - that's still within skew 1. But if you also have anti-affinity per host, each zone's pods might not fit on separate nodes if the zone has only 2 nodes.

Lucas: That's a good point. You need to plan the headroom. Another thing: node labeling consistency. Make sure all your nodes have the same topology labels.

Use a mutation webhook or a node-labeling controller to ensure that new nodes automatically get the required labels. Otherwise, you'll get the 'unknown domain' problem we mentioned. Luna: And what about when you have a mix of node sizes? Should you adjust pod resource requests to match?

Lucas: That's one approach. If you have nodes with different capacities, you can set pod resource requests so that each pod consumes a relatively uniform fraction of node capacity. But that's not always feasible. A more robust solution is to use node selectors or taints to segregate workloads by node type, and then apply topology spread constraints within each group.

Luna: So you essentially create node pools and apply constraints per pool. That does add operational overhead. Lucas: It does, but it's the cleanest way to avoid silent skew. There's also the option to use the 'nodeAffinity' with 'requiredDuringSchedulingIgnoredDuringExecution' to force pods onto specific node groups, but then you lose some flexibility.

Luna: Let's circle back to debugging. If you're seeing pending pods and suspect topology constraints, what's your step-by-step? Lucas: First, check the pod events: 'kubectl describe pod <pod-name>' and look for the 'FailedScheduling' event. It usually tells you the number of nodes that didn't match constraints.

Then check the node labels: 'kubectl get nodes - show-labels'. Ensure consistency. Next, look at the deployment's topology spread constraints: 'kubectl get deployment <name> -o yaml | grep topologySpreadConstraints'. Verify the keys and maxSkew values.

Luna: And if the events are not clear? Lucas: Then enable scheduler profiling. You can set the scheduler's profiling flag to 'true' and get detailed traces. But that's heavy.

Simpler: use 'kubectl get pods -o wide' to see which nodes the running pods are on, and manually compute the distribution. Compare it to your expectation. If you see significant imbalance, look at resource requests per pod - that might be the culprit. Luna: I also like to use the 'kube scheduler simulator' tool - it's a web UI that lets you simulate scheduling decisions.

You can input your cluster topology and deployment specs, and it shows you exactly where each pod lands. Great for offline debugging. Lucas: That's a good tip. There's also the 'topologySpreadConstraints' simulator in the Kubernetes SIG scheduling repository.

Very useful for complex scenarios. Luna: So to wrap up, the key takeaways: topology spread constraints are count-based, not resource-aware. Multiple constraints can conflict. Node label consistency is critical.

And maxSkew of 0 is stricter than most people realize. Lucas: Right. And one more: always test your constraints with a non-production deployment first. Simulate the exact number of replicas you plan to run.

It takes five minutes and can save you a production incident. Luna: Quick honest thing - if today's tech conversation gave you something usable, we should mention that a handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making this many of these shows possible. Lucas: Yeah, no ads, no sponsors - it's listener supported. A small group already keeps it going, and that's the only way we can keep digging into these Kubernetes quirks.

Luna: Exactly. And we're back to it - let's talk about one more edge case that came up in the Kubernetes community recently: topology spread constraints with the 'whenUnsatisfiable: ScheduleAnyway' option. That's a whole different beast - the scheduler will relax the constraint if it can't satisfy it, but the behavior isn't always what you'd expect. Lucas: Good point.

That option essentially tells the scheduler to try its best but not block scheduling. However, it still imposes a soft constraint that influences node ranking. The scheduler will prefer nodes that keep the skew low, but if no node can improve it, it will still schedule. That can lead to even more imbalanced clusters if not monitored.

Luna: So it's more like a hint than a guarantee. I've seen teams use that mode to avoid pod pending, but then wonder why their distribution drifts over time. Lucas: That's the trade-off. You trade strict balance for availability.

In practice, I'd recommend using 'DoNotSchedule' with a reasonable maxSkew - like 1 or 2 - and ensure your cluster has enough headroom to handle the constraint. If you need to overcommit, then use 'ScheduleAnyway' but set up monitoring alerts for skew exceeding a threshold. Luna: So, next time you see a perfectly balanced pod distribution on paper but unbalanced load in practice, check the resource profiles and node labels. And remember, the scheduler is doing exactly what you told it - even if it's not what you meant.

Lucas: Exactly. That's the Kubernetes way: it does exactly what you configure, no more, no less. The hard part is knowing what to configure. Hopefully today's episode helps you avoid a few hours of debugging.