Kubernetes inPlace Pod Resize Breaks Cluster Scheduler

Lucas: So Kubernetes 1.27 shipped in-place pod resize as a stable feature, and a lot of teams were excited — finally you could adjust CPU and memory requests without killing the pod. Luna: And that sounds like a dream for stateful workloads, right? No more draining a database just to add a bit more memory. Lucas: Exactly. But the reality is that in-place resize is quietly breaking the scheduler's core assumptions. I've been looking at a case from a mid-stage fintech — they do real-time payment routing — and they started seeing pods get preempted for no obvious reason after they enabled resize. Luna: Preempted? As in the scheduler kicked out lower-priority pods to make room for something else? Lucas: Yes. But here's the thing: the pods being preempted weren't the ones being resized. It was completely unrelated pods — batch jobs, sidecars — that got evicted. And the team couldn't figure out why. Luna: So what's the mechanism? How does changing one pod's resources cause another to get preempted? Lucas: It comes down to how the scheduler does bin-packing. When a pod is scheduled, the scheduler assumes its resource requests are static for its entire lifetime. It builds a plan — this pod fits here, that pod fits there — based on those numbers. Luna: And in-place resize changes the numbers mid-flight. Lucas: Right. When you increase a pod's CPU request, the kubelet applies it on the node, but the scheduler doesn't automatically recalculate the node's available resources. So now you have overcommit — the scheduler thinks there's more free capacity than there actually is. Luna: And that overcommit can trigger preemption when a new pod comes along, because the scheduler thinks it fits, but the node can't actually accommodate it. Lucas: Exactly. The fintech team saw this happen during a traffic spike. They resized a payment-processing pod from 1 CPU to 2 CPUs, and the scheduler's view of that node never updated. Then a new pod for a different service was scheduled onto the same node — and got immediately preempted because the node was actually at capacity. Luna: That's the kind of silent failure that makes on-call engineers pull their hair out. Was there any logging to trace it back? Lucas: Not really. The scheduler logs showed a normal preemption event — 'pod X preempted pod Y due to priority' — but there was no mention of the resize. The team had to correlate the timestamps with their resize audit logs to figure it out. Luna: So the scheduler is effectively blind to in-place resource changes. That feels like a fundamental design gap. Lucas: It is. And it's not just preemption. There's also resource fragmentation. When you resize a pod down — say reduce memory from 4 GB to 2 GB — the freed capacity is still accounted to that pod from the scheduler's perspective. So new pods can't use that memory unless they happen to land on that exact node. Luna: So you end up with nodes that have free capacity but the scheduler won't schedule new pods there because its view is stale. Lucas: Exactly. The fintech team had one node that was reporting 30 percent allocatable CPU free, but the scheduler never placed anything on it. They had to manually drain the node and let it reschedule everything fresh. Luna: That sounds like something you'd want to avoid in production. What's the workaround? Lucas: There's a few options. One is to avoid in-place resize for pods that are part of a larger deployment — just recreate them. Another is to use a mutating admission webhook that synchronises the resize with the scheduler by updating the pod's resource spec in the API server, which triggers a re-schedule. Luna: But doesn't that require custom code? Not everyone has the bandwidth for that. Lucas: True. Another simpler approach is to set the resize policy to 'NotRequired' for the resources you care about — that way the pod is still restarted when resources change, which forces a fresh scheduling decision. Luna: So you lose the no-downtime benefit, but you avoid the silent fragmentation. Lucas: Right. And honestly, for most stateless workloads, a rolling restart is fine. The real value of in-place resize is for stateful workloads like databases, but that's precisely where the scheduling blind spot hurts the most. Luna: It's almost like the feature was designed for a specific use case but not hardened for mixed workloads. Lucas: That's a fair assessment. And speaking of things that deserve more hardening — if today's conversation helped you think about your own cluster's scheduling quirks, maybe it was worth a coffee. You can find the show at buy me a coffee dot com slash fexingo. No pressure, but listener support is what keeps this ad-free. Luna: Yeah, and we dig into these edge cases so you don't have to discover them in a Friday night incident. Lucas: So back to the fragmentation problem — there's also a timing issue with the horizontal pod autoscaler. If HPA triggers a scale-up at the same time you're resizing pods, you can get a cascade of preemptions. Luna: Because the autoscaler sees the old resource requests and thinks it can fit more pods than it actually can. Lucas: Exactly. The fintech team had a spike in traffic, HPA wanted to add three more pods, but because some existing pods had been resized up, the node was already full. The scheduler started preempting lower-priority pods to make room, but those pods were part of the same application — so it made the traffic problem worse. Luna: So you get this feedback loop where the system tries to handle load but ends up cannibalising itself. Lucas: Yes. And the fix they ended up with was to disable in-place resize for any pod that was managed by an HPA. They used a label selector in their resize webhook to exclude those deployments. Luna: That seems like a pragmatic middle ground. Use resize for low-priority background pods, but not for your core traffic-handling ones. Lucas: Exactly. And there's a Kubernetes Enhancement Proposal — KEP-1287 — that addresses this scheduler awareness issue. It's been in discussion for a while, but as of June 2026, it's not yet stable. The proposal would make the scheduler subscribe to resource update events from the kubelet, so it can adjust its view in near real-time. Luna: That would solve the blind spot. But until then, teams need to be careful about how they use resize. Lucas: And careful about their monitoring. The team I talked to now has a Prometheus alert that fires whenever a pod's resource request changes — they log it and check if the scheduler's node allocation metrics deviated from the actual node usage. It's extra work, but it's caught several near-misses. Luna: What about the resize policy parameter itself? You mentioned 'NotRequired' — does that mean the pod just gets the new resources on restart? Lucas: Right. There are two resize policies: 'NotRequired' means the pod will continue with the old limits until it's restarted, while 'Required' means the kubelet will try to apply the new limits in-place — and if it can't, it will fail the resize. Most teams use 'Required' because they want the immediate effect, but that's exactly when the scheduler gets out of sync. Luna: So setting it to 'NotRequired' and then triggering a rolling restart is actually safer. Lucas: Safer for the cluster, but more disruptive for the pod. There's no free lunch. I think the key takeaway is: in-place resize is a powerful tool, but it's not a magic bullet. You need to understand how it interacts with the scheduler, and you need to test it in a non-production environment first. Luna: And maybe keep an eye on that KEP. I've seen a few community discussions about it — people are pretty passionate about getting this fixed. Lucas: Absolutely. If it lands in the next year or so, it'll make resize a lot more predictable. Until then, treat it like a surgical instrument, not a default workflow.