DevOps Daily with Fexingo · 2026-06-24 · 8 min
Key moments - from our scoring
Substance score
55 / 100
Five dimensions, 20 points each
Kubernetes 1.27 introduced in-place pod resize as a stable feature, promising zero-downtime resource adjustments for stateful workloads - but Lucas reveals a critical scheduling blind spot that's causing silent failures in production clusters. When pods are resized in-place, the scheduler's bin-packing algorithm doesn't recalculate node capacity, creating resource overcommit that triggers unexpected preemptions of unrelated workloads. A mid-stage fintech firm discovered this the hard way: after resizing a payment-processing pod's CPU request, the scheduler placed new pods on nodes it thought had capacity but were actually full, causing cascading evictions of batch jobs and sidecars. The problem compounds with horizontal pod autoscaling, where traffic spikes trigger preemption feedback loops. Luna and Lucas discuss practical mitigation strategies - disabling resize for HPA-managed pods, using mutating webhooks to force rescheduling, or relying on the NotRequired resize policy to trigger rolling restarts. They reference KEP-1287, a pending enhancement proposal that would make the scheduler aware of in-place resize events in near real-time, and recommend Prometheus alerting to catch resource drift before it destabilizes production clusters.
When a pod's resources are resized in-place, the kubelet applies the change on the node but the scheduler doesn't recalculate available capacity. This creates overcommit - the scheduler still thinks nodes have free capacity they've lost - so when new pods are scheduled, the scheduler incorrectly places them on full nodes, triggering preemption of lower-priority workloads to make room.
The team resized a payment-processing pod from 1 CPU to 2 CPUs, but the scheduler's node capacity view never updated. When traffic spiked and HPA tried to schedule new pods, the scheduler placed them on the already-full node, causing cascading preemptions of batch jobs and sidecars - making the traffic problem worse instead of solving it.
Options include: avoiding resize for pods in deployments (recreate instead), using a mutating admission webhook to synchronize resize with the scheduler, disabling resize for HPA-managed pods via label selectors, or setting the resize policy to 'NotRequired' to force rolling restarts instead of in-place updates.
KEP-1287 is a Kubernetes Enhancement Proposal that would make the scheduler subscribe to resource update events from the kubelet, allowing it to adjust node capacity calculations in near real-time instead of remaining blind to in-place resize operations. As of June 2026, it remains in discussion and not yet stable.
The fintech team implemented a Prometheus alert that fires whenever a pod's resource request changes, then checks if the scheduler's node allocation metrics deviate from actual node usage to catch near-misses before they cause production incidents.
Our reviewer’s read on each dimension, with quotes from the episode.
For an 8-minute episode, the density is genuinely high: scheduler overcommit mechanics, resource fragmentation on downsize, HPA cascade failure, and three distinct workarounds are all covered with explanatory depth. A mid-episode donation plug is the only meaningful padding.
When you increase a pod's CPU request, the kubelet applies it on the node, but the scheduler doesn't automatically recalculate the node's available resources. So now you have overcommit
When you resize a pod down - say reduce memory from 4 GB to 2 GB - the freed capacity is still accounted to that pod from the scheduler's perspective. So new pods can't use that memory unless they happen to land on that exact node.
The scheduler-blindness angle on in-place resize is a legitimately niche and non-obvious framing; the HPA self-cannibalization feedback loop is a fresh and counterintuitive consequence. The workarounds (webhooks, rolling restarts) are standard Kubernetes practice and lower the originality ceiling.
the scheduler starts preempting lower-priority pods to make room, but those pods were part of the same application - so it made the traffic problem worse
you get this feedback loop where the system tries to handle load but ends up cannibalising itself
There are no external guests - Lucas and Luna are co-hosts, with Lucas presenting research on a third-party fintech case rather than speaking from direct personal operator experience at scale. The knowledge is clearly competent but secondhand.
I've been looking at a case from a mid-stage fintech - they do real-time payment routing - and they started seeing pods get preempted for no obvious reason after they enabled resize.
The team had to correlate the timestamps with their resize audit logs to figure it out.
Concrete numbers appear throughout - 1 CPU to 2 CPUs resize, 4 GB to 2 GB memory, 30% allocatable CPU reported free, 3 HPA-triggered pods, KEP-1287, Kubernetes 1.27, and a June 2026 status check. The fintech company is unnamed and described only as 'mid-stage', which limits verifiability.
They resized a payment-processing pod from 1 CPU to 2 CPUs, and the scheduler's view of that node never updated.
The fintech team had one node that was reporting 30 percent allocatable CPU free, but the scheduler never placed anything on it.
Luna's questions reliably advance the explanation and occasionally name the right concept ('So the scheduler is effectively blind to in-place resource changes'), but they function as scripted prompts rather than genuine probes - no pushback, no challenged claims, no follow-up that forces unexpected depth.
So the scheduler is effectively blind to in-place resource changes. That feels like a fundamental design gap.
That sounds like something you'd want to avoid in production. What's the workaround?
Computed from the transcript - who did the talking, and the words that came up most.
Episode 70 of DevOps Daily with Fexingo dives into Kubernetes 1.27's in-place pod resize feature and why it's causing unexpected scheduling headaches. Lucas and Luna unpack a real case from a mid-stage fintech where CPU and memory adjustments triggered pod preemption, resource fragmentation, and silent failures in the scheduler's bin-packing logic. They explain the resize policy, what happens when a pod's resource limits shift mid-lifecycle, and why the kube-scheduler's assumptions about static resource requests break down. Along the way, they offer practical workarounds - like avoiding overlapping resize and autoscaler events - and discuss when you should still recreate pods instead of resizing. A must-listen for any team running Kubernetes with dynamic workload demands. #Kubernetes #InPlacePodResize #K8sScheduler #PodPreemption #ResourceFragmentation #ClusterOps #DevOps #CloudNative #KubeScheduler #ContainerOrchestration #PodLifecycle #Fintech #TechPodcast #FexingoBusiness #DevOpsDaily #BusinessPodcast #Technology #KubernetesTroubleshooting Keep every episode free: buymeacoffee.com/fexingo
Transcribed and scored by The B2B Podcast Index.
Lucas: So Kubernetes 1.27 shipped in-place pod resize as a stable feature, and a lot of teams were excited - finally you could adjust CPU and memory requests without killing the pod. Luna: And that sounds like a dream for stateful workloads, right? No more draining a database just to add a bit more memory.
Lucas: Exactly. But the reality is that in-place resize is quietly breaking the scheduler's core assumptions. I've been looking at a case from a mid-stage fintech - they do real-time payment routing - and they started seeing pods get preempted for no obvious reason after they enabled resize. Luna: Preempted?
As in the scheduler kicked out lower-priority pods to make room for something else? Lucas: Yes. But here's the thing: the pods being preempted weren't the ones being resized. It was completely unrelated pods - batch jobs, sidecars - that got evicted.
And the team couldn't figure out why. Luna: So what's the mechanism? How does changing one pod's resources cause another to get preempted? Lucas: It comes down to how the scheduler does bin-packing.
When a pod is scheduled, the scheduler assumes its resource requests are static for its entire lifetime. It builds a plan - this pod fits here, that pod fits there - based on those numbers. Luna: And in-place resize changes the numbers mid-flight. Lucas: Right.
When you increase a pod's CPU request, the kubelet applies it on the node, but the scheduler doesn't automatically recalculate the node's available resources. So now you have overcommit - the scheduler thinks there's more free capacity than there actually is. Luna: And that overcommit can trigger preemption when a new pod comes along, because the scheduler thinks it fits, but the node can't actually accommodate it. Lucas: Exactly.
The fintech team saw this happen during a traffic spike. They resized a payment-processing pod from 1 CPU to 2 CPUs, and the scheduler's view of that node never updated. Then a new pod for a different service was scheduled onto the same node - and got immediately preempted because the node was actually at capacity. Luna: That's the kind of silent failure that makes on-call engineers pull their hair out.
Was there any logging to trace it back? Lucas: Not really. The scheduler logs showed a normal preemption event - 'pod X preempted pod Y due to priority' - but there was no mention of the resize. The team had to correlate the timestamps with their resize audit logs to figure it out.
Luna: So the scheduler is effectively blind to in-place resource changes. That feels like a fundamental design gap. Lucas: It is. And it's not just preemption.
There's also resource fragmentation. When you resize a pod down - say reduce memory from 4 GB to 2 GB - the freed capacity is still accounted to that pod from the scheduler's perspective. So new pods can't use that memory unless they happen to land on that exact node. Luna: So you end up with nodes that have free capacity but the scheduler won't schedule new pods there because its view is stale.
Lucas: Exactly. The fintech team had one node that was reporting 30 percent allocatable CPU free, but the scheduler never placed anything on it. They had to manually drain the node and let it reschedule everything fresh. Luna: That sounds like something you'd want to avoid in production.
What's the workaround? Lucas: There's a few options. One is to avoid in-place resize for pods that are part of a larger deployment - just recreate them. Another is to use a mutating admission webhook that synchronises the resize with the scheduler by updating the pod's resource spec in the API server, which triggers a re-schedule.
Luna: But doesn't that require custom code? Not everyone has the bandwidth for that. Lucas: True. Another simpler approach is to set the resize policy to 'NotRequired' for the resources you care about - that way the pod is still restarted when resources change, which forces a fresh scheduling decision.
Luna: So you lose the no-downtime benefit, but you avoid the silent fragmentation. Lucas: Right. And honestly, for most stateless workloads, a rolling restart is fine. The real value of in-place resize is for stateful workloads like databases, but that's precisely where the scheduling blind spot hurts the most.
Luna: It's almost like the feature was designed for a specific use case but not hardened for mixed workloads. Lucas: That's a fair assessment. And speaking of things that deserve more hardening - if today's conversation helped you think about your own cluster's scheduling quirks, maybe it was worth a coffee. You can find the show at buy me a coffee dot com slash fexingo.
No pressure, but listener support is what keeps this ad-free. Luna: Yeah, and we dig into these edge cases so you don't have to discover them in a Friday night incident. Lucas: So back to the fragmentation problem - there's also a timing issue with the horizontal pod autoscaler. If HPA triggers a scale-up at the same time you're resizing pods, you can get a cascade of preemptions.
Luna: Because the autoscaler sees the old resource requests and thinks it can fit more pods than it actually can. Lucas: Exactly. The fintech team had a spike in traffic, HPA wanted to add three more pods, but because some existing pods had been resized up, the node was already full. The scheduler started preempting lower-priority pods to make room, but those pods were part of the same application - so it made the traffic problem worse.
Luna: So you get this feedback loop where the system tries to handle load but ends up cannibalising itself. Lucas: Yes. And the fix they ended up with was to disable in-place resize for any pod that was managed by an HPA. They used a label selector in their resize webhook to exclude those deployments.
Luna: That seems like a pragmatic middle ground. Use resize for low-priority background pods, but not for your core traffic-handling ones. Lucas: Exactly. And there's a Kubernetes Enhancement Proposal - KEP-1287 - that addresses this scheduler awareness issue.
It's been in discussion for a while, but as of June 2026, it's not yet stable. The proposal would make the scheduler subscribe to resource update events from the kubelet, so it can adjust its view in near real-time. Luna: That would solve the blind spot. But until then, teams need to be careful about how they use resize.
Lucas: And careful about their monitoring. The team I talked to now has a Prometheus alert that fires whenever a pod's resource request changes - they log it and check if the scheduler's node allocation metrics deviated from the actual node usage. It's extra work, but it's caught several near-misses. Luna: What about the resize policy parameter itself?
You mentioned 'NotRequired' - does that mean the pod just gets the new resources on restart? Lucas: Right. There are two resize policies: 'NotRequired' means the pod will continue with the old limits until it's restarted, while 'Required' means the kubelet will try to apply the new limits in-place - and if it can't, it will fail the resize. Most teams use 'Required' because they want the immediate effect, but that's exactly when the scheduler gets out of sync.
Luna: So setting it to 'NotRequired' and then triggering a rolling restart is actually safer. Lucas: Safer for the cluster, but more disruptive for the pod. There's no free lunch. I think the key takeaway is: in-place resize is a powerful tool, but it's not a magic bullet.
You need to understand how it interacts with the scheduler, and you need to test it in a non-production environment first. Luna: And maybe keep an eye on that KEP. I've seen a few community discussions about it - people are pretty passionate about getting this fixed. Lucas: Absolutely.
If it lands in the next year or so, it'll make resize a lot more predictable. Until then, treat it like a surgical instrument, not a default workflow.
Other episodes covering the same guests and topics, from across The B2B Podcast Index.
How Idempotency-Key Design Prevents Payment DisastersThe Developer Tools Podcast with Fexingo · features Luna98 / 100
Why Pipeline Velocity Trumps Deal Size Every TimeThe Growth Operator with Fexingo · features Luna95 / 100
Why Enterprise Software Deals Now Include a Vendor AI Model Explainability MandateB2B SaaS Talks with Fexingo · features Luna94 / 100
How B2B Brands Wreck Pipeline with Unsyncroned CRM DataThe Marketing Operator Podcast with Fexingo · features Luna92 / 100
Why Marketing Attribution Misses the Seasonality PatternMarketing Analytics with Fexingo · features Luna91 / 100
How to Sell Against a Competitor Already in the BuildingSales Leadership with Fexingo · features Luna85 / 100
How Kubernetes ServiceAccount Token Expiration Breaks CI Workflows75 / 100How Kubernetes StatefulSet PVC Resizing Causes Node Disk Failures94 / 100How Kubernetes Topology Spread Constraints Create Scheduling Hotspots95 / 100How Kubernetes CRD Versioning Breaks Controller Upgrades90 / 100How Kubernetes Audit Logging Causes etcd Performance Degradation91 / 100