How Kubernetes StatefulSet PVC Resizing Causes Node Disk Failures

Lucas: So you're running a StatefulSet in production - say, a Kafka cluster or a Cassandra ring - and you realize your PersistentVolumeClaims are undersized. You bump the storage template from 50 gigs to 100 gigs, roll out the update, and then watch your nodes start dropping like flies. Luna: That sounds like a nightmare. What actually goes wrong?

Lucas: It's a race condition introduced by the PVC resize feature that's been stable since Kubernetes 1.24. The StatefulSet controller, when it sees a new storage template, doesn't serialize the resize operations across replicas. It kicks off resizes for every pod that needs the new size - all at once.

Luna: So you get a stampede of resize requests hitting the storage backend simultaneously? Lucas: Exactly. And the storage backend - whether it's EBS, GCE Persistent Disk, or a CSI driver like Rook Ceph - has rate limits. When you exceed those limits, some resize calls fail, but the PVC object may already be marked as 'expanding' in etcd.

The kubelet then tries to mount a PVC that's in an inconsistent state, which can lead to mount timeouts, disk errors, and eventually node pressure. Luna: I've seen this happen. The node's kubelet starts reporting 'failed to mount volume' and the node condition flips to DiskPressure. Then pods get evicted, and the cascading effect can take down half the cluster.

Lucas: Right. One of the more famous incidents was a mid-sized SaaS company - they ran a 30-node cluster with a 16-replica StatefulSet for their message queue. They bumped the PVC size from 50GB to 100GB around noon, and within 15 minutes, three nodes entered DiskPressure. The storage backend - in that case, an older version of Rook Ceph - couldn't handle the simultaneous resize operations.

Luna: Three nodes out of thirty doesn't sound catastrophic, but I bet the impact was bigger. Lucas: Because those three nodes each hosted four or five other pods - web servers, caches, other stateful workloads. The evictions caused a thundering herd on the remaining nodes, and the cluster autoscaler took almost 10 minutes to spin up replacements. Total recovery time: about 45 minutes.

Not a full outage, but degraded performance for most of the afternoon. Luna: And the root cause was specifically the StatefulSet controller not throttling resize operations? Lucas: Partly. The controller itself doesn't have built-in throttling for PVC resizes.

But there's also a race in the generic ephemeral volume feature - when a PVC is created or resized as part of a pod template, the kubelet can attempt to mount the volume before the CSI driver finishes the expansion. If the expansion fails midway, the volume ends up in a weird state where the capacity in etcd doesn't match the actual block device size. Luna: So even if you resize one pod at a time, you could still hit issues if the CSI driver isn't atomic? Lucas: Yes.

But the biggest bang for the buck is the concurrent resize problem. Some teams work around it by using a rolling update with a custom script that updates one PVC at a time and waits for the resize to complete. But that's fragile - you have to manually coordinate with the StatefulSet's pod management policy. Luna: Are there any built-in mitigations now?

I know the CSI spec has a 'volume expansion' capability that includes a 'volume_expansion_policy' parameter. Does that help? Lucas: It does, but only if the CSI driver supports the 'Online' expansion policy. That lets the kubelet expand the volume while the pod is still running, without needing to remount.

But most drivers still default to 'Offline' - meaning you need to restart the pod to complete the resize. And the old volume mount has to be released first, which is where the race creeps in. Luna: So what's the practical advice for someone who needs to resize a StatefulSet's storage today? Lucas: First: never change the storage template and trigger a rolling update simultaneously.

Instead, manually scale down the StatefulSet to zero replicas, update the template, then scale back up. That way, each PVC resize is isolated to the pod creation sequence. Luna: But scaling to zero means downtime for that stateful service. Not always an option for production.

Lucas: Right. So the next best approach is to use a pod disruption budget that allows only one pod to be unavailable at a time, combined with a phased rollout using the 'OnDelete' update strategy. You manually delete each pod after confirming its PVC resize has completed - using 'kubectl get pvc' and checking the status field for 'FileSystemResizePending' or 'true'. Luna: And you can also set a resource quota on the storage backend's API calls?

I've seen teams throttle CSI requests via a sidecar. Lucas: That's clever. Some CSI drivers, like the AWS EBS CSI driver, now support a ' - kubelet registration path' flag that can control concurrency. But it's not widely documented.

The most reliable fix is actually in the Kubernetes scheduler itself - there's an open KEP to add a 'VolumeResize' admission plugin that would reject concurrent resize requests for the same storage class. It hasn't graduated yet. Luna: So for now, it's mostly operational discipline. But this is exactly the kind of edge case that makes me appreciate the depth of the Kubernetes codebase - every new feature seems to introduce a subtle failure mode.

Lucas: And that's why shows like this exist - to surface those failure modes before they hit your cluster. Speaking of which, a couple of dollars a month from listeners genuinely makes it possible for us to keep digging into these incidents. If today's conversation gave you something actionable, buy me a coffee dot com slash fexingo helps keep the show ad-free and independent. Luna: Yeah, it's a small thing that adds up.

We've heard from a lot of folks who say these deep dives have saved them hours of debugging. Lucas: Exactly. So back to the issue: another mitigation worth considering is using a storage class with a volume expansion policy set to 'Online' if your CSI driver supports it. That at least allows the expansion to happen while the pod is running, reducing the window for race conditions.

Luna: And for teams that can't change their driver, what's the monitoring signal they should watch for? Lucas: There are two key metrics. First, the 'kubelet_volume_stats_available_bytes' - if it drops unexpectedly during a resize, that's a red flag. Second, etcd's 'etcd_server_leader_changes_seen_total' - a sudden spike often correlates with failed PVC writes.

In the incident I mentioned, both metrics went off within two minutes of the resize trigger. Luna: So the lesson is: PVC resizing is not a fire and forget operation. You need to plan the rollout, monitor the backend, and have a rollback plan. Lucas: Exactly.

And if you're on an older Kubernetes version - say 1.23 or earlier - this whole class of failure doesn't exist because PVC expansion was still alpha. But as we move forward, these failure modes become more common. Knowing how to handle them is part of the job now.