How Kubernetes PodDisruptionBudgets Cause Rollout Stalls

Lucas: You set up a PodDisruptionBudget to protect your service during node drains. Then your next rollout stalls for forty minutes and nobody can tell you why. Luna: Right, the thing that was supposed to keep you safe is now the thing that's blocking you. Lucas: Exactly.

And it's not a bug. It's a feature working exactly as designed - just not the way you expected. So today I want to walk through a specific scenario that I've seen trip up teams, especially around peak traffic events like Black Friday. Luna: Let me guess - the e-commerce platform with two replicas and a PDB that said maxUnavailable: 1.

Lucas: You already know this story, but let's walk through it for everyone else. So picture this: you've got a deployment running two replicas of your checkout service. It's Black Friday week, traffic is spiking, and you push a critical fix. Your PDB says maxUnavailable: 1 - meaning at most one pod can be unavailable during voluntary disruptions like node drains or rolling updates.

Luna: Sounds reasonable. You don't want both pods down at the same time. Lucas: Totally reasonable. Except what happens during a rolling update?

The deployment controller tries to bring up a new pod. It evicts an old pod. But now you have one old pod terminating, one new pod starting, and the PDB says you can only have one unavailable pod. The new pod isn't ready yet - maybe it's still pulling the image or running startup probes.

So the controller sees two unavailable pods - the terminating one and the not yet ready one - and the PDB blocks further evictions. Luna: So the rollout stalls because the new pod can't become ready fast enough, and the old pod is stuck terminating. Lucas: Exactly. The deployment gets stuck.

No progress for minutes or even hours. And the worst part is, from the outside, everything looks fine. The PDB is working - it's preventing more than one unavailable pod. But the rollout is frozen.

I've seen this happen in production during a flash sale, and the team spent an hour debugging before someone noticed the PDB was the culprit. Luna: And the fix? Or rather, the better configuration? Lucas: There are a few approaches.

One is to use minAvailable instead of maxUnavailable. In this case, minAvailable: 1 means at least one pod must be available at all times. With two replicas, that still allows one pod to be unavailable - but the semantics are slightly different. The controller interprets it more conservatively during rollouts.

Another approach is to increase the replica count. If you have three replicas and maxUnavailable: 1, then during a rolling update you can have one terminating, one starting, and one still serving - the PDB is satisfied because only one is unavailable. Luna: Right, so the minimum safe replica count for a rolling update with maxUnavailable: 1 is actually three, not two. Lucas: That's the key takeaway.

And if you absolutely can't run three replicas due to resource constraints, you can tune the pod's readiness probe to be faster, or use a startup probe to give the pod more time to initialize without being considered unavailable. But the real fix is understanding that PDBs interact with the deployment controller in ways that aren't obvious. Luna: And this is just one scenario. What about node drains?

That's another common case. Lucas: Node drains are interesting because they use eviction API directly. When you drain a node, kubelet evicts pods one by one, respecting PDBs. If you have a PDB that says maxUnavailable: 1, and you try to drain a node that hosts the only two replicas of a service, the drain will get stuck because evicting the first pod makes one unavailable, and then evicting the second would make two unavailable.

So the drain waits forever - or until the PDB is deleted or modified. Luna: So you need to either scale up before the drain, or use a higher maxUnavailable, or ensure pods are spread across nodes. Lucas: Exactly. And pod anti-affinity helps with that.

If you spread pods across nodes, draining one node only affects one replica. But if you have two replicas and two nodes, draining one node still leaves one replica - so maxUnavailable: 1 works fine for drains in that case. The problem is specifically when multiple replicas land on the same node, or when the replica count is too low. Luna: So the lesson is: PDBs are not a set-and-forget configuration.

You have to think about how they interact with your rollout strategy, your replica count, and your pod scheduling. Lucas: Exactly. And this is where monitoring comes in. You should have alerts for stalled rollouts - something that watches the deployment's progress and fires if it doesn't complete within a reasonable time.

Also, you can expose PDB metrics via the kube state metrics. There's a metric called kube_poddisruptionbudget_status_current_healthy that tells you how many pods are healthy. If that drops below your desired threshold and stays there, you know something's wrong. Luna: And don't forget the PDB conditions.

Since Kubernetes 1.26, the PDB object has a status field that shows if the PDB is healthy or not. You can use that in your alerting. Lucas: Good point.

The DisruptionAllowed condition tells you whether the PDB currently allows evictions. If it's false, you know you're blocked. So you can set up an alert on that. But the tricky part is that during a rolling update, the PDB might temporarily be in a state where evictions are not allowed, and that's normal - but if it persists, it's a problem.

Luna: So what's your recommended setup for a production service that needs high availability during rollouts and node drains? Lucas: I'd say: run at least three replicas, use pod anti-affinity to spread them across nodes, and set your PDB to maxUnavailable: 1. That gives you one pod down during updates or drains while still having two serving traffic. If you need more resilience, go with five replicas and maxUnavailable: 2.

But always test your rollout with the PDB in place in a staging environment - because the behavior is subtle and you don't want to discover it during a real incident. Luna: And if you're stuck with two replicas due to cost or resource limits? Lucas: Then use minAvailable: 1 instead of maxUnavailable. That's a bit safer for rollouts, though it can still have issues during node drains if both pods are on the same node.

Better to use a startup probe that's long enough to let the new pod initialize without being marked unavailable. Also consider using a rolling update with a maxSurge of 1, so you can bring up a new pod before terminating an old one. That way you always have two pods running during the update - the old one and the new one. Luna: Right, maxSurge allows you to temporarily have more than the desired replicas, which gives you headroom.

Lucas: Exactly. And that's actually the recommended approach for critical services. Use maxSurge: 1 and maxUnavailable: 0. That ensures during a rolling update, you always have at least the desired number of pods serving traffic.

The trade-off is you need enough cluster capacity to schedule the extra pod temporarily. Luna: So the PDB itself isn't the enemy - it's the combination of configuration choices that creates the stall. Lucas: Right. And this is a pattern I see a lot: teams adopt PDBs because they're a best practice, but they don't think through the edge cases.

The result is a system that's more fragile, not less. So the takeaway for today is: know your replica count, know your PDB semantics, and always test the interaction with rolling updates and node drains before you rely on it in production. Luna: If today's tech conversation gave you something usable - maybe saved you from a future rollout stall - and you'd like to support the show staying ad-free, you can find us at buy me a coffee dot com slash fexingo. Lucas: Appreciate that.

And as we head into the summer conference season, I think we should also talk about how PDBs interact with cluster autoscaler - because that's another common pain point. Luna: Yeah, that's a good follow-up. When the autoscaler tries to scale down a node and the PDB blocks eviction, the node stays up and you waste resources. Lucas: Exactly.

So that's a perfect topic for a future episode. For now, the key thing to remember: test your PDB configuration with your actual rollout and drain workflows. Don't assume it just works. And if you find yourself stuck in a rollout stall, check your PDB status first - it might save you an hour of debugging.

Luna: Great advice. Thanks, Lucas. Lucas: Thanks, Luna. See you next time.