DevOps Daily with Fexingo · 2026-06-29 · 8 min
Key moments - from our scoring
Substance score
82 / 100
Five dimensions, 20 points each
ResourceQuotas are a standard Kubernetes multi-tenancy feature, but when misconfigured or poorly monitored, they become invisible killers. This episode walks through a real production incident where an e-commerce team's checkout service pods were evicted during a flash sale because the namespace's ResourceQuota was exhausted. The cluster showed 60% overall CPU utilization, but the checkout namespace hit its hard limit of 4 CPU cores and 8GB memory. When the Horizontal Pod Autoscaler tried to scale up, the kube-apiserver rejected pod creation at the API server level - before the scheduler even saw the request. The kubelet then evicted existing pods randomly to bring usage below the quota, killing critical checkout service pods. The team spent four hours correlating the failure to kube-apiserver audit logs. Solutions discussed include monitoring quota utilization with Prometheus metrics (kube_resourcequota), setting alerts at 80% usage, implementing LimitRanges for pod-level defaults, building custom admission webhooks for better error messages, and treating quotas as a safety net rather than a capacity planning tool. The episode also covers the interaction between ResourceQuotas and cluster autoscaler - if quotas block pod creation, the autoscaler never sees unschedulable pods and won't add nodes.
When a namespace hits its ResourceQuota hard limit, the kube-apiserver rejects new pod creation at the API level. The ResourceQuota controller then instructs the kubelet to evict existing pods to bring usage below the limit, with evictions chosen randomly if all pods have the same priority.
Enable audit logging on the kube-apiserver to capture 'forbidden' quota exceeded messages, monitor the kube_resourcequota Prometheus metric per namespace, build custom admission webhooks for clearer error messages, and create Grafana dashboards showing quota utilization relative to hard limits.
The cluster autoscaler only triggers when pods are unschedulable. If a ResourceQuota blocks pod creation before the pod is created, the pod never enters an unschedulable state, so the autoscaler never sees a reason to add capacity.
LimitRanges set default resource requests/limits for individual pods within a namespace, while ResourceQuotas set hard aggregate limits for the entire namespace - used together, they prevent runaway consumption and provide pod-level enforcement below the namespace ceiling.
Set quotas generously at double the expected peak usage per namespace, monitor utilization continuously, and test quota limits during load tests rather than relying on single baseline load tests that may not reflect actual service growth.
Our reviewer’s read on each dimension, with quotes from the episode.
The episode delivers a densely packed exploration of a specific, non-obvious Kubernetes failure mode with concrete mechanics, unexpected interactions, and actionable mitigations. Nearly every exchange introduces a new layer of complexity (API server rejection vs. scheduler events, quota interactions with cluster autoscaler, namespace vs. cluster-level quotas, ephemeral storage pitfalls), with minimal filler or obvious advice.
The only clue was in the kube-apiserver audit logs - a flood of 'quota exceeded' messages mixed in with normal traffic. It took them four hours to correlate.
Because ResourceQuotas are enforced at the API server level, the scheduler never even sees the request. It's a rejection before scheduling. That's why no scheduler events showed up - no 'FailedScheduling' warnings, no node-pressure evictions. Just silent denial.
The framing of ResourceQuotas as a silent failure mode is fresh and contrarian - counterintuitive enough that operators with basic Kubernetes knowledge wouldn't have encountered this specific chain of causation. The interaction between quotas and cluster autoscaling, and the absence of visible scheduler warnings, represent first-principles thinking about Kubernetes' API server architecture that goes beyond recycled DevOps wisdom.
Existing pods were then evicted to make room for the new ones.
The scheduler never even sees the request. It's a rejection before scheduling.
Lucas presents as a practitioner with direct experience triaging this issue at scale, drawing on a real e-commerce team's production incident with quantified impact ($200k revenue loss, 50-node cluster, 200 microservices). While the transcript doesn't establish his seniority explicitly, the depth of debugging knowledge (audit logs, kubelet eviction logic, cluster autoscaler interactions) and the specific case study suggest hands-on operational experience rather than purely theoretical knowledge.
I picked up this story from an e-commerce team that ran into it during a flash sale last quarter.
They estimated around $200,000 in missed transactions. The root cause was a ResourceQuota set months earlier by an engineer who'd guessed the namespace's needs based on a single load test.
The episode is anchored in a specific, named case study with concrete numbers: 50 nodes, 200 microservices, 30 namespaces, 4 CPU cores / 8 GB quota, five replica request, $200k revenue loss, four-hour debug time, 60% cluster utilization. Metrics like 80% quota alert thresholds, weekly reports, and Monday triage cadence are provided. The only significant absence is the company name itself, but the operational details are granular enough to be highly credible and actionable.
They had a 50-node cluster, running about 200 microservices across 30 namespaces.
The checkout namespace had a ResourceQuota set to 4 CPU cores and 8 gigs of memory. During the flash sale, the deployment tried to scale up - the Horizontal Pod Autoscaler requested five more replicas.
Luna asks sharp clarifying questions that expose the non-intuitive mechanics ('Wait - Kubernetes evicts running pods because a new pod can't be scheduled? That feels backward.') and pushes Lucas to explain root causes and cascading failures rather than accepting surface-level answers. The back-and-forth reveals multiple layers of the problem systematically. However, there are moments where Luna could probe deeper on implementation details (e.g., how the webhook error message format was chosen, trade-offs in audit logging overhead), and a few promotional asides slightly interrupt momentum.
So what was triggering the evictions?
Wait - Kubernetes evicts running pods because a new pod can't be scheduled? That feels backward.
Computed from the transcript - who did the talking, and the words that came up most.
In this episode of DevOps Daily with Fexingo, Lucas and Luna dive into a subtle but destructive Kubernetes behavior: how ResourceQuotas can silently evict pods when namespace limits are reached, even when the cluster has ample capacity. They walk through a real incident at a mid-sized e-commerce company where a single namespace's quota misconfiguration caused cascading evictions across 12 microservices during a flash sale. Lucas explains the mechanism - how the kube-apiserver rejects pod creations when quota is exceeded, and how the scheduler's lack of awareness leads to orphaned pods. Luna raises the issue of observability gaps, noting that standard dashboards often miss quota-related denials. They discuss mitigation strategies: setting explicit deny messages, monitoring quota metrics via Prometheus, and using admission webhooks for early warnings. The episode delivers a concrete lesson for any team running multi-tenant clusters. A brief donation segment highlights listener support for the ad-free show.
Transcribed and scored by The B2B Podcast Index.
Lucas: Let's talk about a Kubernetes failure mode that's surprisingly easy to miss - ResourceQuotas causing silent pod evictions. I picked up this story from an e-commerce team that ran into it during a flash sale last quarter. Luna: The kind where you think the cluster has plenty of room, but pods just vanish? Lucas: Exactly.
They had a 50-node cluster, running about 200 microservices across 30 namespaces. The flash sale started, traffic spiked, and suddenly the checkout service started throwing errors. Pods were being terminated, but not by the scheduler or a node failure. Luna: So what was triggering the evictions?
Lucas: The checkout namespace had a ResourceQuota set to 4 CPU cores and 8 gigs of memory. During the flash sale, the deployment tried to scale up - the Horizontal Pod Autoscaler requested five more replicas. The kube-apiserver rejected the pod creation because the namespace would exceed its quota. But here's the kicker: existing pods were then evicted to make room for the new ones.
Luna: Wait - Kubernetes evicts running pods because a new pod can't be scheduled? That feels backward. Lucas: It is backward, but it's how the system works when you hit the quota hard limit. The ResourceQuota controller marks the namespace as 'terminating' and tells the kubelet to evict pods to bring usage below the limit.
The kubelet picks pods to evict based on priority, but if all pods are the same priority, it's random. So critical checkout pods got killed. Luna: And the team must have been looking at cluster-level metrics showing 60 percent CPU usage, thinking everything was fine. Lucas: That's exactly what happened.
Their Grafana dashboard showed overall cluster utilization at 60 percent. They had no namespace-level quota monitoring. The only clue was in the kube-apiserver audit logs - a flood of 'quota exceeded' messages mixed in with normal traffic. It took them four hours to correlate.
Luna: Four hours during a flash sale. That's a lot of lost revenue. Lucas: They estimated around $200,000 in missed transactions. The root cause was a ResourceQuota set months earlier by an engineer who'd guessed the namespace's needs based on a single load test.
He never updated it as the service grew. Luna: So the quota became a bottleneck without anyone realizing. Lucas: Right. And because ResourceQuotas are enforced at the API server level, the scheduler never even sees the request.
It's a rejection before scheduling. That's why no scheduler events showed up - no 'FailedScheduling' warnings, no node-pressure evictions. Just silent denial. Luna: I've seen teams set quotas as a best practice for multi-tenancy, but this shows the downside.
What's the mitigation? Lucas: First, don't rely solely on ResourceQuotas. Use LimitRanges to set default resource requests per pod, so you don't get runaway consumption. Second, monitor quota utilization with Prometheus - the 'kube_resourcequota' metric gives you used and hard limits per namespace.
Set alerts at 80 percent. Luna: And the 'quota exceeded' API server events - are those easy to capture? Lucas: They are if you enable audit logging and filter for 'forbidden' reasons. But that adds overhead.
A simpler approach is to use an admission webhook that gives you a clear error message when a pod is rejected due to quota. The default error message from the API server is vague - just 'exceeded quota' without telling you which resource. Luna: So you could build a webhook that says something like 'Pod rejected: namespace checkout is at 100 percent CPU quota'. Lucas: Exactly.
That team now has a custom webhook that returns structured errors. They also added a Grafana panel showing quota usage per namespace, right next to the cluster-level graphs. Luna: It's interesting - the problem isn't quotas themselves, it's the lack of visibility into when they're close to the limit. Lucas: Bingo.
Quotas are a useful guardrail, but they become a trap if you don't watch them. Another thing - the team realized they'd set quotas on CPU and memory but not on ephemeral storage or object counts. During the same flash sale, a logging sidecar filled up a pod's emptyDir volume, which also triggered evictions. Luna: So the quota saved them from one problem but exposed another.
Lucas: Right. They now set comprehensive quotas covering CPU, memory, storage, and even the number of pods and services. They test quota limits during load tests, not just baseline runs. Luna: That's a good lesson.
Before we go deeper - Lucas, you mentioned earlier that this kind of deep-dive episode is possible because of listener support. Want to say a quick word? Lucas: Yeah, sure. If today's tech conversation gave you something usable - we keep the show ad-free and listener-supported.
A handful of folks chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making these deep dives. Luna: It's a small group, but it makes a big difference. No pressure - just if you find value, it helps us keep going. Lucas: Alright - back to quotas.
One more angle: the team also discovered that their quota was set at the namespace level, but they had a few services that ran in a shared namespace. When one service's quota was exhausted, it affected unrelated services in the same namespace. Luna: That's a multi-tenancy issue. They should have separated critical services into dedicated namespaces.
Lucas: Exactly. They moved the checkout service to its own namespace with a higher quota. But then they hit the cluster-level quota - the total sum of all namespace quotas can't exceed the cluster's capacity. That's where ResourceQuota and LimitRange interact with cluster autoscaler.
Luna: So if you overcommit quotas, the cluster autoscaler won't add nodes because it sees the quota as the ceiling, not the actual usage. Lucas: Right. The cluster autoscaler only triggers when pods are unschedulable. If the quota blocks pod creation, the pods never exist to become unschedulable.
So the cluster stays at 50 nodes while the checkout service starves. Luna: That's a cascading failure pattern. What's the fix? Lucas: The fix is to think of quotas as a safety net, not a capacity planning tool.
Use the cluster autoscaler for capacity, and set quotas generously - say, double the expected peak usage - while using LimitRanges to enforce individual pod limits. That way, quota evictions only happen if something goes wildly wrong. Luna: And monitor the gap between quota limits and actual usage. Lucas: Exactly.
The team now has a weekly report that flags namespaces where quota utilization exceeds 80 percent for more than an hour. They review it in their Monday triage. Luna: So the key takeaway: ResourceQuotas aren't evil, but they're invisible until they break you. Lucas: And when they break, they break silently.
If you're running multi-tenant Kubernetes, spend an hour this week checking your quota utilization. It's one of those things that's easy to ignore until you're debugging a production incident at 2 AM. Luna: Good advice. I know I'll be checking ours after this conversation.
Lucas: That's the spirit. Next time, we'll talk about how PersistentVolumeClaims can cause similar silent failures when their storage classes aren't configured correctly. Luna: Looking forward to it. Until then, keep your quotas visible and your pods running.
How Kubernetes StatefulSet PVC Resizing Causes Node Disk Failures94 / 100How Kubernetes Topology Spread Constraints Create Scheduling Hotspots95 / 100How Kubernetes CRD Versioning Breaks Controller Upgrades90 / 100How Kubernetes Audit Logging Causes etcd Performance Degradation91 / 100How Kubernetes Volume Snapshots Cause Storage Backend Data Corruption85 / 100