DevOps Daily with Fexingo · 2026-07-21 · 10 min
In this episode of DevOps Daily, Lucas and Luna dive into a notoriously overlooked Kubernetes failure mode - API server overload caused by excessive watch events. They break down how controllers, informers, and custom operators all lean on watch-based architectures, and why a single misconfigured watch can cascade into cluster-wide latency or even outages. Through the lens of a real incident at a mid-sized fintech, they explain what happened when a poorly tuned deployment triggered millions of watch events per minute, flooded etcd, and brought the control plane to its knees. Lucas outlines the specific metrics to monitor (watch event rate, request latency, etcd db size) and the architectural patterns - like using DeltaFIFO, filtering with field selectors, and throttling informer resync intervals - that prevent these meltdowns. Luna brings her own war story about a CI/CD operator that hammered the API server during a Helm release, and the two discuss how to audit your cluster for watch-heavy workloads. By the end, listeners will know exactly how to diagnose and prevent one of the most common yet invisible causes of Kubernetes instability.
How Kubernetes ServiceAccount Token Expiration Breaks CI Workflows75 / 100How Kubernetes StatefulSet PVC Resizing Causes Node Disk Failures94 / 100How Kubernetes Topology Spread Constraints Create Scheduling Hotspots95 / 100How Kubernetes CRD Versioning Breaks Controller Upgrades90 / 100How Kubernetes Audit Logging Causes etcd Performance Degradation91 / 100