How Kubernetes Custom Resource Definitions Cause Controller Memory Leaks

Lucas: So you deploy a custom resource definition, write a controller to reconcile it, and everything runs fine for days. Then, somewhere between day twelve and day fifteen, the controller gets oom killed. Luna: Classic Kubernetes memory leak pattern - but the controller isn't doing anything obviously wrong. The code looks clean.

Lucas: Right. And the root cause is almost never in the controller logic itself. It's in the client-go informer cache and how it interacts with custom resources under churn. Luna: Can you unpack that?

The informer cache is supposed to be a thread-safe store. What's leaking? Lucas: The cache itself doesn't grow unbounded - it's keyed by namespace and name, so in theory it's bounded by the number of custom resource instances. The problem is what the controller does with the objects it gets from the cache.

Luna: So it's not the informer, it's the controller holding references. Lucas: Exactly. The typical pattern: a controller's reconciliation loop receives an object from the informer, processes it, and then - often implicitly - stores a pointer to that object in a local map or slice for later use. Luna: And because the informer cache shares the same underlying object pointer, the controller is effectively pinning that memory, preventing the garbage collector from freeing it.

Lucas: Yes. The shared informer returns a pointer to the object in its cache. If you store that pointer in your own data structure, the cache can never release the memory for that key-value pair. Luna: Even after the custom resource is deleted from the cluster?

Lucas: The informer cache will evict the entry when it receives a delete event. But your controller's local map still holds a reference to the old object, so the GC can't collect it. That's a leak. Luna: So the controller accumulates stale objects over time.

How bad can that get? Lucas: I worked with a fintech that ran a controller managing about fifty thousand custom resources representing payment transactions. Transactions are created and deleted constantly - high churn. After two weeks, the controller's heap was three gigabytes, and it got oom killed.

Luna: Three gigs for fifty thousand objects? That's about sixty kilobytes per object. For a CRD that probably has a handful of fields, that's a lot. Lucas: It's because the controller wasn't just holding the latest state.

It was retaining every version of each object it had ever seen, because it was caching reconciliation results by object UID, not by name. Luna: Ah, the UID changes on each update - even a spec change generates a new UID if the resource is recreated. So the controller accumulated entries for every version. Lucas: Exactly.

The fix was simple: key the local cache by namespace and name, not UID, and explicitly deep-copy objects before storing them. But nobody thinks to do that until they see the memory graph. Luna: Let's talk about detection. If you're not profiling the controller, how do you spot this pattern?

Lucas: First sign: the controller's memory usage grows linearly over time, with no plateau. Second: after a delete event for an old resource, memory doesn't drop. Third: the controller's reconciliation latency increases because it's iterating over an ever-growing local cache. Luna: And you can confirm by taking a heap dump and looking for objects of the custom resource's Go type that are referenced from your controller's internal maps.

Lucas: Right. Use `go tool pprof` with the controller's profiling endpoint. If you see hundreds of thousands of instances of your CRD struct that are still reachable from the main controller struct, you've found the leak. Luna: What about the informer cache itself?

Could that ever leak? Lucas: It can, in a different way. If the controller registers a custom event handler that does not properly filter or if it watches custom resources across all namespaces without setting a label selector, the informer cache will hold every instance of that CRD in the cluster. Luna: But that's not a leak, that's just a large cache.

It's bounded by the number of CRD instances. Lucas: True. But if you have a cluster with hundreds of thousands of custom resources, that cache alone could be gigabytes. And if the controller doesn't need all of them, you're wasting memory.

Luna: So the lesson is: always use field selectors or label selectors to scope the informer's watch, and always deep-copy objects before storing them in your own data structures. Lucas: I'd add one more: set a memory limit on the controller's container, and monitor the `container_memory_working_set_bytes` metric in Prometheus. If it trends upward without bound, you have a leak. Luna: And if you're using an operator framework like the Operator SDK, does it protect against this?

Lucas: Partially. The controller-runtime library that the Operator SDK uses does some caching internally, but it doesn't prevent the controller from storing pointers to informer objects. The developer still has to deep-copy. Luna: So it's a discipline issue.

Let's talk about a real-world case - the fintech you mentioned. How did they debug it? Lucas: They noticed that the controller's memory usage grew from 200 megabytes to 3 gigabytes over two weeks. They added profiling endpoints, took a heap dump, and saw that 90% of the heap was occupied by their CRD struct.

Luna: And the struct was mostly strings and timestamps, so the memory was from the sheer number of objects, not from large fields. Lucas: Right. They had stored a reference to the object in a map keyed by UID for deduplication. They changed the key to namespace/name and added a deep copy.

Memory dropped to 400 megabytes and stabilised. Luna: That's a ten-x improvement from one code change. I'm surprised this isn't more widely documented in best practices guides. Lucas: It's one of those things that seems obvious in hindsight but is easy to miss when you're focused on business logic.

The Kubernetes documentation does mention that informer objects should not be modified, but it doesn't explicitly say 'deep-copy before storing'. Luna: And if you modify the object in place, you corrupt the informer cache for other controllers watching the same resource. Lucas: Exactly. That's an even worse bug - silent data corruption.

But the memory leak is more common because many developers assume the informer cache is immutable and don't even think about storing references. Luna: So what's the recommended pattern? Should every controller deep-copy the object at the start of the reconciliation loop? Lucas: Yes.

Use `obj.DeepCopy` or, if you're using client-go's `Object` interface, use `runtime.DeepCopyObject`. Then work with the copy.

Never store the original pointer. Luna: Does deep-copying add significant overhead? Lucas: It adds CPU and memory allocation, but it's usually negligible compared to the cost of a leak. In the fintech case, the deep copy added about 5% to reconciliation time, but prevented unbounded growth.

Luna: That's a good trade-off. What about controllers that need to cache state across reconciliations? Like an operator that maintains a list of all resources it has processed. Lucas: Then you definitely need a separate cache with deep-copied objects.

And you should periodically prune that cache - remove entries for resources that no longer exist in the cluster. A common approach is to use a finalizer to ensure cleanup. Luna: But finalizers can cause their own issues, as we covered in episode 67. Lucas: Fair point.

But for this use case, a finalizer on the custom resource ensures that when the resource is deleted, the controller's cache entry is cleaned up. Without it, the cache entry becomes a zombie. Luna: Let me ask a broader question: are there any tools that can detect this pattern automatically? Lucas: Not specifically for this leak pattern.

But you can write a custom Prometheus rule that alerts if the controller's memory grows by more than, say, 10% per day over a week. That would catch it. Luna: Or use the Go race detector? No, that's for concurrency bugs.

Lucas: Right. The memory leak is a logic bug, not a race. The best detection is monitoring and profiling. I actually wish the Kubernetes controller-runtime project would include a built-in memory profiler endpoint by default.

Luna: It does include a metrics endpoint, but not a pprof endpoint. You have to enable it yourself. Lucas: Which is a one-liner: `import _ "net/http/pprof"` and expose an HTTP server. But many production controllers don't have it enabled because it's considered a security risk.

Luna: You can expose it on a separate, internal port. That's what we do at my company. Lucas: Good practice. Let's wrap up with a practical checklist.

When writing a controller for a custom resource: one, always deep-copy the object before storing it. Two, key your internal caches by namespace and name, not UID. Three, set a memory limit and monitor it. Four, enable profiling endpoints on an internal port.

Luna: And five, if you're using finalizers, ensure your cache cleanup runs even if the controller crashes and restarts. Lucas: Absolutely. The pattern is subtle, but once you've seen it, you'll never unsee it. I've had three separate debugging sessions over the past year where the root cause was exactly this.

Luna: I think it's worth doing a follow-up episode on how to properly design operator caches. Lucas: That's a good idea. We could talk about using work queues, rate limiting, and how to avoid storing state altogether by using the API server as the source of truth. Luna: Yeah, the less state you keep in the controller, the fewer leaks you can have.

Lucas: I think we've covered the core issue. If you're running a controller for a CRD, go check your code for stored pointers. It might save you a 3 a.m.

pager call.