How Kubernetes Vertical Pod Autoscaler Misallocates Memory

Lucas: So you set up the Kubernetes Vertical Pod Autoscaler - VPA - because you want it to auto-tune your pod resource requests. You think, great, no more manually guessing CPU and memory. But then you check the recommendations after a week, and something feels off. Memory requests are way higher than actual usage, and CPU requests are still too low.

What's going on? Luna: I've seen that exact pattern. It's like VPA is overcorrecting for memory but not keeping up with CPU. Is it a configuration issue or a fundamental design problem?

Lucas: A bit of both. Let's start with how VPA's recommender works. It collects historical CPU and memory usage from the Metrics Server - typically over an eight-day sliding window by default. Then it calculates a target recommendation based on a percentile of that usage.

For memory, the default is the 95th percentile. For CPU, it's also the 95th percentile, but CPU is bursty and gets smoothed differently. The thing is, memory usage patterns in many applications - especially those with garbage-collected runtimes like Java or Python - have a lot of short-lived spikes that inflate that 95th percentile. Luna: Right, so if your Java app has a full GC event once every few hours that pushes heap usage to 2GB for a second, VPA sees that as the 95th percentile and recommends 2GB as the request.

But the actual steady-state usage is more like 1.2GB. Lucas: Exactly. And the problem compounds.

Let's take a real case - I worked with a team running an e-commerce platform on a cluster with about 200 pods. They had VPA enabled on all stateless workloads. After a month, they noticed memory requests were on average 42% higher than actual peak memory usage across the fleet. Meanwhile, CPU recommendations were consistently about 30% lower than what their Horizontal Pod Autoscaler needed to keep latency under control.

The result: they were paying for a ton of unused memory reservation, and their pods were getting CPU throttled under load. Luna: So VPA was basically wasting memory capacity and starving CPU. What was the root cause for the CPU side? Is it the same percentile issue?

Lucas: Partially, but CPU has a different dynamic. The recommender uses a sliding window that, by default, looks at usage over the past eight days. But CPU usage tends to be spiky in short bursts - think request handling that lasts milliseconds. The Metrics Server scrapes every 15 seconds, so it captures a point-in-time sample.

If your app has a traffic pattern where peak CPU happens for 100 milliseconds every minute, the 15-second scrape interval might miss most of those spikes. The result is that the 95th percentile of sampled CPU usage is much lower than the actual peak demand. VPA then recommends a CPU request that's too low. Luna: So the VPA is blind to micro-bursts.

That explains why some teams see CPU throttling despite VPA being in place. Lucas: Right. And there's another subtlety: VPA's memory target percentile is configurable - you can set `targetMemoryPercentile` in the VPA configuration. But the default is 95, and most teams don't change it.

For CPU, there's no equivalent configurable percentile; it's hardcoded to use the same logic. That makes it harder to tune. Luna: So what can you do? Lower the memory percentile to something like 90 or even 85 to avoid over-provisioning?

And for CPU, maybe you should rely more on HPA and just use VPA for memory? Lucas: That's one pattern - run HPA based on CPU or custom metrics, and let VPA only adjust memory requests. But if you want VPA for both, you can tweak the memory percentile. For the e-commerce team I mentioned, they dropped `targetMemoryPercentile` from 95 to 90.

That alone cut memory waste by about 25%. They also changed the CPU recommendation policy to use a lower percentile - essentially by adjusting the VPA recommender's `cpuHistogramBucketSize` and the `memoryAggregationInterval` to capture smaller spikes. But those are advanced knobs and not well documented. Luna: And the other approach: use a sidecar that exposes real-time RSS or working set size metrics, then feed those into VPA via custom metrics?

Or is that overkill? Lucas: It's not overkill for critical workloads. Some teams run a sidecar that collects /proc/self/status or uses cAdvisor endpoints to get actual resident set size every few seconds, then pushes that to a custom metrics API. Then you can configure VPA to use that custom metric instead of the default memory usage from Metrics Server.

That gives you much finer granularity and avoids the 15-second scrape lag. But it adds complexity. Luna: Complexity vs. waste - it's a trade-off.

But for a cluster with hundreds of pods, the cost of over-provisioned memory can be significant. I've seen cases where teams were reserving 50% more memory than needed, which translates to extra node costs. Lucas: Exactly. And the other thing: VPA's recommendations are only as good as the historical data.

If you deploy a new version of your app that uses less memory, VPA will take days to lower the recommendation because it keeps the old data in the sliding window. You can force a reset by deleting the VPA object and recreating it, but that's a manual intervention. Luna: So VPA works best for stable workloads with consistent patterns. For quickly changing apps, it might actually hurt.

Lucas: Right. And there's one more nuance I want to highlight: memory fragmentation in languages like Python or Node.js. The os level RSS can be significantly higher than the actual heap usage because of memory fragmentation or unreclaimed pages.

VPA sees RSS, not heap, so it might over-allocate memory due to fragmentation spikes. That's another case where a sidecar exposing heap metrics helps. Luna: So the takeaway is: understand what VPA actually measures, change the memory percentile, and consider supplementing with custom metrics. Otherwise you're paying for air.

Lucas: Exactly. Let's talk about a concrete fix: in your VPA spec, you can set `targetMemoryPercentile: 85` instead of the default 95. That alone can reduce waste by 15-20% without risking OOM kills in most cases - because the VPA also sets a limit based on the max observed, so you're still protected. Of course, test it first.

Luna: And for CPU, you might want to use HPA with target utilization of 70-80% and let VPA only handle memory. Or if you must use VPA for CPU, consider reducing the `cpuHistogramBucketSize` to capture more granular spikes. I think that parameter defaults to 0.1 CPU cores - you could try 0.

05. Lucas: Yeah, that's a good tip. But be careful - smaller buckets increase memory usage for the recommender. It's a trade-off.

Luna: You know, this kind of deep dive reminds me why I love this show - we get to untangle stuff like VPA internals that most docs gloss over. And honestly, it's only possible because of listeners who support us directly. Lucas: Absolutely. We keep this ad-free, and a small group of listeners chip in monthly through buy me a coffee dot com slash fexingo.

That's what lets us spend time on episodes like this rather than chasing sponsors. Luna: It's a small gesture that makes a big difference for us. And we're grateful - it keeps the show exactly what we want it to be. Lucas: Alright, back to VPA.

So the other thing to check: if you're using VPA in Auto mode, it will actually update your pods' resource requests - and evict them if needed. That can cause disruptions if the recommender oscillates. We've seen cases where VPA flips between two memory recommendations after a deploy, causing constant pod restarts. Luna: That sounds like a nightmare.

How do you stabilize it? Lucas: One approach: use VPA in Initial mode - it only sets requests at pod creation, never updates running pods. Then you can roll out new deploys to apply new recommendations. That gives you control over when changes happen.

Or you can set a large `updateStategy` minReplicas to avoid too many simultaneous evictions. Luna: So the key is: don't trust the defaults. Tune the percentiles, watch the metrics, and decide if VPA is even the right tool for your workload. Lucas: Right.

For batch jobs or cron jobs, VPA doesn't help because it needs historical data. For stateful workloads with persistent volumes, VPA can't resize the pod without restarting it, which might not be acceptable. So know where VPA fits. Luna: I think we've given listeners enough to start debugging their own VPA setups.

Check your memory request vs. actual usage ratio - if it's over 1.3, you're probably over-provisioned. Lucas: That's a good heuristic.

And if you're curious about the exact percentile, you can query the VPA recommendation status - it shows the lower bound, target, and upper bound. Compare the target to your actual 95th percentile from Prometheus. If they're mismatched, you know where to dig. Luna: Great episode.

Let's do a follow-up on VPA with custom metrics soon - I think there's more to explore there. Lucas: Agreed. Next time, we'll look at how to build a custom metrics pipeline for VPA that actually captures micro-bursts. For now, check your memory percentile and save some cluster capacity.