How Kubernetes Audit Logging Causes etcd Performance Degradation

Lucas: So you enable audit logging on your Kubernetes cluster - standard best practice, right? Security teams love it, compliance requires it, and then suddenly your API server response times go from five milliseconds to two hundred. Your etcd starts throwing leader election timeouts. Nobody changed any workloads.

Luna: Wait - audit logging did that? I've seen clusters with audit enabled and they were fine. What's different? Lucas: It's not about having it on or off - it's about how much you log and where you send it.

By default, the audit policy logs every request at the Metadata level, which is relatively lightweight. But a lot of teams, especially ones coming from compliance-heavy environments, crank it up to Request or RequestResponse level. And that's where the trouble starts. Luna: So each API request generates multiple audit events, and those get written to etcd?

I thought audit logs went to a file or stdout. Lucas: They can. The default backend writes to a local file on the API server node. But the more common setup in managed or multi-node clusters is the webhook backend - you send audit events to an external service, like an Elasticsearch cluster or a Splunk instance.

The problem is, when that webhook backend is slow or unavailable, the API server blocks and queues events. And if the queue fills up, it starts dropping requests. Luna: Right, I've seen that - the audit webhook becomes a bottleneck. But you mentioned etcd specifically.

How does audit logging affect the underlying key-value store? Lucas: Here's the chain. Every Kubernetes object - pods, services, configmaps - is stored in etcd. When you create or update an object, the API server writes to etcd.

That's one write. But with audit logging at RequestResponse level, the API server also has to serialize the entire request and response body into an audit event and write that event to the audit backend. If that backend is a file, it's fine - local disk is fast. But if it's a webhook, the API server makes an HTTP call to the audit backend.

That call can fail, timeout, or just be slow. Luna: And if the webhook is slow, the API server holds up the main request, which means etcd writes are delayed. So etcd sees increased latency and starts timing out on heartbeats. Lucas: Exactly.

I worked with a fintech team last year - they had a three-node etcd cluster, fairly standard. After they enabled audit logging with the webhook backend pointed to their existing Elasticsearch cluster, they saw p99 API response times go from 8 milliseconds to 450 milliseconds. Their etcd leader was re-electing every few minutes. The root cause?

The Elasticsearch cluster was under heavy load from application logs, and the audit webhook requests were queuing up. The API server's default audit webhook batch size is 10 events, with a maximum wait of 10 seconds. So under load, the API server would batch events, but the webhook call would take 12 seconds to complete, causing backpressure. Luna: So the fix was to either scale the Elasticsearch cluster or reduce audit verbosity?

Lucas: Both. But first, they dropped the audit policy from RequestResponse to Metadata for most resources. That alone cut the audit event size by roughly 90 percent. Then they tuned the webhook batching parameters - increased the batch size from 10 to 100, and reduced the maximum wait from 10 seconds to 5 seconds.

That way, the API server sends fewer, larger batches, and the webhook calls complete faster. Luna: Interesting. So the lesson isn't 'don't audit' - it's 'know your audit policy and your backend capacity'. Lucas: Right.

And there's another layer. Even with a local file backend, you can still hurt etcd if you're not careful. Because the API server writes audit logs synchronously by default. That means every API request blocks on the file write.

If the disk is slow - say, a shared NFS volume - you get the same latency spikes. The fix there is to use a buffered logging library or write to a local SSD, not network storage. Luna: What about the new audit log dynamic configuration feature in recent Kubernetes versions? Does that help?

Lucas: It does. Since Kubernetes 1.27, you can change the audit policy without restarting the API server. That's huge for incident response - you can dial down verbosity in seconds.

But it's still experimental in some distributions. The team I mentioned was on 1.25 at the time, so they had to restart the API server to change the policy. That meant a brief blip in availability.

Luna: Let's talk about the actual audit policy structure. What are the key knobs? Lucas: There are three levels: None, Metadata, Request, and RequestResponse. None means no audit events for that resource.

Metadata logs the user, timestamp, resource type, and verb - but not the request body. That's usually enough for compliance. Request logs the request body but not the response. RequestResponse logs both.

The size difference is dramatic: a typical pod create event at Metadata level might be 500 bytes. At RequestResponse, it could be 5 kilobytes or more, depending on the object size. Luna: So for high-volume resources like pods or events, you'd want Metadata only. What about sensitive resources like Secrets?

Lucas: You can set a separate rule for Secrets - log at RequestResponse but omit the request body for non-secret resources. The policy supports per-resource overrides. The common pattern is: log Metadata for all resources, then RequestResponse only for a few high-value ones like RBAC changes or Secret mutations. And you can use OmitStages to skip logging during certain phases, like when the controller manager updates status.

Luna: That reminds me - we should talk about the audit log sidecar pattern. A lot of teams run a sidecar container in the API server pod to handle audit log shipping, so the API server itself doesn't block on the webhook call. Lucas: Yes, that's a solid pattern. Instead of the API server sending audit events directly to a webhook, it writes to a local file, and a sidecar like Fluentd or Filebeat tails that file and ships it to the backend.

That decouples the API server from the log pipeline. The API server never waits for the network call. The sidecar handles retries and buffering independently. Luna: But doesn't that introduce a risk of losing logs if the sidecar crashes?

Lucas: It does. But you can mitigate with a persistent volume for the audit log directory, and the API server's default audit log backend rotates files anyway. So even if the sidecar is down for a few minutes, the logs accumulate and get shipped when it comes back. It's a trade-off: lower latency for the API server versus potential log loss.

In most production environments, the latency improvement is worth the small risk. Luna: So what's the one thing every Kubernetes operator should check tomorrow? Lucas: Their audit policy. If it's set to RequestResponse for all resources, change it to Metadata.

Then measure your API server response times before and after. I've seen p99 drop by 80 percent just by reducing verbosity. Also, check if your audit backend - whether it's a file or webhook - has enough capacity. And if you're using a webhook, enable batching and tune the batch parameters.

That alone could save you from a cluster-wide incident. Luna: And if you found this episode useful, you know what keeps these deep dives ad-free - listener support. If today's tech conversation gave you something usable, head over to buy me a coffee dot com slash fexingo. It genuinely makes a difference.

Lucas: Yeah, absolutely. No pressure, but if you've ever wondered how to support the show, that's the way. And it means we can keep going into these nitty-gritty topics without any sponsor interruptions. Luna: Exactly.

So back to audit logging - one more thing: what about the etcd compaction and defragmentation? Does audit logging affect that? Lucas: Indirectly. If audit logging causes the API server to write more objects to etcd - for example, if you're logging events as Kubernetes Event objects - then etcd's history grows faster, and compaction becomes more important.

But the main impact is on the API server's ability to serve requests quickly. A slow API server means longer etcd transaction times, which can cause the etcd leader to appear unresponsive to followers. That triggers elections. Luna: So the takeaway: audit logging is essential, but treat it like any other system - monitor its performance, tune its parameters, and plan for capacity.

Lucas: Exactly. And if you're running a cluster with thousands of nodes, audit logging at RequestResponse level is almost certainly going to cause issues. Start lean, add verbosity only where you need it, and always test under load before rolling out.