How Kubernetes ServiceAccount Token Expiration Breaks CI Workflows

Lucas: So you push a code change, your CI pipeline kicks off, it tries to deploy to your Kubernetes cluster - and it just sits there. The pod never comes up. No obvious error in the YAML. No memory or CPU limit hit.

What happened? Luna: I've seen that movie. The team spends an hour checking RBAC, then the namespace, then the image pull secret. But it's something more fundamental.

Lucas: Exactly. This is Episode 141 of DevOps Daily, and we're talking about the quietest breaking change in recent Kubernetes history: ServiceAccount token expiration. If you've upgraded your cluster to Kubernetes 1.24 or later - and by now, July 2026, most production clusters are at 1.

27 or higher - you've already got time-bound tokens by default. And if your CI/CD system or application relied on a long-lived token mounted automatically, it's probably broken. Luna: Let me guess - the default token time to live is one hour? And nobody told the developers?

Lucas: You nailed it. Starting in Kubernetes 1.24, the legacy auto-generated tokens with no expiry were disabled. The new default is a token with a one-hour TTL, and it's bound to the pod's lifetime.

But here's the catch: the token is mounted into the pod as a projected volume, but many SDKs and tools weren't built to handle token rotation. They grab the token at startup and never refresh. Luna: So the pipeline starts, pulls the token, and runs for maybe 45 minutes. Then the next API call hits the cluster and gets a 401.

The token expired while the pod was still alive. Lucas: That's exactly the pattern. I spoke with a listener who runs a data processing job that takes about two hours. It kept failing mysteriously at the 55-minute mark.

They blamed the network, the storage class, even the phase of the moon. The actual fix was switching to an explicitly created ServiceAccount that uses a projected mount with a sidecar that refreshes the token. Luna: Right, because the token itself is refreshed on disk by the kubelet. But the application has to actually re-read the file.

Most Go clients using client-go with the 'InClusterConfig' function do that automatically - but the Python and Java SDKs? Not always. Lucas: Let's break down the anatomy of the fix. The recommended approach is the 'Bound Service Account Token Volume' - you explicitly project a token with a desired expiration and a target audience.

Here's the YAML you'd add to a pod spec: volume with 'projected' sources, and inside that, a 'serviceAccountToken' entry. That mounts a token that the kubelet rotates before expiry. Luna: And you can set the 'expirationSeconds' down to as low as 10 minutes, or up to a day. But the key is that the file on disk changes, and your app must watch for that.

Some libraries like 'client-go' handle it; others you need to implement a file watcher. Lucas: Yeah. And then there's the sidecar pattern. You run a container that pulls the token via the TokenRequest API and dumps it to a shared volume, restarting your primary process or signaling it when the token refreshes.

It's heavier, but gives you control. Luna: Before we go deeper, quick honest thing - a handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making episodes like this. If today's tech conversation gave you something usable, that's where you can keep it ad-free. We don't run sponsors, so it's listener-supported.

Okay, back to tokens. Lucas: So the biggest trap is the legacy token automount. Before 1.24, every pod got a token in '/var/run/secrets/kubernetes.

io/serviceaccount' that never expired. Many CI runners, like Jenkins agents or GitLab runners, still rely on that. If you're on a newer cluster, you either need to create a ServiceAccount with 'automountServiceAccountToken: false' and handle it differently, or use the projected volume approach. Luna: But there's also the issue of controllers and operators that were written before the change.

I've seen ArgoCD sync fail because the Application controller's service account token expired mid-sync. Lucas: That's a great point. ArgoCD actually handles token refresh correctly since version 2.4, but older versions?

You'd see sync stuck at 'OutOfSync' because the controller lost authentication. The fix is to upgrade the controller or explicitly set a bound token with a longer TTL, which is allowed if you create the ServiceAccount manually. Luna: What about external access? Like, I'm running a CI job outside the cluster that uses kubectl with a token from a secret?

Lucas: That's a separate path. Secrets that hold tokens are themselves deprecated but not removed. If you manually create a Secret of type 'kubernetes.io/service account token', it will have a token that doesn't expire - but that's officially deprecated.

The better approach is to use the TokenRequest API from outside to get a time-bound token, or use a kubeconfig with a client certificate. Luna: So the overall message is: if your cluster was upgraded to 1.24 or later, and you haven't proactively updated your CI/CD or your in-cluster applications to handle token rotation, you're sitting on a time bomb. Lucas: Exactly.

And the worst part is that failures are intermittent - only when the token expires during a long-running operation. So teams often blame the network or the cloud provider. Let's talk about how to audit your cluster for this. You can check the kube-apiserver logs for 401 errors with 'serviceaccount' in the user field.

Also, list all pods that mount a secret with 'kubernetes.io/service-account' - those are using the legacy path. Luna: And for new code, what's the best practice? I'm thinking: use the client-go library with 'InClusterConfig', and if you're writing Python, use the 'kubernetes' client with token refresh logic or a sidecar.

Lucas: Right. The Python client has a helper called 'load_incluster_config' that reads the token from the projected volume, but it doesn't refresh automatically - you need to reload on a timer. Some teams wrap it with a file watcher that calls 'load_incluster_config' again when the token file changes. Luna: One more thing: what about third-party tools like Istio or Linkerd?

Do they handle token refresh? Lucas: Service meshes are actually a good example. Istio's sidecar proxy has its own identity using the SPIFFE standard, so it doesn't rely on Kubernetes ServiceAccount tokens for control plane auth. But the application itself still uses the projected token for its own calls.

So the sidecar might be fine, but your app code needs to handle it. Luna: Let me ask a controversial question: was this change too abrupt? The Kubernetes community removed the legacy token automount with only one minor version deprecation. A lot of teams got burned.

Lucas: I think it's a valid criticism. The security rationale is solid - stolen static tokens were a huge attack vector. But the migration path assumed that everyone would adopt the SDKs that refresh automatically. In practice, many internal tools and old containers were caught off guard.

The kubelet does rotate the token on disk, but if the code doesn't re-read it, it doesn't help. Luna: So the lesson: after any major upgrade, test your longest-running workloads. And make sure your CI pipelines run for at least an hour and ten minutes to catch token expiration. Lucas: That's a solid testing heuristic.

Also, use the ' - token-ttl' flag when you create a ServiceAccount to set a longer TTL if appropriate - but be careful, because longer-lived tokens are more dangerous if leaked. Luna: Alright, so recap: check your cluster version, audit your pods for legacy token mounts, update your CI runners, and implement token refresh in your code. That's the episode. Lucas: One final thought: if you haven't yet, test your cluster with a workload that runs for 90 minutes with minimal logging.

See if it still authenticates. That's the cheapest way to find out if you have this issue. Luna: Good advice. Thanks, Lucas.

Lucas: Thanks, Luna. We'll catch you next time on DevOps Daily.