How Kubernetes CRD Versioning Breaks Controller Upgrades

Lucas: Luna, I want to talk about a Kubernetes trap that almost never gets attention until it breaks a production rollout: CRD versioning. Specifically, what happens when you update a Custom Resource Definition and your controllers can't keep up. Luna: I've definitely seen controllers crash after a CRD update. Usually the team just restarts the controller and hopes it works.

But the underlying issue is deeper, right? Lucas: Much deeper. The problem starts with how CRDs handle multiple API versions. You can define v1, v1beta1, v1alpha2 - but only one version can be the 'storage version' at any time.

That's the version Kubernetes uses when it writes objects to etcd. When you change the storage version, existing objects get converted only if you have a conversion webhook or if all versions share a common schema. Luna: And that's where teams get tripped up. They update the CRD, add a field, mark the new version as storage, and suddenly old objects lose data or become unreadable by older controllers.

Lucas: Exactly. I saw this happen at a company - let's call them Cloudlytics - that runs a Kubernetes operator for managing data pipelines. They had a CRD called PipelineRun with v1alpha1 as the storage version. They wanted to add a timeoutSeconds field, so they created v1alpha2, marked it as storage, and rolled out the new operator.

Luna: Classic. And then something went wrong, they tried to roll back, and the old operator couldn't read the PipelineRun objects because they were now stored as v1alpha2? Lucas: Exactly that. The old operator expected v1alpha1 schema, but etcd had v1alpha2 objects.

The controller manager crashed on startup with a deserialization error. The team had to manually edit the CRD to change the storage version back - but Kubernetes doesn't allow changing the storage version on an existing CRD without a conversion webhook or a full migration. Luna: Wait, so they were stuck? How did they fix it?

Lucas: They ended up writing a one-time script that queried all PipelineRun objects from etcd directly, unmarshalled them as v1alpha2, dropped the new field, and re-encoded them as v1alpha1. Then they patched the CRD to set v1alpha1 back as storage. It took about four hours of downtime. Luna: That's a nightmare.

And it's not a rare edge case - any team that iterates on CRDs for operators or controllers is at risk if they don't understand storage version semantics. Lucas: Right. And the Kubernetes docs do cover this, but it's buried in the CRD versioning section. Most teams just think, 'I'll add a new version, mark it as storage, and the API server handles conversion.'

That's only true if the versions are structurally identical - same schema, same field types. If you add or remove fields, the built-in conversion won't work. Luna: So the golden rule is: never change the storage version unless you have a conversion webhook or you can guarantee backward compatibility. But what about the common pattern of adding a field with a default value?

Does that break? Lucas: It can, if the field is in the new version spec but not in the old. When the old controller reads the object, it gets an unknown field - if the CRD has preserveUnknownFields set to false, the API server will strip it. But if preserveUnknownFields is true, the old controller might ignore it.

The safer path is to use conversion webhooks from the start if you expect schema evolution. Luna: And conversion webhooks come with their own risks - latency, failure handling, resource usage. But they're better than a four-hour outage. Lucas: Absolutely.

And there's another angle: even if you don't change the storage version, simply adding a new served version can cause issues if your controller's informer cache is watching a specific version. If the watch breaks or the cache gets stale, you might miss updates. Luna: I've seen a related problem where a controller was using a client-go code-generated client that only understood one version. When a new version was served, the client couldn't list or watch the resources at all.

They had to regenerate the client and redeploy. Lucas: If today's tech conversation gave you something usable - maybe a way to avoid your next rollback nightmare - the way this show stays ad-free and focused is through listener support. You can find that at buy me a coffee dot com slash fexingo. Just a simple way to keep these conversations going without interrupting them.

Luna: Yeah, it really does make a difference. And it keeps us independent, so we can dig into exactly these kinds of traps without worrying about sponsorship constraints. Lucas: So back to practical advice. One thing every team should do: before deploying a CRD version change, run a dry-run apply against a test cluster with real objects.

The API server will tell you if any existing resources would fail conversion. There's a kubectl command - kubectl apply - validate=strict - dry-run=server - that catches schema mismatches. Luna: And add that to your CI/CD pipeline. I'd also recommend auditing your CRDs regularly to check for unused served versions.

Each served version adds overhead to the API server, and old versions can confuse operators. Lucas: Good point. And if you're building an operator with the Operator SDK, it generates conversion webhooks for you if you define multiple versions. But many teams skip that because it adds complexity.

They just delete and recreate the CRD - which is fine for development, but not for production with existing custom resources. Luna: Right. Deleting a CRD removes all its custom resources from etcd. That's a hard delete, not a graceful migration.

I've seen teams accidentally wipe out months of configuration data that way. Lucas: So let's summarise the key takeaways. One: never change the storage version without a conversion webhook or a migration plan. Two: if you add a served version, ensure all controllers that watch the CRD are updated to handle the new version.

Three: test CRD changes with - dry-run=server against realistic data. And four: if you must roll back, have a script ready to convert objects back to the old storage version. Luna: And one more: consider using a single version and evolving it with backward-compatible fields and defaults. That avoids the whole multi-version headache, at the cost of less schema rigor.

Lucas: That's a valid trade-off. Some projects like the Kubernetes Gateway API use multiple versions extensively, but they invest heavily in conversion webhooks and testing. For most teams, simpler is safer. Luna: Alright, next time we should talk about the interaction between CRD pruning and defaulting - there's a subtle bug when you set a default value on a field that gets pruned.

Lucas: That's a great topic. We'll get into how defaulting and pruning interact, and why your custom resource might end up with nil values where you expected defaults. For now - thanks for listening, and we'll see you next time.