DevOps Daily with Fexingo · 2026-07-01 · 9 min
Key moments - from our scoring
Substance score
65 / 100
Five dimensions, 20 points each
The Kubernetes volume snapshot feature presents a critical but often-overlooked data corruption risk for stateful workloads running on PVCs. Lucas and Luna explain how the CSI snapshot controller initiates snapshots without coordinating with the filesystem or application layer, potentially capturing partially-written blocks when writes are in flight. This is particularly dangerous for databases like PostgreSQL and MySQL, where torn writes can corrupt pages silently if checksums aren't enabled. The CSI spec v1.9 lacks consistency group parameters, leaving it to individual storage vendors - NetApp Trident and Portworx have protections, but many don't - to implement quiesce mechanisms. The discussion covers concrete failure modes (half-updated blocks frozen in snapshots), architectural gaps (missing consistency-group support in KEP 3017, stalled since 2023), and VolumeSnapshotClass DeletionPolicy risks (defaulting to Delete, erasing backups on accidental deletion). The pragmatic workaround is using Velero with pre-hooks like pg_start_backup or MySQL flush commands, combined with Retain policies and periodic test restores to catch zero-length snapshots and data inconsistencies before production incidents.
The CSI snapshot controller calls CreateSnapshot immediately without coordinating with the filesystem or application, potentially capturing partially-written blocks. If writes are in flight or in page cache when the snapshot is taken, the snapshot contains torn writes - incomplete updates across blocks - that corrupt data on restore.
Crash-consistent snapshots capture all data on disk at a point in time but may miss in-flight writes and buffered data. Application-consistent snapshots require the application or filesystem to quiesce before capture, ensuring all pending writes are flushed to disk first.
Set VolumeSnapshotClass DeletionPolicy to Retain instead of the default Delete, use Velero with pre-hooks (like pg_start_backup) to quiesce applications before snapshots, and periodically restore snapshots to a test cluster to validate the data.
PostgreSQL can detect it if data checksums are enabled, but many deployments don't use them; MySQL may fail to start or silently serve incorrect data depending on the corruption extent.
KEP 3017 has been in discussion since 2023 but remains unreleased because different storage backends have divergent consistency implementations, making it difficult to define a generic API that works across vendors.
Our reviewer’s read on each dimension, with quotes from the episode.
The episode delivers high-density, actionable technical insights about a specific failure mode (torn writes, data corruption from inconsistent snapshots) that most operators won't have encountered directly. It covers the root cause (CSI spec limitation, lack of quiescing), concrete manifestations (partially written blocks, PostgreSQL checksums), and multiple mitigation strategies. Minimal filler; nearly every exchange adds technical substance.
The snapshot controller calls CreateSnapshot as soon as the VolumeSnapshotContent is bound. There's no built-in hook to run 'fsfreeze' or flush the application's write buffers.
The snapshot contains block one with the new data, block two with the old data - the page is corrupt.
The episode tackles a genuine edge case (snapshot-induced silent corruption) that isn't widely discussed in mainstream K8s content, and correctly identifies the root cause as a CSI spec gap rather than user error. However, the mitigations (pre-hooks, Velero, CSI driver selection) are standard practices; the originality lies in connecting them to this specific failure mode rather than discovering a novel solution.
The CSI spec doesn't guarantee that the snapshot is taken at a consistent point.
KEP 3017 - that's been in discussion since 2023 about adding consistency group support to the CSI snapshot API. But as of mid-2026, it's still not released.
Lucas and Luna appear to be experienced ops practitioners with deep Kubernetes and storage knowledge (they reference CSI specs, PostgreSQL behavior, Velero hooks, and KEPs with accuracy). However, they are not identified as operators from major infrastructure teams, and the conversation feels more like peer discussion than interviewing a recognized authority who has shipped at scale.
I've seen this bite teams running databases on PVCs.
I've seen cases where the CSI driver returned success but the backend had an internal timeout, resulting in an empty snapshot.
Strong on technical specifics: CSI spec version 1.9, KEP 3017, specific failure vectors (8 KB PostgreSQL pages, torn writes across two blocks), concrete tools (Velero, NetApp Trident, Portworx), and named mitigations (pg_start_backup, MySQL table locks). Light on metrics (no numbers on incident frequency, customer impact, or adoption rates); could have named affected organizations or provided quantitative examples.
I looked at the CSI spec version 1.9 - the CreateSnapshot RPC doesn't have a 'consistency group' parameter.
Take a PostgreSQL database running on a PVC backed by a SAN. The database has an 8 KB page in memory and another on disk. It does a write that spans two blocks.
Luna asks sharp follow-up questions ('What does a torn write look like in practice?' 'What's the hold-up?') and catches important details (DeletionPolicy gotcha). Lucas responds with concrete examples and methodical explanations. However, the conversation lacks productive disagreement or tension; both parties largely agree, and there are no moments where one pushes back or challenges an assumption, which would deepen the discussion.
What does a torn write look like in practice? Give me a concrete example.
Any other gotchas with volume snapshots? I recall something about the 'VolumeSnapshotClass' and the 'DeletionPolicy'.
Computed from the transcript - who did the talking, and the words that came up most.
Episode 84 of DevOps Daily with Fexingo. Lucas and Luna investigate a subtle but dangerous bug in Kubernetes volume snapshot workflows. They break down how the CSI snapshot controller, combined with certain storage backends, can lead to silent data corruption when snapshots are taken during active I/O. Lucas walks through the sequence of events: the VolumeSnapshot CRD, the VolumeSnapshotContent object, and the timing gap where the CSI driver creates a snapshot before the filesystem is quiesced. Luna challenges whether this is a Kubernetes issue or a storage vendor problem, and they discuss real-world mitigations like fs-freeze hooks and snapshot consistency groups. The episode ends with practical advice for cluster operators running stateful workloads. #Kubernetes #VolumeSnapshots #DataCorruption #CSIStorage #StatefulWorkloads #CloudNative #DevOps #Infrastructure #StorageBackend #Snapshots #K8sStorage #ContainerStorage #Backup #DisasterRecovery #Technology #DevOpsDaily #FexingoBusiness #BusinessPodcast Keep every episode free: buymeacoffee.com/fexingo
Transcribed and scored by The B2B Podcast Index.
Lucas: If today's tech conversation gave you something usable - maybe a storage pattern you hadn't thought about - that's exactly why this show exists. So before we dive in, a quick honest thing: a handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making this many of these. No ads, no sponsors, just people who find the deep dives useful. More on that later - let's get into today's angle.
Luna: Yeah, and if you've ever lost data because a snapshot was inconsistent, you'll want to hear this. Lucas: So today we're talking about Kubernetes volume snapshots. Not the high-level 'how to take them' - the specific, nasty failure mode where your snapshot silently corrupts data on the storage backend. I've seen this bite teams running databases on PVCs.
Luna: Silent corruption is the scariest kind. No crash, no error - just wrong data when you restore. Lucas: Exactly. And it's not a Kubernetes bug per se - it's a timing and ordering problem between the CSI snapshot controller and the storage backend.
Let me paint the picture. You create a VolumeSnapshot custom resource. The snapshot controller creates a VolumeSnapshotContent, which triggers the CSI driver to call the storage backend's CreateSnapshot API. Luna: And the problem is that the filesystem on the volume might still have writes in flight.
Lucas: Right. The CSI spec doesn't guarantee that the snapshot is taken at a consistent point. Most storage backends will take a crash-consistent snapshot - meaning all data that's already on disk is captured, but anything in the application's buffer, or in the kernel's page cache, might be lost. And worse, if the volume is mounted with writeback caching, the snapshot could capture a partially written block.
Luna: That's the corruption vector. A block that's half-updated gets frozen in the snapshot. Lucas: Precisely. Now, some storage backends - like NetApp's Trident or Portworx - have mechanisms to quiesce the filesystem before taking the snapshot.
But the vanilla CSI snapshot flow doesn't enforce that. The snapshot controller calls CreateSnapshot as soon as the VolumeSnapshotContent is bound. There's no built-in hook to run 'fsfreeze' or flush the application's write buffers. Luna: So it's up to the storage vendor to implement consistency.
And not all do. Lucas: That's the key. I looked at the CSI spec version 1.9 - the CreateSnapshot RPC doesn't have a 'consistency group' parameter.
It's just a point-in-time copy of blocks. If the backend doesn't coordinate with the filesystem, you get what's called a 'torn write'. Luna: What does a torn write look like in practice? Give me a concrete example.
Lucas: Take a PostgreSQL database running on a PVC backed by a SAN. The database has an 8 KB page in memory and another on disk. It does a write that spans two blocks. The disk write for block one completes, but block two is still in the SAN's write cache.
You take the snapshot at that exact moment. The snapshot contains block one with the new data, block two with the old data - the page is corrupt. Luna: And PostgreSQL's checksums will catch that on restore, but only if you have data checksums enabled. A lot of people don't.
Lucas: Right. And even if they do, the database might refuse to start, or worse, it might start and serve incorrect data. So the mitigation is twofold. One: use a storage backend that supports application-consistent snapshots.
Two: if you're taking snapshots for backup, make sure you either quiesce the application beforehand or use a CSI driver that integrates with the filesystem. Luna: There's also the option of using VolumeSnapshots as building blocks for backup tools like Velero. Velero can run pre- and post-hooks to flush databases. Lucas: That's exactly what I was going to say.
Velero's hook system is the pragmatic workaround. You define a pre-hook that runs 'pg_start_backup' or flushes MySQL tables with read lock. That ensures the data on disk is consistent before the snapshot is taken. But that relies on the user configuring those hooks correctly - and I've seen plenty of Velero schedules without any pre-hooks.
Luna: So the burden is on the operator. Kubernetes doesn't protect you from yourself. Lucas: It's a classic 'Kubernetes provides the primitive, but you have to compose it safely' problem. There's a Kubernetes Enhancement Proposal - KEP 3017 - that's been in discussion since 2023 about adding consistency group support to the CSI snapshot API.
But as of mid-2026, it's still not released. Luna: What's the hold-up? Consistency groups are table-stakes for enterprise storage. Lucas: The challenge is that different storage backends have wildly different implementations.
Some support multi-volume consistency groups, some don't. The KEP tries to define a generic API, but it's hard to get consensus. Meanwhile, users keep losing data. Luna: Any other gotchas with volume snapshots?
I recall something about the 'VolumeSnapshotClass' and the 'DeletionPolicy'. Lucas: Good catch. The DeletionPolicy is a silent data loss vector too. If you set it to 'Delete', the default, when you delete the VolumeSnapshot object, Kubernetes also deletes the actual snapshot on the storage backend.
One wrong 'kubectl delete' and your backup is gone. Many teams set it to 'Retain' for critical backups. Luna: And the 'VolumeSnapshotContent' stays around as a remnant, but at least the data is safe. Lucas: Exactly.
But the bigger point is: test your snapshot restore. Don't assume a snapshot is valid just because it completed without errors. I've seen cases where the CSI driver returned success but the backend had an internal timeout, resulting in an empty snapshot. Luna: Ah, the 'zero-length' snapshot.
That's terrifying. Restore it and you get a clean empty volume. Lucas: Right. And since Kubernetes doesn't validate the content of the snapshot, the restore succeeds.
You only discover the problem when your application fails to start because the data directory is empty. Luna: So what's your recommended approach for teams running stateful workloads on Kubernetes today? Lucas: One: use a CSI driver that supports snapshot consistency - check the vendor's documentation for whether they quiesce the filesystem. Two: for databases, always use Velero or a similar tool with pre-hooks.
Three: set DeletionPolicy to Retain for all production snapshot classes. And four: periodically restore a snapshot to a test cluster and validate the data. Luna: And maybe don't rely solely on snapshots for backup. Have a separate backup pipeline - like pg_dump or mysqldump - that runs in parallel.
Lucas: Absolutely. Snapshots are great for rapid recovery from cluster failures, but they're not a substitute for logical backups. And if your storage vendor doesn't support consistent snapshots, you're gambling every time. Luna: One final thought: the Kubernetes documentation recommends using CSI driver 'capabilities' to check if the driver supports 'VolumeSnapshot' and 'VolumeSnapshotContent'.
But there's no capability for 'consistent snapshot'. That gap is telling. Lucas: Yeah. The CSI spec doesn't have a capability flag for consistency.
So as an operator, you have to trust your vendor or test it yourself. And testing is hard because the corruption might only show up weeks later when you need to restore. Luna: All right, so for listeners: if you're using VolumeSnapshots in production, go check your pre-hooks. And make sure you've restored at least once.
Lucas: And if you've got a story about a snapshot that went wrong - or a fix you've implemented - we'd love to hear it. Reach out. That's all for today. We'll see you next time on DevOps Daily.
Other episodes covering the same guests and topics, from across The B2B Podcast Index.
How Idempotency-Key Design Prevents Payment DisastersThe Developer Tools Podcast with Fexingo · features Luna98 / 100
Why Pipeline Velocity Trumps Deal Size Every TimeThe Growth Operator with Fexingo · features Luna95 / 100
Why Enterprise Software Deals Now Include a Vendor AI Model Explainability MandateB2B SaaS Talks with Fexingo · features Luna94 / 100
How B2B Brands Wreck Pipeline with Unsyncroned CRM DataThe Marketing Operator Podcast with Fexingo · features Luna92 / 100
Why Marketing Attribution Misses the Seasonality PatternMarketing Analytics with Fexingo · features Luna91 / 100
How to Sell Against a Competitor Already in the BuildingSales Leadership with Fexingo · features Luna85 / 100
How Kubernetes StatefulSet PVC Resizing Causes Node Disk Failures94 / 100How Kubernetes Topology Spread Constraints Create Scheduling Hotspots95 / 100How Kubernetes CRD Versioning Breaks Controller Upgrades90 / 100How Kubernetes Audit Logging Causes etcd Performance Degradation91 / 100How Kubernetes Custom Resource Definitions Cause Controller Memory Leaks100 / 100