How MongoDB Rebuilt Its Document Store for Multi-Terabyte Workloads

Episode notes

Episode 69 of The CTO Podcast. Lucas and Luna dive into how MongoDB's engineering team redesigned their core storage engine to handle multi-terabyte workloads without sacrificing developer velocity. They break down the specific challenge: a single customer complaint about a 12-terabyte shard that took 47 hours to rebalance. That complaint triggered a two-year effort to rewrite the WiredTiger storage engine's write-ahead log and compression layer. Lucas explains why the old architecture hit a wall at around 8 terabytes per node, and how the team's decision to switch from per-document compression to page-level compression made rebalancing 40x faster. Luna brings in a parallel from her own experience at a fintech startup that hit similar scaling pains. They discuss the trade-offs the MongoDB team made - accepting a 15 percent increase in storage cost per gigabyte in exchange for predictable rebalance times. A concrete look at how one of the most popular NoSQL databases evolved to meet the demands of modern workloads.

• How Airbnb Rebuilt Search for 8 Million Listings42 / 100
• How GitLab Built a Single Codebase for One Million CI Pipelines45 / 100
• How Slack Rebuilt Its Search Index for 10 Million Daily Queries37 / 100
• How Notion Rebuilt Its Sync Engine for Offline-First
• How Notion Rebuilt Its Block Engine for Hybrid Local-Sync