How Datadog Rebuilt Its Observability Pipeline for 100 Trillion Events Daily

Lucas: So every time you open a dashboard at work - whether it's Datadog, Grafana, New Relic - you're looking at a live visualisation of trillions of machine-generated events that had to be ingested, processed, stored, and indexed in near real-time. And the engineering challenge behind that pipeline is absolutely brutal. Luna: Brutal is the right word. I've talked to infra teams who say their observability bill is one of the top three line items in their cloud spend.

The scale is just - it's becoming its own problem. Lucas: Exactly. And Datadog is probably the most visible player here. As of mid-2026, they're processing over 100 trillion events per day.

That's not a typo. One hundred trillion. And that number has roughly doubled every year for the last three years. So a couple of years ago they had to fundamentally rethink their entire pipeline - from the ingestion layer all the way down to storage.

Luna: Was there a specific breaking point? A moment where they said okay, the current architecture won't survive another year? Lucas: There was. Around late 2023, their legacy intake was built on a monolithic set of Kafka clusters with custom consumers.

It worked fine at 10 trillion events a day. But at 30 trillion, they started seeing backpressure cascades. A single spike from a large customer - think a Black Friday for an e-commerce client - could cause latency to balloon across all tenants. The p99 for event ingestion, which had been around 200 milliseconds, started creeping toward two seconds.

Luna: And in observability, two seconds is an eternity. If your alerting is delayed by two seconds during an incident, you might as well not have it. Lucas: Right. So they embarked on a multi-phase re-architecture.

Phase one was about sharding the ingestion layer. Instead of one giant Kafka cluster, they moved to a set of regional, tenant-aware shards. Each shard handles a subset of customers, so a spike in one shard doesn't affect the others. Luna: That sounds like a classic sharding pattern, but the hard part is - how do you decide which customer goes in which shard without creating hot spots?

Lucas: That's the tricky bit. Datadog uses a two-level hash. First, they hash the customer ID to assign them to a shard group. Then within that group, they use a consistent-hash ring to distribute across actual Kafka brokers.

The key insight was that they also built a load-shedding mechanism: if a shard's CPU or memory crosses a threshold, they can dynamically move a customer's traffic to a different shard. They call it 'live rebalancing.' It happens transparently with a small latency blip - maybe 50 milliseconds - but no dropped events. Luna: Okay, but ingestion is only the front door.

What about processing? Once events are in Kafka, they have to be parsed, enriched, and indexed. That's where the real compute lives. Lucas: Right.

That's phase two. Datadog's processing layer was originally a set of Java-based stream processors running on YARN. By 2024, that setup was hitting memory walls. Each event - say a single metric or a span - has to be parsed into a structured format, enriched with metadata like host tags and service names, and then written to multiple indexes: one for the live tail, one for search, one for analytics rollups.

Luna: And each of those indexes is a different data structure with different write patterns. So the processing step isn't just ETL - it's ETL to three different targets simultaneously. Lucas: Exactly. Datadog's solution was to move from a record at a time model to a micro-batch model.

They aggregate events into small batches - about 1,000 events or 50 milliseconds of data, whichever comes first - and then process the entire batch through a pipeline of stateless operators. That gives them better cache locality and fewer memory allocations. They also rewrote the parsing engine in Rust, because Java's garbage collection was causing unpredictable pauses under high throughput. Luna: So phase two is a Rust rewrite.

That's a pretty serious engineering investment. Was it worth it? Lucas: They claim a 40 percent reduction in per-event CPU cost and a 60 percent reduction in p99 tail latency. The Rust parser doesn't need a GC pause, and it uses a fraction of the memory.

But the real win is that it allowed them to handle much more complex parsing - things like custom log formats - without a performance penalty. Luna: Quick honest thing - a handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making this many of these episodes. If these conversations are useful for what you're building or running, that's where the support lives. Lucas: Yeah, it's a small group, but it's the whole reason we can keep this ad-free and focused on the real engineering details.

Appreciate every one of you. Luna: Alright, back to Datadog. So they've solved ingestion and processing. But storage is the next monster.

How do you index 100 trillion events per day and still make them searchable in seconds? Lucas: This is where things get really interesting. Datadog's storage layer is built on a custom columnar store they call 'Vulcan' - not the database, but an internal project. Vulcan is designed specifically for observability data, which has a few unique properties: events are immutable, writes are append-only, and queries are almost always time-range bounded.

Luna: So time-series characteristics, but with arbitrary metadata dimensions. That's a hard fit for traditional time-series databases. Lucas: Exactly. Vulcan organises data into time-partitioned segments - each segment covers a fixed time window, like 15 minutes.

Within a segment, events are sorted by timestamp and stored in a compressed columnar format. The compression is the secret sauce: Datadog developed a custom scheme that takes advantage of the fact that many fields - like hostname or service - repeat frequently within a short time window. Luna: So they're doing dictionary compression, but tuned for streaming? Because you don't know the dictionary ahead of time when you're ingesting in real time.

Lucas: Precisely. They use a technique called 'streaming dictionary compression.' As events come in, they build a dynamic dictionary per segment. New values get added on the fly.

The dictionary itself is compressed and stored alongside the data. For fields with low cardinality - say, a data center name - they achieve compression ratios of 30 to 1. For high-cardinality fields like request IDs, it's closer to 2 to 1. But overall, they claim a 10x reduction in raw storage size compared to uncompressed JSON.

Luna: And that 10x directly impacts query speed. Less data to scan means faster searches. Lucas: Right. But queries aren't just about scanning less data.

They also need to skip irrelevant partitions quickly. Datadog uses a technique they call 'bloom filter indexes' on common dimensions. For example, if you query for errors in service 'checkout' in the last hour, the system first checks a bloom filter on the service dimension. If the filter says 'no checkout events in this segment,' it skips the entire segment without decompressing anything.

Luna: That's clever. But bloom filters have false positives. So they can still hit segments that don't have the data. Lucas: Right, but false positives just mean an extra decompression and scan - not wrong results.

The trade-off is worth it because the bloom filters are tiny: about 2 bits per event per dimension. With a 1 percent false positive rate, they skip 99 out of 100 irrelevant segments. That's huge for query performance. Luna: What about the query engine itself?

I know Datadog's query language is proprietary, but how does it execute across these distributed segments? Lucas: Phase three of the re-architecture was rewriting the query engine. The old engine was a simple pushdown: it sent the query to every segment, each segment evaluated it locally, and then results were merged. That worked when segments were few, but with 100 trillion events a day, you have tens of thousands of segments per query.

The network overhead was killing latency. Luna: So they needed to reduce the number of segments they even touch. That's where the bloom filters come in, but also - they must have added some kind of metadata-based pruning. Lucas: Exactly.

Datadog built a 'segment metadata index' that stores summary statistics for each segment: min and max timestamps, cardinality estimates for each dimension, and a list of distinct values for low-cardinality fields. The query planner uses that metadata to prune segments before sending any execution plan. They claim this eliminates 80 to 90 percent of segments before any data is touched. Luna: That's dramatic.

So the actual execution only hits the segments that could possibly contain relevant data. Lucas: Yes. And within those segments, the engine uses vectorized execution. Each segment is read in batches of 4,096 events, decompressed into columnar format, and then processed using SIMD instructions.

For aggregations like count or avg, they can process an entire batch in a few microseconds. Luna: What about tail latencies? In observability, the p99 matters more than the average. A dashboard that sometimes takes 10 seconds is useless.

Lucas: They focused heavily on that. One technique is 'speculative execution.' If a query involves multiple segments, the engine sends the query to all of them in parallel, but it also sends to a few redundant segments that might have overlapping data. If one segment is slow, the redundant responses can mask it.

They also set a timeout per segment - typically 100 milliseconds - and if a segment doesn't respond in time, the engine falls back to a precomputed rollup. Luna: So they degrade gracefully. Instead of showing nothing, they show approximate data. That's a good product decision.

Lucas: Exactly. They call it 'approximate querying.' For dashboards, the default is to use approximate mode for any query that scans more than 10,000 segments. Users can opt into exact mode, but it takes longer.

Most customers never notice the difference because the approximations are within 1 percent for common queries. Luna: Let's zoom out a bit. This entire re-architecture - sharded ingestion, Rust stream processors, Vulcan storage, metadata pruning, speculative execution - how long did it take, and what's the impact been? Lucas: Started in early 2024, and most of it was in production by Q2 2025.

So about 18 months of major engineering work across multiple teams. The result: they went from handling 30 trillion events per day in early 2024 to over 100 trillion today, with p99 ingestion latency under 200 milliseconds and query latency under 500 milliseconds for the vast majority of dashboard queries. Luna: And they did it without breaking backward compatibility. Customers didn't have to change a single agent configuration.

Lucas: That's the part that's hardest to appreciate. When you're re-architecting a live system that processes trillions of events a day, you can't just flip a switch. They ran the old and new pipelines in parallel for months, gradually shifting traffic. And they built extensive automated canary testing - replaying production traffic against the new pipeline and comparing results.

Luna: I think the takeaway for engineering leaders is that scaling observability is not just about throwing more hardware at the problem. It requires deep rethinking of every layer - ingestion, processing, storage, query - and a willingness to build custom infrastructure when off-the-shelf solutions don't cut it. Lucas: Absolutely. And the decision to invest in a custom columnar store and a custom compression scheme, rather than just scaling up Elasticsearch or ClickHouse, is a bet on owning the entire stack.

It's a huge engineering cost, but for a company whose product is observability, it's their core competitive advantage. Luna: So the next time you open a Datadog dashboard and see a graph render in under a second, you'll know there's a whole pipeline of Rust code, dictionary compression, and bloom filters making that happen. Lucas: Exactly. And the interesting question is: where does the next bottleneck appear?

We're already seeing hints that the network itself is becoming the constraint. Datadog has started experimenting with edge-based pre-processing - running lightweight aggregations on the agent itself before data leaves the customer's environment. That could be phase four. Luna: That would be a fascinating episode.

Maybe when they ship it. Lucas: Definitely worth covering. For now, though, the takeaway is clear: when your data grows 2x every year, you have to rethink everything, and you have to keep rethinking it.