How Slack Rebuilt Its Search Index for 10 Million Daily Queries

Lucas: Luna, I want to talk about a piece of infrastructure that most of us use every day but rarely think about until it breaks. Luna: Let me guess - search on Slack? Lucas: Exactly. Slack handles over 10 million search queries a day. That's more than what many dedicated search engines see. And for years, it ran on a single Elasticsearch cluster that was starting to buckle. Luna: I remember the days when searching for an old message felt like a coin flip - sometimes it was instant, sometimes it timed out. Lucas: Right. And the engineering team knew they had to rebuild. But this wasn't a simple upgrade - it was a full migration to a custom search engine they called 'Search It'. The whole thing took over two years. Luna: Two years is a long time to be running a system you're actively replacing. How did they keep the old one running while building the new one? Lucas: That's the fascinating part. They ran both systems in parallel for over a year. Every query went to both the old Elasticsearch cluster and the new Search It system. They compared results, latency, and reliability before they ever let Search It serve production traffic. Luna: So they were literally paying for two search infrastructures simultaneously. That's expensive. Lucas: Absolutely. But it was the only way to guarantee zero downtime. The team knew that if they flipped a switch and something went wrong, millions of users would lose access to their message history. So they took the cost hit. Luna: What was wrong with Elasticsearch? It's not like it's a bad product. Lots of companies run it at scale. Lucas: Elasticsearch is great - for certain workloads. But Slack's data has a very specific pattern. Users search mostly for recent messages, but they also need to occasionally find something from years ago. The access pattern is skewed, and Elasticsearch's sharding model wasn't a great fit. Luna: How so? Lucas: Elasticsearch uses a fixed number of shards per index. If you pick too few, you can't scale. Too many, and you waste resources on metadata overhead. Slack's data grows every day - billions of messages - so they were constantly reindexing, which caused performance spikes. Luna: And the custom solution? How did Search It differ? Lucas: Search It was designed around a 'time-based sharding' model. Each shard covers a specific time range - say, one week of messages. New data goes into a new shard automatically. Old shards become read-only. That means write traffic is always going to a small set of active shards, and reads can be distributed across many. Luna: So the hot shards stay small and fast. That's clever. Lucas: It's a pattern that a lot of time-series databases use. But applying it to a general-purpose search engine required building a lot of custom infrastructure. The team wrote their own query router, their own index management, and their own replication layer. Luna: And all of that had to be reliable from day one. How did they test it? Lucas: They used a technique called 'shadow traffic'. For months, every search query was sent to both systems, but only the Elasticsearch result was shown to users. The Search It result was logged and compared offline. They looked for differences in result sets, latency, and error rates. Luna: I imagine they found a lot of edge cases. Queries with special characters, misspellings, that sort of thing. Lucas: Exactly. Slack search has to handle all kinds of messy human input - emoji, code snippets, file names, channel names. The team had to tune the tokenizer and scoring algorithm for months to get parity. Luna: And once they were confident, how did they actually migrate users? Lucas: They did it gradually - workspace by workspace. Each workspace was a tenant. They could flip a single workspace to Search It and monitor its behavior before moving the next. That way, if something went wrong, only that workspace was affected. Luna: So the migration took months, not a weekend. Lucas: Right. And the team had to manage a lot of complexity: keeping the two systems in sync, handling backfill for older data, and maintaining backwards compatibility for the API. It was a massive engineering effort. Luna: What about the human side? Two years is a long project. How did they keep the team motivated? Lucas: That's a great question. I spoke with one of the engineers who said the key was to ship incremental value. They didn't wait until the whole system was done. They released pieces - like a faster search for recent messages - and celebrated those wins. That kept the team energized. Luna: And leadership? Getting buy-in for a multi-year, expensive migration can't be easy. Lucas: They had data on their side. The old system was hitting limits: p99 latency was over 2 seconds during peak hours, and they had a growing number of timeouts. Leadership understood that if they didn't fix it, the product would suffer. Luna: So it was a bet on the future. And it paid off - Slack search today is much faster and more reliable. Lucas: Yeah, the new system handles 10 million queries a day with p99 latency under 200 milliseconds. That's a 10x improvement. And it's horizontally scalable - they can add capacity by adding shards, not by reindexing. Luna: It's a great case study in how to think about search infrastructure at scale. But also in how to manage a long-running migration. Lucas: Honestly, that's the part that often gets overlooked. The technical design is important, but the project management - running two systems, shadow testing, gradual rollout - that's what made it successful. Luna: And it's a reminder that sometimes the right solution is to build your own, even when off-the-shelf tools exist. Lucas: Yeah, but you have to be honest about the cost. Building a custom search engine is not something most companies should do. Slack had the resources and the specific need. For many teams, Elasticsearch or Algolia is perfectly fine. Luna: So the lesson is: understand your access patterns before you commit to a technology. Lucas: Exactly. And be willing to invest in the right long-term solution, even if it means a painful multi-year migration. Luna: I want to quickly mention something that's top of mind for us. This podcast exists because of listeners like you - and a small group of them actually chip in monthly to keep it ad-free. Lucas: Yeah, and we really appreciate that. If you find these deep dives useful for your own work, you can support us at buy me a coffee dot com slash fexingo. It's what keeps us going. Luna: And we're able to focus on exactly the topics that matter to operators and builders. So thank you to everyone who already does. Lucas: Alright, back to search. One thing I didn't mention is that the migration also forced Slack to rethink its backup and disaster recovery strategy. The old Elasticsearch cluster had a single point of failure. The new system is fully distributed, with data replicated across availability zones. Luna: That's a nice side effect of the redesign. You fix one problem and you end up fixing others. Lucas: Exactly. And that's the kind of outcome that makes the multi-year effort worth it. Thanks for listening, and we'll be back next week with another deep dive.