How GitLab Built a Single Codebase for One Million CI Pipelines

Lucas: So it's mid-2026, and GitLab is running north of a million CI pipelines a day. That's not just a flex, it's a genuine infrastructure challenge - how do you build a system that can schedule, queue, and execute those jobs reliably without collapsing under its own weight? Luna: Especially when each pipeline can have hundreds of jobs, and users expect results in minutes, not hours. Lucas: Right. And what's interesting about GitLab's approach is that they've largely stuck with a monolithic application architecture. They didn't break out CI into a thousand microservices. Instead, they optimized a single Rails codebase to handle that scale. The engineering team wrote a lot about this around 2023 and 2024, and it's still the backbone today. Luna: That feels almost contrarian. Everyone talks about microservices for scale, but GitLab doubled down on the monolith. Lucas: Exactly. And the reasoning is pragmatic. They realized early that the bottleneck wasn't the application code per se - it was the database and the job scheduler. So they poured effort into PostgreSQL partitioning and Redis Cluster for queue management. For example, they partition the `ci_pipelines` table by project ID and by creation date, so queries for recent pipelines on a busy project don't scan the whole table. Luna: That makes sense. And Redis Cluster - are they using it for the job queue itself? Lucas: Yes. The job queue is essentially a set of Redis lists and sorted sets. But they ran into issues with Redis Cluster's cross-slot operations. So they had to design their queue keys to hash to the same slot for related jobs. That meant careful key naming - using the same hash tag for all jobs belonging to the same pipeline. Luna: Hash tags - that's the `{pipeline_id}` pattern, right? So all keys for a pipeline go to the same node. Lucas: Exactly. It's a small detail, but it avoids multi-key operations that fail in cluster mode. They also moved from a single global queue to a 'fleet-wide coordination' model where each runner picks up jobs based on tags and capacity. That's more complex but more scalable. Luna: And they have to handle retries, cancellations, and dynamic scaling of runners. It's not just queuing - it's state management. Lucas: Right. And one of the hardest parts was eliminating global locks. In early versions, they had a mutex around job scheduling to prevent duplicate assignments. But at a million pipelines, that lock became a bottleneck. So they moved to a lock-free design using atomic Redis operations - like `WATCH` and `MULTI` for optimistic locking, and Lua scripts for atomic updates. Luna: That's clever. But it also means more complexity in the application code to handle retries on conflicts. Lucas: Exactly. And they embraced that. The thing I respect is how they used feature flags to roll out these changes gradually. They'd enable the new scheduler for 1 percent of projects, monitor for regressions, then ramp up. That let them catch edge cases without blowing up the whole fleet. Luna: It's a good reminder that scaling isn't just about new architecture - it's about safe deployment. One thing I'm curious about: did they ever consider rewriting CI as a separate service? Like, a dedicated CI microservice? Lucas: They did. There was a lot of internal debate around 2021. But the conclusion was that the cost of splitting - especially the data consistency challenges - outweighed the benefits. You'd have to replicate pipeline data between services, deal with eventual consistency, and manage two deployment pipelines. It's not trivial. So they kept the monolith but made it modular internally. Luna: Modular monolith - that's the term. And they used Sidekiq for background jobs, which is another layer of queuing. Did they run into issues there? Lucas: Yeah, Sidekiq was fine for a while, but as job volumes grew, they had to shard Sidekiq queues by priority and by job type. They also moved some critical jobs to a separate Redis instance to avoid noisy neighbor effects. For example, pipeline creation jobs went to a dedicated queue with its own Redis, so a burst of low-priority jobs wouldn't delay a new pipeline from starting. Luna: That's a classic pattern - separate infrastructure for critical vs. non-critical workloads. Lucas: And the results speak for themselves. GitLab's CI system handles over a million pipelines daily with p99 job scheduling latency under one second. That's from a single Rails app, heavily optimized. It's a great case study for anyone building developer tools. Luna: Speaking of building and running things - Lucas, I know we usually just dive into the technical details, but I want to pause for a second. This show exists because a group of listeners chip in monthly, and that's what keeps it ad-free and focused on real engineering stories. Lucas: Yeah, it's a small group, but it makes a huge difference. If these conversations are useful for what you're building or running, you can join them at buy me a coffee dot com slash fexingo. No pressure, just mentioning it because it's the only reason we can keep doing deep dives like this. Luna: And we really mean that - no ads, no sponsor reads, just the content. So thanks to everyone who already supports it. Lucas: Alright, back to GitLab. One more angle I want to touch on: how they handle database migrations at scale. When you have a million pipelines a day, even a simple schema change can cause downtime. Luna: Right. They use a process called 'zero-downtime migrations' from Day One. All migrations must be reversible and batched. Lucas: Exactly. They break large changes into multiple steps. For example, adding a column with a default value: first they add the column as nullable, then backfill default values in batches of 1000 rows, then make it not null. This avoids long table locks. They also use `pt-osc` or `gh-ost` for online schema changes, but built their own tooling around it. Luna: And they also have a policy of 'database review' for every merge request that touches the data layer. That is a huge culture investment. Lucas: Yeah, they have a team of database reviewers who are assigned to every MR that includes a migration or a performance-sensitive query. That catches issues early. For example, they caught a migration that would have locked the `ci_job_artifacts` table for hours on GitLab.com. They redesigned it to use incremental cleanup instead. Luna: It's that combination of technical rigor and process that makes it work. The monolith itself isn't magical - it's the discipline around managing it. Lucas: Exactly. And that's the takeaway for me. GitLab didn't need to rewrite everything. They invested in understanding where the real bottlenecks were - database, queueing, deployment - and solved those specifically. The result is a system that's still growing. Luna: And it's a reminder that sometimes the best architecture is the one you already have, if you're willing to optimize it relentlessly. Lucas: Well said. That's all for this episode of The CTO Podcast. We'll be back next time with another deep dive.