How Coinbase Rebuilt Its Blockchain Node Infrastructure

Lucas: If these conversations are useful for what you are building or running, that is the whole point. We keep this show ad-free by design, so if the work here helps you think through an architecture decision or avoid a production outage, and you want to support that, the link is buy me a coffee dot com slash fexingo. That is it. No tiers, no perks.

Just keeping the podcast a resource that respects your time. Luna: Yeah, I like that framing. It is a straight trade: if you get value, you can throw in a coffee. No friction.

Lucas: Exactly. And with that out of the way, let us talk about something that does not get enough attention in the infrastructure space: how Coinbase rebuilt its blockchain node layer from the ground up. Luna: Node infrastructure is one of those things that sounds niche until you realize it is the backbone of everything crypto exchanges do. Lucas: Right.

Coinbase started out running a handful of full nodes for Bitcoin and Ethereum. That was manageable. You could SSH in, restart a service, maybe patch a client. But by 2024, they were supporting over 200 blockchain networks, each with its own consensus rules, client software, and hardware requirements.

Luna: Two hundred networks. That is not just more of the same. That is a fundamentally different operational problem. Lucas: Exactly.

And the old approach - dedicated machines per network, manual configuration, a lot of tribal knowledge - was not going to scale. So they set out to build what they call internally a 'node orchestration platform.' The goal was to abstract away the differences between chains and provide a unified interface for consuming on-chain data. Luna: What was the first big design decision they made?

Lucas: They committed to microservices early. Each network gets its own set of containers: one for the consensus client, one for the execution client, one for a health-check agent. Those containers run on Kubernetes, and the orchestration layer handles lifecycle management - spinning up new nodes, resyncing state, failing over when a node stalls. Luna: Stalling is a huge problem in this space.

A node can appear healthy - the process is running, memory is fine - but it has actually fallen behind the network tip by hundreds of blocks. Lucas: Right. And that is the worst kind of failure because it is silent. So Coinbase built a monitoring system that tracks each node's block height against the network's canonical chain.

If the lag exceeds a threshold, the orchestrator automatically triggers a resync from a recent snapshot. Luna: Snapshots are interesting. Traditionally, syncing a full Ethereum node from genesis can take days. Coinbase uses periodic state snapshots - think weekly checkpoints - so a new node can be operational in hours.

Lucas: They host those snapshots on a content-addressable storage layer so that any node can fetch them without hitting a centralized server. That also helps with data integrity: each snapshot is hashed, and the hash is verified before the node starts consuming it. Luna: But snapshots only work if they are fresh. A snapshot from two weeks ago still requires replaying a lot of blocks to catch up.

Lucas: True. So they generate snapshots on a rolling basis, and the orchestrator decides which snapshot to use based on the target network's block time. For Ethereum, which produces a block every twelve seconds, they use a snapshot no older than 24 hours. For Solana, which is much faster - around 400-millisecond block times - they use snapshots every few hours.

Luna: Solana is a beast. The data throughput is insane. How do they even store that efficiently? Lucas: They do not store the full transaction history on every node.

Solana's protocol already requires validators to prune old state. Coinbase follows that - they run what are essentially archive nodes only for a handful of high-value networks. For everything else, they run full nodes with pruning enabled, keeping maybe the last 100,000 blocks. Luna: That cuts storage costs dramatically.

But it also means if a customer needs historical data beyond that window, you need a different strategy. Lucas: Right. For that, they built a separate data lake that ingests every block from every network and stores it in Parquet files on object storage. That is the source of truth for backfills and audits.

The nodes themselves are just the hot cache. Luna: So the node layer is a cache. That is a very different mental model from the traditional database world, where the database is the source of truth. Lucas: Exactly.

And that shift in mindset is what allowed them to treat nodes as ephemeral. If a node dies, you do not try to fix it. You kill it, spin up a new one from the latest snapshot, and point the load balancer at it. Total downtime is under a minute.

Luna: That requires a lot of automation. Did they build everything in-house, or did they use existing tools? Lucas: They evaluated a few third-party node management platforms, but nothing gave them the control they needed for the diversity of networks. So they built their own orchestrator in Go, with a plugin system for each chain's specific quirks.

The API exposes a single endpoint: 'give me a node for network X, region Y, with these capabilities.' The orchestrator handles the rest. Luna: What about client diversity? On Ethereum, there are multiple execution clients - Geth, Nethermind, Besu - and they have different performance characteristics.

Lucas: Coinbase runs a mix. They actually use client diversity to avoid single points of failure. If Geth has a bug that stalls all nodes running it, they want some nodes on Nethermind to stay live. The orchestrator assigns a client type per pod based on a weighted distribution that they tune over time.

Luna: That is smart. But it also means more surface area for bugs. You have to test against multiple client implementations. Lucas: They do.

They have a staging environment that mirrors production, and they run each new client release through a battery of integration tests before rolling out. The whole deployment is canary-based: start with 5 percent of nodes, monitor for 24 hours, then ramp up. Luna: Let us talk about the Ethereum merge to proof of stake. That was a massive architectural shift.

How did Coinbase handle it? Lucas: They started preparing over a year in advance. The key change was that each validator now needs to run both a consensus client and an execution client - two separate processes that communicate via a json rpc API. Coinbase rearchitected their node deployment to always pair a consensus client with an execution client, and they added health checks that verify the two are in sync.

Luna: I remember there was a lot of concern about the merge causing exchange downtime. How did they avoid it? Lucas: They ran a shadow fork weeks before the mainnet merge - a full reproduction of the Ethereum network using their own infrastructure. That caught several issues, including a bug where the consensus client was sending invalid attestations because of a clock skew in the container's time zone.

They fixed that before the real merge. Luna: Clock skew. Such a mundane problem, but it could have caused a slashing event. Lucas: Exactly.

And that is the reality of running infrastructure at this scale: the hardest problems are often not the fancy distributed systems puzzles, but the boring stuff like time synchronization, disk I/O contention, and network bandwidth limits. Luna: What about the cost side? Running hundreds of nodes, each with significant compute and storage - that is not cheap. Lucas: They reduced costs by about 30 percent through three main levers.

First, they aggressively right-sized instances. Many networks do not need a full 64-gigabyte machine. They benchmarked each network's peak resource usage and matched instance types precisely. Second, they use spot instances for non-critical nodes - anything that is not handling live customer traffic can run on spot and tolerate interruptions.

Third, they consolidated storage: instead of giving each node its own SSD, they use a distributed file system that allows nodes on the same Kubernetes worker to share a common data volume with caching. Luna: Does that cause I/O contention? Lucas: It can, but they mitigate it with quality of service controls. Each node gets a guaranteed IOPS floor, and the orchestrator schedules nodes so that high-throughput networks like Solana do not share a worker with each other.

Luna: One thing I find interesting is that they chose to keep node operations in-house rather than using a service like Alchemy or Infura. Why not outsource? Lucas: Control and latency. When you are an exchange, every millisecond of delay on a trade matters.

If your node is behind, you might execute a trade at a stale price. Also, if a third-party provider has an outage, your entire platform goes down. Coinbase decided that owning the full stack gives them a competitive advantage. Luna: But that advantage comes with a massive operational burden.

They have to maintain expertise in every chain's protocol. Lucas: They do. They have a dedicated protocol engineering team whose job is to track each network's development, participate in testnets, and update internal tooling before mainnet upgrades. It is a non-trivial investment, but they see it as core to their business.

Luna: And it has paid off. The infrastructure handled the recent surge in on-chain activity during the meme-coin frenzy last spring without any major incidents. Lucas: That is the proof point. They were able to scale node count by 40 percent in a week just by tweaking autoscaling policies.

No manual provisioning, no panicked calls. Luna: So what is next for this platform? They support 200 networks now. Is the goal to support every chain that has any liquidity?

Lucas: They are moving toward a self-service model where any Coinbase team - or even external partners - can request a new network and the platform will spin it up automatically, assuming the chain meets certain security and stability criteria. The vision is 'infrastructure as a service' internally. Luna: That is ambitious. It requires a lot of abstraction, which brings us back to the plugin architecture we mentioned earlier.

Lucas: And it is still a work in progress. The hardest part is handling chains that have fundamentally different consensus mechanisms - like DPoS, PoA, or even bft based - within the same orchestration model. But if anyone can pull it off, Coinbase's engineering team has the track record. Luna: It is a good reminder that even in a fast-moving space like crypto, the fundamentals of infrastructure engineering still apply: automation, monitoring, and treating nodes as cattle, not pets.

Lucas: Exactly. And by the way, if you are building something similar or just thinking through your own infrastructure decisions, that is exactly why we keep this show ad-free. If the content helps, you can support it at buy me a coffee dot com slash fexingo. Luna: And we will keep drilling into the specifics that matter.

Next episode, we are looking at how Stripe rebuilt its identity verification pipeline for global compliance. Lucas: That is a good one. Lots of trade-offs between fraud detection and user friction. See you then.