Many Databases 1 LSM Engine - OpenData

Speaker A: Here's the question that should bother you more than it does. Why does every modern database, your vector store, your time series engine, your log search, your stream processing state store, why do they all reinvent the same distributed systems, plumbing the same replication heartbeats, fencing protocols, same cache hierarchies, and then charge you more? My guest today is Almog ah Gavra, uh, uh, co founder of Responsive and he's convinced that the most expensive part of your database bill is in the hardware. It's the operational tax on running distributed systems that all look suspiciously alike under the hood. So he's building open data, which is basically one object storage native LSM foundation and many databases on top. The object storage native LSM foundation is provided by Slate DB Vector, time series logs, key values, all sharing the same storage engine, the same tuning knobs, the same failure modes, all on top of LSM foundation. So we get into compare and set fencing on S3. Why single writer is actually easier to make highly available than Raft. Um, LSM trees actually provide you tunable dial between read write and space amplification, and the one architectural decision that gives you free database snapshots almost as a side effect. So if you have ever been on call for different variety of databases and you understand the operational pain, this episode is for you. Before we go, please hit the like button if you like this episode. If you haven't subscribed to the channel, please do that and share it more. So everyone learns about databases and distributed systems. Let's get into it. Hey, Al Moh, welcome to the show. I'm really excited to have you today. Such an interesting project to discuss that you've been working on. So we'd love to learn more about it, the philosophy behind it, the technical aspects, the architecture, some of the use cases. Let's start with a little bit of introduction about yourself, about responsive, and you know, a little bit about how you got to be working with responsive and on open data.

Speaker B: Yeah, thanks for having me on the show kv. I've watched quite a few of the episodes. They're all excellent. Most recently I watched the one with uh, Kishore from Star Tree. Uh, he's a brilliant guy and you did a great job interviewing him. So hopefully I can give you guys similar level of entertainment and information about me and kind of my background, how I got to responsive and open data and all the things we're working on now. Um, I started my career at LinkedIn about a decade ago working on search infrastructure and I've kind of been working on distributed systems ever since then, for search specifically, I worked on the core search engine as well as the ingestion into the search engine. Uh, that got me into the Kafka space. Um, one of the biggest problems with search at LinkedIn was actually making sure that we can ingest and index all of that data live. And kind of one of the biggest problems is that if people create a connection and then they immediately go search for that connection, they want that connection to show up in search. So there's basically a lot of things we had to do to make sure that that type of workflow worked. All of that introduced me to Kafka, which then eventually brought me to Confluent, where I spent most of my time on stream processing. So I worked on KSQL and Kafka Streams a little bit on the Kafka, uh, kind of brokers themselves, not too much. I think one of the biggest challenges of Kafka Streams though was how do you provide that as a service? Because it's really a heavyweight client library that clients run and kind of deploy that own infrastructure. And even though you can manage, you know, you can provide a managed Kafka service, which, you know, there were quite a few of at the time. Confluent was obviously the leading one. The Kafka streams applications themselves were challenging. So at, uh, Responsive, we started the company in order to basically how do you make a client side library easy to operate? And the kind of conclusion we came to, which it's not groundbreaking, a lot of people kind of figure this out, is that compute like is a difficult thing to run, but it's mostly solved with kubernetes and all these things. You get pretty nice scaling and you know, pretty smooth auto scaling from your compute instances. Uh, the problem really comes when you couple that with storage. And that's how kind of Kafka Streams architecture works. That's how Flink architecture works. Both of them coupled the compute with RocksDB storage because you really wanted this like low latency materialization of a topic, kind of of a Kafka topic into, in your local state in order to be able to run the computations that you wanted to run. This caused a whole bunch of problems like you can't easily scale out because if you do you have to shuffle a bunch of data. If one of your nodes crash, you have to rest all of that data from some source of truth that was replicated elsewhere. And this makes the whole architecture like incredibly complicated. So Responsive really started with this one idea of like, okay, let's just separate compute and storage have uh, compute nodes run and then basically just make sure that you have the caching strategies, the bashing strategies that you need in order to be able to like process chunks of data locally, commit them to Kafka and then commit them to your, your remote storage like in line so that you don't have any of those problems. So that's where responsive started. We had, you know, varying degrees of success. I think the architecture was, was wildly successful. We had some really big customers that ran terabytes of storage with multiple tens of thousands of events per second. Uh, and they worked without a hitch. You know, we were able to auto scale. I think the reality that we came to, which ended up causing us to pivot was that low latency, high mission critical use cases are actually extremely rare.

Speaker A: Right.

Speaker B: Most stream processing use cases that we saw out there, and especially when we were talking to a bunch of customers, were primarily data pipelines where you can actually tolerate like an hour or two of downtime. It sucks. Maybe you're like search relevance is low. Maybe, you know, going back to that example I had with LinkedIn, some people can't immediately find the new connections that they were connecting to, but the experience degrades in a way that mostly still works. Like, you know, you can still find 99% of the data out there using search or whatever. So it's really just these like fraud detection or, or real time billing use cases that needed this kind of high scale, high throughput, low latency with extremely high availability guarantees. Um, as part of this journey we were building, we noticed that the kind of biggest cost angle for us and the biggest operational overhead for us running this was actually the stateful storage system. Right. We got the compute, as I said, pretty simple. But the stateful storage was both extremely expensive and extremely difficult to operate. So during that time we were talking to Chris Riccimini, who's kind of, you know, now author of the second edition of the uh, with Martin Kleppman of the second edition of Designing Data Intensive Applications. Awesome book. If anyone of your audience hasn't, hasn't read that yet, I highly recommend it. So we were talking to him about this problem because we were using Scylla, which is a great technology, but uh, you know, it's basically a rewrite of Cassandra in a way that's like a little bit more performant in certain bits and bits and stuff. I won't get into too much detail there's. But we were using that and that was a massive portion of our bill. And we've seen a lot of this like, architecture coming online, like Warp Stream and you know, these systems that really leverage object storage as the primary source of durability. And we wanted to build something for Kafka streams where the State Store could be offloaded onto object storage. So that was the start of slatedB. And why we started building that because it was like, oh, well if we can do this, then it's a very similar model to the embedded rocksdb except everything's backed up to cloud and you can actually scale in, scale out and restart with only cache misses. So sure, you'll be processing a little bit slower during a restart of a node if you need to like rebootstrap your cache, but you can continue processing and you know you'll be able to process while you're warming up that cache. So that led to Slate DB and kind of this architecture which uh, is a very long winded way to get us to where we are now, which is basically, we noticed that most Object Store native architectures out there end up relying on something like an LSM tree, right? So if you look at uh, Warp Stream, if you look at turbopuffer, if you look at even quick Wit, all of these have something that looks a lot like an LSM tree in the root. And we can get to the technical reasons for why, but with that realization we decided that kind of perhaps a more compelling thing for us to build rather than specifically look into how do you replace Kafka Stream State storage is how do you build a framework for building new data systems on top of LSM trees? And that's effectively how we got to open data. Uh, and I'll pause there because I gave a very long winded spiel there. But I'll get to also why I think that's kind of interesting from an economic perspective as well. I don't know if you have any questions about how we got to there.

Speaker A: Yeah, sure. Um, so yeah, that's an interesting background. A lot of experience lies in databases, managing databases, stream processing and so on. And um, I've also read the Manifesto of Open Data, uh, and it makes a lot of sense, but there are some, you know, nuances that I would like to discuss today. Like one of the first things that you also mentioned that, you know, operating databases is a tough thing, right? And uh, it requires a lot of engineers, a lot of experienced engineers, and also cost and it's not easy. It ends up impacting your customers, uh, as well, depending on what kind of themes you have and what kind of databases you Have. So I think the argument behind the manifesto is that a lot of operational problems and cost can be abstracted away so that they stay similar even if the type of database changes. And also the cost reduces because of the usage of object storage. And I think that makes sense. But I would like to understand because for a lot of databases, the query engine is the mode for them, for example, Snowflake or Clickhouse, uh, vectorized execution and so on. So is it not going to be that important when people start moving to object storage? Or are we saying that the query layer itself, the surface area of the query layer itself is going to shrink so that that is not going to be the main problem or main operational problem for managing or running a database. So are we saying that is going to happen or are we already seeing it with existing databases?

Speaker B: Yeah, that's a great question. So I think there, there are a few things kind of baked in there. Let's. So let me kind of break that down into three different questions, if I understood it correctly. The first is like, how is it that we can use a similar storage engine for many different types of databases? The second there was how much of a moat is kind of in the query engine itself? And then third is kind of where do we see the puck going as far as databases and where they're going? And I think a lot is changing recently. The two obvious headwinds or uh, tailwinds are, um, the rise of popularity of object storage, then obviously the ability to use AI to dramatically, you know, speed up your coding and development. Um, so let's start with the beginning, which is, uh, how is it possible to basically have many different data systems on top of the same storage? And why might we want to do that? Right, like you mentioned, the operational burdens and things like that. And this kind of, I guess answers part of both the first two questions, uh, which is that the query engine has a lot of smart in it, but once it works, it more or less works. And this is kind of what we've seen, you know, across the board with whether it's stream processing, search, or recently like we just released the time series engine, which is that it's easy to test a compute bottleneck system because you have inputs and you have outputs and the compute is more or less like always the same. It runs on a node and then that node does this computation and there's a lot of like extremely complicated logic that goes into it, like vectorized compute or you know, like these really complicated query planning and you know, physical plan building and Logical plan, building things. But once it works, at least in my experience, it kind of works really. Where all of the hard problems are in databases, in my opinion is in the distributed system side, right? Because that's where a lot of like unpredictable things happen, right? Like a single machine more or less does what you tell it to do. When you start having two or three machines need to talk to each other, then they start doing things that you're, that are weird. You know, packets might drop when you're connecting to one another. You might think one of the nodes is online so you behave in a certain way, but it's actually not. Um, a lot of these kind of tough problems come up from the distributed systems aspect of kind of building databases. Um, more so in my opinion than the query engine itself, which is just kind of like a really, it builds itself on itself quite a bit, but it's a really kind of well contained module. So that's kind of the first realization. And then the second realization is that all the operational problems downstream of that a lot of them come from operating distributed systems. So like you know, tuning your heartbeats, tuning your, scaling up and scaling down, tuning, replication, figuring out what goes, like how to um, uh, how to handle stale uh, data or out of sync replicas. A lot of the problems in tuning knobs that you have come from those configurations. And that's really kind of what we've noticed with all of these systems in production that once you get the query engine more or less working, it's stable and then the storage part is where it gets complicated. So that leads us to the kind of object store native first architecture. I think there's a few really big benefits from that architecture. The first of it is that, well, object storage solves all of these hard replication problems for you so that you can build a database that kind of treats all of these replication and distributed system problems as a black box. And then you can kind of, you know, punt that problem off to the hundreds of AWS engineers that are running S3 for you. So if you can build a database that relies on that, and that's not easy, we'll get to that in a second. Many of the hard problems are solved and you can really think about your system almost like a single node system that's not replicating, it's not doing any fan out, it's not doing any of these kind of complicated systems, uh, behaviors. So then the question is, and you know, we're already seeing this play out like you know, there's warp stream, which is like A Kafka alternative, uh, which builds, you know, the Kafka protocol on top of object storage. TurboPuffer, which is vector, that builds it on top of object storage and you know, quick wit did log search. So it's kind of becoming clear that it's possible to build these types of systems that you know, really don't have to handle many of the distributed systems aspects of like traditional storage systems. Um, but the second question is, can you build them on the same foundation and if you can, does it help? So that gets to the Open Data manifesto which really I think when you look at the cost of running a database, there's two portions of that cost. There's the hardware cost, right? Like the literal machines that you have to run or you know, aws what it charges you, the networking cost, the memory compute, but there's also the operational cost which is like hiring engineers to actually run it and keep it online for you. And that second thing is a little bit harder to measure, right? It's like, okay, well you know, it costs me this much and you know, well I could hire one more engineer. I could not or maybe I can get away without it. So the best way that I have found to estimate that cost is to look up the markups m that the vendors basically charge to sell you for it. So like if you buy, you know, Kafka managed by Confluent or if you buy like any other of these infrastructure products, they typically give somewhere between like a 60 and 80% markup on um, the underlying hardware. So what they're saying is that, you know, two things they're saying, one, we believe that it is worth it for you to spend that much more than the underlying hardware to get our management expertise. And two, they're saying we can actually offer it cheaper than you would even yourself because we have multi tenancy and so on and so forth, right? So if you look into that, what that tells me is that the most interesting part of reducing the cost of databases is actually not reducing the hardware cost of databases. It's seeing if we can reduce that operational cost of databases. Because if you cut the hardware cost by 50% which is dramatic, that's a whole new generation of architecture. If you can really 2x the performance out of the mature database, if you can do that, that'll only save you 15% of your total bill. But if you can cut the operations down by 50%, right, which looks like it's possible with all this object storage kind of based uh, uh, architecture, then you cut your overall bill by 35%, which is more than 2x what it would be if you cut the hardware cost down. So that's kind of, uh, the heart of Open Data, which is, what can we do that builds a system that really is simple enough for people to actually just run on them, run themselves, that they don't need a managed service to run, but. And if you can do that, you can cut the biggest part of the operations. Now, of course, we also want to do it in a way that reduces the hardware cost. We don't want to have an extremely inefficient system that's easy to run. But that's kind of the motivation behind Open Data. Now, how do you unlock that? How do you get a system that's actually dramatically easier to operate? I, uh, hinted at one of them is object storage. A lot of the distributed system problems go away. But the second thing that we noticed is that all of these systems have very similar underlying architectures, but expose it in a completely different way to you, right? So just because you know how to tune Warp Stream doesn't mean you can then go pick up turbopuffer and know how to tune turbopuffer. Just like Kafka and Cassandra, both of them actually had pretty similar kind of concepts around replication and partitioning and all of these things. But if you tried to like pick up one and become an expert in it, almost none of that expertise then transfers to operating the other system. So then that 70% kind of operational burden compounds with every system that you need. So if you're running a full company and you need a vector database and a log database and a time series database, well, now you need to actually build up expertise in every single one of them. So the next step is, can we actually amortize that cost? So if you can build one foundational layer that's shared across all of these systems, then you need to learn the query engines for each of them. But the knobs for tuning the hard stuff, the distributed system stuff, are actually shared. So you already, like, once you've learned log or vector, you can then move to time series and say, oh, look, I know, I understand how heart beating works. I understand how the flushing mechanism M works, I understand how to tune my, like read and write throughput or my reader replicas. It's all the same. All I have to learn is the query engine part, which is, you know, a big deal. I'd say it's probably like tuning to query engine is like 30% of the work, maybe 40% of the work, but it kind of transfers over. So in aggregate you not only save that like 70% operational burden for each database, but hopefully the theory is that you can save it across all of your databases. So that's really what led to the rise of kind of open data and the philosophy and the manifesto that we have there.

Speaker A: Yeah, I think it makes sense. And since you mentioned like turboper and Warp Stream and databases that are built on top of object storage, um, so as a, let's say as a user or as a company, let's say I have a use case for stream processing, I have a use case for vector. So to make that decision, where should I go to like open data path or should I look for warpstream and turbopuffer which are more specialized systems into one? Um, what is the like the decision making or the mental model one should have when it comes to like using really specialized databases versus hosting like a portfolio of databases where multiple databases can be hosted with similar operational challenges and the tuning required is going to be low. But it's still operational workload or operational stuff that I need to know. So what is the mental model that companies should be following or uh, are following currently?

Speaker B: Yeah, so I think the way that I like to kind of think about the database space and how the different systems are laid out, I like to split it into like three domains. There's like the oltp, you know, core database typically like postgres.

Speaker A: Right.

Speaker B: And that works up until a certain scale. And I think if you can get away with that you should. Right? Like a lot. There's been a lot of articles recently that came out which was like just use postgres for basically everything and at smaller uh, scales that just works and that keeps your life really simple. Right. If you can do that, get away with that. The M all the way on the other end of the spectrum are like the data warehousing use cases, right. Extremely high scale. Dump all of your data in there, run a really, you know, a bunch of like analytical workloads and, and you know, kind of have that managed there. And I think the systems there are typically they benefit a ton from like multi tenancy because the you know, compute workloads are typically a little bit like more spiky. You know, maybe they're daily, you're running like jobs. It's so huge multi tenancy. Shifting all of those workloads to an existing vendor kind of makes sense. Then there's the category in the middle which is kind of what I think about like scaling out your application infrastructure.

Speaker A: Right.

Speaker B: And this is where like vector search databases, um, you know, time series databases, all of these kind of lie. And that gets to your question, which is when would you want to choose something like open data versus a specialized product? And I think there's always going to be room for both of them. But the way that I think about it is that there's like this, uh, I like to call it the scale complexity inversion principle, which is that as you go up in scale, you necessarily have a more complex system in order to manage that scale. And there's a little bit of this 80, 20% rule, which is that if you are at that 20% top use case that really needs to drive the infrastructure to a next level of scale that is not typical, then the niche systems are always going to be ahead. But what you pay for that is typically in complexity, right? So like I would say Warp Stream and turbopuffer are an order of magnitude simpler in complexity than predecessors. But because they are dedicated systems, you know, they are necessarily going to be more complicated than the open data alternative for it. But they can really push those bits, push networking, push CPU for the specialized use case better than what we'll be able to do. So the idea here, again, going back to the kind of the root of this problem, is that if you are that P90 use case, you probably want a specialized system. But most use cases out there can get away with a non specialized system. And I saw this a lot kind of when we were working in Kafka streams, in Kafka, many people just liked the Kafka API, but they were pushing like kilobytes of data per second. So using Kafka was really overkill, right? What they needed from Kafka and what they really liked was, was the fact that you got high availability, you got replication, you got all of these wonderful things like right out of the box, even if you didn't need the scale. And there was no really alternative for that, right? You either choose this distributed system that was built for these like P99 LinkedIn use cases, or you choose a single node system that doesn't really give you the option to get rep, like replication and high availability and all of that. So this middle ground, it just didn't exist, right? Because you basically, if you're already building a system that like replicates and solves all these hard problems, you might as well build the system that solves that P99 use case. So that gets to kind of answer your question specifically, which is, what are we building open data for? We're solving it for the like 80% use case where we think honestly most of these use cases can probably fit on a single node. Right? Single nodes can handle terabytes of data now pretty easily, especially if you just need to store the warm cache part of it. Um, and uh, we can get into that when we get into the architecture. But if you're one of those, you can typically get away with a much simpler system, uh, and it'll end up costing you less. That's kind of the way that I like to think about it. Where you draw the line specifically. You kind of have to go and test the systems and see where you're, you know, where your workload lands.

Speaker A: Yeah, yeah, I think there's a lot of nuance. But at a high level this mental model makes sense. And um, talking about like different layers of abstraction, right? Uh, so if I understand correctly, open data is abstracting away the storage part of it, right? So it's much lower level of abstraction. Whereas something like Ellock Data, uh, which tries to create a common data substrate, uh, which has like query engine, the buffer pool management, the transaction management and durability guarantees, which stays pretty much the same across different databases. And what they do is they plug in like different storage engines like Cassandra, DynamoDB, Postgres and so on. And then on top they have like different front end mechanisms too so they can plug. So they have like a higher level of abstraction, not at the storage level, but the query processing and transaction management. And what you have is a lower level of abstraction which is at the storage level. So first of all, what do you think of these two abstraction levels? Second question, once open data, uh, gets the shape it wants and you know, gets really popular, is the next step having something like an open engine where there is also a higher level of abstraction, where other parts of databases which are common, like transactions, wall management and so on, they also get, get to that layer and get the abstraction. So it even reduces the operational cost and the burden further.

Speaker B: Yeah, that's a good question. So yeah, let's get into the architecture of open data because I think there is quite a bit of overlap between what we're doing and kind of what Ellock Data is doing. As far as the way that I understand it, I'm not super familiar with what they're doing, so don't take what I say as comments on their architecture. But the way that we view the architecture for Slate DB for open Data is that we've been spending basically the last two years developing Slate DB to be the kind of shared storage foundation um, we chose to build a new storage foundation instead of using something off the shelf that already exists because once. And uh, this is something that actually Kishore said on one of your episodes that like every 10 years there's a new advancement in kind of technology that requires you to rethink your systems. Right. Going from disk to ssd. And he even mentioned going from SSD to object storage.

Speaker A: Right.

Speaker B: And I think there simply wasn't anything out there that we could use as a really object store native foundation for all of these different systems. So if we were to pick something like Cassandra off the bat, we wouldn't be able to leverage object storage in the way that we wanted to, where it is the true only source of durability. It is the only kind of thing that does replication for us. We don't manage any of that ourselves. Uh, in order to do that we had to kind of build a new storage layer, the first iteration of it. We actually wanted to just use RocksDB off the bat and plug in kind of instead of a file system, a wrapper kind of over S3 that would delegate everything to it. But it turns out that when you want to build a high performance database like RocksDB, which is embedded key value store for people who aren't familiar, you make a lot of dependence like you make, you take a lot of dependency and assumptions on the underlying systems that you're depending on. So like they had a lot of page cache management things and a lot of like, you know, OS buffering that they kind of depended on. And when we tried to plug um, concept in, we uh, noticed ourselves trying to fill so many gaps because S3 just couldn't handle that. You know, maybe with S3 files it's worth revisiting this and seeing how it goes. But my suspicion, and kind of my intuition tells me that you really need to design a database for the underlying primitives that are available for you instead of just trying to like, you know, squeeze a system that was designed for the last generation of things. So we would have been happy taking something off the shelf. It just didn't seem like anything existed at the time. So then when it comes to, okay, well what do we want to choose for the underlying storage system we wanted. Actually, uh, let me answer the second part of your question first, which is like, where, where are the other levels of abstraction? So slatedb has, at its core, it's basically an LSM tree management system. So it's like for people who aren't aware, an LSM tree is effectively just a way to ingest and query data in a way where you're only ever creating immutable files. Unlike a B tree, which is the other kind of popular uh, indexing structure for key value storage. An LSM tree writes immutable files and then merges them later. Whereas a B tree will replace kind of a block within a file whenever you make enough updates to that file. And it's all about read write and space amplification kind of characteristics. The LSM tree has excellent read amplification characteristics, um, uh, write amplification characteristics because you're basically just like writing the data immutably and then later you're changing it so you're not rewriting data as often as you would. In a B tree where you're constantly rewriting any kind of touched blocks where you pay for that is in read amplification because you have a bunch of immutable blocks, there might be data that's kind of overshadowing old data. So you basically have to just scan the uh, SSTs until you find the data that you need and that might end up reading a lot more kind of data that you don't need to read. Uh, so that is what slatedb is at its core. It also offers on top of that a few other things. So it gives you a wall which is really kind of object store native uh, it gives you a transactional object store uh, mechanism where basically you can leverage the object store compare and set mechanism which is now recently available. I say like in the last year available in S3 where you can actually like ensure that you can fence other nodes that are writing the same data. I can get into more detail there later. I think it's, it's a pretty interesting protocol. It also gives you like a multi version concurrency control MVCC mechanism for, with like transactions within Slate db and it gives you a few other kind of primitives. So that is the common layer that we see for open data. Open data itself is a project that basically maps a whole bunch of other database types, you know, vector time series, log key value graph onto a key value model. If you look at the existing databases, many of them do exactly this, right? Like if you look at Prometheus and time series, it doesn't have a, an explicit key value engine underneath the hood, but it looks really similar to that, right? Like all of the uh, inverted indices that they use are actually keyed by the terms or the like forward index they use are keyed by series. And then you look things up in keys because this is really the only on disk Storage system that works, it kind of looks, everything ends up looking kind of like a key value store in one way or another. Even columnar engines which we can get into. And that's a full diversion that we can get into later. But open data just maps all of these different databases onto a key value structure. So like, if you look at the actual open data code on GitHub, um, the first RFC that we have for each one of those is, is a really detailed storage layout which are like, what do you need to store in order to serve the query workload? So for like time series, and I'll have a detailed blog coming out about this in a little bit, you basically need the samples. So all the measurements that you've taken, you need an inverted index which says, okay, given these labels, like container is abc, what series exist for that container? That way you can really efficiently intersect, uh, different things to find like, okay, I need this label set and this label and it needs a forward index which is basically, okay, well now that I found the series that I need, I want to aggregate them together. So I need to know what labels are in or exist in these series and then I need to group based on those label sets. Once you have those three primitives mapped onto a key value store, then you can just build the query engine just like you would if you had kind of native storage for these concepts underneath the hood. So that is what open data is. Now to get to your question, kind of comparing it, contrasting it to a little bit more of like a higher level abstraction. This I think really leads in, well to what we were talking about earlier, which is where do you want your, your shared complexity to be right? And where do you want the kind of system specific knowledge to be? Our hypothesis is that anything that kind of interacts directly with the storage layer should be shared across the systems because those are, you know, really well abstracted into this key value model. However, a lot of the like system specific caches or the system specific workloads really depend on the query workload, right? Like a log, um, almost append, only read from tail workload looks very different from a time series one which ends up, you know, doing a lot of kind of random reads of series which ends up looking very different from a vector one because vector always scans all of the data as opposed to time series where you typically scan just like a window of your data that's organized by time. So all of those have kind of very different query characteristics. So we thought that a lot of the things related to the query engine so the way that you choose to set up your cache, the way that you choose to set up your compaction policies, things like that, should live specific to the database itself that we're building. Um, did that answer your question?

Speaker A: Yes, I think so. It did. And thanks for giving that a view of the architecture as well, because it makes sense. Because I was also thinking, like, even if the underlying storage engine, which is the Slate DB LSM engine, is the same, the workloads I'm serving, the customers I'm serving, and the expectations that they have are completely different when it comes to time series and key value and vectors and something else. So are we saying that apart from all the storage layer understanding and the operational burden, you still need to understand the query patterns and tune accordingly, but the interface for you to tune, all those things are going to be the same, Is that what you're saying?

Speaker B: Yeah. I'm so glad you asked this question, because I think this is my favorite thing about LSM trees in general. There's this awesome paper out there by Niv Davian, I think it's his last name, it's called, uh, Dotovsky. And it basically describes an LSM tree not as a data structure, but as a mechanism for tuning between read write and space amplification. Right? And it kind of. This is a little bit of a long winded way to answer your. Answer your question, but basically the thesis of that paper is that on one end of the spectrum you have the best system of all time for write amplification. And that's just a log, right? Basically, you only write new files, you never compact, you never delete old files. But that system is the worst system in the world for reads, because whenever you want to find a key, you have to search all of the log files you've ever written. On the other end of the spectrum is the best system in the world for read amplification. It's just a sorted array. But that's the worst for write amplification, because every time you write new data, you rewrite your entire sorted array in order to keep it sorted, basically, because it's just one file. And now the question is, can you build one data structure that allows you to tune between these two, um, kind of extreme endpoints? And that's what an LSM tree really is. If you look at the structure of an LSM tree, the top of the LSM tree is a log, and then the bottom of the LSM tree are basically different levels of sorted arrays that you rewrite with increasing levels of frequency. What's really cool about this is that you can really tune it for a wide variety of use cases with a kind of out of band process that's called compaction. You can choose how often you run it and you can choose what the shape of the data looks like once you have this LSM tree data structure set up. Getting to your questions about how can you tune for different workloads, I think there's a split where, and this is why we don't have like one database that does everything, but we chose to actually expose dedicated like single purpose systems instead of having like one database that just does everything for you. And that split is that about, I'd say like 70% of that tuning work is common for a workload. So like uh, let's take time series as an example. Almost everybody who runs a time series engine like Prometheus for their metrics kind of has a very similar workload where 90% of the time you're looking at the last like 24 hours of data, right? And you're typically scanning that data in order to plot graphs and you're aggregating in a very specific way. Sometimes you want historical queries that look like at longer windows in order to find things typically you're also okay if those queries are a little slower, right? So that profile of use case is pretty constant across lots of different uh, kind of time series workflows. And the way that you map that to an LSM tree is via the compaction policy. So what we do is there's this different compaction policy that's called like time windowed compaction strategy. In Cassandra, uh, we're building something very similar to that in Slate that basically says I'm going to compact data in time windowed chunks. Right? So what that means is that as soon as data gets like two hours old, I know that no new data is coming in, right? At that point I'm not getting like metrics that were two hours old at that point. Like most people just you know, ignore the data that comes in from two hours ago because you already have most of the data that you want. So you can structure your LSM tree in a way where the older data is just a log that's compacted nicely because you know that data is never going to change. So you can target that section of the LSM tree for queries. However, for vector, the kind of structure looks a lot different. You always want to maintain a very kind of read optimized data structure so that you can have kind of very effective recall in your queries. So then you have a different compaction strategy, typically like a size tiered compaction strategy that makes sure that your tree never grows too far so that you can quickly and efficiently find the vectors that you need, or like the centroids that you need to serve the queries that you want. And you keep a section of that kind of really hot in memory. So a lot of those tuning kind of knobs are baked in to the vertical database that we're doing. However, once you understand those knobs, the like read, write and space amplification knobs. So for example, like, let me get, you know, specific one of those knobs is like how often you flush an L0 SST. So the way LSM trees work, you basically buffer some data in memory and then you flush that as a sorted array to disk in what's called an L0. How often you flush those makes a pretty big difference on your performance structure. The more often you flush them, the less memory you need, because you can flush, you know, from memory to disk faster, but the more pressure you're putting on your read side. Because if you have a lot of l zeros, you might need to scan all of them in basically what looks like a log, right where you need to kind of consult all of your l zeros because none of them have been compacted with any other data. So your read profile is going to look a little worse. Once you understand this, you can know, okay, I understand how to tune my L0 max SST size. That concept is the same. And I can apply that to all of the different systems that I have. Uh, and, you know, I can get into some other kind of common tuning knobs. Things like how, what happens when, like your l zeros fill up and you get write stalls? How can I manage that? How often do I want to run compactions? Things like that are all kind of common new knobs that apply. Even though the fundamental LSM structure looks very different for different use cases. I hope that that kind of covers it.

Speaker A: Does, it does make sense. So it's like you have like a, uh, entire range of options, as in on a spectrum, but they are similar options. What you have to tune and you can go from like the leftmost end to the rightmost end and get different guarantees, uh, read performance, write performance, and so on. And then you can also tune it specific to given use case, for example, time series and key value and vector, as you mentioned. And those are great examples because, like, taking the domain understanding of, as you said, like, no one is going to be, you know, ingesting two hours old data or even five hours old data. And that depends on the, of course, the use case and the domain. So getting that domain understanding and having fewer tuning options makes a lot of sense because then I know my domain, but I may not be knowing a lot about the databases. But if I have fewer options, I think that gives me the power to tune and test quickly because those options are not changing. Like one problem I have had with let's say systems like Kafka, they are great system, but there are just so many tuning options available. And if I have so many tuning options for different technologies, it gets, you know, exponentially difficult for me to understand. So I think I understand, you know, the abstraction layer. I think it makes a lot of sense. And taking that domain knowledge into the tuning and then testing on top of it, it makes a lot of sense. But even if I have, let's say, an understanding of my domain and I'm able to tune these parameters, how do I connect the dots when it comes to the cost aspect of it? For example, tuning the, how frequently the compaction is done may mean that we are calling more number of times to the object store, for example S3, so that in a way translate to the cost of gets and puts on S3, right? So how do I get to that level of understanding and what kind of transparency do I have when it comes to cost with tuning of these, uh, different knobs?

Speaker B: Yeah, yeah, that. Good question. Actually, a big thing that's influenced my thinking about this is that when Ryan and Richie were on your podcast, they talked about specifically about this, right? Like what knobs do you want to actually expose to your users and this problem of like config overload. And they mentioned something that stuck with me, which was that you want to expose the knobs that really affect your cost profile, right? Because then it becomes a choice to the user of what trade offs do I want for my workload? Am I willing to pay more to get stronger performance characteristics or is this workload less important for me? And I just want to save money. So that's really stuck with me. And I think that's the most important configurations that you want to offer to your user and other configurations you only kind of want to be there in case of emergency, right? You have an incident, you need a knob to tune in order to get out of that incident. And those are kind of the two knobs that I think are the most important, uh, basically to offer and to have well documented. I also think that this is a section, you know, I don't want to get too much into AI, but I think this is a section that agents can really help with because the configuration overload problem is like a context problem for humans, right? Which is like, how do I know which config to tune in this section? And then also, well, how do these configs interact with one another, right? Storing all of those is like an N squared problem in your mind because you're like, okay, I need all of these configs and how they all interact with one another. But I do think that like, well documented configs and ones that like, have minimal interactions between one another is something that like, agents can help you a lot with during an incident. So putting that aside, like, which knobs do you want to think about in order for tuning your cost profile? I think the first thing to remember is that the goal of open data is first and foremost to reduce the operational overhead, right? Because even if your cost profile from your hardware is a little higher, if you can save that cost from the operational burden, it more than makes up typically for the hardware cost, right? So maybe that's kind of the first thing to think about. And then if you come at it from that perspective, right, Then it's like, okay, well, now we want to tune the knob. We want to expose knobs that allow you to tune the hardware without making it more complicated to like run the system. And that all just kind of comes down to. And again, this is the benefit of, I think like not abstracting away storage and really kind of diving deep in really depending on one storage backend of Slate db, which is that, you know, you're running with object storage, right? And object storage has a few kind of cost trade offs that you have. Chris wrote an awesome article about this. He called it the, um, uh, uh, the cloud, I think, um, triad latency, cost and durability, right? And those are kind of where you want to tune your costs. And basically what, what he was saying is that you can pick two. I want low, right, latency, I want high durability guarantees, or, and I want low cost, right? Those are kind of what you get to choose from. And that's the, and everything behind that is kind of where your, your interesting knobs come along and let me kind of flush that out in a little bit more detail. Basically, if you want low latency and you want that data to be, uh, durable as quickly as possible, then you have to have a ton of S3 puts, which is just going to cost you money, right? So that's going to drive that aspect down. However, if you're okay with kind of loosey goosey durability guarantees, but you want to have extremely fast write latency, then what you do is you simply ack requests before they're durable, right? And this sounds crazy, like why would you want to do that? But actually a lot of systems can do that, especially if you have something durable up front, right? And that's a classic like Kafka architecture where you have one system of record that's extremely durable and you pay for it, but then everything downstream of that system of record can actually deal with laxer durability guarantees that because if you fail a write, you just go back to your system of record and re ingest it, right? So this allows you to pay once for that durability guarantee and later down the line you don't pay for that durability guarantee, but you can still have low latency ingest. You can still query that data extremely quickly. So those are kind of the most important knobs on the right side and how you want to tune those and how that kind of translates to the literal config knobs that are available in open data. On the other end of the spectrum are your reading guarantees, right? And I think this is actually where most of the cost ends up coming and where the biggest range of trade offs are. Because what this object storage architecture gives you, right, uh, and this kind of gets to the tail latencies that I think you mentioned at some point earlier on is that it really gives you this flexible trade off of everything just becomes a cache, right? So how much cache do I want to keep warm in order to serve low latency queries, right? Like I see a lot of people online comparing object storage native systems with full NVME backed systems and they're like, look, these object storage systems can never keep up with these NVME backed systems because a cold miss is just catastrophic. And that's true. But I view that as a knob that you can tune in the object storage system where you cannot tune that in the NVME system. So for example, if you want an extremely high performance object, uh, store baked, uh, backed time series engine, you can simply keep the entire thing warm on NVMe, right? And you we have a system baked into slatedb that allows you to prefetch SSTS. So basically what you do is you would have two replicas that are warm that are constantly fetching the SSTS onto your NVMe disk and then boom, you have zero cold misses to S3, of course you pay for that, right? You pay for that in NVME storage. You pay for that in your large memory caches. You pay for that in kind of more aggressive fetches from S3, which is more data transfers and more like S3 get requests. But you have the knobs to choose that on the other end of the spectrum, let's say you really don't care about your latency because you're spinning up a one off reader that is doing one like batch job that wants a big historical query and you want to just, you know, have. You can run it with basically no disk, right? And just process that data as it comes directly from S3. You're paying a little bit more in S3 gets. But you can run that on like a stupid spot instance, right? Because you know, you don't really care about having that NVME hot for you. So I think summarizing that and answering your question specifically, the write side of the pipeline is all about tuning between latency, cost and durability. On the read side of the pipeline, it's all how much resources do you want to throw at your cache and how much work do you want to spend keeping that cache warm? But the scale that you can tune between is really like a much wider spectrum than you were ever able to tune beforehand when you kind of were forced into NVME replicated disks.

Speaker A: Makes sense.

Speaker B: Yeah.

Speaker A: I've also read some of the articles where the comparison is made between object storage and you know, NVME based. Talking specifically about open data. Is this the cache that we are talking about? Is this some um, like slate DB tuning knob or is it a higher level, let's say open data, time series database or some other database. So do I get these knobs specific to certain databases or is it a common knob that I can tune for at the slate DB?

Speaker B: Yeah, uh, it's both. I think caching is probably the most critical part of any object store native system. Because if you want a high performance object store native system, a coldness kills you, right? Like if you look at all of these architectures, Turbopuffer and Warp Stream and whatever, they spend a lot of time talking about all the different cache layers and how they can maintain the cache and how you can make sure you keep the hot data set in cache. So that really means that this is like one of the most critical problems for you to understand in object store native world and give the options to tune those. So right now we have kind of three and a half layers of caches the first layer of caching is what we call the object store cache. And this cache is like very wide chunks of data, right? Like, you know, it's kind of indexed in 4 megabyte chunks. And whenever you make a request to object storage, it caches 4 megabytes. So that a reduces the amount of like s, uh three calls that you get. Of course, it makes all of your S3 calls a little bit more expensive, but it's kind of a very, you know, blunt cache. One layer of caching. On top of that we have what we call the block cache. So blocks are four kilobyte chunks in Slate DB and they're like a little bit more, you know, special purpose and there's like a little. And we can cache those in an already decoded deserialized way. So that access to the block cache is much faster than access to the like, uh, object storage cache, which is much faster than access to direct object storage. Then on top of that we have individual caches that are baked into each system, right? And then these are kind of tailored to the use case that you have. So in Vector, for example, having access to the centroid graph is critical because every single query basically needs to traverse that centroid graph and it's about 1/100th of the size of the entire data set. So that gives you kind of this understanding that you want that cached 100% of the time and you want it cached very aggressively because every hit is going to get there. So that is actually kept typically in memory, right? Like that is a dedicated memory layer of this cache for time series. You want to cache basically a few things that are used for all of your queries. So typically you want the last like X hours of your indexes cached because every query is going to hit those indexes. You might not need the samples cache because once you have the indexes, you can parallel fetch from S3 very aggressively and you can saturate your bandwidth just like fetching samples from S3. And you know, the latency characteristics of those queries are typically okay for that. So the way that I see this going, this is just the beginning. The way that I see this going for caching in general is that that's probably going to be the biggest kind of per system exposed knobs to the end users because it's also the most important problem, I think, in an object store native world. Um, so the caches are where things are going to get, I think, a little complicated. The benefit of these configurations is that they're all just kind of ultimately caches, right. If you lose your cache, if it gets blown out, whatever, you can always rebootstrap that cache from the object storage kind of durable source of truth of data. So it's really a kind of performance characteristic and not so much a like correctness characteristic. And you can tune it on the fly without really worrying about, you know, what are the implications to the other nodes in the system. The other big kind of tuning knob, uh. Were you going to say something?

Speaker A: No, no, no, go ahead.

Speaker B: I think the other big tuning knob, and this applies to all of the different systems, is that when Object Store is the source of truth, all of the open data systems allow you to run separate readers and writers, right? So you can maintain different types of caches for your readers and your writers and you can scale them totally independently. Right. So SlatedB, this comes from SlatedB, naturally, where SlatedB has a split reader writer, the writer is a single writer. This is how we can guarantee kind of all the transactions, all the consistency, all of the guarantees that we have. And if you want to scale out writers, then it's partitioning readers. However, you can spin up any number of readers that you want. They just basically list the manifest, which is how Slate DB basically uh, uh, organizes all of its data in S3. It discovers new data and it pulls it in. There's a little bit of a latency hit because you know, it takes like 50ms. Or whatever to flush data to S3. So the most aggressive pipeline will discover that data in 100 milliseconds, 200 milliseconds and pull it into the reader. So if you're reading from the reader, you're always going to have that ingest latency hit. But you can maintain different caches for different readers. If you have, for example, you know that you have a hot reader for alerts in time series that always read the last 10:20 minutes of data. You can very aggressively cache those last 10:20 minutes of data to, to serve your alert workload. And then you can have a different reader that perhaps has a much wider cache with like EBS storage because you're okay if it runs a little slower, Right. Instead of like NVME storage that's very aggressively serving alerts. So that's the kind of other I guess, fundamental architecture knob that's available for you that isn't really available in traditional like you know, non object storage based systems. Because the readers typically are the writers, Right?

Speaker A: Yeah. This is very interesting because I also wanted to ask you this exact same scenario where specific to time series type of data. Let's say you are serving a dashboard, sort of grafana dashboard, where some the latency guarantees are a bit relaxed. Right. But if you're looking at an alert monitoring, I think that requires fresh data. So having like different readers for these use cases makes a lot of sense. And if I have the power to, you know, tune them separately, I think it makes a lot of sense because then at a higher level of abstraction I can just control how reads are happening. So that, that makes a lot of sense. So for the single writer mechanism, I wanted to understand a bit more about this decision. Right. SlatedB has a single writer with fencing. How does it compare with having to do the partitioning at a higher level plus allowing multiple writers and then, you know, having some other mechanism to make sure that transaction guarantees are also fulfilled? So how does that those two approaches compare?

Speaker B: Yeah, so I think there's a few points of nuance here. First, let me define what it means to have a single writer. Right. Because I think that that's kind of important to understand. A single writer just means that ultimately you have one point of like one node that is determining the order of what is happening in the world. Right. So if you get multiple writes, what write actually came first?

Speaker A: Right.

Speaker B: That's all it means to have a single writer and that a single node is making that decision. Uh, in a multi writer setup you still have to make that decision, but typically use something like raft or whatever to do all of this coordination and choose, maybe select a leader or actually do consensus per write. There's lots of different options or you can choose to resolve it after the fact. Um, I think this third option is actually interesting because it plays really nicely into Slate DB's architecture. And I'll get into the trade offs a little bit more and why we chose it. But you can have multiple writers if you don't care about the ordering guarantee. And this is actually true of a lot of systems. So like for time series, for example, everything is append, only you don't actually care about what write happened first because every metric is kind of an event that makes sense for itself. So the way that we've set up our time series deployment with open data is that we have a layer that sits in front of slatedb that accepts writes from many writers and that's what we call stateless ingest. And basically the suggestion that we have for users is that you deploy one of these in each availability zone so that you don't have to go across availability zones to Talk to the single writer, and it just basically collects batches of data and dumps it into S3. So now you have multiple writers that are all collecting data, dumping it to S3. What the Slate DB node is actually set up to do is read from this S3 bucket. So you don't have any writers directly writing to slatedB. Slate DB just pulls from this S3 bucket, indexes itself in that single writer. So now you get a lot of the benefits of multiwriter, but you also get the benefits of kind of the simplicity of architecture. Not needing to run raft, not needing to do leader election, not needing to do all of that when you're running your actual kind of core system. Now, the fundamental trade off here is that you're bottlenecked by a single node for writes, right? So kind of what can you do to scale that out? I think this also kind of ties back into something that I mentioned earlier, where we're really building for the 80% use cases. And I really think that most use cases, many use cases, not most, but many use cases, do really well. Scaling vertically, we could handle some obscene amount of events per second on ingest. And remember, you can scale your readers independently. So really we're just talking about write ingestion, right? Readers you can scale ad nauseam as high up as you want. So the question is like, how much can I handle on a single node for ingest? And what we've noticed is that because LSM trees are so optimized for this write throughput, you can easily handle tens of thousands, if not hundreds of thousands of samples per second on a relatively common AWS node. I think we ran our test on i3 Extra Large, something like that, and you can easily handle 60 to 70,000 samples per second on the ingest, which is a pretty decent workload, right? Many companies never get really much beyond that. Uh, and then it just becomes like a read concern. How can you scale out your reads to serve that? But that's easy to scale. So that's kind of the mental model approaching that. Um, now, of course, we don't want to say if you get to that scale, you have no option and you have to leave open data. Like, that's not a great solution because everyone hopes to scale past the single node eventually. So that's where we kind of have to make this decision, which is, do we want to push users toward partitioning or do we want to try and do something a little bit fancier, like Cassandra's hash rings or anything that kind of more dynamically distributes data. There are a few primitives in Slate DB that we're building in order to help with the situation. Right. Because if anyone who's used Kafka knows how hard it is to decide on a partitioning strategy upfront, it's like, oh, how many partitions do I want? And that's going to bottleneck my scaling and there's kind of problems here and there. Um, so there are a few knobs and we can get into the details of those that allow you to basically split and merge Slate DB instances. Right. And that's kind of like a one time operation. Uh, that's actually um, a stateless operation. It's actually just you create a new manifest, uh, too much detail. We can get into that one kind of in a follow up, but basically. So answering the question of why we chose partitioning, many of these workloads can actually be very naturally partitioned. Right? Like time series, you can partition on the kind of series themselves. So like certain label sets you can see, you uh, can, you can hash and send out to different ingestors and this is exactly like what Victoria Metrics does. You know, you can kind of have a scatter gather based on multiple different ingestors and kind of this static partitioning scheme is just so much easier to think about reason and scale if you don't, since you don't have to move data around. As soon as you have to start like moving data around, everything just becomes a lot more difficult because then you have this like operation. And I'll reference kind of the listeners back to the warp stream, uh, kind uh, of podcast that they had with you where they talk about all of these challenges with Kafka and moving data around and coordinating that and all of the tooling that you need to make sure that this operation is complete. Um, and we just kind of felt that that challenge um, is really only justified in a small niche of use cases. So that's kind of the way that we think about scaling and partitioning the right workloads.

Speaker A: That makes sense. And since we are focusing on 80% of the use cases and this is backed up by an LSM engine which is really fast because it's a pen only you can also tune it. So having a larger box can solve a lot of problem and handle a lot of scale. And um, for anyone going beyond that scale, I mean it's a good problem to have. So probably they have more resources also to spend more effort into partitioning and or managing the database. Having one box though. So of course it's about scaling writes, but it's also about having higher availability. Right. So how does that compare it with what are the failure modes? If I have a single writer, what are the failure profiles? What kind of tuning do I have to do to handle failures gracefully? Whenever a new machine is coming up, how do I make sure that the cache is warmed up and it's not really creating a huge spike in my writes or reads? So yeah, what kind of options do I have? How should I think about it as compared to the traditional systems where there are a lot of failure modes and for example Kafka or Cassandra and so on. So what changes, uh, in my mental model here?

Speaker B: Yeah, so this gets kind of back to something that I discussed earlier, which is M. Something that was really influenced by Kafka Streams, which is. Kafka Streams is a highly available system that is actually a single writer system. Uh, this really influenced my thinking around how you can handle single writer systems and also what the failure modes look like for traditional distributed systems. Let's take a step back and consider what it means for a node to fail. When a node fails, there's lots of different failure modes. Let's just assume that it's dead. It's no longer responding to requests. In that failure mode, you have to detect that. Typically you do that through heartbeats. And the heartbeat says, oh, I have like, I want a heartbeat every 500 milliseconds, every second, every 30 seconds. Whatever your SLAs are, the more frequently you heartbeat, the more you risk basically, uh, detecting a false failure where maybe there's just a pause for whatever reason, a network partition that's kind of a blip and would have recovered if you tune it to be really aggressive, then you might see those. Now the question then becomes what happens when you detect a failure in a traditional distributed system? Assuming you're doing a single leader and then follower, not like a multi leader, which is a whole different world you can get into there later. But if you're doing a single leader system, which is like Kafka and many other popular distributed systems, you still have the same failure mode where you have to detect that the leader happened. And after you've detected the leader failure happens, you have to select a new one and then reroute rights there. So even in the most high available systems, you still have these windows of true unavailability. And the way that they're handled is typically clients retry, right? Like the clients will retry and then they'll hit a load balancer and the load balancer will realize that there's a new leader and it'll send it over to the new leader and then everything looks good. It's actually no different in a single writer system like slatedb because the durability is all up, uh, in object storage and you don't actually need to read old data typically to serve requests in many workloads, maybe you have to bootstrap a small amount of data. So what that means is that as soon as you detect a leader failover, like the single writer fails over, you use Kubernetes, you spin up a new reader, a new writer, and that writer can immediately start accepting requests. Because I don't have to bootstrap all of the data from object storage onto my local node. All I have to do is bootstrap the minimum amount of data. And you know, for example for um, vector, that's the centroid graph. Like I need to have the centroid graph to even ingest any data. Um, I can buffer data while I like, I can accept writes, buffer it, write it into the wall while I'm loading things up. So there's like ways to even reduce that time down even lower. But basically your, your downtime kind of exposure is how long does it take you to identify that failure and then spin up a new node, uh, that, that can kind of start serving write requests. Now this is where it kind of ties back into Kafka streams. And what's interesting is that you don't actually have to run any replication system. All you have to do if you want to turn that dime like that dial down to as low as possible, is run what we call a standby replica. Basically you just have to have a replica that's sitting there waiting to become the leader in case the leader fails over. All it does is it keeps that necessary write state warm in memory. And then as soon as a leader kind of fails, it is ready with the state that it needs to start serving requests. So you can again getting back to like the knobs that are available for you to tune your cost profile if you want, extremely high availability guarantees you run a warm replica, right? But that's two warm replicas instead of your typical three for your replication cluster. But uh, if you're okay with like a 1 second failover time, right, or like a 5 second failover time or whatever you want, then you might not even need to run a hot replica because that's how long it takes you to start up that new replica. So that's kind of the most obvious failure mode, like the Node dies and it's just never going to come back online. And again, I think this architecture just gives you more tuning knobs than you would have beforehand. Uh, the other failure mode, which I would say is the bane of distributed systems existence, is a node actually becomes a zombie. So the leader appears to be dead, but then it comes back online thinking it's still the leader, like X minutes or hours after it died. And a single writer system makes this fencing protocol so much easier because you don't have to then have like a RAFT system where you then communicate, and then the writer knows that it's no longer the leader. You'd simply have object storage. Like, you basically delegate all of that to object storage, where you write a single file as the manifest, and whenever anyone tries to write a new file, they verify that they are the first to write it. So when the old leader comes back online, it tries to write a manifest that already exists and it notices that it's been fenced and it dies. So the fencing protocol in that failure mode just becomes way easier to handle than a traditional distributed system. That's basically how we think about the failure modes. Also note that in many situations you can get high write availability if you don't care about the single writer through something like stateless ingest, where you have actually a layer in front of your single writer that can just accept writes and dump directly to S3. Um, so that's like a. In their state list, they literally need zero state in order to spin up. So you can just run a Kubernetes engine with the load balancer in front of multiple nodes and boom, you get extremely high write availability. That's just the availability of object storage. So all of that discussion was really just if you want basically transactional writes.

Speaker A: Yeah, yeah, I think it makes sense. And also the failure modes, and based on what kind of guarantees you need, the handling of those failure modes is also simpler because there are like less moving parts. Right. So this makes sense talking about other parts of the architecture. So of course you can separate out like the ingest and, you know, the readers. And also between readers, you can have multiple readers like we talked about. I was also looking at like a separate component called the compactor. And since now the data is in the object storage, you can also remove the compaction load from the machine that is actually doing the ingestion, which is quite powerful. Right. Because a lot of other LSM engines that I've, uh, read about, there's always this constant tug of war between, okay, doing more compaction or doing more ingestion, what is taking more resources and how do we tie that back together? So when we have peaks of write or ingestions, how do we make sure that we take the compaction down and not impact the uh, writes and so on. But this makes it a lot easier, right? Because there is no competition. The data is in uh, object storage. Right. So what has been your experience with that?

Speaker B: Yeah, that's one of my favorite things about the architecture. I wouldn't say it removes all the trade offs, but it makes the trade offs way easier to reason about. Um, like Scylla for example does this. They are really cool in the sense that they dedicate a few threads, they have this thread per core model and they just dedicate a few threads to compaction so that the compaction threads aren't interrupting your main workload. But there is a trade off. When compaction is not running, you're just wasting those cores. The really cool thing about this architecture that you just discussed, where compaction can run on a totally separate node, is that you can run as many cores or as few cores as you want for compaction and only during the process of compaction. That being said, compaction still isn't free. Like you can run extremely aggressive compaction, but whenever you do you're basically replacing ssts, which means the next query would be a cache miss. Because basically all of the blocks that used to be uh, uh, valid are no longer valid because compaction ran in produced new SSTs. Now the optimized SSTs are on, on S3. So basically you need to fetch that data from S3, you need to hydrate your cache and ideally you want to do this in a way that never gives you a cache missed. So I think to answer your question, it is a step up in simplicity of the architecture because now your compaction is actually like stateless. You can just run it wherever you want and run it as a one time job. You can run it as a long running job, whatever you want. And it just coordinates with the main writer over object storage. Nothing goes over the network, it uh, all goes over object storage and manifest writes. So it dramatically simplifies your architecture. But again running more aggressive compaction is not free operation. We are working on ways to make that like easier where you can like prefetch things as a compaction happens, but it's still not quite like free.

Speaker A: Mhm, absolutely does that. So talking about uh, like bringing back the 80% discussion again that for let's say 80% of the use cases running a Separate compactor may not make sense. If. Is it. Is it possible to run the compaction on the same ingestion node and get better cache hit ratios because the compaction process can also hydrate the cache more effectively? Um, is that possible? Is there also some decision making on the engineer who is setting this up? So what is the recommendation basically? Is it to start compaction and ingestion on the same node and then separate it out when it really becomes a problem, or what is the mental model?

Speaker B: Yeah, the default is to run compaction on the same node. Right now, we don't wire compaction directly into the cache. We should, uh, I think that's just like a work item that we have to do for Slate tv. But the process of extracting compaction onto its own worker node is actually an extremely smooth thing. It's like a single role. In fact, we have an RFC out there that's going to allow you to even run it hybrid. You can have compaction running on a node and other compactors running separately so that you can basically first spin up your other compactors and then roll your node if you wanted to. It's extremely flexible where these run. And that's because it all just communicates over object storage at the moment. Right. So, like, whether it runs on the same node or is a separate process is kind of totally agnostic to us. And same thing with garbage collection, by the way, which is another kind of really exciting thing. The compactor never actually deletes any files, which, uh, gets to. Oh, my God, I can't believe we haven't even started discussing this aspect of the architecture. But because of that, it gives you the ability to have basically free checkpoints and snapshots of your data. All you have to do is say garbage collection, don't clean up the old things. And if you do that, those snapshots can stay around forever. And because object storage is really cheap, you can then actually spin up a new reader that points to an older version of your data that was compacted but never garbage collected because you signaled the garbage collector to never clean up that old data. Um, so all of these, like, components, they all communicate over just S3. So whether they run on a single node, different nodes, or even different, like, availability zones really doesn't matter to the. To the architecture.

Speaker A: Makes sense. I know. I mean, there is a lot of things that we can go deep into and also some of the components that we can talk about, but in the interest of time, I would say let's talk about the. So we have Covered a lot of use cases where this would make sense and save a lot of cost and operational uh, burden. What are some of the use cases where it would not make sense at all? Like are uh, there cases like this that we already know of?

Speaker B: Yeah, I hinted at this at the beginning where I talked about like the three domains of databases. I really think that if you can handle your workload on postgres, you should just handle your workload on postgres. The kind of next step for that is I think this architecture works best for situations where it's not your primary data store. Right? Because a lot of these like extremely high write throughput use cases are downstream of other systems. So like your main transactional workload probably should remain on a system like Postgres or you know like whatever other kind of main store that you have that there's nothing actually in this architecture that prevents us from building a like a postgres compatible like system. We just don't think it's worth it because that like class of databases is so well understood and so like well developed and typically handles the scale that you need for those use cases pretty well. That it just kind of getting back to this 80, 20% use case. We think 80% of the use cases do just well on just postgres. So I think that's kind of the first class of things that don't particularly make sense. I think the next thing that doesn't really make sense are extraordinarily high scale things where you really want like a bunch of multi tenancy, you want to really optimize every little part of your, of your system so that you're utilizing like every core 100% of the time or systems that require basically that, that can't have cold misses. Right? Like if you're serving a like fraud detection pipeline on Kafka, we don't recommend using our log because every millisecond that you miss that fraud is potential exposure to you. Right? So systems that like want that really really really low level, you know, like sub 50 millisecond uh, latencies. A cold miss if it kills you because it has to hit object storage is basically going to be unacceptable. In those situations it probably makes sense to run a um, kind of legacy. I'm putting that in quotes because there's a lot of really awesome systems out there but ones that don't depend on object storage. And the first layer really is hot replication across the network because you're always just going to be able to beat out what open data systems can do in terms of tail latencies. So I would say those are probably the two biggest categories, again, like core, core postgres, SQL, OLTP stuff, and extremely low latency kind of, you know, types of workloads.

Speaker A: Makes sense. Cool. Um, talking about LSM engines and systems around it. This is one of my favorite topics and there's just so much to talk about. But I think we have covered a lot of ground today and, you know, given our audience a good understanding of why open data exists and what is the goal and comparing it with the existing systems and, you know, the use cases that can really benefit from this and the use cases that may not benefit at all. So I think it makes sense. And if there are more questions and there are more topics, of course we can do like volume two of the discussion. But I think for now, I think there's a lot of information that we have packed in. And, uh, yeah, I'll add all the important blogs and resources in the YouTube description so, you know, viewers can look at it. Uh, but yeah, I love the discussion. So thanks a lot, Al Mukh, for joining me today. And, uh, I hope you also had a good time.

Speaker B: Yeah, this was great. Thanks so much for having me on the podcast, kv.