TCG079: Why Your State File is Actually a Distributed Systems Problem

Speaker A: Foreign.

Speaker B: AI is here. AI is here. And terraform's dead. Open tofu. Dead, dead, dead, dead things. So reading the next wave of hot takes out there might have you believe that this is actually true. Well, here's a news flash. The, the value was never this, this DSL thing that people see that we call hcl, or Terraform, uh, configuration language. It was the State. And for like, over a very long time, over a decade, we've been storing that state and a flat JSON file protected by a, a, uh, single lock. And then. And then when it tips over it, like, broader scale, we feign surprise and pain. Like all the soccer players you see when you watch the World Cup. You know, when someone makes, like, hardly any contact, minimal contact, and they go diving with, like, these theatrics, you know, similar to what you might see in Hollywood. Um, but. But back to the problem, though. What if State was never a file problem at all, but more of a distributed systems problem? You know, kind of dressed up in some, some JSON costume.

Speaker A: What if.

Speaker B: To talk about this and to unpack a lot of things around this, we are welcoming back to the podcast Malcolm Matalka. Um, how are you doing today, Malcolm?

Speaker A: I'm doing great. How are you doing?

Speaker B: Excellent. Have you been watching the World Cup?

Speaker A: I have not. I'm not a soccer man.

Speaker B: I'm a hockey fan. So I've tried watching, and it's very painful for me to see, um, everybody flopping around like fish. And then as soon as, like, it's over and they realize they didn't get a red card, they hop right up and, like, run back to their bench like nothing ever happened, really. If I have to watch this a hundred more times today, I can't watch another soccer game. It's too much. Um, how are you doing today, Yvonne?

Speaker C: I'm great. And I have not been watching World cup games, but what I have been watching are European World cup and even African World cup attendees. I've been watching them explore, uh, the United States of America on social media. And that has been an absolute treat. So that. That, that has fed my soul this week. It has. It has given me new hope in humanity and the state of our world and has brought me endless joy.

Speaker B: There's some. One of them, like, came out, I think it was from Switzerland, and he went into, like, Walmart, and he kept going back to Walmart. He's just like, I can't believe this place. Like, he just kept going back.

Speaker C: And then he's like, gun and cereal and a tent and, And A sleeping bag and paper towels and a, uh, full outfit all in one place. Absolutely. God bless.

Speaker B: It was so crazy for him. He got the Walmart, like, logo tattooed on his arm. Went viral. And then Walmart like, retweeted him. They're like, you're our new ambassador in the, uh, in Europe. You know, I thought it was hilarious. Like, wow, what a. I'm short of

Speaker C: permanent body markers, but you do you.

Speaker A: Yeah, that's a bridge too far, I think.

Speaker B: So, Malcolm, um, returning guests. So the last time you were on, you were like one of the, the OGs on this podcast. It was like when I was the only person. I was trying to figure out, you know, what my audience even was. And we had a great talk about, um, oh, Camel Open source. Just. You had a lot of hot takes. It was great.

Speaker A: You know, you've blossomed so well too.

Speaker B: Thank you. I'm working on it. I spend 10 hours a day or 10 hours a week at the gym, let's say. Just kidding.

Speaker C: No, we need some floating flower emojis.

Speaker B: Yes. So while our last whole thing was, I think, like open source and you know, very like language specific stuff, I want to go somewhere different today. You know, something that's getting a lot of hype, um, a lot of just attention. But I really want to talk about State, like, what kind of like what actually makes or made infrastructure as code good in the first place and kind of like looking at what you think the next, uh, phase looks like on the graph, you know, thing instead of a file thing. So I'd love to dig in on this because, um, and before we get into, you know, everything, do you want to just give us a. Not like a lexicon of Malcolm, like we don't have to spend 10 minutes talking about all your accolades, but just kind of a little background in your career and like what you're doing today, um, and where you're at.

Speaker A: Yeah. So I think, um, the most relevant to the discussion we're having right now is, uh, my background's in software engineering. Uh, more from like a theoretical perspective, but so doing a lot of programming, Jack of all trades there. But my, uh, employment history has been more around building internal dev tools for various companies. Probably the biggest on that list was Spotify, where I worked on the team developing the internal dev tools for Menace, managing M. Cassandra clusters. Uh, and then I also have a background, a bit in doing, um, like scientific computing. Big pipelines around, analyzing DNA and all that sort of stuff. So a bit of a mix there and all of that. Has really brought us here today in that seeing um, how you solve in scientific computing, you need to not rerun computations that you know you don't need to rerun. So there's a lot of effort put into like managing it as a workflow, making sure you're not repeating steps and so on. And when you look at how managing infrastructure as code works right now, you see a lot of wasted time on operations that you don't have to waste time on. And that is just based on the model that we have to work with. Um. M. My background, mostly infrastructure code is around Terraform. So anything I say will mostly be focused on Terraform. I don't think things are that different if you look at the other tools like CDK or Plumie or anything like that. I think a lot of them are doing the same sort of high level model there.

Speaker B: I love it. So one, one thing specifically I think in your, your background, like I think you know, uh, you've got years of like GitOps style, like automation just kind of like on top of the, you know, Terraform. Um, has this given you like working in this type of workflows, it kind of given you a ground level view. And I'd like to understand that like if your conversations with companies that have really hit that point where it breaks for teams. You know, so starting off small, you get bigger this j, you know, JSON files and things get bigger and bigger and bigger and then you have, you know, I know how you love state splitting. It's one of your favorite activities. So you know, what is your experience on just kind of like where this begins to break down in the first place.

Speaker A: Yeah. So if you take our, our first episode together where we talked a lot about OCaml, a very niche programming language. Um, the theme here is I will continue to be weird and say that I see best practices. I'm probably like the most vehemently against them and think they're just kind of stupid. And what are those best practices? Well, when your state grows. So uh, I'm sure most people listening are uh, like aware of how Terraform works. But just like a quick reminder, you have what's called the root module where is where you define all of the infrastructure that you want to manage inside of that particular unit. And that root module has a state file and that state connects all of the identifiers in your cloud to your code. So you as the programmer say I want an EC2 host named my favorite EC2 host. But that's not what it looks like on the cloud. The cloud gets some crazy identifier depending which cloud you're on. And you need something to manage to connect those two things. And there's also information that you as a programmer want to define and information that the cloud wants to define. And then you need a place to store that connection as well. So that's what the state file is. In Terraform, the challenge becomes when you grow, um, in a few ways, both in the amount of resources that you want to manage, but also in the amount of team members that you want operating on those resources. So Terraform is very good at being a fast starting tool. You just download a cli, you write your code and you go. But once you start having multiple people operating on it now you're in the distributed systems world and you need a way to manage access to that cli because only one person can perform an operation at any single time. And that's because that state we talked about is managed as one single blob and it is rewritten entirely on any operation you do. And there's no real way to say two people wanting to do an operation and they get their own blob and then combine them. So we have to serialize access, uh, basically a big file lock. And say this person gets to do this, then after that this person gets to do that. So that becomes a real issue at scale, uh, both in terms you have lots of teams, you have lots of people and you have lots of resources, because more resources means things take more time, more team members means more people want to do things at the same time. The recommended best practice now, which I think is silly, is, but to be fair, it's the best we can do given the existing model of Terraform, is you take those resources and you break them up into smaller root modules, which means smaller states. And then people can work on the resources they want in state one and then people, other people can work on the resources they want in state two. So you still have this serialization issue, but if you've broken up your root modules, hopefully, well then you're m reducing it quite a bit because individual, uh, teams, for example, can operate on their own root modules.

Speaker C: Well, and that hearkens back for me. I know it's not the same thing, but it makes me think about cap theorem, you know, consistency, availability, partitioning. And what you're doing there is you're partitioning so that you can enable more people to work on it at the same time. But then you're giving up unification. If I'm understanding what you're saying correctly.

Speaker A: Exactly. You're kind of partitioning for availability, if you want to think it that way. Uh, otherwise people are blocked from doing the operation without the partitioning.

Speaker C: Right.

Speaker B: So yeah, that was a great call out of Von 1. One thing that I'm seeing a ton of and this, this is the thing that drives me probably the most not you know, just nuts lately is hey, like to get to the whole point of like this stuff is dead. It's like okay, let's compare HCL to like Python or something or HCL to like running an agent to do X, Y and Z. And this is why Terraform is dead. So they're, they're almost comparing like HCL is like the, the D. Okay, this DSL is not that great. So obviously this whole paradigm is dead. And so if you know, to me like if the syntax isn't that magic, um, and you look at the stickiness of infrastructure as code across every, every company that I ever worked with or for really. Um, I guess the case to be made is that the, the syntax has basically been commoditized with AI. Like anybody can write Terraform now with a, you know, coding assistant. But State is really that load bearing concept here. Um, and it's, it's very important and that's where to me like if you can't figure out how to solve that in a better way, this is kind of here to stay. So it's not just going to die and then you get into more um, distributed systems thinking. So I think so Terraform uses file system semantics for what is really like a concurrent, like multi, you know, like you were saying, like a multi, uh, multi actor coordination ish problem of, of sorts. So I think you've, I think, I think it was you. I, I hope so. But you, I think you described Terraform State as kind of a distributed systems problem. Kind of like thinly veiled in a JSON costume or something. Maybe or something along the sort. But I, I'd like you ah, know if possible for you to unpack that for us. Like what's the mismatch between like treating State like a file and what's actually happening? Like when multiple people and pipelines touch the infrastructure. Um, like when did. And also to kind of add on that, like when did it click for you that this was a distributed systems problem? Like more theory than just replacing like tools. Like tools are tools at the end of the day but there's actual theory underneath this that kind of sets you up for success.

Speaker A: Yeah, I'm sure I said that because I say a lot of snarky stuff online. Um, but to put some meat behind that, um, I think first, like so anyone in the audience listening? I think there's two things that I hope to successfully communicate here. One is that the best practices we have right now in ISE are really inadequate for the modern world. And whether that's just, you know, AI is accelerating things, we even take AI out of it. Nobody at any point has said I, uh, want to go slower, like AI is helping going faster. But even just any other company has always wanted to go faster and faster. And the amount of infrastructure you can take and manage is only increasing and that directly correlates to slower infrastructure management through your iac. So those best practices just really aren't keeping up with the modern world. The other one is that there's nothing wrong with the interfaces we have right now. So by interfaces I mean like using hcl, iac, any of these tools, those are actually fine and you can debate whether they're a great solution or not a great solution. But there's a lot of existing knowledge behind them, there's a lot of development behind them. We know that they work at scale in various ways and the those are functional. We just have to find uh, it's really the backend that needs to change. It's not what the user is interacting with, it's the backend that has to change so we can be backwards compatible and also pick up in the modern world. So the, if you look at all existing infrastructure as management tooling right now, effectively it's a very complicated way to write a file lock and there's a few, there's a few things behind that. One of them is because Terraform is so low level as the cli, you don't really want to give people access to it on their laptop. So most people go through a gitop solution or there's things that require, you know, you can click around on the UI but some sort of third party thing where you have to interact with it and it's separated from your actual development environment. And that adds a lot of friction around things like developing with AI. So I think a lot of people coming at this saying oh, I don't need IAC because I have these AI solutions. I think they're actually kind of experiencing this separation and the tool kind of feeling foreign and all the existing infrastructure around using that tool kind of having all this friction there. And they say well I can just use Python and Boto and bring stuff m up myself. And they're kind of missing the fact that this friction is the actual problem. It's not that HCL is a problem or anything like that. And because the solution we've chosen around solving distributed systems problem is really just a fancy file lock, we're actually blocking people out from being much more productive using these tools. If we can just give access to Terraform or a Terraform like experience just on your laptop. Because if you think about how an LLM writes code very well, generally it requires unit tests or some sort of testing framework and you say, hey, go implement this, implement some tests. And it bounces back and forth and it says, okay, I did this, but the test didn't work. Let me go real quick. And if you have this massive step between writing the thing and then running the thing, either you have to make a pull request if you're in the GitOps world or you have to like click a button somewhere. One, the LLM is not very good at that and two, it's not very good at getting the context back out. So if we can give this new model where you can actually just run things locally and we're going to look at it more like a, there's a backend service that is giving you permission to do these certain operations now you can go a lot faster in this AI world and a lot safer and you don't have to give up all the great things we have with a declarative system. Um, you know, complain as one might about hcl. It's actually very simple. And the core challenges there are really around like how do you write correct cloud infrastructure and not how do you write the correct resource block?

Speaker C: Well, and so you make me think of this, uh, this talk that I listened to from IO and the subject of the talk was software engineering at the tipping point. And the whole premise of the talk was yes, we're writing code faster with AI, but just writing the code isn't what you need to ship a full system. And so in the talk he talked about not only the infrastructure you need for unit testing, he talked about the impact that all that development is having on our software development Systems. Think Git, GitHub, uh, our code repository. What you're talking about here with Terraform and this state file is just another instantiation of that problem is that we, because we, our velocity of development is increasing, our systems can't keep up. And um, I would recommend anybody watch that talk even if you're not a developer because it shapes up the problem for that, that infrastructure folks are going to have to solve, um, and it has to do with the velocity of development and how quickly we're building solutions and how the rest of the infrastructure, and I use that term loosely, that we have around that work, um, isn't ready, isn't sufficient, can't operate at that same speed. Um, and we're going to see this in this arena as well. Um, yeah, I think that's. You're not a developer. Yeah, go ahead.

Speaker A: Yeah, I think that's exactly right. And in particular, I think that software development has gotten a lot more attention in terms of speed here because for a lot of people, infrastructure changes a lot slower and you're doing things kind of at the speed of a human making a change and then testing it and then. But people don't want infrastructure to change that fast. But again, things are speeding up. People. It's not necessarily infrastructure changes so much faster these days, but people are spinning things up and spinning things down more frequently. They are trying new services. Um, they doing experiments is much easier now on the application side. And now you translate that over to the infrastructure side where someone's like, hey, I just want to like, spin this, like this, this, uh, my application up real quick with all the infrastructure around it, see if it works, see if it's working for me, see what it's doing. And then if I don't like it, I'll destroy it or I'll keep it up or modify it. So the. Now that things are going faster on the infrastructure side too, I think people are starting to get more frustrated and like, hey, like this didn't feel. I knew this was slow before, but now it's actually preventing me from doing what I want rather than just being kind of annoying.

Speaker C: And what you're identifying is, is the problem isn't, uh, isn't the language that we're using to deploy the infrastructure, it's the supporting services around it. Specifically this state file that really is just a file, not a distributed system.

Speaker A: Exactly, exactly.

Speaker B: This is great, uh, great point.

Speaker C: Great.

Speaker B: Uh, call out of. On. I'm have to watch that talk now. One, one thing I'd like to do. So I, I think the. I have so many questions. Um, so I'd like to connect this to AI a little bit because kind of like AI is when, you know, I've started hearing all these fancy, like, ideas about how things are going to change and you know, back to the dead, dead, dead thing. So what, what I've realized is there's actually a camp of startups, there's a camp of a Lot of things. Opinions that are, um, basically asserting that agents are just going to call Cloud APIs directly and kind of skip this whole IAC paradigm entirely. Um, I, I kind of believe this is a trap, but I'd like to get your thoughts on it. Like what, what do you lose the moment that you drop state and item potency and just go agent to the walls here?

Speaker A: Uh, I think you get probably like woken up a lot at times. You don't want to get woken up. And, uh, I, I think it's fine if you're the first person, but your second hire is going to hate you. They're going to be like, wait, wait, uh, how do I make a change here? Uh, but seriously, the thing that we see is that you do want, uh, so why do we have ISE in the first place? Right? Well, originally humans were not making those API calls, but they were doing the equivalent by clicking around and setting stuff up and being like, okay, this is it. But you lose a few really important things. One of them is if anything gets destroyed or modified in a way that you didn't want, you don't really have that record. Whereas if you have infrastructure as code, you can, at least it's in, hopefully it's version controlled as well. You can go back in the history and say, oh, this is what it looks like. We can revert that, or we can see how it's changed from there. And especially if it's been destroyed, we can bring it back as well just by rerunning that code. So one thing is you get reproducibility, you get auditing, uh, you get someone, you get a place where you can add comments as well. Just like it's code, you can say, hey, I did it this way for a reason. There's no real place to do that in any sort of API or, uh, ui. Um, and the other thing is that you get determinism. Now you can do a lot of stuff which is, you know, Boto, running Python or even like, I've seen some tools that just wrap like the aws, CLI or whatnot. But at the end of the day, you have this intent that you wanted, right? You're like, I have a server, I wanted to run. This server has meaning to me as a human and it exists in the cloud somewhere and that has meaning to the cloud. And you do need state somewhere to connect those two things, because otherwise, the next time you run that script, how's it going to know whether it needs to create the thing or modify the thing? And you can uh, you can like it's all software, right? You can build all this if you want, but we already have all of it built and it has really great coverage and it has on like APIs and all that. And it's very battle tested and it works pretty well and it's deterministic after the point of your intent. So the model that we promote uh, with our customers is that you can use AI on the planning side to say, hey, I'm trying to do this thing, can I, can you plan it for me? Can you write the code for me? Can you, uh, you know, I want to do the thing but like, you know, write the part out for me to see what it'll look like. But at that point when you said this is what I want, becomes entirely deterministic after that. And you're guaranteed that the end result is going to match that front result. So you get your human review on one side and then the determining system on the other side. And I think that is incredibly valuable because you just know it works like you planned it was going to work.

Speaker B: You frame that very nicely. And when I think about that too, like, those are all great points. I think too of like some of the bigger, bigger, uh, environments I've worked in personally, like where you have a large business and you know, maybe the way that, that, that business chunks things up. I'll try to use a simple example. So say, say that you have like a line of business and maybe there's like a production VPC and a non production vpc, you know, just for simplicity. And within that VPC you have, you know, more subnets than you can count, you know, on both hands. You have constant reservations and building, you have multiple teams that are building within the, the guys of that VPC under that same overarching cider address. Well, you run into a scenario where folks are grabbing the same stuff or they're saying it's available at the same time and you know, again, you break things. Um, so you have race conditions and then eventually you have, you know, API rate limiting to think about, you know, as the environment gets bigger. And then even from there just thinking about, okay, like if you come in with a fresh agent and it has to go out and rediscover your whole estate. Again, like it has to go out and figure out like you were saying, like what's there, what's available to use. It has to do a gigantic amount of reasoning which you might as well chop down a tree for the amount of tokens that you're going to Be using on the daily. It's just not like from an efficiency perspective. It seems to me like it's one of those things that just sounds, you know, a lot of things sound really good when you first hear like a um, someone talk with fire about them. Like, yes, this is it. But then like you start digging down into some of these small details and you start just falling into, you know, holes. So I don't know.

Speaker A: No, exactly. And I think that, I think one way to think about it is like where do you want your creativity? I think you want it in that I have a problem to solve my infrastructure. Be creative in how to solve that problem. But once we've agreed that that's the correct solution, be deterministic after that point, you don't want creativity on actually like starting an EC2 instance. Right. That is just like. Just do it.

Speaker B: Yeah. Table stakes in 2026, you know. Come on. Um, so you started state graph and I'm seeing an outcropping of just kind of taking. And again kind of like you alluded to you, and this is kind of the way I see it again, like the way that I see it just personally is like this uh, this flat file decade that we've had of these JSON files. It's not ending. It's like here to stay. But it's going to be like a good enough default for a lot of people. But when you want to start scaling and you want to start doing um, that infrastructure, that file needs a glow up is the younger generation glow up. You know, so the glow up of the state file is this graph database stuff. Do you want to um, kind of go into just kind of what it means to take that flat state file to a graph database and like what benefits someone might see, like why it's even important to think about.

Speaker A: Yeah, yeah. And I want to be clear. Like I've been complaining a lot about the state of things, but that's also trying to do something about it. So it's not just like ah, you know, old man yells at cloud situation. It's like trying to actually fix it. And as I said before, like uh, a lot of the options out there and you can look, uh, really all of the options in my opinion are just really sophisticated forms of a file exclusion lock. Know one person operating this file at a time and there's this great quote that's perspective, um, is worth 20 IQ points. And what we saw is just, you know, we're not smarter, we don't know more about terraform than anyone Else, but we just turn the problem a little bit and we're like, hey, if you look at it from this perspective, this actually can unlock a lot of things. And the perspective was, what if you got rid of the file and just stored everything in a database? And it sounds like such a simple thing. And, uh, people familiar terraform be like, oh, but it already has a, uh, postgres backend. And that's not what I'm talking about. I'm really talking about deconstructing the state file and putting your instances in its own table, your resources in its own table, everything in a proper relational database. The things that you can do just with that little insight end up really growing and growing. So an initial one is that the difference between a resource, uh, in state A and a resource in state B is really just the value in a column. And if you want to look at both these resources, that's not a big deal. You just do a select where State A equals 1 and State B, or state equals A and state equals B. And now you can see both of those resources. So these boundaries really disappear, which were previously a hard boundary, now becomes a very soft boundary, and it becomes a boundary that you can decide to enforce, where you can decide not to enforce depending on the operation that you want to do. So, for example, um, one thing that we promote is looking at your infrastructure as a single unit and not really as these, uh, pieces of data in different states. So if you want to say, hey, how many S3 buckets do I have? That is a query you'd want to do across every state and you don't care where that data is exactly, or you don't care like which state file it's actually in, you just want to say, across all my infrastructure, how many of these things do I have? And then maybe once you get one of those things and say, okay, well, actually I'm curious about things in this particular state. Now, if you look at how it works in really the current world is that you have all these individual states, and if you aren't using certain tooling that supports, um, asset management, then you probably have to download each one, do some, um, bespoke querying on the state, see if it has what you want and then put it back. Whereas once you have it in a database, it's all. That's what databases let you do. They let you query stuff. The other big thing, which I think is really sort of really like a fundamental shift in how to think about infrastructure and how to manage infrastructure is right now you Have a uh, change. It can be a big change, it can be a small change, but it has to take out the whole file, right? It has to control that whole file when it's doing it. But when you think about like imagine a database where it only let one connection at a time and do its query, then had to disconnect and then the next connection could come in and be like, hey, it's my turn. That like nobody would use that database, right? So if you think about it in terms of, well, uh, how do databases work? Well, databases let multiple users operate in a transaction on a single table. And as long as those transactions don't overlap in the modifications they make to that table, then both can commit without an issue. So one thing you can do is now instead of only one person can plan and apply at the same time. Anyone, as many people as you want can apply at the same time. And as long as they don't overlap in the resources that they're modifying, those transactions commit without a conflict. So you've like unblocked this massive reason that people end up breaking up their state files in the first place.

Speaker B: That makes me smile. Just uh, just like the thinking of drift detection and kind of like queryability as you're, as you're talking through it. Um, if I'm, if I'm in a big company and I have lots of teams and you know, it's really hard to narrow things down. Like if I want to ask, hey, you know what changed in prod in the last two hours because something blew up. Like what that means sort of today is some, some measure of like grepping some, a lot of like CI logging and stuff and maybe finding some drift at the next plan. If like um, I'm like really having a good day. Today's been a little salty. So if I was doing this today, everything would be broke. But um, this, this kind of changes the conversation when you can kind of think about like okay, what changes when state is something you can actually query in real time and get answers like you're saying across all the different things that used to be separate, you know, how does, how does a graph change? Blast radius analysis, if that makes sense. Like going from tribal knowledge and all this grapping and I think hey this, this might touch this, you know, but I have to spend like 30 minutes and figure it out to something that you can actually compute instantly. And a follow up question to that is, is there like some shared fate between environments when you look at a graph database, since it's like one you know, in my mind I'm thinking like one database. So what, what would you say there?

Speaker A: I think, um, once you go down this path, you end up seeing all these places you can just add more and more information, uh, rich information. So at first we thought, okay, well, what if the state is in a database? That's great. And then you start thinking, well, if I have the code in there as well, I can actually start correlating code changes to state changes. And because I'm tracking all these things through, uh, we have like application level transactions effectively. So you can see I want to do this change. Here's the code change. And then the state gets populated into that same transaction. As you modify it, everything comes together. So the effect here is that, hey, something's broken or something weird happened. Let me go see. When that change, when that actual piece of code change, what was the code change? You can correlate that back to your git, uh, repository too, if you want. But the git repository won't have how the state actually changed in it as well, where your database will. So you can in the worst case. So I mean, uh, people do like state surgery now and then. Most people try to keep state a bit of an abstraction, but it doesn't always happen, especially because, um, you know, state as a JSON file is not a great abstraction in the first place. But now that you have all these things in there, you can see a history because your JSON file doesn't have your history either, right? You, maybe you store it someplace that keeps history. But again, are you connecting that to the actual code change? You have all this sort of, I want to look in the past and see what happened in the past just becomes like a trivial select versus a either a thing you have to build internally or a sort of a duct tape of an ex, uh, of another tool that's like running your terraform. And then it's saying, okay, I'm going to try to like index maybe the state here and there and see what happens and present that to you. Everything is just sort of. It's much more natural in this world. In this world.

Speaker B: Yeah, yeah, that makes sense. Um, and to kind of add on that too. So we have this graph now. It's really important to, you know, my business and I have all my infrastructure that's depending on it. Uh, you know, Neo4J isn't a lightweight little thing that you can just kind of run in a tiny little container and just run with it. It's a little heavy. You know, there's A little bit of maybe some engineering involved behind the scenes. So when you think about like blast radius for the database because you know in the past I uh, I've seen it done so many ways but stuff is so particular is like every environment based on every technical thing. So like networking would have its own state file and its own git repo and those were separate like everything is, it's like a state splitting nightmare. M on steroids. It's just crazy. Now you have this one graph database that all this stuff that you depend on is in. So is thinking about like providing a service like state Graph does. Like is this kind of a thing of using that cloud, high availability, um, reliability and doctor patterns that underpin cloud to scale that database and make it reliable.

Speaker A: Uh so we put a lot. One of sort of our ethos is, is that we want to be as simple as possible. We're not trying to win like complicated architecture awards. So we actually like a dirty secret here is we don't use a graph database. We just use a plain old relational database. We use postgres and just do some like, we do some wild SQL on it but it's all like pretty normal SQL update at the end of the day and scaling that is scaling uh postgres for whatever challenges it has. It's actually like a pretty well understood beast. So it's not so bad. We also have um, we focus a lot on self hosted customers so that even a very large self hosted customer on a pretty moderate hardware um, that runs like, we're not talking like terabytes of data here. So that runs pretty efficiently and we just do standard replication out and all that. So we really try to keep uh, everything simple not only for our sanity and testability sanity but also there is the sense of well now I have a service between me and my infrastructure. Like what happens if that goes down for some reason and keeping that as simple and understandable as possible is really important to us because we want both the systems that we manage to not wake up at night but also to instill that trust in any customer because we understand that concern. And uh, being just like a uh, basic, no frills architecture is rock solid in reality.

Speaker B: I love that uh, connecting the dots to just you know, again problems and solutions. This is the problem and this is a very you know, elegant solution without a lot of unnecessary overhead. Because I've, I've talked to some folks that are just like yeah, we're gonna basically we've chosen like Neo4J and we're gonna like work our way backwards. And you've really done um, you know, what you just kind of framed out is like, hey, this is actually what the problem is. We have a lot of known patterns that we can kind of build our, you know, our architecture on that already scale and you know, this is, meets the problem where it sits. Like there's no reason to take on unnecessary burden or overhead here.

Speaker A: Yeah, definitely. And I think one thing that's really um, what I'm looking for here, really uh, satisfying about Statecraft is that it is a very real customer facing problem. Like the people we talk to are just like I, you know, sometimes I have to wait two days in the queue because like this thing takes like two hours to plan, another two hours to apply. And so you have two people in front of me and that's the whole day. And that's assuming they don't run into any issues. So it has this very real uh, customer facing benefit. And it's also just a really cool like thing, theoretical graph problem to solve. So you get this good mix of like really fun engineering that also once you present it to people like the, their eyes light up. Like we've been talking just doing like a basic demo and they're like holy cow. Really? Who, who does, who says holy cow really? In like an ISE demo?

Speaker B: Yeah, yeah, yeah, very fair. It's just, it's, it's also interesting to me too because um, the like all of these like the database angle, like we've, there's been a ton of problems solved by databases for decades and I think infrastructure tooling kind of like I don't think it ignored it out of, I don't know, um, none of these ideas here are really new. In fact a lot of technology, you know, barring some of the wizardry stuff you see with AI stuff, a lot of it's just patterns that have been sort of repackaged and have a lot of intelligence built on them to solve newer problems in different ways. And that's almost the point, you know. So like the um, you know, right ahead logging like all these things with you know, these patterns are like well trodden with database theory. Um, I wonder. And I'm just kind of like thinking to myself like why, why this is like why this is only happening now, you know, and why it didn't happen um, you know, long ago. Just interesting to think about.

Speaker A: No, definitely. And I think that so Terraform, uh, I think you can rightly argue is the like the first iac, at least the first IAC with mass adoption. I know there's other things before that depending on how you want to look at it, but really that's like the one people go to and think about. And it did this thing that was very different and very successful. And really I uh, think you could say it's like a game changer in how people worked with the cloud. That's fair. And the model it provided works very well but it's kind of stagnated in a way which I don't know why it's stagnating that way because the people I talk to must be the same people that the terraform developers talk to and the tofu developers talk to and they must be telling them the same thing. And I think part of it might be that we're just kind of like ridiculous, uh, enough people where we were like, well like we'll just do it and because part of what you have to do to make this work is you kind of have to re implement large chunks of terraform and Tofu on your own because you want to be able to do all these really powerful evaluations of it. Uh, we use the same providers underneath and all that. So it's the same terraform at the end of the day or topo at the end of the day. But the thing you interact with is completely different. And that's kind of scary from a new product perspective. But uh, it's, it's working extremely well. And, but, but I don't know why like the existing tooling isn't like well we, we can go this next step here. I think part of it is that and, and I think this is, this is to a true benefit of Terraform and Tofu is if you're uh, if you're a starting out all you have to do is download a cli, right? And you have to figure out a place to put that state file. But all you do is down the CLI and you run it. There's no infrastructure you need and that becomes kind of a later problem. Whereas if you switch to infrastructure as a database, well now you have to run a database somewhere. And so I think like terraform is great at that starting out point and it's, and I don't know if we would ever want to lose that, you know, like, like I think that there is true benefit there but I do think that if you succeed you need to, you're going to graduate up into something that is infrastructure in a database. Whether it's something we're doing or this idea catches on, which I hope it does because there's a lot of people out there feeling like a lot of pain in how they're blocked every day and how slow everything is. And this is the only viable solution that I've seen this model.

Speaker B: Yeah, you. You crushed that answer. You actually answered my follow up question in there as well. So. Awesome. So I guess as we, as we kind of wrap up, um, where should people go to follow your writing? Hot takes? Any work that you're doing, where can people find you?

Speaker A: I am most active on, uh, and this is like, feels a little cringe to say, but like on LinkedIn.

Speaker B: Ah.

Speaker A: I have very little filter there and I just let it rip. Otherwise, um, if you check out statecraft.com I have blog posts there and there's also a link to the Discord if you want to have, uh, a live chat about what we're doing, uh, what the model is, how to think about things. Um, so those are the three places I recommend.

Speaker B: Thank you so much for your time. It's been fun. Always a pleasure. Yeah, thank you very much.