Bryan Cantrill on Oxide Computer

Speaker A: Today I'm talking to Brian Cantrell. He's the co founder and CTO of Oxide Computer Company and he was previously the CTO of Joyent and he also co authored the DTrace tracing framework while he was at Sun Microsystems. Brian, welcome to Software Engineering Radio.

Speaker B: Awesome, thanks for having me. It's great to be here.

Speaker A: You're the uh, CTO of a company that makes computers but I think before we get into that a lot of people who build software now that the actual computer is abstracted away, they're using AWS or they're using some kind of cloud service. So I thought we could start by talking about data centers because you were previously working at Joyent and I believe you got bought by Samsung and you've previously talked about how you had to figure out how do I run things at Samsung's scale? So yeah, how was your experience with that? What were the challenges there?

Speaker B: Yeah, I mean so at uh, Joyent and uh, so Joyent was a cloud computing pioneer. Uh, we competed with the likes of AWS and then later GCP and Azure. Uh, and we were operating at a scale right? We had a bunch of machines, a bunch of DCs but ultimately we were a VC backed company and a small company by the standards of, certainly by Samsung standards. And so when Samsung bought the company, the reason by the way that Samsung bought Joyent is Samsung's cloud, uh, Bill was uh, let's just say it was extremely large. They were spending an enormous amount of money every year on the public cloud and they realized that in order to secure their fate economically they had to be running on their own. Infrastructure just did not make sense and there was not really a product that Samsung could go buy that would give them that on prem cloud. Uh, I mean in that regard like the state of the market was really no different. And so they went looking for a company uh, and bought Joyent. And when we were on the inside of Samsung that we learned about Samsung scale and Samsung loves to talk about Samsung scale and I gotta tell you it is more than just chest thumping like Samsung's scale really is. I mean just the sheer, the number of devices, the number of customers, just this absolute size. They really wanted to take us out to, to levels of scale certainly that we had not seen. The reason for buying Joyent was to be able to stand up on their own infrastructure so that we were going to go buy, we did go buy a bunch of hardware. And I remember just thinking God, I hope Dell is somehow magically better. I hope the problems that we have seen in the small. We just, I, you know, I just remember hoping, and hope is hope. It was of course a terrible strategy, and it was a terrible strategy here too. Uh, and the problems that we saw at the large were when you scale out the problems that you see kind of once or twice you now see all of the time, and they become absolutely debilitating. And we saw a whole series of really debilitating problems in many ways, like comically debilitating, uh, in terms of showing just how bad the state of the art app is. And we had, I mean it should be said we had great software and great software expertise. Um, and we were controlling our own system software. But even controlling your own system software, your own host os, your own control plane, which is what we had at Joyent. Ultimately you're pretty limited. You've got the problems that you can obviously solve the ones that are in your own software, but the problems that are beneath you, the problems that are in the hardware platform, the problems that are in the componentry beneath you become the problems that are in the firmware. Those problems become unresolvable and they are deeply, deeply frustrating. Um, and we just saw a bunch of them again, they were comical in retrospect. And I'll give you like a couple of concrete examples just to give you, you an idea of what kind of what you're looking at. One of the, our data centers had really pathological IO latency. We had a very database, uh, heavy workload. And this was kind of right at the period where you were still deploying on rotating media on hard drives. So this is like, so an all flash by, did not make economic sense when we did this in 2016. This probably be m interesting to know like when was the kind of the last time that, that actual hard drives made sense? Because I feel this was close to it. So we had a, a bunch of pathological, uh, IO problems. But we had one data center in which the outliers were actually quite a bit worse. And there was so much going on in that system, it took us a long time to figure out like why and because when you, when you're IO you're seeing worse IO, I mean you're naturally, you want to understand like what's the workload doing? You're trying to take a first principles approach. What's the workload doing? So this is a very intensive database work to support the object storage system that we had built called Manta. And the metadata tier was stored in. We were using postgres for that. And that was just getting absolutely slaughtered. Um, and ultimately very IO bound with these kind of pathological IO latencies. Uh, and as we trying to peel away the layers to figure out what was going on. And I finally had this thing. So it's like, okay, we are seeing at the device layer, at the disk layer, we are seeing pathological outliers in this data center that we're not seeing anywhere else. And that does not make any sense. And the thought occurred to me, I'm like well maybe uh, do we have different rev of firmware on our HGST drives? HGST now part of WD Western Digital were the drives that we had everywhere. And um, so maybe we had a different, maybe I had a firmware bug. Not be the first time in my life at all that I would have a drive firmware issue. Uh, and I went to go pull the firmware rev and I'm like, Toshiba makes hard drives. So we had, I mean, and I had no idea that Toshiba uh, even made hard drives, let alone that they were, they were in our data center. I'm like what is this? And as it turns out, and this is you know, part of the challenge when you don't have an integrated system which not to pick on them, but Dell doesn't. And what Dell would routinely put just sub, make substitutes. And they make substitutes that they, you know, it's kind of like you're going to like, I don't know, instacart or whatever and they're out of the thing that you want. So you know, you're someone makes a substitute and like sometimes that's okay, but it's really not okay in a data center. And you really want to develop and validate an end to end integrated system. And in this case like Toshiba doesn't, I mean Toshiba does make hard drives, but they are a, or the data they did, uh, they basically were uh, not competitive. And they were not competitive in part for the reasons that we were discovering. They had really serious firmware issues. So these were drives that would just simply stop, stop acknowledging any reads from the order of 2700 milliseconds. Long time, 2.7 seconds. Um, and that was, it was a drive firmware issue, but it was highlighted like a much deeper issue which was the simple lack of control that we had over our own destiny. Um, and it's an example among many where Dell is making a decision that lowers the cost of what they are providing you marginally, but it is then giving you a system that they shouldn't have any confidence in because it's not one that they've actually designed. And they leave it to the customer, the end user, to make these discoveries. And these things happen up and down the stack and for every, for whether it's. And not just to pick on Dell because it's true for hpe, it's true for Super Micro, uh, it's true for your Switch vendors, it's true for storage vendors where the one that is left actually integrating these things and trying to make the whole thing work is the end user sitting in their data center. There's not a product that they can buy that gives them elastic infrastructure, a cloud in their own dc. The product that you buy is the public cloud. Like when you go in the public cloud you don't worry about the stuff because it's AWS's issue or it's GCP's issue. And they are the ones that get this to ground and they, and this is kind of, you know, the eye opening moment, not a surprise. Uh, they are not Dell customers, they are not HPE customers, they are not Super Micro customers. They have designed their own machines and to varying degrees depending on which one you're looking at. But they've taken the clean sheet of paper and the frustration that we had kind of at Joyent and beginning to wonder and then Samsung and kind of wondering what was next. Uh, is that what they built was not available for purchase in the data center, you could only rent it in the public cloud. And our big belief is that public cloud computing is a really important revolution. Infrastructure doesn't feel like a different deep thought. But cloud computing is a really important revolution. It shouldn't only be available to rent, you should be able to actually buy it. And there are a bunch of reasons for doing that. Uh one, and the one we saw at Samsung is economics, which I think is still the dominant reason where it just does not make sense to rent all of your compute in perpetuity. But there are other reasons too. There's security, there's risk management, there's latency. There are a bunch of reasons why one might want to own one's own infrastructure. But uh, that was very much the, so the genesis for Oxide was coming out of this very painful experience and a painful experience that because I mean a long answer to your question about like what was it like to be at Samsung scale? Those are the kinds of things that we, I mean in our other data centers we didn't have Toshiba drives, we only had the HDSC drives. But it's only when you get to this larger scale that you begin to see some of these pathologies. But these pathologies then are really debilitating in terms of those who are trying to develop a service on top of them. So it was, it was very educational in, in that regard. And, uh, you're very grateful for the experience at Samsung in terms of opening our eyes to, to the challenge of running at that kind of scale.

Speaker A: Yeah. Because I think as software engineers, a lot of times we treat the hardware as a given. And, um, it sounds like in this case, I mean, maybe the issue is not so much that Dell or HP as a company doesn't own every single piece that they're providing you, but rather the fact that they're swapping pieces in and out without advertising them. And then when it becomes a problem, they're not necessarily willing to deal with the, the consequences of that.

Speaker B: Or they just don't know. I mean, I think they just genuinely don't know. I mean, I think that they. It's not like they're making a deliberate decision to kind of ship garbage. It's just that they are making. I mean, I think it's exactly what you said about, like, not thinking about the hardware. It's like, what's a hard drive? Like, what's. I mean, it's a hard drive. It's got the same specs as this other hard drive. And it's like, you know, it's a little bit cheaper, so why not? It's like. Well, there, uh, are some reasons why not. And one of the reasons why not is like, uh, even a hard drive, whether it's rotating media or flash, like, that's not just hardware. There's software in there. And that the software is like, not the same. I mean, there are components where it's like. There's actually. Whether, you know, if you're looking at like a resistor or a capacitor or something like this. Yeah. If you've got two parts that are within the same tolerance, like, sure, maybe. Although even the double E's, I think, would be, uh, objecting to that a little bit. But the more complicated you get, and certainly once you get to kind of the hardware that we think of, a microprocessor, a network interface card, a hard Drive or an NVMe drive, those things are super complicated. And there's a whole bunch of software inside of those things, the firmware, and that's the stuff that, that you can't. I mean, uh, you say that software engineers don't Think about that. It's like you. No one can really think about that because it's proprietary, that's kind of welded shut and you've got this abstraction into it. But the, the way that thing operates is very core to how the thing in aggregate will behave. And I think that you. The kind of the fundamental difference between Oxide's approach and the approach that you get at a Del, hp, Suit, Micro, whatever, is really thinking holistically in terms of hardware and software together in a system that ultimately delivers cloud computing to a user. And there's a lot of software at many, many, many, many different layers. And it's very important to think about, about that software and that hardware holistically as a single system.

Speaker A: And during that time at Joyent, when you experience some of these issues, was it more of a case of you didn't have enough servers experiencing this? So if it would happen, you might say, like, well, this one's not working, so maybe we'll just replace the hardware. What was the thought process when you were working at that smaller scale and how did these issues affect you?

Speaker B: Yeah, at the smaller scale, uh, you see fewer of them, right? You just see, it's like, okay, what you might see is like, that's weird. We saw this in one machine versus seeing it in 100 or 1,000 or 10,000. Um, so you just, you just see them, uh, less frequently. As a result, they are less debilitating. Um, I think that it's when you go to that larger scale, those things that become, that were unusual now become routine and they become debilitating. Um, so it really is in many regards a function of scale. Uh, and then I think it was also, you know, it was a little bit dispiriting. That kind of the substrate we were building on really had not improved. Um, and if you look at, you know, if you buy a computer server, buy an x86 server, there is a very low layer of firmware. The bias, the basic input output system, the UEFI bias. And this is like an abstraction layer that has existed since the 80s and hasn't really meaningfully improved. Um, the transition to UEFI happened with, I mean, ironically, with Atanium, um, you know, two decades ago. But beyond that, like this low lay, this lowest layer of platform enablement software is really only impeding the operability of the system. Um, you look at the baseboard management controller, which is the kind of the computer within the computer, there is a, uh, there is an element in the machine that needs to handle environmentals, that needs to handle Operate the fans and so on. Uh, and that traditionally has, is the spaceport management controller. And that architecturally just hasn't improved in the last two decades. And you know, it's a proprietary piece of silicon generally from a company that no one's ever heard of called a speed, which has to be is written all in caps, so I guess it needs to be screamed. Um, ASPEED has a proprietary part that has a. There is a root password. Infamously there is. The root password is encoded effectively in silicon. So uh, which is just. And for um, anyone who kind of goes deep into these things like oh my God, are you kidding me? Um, when we first started oxide, the WI FI password was a fraction of the ASPEED root password. For the bmc, it's kind of like a little bit, little BMC humor. Um, but those things, it was just dispiriting that the state of the art was still basically personal computers running in the data center. Um, and that's part of what was the motivation for doing something new.

Speaker A: And for the people using these systems, whether it's the baseboard management controller or it's the BIOS or UFO UEFI component, what are the actual problems that people are seeing?

Speaker B: Oh man, I, ah, you're going to have like some fraction of your listeners, maybe a big fraction or like yeah, like what are the problems? That's a good question. And then you're going to have the people that actually deal with these things who are like their heads already hit the desk being like what are the problems? Like what are the non problems? Like what works? Actually that's like a shorter answer. Um, I mean there are so many problems and a lot of it is just like, I mean there are problems just architecturally. These things are just so, I mean, and you could the problems spread to the horizon so you can kind of start wherever you want. But I mean it's like as a really concrete example, okay, so the BMCs that the computer within the computer that needs to be on its own network. So you now have like not one network, you got two networks that and that network by the way, that's the network that you're going to log into to like reset the machine when it's otherwise unresponsive. So that going into the BMC you can uh, are you're able to control the entire machine? Well, it's like, all right, so now I've got a second network that I need to manage. What is running on the bmc? Well, it's running some ancient, ancient version of Linux Ah, that you got. It's like well how do I, how do I patch that? How do I like manage the vulnerabilities with that? Because if someone is able to root your bmc, they control the system. So it's like this is not you've and now you've got to go deal with all of the operational hair around that. How do you upgrade that system? Updating the bmc, Uh, I mean it's like you've got this like second shadow bad infrastructure that you have to go manage. Generally not open source. There's something called openbmc, um, which um, people use to varying degrees. But you're generally stuck with the proprietary bmc. So you're generally stuck with ILO from HPE or IDRAC from Dell or uh, the uh, supermicros bmc, the acpbmc. And uh, it is just excruciating pain. Um, and this is assuming that by the way, that everything is behaving correctly. The problem is that these things often don't behave correctly and then the consequence of them not behaving correctly is really dire because it's at that lowest layer of the system. So I mean, I'll give you a concrete example. A customer of theirs reported the Mesa won't disclose the vendor, but let's just say a well known vendor had an issue with their temperature sensors were broken, um, and the thing would always read basically the wrong value. So it was the BMC that had to invent its own a different kind of thermal control loop. And it would index on the, on the actual inrush current. They would look at the current that's going into the CPU to adjust the fan speed. That's a great example of something like that's an interesting idea that doesn't work because that's actually not the temperature. So that software would crank the fans whenever you had an inrush of current. And this customer had a workload that would spike the current and when it would spike the current, the, the, the fans would kick up and then they would slowly degrade over time. Well, this workload was spiking the current faster than the fans would degrade, but not fast enough to actually heat up the part. And ultimately over a very long time in a very painful investigation, this customer determined that like my fans are cranked in my data center for no reason, we're blowing cold air and it's like that this was on the order of like 100 watts, a server of energy that you shouldn't be spending. And ultimately that comes down to this Kind of broken software hardware interface at the lowest layer that has real meaningful consequence, uh, in terms of hundreds of kilowatts, um, across a data center. So this stuff has very, very, very real consequence. And it's such a shadowy world. Part of the reason that your listeners that have dealt with this, their heads will hit the desk is because it is really aggravating to deal with problems at this layer. You feel powerless. You don't control or really see the software that's on them. It's generally proprietary. You're relying on your vendor. Your vendor is telling you that, like, boy, I don't know, you're the only customer seeing this. I mean, the number times I've heard that and I have pledged that we're not going to say that at oxide because it's such an unactionable thing to say. Like, you're the only customer saying this. It's like it feels like, are you blaming me for my problem? Feels like you're blaming me for my problem. Um, and what you begin to realize is that to a degree, these folks are speaking their own truth. Because the folks that are running at real scale on hyperscale, those folks aren't Dell, hp, Super Micro customers. They are actually. They've done their own thing. So it's like, yeah, Dell's not seeing that problem, um, because they're not running at the same scale. Um, but when you do run, you only have to run at modest scale before these things just become overwhelming in terms of the headwind that they present to people that want to deploy infrastructure.

Speaker A: Yeah. So, uh, maybe to help people get some perspective. At, at what point do you think that people start noticing or start feeling these problems? Because I imagine that if you just have a few racks or do you

Speaker B: have a couple racks or the. Are you just wondering? Because. No, no, no. I would think. I think anyone who deploys any number of servers, especially now, especially if your experience is only in the cloud, you're gonna be like, what the hell is this? I mean, just, again, just to get this thing working at all, it is so. It's so hairy and so congealed. Right. It's not designed, um, and it's accreted. And it's so obviously accreted that you are. I mean, nobody who is setting up a rack of servers is going to think to themselves like, yes, this is the right way to go do it. This all makes sense because it's just not. It feels like the kit, I mean, kit car is almost too generous because it implies that there's like a set of plans to work to in the end. Uh, I mean, it's a bag of bolts. It's a bunch of parts that you're putting together. And so even at the smallest scales, that stuff is painful. Just architecturally, it's painful at the small scale, but at least you can get it working. I think the stuff that then becomes debilitating at larger scale are the things that are worse than just like, I can't, like this thing is a mess to get working. It's like the fan issue that, um, where you are now seeing this over hundreds of machines or thousands of machines, um, so it is painful at more or less all levels of scale. There is no level at which the PC, which is really what this is, this is the personal computer architecture from the 1980s. And there is really no level of scale where that's the right unit, I mean, where that's the right thing to go deploy. Especially if what you are trying to run is elastic infrastructure, A, uh, cloud. Because the other thing is like, we've kind of been talking a lot about that hardware layer. Like, hardware is, is just the start. Like you actually got to go put software on that and actually run that as elastic infrastructure. So you need hypervisor. Yes, but you need a lot more than that. You need to actually, you need a distributed database, you need web endpoints, you, uh, need a cli. You need all the stuff that you need to actually go run an actual service of compute or networking or storage. I mean, and for compute, even for compute, there's a ton of work to be done. And compute is by far, I would say the simplest of the three. When you look at like networks, network services, storage services, there's a whole bunch of stuff that you need to go build in terms of distributed systems to actually offer that as a cloud. So I mean, it is painful at more or less every level if you are trying to deploy cloud computing on prem.

Speaker A: And for someone who doesn't have experience building or working with this type of infrastructure, when you talk about a control plane, what, what does that do in the context of the system?

Speaker B: So control plane is the thing that is everything between your API request and that infrastructure actually being acted upon. So you go say, hey, I want to provision a vm. Um, okay, great. We've got a whole bunch of things we're going to provision with that. We're going to provision a VM M, we're going to get some storage that's going to go along with that, that's got a network storage service that's going to come out of. We've, uh, got a virtual network that we're going to either create or attach to. We've got a whole bunch of things we need to go do for that, for all of these. There are metadata components that we need to keep track of this thing that, beyond the actual infrastructure that we create. And then we need to go actually like act on the actual compute elements, the host os, what have you, the switches, what have you, and actually go create these underlying things and then connect them. And there's of course, the challenge of just getting that working is a big challenge. Um, but getting that working robustly, getting that working. You know, when you go to provision a vm, all the, the steps that need to happen and what happens if one of those steps fails along the way? What happens if. You know, one thing we're very mindful of is these kind of, you get these long tails of like, why, you know, generally our VM provisioning happened within this time, but we get these long tails where it takes much longer. What's going on, where in this process are we, are we actually spending time? Uh, and there's a whole lot of complexity that you need to go deal with that. There's a lot of complexity that you need to go deal with this effectively, this workflow that's going to go create these things and manage them. Um, we use a pattern that are called sagas, actually a database pattern from the 80s. Uh, Katie McCaffrey is a database researcher who I think, uh, reintroduced the idea of sagas, um, in the last decade. Um, and this is something that we picked up. Um, and I've done a lot of really interesting things with, um, to allow for this kind of these workflows to be, to be managed and done so robustly in a way that you can restart them and so on. Uh, and then you got, so you get this whole distributed system that can do all this, that whole distributed system that itself needs to be reliable and available. So if you, you know, uh, you need to be able to. What happens if you, if you pull a sled or if a sled fails? How does the system deal with that? How does the system deal with getting another sled added to the system? Like, how do you actually grow this distributed system and then how do you update it? How do you actually go from one version to the next? And all of that has to happen across an air gap where this is going to run as part of the computer. So there are, it is fractally complicated. There, there is A lot of complexity here in, in software, in the software system and all of that we kind of, we call the control plane. Um, and this is what exists at aws, at gcp, at Azure. When you are hitting an endpoint that's provisioning an EC2 instance for you, there is an AWS control plane that is doing all of this and has uh, some of these similar aspects and certainly some of these similar challenges.

Speaker A: And for people who have run their own servers with something like say VMware or Hyper V or Proxmox. Are those in the same category?

Speaker B: Yeah, I mean a little bit. I mean to kind of like vSphere, yes. VMware, no. So it's like you uh, VMware ESX is kind of a key building block upon which you can build something that is a more meaningful distributed system. When it's just like a machine that you're provisioning VMs on, it's like, okay, well that's actually you as the human might be the control plane. Like that's, that's a much easier problem. Um, but when you've got, you know, tens, hundreds, thousands of machines and you need to do it robustly, you need something to coordinate that activity and you know, you need to pick which sled you land on and you need to be able to move these things, you need to be able to update that whole system. That's when you're getting into a control plane. So you know, some of these things have kind of edged into a control plane. Certainly VMware, um, now Broadcom, um, has delivered something that's kind of cloud ish. Um, I think that for folks that are truly born on the cloud, it's, it still feels somewhat uh, like you're going backwards in time when you, when you look at these kind of on prem offerings. Um, but it's got these aspects to it for sure. Um, and I think that we're um, some of these other things when you're just looking at KVM or just looking at proxmox, you kind of need to connect it to other broader things to turn it into something that really looks like manageable infrastructure. And then many of those projects are really, they're either proprietary projects, proprietary products like vSphere, um, or you are really dealing with open source projects that are not necessarily aimed at the same level of scale. Um, you know, you look at a Proxmox or uh, um, you'll get an OpenStack. Um, and you know OpenStack is just a lot of things, right? I mean OpenStack has got so many. OpenStack was kind of a free for all, for every infrastructure vendor. Um, and I, you know, there was a time people like, don't you. Aren't you worried about all these companies together that you know are coming together for OpenStack? I'm like, haven't you ever worked for like a company? Like companies don't get along, by the way. It's like having multiple companies work together on a thing. That's bad news, not good news. And I think, you know, one of the things that OpenStack has definitely struggled with, kind of with what actually the. There's so many different kind of bender elements in there that it's very much not a product, it's a project that you're trying to run, but that very much is in. I mean that's similar certainly in spirit.

Speaker A: And so I think this is kind of like you were alluding to earlier. The piece that allows you to allocate compute, storage, manage networking, gives you that experience of I can go to a, uh, web console or I can use an API and I can spin up machines, get them all connected. At the end of the day, the control plane is allowing you to do that in hopefully a user friendly way.

Speaker B: That's right, yep. And uh, I mean in order to do that in a modern way, it's not just like a user friendly way. You really need to have a CLI and a web UI and an API. Those all need to be drawn from the same kind of single ground truth. Like you don't want to have any of those be an afterthought for the other. You want to have the same way of generating all of those different endpoints and entries into the system.

Speaker A: If you take your time at joyent as an example, what kind of tools existed for that versus how much did you have to build in house as far as the hypervisor and managing the compute and all that?

Speaker B: Yeah, so we built more or less everything in house. I mean what you have is, um. And I think over time we've gotten slightly better tools. Um, I think. And maybe it's a little bit easier to talk about the kind of the tools. We started at Oxide because we kind of started with a clean sheet of paper at Oxide. We knew we wanted to go build a control plane, but we were able to kind of go revisit some of the components. So actually. And maybe I'll talk about some of those changes. So when we, for example, at Joyent, when we were building a cloud at Joyent, there wasn't really a good distributed database. Um, so we were using Postgres as our database for metadata and there were a lot of challenges. And Postgres is not distributed database. It's running with a primary secondary architecture. And there's a bunch of issues there, many of which we discovered the hard way, um, when we were coming to Oxide. You have much better options to pick from in terms of distributed databases. There was a period that now seems maybe potentially brief in hindsight, but of a really high quality open source distributed databases. So there were really some good ones to pick from. Um, we built on cockroachdb on crdb. Um, so that was a really important component that we had at Oxide that we didn't have at Joyent. Um, so I wouldn't say we were rolling our own distributed database. We were just using Postgres, uh, and dealing with an enormous amount of pain there in terms of the surround. Um, on top of that a control plane is much more than a database obviously. Uh, and you've got to deal with, um, there's a whole bunch of software that you need to go right, um, to be able to transform these kind of API requests into something that is reliable infrastructure. Right. And there's a lot to that. Uh, especially when networking gets in the mix, when storage gets in the mix, uh, there are a whole bunch of like complicated steps that need to be done. Um, at Joyent. Um, we in part because the history of the company and like look, this, this just is not going to sound good, but it just is what it is and I'm just going to own it. We did it all in Node, um, at Joyent, which I, I, I know it sounds really, right now, just sounds like you, you built it with Tinker Toys. You okay? Uh, did you think it was you built the skyscraper with Tinker Toys? It's like. Well, okay, we actually, we had greater aspirations for the Tinker Toys once upon a time and it was better than, you know, than Twisted Python and Event Machine from Ruby. And we weren't going to do it Java. All right, so, but let's just say that that experiment, uh, that experiment did ultimately end in a predictable fashion. Um, and uh, we, we decided that maybe Node was not going to be the best decision long term. Um, Joyent was the company behind Node js, uh, back in the day Ryan Dahl worked for Joyent. Uh, and then uh, we uh, landed that in a foundation in about uh, what, 2015, something like that. Um, and began to consider our world, uh, beyond Node, a big tool that we had in the arsenal when we started Oxide is Rust. Um, so indeed the name of the company is a tip of the hat to the language that we were pretty sure we were going to be building a lot of stuff in, namely Rust. And uh, Rust is uh, has been huge for us, a very important revolution in programming languages. You know there have been different people kind of coming in at different times and I kind of came to Rust in what I think is like this big kind of second expansion of rust in 2018 when a lot of technologists were think sick uh, of Node and also sick of Go and also uh, sick of C and wondering is there going to be something that gives me the performance that I get out of C? The robustness that I can get out of a C program but is often difficult to achieve. But can I get that with kind of some of the velocity of development, although I hate that term, some of the speed of development that you get out of a more interpretive language. Um, and then by the way can I actually have types? I think types would be a good idea. Uh, and Rust obviously hits the sweet spot of all that. Um, and has been absolutely huge for us. I mean we knew when we started the company again, zoxide, uh, we were going to be using Rust in quite, quite a few places, but we weren't doing it by fiat. Um, we wanted to actually make sure we're making the right decision um, at every different, at every layer. I think what has been surprising is the sheer number of layers at which we use Rust. In terms of we've done our own embedded firmware in Rust. We've done um, in the host operating system which is still largely in C, but very big components are in Rust. The hypervisor propolis is all in Rust, uh, and then of course the control plane distributed system on that is on Rust. So that was a very important thing that we very much did not need to build ourselves. We were able to really leverage a terrific community. Um, we were able to use, and we'd done this at Joyent as well. But at Oxide we've used Illumos as a host OS component which uh, our variant is called Helio. Um, we've used ah, Beehive, um, as that kind of internal hypervisor component. We've made use of a bunch of different open source components to build this thing, um, which has been really, really important for us. Uh, and open source components that didn't exist even like five years prior. That's part of why we felt that 2019 was the right time to start the company.

Speaker A: So when we started Oxide, and you had mentioned that at Joyent, you had tried to build this in Node, what. What were the issues or the challenges that you had doing that?

Speaker B: Oh, boy. Yeah. Again, we kind of had higher hopes in 2010, I would say, when we sat on this, um, the problem that we had just writ large. Um, JavaScript is really designed to allow as many people on earth to write a program as possible, which is good. I mean, that's a laudable grade goal. That is the goal ultimately of such, uh, as it is of JavaScript. It's actually hard to know what the goal of JavaScript is, unfortunately, because Brendan Eich never actually wrote a book. So there is not a canonical. You've got kind of Doug Crockford and other people who've written things on JavaScript, but it's hard to know kind of what the original intent of JavaScript is. The name doesn't even express original intent. Right. It was called Livescript and it was kind of renamed to JavaScript during the Java frenzy of the late 90s. A name that makes no sense. There is no Java in JavaScript. That is kind of, I think, revealing to kind of the, uh, unprincipled Mess that is JavaScript. It's very pragmatic at some level, um, and allows anyone to. It makes it very easy to write software. The problem is it's much more difficult to write really rigorous software. Uh, and this is what I should differentiate JavaScript from TypeScript. This is really what TypeScript is trying to solve. TypeScript is like, how can I think TypeScript is a. Is a great step forward? Because TypeScript is like, how can we bring some rigor to this? Like, yes, it's great that it's easy to write JavaScript, but that's not. We don't want to do that for. Absolutely. I mean, that's not the only problem we're trying to solve. We actually want to be able to write rigorous software and it's actually okay if it's a little harder to write rigorous software. That's actually okay if it gets leads to more rigorous artifacts. Um, but in JavaScript, I mean, concrete example, you know, there's nothing to prevent you from referencing a property that doesn't actually exist in JavaScript. So if you fat finger a property name, you are relying on something to tell you. By the way, I think you've misspelled this because there is no type definition for this thing. And I don't know that you've got one that's spelled correctly, one that's spelled incorrectly, that's often undefined. And then the, when you actually go, so you've got this typo that is lurking in your, what you want to be rigorous software. And if you don't execute that code, like, you won't know that's there. And then you do execute that code and now you've got a, you've got an undefined object and now that's either going to be an exception or it can again, depends on how that's handled. It can be really difficult to determine the origin of that, of that error, of that programming. That is a programmer error. And one of the big challenges that we had with Node is that programmer errors and operational errors, like, you know, I'm out of disk space as an operational error, those get conflated and it becomes really hard. And in fact, I think the language wanted to make it easier to just kind of, uh, drive on in the event of all errors. And it's like actually not what you want to do if you're trying to build a reliable, robust system. So we had no end of issues. We've got a lot of experience developing rigorous systems, um, again, coming out of operating systems development and so on. And we, we brought some of that rigor, if, strangely, to JavaScript. So one of the things that we did is we brought a lot of postmortem diagnosability and observability to Node. And so if, if one of our node processes died in production, we would actually get a core dump from that process, a core dump that we could actually meaningfully process. So we did a bunch of kind of wild stuff, I mean, actually wild stuff where we could actually make sense of the JavaScript objects in a binary core dump, um, things that we thought were really important. And this is the, the rest of the world just looks at this being like, what the hell is this? I mean, it's so out of step with it. The problem is that we were trying to bridge two disconnected cultures of one, developing really rigorous software and really designing it for production diagnosability, and the other, really designing it to software to run in the browser and for anyone to be able to kind of liven up a web page, right, is kind of the origin of Live script and then JavaScript. And we were kind of the only ones sitting at the intersection of that. And you begin when you are the only ones sitting at that kind of intersection, you just are. You're kind of fighting a community all the time. And we just realized that we are. There were so many things that the community wanted to do that we felt are like, no, no, this is going to make software less diagnosable. It's going to make it less robust. And then you realize like, um, we're the only voice in the room because we have got, we have got desires for this language that it doesn't have for itself. And this is when you realize you're in a bad relationship with software. It's time to actually move on. And in fact, actually several years after we'd already kind of broken up with Node, um, and it was like, it was a bit of an acrimonious breakup. There was a, ah, famous infamous fork of Node called IO js. Um, and this was viewed because the community thought that Joyent was being, was not being an appropriate steward of Node Js and was, uh, not allowing more things to come into Node. And of course the reason that we of course felt that we were being a careful steward and we were actively resisting those things that would cut against its fitness for a production system. But somebody, the community saw it and they forked. Um, and I think that we knew before the fork that's like, this is not working and we need to get this thing out of our hands and we're the wrong hands for this. This needs to be in a foundation. Uh, and so we kind of gone through that breakup. Uh, and maybe it was two years after that that friend of mine who was um, running the uh, Node Summit unfortunately now passed away. Charles B. Um, but Charles, venture capitalist, great guy and Charles was running Node Summit and came to me in 2017. He's like, I really want you to keynote Node Summit. And I'm like, charles, I'm not going to do that. I've got nothing nice to say. Like, this is. You don't. I'm the last person you want to keynote. He's like, oh, if you have nothing nice to say, you should definitely keynote. You're like, oh God, okay, here we go. He's like, no, I really want you to talk about, like you should talk about the joint breakup with Node Js. I'm like, oh man. And that led to a talk that I'm really happy that I gave because it was a very important talk for me personally called, uh, Platform as a reflection of Values. And really looking at the values that we had for node and the values that node had for itself and they didn't line up. And the problem is that the values that node had for itself and the values that we had for node are all kind of positives, right? Like there's nobody in the node community who's like, I don't want rigor. I hate rigor. It's just that if they had to choose between rigor and making the language approachable, they would choose approachability every single time. They would never choose rigor. And you know, that was a, that was a big eye opener. I do. I would say if you watch this talk, because I knew that there's like the audience was going to be filled with, with people who had been a part of the fork in 2014, I think was the fork, the IO JS fork. And I knew that there were, there were some, you know, some people that were, um, had been there for the fork. And I set a little bit of a trap for the audience. But the, and the trap, I said, you know, I kind of talked about the values that we had and the aspirations we had for Node, the aspirations that Node had for itself and how they were different. And you know, I'm like, look, in hindsight, like a fracture was inevitable and in 2014 there was finally a fracture. And do people know what happened in 2014? And if you, if you, you could listen to that talk, everyone almost says in unison, like, iojs. I'm like, oh, right, IO js right. That's actually not what I was thinking of. And I go to the next slide and is a tweet from a guy named T.J. holloway, Chuck, who was the most prolific contributor to Node. And it was his Tweet also in 2014, before the fork, before the IHS fork, explaining that he was leaving Node and that he was going to go. And you, if you turn the volume all the way up, you can hear the audience gasp. And it's just delicious because the community had never really come, had never really confronted why TJ left, um, there. And I went through a couple of folks, Felix, a bunch of other folks, early Node folks that were there in 2010 were leaving in 2014 and they were going to go primarily. And they were going to go because they were sick of the same things that we were sick of. They had hit the same things that we had hit and they were frustrated. I really do believe this, that platforms do reflect their own values. And when you are making a software decision, you are selecting value. You should select values that align with the values that you have for that software. That is, those are, uh, that's way more important than other things that people look at. I think people look at, for example, quote unquote, community size way too frequently. Community size is like, eh, ah, Maybe it can be fine. I've been in very large communities, Node. I've been in super small open source communities like Illumos and D Trace, a bunch of others. There are strengths and weaknesses to both approaches. Just as like there's a strength to being in a big city versus a small town. Me personally, I'll take the small community more or less every time, because the small community is almost always self selecting based on values. And just for the same reason that I like working at small companies or small teams. There's a lot of value to be had in a small community. It's not to say that large communities value us, but again, long answer to your question of kind of where did things go south with Joyent and Node? They went south because the values that we had and the values the community had didn't line up. And that was a very educational experience, as you might imagine.

Speaker A: Yeah. And given that you mentioned how because of those values, some people moved from Node to Go, and in the end, for much of what Oxide is building, you ended up using Rust. What would you say are the values of GO and Rust? And how did you end up choosing Rust, given that?

Speaker B: Yeah, I mean, well, so the value for yeah, Go. I mean, I understand why people move from Node to Go. Go to me was kind of a lateral move. Um, there were a bunch of things that I Go is still garbage collected, um, which I didn't like. Um, Go also is very strange in terms of there are these kind of like, these autocratic kind of decisions that are very bizarre. Um, I mean, generics is kind of a famous one, right? Where GO kind of as a point of principle, didn't have generics, even though GO itself. Actually the innards of Go did have generics. It's just that you, a Go user, weren't allowed to have them. And there was an old cartoon years and years ago about when a technologist is telling you that something is technically impossible, that actually means I don't feel like it. Uh, and there was a certain degree of generics are technically impossible and Go. It's like, actually there are. And I just think that the arguments against generics were kind of disingenuous. Um, and indeed they ended up adopting generics. And then there's some super weird stuff around, very anti assertion, which is like, what, how are you? Why are you? How is someone against assertions? It doesn't even make any sense, but it's like, oh, nope, okay, there's a whole screed on it. Nope, we're against assertions and you know, against versioning, there was another thing like, you know, Rob pike has kind of famously been like, you should always just run on the way to commit. And you're like, is that, is uh, does that make sense? I mean we actually built it so there are a bunch of things like that you're just like, okay, this is just exhausting. And I mean there's some things about GO that are great and plenty, uh, of other things that I just, I'm not a fan of. Um, I think that the, in the end, like GO cares a lot about like compile time. It's super important for go, right? Is very quick compile time. Like okay, but that's like compile time is not like it's not uh, unimportant. It doesn't have zero importance. But I've got other things that are like lots more important than that. Um, what I really care about is I want a high performing artifact and I wanted garbage collection out of my life. I got to tell you, garbage collection to me is an embodiment of this larger problem of where do you put cognitive load in the software development process and what garbage collection is saying. To me, it is right for plenty of other people and the software that they want to develop. But for me and the software that I want to develop, infrastructure software, I don't want garbage collection because I can solve the memory allocation problem. I know when I'm like done with something or not, I mean it's like I, whether that's in, in C with, I mean it's actually like it's really not that hard to not leak memory in, in a C based system and you can give yourself a lot of tooling that allows you to diagnose where memory leaks are coming from. So it's like that is a solvable problem. There are other challenges with that. But like when you are developing a really sophisticated system that has garbage collection is using garbage collection, you spend as much time trying to dork with the garbage collector to convince it to collect the thing that you know is garbage. You're like, I've got this thing, I know it's garbage. Now I need to use these like tips and tricks to get the garbage collector. I mean it's like it uh, feels like every Java performance issue goes to like minus XX colon. Use the other garbage collector, whatever one you're using, use a different one. And using a different uh, approach, it's like, so you're in this. To me it's like you're in the worst of all worlds. Where the reason that garbage collection is helpful is because the programmer doesn't have to think at all about this problem. But now you're actually dealing with these long pauses in production. You're dealing with all these other issues where actually you need to think a lot about it. And it's kind of. It's witchcraft. It's this black box that you can't see into. So it's like, what problem have we solved exactly? I, uh, mean, so the fact that Go had garbage collection, it's like, uh, no, I do not like. And then you get all the other, like, weird fatwas and, you know, everything else, like, no, thank you goes to no thank you. For me, I get why people like it or use it, but it's. It's just that was not going to be it. I'm like, I want C. But I. There are things I didn't like about C2. I was looking for something that was going to give me the deterministic kind of artifact that I got out of C. But I wanted library support in C is tough because there's the. It's all convention, you know, there's a bunch of other things that are just thorny. And I remember thinking vividly in 2018, I'm like, well, it's rust or bust. I'm gonna go into rust. And, uh, I hope I like it, because if it's not this, it's gonna, like, I'm gonna go back to C. I'm like, literally trying to figure out what the language is for the back half of my career. Um, and when I, you know, did what a lot of people were doing at that time, and people have been doing since of, you know, really getting into rust and really learning it. Appreciating the difference in the. The model for sure. The ownership model people talk about, that's also obviously very important. It was the error handling that blew me away. And the idea of, like, algebraic types. I never really had algebraic types. Um, and the ability to have error handling is one of these really, uh, you really appreciate these things where it's like, how do you deal with a function that can either succeed and return something or it can fail? And the way C deals with that is bad with these kind of sentinels for errors. And does negative one mean failure? Does zero mean failure? Some C functions. Functions. Zero means failure. Traditionally in Unix, zero means success. And what if you want to return a file descriptor? It's like, oh. Then it's like, okay. Then it'll be like zero through positive N will Be a valid result. Negative numbers will be. And was it negative one? And I said, er, no. Or is it a negative number? That didn't. I mean, it's like. And that's all convention, right? People do all those different things and it's all convention and it's easy to get wrong, easy to have bugs, can't be statically checked and so on. Um, and then what Go says is like, well, you're going to have like two return values and then you're going to have to just constantly check all of these all the time. Um, which is also kind of gross. Um, JavaScript is like, hey, let's toss an exception if we don't like something. If you see an error, we'll throw an exception. There are a bunch of reasons I don't like that. Um, and you look at what Rust does where it's like, no, no, no, we're going to have these algebraic types, which is to say this thing can be a this thing or that thing. But. But it has to be one of these. And by the way, you don't get to process this thing until you conditionally match on one of these things. You're going to have to have a pattern match on this thing to determine if it's a this or that and if it in. In the result type that you. The result is a generic. Where it's like, it's going to be either the thing that you want to return, it's going to be an okay, that contains the thing you want to return, or it's going to be an error that contains your error. And, and it forces your code to deal with that. And what that does is it shifts the cognitive load from the person that is operating this thing in production to the actual developer that is in development. And I think that to me is like, I love that shift. Um, and that shift to me is really important. Um, and that's what I was missing. That's what Rust gives you. Rust forces you to think about your code as you write it, but as a result, you have an artifact that is much more supportable, much more sustainable and much faster.

Speaker A: Yeah, it sounds like you would rather take the time during the development to think about these issues, because whether it's garbage collection or it's air handling at runtime, when you're trying to solve a problem, then it's much more difficult than having dealt with it to start with.

Speaker B: Yeah, absolutely. And I just think that, like, why also, like, if it's software, if it's. Again, if it's infrastructure, Software, I mean, the kind of, the question that you should have when you're writing software is how long is the software going to live? How many people are going to use this software? Uh, and if you are writing an operating system, the answer for this thing that you're going to write, it's going to live for a long time. Like if we just look at plenty of aspects of the system that have been around for a, for decades, it's going to live for a long time and many, many, many people are going to use it. Why would we not expect people writing that software to have more cognitive load when they're writing it? To give us something that's going to be a better artifact. Now, conversely, it's like, hey, I kind of don't care about this. And like, I don't know, I'm just like, I want to see if this whole thing works. I've got, I like, I'm just stringing this together. I don't like, no, the software like will be lucky if it survives until tonight. But then like, who cares? Yeah, yeah, garbage clock, you know, if you're prototyping something, whatever. And this is why you really do get like, you know, different choices, different technology choices, depending on the way that you want to solve the problem at hand. And for the software that I want to write, I do like that cognitive load that is upfront. Um, and although I think, I think the thing that is really wild, that is the twist that I don't think anyone really saw coming, is that in a, in an LLM age that like the cognitive load up front almost needs an asterisk on it because so much of that can be assisted by an LLM. And now, I mean, I would like to believe, and maybe this is me being optimistic that the, in the LLM, um, age we will see. I mean, Rust is a great fit for the LLM age because the LLM itself can get a lot of feedback about whether the software is written is correct or not. Much more so than you can for other environments.

Speaker A: Yeah, that is a interesting point in that I think when people first started trying out the LLMs to code, it was really good at these maybe looser languages like Python or JavaScript and initially wasn't so good at something like Rust. But it sounds like as that improves, if it can write it, then because of the rigor or the memory management or the error handling that the language is forcing you to do, it might actually end up being a better choice for people using, yeah, LLMs.

Speaker B: Yeah, absolutely. It gives you more Certainty in the artifact that you've delivered. I mean you know a lot about a Rust program that compiles correctly. I mean there are certain classes of errors that you don't have, um, that you actually don't know on a C program or a Go program or uh, a JavaScript program. I think that's going to be really important. I think we are on the cusp. Maybe we've already seen it, this kind of great bifurcation in the software that we write where the rigorous software becomes much more important. We have this foundational software that we're going to rely on as much more bedrock and then we're going to have much more software where that rigor is not as m much of a constraint because the constraints are the ability for it to be uh, customized to my need and done very quickly. And it's going to feel like I think two different worlds. Um, and I think in an exciting way, I think it's going to be, I think that the future is definitely exciting for software development.

Speaker A: Another interesting decision about the oxide computer is you chose an operating system that I think most people aren't familiar with. Rather than a Linux or a FreeBSD, you chose, I believe it's Alumos. Is that how you pronounce it?

Speaker B: Yeah, yeah, yeah, yeah. I mean, yes. So Illumos, which very much inherits from uh, Open Solaris, um, which inherits from the Solaris, uh, heritage which was sunoss4x before that was the Solaris itself is this unholy love child of uh, sonos4x and the true kind of BSD lineage and the AT&T lineage in Unix and SVR4 in many ways I know that people haven't necessarily heard of it, um, but it is true Unix in a real historical sense, um, in a way that Linux and even the bsds are not. Uh, and there are aspects where it shows it, but in terms of. It is an idiosyncratic decision. It's not one that we took lightly. People assume like, oh, it's a bunch of old sun folks like of course they're going to pick lumos like this. That's. And we had a lot of operational experience with smart OS at ah, join. But that's, it's more nuanced than that. I would say it is true that a bunch of the technologies that we have developed over the years, uh, we developed them for good reasons, um, and we don't want to be without them. I mentioned the post mortem diagnosability of JavaScript, uh, postmortem diagnosability is really important to us and to me debugability is really important. Um, and debugability is not something that other operating systems have taken that seriously. Just to put it bluntly, you can judge a lot about an operating system by its built in debugger. And uh, there's not really a built in debugger present for Linux. But it's more than just that. If you look at Linux in particular. Linux is a kernel. It is not. And this is something that Torvalds makes clear in his defense at more or less every juncture. Linux is a kernel. And this is the kind of the very famous, the stallmanism of what you are calling Linux. I call Canoe Linux. And it's obviously on the one hand a little bit ridiculous, on the other hand not wrong in that the software that you need to actually have a functional system is a lot more than just Linux. You need a libc, right? And there's actually more than one libc. Is it going to be muscle? Is it going to be glibc? There are some other alternate libc's. Um, you need. And part of the problem is if you're going to use a Linux kernel as the basis for a host operating system that's nowhere near enough because you need to now build an entire distro effectively of all these different tools on top of that and the maintenance burden of that is off the charts. Um, and it's actually funny one of our colleagues, Laura Abbott, came from Red Hat and I'm like well this will be interesting because again my mind was open, more open than it had been in a long time about like maybe we should use Linux, I don't know. And Laura felt strongly that we shouldn't. But for reasons that were different, Laura was very concerned about what the distro management burden would be. And she had really seen that upfront at Red Hat and she's like do not m make this decision lightly that you've got to make all these other decisions. It's something I really like I remember at the time being like okay, yeah, you know, I get that. But then in the years since I'm like oh wow, that's a really big deal. That's a bigger deal than I realized. And so we, you know, we don't have to do that. We get an operating system that's got a lot of stuff like built in. Um, and of course it means that like we've also, it's talk about small communities, definitely a small community. Um, and there are Challenges there too. Um, but it is one that has allowed us to get the technologies that we really need. CFS containers, D trace, the virtual networking, a bunch of things that we really needed we had in place and then allowed us to also move the system in a dimension that we needed and wanted to move it. So um, for us it's been the right decision. But I think again the Internet is kind of disbelieving that we actually were at a real juncture there. But uh, but we were, um, we hand on heart, were uh, actually at the moment where I was like most potentially intrigued or at least wanted to explore something like Linux, uh, Torvalds was just on an absolute bender against zfs, which he's done occasionally. And I'm like this is, dude, this is the wrong time to be on this. This is ZFS tamper tantrum, which he's again done a couple of times. Um, but was kind of the. And a problem with Linux is that ZFS is not a first class, first class file system. Um, it's always been on the outside because of this kind of these ridiculous licensing concerns. And it was a reminder of like, oh yeah, write this. Yeah, okay, I think, yeah, we'll go our own way. Thank you.

Speaker A: And you mentioned one of the risks would be managing your own distribution. But for Illumos you actually did create your own distribution.

Speaker B: Yeah, you do, but like it's yes and no. I mean you're creating your own distribution but you're not having to like go be like, okay, God, we got to go get our own version of gdb, we got to go get our own version of Graph, we got to get our own version of tar, we got to get our own version. And it's like you. That's what I mean about like in Illumis and FreeBSD is like this too. By the way, it's not the only operating system like this actually. Windows is like this. It is the operating system is, is the kernel plus the system libraries plus the commands to run. So you're, you're not having these things operate across purposes. You've got and technologies like D Trace that are able to straddle this user kernel boundary in a really important way. So that's what I mean by a distro. We've got our own derivative. Um, but we actually very deliberately, in part because of our experiences at Joyent, uh, we really get stuff upstream. So we are as much as we humanly can, everything is upstream. The other thing we knew we wanted to go do or had a hunch we wanted to go do, we knew that we wanted to eliminate the BIOS completely. And for all the things that we did at Oxide, which is we did our own switch, we did our own compute sled, we done our own microcontroller operating system. In many ways the riskiest thing was this idea of eliminating the bias. And so we knew one of the things we wanted to go do was actually do that lowest level of platform initialization ourselves from the actual operating system, from the host operating system. And that required us to get this thing really under a knife. We managed to pull that off, um, but that would have been much thornier, I think, trying to kind of battle, um, an upstream, because it's a very different architecture. What we have is not a PC architecture, it's an x86 machine AMD. But we have done. We in the host operating system have done the lowest level of platform initialization that's historically done by the bios, um, which was a huge challenge and one, by the way, that AMD didn't think we could do. AMD was very surprised when we were sometimes, you know, uh, that somebody is like, not explicitly underestimating you, but then they're very surprised when you've succeeded and you realize like, oh, okay, yeah, actually I guess you were underestimating me. I didn't realize that. And that was definitely the case here where, uh, AMD was very surprised that we lived, um, and that we actually had a machine that could boot, um, without running a UEFI bias. But that very much was hand in glove with this decision to have our own derivative for Helios.

Speaker A: And it sounded like possibly for this to work you would have to upstream things to Illumos. Is that right?

Speaker B: I mean, for sure, you just don't want to be, you don't want to be battling a. You're going to end up with your own kind of derivative at some level. And this is where it's actually just helpful to have a smaller community, frankly. Um, just makes it easier to go do that stuff.

Speaker A: Yeah. Because with Linux having so many users it would be much more of a fight trying to. Yeah, everything's a fight.

Speaker B: I mean, everything's a fight is the problem. I mean, you get like. I mean, I'll tell you another thing, it's actually. This is like really funny because, uh, this is something that, um, Torvalds and I actually agree on, uh, pretty thoroughly, which is with the presence of rust in the system. So we've been able to add. I mean, I absolutely believed that, like, we want to have rust in kernel modules and we do in Helios, we've got rust in kernel modules in terms of our oxide packet transformation engine Opte, and that will increase over time. That is an absolute civil war in Linux, ironically one that like again Torvalds and I agree but there are people, other people in Linux who are ardently disagree over, ardently disagree over rust in the kernel. And like that is not a fight I want to have. Like I'm just, I am absolutely disinterested in having an argument over Rust versus C. Like no thank you. That is the peril of a larger community is you end up with people and then you know, a larger community that like look, is not exactly an exemplar of positive community behavior. Um, and there are plenty of people that have been uh, that have done Linux kernel development and then have walked away from it because it was not exactly an uplifting experience. And uh, their treatment of rust is really bad right now. That's like one of the big open challenges that I think Linux has. What does the future of the system look like with respect to Rust? I feel it's like it's. The Torvalds position is the only one that's ultimately tenable in the limit. But clearly there are people inside of Linux, maintainers inside of Linux that ardently disagree with that.

Speaker A: And for those who aren't familiar with the fight, what is his opinion versus the rest of the.

Speaker B: I mean Torvalds I think is. Torvalds is pro. Rust I think is the short answer. I shouldn't speak for him obviously but my read on this from the outside looking in is Torvalds is broadly like, yeah, this is like, this is interesting. The same reasons I think it's interesting and that we should have kernel components written in Rust. Um, but you've got um, I think maintainers of subsystems that have, are ardently opposed to it and are kind of holding aspects of the hostage of the project hostage over it. There's going to be, there's going to continue to be friction um, over the future of the operating system there and like I, I don't want to have debates, I really don't. I mean, I guess maybe, you know, okay, maybe I do kick an occasional hornet's nest on the Internet, but I really like don't want to be spending. I would much rather find a community where we're going to broadly agree on our values and then we then I just find that there is, that there is much less discord um, when you have that kind of that value alignment. Um, and I think that that's one of the big challenges for Linux is there is, there is um, some, some uh, disalignment there in terms of what people view as the future of the operating system

Speaker A: and this choice of your operating system as a user of an oxide computer, would they.

Speaker B: No, you don't know. It's just like, you don't know. Like, you know, if you're a user of aws, you've got no idea what Nitro is or Annapurna or these other things that are like, you know. Do you know whether they use KVM at aws? No, you've got no idea. I mean do they use KVM or Zen? It's like the answer is like, well, it depends on your instance type and the age of the thing. But as a user that's opaque to you. Um, what you've got is like you want to hit an API endpoint, you want to provision an instance and you want to run an operating system of your choice. You want to run Windows or Linux or FreeBSD or whatever in your cloud. That's what you actually want to go do. Um, and that's what we enable you to do.

Speaker A: And something I find interesting about the Oxide Project is you not only write software that's general compute to run on x86, there's also parts of the system that are written for FPGAs and written for ASICS. I wonder if you could explain to people who might not be familiar what the difference is and what it's like building for those things versus running something on a VM, um, on an X86 machine.

Speaker B: Yeah, I mean, so you've got, I mean there's a whole spectrum of. And this is all kind of constitutes the platform for compute. I mean you have the x86 cores, uh, amd cores, which are the, I mean that's the kind of compute that you know about, right? That is kind of architecturally present in terms of the compute that's elsewhere in the system. I mean you've got microcontrollers, so we talked about the baseboard management controller. Uh, we have turned that into a much smaller thing, a service processor. That's a microcontroller, a Cortex M, um, microcontroller, um, that is running that you don't see as a user. Um, but we've got our own system software. Appropriately enough called Hubris, um, because we actually thought we were not going to use our, we thought we were going to use an open source operating system, but then realized that Actually we need to go our own way. And the debugger for hubris is appropriately enough called humility. Um, but hubris runs on that microcontroller. It runs on our root of trust microcontroller. Um, but you also have, we've um, got FPGAs in the system. Um, FPGA is doing different kinds of things. And FPGA is a field programmable gate array and you'll hear it referred to as soft logic. And uh, FPGAs are really neat because they are uh, they are hardware that you can reconfigure dynamically. And the thing that's really cool about that is we can take our FPGA payload and put it into a hubris larger hubris payload and then you can sign that and attest to that. And so now you've got confidence about that whole bundle that you're loading onto the system. And then when we start the system we actually load this bitstream onto the FPGA. And in our different platforms the FPGAs have different roles but they're doing low level things like power sequences sequencing. They're doing, we generally are using FPGAs. Historically we're using FPGAs to do this kind of this, these low level things on the board. Because it makes for a simpler board, right? It makes for a simpler board when you can just put the soft logic down. Um, and some of these are crazy small FPGAs that are small but they're enough for what we need. We also have higher performance FPGAs that we are increasingly using uh, for things like NICS, uh, network interface card. So we are some of the kind of the next gen FPGAs are really pretty cool because they have what you call a hard block which is say kind of like one of the components, the hard components on, they're not programmable. That allows you to have like a surdes, that allows you to actually speak over a network which is actually the really hard part of a nic. And then you can do the NIC in softlogic, which is really neat. Um, so we're really excited about that. That's kind of in a forthcoming platform. We're working on a uh, an FPGA there. Uh, and then of course we've got an asic, uh, um, is an application specific integrated circuit and that's ah, not dynamically reconfigurable. Um, and ASIC is where. And we've got any number of asics on the system where you've got a deliberately fabbed part. And so our traditional NIC is an asic. Um, and we haven't done any ASIC development yet at Oxide, but I think any oxide engineer would say would emphasize the yet, uh, that at some point we almost will do an asic. And ASIC is still really, really hairy to do. Um, FPGAs are great because they look a lot like software. Asics are a lot more complicated and for a bunch of reasons that are that own kind of like deep well of complexity. Um, but, uh, not least, the software licensing for an ASIC is brutal because you actually need to get EDA tools from the likes of Synopsys or Cadence. And you're kind of mating yourself onto a FAB and a process. And it's like, there's a whole, like, it's very, very involved to do an asic. Very expensive to do an asic. It's higher performing, consumes less power, it's cheaper when you're actually, when you actually have the artifact. Uh, and there are reasons why we, you know, we can envision ourselves doing asics in the future, but for right now the ASICS we buy are all off the shelf and then we use FPGAs in a bunch of different places of the system. I would, uh, refer, folks, you know, we've got an Oxide Friends podcast. We actually have Raj Khaduri, um, joined us for, uh, one of our episodes talking about exactly this, about these different kinds of compute elements. And that's a great episode that I would steer people to. They wanted to learn more about the different kinds of compute.

Speaker A: The oxide machine itself. I don't think we've really explained physically what it is, you know, what are people going to be receiving? What are the components? What is it? How is it different? Yeah, what would I actually be buying and getting into my data center? So maybe you could talk a little bit about what that actually.

Speaker B: Yeah. So, uh, you get a crate, um, you get a very large crate. Um, and in that crate is a rack. And that rack is a, uh, it's an oxide designed rack. Um, and that rack has two switches and 32 compute sleds that are in it. Um, and that rack has been designed holistically. So we have designed all of these parts to work together. We've got some very basic things. We've got a DC bus bar in the rack. So we've got a power shelf that rectifies from AC to dc and then we run DC up and down the back of the rack and our compute Sleds blind mate into that DC power that is basic. And anybody at uh, all of the hyperscalers have a DC bus power based system. You can't buy a DC bus bar based system from Dell, HP Supermicro, because in order to have a bus bar based system you really need to be designing at at least the rack level because the rack needs to be that. You can't just buy an oxide sled. Um, you need to buy the rack that is gonna, is gonna contain it. And then we very much designed the rack around the switch. So the switch we actually have a uh, cable, the backplane. And in addition to blind mating into power, our compute slots blind made into networking. So the networking is a passive cabled backplane and it means there's no actual cabling in the sled itself. So if you want to take a sled out, you just like take a sled out. There's no cables. One of the challenges with the kind of traditional server design and the baseboard management controller has got to be on two different networks. Well ours is on two different networks too, but you don't see any of those because it's all sitting on that cable backplane. And our switch, which has that intel to Pino at its core, there's uh, actually another switch that serves as the switch for the service processors on the system and that's all uh, coherently interconnected. Um, so what you get is a real product and that thing wheels into the data center, you get power applied and you're on the tech port doing the basic configuration we need, which is basically like, we need to know like who to talk to for time more or less. We got to find, you know, there's some real basics that we need. We need a BGP session and we need to be able to actually like connect to a network. But once we can do that, we're off. And you're provisioning VMs, uh, you're provisioning storage, you're provisioning networks, virtual networks, just like you're on the public cloud. And one of the things that's been really, really fun because this is really painful with traditional infrastructure, it's this kind of bag of bolts approach. It can take a comically long time, uh, I'm talking like easily months to get the gear has arrived. Get to the point where devs are on there, like what takes months. It's like, well, glad you asked. It's like we got the wrong switch from this vendor. This, that server came with the wrong rack rails. We're missing the cabling over here. We got it all together and the software vendor, the software didn't work and the software vendors pointing at the. So it's like these things just like can take a long time to get up and running. With the oxide rack, I think wheels in, you power it on, you plug it in, you get a BGP session and you're provisioning VMs. And one of the experiences that's been really fun is to be with customers as they are doing that for the first time and be like, oh my God, I'm on it. Like, or provisioning infrastructure. I can just let my devs at it. We just got this thing powered on two hours ago and that is really, really neat. Um, and very vindicating of the approach that we've taken. It's really exciting.

Speaker A: And for a listener who's never had to do it the traditional way, what are all the different things that they would need to get into this rack

Speaker B: to have the equipment they would need? You got to go buy switch from a switch vendor. You got to go buy. You're buying your compute from someone else. Maybe you're buying storage for someone else. You're cabling it together. You're get, you've got, you got your service, your BMC is going to be on a network, so you got a separate switch for that you've got to have. And then that's all just the hardware that you've got to get software on it. Like, what are you doing for. Are you getting OpenStack? You're doing VMware? Are you doing something else? And it's like, then that's all got to work. You've got to get that imaged on there. You've got it. And it's like you're just kind of building this thing as you go. I mean, there's just nothing that is. You're the one kind of, uh, literally cobbling this thing together. Um, and it's just not a product experience.

Speaker A: Yeah. And I think the interesting part was something you talked about earlier too, is that if something isn't behaving the way you expect, for example, the IO latency or maybe some kind of compute issue, it seems like in most of those cases you really have no one to turn to. You can go to Dell or HP and they'll probably just shrug.

Speaker B: Um, well, yeah, there was kind of a tweet when we were starting the company that we were all, I think may have made it into a slide deck or two, uh, where someone said that they were, uh, dealing with a support issue with Dell and emc and they felt like they were talking to their divorced parents. It's not necessarily malicious because, like, a lot of the problem is that, like, you have these systems that are each well designed on their own, or they're well designed. Maybe that's putting.

Speaker A: That's.

Speaker B: That is giving them way too much credit. They're designed on their own. And then they have an issue when you put them together that each can plausibly blame on the other, where they're, you know, just like anything. We got like, you know, miscommunication with someone or what have you, where it's like, well, you know, you thought I was to blame and I thought you were to blame. And I don't know, it's kind of like it's in the middle, right? And this happens a lot where you have. It's like, actually these two things are kind of behaving somewhat reasonably on their own, but you put it together and the result is terrible. And the problem is it is the end user that is left being like, well, who helps me with my terrible problem? It's like, great, you each have made a convincing case that it's not your problem. Like, so I guess it's. Who's. I guess it's my problem, I guess. Um, and that is what is really, really frustrating to people.

Speaker A: And I'm curious, back when you were at sun and sun was selling their own servers was like, if somebody had a problem with one of those servers, was that a case where they could go to sun and then sun would actually go and figure out, okay, what's the problem here?

Speaker B: Uh, that was, uh, on its best days. Yeah, on its best days. Uh, its days were not always its best days. Certainly the vision that I always had for Sun Systems is that you'd be able to do that. It was not always the case. I started a storage group inside of sun called, uh, fishworks, um, in 2006. And we partly wanted to really make good on this idea of systems thinking. The, uh, in fish. The FISH in fishworks stands for Fully Integrated Software and Hardware. And we learned a whole set of painful lessons, um, because we were still relying on, quote, unquote, commodity hardware there and got bit by a bunch of firmware we didn't control. Um, so throughout my career I have tried to make good on this promise, but I really feel it's only at oxide that we've been able to control our fate sufficiently from top to bottom to really make good on it. Um, and it shows in the experiences that people are Having, which is really great.

Speaker A: And it sounds like maybe that's primarily because the software running on each of these components is in your control. So you're able to trace through and see where the actual problem is rather than having all these proprietary vendors.

Speaker B: Yes, it is that for sure. It is that we just like have more of the components under our own control. Don't want to understate the importance of that. Um, I think it's also just the sense of responsibility that we take that we like our view is like we're going to own this problem no matter what. And the, you know, we've had now a couple of concrete examples where we have customer issues that like at the end is like not a, you know, is it an oxide problem? It's like. Well no, it fell in a gap between oxide and a system we were talking to. And I think it's a point of pride that we take responsibility for taking that all the way to root cause. Uh, sorry to be plugging our own podcast here. Whenever the Internet jokes that we're not that we are actually a podcasting company that is merely creating computers for content creation and I'm not sure they're wrong. Uh, and so every, you know, whenever we have any kind of like crisis at oxide in terms of debugging a problem or what have you, we were like this is going to be great content. And we had one of these, uh, an episode that we called Hell is Other Networks where uh, we um, had a customer that, where we took this particular switch and uh, talking to ah, the oxide rack and each, the oxide rack and the router were making like reasonable decisions on their own. But you added it up and it was a nightmare. Um, you know we were able to get that debugged to root cause and even though you again like our behavior was reasonable and its behavior was reasonable. Um, but we want to be able to take responsibility for that. So I think it is twofold. It's the fact that we control these layers of the stack. But then it's also very much the sense that we have suffered at the other end of this and what we want to be able to do is really take on that responsibility and get a customer actually righted wherever the problem may lie.

Speaker A: I think that's a uh, good spot to end it on but anything else you wanted to mention?

Speaker B: No, it's great. I uh, know we hit on a bunch of things so hopefully thanks for the wide ranging conversation. This was great. Sorry to go on and probably went on a couple, one too many rants but, you know, as I'm want to

Speaker A: do, it's what the people want to hear.

Speaker B: You got to give the people what they want. You got to give the people the. You know, what I actually didn't do at all is I didn't go on any anti Oracle screed, which is now like, I am somewhat Internet famous for my unvarnished opinions on Oracle. And uh, uh, as the, uh, David Ellison and Paramount has been more in the news, I've had journalists calling me up because they see hacker news comments where I'm talking about Oracle. So it's like, this is what I'm famous for now. It's my unvarnished friends and Oracle. So I'm sorry to not give the people what they want in that regard. I'm sorry.

Speaker A: You'll always be tied to Larry.

Speaker B: I guess minute 33, I know is always going to. It's fine. Uh, I'll take it. Truth to power. I don't mind speaking it.

Speaker A: All right, so if people want to check out what's going on at Oxide, check out your podcast.

Speaker B: Yeah, Oxide and friends, um, you can join us live in the discord. We'd love to have people, uh, join us live. Um, but check out the podcast feed, um, and check out the rfds. Those are all out there as well. And then check out the repos. Um, we're a d. Everything we've talked about today is open source. So, um, people can dive in there and see what we're actually doing. We are really pride ourselves on that transparency. So we leave nothing to the imagination. I think that sometimes people are like, okay, maybe you guys could share just a little bit less, you know, so maybe transparency to a fault, but, um, there's a lot to go dive into if you want to dive into Oxide.

Speaker A: Yeah. I think for the people who are actually managing these data centers and working with traditional vendors, transparency is exactly what they want to hear.

Speaker B: It is. It is, yes. I think that that transparency is. We did have an early customer that had an issue and we wanted to get back to them, let them know, like, hey, we're working on your issue. Customer's like, no, no, I know you're working on it. I'm watching the GitHub issue, actually. It's great. It's like, oh, okay. So, you know, uh, it's, um. We love that. We love the fact that people can see what we're working on. We have always felt there's much more to gain than there is to lose.

Speaker A: Very cool. Well, Brian, thank you for chatting with me today. This is fun.

Speaker B: Absolutely. Thanks for having me. This is. This was a lot of fun. Uh, and sorry to take you into the filth that is the modern data center.