Securing Kubernetes for the Mythos Era

Speaker A: You are listening to Kubernetes Bytes, a podcast bringing you the latest from the world of cloud native data management. My name is Ryan Walner and I'm joined by Bhavan Shah, coming to you from Boston, Massachusetts. We'll be sharing our thoughts on recent cloud native news and talking to industry experts about their experiences and challenges managing the wealth of data in today's cloud native ecosystem.

Speaker B: Good morning, good afternoon and good evening wherever you are. We are coming to you from Boston Massachusetts. Uh, today is 7-2-2026. I hope everyone is doing well and staying safe. Uh, today we have another great episode lined up for you guys. Um, an interview with Alex Zehnler, Founder and Chief Technology Officer at Adira. Uh, it will be a great conversation around Kubernetes security, container security and how we need to think about security security, especially when using GPUs inside our Kubernetes clusters, both for training and for inference use cases. Should be a great conversation. So without further delay, let's get Alex on the pod. Hey Alex, welcome to the Kubernetes Bytes podcast. Um, why don't you take a minute and introduce to our listeners who you are and how did you get here?

Speaker A: Yeah, thank you for having me on. I'm super excited to chat. Um, so yeah, I'm Alex Zindla, I'm the CTO of Adera. Uh, Adara, uh, is a security company. We build uh, things especially in the container security space but um, more generally infrastructure security. Um, and uh, I am, what am I 27 years old now as of this month of recording. Um, and uh, I have something like 13, 14 years in tech. Um so I started very young in tech. My uh, first full time job. It wasn't technically legally a full time job but I was working full time hours, uh, was uh 14. So I started basically on my 14th birthday uh, working in tech and uh, first in the Internet of things, uh, and then eventually uh, I kind of retired after 10 years and IOT and started uh, Adera, um, in order uh, to make computing more secure. So uh, I got my start in tech uh from Minecraft actually. Um, that was really where I got programming experience I would say before that it was like I was a Linux sysadmin when I was like 10 trying to uh, host Linux servers on the beat up laptops that I had and um, and that was kind of where my interest I guess in computing and like oh the internals of stuff matter. Um, but um, really programming started when Minecraft came out and I started making mods and um, that led me into the Java ecosystem which actually then led me into the Dart ecosystem which then led me into Iot, which uh, was a mix of Java and Dart. Uh, and then, yeah, now I'm here.

Speaker B: That's awesome. Like that, that's, I think so before each guest, right. I do use the AI tools and ask them like, hey, can you help me do some background on the guests? And like one of the things even, even Claude was like, yep, you should definitely ask Alex about her, about the early stages because I think you are at IO like super young. And then obviously the IoT experience, it's super cool. Like damn. Yeah, tech at 14, it's awesome.

Speaker A: Yeah, I, I'll tell you, like I grew up in a lower, uh, middle class environment. I did not have tech. Like tech was not like, uh, oh, I'm just got all these computers. No, like I, I had a hand me down laptop that my dad got for free. Um, my dad was, was worked it at the university. So my inroad to tech, like messing with tech was like at the university itself. And so that was kind of where I got to mess around with things. Um, and basically like I've told this story before on the podcast, which is probably how Claude knows about it, because it's one of those things where I, when I said it for the first time publicly I was like, should I be sharing this? Because I'm not sure if I signed something when I was a kid or not. But uh, basically I got in trouble with Google when I was 12 because I had a Google account before kids were. Before you were supposed to have a Google account. And actually I think it's even more restrictive now, I think in some states, but um, especially in the UK now. Um, but you know, back in, back in this time, I think uh, it was 13 as the earliest you could have Google account. Well, I had been working on Chrome OS stuff at the time and uh, basically what happened is I was on the forums talking about stuff and I mentioned my age and the forum moderators or community managers at Google as I now know, um, notice this and they go hey, like this isn't right. They talked to their lawyers and basically I got a phone call with a Google lawyer and the community manager and they were like, legally we have to take away your Google account. However, because you're so interesting with this Chrome OS stuff, which I was just doing like shell scripting and stuff. Like nothing actually interesting. Um, but you know, for a kid I guess it was interesting.

Speaker B: Yeah.

Speaker A: So uh, uh, they invited me to Google I O. So I got to go to Google I O 2012 for free, all expenses paid. And that was where I got my first wholly owned computer. It was a Chromebox. And I ran a sea it root on the Chromebox, uh, to play Minecraft. And I think I installed Ubuntu at the time. So um, yeah, anyway, that's uh, I'm sure that's how Claude knows about that.

Speaker B: Oh yeah, it's in the model training now. Yeah, I didn't even have that search on. So some. The podcast might have been a while back because the lead about it.

Speaker A: Yes, absolutely.

Speaker B: So let's talk about Adira. Right. So, uh, can you, like you said, it is in the container security ecosystem. Uh, we have had folks, everybody from let's say Armo Security to Viz before they were acquired by Google to Chain Guard very recently and even they referred to Adira during the episode. So uh, what does Adira do that other image scanning tools don't? Can you talk about that a bit?

Speaker A: Yeah. So I think it's really interesting. Adera is different in so many ways. And uh, I always think it's funny that companies often model the type of person who runs them. Um, and uh, we're very different as a company and we are also people, uh, wise. And we're also very different product wise. Um, so uh, I guess an important point is that Adara was uh, my co founders. Ariadne and Emily actually came from Chainguard prior to Hedera. And um, they had built Ariadne in particular built Wolfie, which is the base of Chainguard images. And Wolfie is a open source GNU Linux distribution, um, a la Alpine. Uh, she also worked on Alpine. And what we realized is like okay, we've patched the vulnerabilities, we've made the pipelines to do that, we built the tech. And I say we as in I wasn't part of Chain Guard, so I should clarify that. But um, uh, Ariadne thinking from that perspective was like, what's next? And I had been working in IoT at the time and IoT the S& IoT stands for security, as everyone says, there is no S in iot. Um, so you know, I was really trying to solve like how can we make edge devices more secure. So I started building what eventually became Adara's foundational tech. And then I started talking to Ariadne and we realized like, all this time is being spent on patching vulnerabilities and there's recent stuff now about like, you know, you can now get patches from vulnerability before it's even disclosed and all these things. And I think, uh, that's great. Like I'm, uh, not. There's nothing wrong with that, but I fundamentally believe that the way that we actually do computing is fundamentally insecure. The fact that a patch is required to make sure you don't get completely owned is a side effect of how we build computing and infrastructure today. The primary reason for that is in the switch from VMs to containers. We forgot that boundaries matter, that security boundaries matter. And I wanted to do something where I wanted a technology that actually gave security boundaries to containers and other things, uh, applications in general, that actually protected the system and your application from each other and prevented lateral movement without the sacrifice of the various different problems that can occur when you provide isolation. Um, you know, if you've heard of a project like KATA or gvisor, conceptually what I wanted to do was similar, but there was a major problem with those projects. A, they weren't really production focused, as in, they weren't really trying to make it like usable for a downstream user. It was a tool in the toolbox and you sort of got to build it yourself. Um, and then in Gvisor's case especially, but also in kata, performance was kind of an afterthought. Like it was like, hey, we need it to not be completely awful. But 30% performance impact was the norm

Speaker B: and it's so silly.

Speaker A: Yeah, and no one's going to deploy it. When I see GVISOR deployed in particular, it's usually as a last resort of we have no idea what to do. Um, I also think gvisor's design, by the way, has some fundamental design flaws. Implementing the Linux kernel and go is not a particularly great idea because of garbage collection and other things. Um, then there's also the problem of it's running on top of the Linux kernel. Anyway, so, so there was all these things and with KATA you need a nested virtualization and a bunch of instances didn't have that. So I tried to build a technology that solved uh, these problems. And so ultimately what Adara is, is we're on the Kubernetes podcast. So the most important thing is Kubernetes itself and like how do you integrate that cleanly? So we did a bunch of work on building out a from the bottom up, from the hardware up. How could we run a Kubernetes Kubelet effectively a Kubernetes node that is properly isolated. And so if you use Adara today, what you get is effectively Every single Kubernetes POD is completely isolated into its own system security boundary which gives a high degree of uh, security and um, protects you against vulnerabilities that uh, provide lateral movement or privilege escalation. Um, then we don't just do VMs. It's not just about oh, now we have the VM UM boundary and that's it. Um, we build on top of that by building out functionality that a secures uh, the system as it is. So implementing network in a secure way and providing all of these additional functionalities and things that you can consume in your platform to be more secure. And I think that's what makes us so different is we're trying to solve it from the bottom up without making you rethink your application. Because the same image that runs a normal Kubernetes will run on the DARA Kubernetes.

Speaker B: So then a follow up question, right If I've been using Kubernetes in production for the last five years, let's say, um, if I wanted to get started with the data, does that mean I have to redeploy a brand new Kubernetes cluster widget installed and migrate my application like a blue green deployment and then move it over? Or can I do like a brownfield onboarding of my existing clusters and make them more secure?

Speaker A: So this is the cool thing we don't, we're not a Kubernetes distribution. Instead we distribute the node images or uh, allow you to install on top of an existing node. What that means is that you can effectively upgrade your existing cluster into an Adara enabled system and get that isolation and security by default without ever having to change how you actually do things. This is super important to the way that we think about it because other solutions that try to do any sort of security or of this form or more often than not it's like dev environment type stuff. When you try to do that stuff they either a just install on the normal system, some binaries and then try to run um, uh, additional namespaces which doesn't really solve the problem because you still have the Linux kernel fundamentally that is monolithic and is a security nightmare in a lot of ways. Um, then you have the other systems which try to basically get you to use their cloud environment and then you plug it in as an additional node. And so this creates like kind of the worst of all things, um, kind of set up. And so we fundamentally believe that if you're going to do this the right way, you need to make it work with the way that people actually think about production today. And this has proven true in the market because people who would normally not consider this to be an important thing for them because, oh, it's too hard, we've made it easy enough to adopt that it actually becomes feasible for them to make their security better and sorry,

Speaker B: go ahead, finish it though.

Speaker A: No, and I think that like, in today's environment, making it easy is super important. Like, and the other thing that's like kind of funny and uh, you know, I'm sure we'll talk about this, uh, later, but like with AI and Mythos and all these things, people are. I don't know what's going on, but we've gotten into a little bit of psychosis where we think that like an AI agent is not just an application, but it is. And the gap for me that I think is really funny is people think, oh, well, AI agents have this need for a completely different permissions model and a completely different way of running them. But actually fundamentally the problem was that our other applications needed this too. We just didn't care.

Speaker B: Yeah, that amount of risk. Yeah, it's okay, I'll put some perimeter security. And

Speaker A: we had a developer to blame. Right. And now we don't. And that's the difference. And so for me, it is very important that we think of these things as on the same playing field.

Speaker B: Mhm.

Speaker A: And because I think if AI agents are going to be real and last a long time, they have to be no different than anything else fundamentally from how we run them. I don't think that's making the AI agent as insecure as our normal applications. I think that's making everything more secure. That's my fundamental difference in thinking in today's day and age.

Speaker B: Interesting. People can onboard their existing clusters by installing this or onboarding this. Um, are there amis available if you say you have OSS that customers can use and update their machine configs and then just be up and running?

Speaker A: Yes, we provide machine images so that people can consume Adera and effectively it's just an AMI swap and you're good to go. We also for people who build their own amis, because there are a lot of people out there that still do that. Um, you can just hook in our installer as a phase in your packer build. Actually we build our AMIs using AWS's Packer builds and then we build on top of them. Um, and actually coming, uh, soon is support for Bottle Rocket and things like that. So you can, um, utilize those technologies as well, and I think making that easy and flexible is super critical because if I didn't do that, I am making the infrastructure or people's lives harder. To me, the thing is that you need to have this be an actual technology that people can consume. Like, the problem with something like KATA is that it really is build it yourself. I can't tell you the number of companies I've talked to who have teams of people that do kata and it's. And they don't want to be. I know. Uh, but that's, that's what they have to do because the tools that they provide are, aren't good enough. And I really feel like we have to think about the way that we build tech in a way that is actually easy and flexible at the same time.

Speaker B: Gotcha. And that's a great, like, I don't know, a thesis to have and obviously help customers adopt it easily and simply without a lot of friction. Um, but if we, if I ask you to go into the next level of detail. Right. So, uh, I did look at your documentation. There are concepts called zones and driver zones and then workloads. Can you help us map this to things that Kubernetes users might already be familiar with so that it's easy for them to just make those connections? This is an audio format. Majority of our audience listen it on as the audio format. So it would be helpful.

Speaker A: Uh, yeah, absolutely. I think there's a few interesting things that we did that are very different. One is that Adara as a platform isn't tied to Kubernetes itself. Um, we built our compute platform so that it is basically an API that then Kubernetes uh, can build on top of. Um, the reason why we have our own concepts is because we're applying this to places that other people don't even think of. For example, we have Kubert support coming where you can actually run, um, Kubert. You can use qvrt, but actually the entire back end is Adara. Applying the same technology to multiple different areas is really important. Um, uh, the key, uh, technologies that we built, um, are zones and workloads. A zone is what you would usually call a virtual machine. The reason why we call it a zone is that it is, uh, it's got additional things on top of it that integrate with our platform. For example, memory management and CPU usage management are more than just a typical vm. We have full external control over how much memory something is using and we can preempt and move memory around, um, then networking, uh, as well. Similar story there. Uh, then we also provide a concept called workloads that run inside of zones. And these are just containers effectively. But there's also things where you could for example, mount the virtual machine image as a workload and run a vm, but inside of another vm. So um, we provide these concepts uh, to make things uh, flexible because some people also just want to use our API. But when it comes to Kubernetes, what does this map to? Well, you use the same Kubernetes pods and containers that you normally do. But what we do is we map a Kubernetes pod to a zone and we map a Kubernetes container to a workload. We dynamically reconfigure your resources into a Darius, uh, system and we do all the work to map everything. Uh, this is really powerful because if you have those downstream concepts, let's say that you build a workload that actually knows it's running on the Dara. You can consume additional features by running inside of Dara. For example, we have private zone to zone communication which is quite literally like the hypervisor can mediate data transfers between zones without the host ever seeing. And those are some very powerful features, especially when we talk about privacy and security, um, that people can consume on the platform.

Speaker B: And like these nuggets that you're sharing, right? Like it's just, I don't know, blowing my mind in different because before if I, if I take your hypervisor based communication and controlling that communication, right, we had to go into the EBPF level and then control packet transfers through EBPF rules. But now you're saying we can do it at the hypervisor layer before even the host finds out about it. So that's just. Wow.

Speaker A: Yes. Yeah. So if two things shared the same machine and secretly like also my goal is to build machine to machine communication because that's basically what IoT is. And so in my mind just immediately goes to that. But uh, the uh, and kind of, by the way, some of our tech is heavily inspired by mainframes, which I think is a really funny fact for a lot of people. Um, particularly message passing and various things. But yeah, like EBPF is effectively the only like in kernel thing that you have. And what we've thought about is if you build the hypervisor tech properly, you can do these really interesting things that especially in the age of AI, are super powerful. Like why does an AI agent need to pass data over the network? Why can't it just create shared memory between two zones and communicate data that way. The truth of the matter of why can't it do that is it can, but no one does because it's hard. And so we did the hard work. Um, we have a mechanism internally called Inter Domain Messaging. It's basically an RPC system that allows you to communicate between, um, different zones. And this is how our internal tech, when we map the Kubernetes, actually works. Um, anyway, there's a lot of nerd stuff at that point, but you mentioned one more thing.

Speaker B: You said you're also adding keyboard support. Because, again, that was another thing that just caught my eye because, yeah, there are a lot of customers, even as part of my day job that I talked to, that are moving from VMware, where VM, um, had its own set of resources and complete isolation, to something like Kubevord and OpenShift virtualization. But now I'm making that connection that those are just pods. And if you don't treat those, uh, quality aspects cleanly, you might come in with the expectation that I'm running a vm, but at the end of the day, you're not. You're running it as a pod, so.

Speaker A: Oh, wow.

Speaker B: Okay. Yep.

Speaker A: Yeah. So, so this is the interesting thing about qvr. And, okay, so lots of things going through my head about qvr, but the biggest important point is that Kubert is just Libvert.

Speaker B: Yeah.

Speaker A: And if you've ever used Libvert before, it's great. Like, I. I love Vert Manager. If you love VMs, you probably have used it. But the problem is that Libvert is a sea demon that runs qmu. Now, QEMU is great, and I love people, I know a lot of qmu developers, but QEMU is very heavyweight. It is really. It is a workhorse, and that's part of the reason why it's heavyweight. But it also can do everything under the sun. And when something can do everything under the sun, it usually means it's insecure in some way. There's a lot of foot guns and things in qemu, and those things lead themselves to libvirt, and they lead themselves to kubevirt. Kubert recently introduced a new hypervisor interface, and so we're hooking into that to actually deploy, um, our system, um, into Kubert. And, um, and I think that's super powerful because, um, one thing I did not mention yet is that everything in Hedera is rust. Like, literally everything. Uh, and that's not just, like, rewrite it in rust or whatever. This Is like we did this because this was the best option. Um, and we care about memory safety. And to be honest, I think not enough in the lower levels of the infrastructure, um, today are written in rust. Um, that's an important thing. We built this platform to be flexible for a reason. Um, the fact that we can do Kubert on our platform is a testament to how we built this thing. Um, that really just comes from. I wanted to build fundamental technology building blocks effectively and then build on top of that rather than a very thin opinionated layer.

Speaker B: Yeah. And okay, so we spoke about containers, we spoke about virtual machines with cube word. Obviously we have to talk about GPUs. Like GPUs are an expensive resource. It's difficult to get the latest and greatest unless you commit to like a three year usage. Customers that actually get their hands on GPUs to get the most utilization out of them. Right. We see even the Nvidia GPU operator have things like multi instance GPU and mig and they allow different pods to share the same underlying physical resource. So how do we think about obviously, uh, uh, uh, technical details on how GPUs have evolved from their early VDI days to gaming days, to crypto and to AI and now how they're being used, but also how do we secure them? How do we secure the applications sharing the resource?

Speaker A: Yes, this is one of my favorite topics because as I mentioned with Minecraft, was a gamer and so my frame of reference for GPUs before they got taken over for all the things that they do now, uh, was rendering. And I love rendering and I love gaming tech and game engines and stuff like that are. If you ever look at my GitHub, you'll see that I have a Minecraft clone called Voxolotl. Um, and the uh, goal was basically I want to make a metal renderer for Minecraft like systems. Uh, I care a lot about rendering. The truth is that GPUs, the same ones they deploy, the data centers today are still rendering machines. They fundamentally are designed to do that. Like if they don't care about tenancy, they don't care about multiple pods, they are just there to run a single instruction, um, tens of thousands of times really quickly. Um, one of my favorite like things trying to explain GPU is to people who don't know the difference between a CPU and the GPU is that there is no proper branching on the gpu. At least as of the latest um, information I have gathered. I'm sure that people are trying to do all sorts of crazy stuff these days, but there really is no branching on GPUs. If you uh, say you write an if statement and that if statement doesn't execute, um, what happens on the GPU is it just runs a bunch of no op, uh, things, uh, instructions instead of uh, actually executing the code. These things are designed for parallel computation. They don't really care about processes. They don't have a kernel. I'll never. Fun fact for you. I have to work a lot in the business space. Obviously being a founder, uh, it's always fascinating to me when I talk about the kernel and how many people think I mean GPU kernels. When I'm like no, no, no, no, the Linux kernel. Yeah, uh, it's a little personally annoying. If you're a GPU user out there, I would love if you would put the GPU in front of kernel when you talk to business people. That would help me a lot. Uh, but importantly, the kernels that run on there are just basically just code. And a GPU is just basically receiving RPCS and executing code. It doesn't care about processes. When you go and apply this to a container space, you have this natural problem of you either have to assign one container to the entire gpu, you have to do mig, which gets you a slice of the GPU, which is basically completely fake. All it is is basically you can consume up to this amount of memory, but it doesn't necessarily say this amount of memory is only for you. Same thing for time slicing where it's slicing up the actual execution time. All of this stuff is a software abstraction on hardware that doesn't have any idea what's going on. X86 and ARM hardware, by the way, on CPUs do actually understand the concept of processes. They have page tables and they have all these different functionalities that actually understand and build on top of it. GPUs have none of that. The problem that you run into is that all this data is just shared. If you share a single GPU between multiple containers. There are so many attacks that are possible. I'm not even talking about data leaks. I mean denial of service attacks. A great example is that it's very common that GPUs just crash. Like you can write a GPU kernel that literally just reboots the GPU every time, every time. Because fault isolation is not a thing on these things. And so the problem becomes. And actually if you remember the early days of like WebGL, like if you were around for the early browser, 2011, 2012, space uh, you may remember that it was not uncommon for GPU processes on a browser to crash because that was effectively the first multi tenant use of a GPU was bringing um, GPUs to the web. Because before that it was literally I'm just running my operating system or my game on the gpu. But now it's like, oh, I have multiple contacts and uh, now I have untrusted code running on it. Well, we're here right now, we're hitting this point in the data center space, except it's much worse because the web browser space had the advantage of controlling the API, but we don't control the API today. If you load a CUDA kernel, you have basically full execution on the gpu. Fundamentally this is just a completely flawed way of doing computing if you care about security. And so what does Adair I think about this and how are we trying to solve it? Well we're solving it on two fronts. One is that basically nobody in production actually shares GPUs. Like if you go talk to um, any of the major GPU clouds, nobody is doing like uh, I'm going to partition a GPU out except for at their lowest band where they don't really care about the security, if we're being honest. Um, but the big customers, they're getting whole GPUs now part of that is that one GPU doesn't really produce all that many tokens. Um, and so people want the whole gpu. Realistically. Um, there's also the fact that models are getting bigger. I mean have you seen the terabyte model that just got released? Uh, I think it's glm. Okay, I want to say is what it's called. Um, I think it's actually like 600 gigabytes or something like that, but it's huge. Um, and how do you deal with that? Right, so these GPUs are being consumed in whole. So the problem is that when GPUs become a cost constraint and people either have a lot of them to share, which is a bubble possibility, or they become so scarce, which they arguably are right now for a certain class of people, um, that you need to actually partition them. You have to have a backup plan for that. What we are doing is focusing on pass through. Can we pass through a GPU into a, ah, isolated environment so that at least you don't have all eight GPUs on your GPU server, all on one Linux kernel? Because by the way, one major problem is that you can like seg fault effectively or really kernel panic the entire Linux system through a bad gpu, uh, kernel because uh, there's so many different possible ways to break the Linux system kernel, um, when it comes to the Nvidia driver. Now some of this is Nvidia driver related, some of this is just, this is how Linux is. Um, but you can't really trust untrusted devices, right? Like if, if you are connecting a container to a device, that device is now untrusted. Linux has no concept of an untrusted device technically. So you have to have a zone in order to isolate that. That's one front that we've been focused on is just a simple use case of how can you create a VM with the GPU without it ever touching the host or having the ability to break the host. I call that fault isolation. Then the second category uh, of how you would run a GPU is this idea of separating the GPU driver and all the device related stuff from the actual running of the application which needs the gpu. Uh, you mentioned driver zones earlier and driver zones are what that is. It allows you to run a zone that runs the Nvidia driver and is connected to the Nvidia gpu, but provides the CUDA API outwards into another zone that is actually the one controlling it. And so you have a fault isolation in the sense that uh, the GPU driver can't uh, be exploited. The vast majority of all of the code that is risky is actually in the GPU driver zone. Um, and the application is separate from that. And it also means that you can mitigate things like overriding the firmware of the gpu which by the way is a really terrifying thing that you can do. You can just downgrade the firmware of the Nvidia M GPU if you have full access.

Speaker B: Okay, but Alex, I do need your help, right? Because um, I'm not a Linux kernel expert so I'm trying to just ask very basic questions. We spoke about MIG and how that's not true isolation because the GPU doesn't know that these are different tenants. Putting your applications in zones is Adira taking on the responsibility to carve out the gpu and do I need, do I still need mig or do I just use Adira? And it knows, hey, this is section A of the GPU and this zone and the workload in it is only accessing this zone. Uh, this section.

Speaker A: So the first uh, example that I provided is about full pass through. Yeah, so that's where one application provides the entire, or gets access to full gpu. This is the vast majority of where the industry is when it comes to GPU security. And that's purely just because no one has built anything better. Um, uh, the second thing that we are building that is better is about driver zones, which is this idea of if you separate the driver aspect, then you can actually portion out the GPU to uh, multiple different zones and communicate through a virtual layer. By doing that, Adera is effectively taking on the responsibility of managing the differences between everything. Gotcha.

Speaker B: Okay, that makes more sense now. Thank you for clarifying that.

Speaker A: Yeah, yeah, of course, no problem.

Speaker B: And then obviously recently we also see the Kubernetes community make progress on things like dra. Can you give us, uh, um, I don't know, an overview of what DRA is? How does that help users as well improve their posture?

Speaker A: Yeah, so DRA stands for dynamic, uh, resource allocation. It's a feature in Kubernetes, which is really cool. M. The primary driver of it is that Kubernetes doesn't really understand devices. Um, before this the state of the art was something called device plugin, which is bad, let's just put it that way. But the simple idea is that you write an annotation for a device that you want to claim and then some magical stuff in the background, uh, eventually tells the runtime that hey, this device node, like literally dev Nvidia zero needs to be passed through into the container.

Speaker B: Mhm.

Speaker A: The problem is that this basically doesn't work for any advanced runtimes like Adara. Um, we have to do a lot of translation and a lot of work to make that work. Um, because it's very single kernel focused. Um, also it doesn't really understand the GPU or the device. Like the GPUs can't represent themselves in a way that makes sense to Kubernetes using this model. Um, and actually even worse is things like TPUs where you have these very complex topologies that need to be considered. DRA is about providing a new resourcing system to Kubernetes for telling Kubernetes how to schedule complex device topologies. Um, so what on earth does that mean? Well, if you have a GPU, right, um, that GPU is physically connected, usually over PCIe. Um, and if it's connected over PCIe, that usually is tied to the MMU or the memory, uh, management unit of the machine. And I'm going deep because that's how you have to be with the hardware stuff. But the uh, memory management.

Speaker B: I'll ask you for an explanation Again, but go ahead.

Speaker A: Yes, yes. Well that's where I'm about to go, so I got you. So uh, the memory management Unit though and CPUs are not really all in the same place. So for example you have these dual socket CPUs which basically means you have two CPU cores that have each their own kind of memory that's close to them for them to access memory that is far away from them, that takes more time. But some memory is closer to them and that takes less time. In the same way some GPUs and the actual PCIe device are closer to some CPUs and some are further away. This is this complex topic called non Uniform Memory Access Access. Now NUMA is a huge mess. And shout uh, out to Stephen on our team who just wrote a bunch of blog, uh, stuff about numa, uh, and how we think about it and how we uh, work on it. Because it's stuff that like a lot of people just kind of goes in one ear, out the other but actually has a massive impact. Um, just an interesting stat for you. If your application is using memory um, that is from the opposite NUMA node. So for example if you're on CPU one but you're talking to a memory thing that's on memory two that uh, can result in a 30% reduction in performance. This can be very severe and people are like I don't know what's going on. Well, in the same way GPUs have the same problem, right? GPUs can be connected closer to one CPU or the other. Well DRA has to deal with that and so does Kubernetes, because it needs to know where numa, ah nodes are. Basically numa, um, distance as it's called, um, between two things. NUMA is effectively represented as a number. So it's very simple. NUMA node zero and NUMA node one are one distance apart. If you have NUMA node five that means it takes approximately five times longer to get from node zero to node five. Now this is a simplification but that's basically how it works. This stuff uh, really matters and Kubernetes has no idea about any of it. DRA is an attempt to get the stuff better and actually treat uh, have systems that actually understand these resources more effectively. Uh, in TPUs it's even more interesting because they have these things like you can get um, basically uh, uh, kind of like a matrix of section of the tpu. And how on earth do you represent that in Kubernetes? And that's uh, effectively what DRA does. And so um, using DRA with a GPU system either like Hedera or not enables uh, you to get better locality and better understanding of the hardware so that things can be scheduled more efficiently.

Speaker B: Okay. And then the same logic of uh, the previous two points that you mentioned on how can help with GPUs uh, applied to a DRA enabled system as well. This just makes sure that you don't take like a second. Not second, just a penalty hit if you don't provision your workloads correctly. Because again, at the end of the day Kubernetes users don't get to make that choice. The API server and the Kubernetes control plane does so. Okay, got it.

Speaker A: Yeah. I think an important point there is that Kubernetes wants you to forget about all this stuff, but you really can't forget. Right? And that's the fundamental issue of Kubernetes is like when it was all just CPU and memory, that stuff is easy to ignore. But the rest uh, of the system like Even coming from IoT we have serial devices because things still talk. 1980s protocols. How do you do that with Kubernetes, Good luck. There really is no solution except to literally just pin something to a specific node and then you just happen to know that this, this exact dev node is what you need to pass through. And so Kubernetes is dra, uh, is about Kubernetes growing up and understanding the world is actually a little bit too complex.

Speaker B: Gotcha. Okay. No, thank you for that I don't know lesson. And I'll say if somebody would have explained me NUMA the way you did, I would have gotten it like 10 years back.

Speaker A: Thank you.

Speaker B: Where they did dedicated sessions around memory management and NUMA when it comes to specifically focused on VMs, I, I tried to attend one of those and it just like went over my head. I was like, I don't know how this helps me coming from a storage background. So I just walked out and no, now it makes sense.

Speaker A: I'll, I'll just one point on that. Like I think uh, we. So I consider myself a systems programmer in the sense that what I enjoy about programming programming is like the operating system layer, it's the hardware layer, it's the all of that. And we are awful at communicating concepts and uh, I have tried my best to be good at it and I hope that I have made a dent. But like NUMA is not complex on this. But it is like the implementation is complex but the way that we should Explain it as system engineers should not be. And like, I think we're gonna do more content on this because, um, it's, it's a critical part of the system,

Speaker B: you know, for sure. Okay, now let's talk about Mythos, right? And Fable and all of the new models and maybe whatever comes after those. Uh, yeah, like we have all seen the timeline around Anthropic and how they, uh, made it like announced Mythos but didn't release it, worked with partners, announced Fable, pulled it back. How do you, uh, as a founder of a security company, keep up with this? And if you have like the customers that you work with, when they ask you questions, how do you answer those? What are the things that people should be looking out for, especially with these models getting more smarter on the cybersecurity front?

Speaker A: So, uh, I think this drives home everything that I care about and security. And you know, the thing, I keep repeating this to people, but I think it's really an important point. Mythos and the like, fear mongering that people felt, um, of course some of it is hype. Like anyone who doubts that it's like there's no hype in it is, is not being truthful. But the fact is that I don't care. I don't care if it's hype or not. Because what will happen is psychologically people will care more. And the fact is that not enough people actually care about vulnerabilities.

Speaker B: Mhm.

Speaker A: And so there's like, um, I will say I am very deep in Linux kernel stuff and every day I find something in the Linux kernel where I'm like, I cannot believe then this is how this works. And there have been multiple times we've. I think I have at this point like three or four kernel CVEs, um, that are like properly mapped to me. Um, I think there's probably more that haven't been. Um, but like, what's interesting to me is like, we've forgotten how bad everything is. And so my argument for why people should care about Mythos and evil is very simple. That even if it's hype, people are not going to treat it like it is. And so that results in two things. One, vulnerability researchers who are humans are going to find more vulnerabilities and basically you just up the competition. And let me tell you, I know some vulnerability researchers, they need their $200,000 payment so they can go party next year, right? So they're going to go find more vulnerabilities, AI or not. And the other thing that's really critical is that if Mythos does find bugs, which I think nobody can sit here and tell me it hasn't found anything. Um, all of those bugs are very severe. And my analysis based on seeing um, Ariadne actually had access on our team to Mythos, uh, because of um, some very cool things of the Linux foundation. And some of the things that found are like, oh, okay, like this is a thing that no human would have really thought of and the attack chains that it can come up with are complex. And uh, I think I'm quoted and wired somewhere of something like uh, I said like if you have Mythos is basically 1000 vulnerability researchers, uh, working 24 hours a day. But um, on one single system you multiply that by the number of projects involved in an average Linux distribution and it's just insane, right? So in my opinion it's a very, very big deal. And it's not just because it makes the security stuff more important. It's because I really care about not getting hacked. I don't want my bank to get hacked. And the fact is that if they don't take this seriously, they will. So, uh, and who knows when this is released what will happen with Claude? Because chaos is happening. But what I will say that's really interesting is I, I, first off I'm one of the signature signers I guess of Free, uh, fable or whatever it is. Um, um, and, or I think it's either Free Fable or free Mythos. But um, I firmly believe that the government action that was taken was the wrong move. Um, and the fact is that pausing access to one model is not going to help when someone else goes and develops the same thing. Because the truth of the matter is that uh, there really isn't a really big moat around these models. So you know, there's, I mean I will say that Anthropic is by far the most interesting AI company out there right now in my opinion. But you know, it's just a matter of time before, you know, I think uh, actually two days before this there was like a distillation, um, uh, thing that happened where Alibaba, uh, distilled a uh, part of Claude. And so there's all these things that are happening and the, the risk is high. So I consider the risk real. I think that what's happening uh, regulation wise with it is insane because I um, don't think you solve anything by uh, by restricting access to this stuff.

Speaker B: But how do architects plan for it? Right? Like if I am an infrastructure admin Kubernetes or not, right? Like how do I plan for a world where all of the different layers in my stack will get exposed? How do I, like, I don't know, think about it.

Speaker A: Well, you can never move at the pace that it can find vulnerabilities. And um, that is to say like, sure, you can get, you know, CVE low CV images and you should do that by the way. Like I don't want you running a vulnerability in production just because, you know, some distribution hasn't updated it. But you need to fundamentally build towards a world where everything can go wrong. Because it's always been able to go wrong, but the heightened ability for it to go wrong now is real. And so that means architecting your systems as securely as you possibly can. Um, secure by design is more important than ever. And when I think, by the way, that secure by design has had a huge marketing problem because it wants you to do this shift left thing and all of that, which I think was probably a bit of a failure. And the thing is that putting the onus on developers or whatever doesn't help what we should be doing. And I get in trouble a lot for this because I realize sometimes that I'm talking to the developers, but I'm um, just as often talking about the systems engineers who I wish would do things differently. Um, when I say these things I don't mean you as an application developer are the problem. Um, but I think we as ah, software engineers have to think about how we build software differently and build in a more secure way. And that means infrastructure. Doing things like what we do at Adhera, where we think very low level and we reduce layers of the stack, adding um, complexity and abstraction helps no one when it comes to security. We have gotten to the point now where the average application developer has next to no idea about the actual low level parts of their system. To some degree I'm glad because that opens up the ability for more people to understand and build applications. But it also means that when you have seven layers of stack in between things, no one cares about the security. Uh, I gave a really interesting talk once about how layers of the stack lead up to good or bad security outcomes. If the Linux kernel is bad security wise, it makes your application bad security wise. Um, so to me we have to start caring about all the things that can go wrong and find and build infrastructure to make it feasible to actually mitigate those things. Yeah, because

Speaker B: not just Murphy's Law, this is like. Yes, I don't know the times too.

Speaker A: Yes, exactly. Like we have hit a point where you can't really wait on this stuff. And by the way, like, I'll tell you one thing that drives me nuts about this is when people say proactive security. Every proactive security tool I have ever seen has always been actually reactive. And, and I believe that Adara, uh, and tools and that space are the real proactive tools. And although I might be biased, I strongly consider, I want people to actually consider it because like, if you build your system appropriately, it is proactive because the vulnerabilities uh, have a reduced impact. But if you are just constantly reacting to every new vulnerability by patching it it, you will never solve a single problem in your life.

Speaker B: No, that makes perfect sense. Uh, I don't know that that's like my next question was going to be what's next for Nadira? Or where do you see this, this growing? But I think that's a great place to like. I don't know for me to bring up the last question unless you wanted to add something. I, uh, know we are almost at time.

Speaker A: No, I'll just give a very brief answer that uh, what's the next for Adera is we're going to continue building real security technologies and infrastructure technologies. And um, I will never uh, give up on making sure that things are easy to use and consume because that is what I think is most critical for our product.

Speaker B: Awesome. No, thank you so much. Alex. Where can people learn more about Adira and any links that you want to share the Minecraft fork that you've created?

Speaker A: Yeah, so if you go to my GitHub, GitHub.com azenla you can find some of that stuff. Uh, you can also go to GitHub.com Adera-Dev um, there's also. I'm on Bluesky as Alex Zenla IO that's probably the best place for you to keep up with my day to day. Uh, if you're interested. Um, it's everything from what movie I'm watching, uh, to what crazy code thing am I doing. Um, and then uh, in terms of Hedera, the best place, uh, aside from our GitHub is adhera.dev and our blog, um, our LinkedIn is also very, very packed full of like actually interesting stuff. Um, we're not just like constantly advertising to you. Like we're actually talking about NUMA and stuff like that. So um, yeah, definitely give that a look and give uh, us the following.

Speaker B: Awesome. Thank you so much Alex for joining us today. Uh, would love to have you again at some point in the future for sure. Okay, that was a great interview. Hope you guys liked it as well. I liked how Alex broke down complex concepts around security. Why we need to assume breach when thinking about newer models like Mythos and Fable. How do we prep for that? Um, and how can Adira help upgrade your security posture, especially, um, on your existing Kubernetes clusters? Right, so no need to create new clusters and then migrate your workloads, but how you can upgrade your nodes, use the new amis, update your machine configs, and get it up and running for a better security solution for your existing production deployments. Uh, we do have similar episodes lined up, uh, with practitioners from our ever evolving ecosystem. So I would really appreciate if you can, like, share and subscribe the podcast. Subscribe to the podcast on Apple Podcasts, Spotify, and even on our YouTube channel. Um, and with that pitch, it brings us to the end of another episode. I'm Pawan and thank you for listening to the Kubernetes Bytes podcast.

Speaker A: Thank you for listening to the Kubernetes Bytes podcast.