Rust for Linux with Alice Ryhl and Greg Kroah-Hartman

It's Rust in Production, a podcast about companies who use Rust to shape the future of infrastructure. My name is Matthias Endler from corrode, and in this very special live episode at RustWeek, we welcome Alice Ryhl and Greg Kroah-Hartman from the Rust for Linux project to talk about saving the Linux kernel with Rust. Welcome everyone to a very special live recording here at RustWeek in Utrecht. This one has been a long time in the making at least for me two of my personal heroes are here today and we will talk about the rust for linux project rust in the linux kernel now if some of you have listened to the podcast before we had danilo krummrich from red hat on the podcast already and we talked about the nouveau driver and yeah we are back with one more episode for rust in production my name is matthias and i have to thank Alice and Greg for coming over the first question is always can you introduce yourself and perhaps the organization you're working for Alice maybe if you want to start.

So i'm Alice i work at google on the android rust team where I work on Rust, you know, in the Linux kernel, primarily. Rust on Android more generally. And I also maintain Tokio, the asynchronous runtime for Rust. And you are known as @darksonn on the internet, if some people remember that.

Exactly, yeah. I'm Greg Kroah-Hartman linux kernel developer and maintainer i work for the links foundation for the past many years i maintain parts of the links kernel like the driver core usb, tty and lots of little things and i also maintain the stable kernels so we do the stable kernel releases once or twice a week and i'm part of the security team and part of the cna team for the CVEs and i've been getting involved in the rust stuff as well. Now the kernel is getting close to i think 30 years of age nowadays which is incredible and of course it has started as a c project and it has been a c project for a very long time when was the time when you got first involved in linux in general.

Me i'm late in the late 1990s so only 25 years almost a newbie yeah a lot of the so no i got involved late 90s i had been using it in work for other things but i wrote firmware for usb devices and I worked on the USB specification. And I made devices like barcode scanners and I had to hook my, write my firmware to work on all other operating systems. So I tested on different ones. Linux did not have good USB support at the time.

My device was supposed to show up as a keyboard, but if I plugged into Linux, it showed up as a 102 key mouse. That was my firmware's bug, but funnily enough, Windows worked fine. So I fixed the firmware and Linux worked fine. but then I started contributing to Linux kernel that way because I saw little bugs I could fix and things like that and got involved and I realized I could get a job doing Linux full time in the late 90s and haven't stopped since.

Can you say you started contributing to linux because you got curious and you wanted to tinker with your own hardware and and you started to take on more and more responsibility in the project. Yeah i wanted my drivers and it was fun it was a fun feedback loop i wrote my first driver my wife went away for the weekend said work on that driver that you want to do i was like okay and i submitted it and i it was for a usb to serial converter and it was like instantly people wrote back this is wrong this is wrong this is wrong and oh have you heard of smp multi-processors i'm like what's that two processors at once?

Locking? I never heard of that. But it was great. It was this feedback loop.

And I learned that the engineering culture is, from an engineer's point of view, you learn so much from people who are much, much smarter than you. And I could become a much better programmer. I'd never learned about multiprocessors and four and 16-wave machines and things like that. And it teaches you to be a better engineer over time.

And I just love that feedback loop. And it got really, really good after time. Funnily enough, one of my first messages to the mailing list was like, how do I make a patch? Cause it didn't, the documentation wasn't there.

And the person who responded to me was very helpful and nice and whatnot. Like 10 years later, he ended up being my boss at SUSE. So it's a small world at times. Wow.

It's pretty incredible. Did you get into any segfaults? Did you have any segfaults? Oh, machines crash all the time.

But that's, I mean, you can crash your own machine and keep on going and whatnot. So it's not a big deal. And Alice, when was the first time you got involved in the Linux project or in the Rust for Linux project? Oh, I've been there for a long time, like three, maybe four years.

But you worked on Rust as well, way prior to that. So when was the time you started with Tokio? That's right. So I've been playing around with Rust for maybe 10 years, but for Tokio, the Async space, I got involved in around 2020, I would say.

And when was the time when you took over the project? Was that pretty much immediately or was there a transition period? Well, so, I mean, the way it all started was I showed up in the chat room. People were asking questions.

How do I do this with Tokio? At some point I was like, I've seen this question a few times, let me go fix the docs. So that was my first pull request to Tokio that was a docs fix. And it kind of just kept going and after I don't actually know how long time it took, but after a while I, started you know becoming part of the maintainer team and you know it all went from there.

What i find interesting in both of your stories is that you started taking on responsibility for extremely important technical projects and both of you came from a position of curiosity you started because you wanted to learn about that ecosystem and it wasn't a big bang moment or so, or you didn't graduate from university to become maintainers of these projects, but it was rather a very gradual process. And I really liked it. Over 50% of the Rust crates depend on Tokio, and I think everyone in this room has used Linux.

Raise your hand who has never used Linux before. Because that would be a lie. Tons of Linux devices in this room alone. The projector.

Sorry? The projector. The projector, even. And probably the technology for microphones and everything else.

Now, when we talk about Rust for Linux specifically, was that a social challenge for the most part, or was it a technical challenge to introduce Rust? I mean... It's a trick question. Let's go with...

I mean, that was both. It is both. It is both. But probably the biggest part was social.

Why is that? Well, I mean, if you are going to introduce a new language to a project, it's a big investment. You have to do a lot of work to, you know, you start to use it in one place and then, you know, and then people have to... People who have been coding for C in C code all of their lives now have to suddenly care about this Rust code that's becoming part of their code base.

And so there's a lot of stuff that you have to get buy-in from a lot of people. It's almost a bit like a leap of faith of sorts. You have to build trust first. Totally, yeah.

Trust is the key here. I mean, technically it was proven five, eight years ago that Rust could work in the kernel, right? They had examples out of tree, it could work. But the kernel works on trust of individuals and we trust people to maintain the code that they're going to contribute and maintain it.

And we don't necessarily trust that you're going to get it right because we all get it wrong. We trust that you're going to be there to fix it when you get it wrong, because we all get it wrong. And that's the trust model. We need to have the trust built up in the community that we have a big enough body of people both that can deal with this infrastructure and deal with this language and that benefits are there.

We all agree that the benefits are there finally. And that we trust the people who are maintaining and working on this that they will maintain and work on it continue. What made Rust unique in that sense? Because there were other languages which were attempting to be adopted in the Linux kernel, which were not adopted before.

I think if you're going to go to the big investment of introducing a new language, I think it has to be memory safe. There's no way it makes sense to spend that much time and effort if you're not going to go to a memory safe language these days. And it also has to be a language that you can use in the kernel. I mean, I'm sure Java would be great.

But, you know, there's a lot of languages out there that are memory safe with a garbage collector or stuff like that but i don't think any of those would work in the kernel and on the other side you have all the memory and safe languages and i think rust is almost the only language that fits it's the combination of things right it has all the things you need i think. It was the right idea at the right time. I think people proved that it could work. I mean, before we always had these ideas, wouldn't it be great if we could use C++?

And then all of us were like, no, please no. And for specific reasons. So nothing came along. Nobody spent the time and effort and energy to prove, hey, here is a solution of some way we can evolve forward.

And the Rust developers did that. And I have to hand it to them for doing that real work. It took them many, many years, but they were the ones that did the work and proved that it could be done. And previously, nobody had ever done that.

It's as simple as that. They did the work. But you could also do the work with another language. Nobody did, though.

Yes, I'm not disagreeing with you, but see, these people did the work and proved to us that this is a good idea, and here are the benefits, and they're going to stick around. This is going to save - I mean, I'll talk in my talk tomorrow about why I think Rust is going to save Linux in the end. And I think that's - we now understand and believe that. and they did the work to prove that to us.

It's a big statement to say rust is gonna safe Linux Yeah, I agree yeah i gotta talk tomorrow all about that. Now of course when when you say rust was the first to stick around and really tackle the hard problems i agree but at the same time i think rust is great because it can take a backseat in comparison to other languages like it can be still c for the majority of the kernel and you can take smaller bits and rewrite them in rust more or less gradually you don't have to do a big bang rewrite Interop Is the new rewrite.

Can you elaborate so. I mean if you have your massive of code base in one language, like as you said, you can't rewrite the entire thing. There's no reason to do that. It doesn't make any sense.

So what can you do instead? Well, what if you add a little bit of the new language, but then you have to talk, and that's interop is when one language talks to another. How can you do that from a technical perspective? How do you facilitate the interop story in the Linux kernel?

Where do you even start? It's hard, and that was my biggest fear. My biggest worry was, you know, see in the kernel, we have these ideas of how we do objects how we do memory how we handle this stuff and Rust says it's very strong ideas of how it does memory and language and objects and references and stuff and those bindings it's called a binding is the magic glue right and getting that right is very very hard and we have very smart people this room and Alice to do that work and without that and that was my big fear that i don't think this can actually happen until we prove that those bindings will work and a lot of people spent the time and energy to prove that it could work.

And we have working systems today as proof of that. But then you can take those bindings and just say from this point forward, hey, maybe we only write all the new graphics drivers in Rust, which the graphics driver maintainers have said. So then over time, new stuff coming in will evolve different language and we can evolve over time. The kernel evolves.

We don't just do massive rewrites. Every little step of the way, every tiny change that goes in work on its own. You can't break anything. So it's slow evolution over time that change.

Will work, and we'll do that. If you get the interop wrong, developer ergonomics will suffer. And we'll fix it. I mean, that's the best part of this stuff.

There's nothing in these interoperability... I mean, we get it wrong all the time. We're human, right? We can fix it.

There's nothing magic here. It's just code. Are there strategies to get it right the first time? Sure, we'd love to.

Does it live from a lot of tension between the old world and the new world? Does it live from a lot of, let's say, inertia to say, okay, this is how we think about this abstraction in the C world, and then now we need to build a safer wrapper around this or an interrupt layer for this to communicate from the Rust side? Like, on the technical side, how do you do that? Where does the rubber hit the road?

Maybe I can give one example. Let's say you have a C function that returns a pointer. Now the question is, what are the semantics of this pointer? So in Rust, there are many different pointer types.

We have the reference, we have the box, we have the arc, we have a bunch of different pointer types. And so when we write a Rust version of this C function, or like a Rust wrapper that calls the C function, we have to think, okay, this pointer, what are the semantics? And then in the Rust version, we have to pick the right pointer type. And so in this sense, we're adding more details to the signature.

We're saying this C function, which didn't say in the signature, what kind of pointer is it? Is it an owned pointer? Is it a borrowed pointer? We have to add that.

And so in some sense, we're adding a lot more semantics in the code that were not there in the C version. And that's the tricky part. But also along those ways, we've seen APIs and interfaces that we have in C that are just simple and easy to use in C. And it turns out it would be very complex in Rust.

Like famously in the driver core, there was some stuff that the driver, people writing the driver core bindings were going through hundreds and hundreds of lines of Rust code when you could do it in one line of C. Because, and I said, well, we can change the C code. And they're like, oh, and that was the big moment that, hey, if we work together, I can make C code simpler, do different, and you get to delete hundreds of Rust code. Great, let's do that.

And a lot of the work we've done on the C side, if Rust were to disappear today, hey, we've made our C code better because Rust has showed up. We've had to think about how these pointers work. We've had to think about, is this a mutable pointer or not? Is it const or is it not?

How do we work on this? And we changed the C code in the APIs because we can. And that's been a huge benefit for everybody. Even if they don't touch the Rust code, the C code has benefited so much from that.

Hinges a lot on the collaboration of a maintainer of a subproject sure. But that's fine i mean yes and. There might be subprojects which are more open to the idea and some which are more resistant. I know where you're going with it so the big so yes there's a lot of maintainers that don't want to deal with another language right for various reasons and that's fine so in the kernel we have the rule that if you bring some code in and maintainer doesn't want to maintain it you have to be the maintainer of it.

It's as simple as that. So then we have to find a responsible person that we trust to maintain this code. So for some subsystems, the maintainer that comes in for the bindings between the C and the Rust code is a new developer and they will work alongside the C maintainer, right? For other subsystems, I'll say like the driver core with me, Danilo came on and he said, I will be the, just maintain the driver core bindings, right?

I said, no, no, no, no. You're going to be the maintainer of all this. So we share it. Me and Rafael and Danilo share it.

We do the binder, we do the stress side, we do the seaside and we work together. So it depends on the subsystem. It's just, it's working with people. It's all just people and interactions.

And so far it's worked out really well. We have some new bindings that are coming in for one subsystem and a maintainer that is like, I don't even like maintaining the subsystem in the seaside. You want to maintain them both? I mean, and maybe the person who's doing the Rust side will say, okay, great, I'll do that.

And you take over and go from there. So it's just, we'll work through it as each individual subsystem. And we have a good, I want to say, a very solid mass, critical mass in the kernel of these bindings from the C to the Rust side to prove you can write real drivers. And Alice has proven you can write real drivers in code in Rust for a device that works today.

So going forward, it should be much easier, much simpler. Writing the Rust driver code is much simpler than the bindings. At least from an outsider's perspective, there used to be some resistance from some subproject maintainers to give Rust a chance in their ecosystem. Has that changed to some extent?

Have they changed their mind? Did you see any change in the way Rust is perceived in the Linux kernel by people who are, let's say, C veterans, very experienced C developers? I mean, I'm a C veteran. I've been doing this for C for 35 years every day.

People always disagree. We have 5,500 kernel developers every year. We can only agree on one thing. And that only one thing is that we want Linux to succeed.

After that, we disagree about everything. We're people, right? We don't have managers. We have the direction.

But we now have a critical mass of core maintainers. We made a public statement last year saying, the Rust experiment is over. It's not an experiment. We're going to do this for real.

we accept this as a project, we're going to go forward with this. Maintainers don't have to work with it and they don't have to interact with it if they don't want to. Just like any individual maintainer doesn't have to accept changes from anybody else, but everybody can be routed around. It's ostensibly this very pretty triangle of pyramid of hierarchy, but if you graph it, code flows in from everywhere, which is good.

We can all modify other people's code. We can maintain other people's code. We can fix bugs in other people's code and get the code merge and get it out to users which is what matters in the end so it's a very flexible, hierarchy system and yeah some people don't want to deal with it some people don't want to deal with people who use ai generated patches and they can refuse those too i mean we have lots of rules of lots of different things not a problem. Alice do you remember the time when you entered the picture and you started to build out all of these really great interrupt abilities between Rust and C?

How was that initial phase of building out all of these data structures in? Things have totally changed, right? So, okay, so one way to see it is there's this yearly conference called Linux Plumbers where all the kernel developers show up and we talk. And I've been going there for a few years now.

And every single time I've gone there, things have totally changed from the last year, right? I've seen this happen like four years in a row or something like that. So, yeah, I mean, like sure, there are some subsystems left, But there are definitely subsystems that, you know, I came to them, we had 0% Rust code, and then we started writing some Rust code. And then the next year, they were like, oh, I guess you have a cute toy there.

And then the next year, they have Rust code in their subsystem. And then the next year, it's no longer an experiment. Why would we not do this at all? I mean, it takes time.

It's human behavior, right? It's just, we're just people. we can't do mandates but we can persuade and discuss and hopefully convince people i was convinced i was astonished this is not going to work at all and then i went to one of the Rust conferences and i think i had a whole presentation aimed directly at me for like an hour and a half to prove hey this can be done and then we went out in the hallway and discussed for another three or four hours and then over drinks that night even more and i was convinced because.

Of the alcohol or. Because no no no what. Was your main concern that people wouldn't stick around until the end. Well my main concern again was back to this bindings is can we come up with a way to make these things interact in a way that's going to work good for both sides and i think Danilo and others have proven that actually it's that interaction and that binding is going to work better than i imagine so i'll call out they've done an amazing job there.

Yeah a quick round of applause for that well deserved. I certainly wouldn't contribute to the C side of the kernel because I would feel intimidated, honestly. I would not know if I could contribute in a safe manner, but I would definitely be interested in contributing to the rust side nowadays. Have you ever noticed any change in maintenance or maybe contribution behavior?

Are there more rust contributors now in comparison to maybe two, three years ago? If there are more people yeah for sure yeah absolutely yeah. And the other side, is that another thing where you say, okay, if you build Rust bindings for a C subset, you have to have a lot of empathy for the C code of that subset. But when you introduce new C code to the library, do you also keep Rust in mind now?

And to say, well, okay, maybe eventually we want to call this from C, so maybe we build out our code very differently now in comparison to before Rust existed. Well, yes and no. So we don't try and add code to the kernel that's not used. So you can't just say, I want to add a binding and here's an interface.

Sometime in the future, somebody can call us from Rust. We try not to add that. It's been a little chicken and egg problem with a lot of the bindings in the beginning because we didn't have a real user. Now we almost have a real user for everything.

But when you try and add something, we need a real user. If you're just adding new functionality to the c side that nobody in the Rust side is using why would you add the binding because nobody's going to use it it's not going to be there just add it when it's needed and away you go just add stuff as needed so you see a lot of the initial bindings that are coming in for there's tiny subsets of the bindings like look at some of the stuff for the file systems that you added it was just a tiny subset of interacting with files for what was needed for one driver and then we'll grow from there or you're doing adding stuff for netlink, tiny subset of, we can't really do a full network packet, but maybe we can do a netlink message that way.

So we'll just grow over time. And again, it's just evolution. You don't want to add giant chunks at once because it's unreviewable. And if it's not being used, why is it even there in the first place?

Can you use a lot of the Rust standard library in the Linux kernel, or do you really have to write everything from scratch? I remember that the Linux kernel has its own allocator and there's a lot of emphasis on linked lists, for example, on the C side, but I think there's also an implementation for it on the Rust side. Is there a huge overlap between what the standard library provides and what the Linux kernel needs and where does that overlap end? So the Rust standard library is split into three parts.

There's core, which is the core stuff, which we do use in the kernel. Then there's alloc, which is the things that only require an allocator and nothing else. We used to use that, but we wrote our own now. And then there's the std, the actual thing most people think of as the standard library, which provides things like files and TCP stream, But they require that there's an operating system below you, which there isn't in Linux.

So it doesn't make sense to use std. Originally, we were using core and alloc, and alloc was our allocator. Eventually, we found out that the Linux kernel does a bunch of interesting stuff for its allocators, and so we wrote our own alloc crate, but yeah. What interesting stuff.

Think about this. Memory needs to be allocated, and you can say, don't swap. Don't do any I.O.

to give me this memory, or give it to me on that NUMA node over there, or give it over here or give it out of this bucket or that bucket or merge these buckets and memory is i say it's just a simple driver for memory chips right but that's a really complex beast underneath there so interacting with that from the rest of the kernel is not just malloc and free it's free but the malloc part is a little more nuanced. That means that allocator is very much specific to the linux kernel itself.

Yes It doesn't. Make sense to use it anywhere else? Or is that a standalone library that I could use for any other purpose? For now, it's specific to the kernel.

It's built on top of the C side. So I mean, it's built on top of what we have in the kernel today. So there's just some, when you call and you ask for memory, you have to give it some hints of what type of memory you want and what you're doing at that moment in time. That in a normal Rust user space program makes no sense at all.

So you have to actually give it more options, more parameters. You started to adopt Rust in the Linux kernel on the, I would say, outskirts or maybe on the driver's side of things. And I think that was very clever because, well, drivers are isolated, less risk, less of a blast radius. You can experiment more there.

They don't have to be in tree. You can pull them back in tree if you like to, but only once they are mature enough. Do you find that Rust finds its way into the core of Linux now with maybe eventually replacing this allocator that was written in C with rust. So actually, drivers...

It makes it sound like drivers are the easiest thing to do for Rust. Drivers are the hardest thing to do because drivers consume from all the kernel. So Alice, to write a driver in Rust, had to make bindings to everything, like to the alloc crate, to do this, to do IO, to touch this file system, to do a memory malloc, to do all this stuff. So drivers are just, they look like a leaf on a tree, but they consume from the whole trunk, right?

It's easier to write a core piece of the trunk in Rust. In fact, the first code in Rust was a standalone bit of code that was added to the kernel. It was to write, do the QR code when the kernel crashes. Do that logic.

And famously, it had a memory bug. You can write memory unsafe code in Rust very easily in the kernel. Memory safety means something different in the kernel. So that standalone piece of Rust code was actually very easy and simple to do.

It was just one C in and one C function out. But to do drivers was very much more complex. And we had to agree as a community to accept this as going to be a possibility even more to do it as a driver. Going forward who knows we'll see we still don't have full support for all architectures that we support we saw what s390 got posted again today.

Yeah one. Of those architectures so we want to get all the main main architectures to have it before we'll be willing to put a core code to it because we just don't want to break systems. Yeah the big question is if you're going to have a core piece of logic is i mean if it's a new core piece then okay but if it's an existing core piece do you now have two implementations no this is this is great right so the architecture thing is pretty, important factor here in whether or not we use it in the call right.

Because the allocator is shared between the rust side and the c side and it's very critical i also thought of schedulers for a moment but again we have schedulers they do work i don't even know what the latest scheduler is that is used in the default Linux kernel. You can write schedulers in BPF. So people are hacking their little steam boxes to write schedulers from user space to do better framework. And that same scheduler actually works really well in giant servers.

So it's kind of fun. So you can write schedulers from user space in Rust, compiled to BPF, injected into the kernel. And people have been doing that for a few years now. What organizations use Rust for is mostly foundational to an organization.

It's in the core, the thing that cannot fail. This is where you get the most leverage from the language. It's memory safe. And this is how you can hook into that system, sort of.

And the thing that, in my opinion, comes closest to that in the Linux kernel would be I.O., I guess. We have a couple of different I.

O. systems in the kernel. I can't list all of them, but they were poll-based ones. There's IO Uring.

And Alice, you will also give a talk on IO. I'm not sure if that is related to your work on the Linux kernel, or if you can maybe share what this work is about. So I have a talk about completion based IO, but that's about user space, async Rust, and the Tokio side. Here the question is, it's actually kind of difficult to use IO Uring from async Rust, because we designed our trades to work really well with ePol and then Linux came around in 2019 and said hey we have this thing called io_uring which does not work in the same way and is not necessarily so easy to just swap them around, so yeah that's what my talk later today is going to be about.

What would have to change in Tokio to support this new I.O. model in Linux better? So in some places we already have started using it actually.

So if you have a file in Tokio, then you can go through, then if you enable the AyoUring stuff, then it will transparently use AyoUring instead of the normal background thread pool to do file IO, which is pretty great. But it's going to be much more tricky to use it for networking, for example. Because for networking, Yeah. Well, because networking is where ePoll is used today.

And ePoll is kind of difficult to translate to our hearing without changing the API we have, right? And that's the thing, right? In user space, if we change the API, are people going to use it? You need some sense of backwards compatibility for people to adopt your new stuff.

That's the mantra of the Linux kernel. Don't break user space. The kernel is always at fault then. Yeah, but we have this boundary of what's user space what's the kernel that's very well defined but within that we change everything all the time and so those are our apis are internally like you say the binding we get it wrong the first time great we'll rewrite it and make it work that's not seen by user space at all so yeah our goal is never to break user space you should always be able to update to the latest kernel and nothing should ever break if so then we did something wrong.

Rust could also make that boundary a bit safer. Has nothing to do with rust sorry that that's not that's not the the issue that api of how you enter into the kernel and how you exit back out is language agnostic let's just say it's just a way of getting data in and out and rust no language matters there we traditionally have c wrappers but rust has rust wrappers and go has go wrappers and all etc how that goes. Now I find it interesting also, Ellis, that you work on both sides.

You work on, say, quote-unquote, a user space application and the kernel space. And if you find a thing that maybe in Tokio could have been done differently on the Linux kernel side, is there a way for you to maybe encourage a discussion around that? To say, oh, can we change this API that would help us on the user space so much? Or do you strictly separate that work between Tokio and the Linux kernel right now?

So obviously, if I want to make some Linux kernel changes motivated by my user space work, it would be a lot easier now that I know how the Linux kernel community works. I don't think we've had user space API changes per se. We have had a case where we had, I think, a kernel bug, and then I already knew the person on the Linux kernel side that I needed to speak to, and then we could figure out what the bug was and get it fixed. So in that sense, the crossover is quite useful.

Now, looking ahead a little bit into the future of the Rust for Linux project, what would you say will land in the next 6 to 12 months? What can people expect there? Or is it mostly going to be fixing small little paper cuts, making the APIs a little nicer? Or are there any big plans?

I mean, everybody asks what's next in Linux. It's whatever happens to show up. We don't know. We don't know what people are working on.

We have hints of what people are working on. I have hints of people saying, I'm going to be doing this and this and this. Like, that's great. I believe it when I see it.

So whatever shows up, we'll deal with and we'll handle it then. So we have ideas that people are, you see hints of what people are writing new drivers in that are in Rust. We have the graphics drivers being written in Rust. We have other things being written in Rust.

Let's see what happens. Let's see what shows up. , what's on your agenda? Well, We have some pretty cool projects ongoing.

So actually here at the conference, we have a booth where you can try to play a racing game on a device running a Rust GPU driver in the kernel. So, you know, that shows that it's working. It's very much a prototype at this point. But, you know, hopefully something will happen with that.

I have one more thing. It might be a little bit political, So just tell me if you don't want to answer to this, but are maintainers now inclined to also review the Rust code or is it still okay to maybe not touch the Rust code whenever you make a patch to the C part of things, if you don't want to look at the Rust at all? There have always been some maintainers who are interested in the Rust code and have, you know, been curious to see, hey, what is this and how does it work?

Remember, we also have 700 maintainers. So to classify everybody is all agreeing to do the same thing is kind of tough. We maybe have 150, 200 core maintainers. Even getting those people to agree is hard, right?

So we all work differently. And the goal of the kernel, any change that comes in should never break the build. And if it breaks the build on the Rust side, we should be able to fix it up. I mean, so far, the things that have come in really haven't broken much.

It hasn't been an issue. it really and it's part of that's one reason we agreed this experiment is over it looks like this is actually working because they put the work in in the past couple years we haven't had a really big roadblocks and things seem to be working okay and let's try it. But if i send in a patch to the c side and it breaks things on the rust side is is there any discussion about that or. It depends on the depends on the maintainer if i even if i notice it i'm like hey you broke that can you fix that so our build bot might just smack you down on the on the mail on the thing saying you broke the bill so you can't do that so like if you break the bill automatically i'm like okay come on let's figure this out if you say i don't know how to do the rust side great i'll offer help and things like that i don't think we've ever nobody's ever turned down help or things like that a lot of people like oh cool i've wanted to try this rust thing let me try this out.

Ah that's a very interesting way to solve it because you take away this objectiveness and you make it objective by making it a technical problem you failed the build doesn't mean you made a mistake or you should have been more careful or why didn't you look at the rust part. You just say we all make mistakes right and we have checklists and that's why we have bots that run on the thing to say hey you did this wrong that's why we have like some subsystems have ai tools that are running on patch reviews and say hey that looks good but what about this what about this what about this it's like back to my very first patch i submitted i'm like oh i got all those things wrong great That's the technical thing I got wrong.

I'll learn from this and move on. My next submission go. On average, it takes at least three tries to get a patch accepted into the kernel. And we're still running a huge, almost 10 changes an hour for the past 20, 15 years.

Our rate of change is huge. And our rate of actually submissions and numbers of times that have to get accepted is a lot. That's normal. That's just how things go.

I've had some patch series that's taken like 25 times to get a patch series in mergeable state. And that's just normal. On the C side, I don't think we've seen anything that's really broken the Rust side yet. What are our Clippy settings?

I don't know. They're there. Do we use Clippy in the kernel? Yes, yeah, and they keep breaking.

But as a new version of the Clippy, it's fine. And then the backport, the stable kernels, and I take patches that will fix those up. Is it already set to pedantic mode? I don't know.

No, it's not set up. So we enable some set of things, right? And we do enable the... We don't enable the pedantic group, but we do enable the...

Group of lints. And so when Rust compiler or Clippy adds a new lint in the default group, then suddenly we have to go update our code because we want it to be lint-free. And of course, as Craig said, then we have to backport the fix as well because we also want the code on the stable releases to be lint-free ideally. But yes, so that's a yes on Clippy.

Isn't it ironic because C developers also, by osmosis, use Clippy now. If they break the build and maybe the Rust side broke and Rust uses Clippy to just enforce certain rules, also just the Rust compiler being stronger or maybe detecting more bugs, does that mean the C code also profits from that ecosystem? Does it mean the C code over time gets better? because there's more and more edge cases which get uncovered by the Rust tooling?

Maybe, but we've had static code analysis in the kernel for decades. And we've fixed up things based on rules and we make sure we don't break the rules. The new version of compilers come out and they find more patterns and we fix those again, backboard those. So we're used to this.

This is good code hygiene. We have no objection to this at all. So if another tool comes along and says, need to fix this up in the C code, great, we'll fix it up. I don't think clippy is so revolutionary like they have lintz in the c code too i.

Think the thing that's more interesting. Is the rust format. Because on. The c side the formatting is.

Very loose we have rules yes we do but they're guidelines and are not as strict as the as the Rust i but i like that but we i mean a code formatting is just there to make things universal and pattern matching you just want to have it consistent not that you agree with it or whatnot but that's why we have a code style. And now we have a Rust code style. Great, we'll follow that. Wonderful, not a big deal.

Yep, just run us through the formatter. Do you use the default Rust format settings? Will you make any changes to it? I think there are some very minor changes to the settings, but almost.

Is there even a C formatter? Yeah. It's been around for decades. 50 years.

But it's harder to introduce it than to start with one. We've had it. We've followed the rules. I gave a talk 25 years.

One of my first talks I ever gave in a public thing was, here's our coding style, and here's why we have a coding style, and we abide by this coding style. Yeah, we've had that. And we've had, there's a bad little script in the kernel that will actually reformat your code to the coding style if you really want to run it. But it's been around for a long time.

So it is used across all subsystems. Yeah, yeah, yeah. Any new code coming in has to follow those rules. We have some really old code in the kernel that doesn't follow the rules, but that's fine.

That's nobody's touching those drivers anyway. But all new stuff, it's been, again, this is 25, 30 years we've been doing this. Yeah, that's not an argument here. It's not like we copied the C code style over to Rust.

Like we're not using tabs. Yeah, I wish. That's fine. Our editors just work.

It's okay. These are common standards for both the kernel side and the Rust side and we just live with them. It's all good. I remember that there was a Linux tool which was very impressive.

It found... Code semantically it wasn't a grab it was more of a semantic code search. Coconut yeah exactly so yes that's what got me to move to europe wait. What that's a very.

Interesting turn of events so julia luau who's a professor in paris has a tool called came up with a tool called coconut and it's a semantic tool that does code transformations and you can write code and rules and it'll, it'll fix problem. It'll transform the code based on these rules and do that long, long time ago. So you realize bugs are follow a common pattern, right? And if you define the pattern, like saying, oh, we forgot to check the error value of this function, always check that and fix it up.

So we do it right. And then we have these set of rules that you can run them and make sure any new code coming in doesn't break these rules. So her work has fixed more security bugs in the kernel than anybody else. Fixed them, got them out of the code, and it prevents them from coming back in.

Now, it turns out we haven't been running some of those recently, so some of those bugs are creeping back in. But there's also a graduate student here. I saw him at the conference. He's working on that tool for Rust.

He's presented at the Rust conferences a few times, and he's doing some really good work at Vrye University in Amsterdam here on that. So that tool is also available for Rust for code transformations of Rust code based on semantic rules. And that's been around for a long time, and that's a good thing to be doing for Rust. And that's all these LLMs do as well.

you can say look at this pattern over here is this pattern match over there and we've been doing that for the stable kernels for decades again we've fallen all the bug fixes that we didn't forget that we forgot to mark as bug fixes and now we know how to backport them just based on the semantic, analysis of the patch and the code it works well in rust now too any. Other tooling that people might not know that is used to yeah write rust code in the linux kernel So. There's a pretty cool project called KLint, which is short for kernelLint, which is a Rust compiler extension, kind of like how Clippy is a Rust compiler extension, but it's a custom run one written for the kernel, which checks some kernel specific things.

One of the ones that I'm the most excited about, which is still a work in progress, but there's a lint for... So in some parts of the kernel code base you're not allowed to sleep. And then klint has this lint that checks in those pieces of code if you call something that sleep. And so basically it's a repository with custom clippy lints.

Are these actually clippy lints or is that a separate system well. It's a it's a program compiler plug-in in the same way as how clippy is a compiler plug-in. Okay yeah i always find it interesting to cool yeah to to hook into the the rust kernel like that and extend it like this Because. It has to do a lot of pretty complex analysis where you run the Rust compiler, you get all the type information, and then based on that, you can then make the lint, right?

You can't just do this check based on, oh, this word is here. It requires much more complex analysis. You need to run the compiler to get the information you need to do the lint. So that's why it's a compiler plugin.

So we have to come to the end. And I thought about the final question for a long time. Typically, our final question is, what's your message to the Rust community? But I won't let you off the hook today that easily.

So I will try to reframe it. I will rephrase it. And I have two questions. In fact, I have one for each of you.

It's the same question, but just with a twist. So who wants to start? Okay. Alice, what can Rust developers learn from C developers?

Empathy, what can Rust developers learn from C developers. I want to hear this it's. A bit of a curveball. That's a hard question.

And your question will be the, if you like you can also answer first. Whoever has to answer first. Put you on the spot there yeah. You really did.

So i i can answer so rust rust free so when switching from c to rust and back to c i'm like oh no and c i gotta okay was this pointer doing who owned this pointer who did this who where did this go and i have to like mentally think a lot more so many years ago a friend of mine said you gotta try this rust language. It's really fun. It makes programming fun again. It's like, nah, it's fine.

I'm fine with C. And I should have taken him up on it. Many years later, he was right. Rust to me makes programming fun again.

So it makes, it takes the, I know what logic and what I want to express and how I want to get something done. And Rust provides me with the tools to know that I'm doing so in a way that I'm not doing all the stupid things I do in C wrong. And I don't have to keep track of that it's like okay it compiles logic looks right okay now i can worry about is my logic right i'm worried about all this other meta stuff i have to keep in my head as a c programmer so my thing is so from a c programming point of view learning from the rust developers is hey thanks for making programming fun again really appreciate it, you.

Can say the same thing. You can say working. With c has made. Rust fun again.

Programming fun again. I mean c is pretty fun right you got to do a, You know, you get to set up all these complex things and keep it all in your head. But I don't really know if I want to learn from that. I think part of it is also maintaining a project like Linux for the better half of three decades.

Is that the way to say that? Anyhow, you know what I mean. For a long time. Because you go through multiple different phases of, you know, different trends, people using different paradigms and different tooling and yeah generations of developers essentially and how do you make that work like the long-term vision of such a project that's the most impressive thing when i think about the linux kernel no matter the language really maybe in 30 years from now it might be a different language that gets added but what won't change is how you think about the project from an engineering perspective and you also think about the users So I think I can speak for all of us when I say thanks for both of your very important work, making the world a better place.

Thanks. Thank you very much. Rust in Production is a podcast by Corrode. It is hosted by me, Matthias Endler, and produced by Simon Brüggen.

For show notes, transcripts, and to learn more about how we can help your company make the most of Rust, visit corrode.dev. Thanks for listening to Rust in Production.