Elizabeth Figura on Wine and Proton

Speaker A: Today I'm talking to Elizabeth Figueira. She's a Wine developer at Codeweavers. And today we're going to talk about what that is and uh, all the work that goes into it.

Speaker B: Thank you, Jeremy. I'm glad to be here.

Speaker A: I think the first thing we should talk about is maybe saying what Wine is because I think a lot of people aren't familiar with the project.

Speaker B: So Wine is a translation layer. In fact I would say Wine is a Windows emulator. That is what the name originally stood for. It re implements the entire Windows or we say win 32 API so that programs that make calls into the API will then transfer that code to Wine. And we allow that Windows programs to run on things that are not Windows. So Linux, macOS, other operating systems such as Solaris and BSD. It works not by emulating the CPU, but by reimplementing every API basically from scratch and translating them to their equivalent or writing new code in case there is no, you know, equivalent.

Speaker A: I believe what you're doing is you're emulating system calls. Could you explain what those are and how that relates to the project?

Speaker B: Yeah, so system call in general can be used to refer to a call into the operating system to execute some functionality that's built into the operating system. Often it's used in the context of talking to the kernel. Windows applications actually tend to talk at a much higher level because there's so much high level functionality built into Windows. When you think about, as opposed to other operating systems that we basically we end up implementing much higher level behavior than you would on Linux.

Speaker A: And can you give some examples of what some of those system calls would be and I suppose how they may be higher level than some of the Linux ones?

Speaker B: Sure. So of course you have like low level calls like interacting with a file system, you know, create a file and read and write and such. You also have high level APIs to interact with a sound driver. There's uh, one I was working on earlier today called Xaudio, where you actually build this bank of sounds that's meant to be played in a game and then you can position them in various 3D space. And the operating system in a sense will take care of all of the math that goes into making that work. That's all running on your computer and then it'll send that audio data to the sound card once it's transformed it so it sounds like it's coming from a certain space. A lot of other things like parsing XML is another big one that uh, there's a lot of things. The space is honestly huge.

Speaker A: And yeah, I can sort of see how those might be things you might not expect to be done by the operating system. Like you gave the example of 3D audio and XML parsing, and I think XML parsing in particular, you would have thought that that would be something that would be handled by the standard library of whatever language the person was writing their application is. So that's interesting that it's built into the os.

Speaker B: Yeah, well, and languages like C, it isn't even part of the standard library. It's higher level than that. You have specific libraries that are um, widespread but not codified in a standard. But in Windows, in Windows they are part of the operating system. And in fact there's several different XML parsers in the operating system. Microsoft likes to deprecate old APIs and make new ones that do the same thing very often.

Speaker A: And something I've heard about Windows is that they're typically very reluctant to break backwards compatibility. So you say they're deprecated, but do they typically keep all of them still in there?

Speaker B: It all still works.

Speaker A: And that's all things that WINE has to implement as well to make sure that the software works as well.

Speaker B: Yeah, yeah. And we also need to make it work. We also need to implement those things to make old programs work. Because there is uh, a lot of demand, at least from people using WINE for getting some really old programs working from the early 90s even.

Speaker A: And that's probably a good thing to talk about. In terms of what are the types of software that people are trying to run with Wine and what operations system are they typically using?

Speaker B: Um, in terms of software, literally all kinds. Any software you can imagine that runs on Windows, people will try to run it online. So we're talking games, office software, productivity software, accounting. People will run, build systems online, build their, just build uh, their programs using on Visual Studio, running on Wine. People will run WINE on servers, for example. Like software as a service kind of things where you don't even know that it's running on wine. Really super domain specific stuff. Like I've run Astronomy software on Wine design, computer assisted design, even hardware drivers can sometimes work on wine. There's a bit of a gray area. Yeah.

Speaker A: Um, I think from maybe the general public or at least from what I've seen, I think a lot of people's exposure to it is for playing games. Is there something different about games versus all those other types of productivity software and Office software that makes supporting those different?

Speaker B: Um, there's some things about it that are different. Games of course have gotten a lot of publicity lately because there's been a huge push largely from Valve but also some other companies to get a lot of huge wide range of games working well under Wine. And that's really panned out in the, in the way I think, I think we've largely succeeded. We've made huge strides in the past several years. 5, 5, 10 years I think. So when you talk about what makes games different, I think one thing games tend to do is they have a very limited set of things they're working with and they often want to make things run fast and so they're working very close to the metal. They're not going to use an XML parser, for example. They're just going to talk directly as directly to the graphics driver as they can. Right. And probably going to do all their own sound design. I did talk about that X Audio library, but a lot of games will just talk directly as directly to the sound driver as Windows lets them. So this is a, um, often a blessing honestly because it means there's less we have to implement to make them work. When you look at a lot of productivity applications, and especially the other thing that makes some productivity applications harder is Microsoft makes them and they like to make a library for use in this one program like Microsoft Office and then say, well, you know, other programs might use this as well. Let's put it in the operating system and expose it and write an API for it and everything. And maybe some other programs use it. Mostly it's just Office, but it means that Office relies on a lot of things from the operating system that we all have to reimplement.

Speaker A: Yeah, that's somewhat counterintuitive because when you think of games, you think of these really high performance things that seem really complicated. But it sounds like from what you're saying because they use the lower level primitives, they're actually easier in some ways to support.

Speaker B: Yeah, certainly in some ways they, yeah, they'll do things like reimplement the Heap Allocator because the built in Heap allocator isn't fast enough for them. That's another good example.

Speaker A: You mentioned Microsoft's more modern Office, uh, suites. I've noticed there are certain applications that aren't supported like for example, I think the modern Adobe Creative Suite. What's the difference with software like that and does that also apply to the modern Office suite or is that actually supported?

Speaker B: Well, in one case you have things like Microsoft using their own APIs that I mentioned. With Adobe, that applies less, I suppose. But I think to some degree, I think to some degree the answer is that some applications are just hard and there's no way around it. And we can only spend so much time on a hard application debugging things. Debugging things can get very hard with wine. Let's let me like, explain that for a minute because normally when you think about debugging an application, you say, oh, I'm going to open up my debugger, pop it in, uh, break at this point, see what, like, all the variables are, or if they're not what I expect, or maybe wait for it to crash and then get a back trace and see where it crashed and why you can't do that with Wine. Because you don't have the application, you don't have the symbols, you don't have your debugging symbols. You, you don't know anything about the code you're running unless you take the time to disassemble and decompile and read through it. And that's difficult every time. It's not only difficult. Every time I've. I've looked at a program and been like, I really need to just. I'm gonna just try and figure out what the program is doing. It takes so much time and it is never worth it. And sometimes you have to, sometimes you have no other choice, but usually you end up. You actually rely on seeing what calls it makes into the operating system and trying to guess which one of those is going wrong. Now, sometimes you'll get lucky and it'll crash in WINE code, or sometimes it'll make a call into a function that we don't implement yet, and we know, oh, we need to implement that function. But sometimes it does something more obscure and we have to figure out, well, like all of these millions of calls it made, which one of them is. Which one of them are we implementing incorrectly? So it's returning the wrong result or not doing something that it should. And then you add on to that the, you know, all this sort of harder to debug things like memory errors that we could make. And it can be very difficult. And so sometimes some applications just suffer from those hard bugs. And sometimes it's also just a matter of not enough demand for something for us to spend a lot of time on it. Right,

Speaker A: Yeah, I can see how that would be really challenging because like you were saying, you don't have the symbols, so you don't have the source code, so you don't know what any of this software you're supporting how it was actually written. And you were saying that a lot of times, uh, there may be some behavior that's wrong or a crash, but it's not because Wine crashed or there was an error in Wine. So you just know the system calls it made, but you don't know which of the system calls didn't behave the way that the application expected.

Speaker B: Exactly.

Speaker A: I can see how that would be really challenging. And Wine runs so many different applications. I'm kind of curious, how do you even track what's working and what's not as you change Wine? Because if you support thousands or tens of thousands of applications, how do you know when you've got a regression or not?

Speaker B: So it's a great question. Um, probably over half of Wine by source code volume. I have to actually check what it is, but I think it's probably over half is what we call is tests. And these tests serve two purposes. The one purpose is AR regression tests, and the other purpose is AR conformance tests that tests how, uh, an API behaves on Windows and validates that we are behaving the same way. So we write all these tests, we run them on Windows and, you know, write the test to check what the Windows returns, and then we run them on Wine and make sure that that matches. And we have just such a huge body of tests to make sure that, you know, we're not breaking anything. And that, uh, every. Every. All the code that we, that we get into Wine, that looks like, wow, it's doing that really well. Nope, that's what Windows does. The test says. So. So pretty much any code that we, Any new code that we get, it has to have tests to validate to demonstrate that it's doing the right thing.

Speaker A: And so rather than testing against a specific application, seeing if it works, you're making a call to a, uh, Windows system call, seeing how it responds, and then making the same call within Wine and just making sure they match.

Speaker B: Yes, exactly. And that is, obviously, that is a lot more automatable.

Speaker A: Right.

Speaker B: Because otherwise you have to manually. You know, these are all graphical applications. You'd have to manually do the things and make sure they work. Um, but if you write automatable tests, you can just run them all, and the machine will complain at you if it fails it. Continuous integration.

Speaker A: And because there's all these potential compatibility issues where maybe a certain call doesn't behave the way an application expects. What are the types of ways that shows when someone's using software? I mean, uh, I think you mentioned crashes But I imagine there could be all sorts of other types of behavior.

Speaker B: Yes, very much so. Basically, anything, anything you can imagine again is what will happen. You can have. Crashes are the easy ones because you know when and where it crashed and you can work backwards from there. But you can also get. It could hang. It could not render. Right. Like maybe render a black screen for, you know, for games, you could, uh, very frequently have graphical glitches where maybe some objects won't render right or the entire screen will be red. Who knows? Uh, in a very bad case, you could even bring down your system. And we usually say that's not wine's fault, that's the graphics library's fault. Because they're not supposed to do that, uh, no matter what we do. But, you know, sometimes we have to work around that anyway. But yeah, there's been some very strange and idiosyncratic bugs out there too.

Speaker A: Yeah. And like you mentioned, uh, there's so many different things that could have gone wrong that I mentioned is very difficult to find. Yeah. And when software runs through wine, I think a lot of our listeners will probably be familiar with running things in a virtual machine, and they know that there's a big performance impact from doing that. How does the performance of applications compare to running natively on the original Windows OS versus virtual machines?

Speaker B: So in theory, and, um, I haven't actually done this recently, so I can't speak too much to that, but in theory, the idea is it's a lot faster. So there is a bit of a joke acronym to wine. WINE is not an emulator. Even though I started out by saying WINE is an emulator and it was originally called a Windows Emulator. But what this basically means is WINE is not a CPU emulator. When you think about emulators in a general sense, they're often emulators for specific CPUs, often older ones like, you know, Commodore Emulator or an Amiga emulator. But in this case, you have software that's written for an X86 CPU and it's running on an X86 CPU by giving it the same instructions that it's giving on Windows. It's just that when it says, now call this Windows function, it calls us instead. So that all should perform exactly the same. The only performance difference at that point is that all should perform exactly the same as opposed to a virtual machine, where you have to interpret the instructions and maybe translate them to a different instruction set. The only performance difference is going to be in the functions that we are implementing themselves. And we try to we try to implement them to perform as well or almost as well as Windows. There's always going to be a bit of a, uh, theoretical gap because we have to translate from say, one API to another. But we try to make that as little as possible. And in some cases the operating system we're running on is just better than Windows and the libraries we're using are better than Windows. And so our games will run faster. For example, sometimes we can, sometimes we can do a better job than Windows at implementing something that's under our purview. There are some games that do actually run a little bit faster in Wine than they do on Windows.

Speaker A: Yeah, that reminds me of how there's these uh, gaming handhelds out now and some of the same ones, they have a, they either let you install Linux or install Windows or they just come with it, ah, pre installed. And I believe what I've read is that oftentimes running the same game on both operating systems, running the same game on Linux, the battery life is better and sometimes even the performance is better with these handhelds. So it's really interesting that that can even be the case.

Speaker B: Yeah, it's really a testament to the huge amount of work that's gone into that both on the Wine side and on the side of the graphics team and the Kerne team. And of course, you know, the years of work that's gone into Linux even before these gaming handhelds were even under consideration

Speaker A: and something. So for people who are familiar with handhelds, um, like the Steam Deck, they may have heard of Proton. Uh, I wonder if you can explain what Proton is and how it relates to Wine.

Speaker B: Yeah, so Proton is basically. How do I describe this? So Proton is a sort of a fork, uh, although we try to avoid the term fork, we say it's a downstream distribution because we contribute back up to Wine. So it is an alternate distribution fork of Wine. And it's also some code that basically glues Wine into an embedding application. Originally intended for Steam and developed for Valve, it has also been used in others, but it has also been used in other software. It. So where Proton differs from Wine, besides the glue part is it has some, it has some extra hacks in it for bugs that are hard to fix and easy to hack around as some um, quick hacks for making games work now that are like in the process of going upstream to Wine and getting their code quality improved and going through review. But we want the game to work now when we distribute it, so that'll, that'll go into Proton immediately. And then once we have, once a patch makes it upstream, we replace it with the version of the patch from upstream. There's other things to make it interact nicely with Steam and so on. And yeah, I think, yeah I think that's. I got it.

Speaker A: Yeah. And I think for people who aren't familiar Steam is like this um, I don't even know what you call it, like a gaming store.

Speaker B: I call it a game store, game distribution service. It's got a huge variety of games on it and you just publish and it's a great way for publishers to interact with their, you know, with the wider gaming community. Uh, after, it's just after paying a cut to Valve of their profits they can reach a lot of people that way. And because all these games are on Steam and Valve wants them to work well on their handheld, they contracted us to basically take their entire catalog, which is huge, enormous and try and just step by step fix every game and make them all work.

Speaker A: Osu. Um, and I guess for people who aren't familiar, Valve uh, Software is the company that runs ah, Steam. And so it sounds like they've asked your um, company to help improve the compatibility of their catalog.

Speaker B: Yes, Valve contracted us. And again when you're talking about Wine using lower level libraries, they've also contracted a lot of other people outside of Wine. Basically the entire stack has had a tremendous, tremendous investment by Valve Software to make gaming on Linux work well.

Speaker A: And when you refer to the entire stack, like what are some of those pieces? At least at a high level I

Speaker B: would see, let me think there is the WINE project, the Mesa Graphics Libraries. That's another uh, open source software project that existed, has existed for a long time but Valve has put a lot of uh, funding and effort into it. The Linux kernel in various different ways, the desktop environment, uh, and Window Manager, um, are also things they've invested in. Yeah, everything that the game needs on any level and that the, and that the operating system of the handheld device needs.

Speaker A: And wine's been going on for quite a while. I think it's over a decade, right?

Speaker B: I believe. Oh more than, oh far more than a decade. I believe it started in 1990, I want to say about 1995, mid-90s. Um, I probably have that date wrong. I believe Wayne started about the mid-90s. So it's been going on for about three decades at this rate.

Speaker A: Wow. Okay. And so all this time how has the project sort of sustained itself? Like who's been involved and how has it been able to keep going? This Long,

Speaker B: uh, I think as is the case with a lot of free software, it just, it just keeps trudging along. There's been times where there's a lot of interest in Wine, there's been times where there's less. And we're fortunate to be in a time where there's a lot of interest in it. We've had the same maintainer for almost its entire existence. Alexander, uh, Juilliard. There was one person who started maintained it before him and uh, left a maintainership to him after a year or two. Bob, uh, Amstadt. And there has been a few, there's been a few developers who have been around for a very long time. A lot of developers who have been around for a decent amount of time, but not for the entire duration. And then a very, very large number of people who come and submit a one off fix for their individual application that they want to make work.

Speaker A: How does Crossover relate to the Wine Project? Like it sounds like you had mentioned Valve Software hired you for subcontract work, but Crossover itself has been around for quite a while. So uh, how has that been connected to the Wine project?

Speaker B: So the company I work for is codeweavers and Crossover is our flagship software. So Code Reverse does a couple different things. We have a sort of a porting service where companies will come to us and say can we port my application usually to Mac? And then we also have a retail service where we basically have our own, similar to Proton, but you know, older but the same idea where we will add some hacks into it for very difficult to solve bugs. And we have uh, a nice graphical interface. And then the other thing that we're selling with Crossover is support. So if you try to run certain application and you buy Crossover, you can submit a ticket saying this doesn't work. And we now have a financial incentive to fix it. You know, we'll try to fix your spend company resources to fix your bug. Right. So that's been. So Codeweaverse has been around since 1996 and Crossover, uh, I don't know the date, but Crossover has been around for um, probably about two decades if I'm not mistaken.

Speaker A: And when you mention helping companies port their software to for example macOS, is the approach that you would port it natively to macOS's APIs or is it that you would help them get it running using Wine on macOS?

Speaker B: Right. So that's basically what makes us so unique among porting companies is that instead of rewriting their software, we just basically stick it inside of crossover and, uh, and make it run. And the idea has always been the more we implement, the more we get correct, the more applications will work. And sometimes it works out that way, sometimes not really so much. And there's always work we have to do to get any given application to work. But yeah, so it's very unusual because we don't ask companies for any of their code. We don't need it. We just fix the Windows API.

Speaker A: And so in that case, the ports would be where, let's say someone sells a macOS version of their software, they would bundle crossover, uh, with their software.

Speaker B: Right. And usually when you do this, it doesn't look like there is crossover there. Like, it just looks like this software is native, but there is crossover under the hood.

Speaker A: And so earlier we were talking about how you're basically intercepting the system calls that these binaries are making, whether that's the executable or the DLLs from Windows. Um, but I think probably a lot of our listeners are not really sure how that's done. Like, they may have built software, but they don't know how do I basically hijack the system calls that this application is making? So maybe you could talk a little bit about how that works.

Speaker B: So there. So there's a couple steps to go into it. When you think about a program that say, that's a big, A, uh, big file that's got all the machine code in it. And then it's got stuff at the beginning saying, here's how the program works and here's where in the file the processor should start running. That's your.exe file. And then your DLL files are libraries that contain shared code. You have a similar sort of file that says, here's the entry point that runs this function, this parsexml function, or whatever have you, and here's this entry point that has the generate XML function and so on and so forth. And then the operating system will Basically take the.exe file and see all the bits in it. Say I want to call the parsexml function. It'll load that DLL and hook it up. So the processor ends up just seeing jump directly to this parsexml function and then run that and then return and so on. And so what WINE does, is it part of wine? That's part of wine is the library implementing that parsexml and readxml function. But part of it is the loader, which is the part of the operating system that hooks everything together. And when we load, we Redirect to our libraries. We don't have Windows libraries, we redirect to ours and then we run R code. And then when you jump back to the program and. Yeah,

Speaker A: so it's the loader that's a part of Wine that's actually not sure if running the executable is the right term.

Speaker B: No, I think that's a good term. It starts in a loader, and then we say, okay, now run the machine code, next executable, and then it runs and it jumps between R libraries and back and so on.

Speaker A: And like you were saying before, oftentimes when it's trying to make a system call, it ends up being handled by a function that you've written in Wine, and then that in turn will call the Linux system calls or the macOS system calls to try and accomplish the same result.

Speaker B: Right, exactly.

Speaker A: And something that I think maybe not everyone is familiar with is there's this concept of user space versus kernel space. Could you explain what the difference is?

Speaker B: So the way I would explain, the way I would describe a kernel is it's the part of the operating system that can do anything, right? So any program, any code that runs on your computer is talking to the processor. And the processor has to be able to do anything the computer can do. It has to be able to talk to the hardware, it has to set up the memory space. So actually a very complicated task has, uh, to be able to switch to another task and, and basically talk to another program. And you have to have something there that can do everything. But you don't want any program to be able to do everything. Um, not since the 90s. It's about when we realize that we can't do that. So the kernel is the part that can do everything. And when you need to do something that requires those, those permissions that you can't give everyone, you have to talk to the kernel and ask it, hey, can you do this for me, please? And in a very restricted way where it's only the safe things you can do. And to a degree it's also like a library, right? It's the, uh, kernel kernels have always existed and since they've always just been the core standard library of the computer that does the, that does the things like read and write files, which are very, very complicated tasks under the hood, but look very simple because all you say is write this file and talk to the hardware and abstract away all the difference between different drivers. So the kernel is doing all of these things. So because the kernel is the part that can do everything. And because, um, when you think about the kernel, it is basically one program that is always running on your computer, but it's only one program. So when a user calls the kernel, you are switching from one program to another and you're doing a lot of complicated things as part of this. You're switching to the higher privilege level where you can do anything and you're switching the state from one program to another. And so it's a. So this is what we mean when we talk about user space where you're running like a normal program, and kernel space where you've suddenly switched into the kernel and now you're executing with increased privileges and a different idea of the process space and increased responsibility and so on.

Speaker A: Uh, and so do most applications. When you were talking about the system calls for handling 3D audio or parsing XML, are those considered, are those system calls considered part of user space and then those things call the kernel space on your behalf or, huh, how would you describe that?

Speaker B: So when you look at Windows, most of the Windows library, the vast, vast majority of it, is all user space. Most of these libraries that we implement never leave user space. They never need to call into the kernel. There's only the core low level stuff. Things like we need to read a file that's a kernel call. When you need to sleep and wait for some seconds, that's the kernel call. You need to talk to a different process, things that interact with different processes in general, not just allocate memory, but allocate a page of memory from the memory manager and then that gets suballocated by the Heap allocator. So things like that.

Speaker A: Yeah. So if I was writing an application and I needed to open a file, for example, does that mean that I would have to communicate with the kernel to read that file?

Speaker B: Right, exactly.

Speaker A: And so most applications, it sounds like it's going to be a mixture. You're going to have a lot of things that call user space calls and then a few. You mentioned more low level ones that are going to require you to communicate with the kernel.

Speaker B: Yeah, basically. And it's worth noting that in all operating systems you're almost always going to be calling a user space library that might just be a thin wrapper over the kernel call. It's going to do like just a little bit of work and then call the kernel. In fact, in Windows that's the only way to do it. Uh, in many other operating systems you can actually say you can actually tell the processor to make the kernel call. There is A special instruction that does this and it'll go directly to the kernel and there's a defined interface for this. But in Windows that interface is not defined, it's not stable or backwards compatible like the rest of Windows is. So even if you wanted to use it, you couldn't. And you basically have to call into the high level libraries, or low level libraries as it were, that, that tell you that create a file. And those don't do a lot, they just kind of tweak ah, their parameters a little and then pass them right down to the kernel.

Speaker A: And so Wine, it sounds like it needs to implement both the user space calls of Windows, but then also the kernel calls as well. But Wine itself, is that only in Linux user space or macOS user space?

Speaker B: Yes, this is a very tricky thing. But all of Wine, basically all of what is Wine, runs in the user space and we use kernel calls that are already there to talk to the kernel, to talk to the host kernel. You have to, you get a sort of second nature of thinking about the Windows user space and kernel. And then there's a host user space in kernel and Wine is running all in the host user space, but it's emulating the Windows kernel. In fact, one of the weirdest, trickiest parts is I mentioned that you can run some drivers in wine and those drivers actually, they actually are, they think they're running in the Windows kernel, which in a sense works the same way. It has libraries that it can load and those drivers are basically libraries and they are making kernel calls and they're making calls into the kernel library that does some very, very low level tasks that you're normally only supposed to be able to do in the kernel. And you know, uh, because the kernel requires some privileges, we kind of pretend we have them and in many cases even the drivers are using abstractions. We can just implement those abstractions kind of over the slightly higher level abstractions that exist in user space.

Speaker A: Yeah, I hadn't even considered the being able to use hardware devices. But I suppose if in the end if you're reproducing the kernel, then whether you're running software or you're talking to a hardware device, as long as you implement the calls correctly, then I suppose it works because you're talking about device. Like maybe it's some kind of USB device that has drivers for Windows, but it doesn't for Linux.

Speaker B: Yeah, no, that's exactly, that's kind of the example I've used. Uh, I think there was, I think one of my best success stories Was, uh, drivers for a graphing calculator.

Speaker A: Uh, oh, wow.

Speaker B: That connected via usb. And I basically just plugged the Windows drivers into Wine and ran it. And I had to implement a lot of things, but it worked. But for example, something like a graphics, um, driver is not something you could implement in Wine because you need the graphics driver on the host. We can't talk to the graphics driver while the host is already doing so.

Speaker A: I see. Yeah. And in that case, it probably doesn't make sense to do so anyways.

Speaker B: Right, right. It doesn't, because the transition from user into kernel is complicated. You need the graphics driver to be in the kernel and the real kernel. Having it in Wine would be a bad idea.

Speaker A: Yeah, I think there's enough APIs you have to try and reproduce that. I think, uh,

Speaker B: it'd be very difficult.

Speaker A: Right. There's so many different calls both in user space and kernel space. I imagine the user space ones Microsoft must document to some extent

Speaker B: is that. Well, sometimes.

Speaker A: Sometimes. Okay.

Speaker B: I think it's actually better now than it used to be. But some. Here's where things get fun, because sometimes there will be, you know, regular documented calls. Sometimes those calls are documented, but the documentation isn't very good. Sometimes programs will just sort of look inside Microsoft's DLLs and use calls that they aren't supposed to be using. Sometimes they use calls that they are supposed to be using, but the documentation has disappeared just because it's that old of an API and Microsoft hasn't kept it around. Sometimes. Sometimes Microsoft's own software uses APIs that were never documented because they never wanted anyone else using them, but they still ship them with the operating system. There was actually a kind of a lawsuit about this because it is an antitrust lawsuit, because by shipping things that only they could use, they were kind of creating a trust. And that got some things documented, at least in theory. They kind of haven't stopped doing it, though.

Speaker A: Oh, so even today you would call those private. Private APIs, I suppose?

Speaker B: Uh, yeah, you could say private APIs. But if we want to get, you know, newer versions of Microsoft Office running, we still have to figure out what they're doing and implement them.

Speaker A: And given that they're either, like you were saying, the documentation is kind of all over the place. If you don't know how it's supposed to behave, how do you even approach implementing them?

Speaker B: And that's what the conformance tests are for. I mentioned earlier, we have this huge body of conformance tests that double as regression tests. If we See, an API we don't know what to do with, or an API we do know, we think we know what to do with because the documentation can just be wrong and often has been. Then we write tests to figure out what it's supposed to behave. We kind of guess until we, and we write tests and we pass some things in and see what comes out and um, see what the operating system does until we figure out oh, so this is what it's supposed to do. And these are the exact parameters in, and, and then we, and, and then we implement it according to those tests.

Speaker A: Is there any distinction and approach for when you're trying to implement something that's at the user level versus the kernel level?

Speaker B: No, not really. And like I, and like I mentioned earlier, like. Well, I mean a kernel call is just like a library call. It's just done in a slightly different way, but it's still got, you know, parameters in. It's still got a set of parameters, they're just encoded differently. And again, like the, the way kernel calls are done is on a level just above the kernel where you have a library that just passes things through almost verbatim to the kernel and we implement that library instead.

Speaker A: And you've been working on, I, uh, think WINE for over six years now.

Speaker B: That sounds about right.

Speaker A: What does uh, your day to day look like? What parts of the project do you work on?

Speaker B: It really varies from day to day and. Some people will work on the same parts of WINE for years. Uh, some people switch around and work on all sorts of different things. And I definitely belong to that second group. If you name an area of wine, I have almost certainly contributed a patch or two to it. There are some areas I work on more than others, like 3D graphics, multimedia. I worked on a compiler that exists, uh, sockets, so networking communication is another thing. I work a lot on day to day. I kind of just get, I kind of just get a bug for some program or another and I take it and I debug it and figure out why the program's broken and then I fix it. And there's so much variety in that because a bug can take so many different forms like I described and, and the fix can be simple or complicated and it can be in really anywhere. To a degree, being able to work on any part of WINE is sometimes almost a necessity because if a program is just broken, you don't know why. It could be anything, it could be any sort of API. And sometimes you can hand the API to somebody who's got a Lot of experience in that. But sometimes you just do whatever. You just fix whatever's broken and you gain experience that way.

Speaker A: Yeah, I mean, I was gonna ask about the specialized skills to work on wine, but it sounds like maybe in your case it's all of them.

Speaker B: There's a bit of that. The skills to work on Wine are very. It's a very unique set of skills because. And it largely comes down to debugging because you can't use the tools you normally. Normally use to debug. You have to be creative and think about it different ways. Sometimes you have to be very creative. And programs will try their hardest to avoid being debugged because they don't want anyone breaking their copy protection, for example, or hacking or, you know, hacking in cheats. They want to be. They don't want anyone hacking them like that. And we have to do it anyway for good and legitimate purposes, we would argue to make them work better on more operating systems. And so we have to fight that every step of the way.

Speaker A: Yeah, it seems like it's a combination of, uh, uh, being able, like you were saying, being able to debug. And you're debugging not necessarily your own code, but you're debugging this behavior. And then based on that behavior, you have to figure out, okay, where in all these different systems within WINE could this part be not working? And I suppose you probably build up some kind of mental map in your head of when you get a type of bug, uh, or a type of crash. You think, oh, maybe it's this, maybe it's here or something like that.

Speaker B: Yeah, there is a lot of that. You notice some patterns. Experience helps. But because any bug could be new, uh, sometimes experience doesn't help. And you just kind of have to start from scratch

Speaker A: at sort of a high level. Can you give an example of where you got a specific, um, bug report and then where you had to look to eventually find which parts of the system were the issue?

Speaker B: One, I think, good example that I've done recently. So I mentioned this Xaudio library that does 3D audio. And uh, say you come across a bug. I'm going to be a little bit generic here. And say you come across a bug where some audio isn't playing right. Maybe there's silence where there should be the audio. So you kind of, you ought uh, to look in and see, well, where is that getting lost? So you can basically look in the input calls and say, here's the buffer. It's submitting. That's got all the audio data in it and you look at the output, you look at where you think the output should be. Like that library will internally call a different library which programs can interact with directly. And this, our high level library interacts with that is the give this sound to the audio driver. Right. So you've got X Audio on top of um, MMDev API, which is the other library that gives audio to the driver. And you see, well, the buffers that X Audio is passing into MMDave API, they're empty, there's nothing in them. So you have to kind of work through the XAUDIO library to see where is that sound getting lost? Or maybe, or uh, maybe that's not getting lost, maybe it's coming through all garbled. And I've had to look at the buffer and see why is it garbled? I'll open it up in Audacity and look at the shape of the wave and say huh, that shape of the wave looks like it's, it looks like we're putting silence every 10 nanoseconds or something, or reversing something or interpreting it wrong, things like that. Um, there's a lot of. You will do a lot of putting in printfs basically all throughout wine to see where does the state change, where is it right and then where do things start going wrong?

Speaker A: Yeah. And in the audio example, because they're making a call to your X Audio implementation, you can see that, okay, the, the buffer or the audio that's coming in, that part is good.

Speaker B: Right.

Speaker A: It's just that later on when it sends it to what's going to actually have it be played by the um, hardware, that's when it's missing.

Speaker B: So we did something wrong in a library that destroyed the buffer. And I think on a very, very high level a lot of debugging WINE is about finding where things are good and finding where things are bad and then narrowing that down until we find the one spot where things go wrong. There's a lot of processes that go like that.

Speaker A: Like you were saying, the more you see these problems, hopefully the easier it gets to narrow down where often.

Speaker B: Yeah, especially if you keep debugging things in the same area.

Speaker A: And WINE supports more than one operating system. I saw there was Linux, macOS, I think FreeBSD. How much of the code is operating system specific versus how much can just be shared across all of them?

Speaker B: Not that much is operating system specific actually. So when you think about the volume of wine, the vast majority of it is high level code that doesn't need to interact with the operating system on a low level. Right. Because Windows keeps putting, because Microsoft keeps putting lots and lots of different libraries in their operating system. And a lot of these are high level libraries. And even when we do interact with the operating system, we're using cross platform libraries or we're using posix. Uh, so all these operating systems that we are implementing are basically conformed to the POSIX standard, which is basically like Unix. They're all Unix based. POSIX is a Unix based standard. Microsoft is, you know, the big exception that never did implement that. And so we have to translate its APIs to Unix APIs. Now that said, there is a lot of very operating system specific code. Apple makes things difficult by diverging almost wherever they can. And so we have a lot of Apple specific code in there.

Speaker A: Another example I can think of is I believe macOS doesn't support Vulkan.

Speaker B: Yes. Yeah, that's a great example of Mac not wanting to use, uh, generic libraries that work on every other operating system. And in some cases we look at it and we're like, all right, we'll implement a wrapper for that too on top of your, um, operating system. We've done it for Windows, we can do it for Vulkan. And then you get to Molten VK project. Uh, and to be clear, we didn't invent Molten bk. It was around before us, but we have contributed a lot to it.

Speaker A: Yeah, I think maybe just at a high level it might be good to explain the relationship between Direct3D or DirectX and Vulkan. And um, yeah, maybe if you could go into that.

Speaker B: So direct 3D is Microsoft's 3D API. The 3D APIs are basically a way to firstly abstract out the differences between different graphics cards which look very different on a hardware level especially. They used to look very different. They still do look very different. And secondly a way to deal with them at a high level. Because actually talking to the graphics card on a low level is very, very complicated. Even talking to it on a high level is complicated, but it can get a lot worse if you've ever been a, if you've ever done any graphics driver development. So you have a number of different APIs that achieve these two goals of, of abstraction and of uh, building a common abstraction and of building a high level abstraction. So OpenGL is broadly the free operating system world, the non Microsoft world's choice back in the day and in direct 3D was Microsoft's API and direct 3D. And both of these have evolved over time and come up with new versions and such. And when any API exists for too long, it gains a lot of cruft and needs to be replaced. And eventually the people who developed OpenGL decided we need to start over, get rid of the crop to make it cleaner and make it lower level. Because to get into maximum performance, games really want low level access. And so they made Vulkan. Microsoft kind of did the same thing, but they still call it Direct 3D. They just, it's, it's their, the newest version of Direct 3D is lower level, it's called Direct 3D 12. And, and Mac looked at this and they decided we're gonna do the same thing too, but we're not gonna use Vulkan, we're gonna define our own. And they call it Metal. And so when we want to translate D3D12 into something that another operating system understands, that's probably Vulkan. And on Mac we need to translate it to Metal somehow. And he decided instead of having a separate layer from D312 to metal, we're just going to translate it to Vulcan and then translate the Vulcan to Metal. And it also lets you things written for Vulkan on Windows, which is also a thing that exists that lets them work on Metal.

Speaker A: And having to do that translation, does that have a performance impact or is that not really felt?

Speaker B: Yes, uh, it's kind of like anything when you talk about performance, like I mentioned this earlier, there's always going to be overhead from translating from one API to another. But we try to, well, we put in heroic efforts to try to make sure that doesn't matter, to make sure that stuff that needs to be fast is really as fast as it can possibly be. And some very clever things have been done along those lines. And sometimes the graphics drivers underneath are uh, so good that it actually does run better, even despite the translation overhead. And then sometimes to make it run fast, we need to say, well, we're going to implement a new API that behaves more like Windows so we can do less work translating it. And that's. And sometimes that goes into the graphics library and sometimes that goes into other places.

Speaker A: Yeah. Something I've found a little bit interesting about the last few years is developers in the past, they would generally target Windows and you might be lucky to get a Mac port or a Linux port. And I wonder, like, in your opinion, uh, now that a lot of developers are just targeting Windows and relying on Wine or Proton to run their software, is there any, I suppose, downside to doing that? Or is it all just upside, like everyone should target Windows as this common platform?

Speaker B: Yeah, it's an interesting question. Uh, there's some people who seem to think it's a bad thing that we're not getting native ports in the same sense. And then there's some people who see, no, that's a perfectly valid way to do ports. Just write for this de facto common API. It was never intended as a cross platform common API, but we've made it one.

Speaker A: Right.

Speaker B: And so why is that any worse than if it runs on a different API on Linux or Mac? Yeah, I guess I tend to, that argument tends to make sense to me. I don't personally see a lot of reason to say that one library is more pure than another right now. I do think Windows APIs are generally pretty bad. Um, this might be, you know, just some sort of, this might just be an effective having to work with them for a very long time and see all their flaws and have to deal with the nonsense that they do. But I think that a lot of the native Linux APIs are better. But if you like your Windows API better and if you want to target Windows and that's the only way to do it, then sure, why not? What's wrong with that?

Speaker A: Yeah, and I think the doing it this way, targeting Windows, I mean if you look in the past, even though you had some software that would be ported to other operating systems without this compatibility layer, without people just targeting Windows, all this software that people can now run on these portable gaming handhelds or on Linux, most of that software was never going to be portable ported. So just having that as an option.

Speaker B: Yeah, that's kind of why WINE existed, because people wanted to run their software, you know, that was never going to be ported. They just wanted. And then the community just spent a lot of effort in making all these individual programs run.

Speaker A: Yeah, and I think it's pretty amazing too that now that's become this official way, I suppose, of distributing your software where you say like, hey, I made a Windows version, but you're on your Linux machine, it's officially supported because we have this much belief in this compatibility layer.

Speaker B: It's kind of incredible to see Wine, uh, having got this far. I mean, I started working on it, you know, six, seven years ago and even then I could never have imagined it would work be like this.

Speaker A: So as we wrap up for the developers that are listening or people who are just users of Wine, um, is there anything you think they should know about the project that we haven't talked about?

Speaker B: I don't think there's anything I can think of.

Speaker A: And if people want to learn uh, more about the Wine project or see what you're up to, where, where should they, where should they head?

Speaker B: We don't really have any things like news, unfortunately. Um, read the release notes, uh, follow some. There's some, there's some people who, from codeweavers who do blogs. So if you, so if you go to codewevers.com blog, there's some, there's, there's some codeweavers stuff, uh, some marketing stuff. But there's also some developers who will talk about bugs that they are solving and how uh, it's easy and the experience of working on Wine.

Speaker A: And I suppose if someone's interested in, let's say they have a piece of software, it's not working through Wine, what's the best place for them to either get help or maybe even get involved with trying to fix it?

Speaker B: Yeah, uh, so you can file a bug on winehq.org or you know, find there's a lot of developer resources there and you can get involved with contributing to the software. And uh, there's links to our mailing list and IRC channels and uh, the GitLab where all places you can find developers. We love to help you debug things, we love to help you fix things. We try our very best to be welcoming community and we have a lot of long. We've got a lot of experience working with people who want to get their application working. So we'd love to have another.

Speaker A: Very cool. Yeah, I think Wine is a really interesting project because I think for, I guess it would have been for decades it seemed like very niche, like not many people were aware of it. And uh, now I think maybe in particular because of the Linux gaming handhelds like the Steam deck, Wine is now something that a bunch of people who would have never heard about it before, now they're aware of it.

Speaker B: Absolutely. I've watched that transformation happen in real time and it's been surreal.

Speaker A: Very cool. Well Elizabeth, thank you so much for joining me today.

Speaker B: Thank you, Jeremy. I've been glad to be here.