Why DIU Ruined Wayne Starr in the Best Way

Speaker A: M. I want to control technology. I don't want technology to control me. If it's closed source software, it could suddenly require a subscription at some point. It could be connected to the cloud and who knows what's happening with the data? Who knows where that's going? And so I try to pull as much back as I can to things that I can control and that I can monitor and use.

Speaker B: Welcome to the Defense Unicorns podcast. Today's guest is Wayne Starr, who took an unconventional path through the Air Force, pushing boundaries in software innovation from the inside out. From DiU to defense unicorns, Wayne's journey shows what happens when you build your own way, but still understand all of the policy. Wayne, welcome to the Defense Unicorns podcast. I'm super excited to have you on. If you don't mind, just sort of tell our listeners a little bit about who you are and why you are. Like I was about to do something terrible and ask you why you are the star of the oh yes, factory ecosystem. But I'm not going to do that. I'm just going to ask you to tell me about your background.

Speaker A: Yeah, so I have a background in software engineering. Got my degree out at Rochester Institute Technology in Rochester, N.Y. really had a passion for that since I was a kid. Did ROTC there as well and got a chance to actually intern at Microsoft and get a little bit of commercial experience before joining the Air Force. My year group was something sort of, kind of special, I guess, in that our training backlog had been backed up so far that Congress mandated that 90% of active duty second lieutenants had to be on active duty by the end of the fiscal year. And so I got the opportunity for my first assignment straight out of college to go to the Defense Innovation Unit out in Mountain View, California. And so that really was a, uh, special, uh, opportunity for me to kind of like explore the software engineering skills that I had learned and then sort of see how all this stuff fit into the Air Force and the broader ecosystem of how software development happens within government systems and specifically Air Force systems out there. What I kind of did was really a blend of information technology stuff for diu, but also a lot of software engineering, kind of logistics and security management. There was, we, we basically I came in right in at the end of 2016 to what was kind of initially called the, the Jigsaw effort or was eventually turned into the Tanker Planner tool. And basically what we were doing was trying to figure out how we could improve the Mission Air Attack Plan toolkit at that time. And then be able to actually deliver something that worked for the warfighters that were out in Al ud, Qatar.

Speaker B: Are you basically trying to take like uh, a legacy software system and turn it into something that delivered like on a more like agile timeline?

Speaker A: Yeah. So basically the previous system, AOC 10.2 or Air Operations Center 10.2 was the update from 10.1. So effectively a minor upgrade had started in 2006. And by 2017, by the time we actually delivered the tanker planner to the first version, it hadn't delivered a single anacoded production. And so they were very hard stuck in waterfall in the way that they developed things. They were also not really looking at biting off small chunks of the elephant. They were trying to do this whole thing and by trying to do that it just really, really slowed them down. This, I think the average ATO time or something was something like 340, 350 days. It was basically almost a year. And normally with waterfall, obviously you're doing your requirements, you're doing your design, you're doing your development, and then you're kind of doing security, all as a back afterthought. And so if anything gets screwed up at any point during that process, you kind of have to go back to the beginning and start again, and start again, start again. And so that winds up really hampering and slowing down development. And what we did is really trying to take a user centered approach, flying out actually to Qatar to go talk to the tanker planners, incorporate that feedback into the system within a matter of weeks, within a matter of a few sprints, and then deliver that back to them. And I still remember one of the times where we had kind of a game breaking bug with our system where when we first deployed it onto siprnet is where they were doing a lot of their development. It would take every time they hovered over a puck on the board. So the way they planned was these pucks, they'd move around basically to like align them with sorties. Whenever they hovered over one of those pucks, it would take 12 seconds to load. This is something we didn't see on our Mac development systems because they were just so fast that they could kind of grunt through that, that particular thing. But we were able to over a weekend, myself and Logan Raeplinger, another developer that was out there TDY with us that we had that team, we basically mapped everything out. All of the different. We did basically the big O notation for the number of sorties and the number of tankers and all the different things that all the variables that were going into it and then mapped out where the Sloidon was, provided a fix, and then had that deployed back to siprnet by that Monday. And that was like the first trip where they were actually able to use the Tagger Planner tool for the first time. Because that was like, the. One of the last bugs that we actually cleared for them. Um, but we would have never gotten that feedback if we never delivered it, put it in front of the users, and actually got feedback from them directly. We'd never known that it existed. We'd never known as a problem or anything. So that's an example of why you need to actually engage with your users and why that stuff is so important.

Speaker B: Did you find the users were excited to engage when you went out there, or were they, like, almost annoyed that these software nerds were hanging out?

Speaker A: Yeah, there was a little bit of skepticism at first, I would say. Basically, obviously, when you say you're from the government, you're here to help. Even the people in the government don't really like that a ton and are skeptical. But there was a lot of sort of hope, I guess, that people had. And then once people started seeing results, then it became kind of like the floodgates were open. Like, we'd have other people from the map cell, the missionary attack plan cell, whatever, come over to the little area that we were working in and try to, like, sell us, um, on their app as well. Like, hey, can you build one for me? Can you build one for me? Can you build one for me? And so that was kind of cool to see as people kind of realized how much it was impacting the tanker planners. And then the other people that were. That were getting apps was. Was definitely pretty cool because it really changed the way that the Tinker planners had to do their stuff. Instead of having to fight between a whiteboard Excel and then the actual weapon system that they were supposed to use, they instead had one tool that they could kind of plan everything out in. They still had to backport things in, especially in those first versions of the TinkerPlanner tool. But it simplified a lot of the way that that worked for them. And so they could make the plan both more efficient because they had more time to think about it. And so there were actual legitimate fuel savings that that came from fewer tankers needing to be put up in the air and a more efficient plan overall. And they were able to spend less time doing it. So we. I think they wound up being able to cut back the crew of tanker planners by. By one person. So uh, you know, a whole person not having to go out and deploy and do all that stuff. And it was basically a sell of about three people there. And so once people started seeing like some of those practical benefits too, less time and a little bit easier thing. Like deployments aren't easy. They put a lot of time pressure on people and all these things. 12 hour days, six and a half days a week for the most part. And so being able to, to benefit them and help them kind of work through that and make something that's more effective, more efficient was, was really cool.

Speaker B: Did you know that? Like, I guess at what point did you realize that your Air Force career was not normal?

Speaker A: Basically right away I was kind of told that DIU was going to ruin me. That was regularly said so. And I was the only second lieutenant that was there at diu. Everybody else was at least a captain. Most were majors and above. And so yeah, I was regularly told that it would ruin me. And it was weird and it was different. And sometimes you'd get those like little flashbacks of, of the regular Air Force with people being either trying to like kill what we were doing or do different things. One of the, one of the things that we wound up developing in was commercial BitBucket. I mean the, the stuff we were developing was all unclass software. There was nothing crazy or secret about it or anything. But we did start off in BitBucket.com uh, for our software repos because that was what was most convenient and that was what the contractor that we worked with used to used. And so even though that technically, like, because it was fully unclass stuff and we, we could technically use it and it was something that was provided to us technically by the contractor. And so there's the whole like NIST 800171 I think is what the, the actual document is. That's. But that's for fou or control unclassified information. But anyway, so even though we were technically in compliance with all those things, people still looked at that with a lot of side eye and different things. And there was one day where I still remember Colonel Odie coming in and saying, hey, I think we're going to need a GitLab instance for ourselves just to kind of like cover some things down a little bit more. And what was funny about that is that I had already kind of like heard some of the rumblings and already had a domain bot and a GitLab instance up and running and we were already migrating our repos over to it when he came in and Said that. So it was fun kind of being in a little bit of a band of renegades. But you did get that like kind of clamp down. This is not normal. Even though we're not like, we're not violating any rules or anything. Everything's fine. Everything's like fine by the letter of policy and especially like the policy that actually matters at ah, the, at the high level because DIU is really connected right there where especially at that time was right under the SEC DEF. And so we were following DoD policy like as much as we could, but there were other aspects of the Air Force that were definitely questioning what we were doing. And you'd get those little sniffs and whiffs of, of some of those antibodies coming through.

Speaker B: How do you deal with, I guess how do you, how do you push for progress in the government in that situation?

Speaker A: So one, the real way I, that I kind of did it throughout my career is really knowing your stuff. Like read the policy documents, go through and read them and figure out how they apply to what you're doing and if something is conflicting or weird or then try to resolve that conflict to try to figure out what's, what's going on. There's a ton of different, different things and you won't always win every single battle. You have to figure out which battles matter. But if uh, you know what NIST853 says FIPS199, FIPS200, FIPS171 or sorry, NIST800171 those will be. You'll be able to use those and apply them in a lot of different ways, especially if you can tie them directly to the technology and how that is actually implemented because a lot of people don't really know. Like I still remember the. One of the wing media policies that I was kind of reviewing and we were talking about how we would get updates into our systems. It referred to zip disks and all these other things that were kind of. You wouldn't actually. A zip disk.

Speaker B: Oh, a zip disk.

Speaker A: Iomega Zip disks. Yes, yes.

Speaker B: Did you feel like you were very well situated to know what a zip disk was given that you were a second had it?

Speaker A: Well, so my dad actually used gave us when we were kids, Zip disk that was viri in a bottle. So he had a program for Ms. DOS that would allow you to create viruses for dos and he put them on a zip disk and gave it to us when we were kids because we used to play with Ms. DOS computers when we were little. So I knew what it was. But the, I mean that probably hadn't been practically used in decades at this point. So. And then that doesn't necessarily even correlate with exactly like what, like from a philosophical standpoint, what is media? What is this concept of media? Obviously most people know it's DVDs, Blu Rays and various things, but really it's any representation of digital data. And that could be even a piece of paper. Right. Like, so when you're, when you're taking media into a skiff, technically you can take digital media on a piece of paper. And one of the things I did kind of to prove that and I didn't show, I didn't really show it off a ton, but to really just prove it is I printed, there's a, there's an application called Paperback that you can print digital files to. And I printed the MP3 of AMIA's paper airplanes, or paper planes, whatever, onto paper. And you could scan that back in and get the original MP3 back out because it's digital data basically in a giant QR code. But that kind of thing and that kind of conception of what actually what is media? What are these things? What is the, what are actually like the technological and policy implications of these, of these things and where does that meet and make sense together? And then if you can figure that out and go forward, usually you can kind of wind your way around a lot of these, these sort of antibodies. There are people that are going to try and stop it, but they're going to base it off of some policy that they maybe know. But if you know the policy better than them, then you can invalidate their argument.

Speaker B: So does was the final conclusion that you can or cannot bring the paper copy of paper planes into the skiff?

Speaker A: Technically you couldn't. Um, because it's media. So. Yes, because it could be turned back into a digital file and you could bit with any, with a scanner.

Speaker B: So do you think the people who were writing that policy thought that through, like, like really thought that far through that, that they would be banning essentially a bunch of paper?

Speaker A: No, no, the, the fact that they mentioned zip disks, I think they copied it from probably an old document and hadn't really revised it recently.

Speaker B: So I just think that's fun. I like that.

Speaker A: Yep.

Speaker B: So. So you, one of the things I, I know about you pivoting a little is that you are a giant open source software nerd. I'm curious if that started before or after you were working for diu.

Speaker A: Yeah. So that price really started in high school for me. So I got. So I obviously I had a lot of computers growing up. Uh, most of them were Windows 95, Windows 98, Windows 311 MS, DOS 622 kind of era machines and so played around a lot with Microsoft's products. At that time we didn't really have the Internet so much. We had dial up until basically until high school, until I was in high school. So didn't have a lot of exposure to those things. But when I was in the eighth grade, I worked for my dad for a summer. One of the, one of the goals there was for me to get a computer. He basically said, if you work for me, I'll kind of pay for half of your computer. Um, and so I got a new Dell thing system with Windows 7. It was a dual core Athlon, I think it was a 6,000 if I remember correctly. But so it was a nice computer that ran Windows Vista and had all that stuff. But I still had one of my old Gateways. It was like a Pentium 3, I think, 800 megahertz. And I wanted to figure out how to get it on the Internet because by that time I now had access to. We moved into town and had a cable subscription or whatever. So I wanted to get it online. And obviously I kind of knew that Windows 98 wasn't really the way to do that in 2007. So I started looking for alternatives and things I could run on it and then found Ubuntu out of that as a way as something to run on old computers. And so Ubuntu 7.10 was the first version of Linux that I wound up using. And then I kind of understood a little bit about what Open Source was at that point, because I was starting to get into a little bit of programming on the computer. But the thing that really kind of struck me back then was how many free applications you could, you could get. And the concept of the software center, because back then you still had like the software center and repositories and you could download applications and you could do it all for free. And that was really interesting. Playing games like Sauerbrotten or Super Tux Cart or some of those like Linux games that are out there and being able to just, just download them. Because that was even before, like, that was before really the iPhone and the App Store became a thing. So like this concept in the general population of having an app Store, a place you can find apps and then download them, was not a thing really. So that was just kind of Cool to experience that first on Linux. Um, and then from there I completely.

Speaker B: Computer games is why you like open source software?

Speaker A: Yes, yes. I still play some of those games to this day too. They're fun to go back to.

Speaker B: But do you still play Super Tux Cart?

Speaker A: Yes, sometimes.

Speaker B: Can you tell me what Super Tux Cart is? Is it like Mario Kart but with a little penguin?

Speaker A: It's Mario Kart with Tux and you've got like the BSD guy. And then I think there's. What is it? OpenBSD also has their little Pufferfish and a couple other GNU characters and stuff.

Speaker B: That is amazing. Can, can I ask you to sort of take, take your love of Open source and then dot dot, dot, many years to where you are right now and talk about like how you choose technology in your current life?

Speaker A: Yeah. One of the biggest, I guess, philosophies that I try to follow is that I want to control technology. I don't want technology to control me. And I think when you have low source software or even just bad terms of service or various things like that, then you're kind of ceding control of the technology around you to somebody else is effectively controlling you. When you're scrolling, if you scroll TikTok or Instagram or whatever, somebody is feeding that algorithm and is feeding you that content and various things. And so, and to some extent you're not in control of, of what's kind of happening potentially there. And the same thing for any sort of device. If it require, if it's closed source software, it could suddenly require a subscription at some point. It could be connected to the cloud and who knows what's happening with the data? Who knows where that's going. And so I try to pull as much back as I can to things that I can control and that I can monitor and use. And so I, and it's not that I don't do a lot of the things that maybe like people have like, people have like IoT devices in their homes. I have IoT devices in my home. It's just that the way that I selected those devices is I wanted ones that I could run custom firmware on. And so I run esphome on all of my light bulbs. They were originally actually Tuya, uh, devices. So they, they actually were like, that's a Chinese company that does OEM stuff for, for even American brands. And so you can actually though cut them away from Tuya services and reflash them with open source firmware that runs locally. And so trying to find ways to do that so that I can control the way that that works and usually also unlock more functionality than was originally there in the, in the original implementation of uh, said product. And so that's generally kind of how I approach things is what, what control might somebody have with this particular device? And then what can I do to claw back that control or what can I do to even exercise freedom in those respects? Another way that I wound up doing that was AT and T. They, I had that their fiber Internet service for a bit and they want to do their whole things. I read through their terms of service. One of the things in there was that they were going to give you an exclusive worldwide or a non exclusive worldwide irrevocable license or uh, they use the word write to all the data you use in their service. And so in order to sort of back away from, from that, even though I was still getting their Internet, one of the things I did was purchased my own modem from them or really from Amazon and then it's got it onto their, their service, which is not something you're normally supposed to do because it's an AT&T branded device and all kinds of things. But. And the installer was very confused when he saw that I already had a modem, a fiber modem. But that was something that I did mostly just to exercise the fact that like this is my device and if you start doing anything weird, like if they tried to collect WI FI information, air location information, which is something that Comcast has in their TOS in terms of service, that I could go into my device that I own and disable the WI FI antennas and kind of if I needed to. So it's, I try to look for ways to do a lot of those, those sorts of things and make sure that uh, that again I'm in control of the technology versus technology controlling me.

Speaker B: If you were giving advice to podcast listeners who maybe aren't quite ready to flash their light bulbs, for example, what's the 8020 on being more free or having more freedom over your software choices?

Speaker A: So I think it starts really with understanding like what are you giving away right now and what are you comfortable with giving away right now? Like if, especially for the, the big services, banking and email and stuff like that, that's, that could really ruin your life if it goes wrong. Read the terms of service and uh, are you comfortable with the terms of service and the privacy policies of those services? And then if you're not, then how can you mitigate that or back away from that? So that's probably the, the biggest thing because you can, you can decide to use these things and then if you're okay with them going offline or, or not working anymore or being locked out of it, then that's just a cost and expense that you've paid. If you bought into some sort of lighting company, they go out of business. It required a cloud connection. Those light bulbs no longer work. As long as you accepted that. Yeah, uh, this is part of the, what I'm accepting here and I'm okay throwing these bulbs away, then that's, that's fine. But just, just get more understanding of, of what might happen with those services.

Speaker B: Yeah. I had a fascinating conversation with a friend recently who had started using the Ask not to track on, um, their Apple iPhone, which is an option now, and were very disappointed by the low quality advertisements that they started getting and

Speaker A: turned it back on.

Speaker B: So to me, like, that's an informed choice. You've decided that you really liked it when they had all your data. So I guess that's the fair summary maybe of what you're saying. Not let them track you, but if you're going to let them track you, at least know what they're tracking and that you allowed it to happen.

Speaker A: Yep. And if you're okay with it, then you're okay with it. For me, I don't like the advertising because I find it to be a form of manipulation and so I just don't like it. But, but one of the, one of the things I actually look for is when I start getting political advertising from both sides, then it starts being like, aha, uh, they know nothing. Which is, which is good. But yeah, it's, it's different things for different folks and different people who have different priorities. So it all depends on what you ultimately want.

Speaker B: I, I'm not quite at the flashing my own light bulb stage yet, but I think I like there's a part of me that wants to be. That seems really fun.

Speaker A: It is fun. It is fun. Yeah. Because you can buy a thing for 10 bucks and then just be like, boom, and then it's mine. So.

Speaker B: And then you can go into like a light bulb manufacturing business if you wanted to, but that's probably a step too far.

Speaker A: Yeah, I could resell them on ebay, I guess.

Speaker B: But so, so coming back to DIU life full circle, back to where, where we were a few minutes ago. What. So you, you kind of had this very unusual Air Force experience. Did your Air Force experience ever get normal?

Speaker A: No, it did not. So after the Kessel Run stuff and all the stuff we did with Tanker Planner Tool, I went to Rogue Squadron, where we worked on small uas, counter UAS stuff, worked on a lot of different applications. And that was a lot of fun because that was kind of in the very early days of really the DoD thinking about small UAs on, uh, mend aerial systems. Because really, up to that point, you had kind of the RPA community and larger systems, but you didn't have things that people in the field could kind of like deploy and maybe do room clearing in a building kind of with or something like that. It just wasn't really there. And at the time, the leader was dgi, which is obviously a Chinese company. So that was fun to be able to work with that. That was kind of my first exposure to reverse engineering, which was a little bit fun. I didn't know what I was doing, but it was fun to experiment with and actually get some stuff working. And then, uh. But that was kind of, again, another renegade group of people just kind of building stuff in really what we called the cages, which is really a storage unit. And then from there I actually had orders to go to Kessel Run in Boston and actually be assigned there, but they wound up canceling those orders. My career field functional did, telling me that I needed to be a real cyber officer in the Air Force. I learned later that that may have not entirely been the career field functionals doing, and that there may have been some strings pulled behind the scenes because when I actually arrived to San Antonio, I basically learned that instead of going to the unit I was supposed to go to, I was going to go work downtown with the airport. Air Force Chief Software Officer Nick Shalon's thing at the. At the time it's called Level Up. It's still. Yeah. And it kind of split into Platform one, but that I was going to work there for six months instead of actually working on, um, what I was assigned to do. And so kind of continued to stay in that, like, software factory world, really, even after leaving, quote, unquote, try the career field. Trying to make me leave it, I guess, at that point.

Speaker B: So basically, they ruined you, and then they continued to ruin you for the

Speaker A: rest of your career.

Speaker B: For the rest of it.

Speaker A: Yep.

Speaker B: At some point you left the Air Force, presumably. Let's surprise everyone with where you went next.

Speaker A: Defense Unicorns.

Speaker B: Shocking. Why I'm curious why you picked. What made you want to go to Defense Unicorns.

Speaker A: Yeah. Uh, so I knew. I've known Jeff McCoy, who's our current CTO, for a while. I actually technically originally hired him when we were working with Kessel Run, so. And what's kind of funny about that is we actually hired him as a UX designer because, uh, during the interview he talked about how much he likes to make UIs and stuff and make people happy. And he didn't talk much about his technical ability. So we didn't actually know how good of a programmer he was. But. But which turns out he's. He actually is a very good programmer and he's, uh, an okay UX designer, but also a fan of Times New Roman. So that's his. His thing, I guess. But yeah, but, uh. But yeah. So I. I've known him basically ever since then, had a lot of fun working with him on a lot of these different projects over the years. And when he left the Air Force and, and kind of helped to find. To found Defense Unicorns, he'd been kind of talking to me about it for about a year or so maybe before I actually joined the company. And over time in the Air Force, especially once I became a captain, it got pushed a lot more into kind of the political and ego battle kind of space where I just didn't really want to be where you have to argue with people that sometimes don't know what they're talking about or are just not technical or whatever. Um, and one of the things that. One of the stories that I still remember kind of from the 90th was the headset battle with wing security. And so we. Which you probably remember from being there, uh, very well. Yes. And so that the headset battle. Yes, the headset battle was, was basically during COVID Obviously lots of people went home, but we worked in a skiff, so we still had some people in actually in the office still. So we still had to have that. But people were. Because half the. About half the folks were out. We had a lot of people on teams calls all day. And so you'd have people basically with phone neck just kind of like resting their ear just constantly. On teams calls. On teams calls, on team calls. And so the unit, wanting to care for its people and try to provide a better solution, decided to buy headsets. And the phones that we were using were like, really, really nice phones. Our, uh, civilian, Garrett Heaton, could explain a lot more about exactly how they were certified and everything, but they were certified to the, to a higher level than what we actually needed in the area that they were operating in. And the headsets we bought, they were not. They were not like USB headsets or anything. They were Just using the standard RJ11 plug that. The actual handset, the thing that you hold in your hand that. That uses. And so electrically they're the same. Right. Like, if you're looking at them, there's a speaker and a microphone connected to an RJ11 plug that plugs into the back of the phone. Like, the handset and the headset are the same. And so we thought, okay, that we're. We. We know the regs very well. We know what these phones are certified for. We're exceeding the regs. We're not just meeting them, and we're. We're just gonna buy this for. For our people. Uh, uh, we wound up doing that, and then we wound up distributing them. And then at some point, I think it was about, like, maybe it was only a couple days, I think later, the wing realized that we had headsets and then basically freaked out just at the word headsets. Didn't look at what the headsets were or how they're being used or anything, and then basically took all of the headsets away. And then that started a battle with the wing commander and the wing and everybody. And eventually we did get the headsets back in that initial battle, but then wing leadership flipped, and then wing security decided to come back and take them away again, and then they were gone basically forever at that point. I mean, at least until I left. I don't know if they. They may have gotten them back at some point, but I never saw them come back.

Speaker B: One satisfying conclusion for listeners is that, yes, we got them back, and as far as I know, they're still there.

Speaker A: Okay, good, good, good, good. Yeah, but for me, I never saw. I mean, that. And this is years. Like, this is literally years. So it was just. It was kind of demoralizing in some ways that that happened and that people just sort of applied the policy in that way. And we used to joke that we should get headbands with the 90th logo on them. And we should have done that. It would have been awesome just to put the handset underneath the headband and then have basically, uh, a headset at that point, because that's what it is.

Speaker B: We joked with getting those with Wing securities logo on them too.

Speaker A: Yes, yes. Have food to thank for those. Yes. But it's that, uh. It's just that kind of thing where you're. You're fighting with these people that don't really know what they're talking about a lot of the time. And I do sympathize somewhat with the wing level in The Air Force, you have kind of a lot of hierarchy, obviously. And so when you're at that level, you're not quite at the level where you can actually make policy that's usually matchcom or even higher, that actually sets policy. You're also not at the level where you actually implement policy that's usually squadron level or lower. So you're kind of stuck in the middle of just trying to figure out what's, what's happening within your units, but then also, like, how is. What is policy and what's going on? Because. Because you're also going to be the one that gets squeezed when a policy violation happens, especially if it happens across many units in your wing. And so you're kind of in a precarious situation where you can't set policy, really. You can kind of do that, but, you know, most of it's set for you and you can't implement policy. You just have to kind of monitor it. And then there's a. There's big consequences for you to get that wrong. And if you. So if you don't understand a lot of the technologies, then the easy answer is no.

Speaker B: So there's, um, there's really no benefit to you either at the wing level for saying yes in those situations. So I'm with you. I actually also empathize a lot with that. Uh, there was a ton of high level oversight on headsets in general. The, the mistake we really made, I think, as a squadron. And if I could go back and do it differently, I'd. I'd call it like alternate handheld device. And people be like, I don't know what that is, but go ahead, like, do whatever. But when we use the word headset, they're like, oh, no, like, we've banned those.

Speaker A: Right, right, exactly.

Speaker B: So. So the headsets were one of your final straws and you, you came over to Defense Unicorns to, to build fun stuff. One of the things we haven't done on the podcast before is really talk about, like, the underlying open source software of the company. And so one of the things I'd love to chat about with you is the first tool you worked on when you came over here is Zarf, which we just donated to the open SSF and has gotten tons of adoption. I'd love to hear what that was like in its early days and what you love about it and maybe what you hate about it. If you really want to get spicy.

Speaker A: Yeah. Yeah. So in 2022, I joined the Zark team, really, as just an individual contributor kind of doing PRs and managing some of the different things that we had coming up there. And then in 2023 I uh, took over as the tech lead when Jeff McCoy, who was leading the project previously, moved on to some other, to some other projects that we, that we've developed. And so yeah, Zarf originally really started it as a way to run K3s in the air gap. It, it actually originally didn't support other Kubernetes distributions. It was just K3s when it first started and it was just to kind of run it on relatively small deployments in the edge on little tiny things. And so I was able to kind of see it technically a little bit after that. It had just, when I joined the team, it had just gotten multi, what we called multi distro support. Although that was still a little bit rocky in the early days of Zarf just because Containerd and Cryo and some of those, the container runtimes were changing around a little bit of stuff. And so we had like issues running in Azure for a little bit because of the way the Container D worked there and a couple other things. But we worked through that over time and I was able to kind of see it grow from, from that little project that was really designed for that little thing to adding in variables and component composability to be able to build larger XARF packages and then take Zarf packages and how do we actually orchestrate them kind of together with other things. And there was a lot of design decisions that happened through all of that. Even things like should we implement Terraform in Zarf and all these other um, interesting things that, that we've kind of experimented with over the years. But it's been fun to kind of see it turn from this little tool that just runs these little clusters to something that you can package up just about any cloud native application now, as long as it's helm based for the most part. I mean Zarf is still largely based around Helm, but you know, you can package it up and then you can deliver it to really any Kubernetes cluster at cloud scale if you, if you need to, or still in the small, little, little deployments as well, wherever you need it. So that's been cool to see. And we've also as a company built a lot more of our own XARF packages over time. Back then we had the concept of, well, UDS didn't exist. Uh, and then the concept like UDS packages and like the UDS registry and us having our own packages and our own little Like App Store thing, none of that had really existed yet. So Zarf got a lot of refinements from a lot of actual use with, with those sorts of concepts coming, coming online. So. But yeah, it's been cool to see that and it's been cool to see the community grow as well. I uh, think I had joined just after Zarf got its first external contributor. So that had done the Git stuff, the a lot of the um, Git repository mirroring functionality in Zarf. But since then I've had like Danny Gershman from Radius Method. He added the whole Argo CD support thing. That's not something we use at Defense Unicorns like at all really. But it was something that he had used and him and his team have contributed a lot over, over the years now now they're like co maintainers under open SSF with UVZarf and just sort of seeing those relationships grow and expand has been really interesting to see from just this thing that really one person and then which is Jeff McCoy and then two people which is John Perry and Jeff McCoy and then me being the third kind of there. It was yeah, kind of cool to see that grow.

Speaker B: How do you build like a community around a tool? Is it more about advocating for the community or is it more about just building a good tool that other people need?

Speaker A: So I think it's more about like building a good tool that people need. I would say though that in that you need to listen and sort of figure out what's what the community is wanting and then kind of solicit feedback and try to figure out what's what. And there's there's a couple different things that, that you can do with just simply kind of running the project, chopping logs and carrying water to try to sort of entice it and make it easy for a community to form around a project. We. And uh, there's a couple things that I'm kind of new to running open source stuff. So a lot of what I learned was from a lot of like the Mozilla has a couple documentation or a lot of documentation about how, what makes a good community and how to kind of like build that and a lot of what they have kind of focused on and some of their things uh, is about like making sure that PRs are reviewed quickly, especially from external contributors. Because if you review it within 48 hours then it's more likely to be to. They're, they are more likely to be a return contributor than if it's something like I think it's like seven days. If they don't get a review then that drops off completely and they're not going to contribute back again. And so trying to kind of run a tight ship around that stuff, make sure that the community is involved and understands the direction of what, where's our uh, where things are going and what, what's. What's going on is very important. One of the mistakes that we made early on was sort of trying to be well, uh, several I would say don't try to be special. There's several of our mistakes that, that I could point to where we tried to be special which is sort of part of the brand of defense unicorns. But we were. Our first few community meetups were run in gather which is not what the rest of the Kubernetes community uses for doing any sort of community meetups. So switching to Zoom under the Linux foundation is a lot better because like people are familiar with that. Using Kubernetes Slack more is because people are familiar with that. All those things are a lot better to, for, for the kind of community to understand. The other thing that is interesting as well is kind of like the. How you name things even in tech in the tech in the technology is interesting too. One of the common features of Zarf that we get a ton of questions about and still is kind of like to this day a mistake in some ways, in some ways just because of what its name is is the Skeleton Zarf package. Basically that was a concept of a uh, partially created Zarf package that you could import into other Zarf packages. But people struggled with, with that concept and kind of almost also struggled with the name Skeleton. One of the things I think I kind of realized is that like that while it's culturally relevant kind of to like us to other people around the world, they might not kind of interpret things in that same way. And it's also not a word that's used in technology a lot in other contexts. And so there's it's. It can be difficult for people without context for what we did and why we built the skeletons ARF package to understand what a skeletons are. Package is just from those words. And so being less special when you're trying to build a community project can actually help you because more people will understand what you're talking about when you, when you say things.

Speaker B: That makes a lot of sense. What are you most excited about? The direction that Zarf is headed right now?

Speaker A: Yeah, so I uh, think I'm most excited probably about the, the feedback that we're Gonna, we're trying to get into the baseline from um, some of the UES work that we've done. So we have kind of the UDS CLI that we use internally, which is also an open source project, but we're going to try to take a lot of those features and pull them from UDS CLI and put them back into Zarf 1. So it's, the features are more in a neutral spot anyway and it's a better, healthier spot for things longer term, but it also enables a lot more things for the community. I think through that we'll see how the XARF enhancement proposed proposals actually work out. But through that we're going to try to lean a little bit more into HELM isms. It's kind of like the way HELM does things. And a lot of that again, kind of goes back to the, like do what people understand. A lot of people coming to ZARF are coming from helm. And so if people are familiar with Hellemisms, if we implement those into ZARF and kind of have a familiar interface there to, to those features, then that will, that will be easier for them to pick up and easier for the community to adopt. So, so that's probably what I'm most excited about with Zarf and some of the future there.

Speaker B: One of the questions I get about Zarf a lot is why we're not at 1.0 yet.

Speaker A: Yes. So that's been a debate for a while. The, there's uh, a couple like, kind of, I guess philosophical questions to that is. One is like why, why does 1.0 kind of matter in the first place? Like what is, what does 1.0 mean in this, in the sense of that? And to me it's really like, in my opinion it's a stability promise for the project. So you're saying like, we're not going to change anything from here on out. And this is how this is. This interface is now stable interface and things are, things are not going to change. And that's something that we haven't been fully comfortable with or the team rather, because it's not really just defense unicorns anymore, but the team hasn't been as comfortable with yet just because there's a kind of a lot of unknowns in the space. Zarf is pretty mature as a product like we used to use in many different production deployments and various things. But we do still think that the schema is going to change and shift around a little bit and there's some things that, that may get deprecated and may happen along the way. And that's not to say that you can't use it in production, but it's more that saying that you need to be willing if you're going to adopt Zarf, to come with us along this journey as things get changed and contribute back and all these different things, rather than kind of expecting that you can take it off the shelf and use it for the next 10 years, which is more of what a 1.0 would, would mean. And so that's kind of largely why Zarf hasn't. Zarp has tried twice to go kind of to outline a 1.0 roadmap, but largely has, has determined that there's still a lot of things that Zarf wants to experiment with and that Zarf team has wanted to experiment with and that they're not quite ready to, to make that really what would probably be a 5 to 10 year like maintenance life cycle promise.

Speaker B: Oh, that makes a lot of sense. What question do you wish I would have asked you that? I haven't asked you yet.

Speaker A: Um, I'm not sure. Well, you haven't asked about any of our other open source projects like Pepper.

Speaker B: What other open source products do we have that you love or. Well, that's not a fair question because I know you love them all.

Speaker A: Yeah, but yeah, there are quite a few that we have and I think Zarf gets a lot of attention and love. But there are others that are, that are pretty important. Pepper is one obviously that we use a lot within UDS core to provide really the UDS operator functionality which allows you to really tailor a package to an environment without needing to have that package be manually configured for that environment. So you can have things like SSO clients or domain names or whatever be created by that operator and it allows you to build any other operator as well in a very kind of like fluent syntax. So it's very easy to look at it and sort of. It basically reads well in English when you actually read the actual syntax. So it'll be like when a POD is updated, then do this kind of thing. And so it's an easy way to build operators and actually build in that functionality. It also embraces TypeScript, which is interesting for the Kubernetes community, which is largely GO based. But I think that's also an interesting opportunity too because there's a lot of people that are familiar with TypeScript and maybe not go, but they can if they needed say like a front end dev, uh, that was maybe doing like a, I don't know, nest JS React app and they needed some sort of Kubernetes operator. They can very easily adopt Pepper because it's in their ecosystem. And so that's kind of an interesting thing that one enables a lot of things for us internally, but as an open source product could provide a lot of value to people out there that especially are maybe underserved, uh, if to use that word by the Kubernetes and kind of go community as right now. And so that's, that's a cool one. Lula I think is also an interesting one and that's something that actually I kind of have some exposure to. Back in the Rogue Squadron days we, There was a thing, there was a project from 18F that used to be called Open Control and it sort of. That was the predecessor to oscal or the Open Security Control Assessment language that NIST currently has and that LULA uses. And there was a lot of stuff that I saw kind of as a developer back then that would have been really interesting and that isn't really fully solved by ATOs today. Being able to tie the security control to an actual implementation kind of in code is very interesting. One of the things that I thought about back in the Open Control days that you could, should be able to do with Lula and other things would be if you had a code annotation above a line above a function that say, said this particular function implements AC3 or whatever, because this is the uh, code that integrates with Keycloak. You could actually like generate not only the actual RMF documentation, but you could actually tie that to code coverage for that function and that actual line of code and actually see that all the way through that was implemented in whatever application that you, that you had and it would provide a lot more transparency. Now whether as a developer that sounds really cool. I know the security folks have kind of a ways to go in terms of like uh, embracing some of that, or understanding rather some of that radical transparency. But that potential to sort of get there and actually point to precisely like this control is implemented exactly by these components, uh, in these functions. And this is exactly how it's working I think is a very potentially powerful thing thing to implement in the future. And to the extent that OSCAL could, could maybe let us get there, I think that's very, just a very interesting thing because we're still kind of struggling even with a layer above that. Where is this? We might bring in Keycloak with UDS core. But then how is it, how important is it that all of Your apps also integrate with Keycloak and is that control truly implemented by Keycloak or is it implemented by a combination of Keycloak and the integration that those apps have with Keycloak? And so there's, there's nuance there that we're starting to like kind of still scratch the surface with. But I think if we can keep going further and further and if Oscar can keep picking at that, that itch, I guess that'll be really, really cool for the security community, I think.

Speaker B: So I've got a just short like lightning round of questions to ask you. Hopefully fun, hopefully quick answers if any of them cause philosophy that's fine too but we'll, we'll just, I'll. I'll start it off and we'll see where it goes. We'll start with Other than defense unicorns tools, what is your favorite open source tool?

Speaker A: Probably GNOME M, it's the desktop environment that I currently use. And so it's, it's probably one of the things that I wind up using. Like it's not uh, maybe it's not a tool depending on how you define tool. And that is also a fun philosophical conversation to get into about what is a tool. But uh, but probably Gnome just because it's very. I, I do like a lot of its philosophies around how you interact with your desktop. It's very kind of like single paned in many ways, which is kind of an issue that I've had with Mac OS and Windows where you have like things sliding out from all the different corners of the screen. And things work differently in different modes. Like if you're in the little dock at the bottom will be different if you're on the desktop versus in Mission Control versus in these other pages on macOS. And to me that inconsistency has been kind of. It gets frustrating sometimes just because I don't. I want the computer to do the thing and be it be consistent and smooth. And even though GNOME has it gets criticism from other Linux users for being maybe overly simplistic sometimes. But I do enjoy that simplicity and the, the ability to just have everything right there. When you go to the. Even on Windows I still like slide my mouse up into the upper right corner or upper left corner sometimes expecting it to like, go to the little like Gnome, um, window screen that shows everything. It doesn't but, but even from that window like you can see all your windows, you can see your dock and you can start typing to open new applications. And like so Everything's kind of right there for you and you can kind of get to things very, very quickly. But that's also because I'm a damn dirty mouse user as well. And so I know a lot of other Linux people would be like, well, why not use i3 or some of these other tiling window managers? And it's just I, I like the mouse. I don't know.

Speaker B: Yeah, that's fascinating.

Speaker A: Yep.

Speaker B: Is it gnome? Like G N O M E?

Speaker A: Yes.

Speaker B: Awesome.

Speaker A: Yep. Gnome. Yep.

Speaker B: That sounds very cool.

Speaker A: Yep.

Speaker B: Next question. When you are like actually sitting and writing code, like in the zone focused, what are you listening to if anything?

Speaker A: Lots of stuff. Usually random electronic things, usually meme music, just, I don't know, random parodies of things that are kind of in the background. Like, oh, uh, what is it? Dog of Wisdom is a meme. But there's a song on Spotify that's called Dog of Wisdom that's like the, I think it's like the funk version of it. But yeah, so stuff like that, that's just kind of like weird electronics stuff but that has like some sort of usually faster beats and then it's just silly.

Speaker B: So I love it. What advice would you give second Lieutenant Wayne about what to do differently in life, if anything?

Speaker A: What to do differently in life? I think one of the things I probably could have done is I, I guess one kind of get out there a little, a little bit more too. I don't know, I kind of view my. And this is something I'm trying to kind of do a little bit more now. But uh, it's to sort of share certain things because one of the things that I'll, I'll do is like hide it in the corner and just be like, well, I'm just over here doing stuff, working away or whatever. And I don't like self promotion at all. But trying to find ways to do that more I think would be useful, not necessarily for self promotion, but just share things publicly. One of the ways I've started to try to do that now is to. Instead of just doing self promotion, like making posts on LinkedIn, but ones that celebrate other people. So specifically mentioning other people and sharing, sharing some of the cool fun things that, that I'm m working on and other people that I work with are working on or that are kind of around me and stuff. So I think that's a big thing because one of the things that is that like you can't really um. If you. Even though you might have unique ideas and you might do Unique things with those ideas and stuff. If you can't, you can have much more impact if you actually shared those ideas. Now, not everybody's going to listen to you. Not everybody's going to care. Not everybody's going to do anything with that. But. But there might be some people that will, and they might be inspired, uh, to do their own thing or they might tweak it to do their. Whatever they want to do. So putting some of those ideas, projects, things that I want to work on and do more out into the open, I think that's. That's one of the big things I would have told my former self.

Speaker B: I think that's great advice. Last question. If you could make most of the world, or maybe we'll just do most of the company. Read a book. What book do you wish everyone would read? Read a book, like, is there like a book that you've read that has improved your life significantly? Maybe that's a better way to ask it.

Speaker A: I do like Zen and arc motorcycle maintenance. I do think that the argument of quality in there is. Is an interesting one. And I think it's one that people don't think about a lot of what makes something quality. The way that that book is written is kind of interesting in and of itself being a chautauqua, but which is kind of written from the perspective of inside of one person's mind is what that means. But, uh, what is quality is a very interesting thing. Uh, one of the things I value a lot is kind of the concept of craftsmanship. Just because it's something that you can. In engineering, you have to kind of put yourself into the thing that you're making a lot and sort of really think about how this thing is structured and what you're. What you're doing with it and sort of learning and having a better relationship and understanding of quality and of what that kind of like looks like, I think is an. Is an interesting philosophical conversation in engineering to have.

Speaker B: I don't have any other questions for you, but I really appreciate your time. I'm super excited to bring your perspective to the Defense Unicorns podcast. And thank you. Thanks so much. I appreciate it.

Speaker A: Yeah, no problem. Thanks for having me.

Speaker B: Thank you for listening to this episode of the Defense Unicorns podcast. If you enjoyed the show, please share it with your network and rate the podcast five stars. Wherever you listen, tune in next time for the latest in DevSecOps.