Breaking the Myths of DevSecOps

Speaker A: M. I write software today for Defense Unicorns and I guess who's responsible for the fort when it breaks? Me. So I want to make sure it doesn't break. I have incentives to make sure it's uh, easy to modify, easy to fix. You know, it's all about incentives. How are you incentivizing quality?

Speaker B: Welcome to the Defense Unicorns podcast. I'm your host, Rebecca Lively. Today we're joined by Brian Finster, distinguished engineer here at Defense Unicorns and and self described DevOps insurgent. Brian prides himself on continuously developing his craft and inspiring others to execute continuous delivery. I'm super excited to have you on. I know when I first started looking to join Defense Unicorns, you were one of the people whose voices I saw very prominently. And I'd love to kind of get your take on all things dev sec ops, if you don't mind. Start with like a super brief introduction of like, who you are and, and why, ideally why you're excited to be on the podcast and talk, uh, to our listeners.

Speaker A: Yeah, so I've been a software Engineer for almost 30 years now, mostly working on high availability systems for large enterprises in the supply chain space. Um, in fact I spent about 20 years in supply chain before moving to platform at Walmart, which from my perspective is just the supply chain of software. It's like, how do we smooth out the, uh, supply chain of information and deliver things more rapidly safely? And you know, the thing I really focus on is how do we improve the, not only the quality of our outcomes of what we're delivering, but the quality of life of the people who are doing it. Which is why I focus so heavily on DevOps and on continuous delivery. Because those things, you know, they have to go together. You're not going to do one without the other. And the outcomes are always better if you take those things seriously and really understand what they are. Dig in, um, and do the hard work to get there.

Speaker B: So you mentioned both rapidly and safely. I think it's a common, I'll just say a common perception that you can't have both.

Speaker A: Yeah.

Speaker B: So how do you do both? Can you do both?

Speaker A: Well, I think people misunderstand rapidly. Number one, there's a, there's two aspects of this. I think people need to understand. This is not just, oh, we need to deliver a whole bunch of features really fast. That's not a goal. The goal we have is, number one, I need to be able to respond to the realities of what's happening in production as quickly as possible safely. I don't want to be making up ways to make change. I, uh, don't want to be cowboying change in when something's going wrong in production, either with security breach or functional problems or whatever it is. The then I don't want to be throwing gasoline on the fire, uh, at 3:00 in the morning. I need to be able to recover from what's currently occurring as quickly and safely as possible. So operational responsiveness is key. The other part is if I'm building something new. Software development is not the same as building a car, unless you think of it as designing the car we're going to build. We're prototyping everything all the time. And the uh, bigger the thing is that we deliver, the more wrong is in that prototype. And so it's not about speed, it's about feedback. And people talk about quality in this process and they only think about, we've written a spec and now we're going to test, see how closely it matches the spec without thinking, hey, could be that our idea behind the spec was wrong. And how do we test the idea? The way we test the idea is trying to get to the core of the idea, deliver that core, get feedback and find out if that's the right idea and then go back and adjust our idea. And so we're always running experiments. We're not building something we know is good. We're doing hypothesis driven development. I believe that if we deliver this, we should see this kind of improvement measured this way. And so that's it's not so much speed as shorter iteration, smaller things delivered more frequently to get faster feedback.

Speaker B: And when you say feedback, like, who do you have in mind as getting feedback from?

Speaker A: Well, I mean, the best case. So the first time I got to operate using CD for real, where we were delivering multiple times a day was I had.

Speaker B: And just to. Just to make sure that everyone's following along. When you talk about cd, you're talking about continuous delivery.

Speaker A: Yeah, continuous delivery. Right. We were delivering to production in a distribution center multiple times a day. I was getting feedback on slack from the people doing the work in the DC because we were building out a new thing and say, hey, how about this? What do you think about this? Can you give us some feedback? And it wasn't about defects, it was about, hey, what if you tweaked it slightly this way it would be less work for me. Right. That's the best possible kind of feedback. But the, you know, anything that I can do to get some idea about how far off the Mark, we are about the, you know, the solution we have to the problem in front of us. You know, how close can I get to that? You know, I, uh, need some sort of subject, uh, matter expert, someone who understands the work, not just somebody who's tasked to test. I need somebody who understands the problem we're trying to solve and can get, give us feedback about how close our solution is to solving that problem.

Speaker B: One of the things I found so frustrating when I was in the government about how we implemented Agile M was it was more about meeting certain goals of delivery timelines and less about the why behind it. So I know one of the most frustrating experiences I had was learning that a team was delivering every two weeks. And I asked them who they were delivering to and they kind of got quiet and they said, well, we do our Sprint review and we demo at the Sprint review. And I was like, great. Who comes to the Sprint review? And long story short, no one came to the Sprint review except for the team and they were delivering to their internal repo. So, like, how do you combat that sort of, I don't know, I call it Agile theater.

Speaker A: By trying to educate people on, um, what the actual problem we're trying to solve is. I mean, the problem we're trying to solve is we lack certainty that we're solving the right problem. You know, this is like using. Was it earned? Oh, crap. What's the method they use to control, uh, I'm blanking on the, on the acronym where you're saying, okay, you have this much work to do and you've delivered this percentage of the work, and so you're going to get paid this much. We're not building tanks. These aren't M4s where we know exactly what it is. You've ordered 100,000 M4s and we've delivered 20,000 M4s. Right. This is, we're inventing a new thing. You don't know how big it is. You have no idea how big it is. You just have some ideas of how big it might be. And so trying to get people to really understand that we're inventing new capabilities. Where's the Gantt chart for Edison? When he was building the light bulb, he had to go and invent how to, you know, how to build an operational light bulb, what that looked like.

Speaker B: I think Edison's such a great example too, because he did not, he did not just sit and draw light bulbs for six months and then build a light bulb that worked the very first time. Yeah, and I think sometimes that's the other perception I've seen, which is this idea that we will spend 18 months designing something that then we can build in just six months and we don't even have to hire any software developers until we've done that 18 month design phase.

Speaker A: Yeah, I mean, going back to my supply chain background, the one thing that I know from 30 years of software development is your ideas age like milk, not like wine. And so the longer that you spend polishing your idea, the more spoilage you're going to get, you know, and so you've got this big idea and it's just like piling up pallets full of milk in the warehouse until just the exact right customer comes and says I want it. And then all you have is spoiled milk. These ideas age and I wrote a blog post a while back about the three rungs which I try to capture this stuff. I'm always trying to find different ways to communicate. This core idea about why you're doing agile wrong is that the requirements are wrong. We've talked about that. Or I'm going to misinterpret what you asked for during implementation or the actual need changes by the time we deliver. And it could be that the actual need changed or when you saw that what you asked for didn't actually meet your need, you went oops. And that's when we started doing editing. Right? It's like, oh, I thought this would solve my need, but it actually doesn't. But if we tweaked it this way. Well, no, I'm sorry, you need to write another contract. We need to start on the project in no, we deliver, we find out and we really focus on how do we make the minimum amount of change required to to have a something that's fit for purpose. We just really have to get feedback on fit for purpose.

Speaker B: How do you do that in a world where the contract is king? Like I know the Agile principle is I think customer collaboration over contract negotiation, but I feel like the government way is often the contract rules. And honestly, like I've seen some cynical views that actually some of the big contractors love it when you change, quote, change the requirements because it means they get to build more. Do you have any insights on how to do that in. In an environment where for better or for worse, like the contract really is

Speaker A: king, you need to find creative ways to work around the contract or educate people how to write better contracts. I mean really the contract. This is one of the things that I, when I joined Defense Unicorns, I started talking to program officers about some of the challenges they were having. I was talking to one guy and he was explaining to me the problems he had and I said, uh, ah, well, Congress is your root cause. I said, you've got a contract for building something over here and then we can't possibly let the people who build it operate it because conflict of interest or something. So we have another contract for somebody else over here to operate it, which means it's the, the builders have absolutely zero incentive to build in a sustainable way where it's operational, which is why you're seeing this problem here. I said that, you know, we've got to figure out a way to work around this contracting problem just so you can have something operational that won't just age out in five years that you have to start over from scratch and spend several million more dollars to rewrite. You know, we need to um. But ultimately it comes down to educating on acquisitions about how do we improve how contracts are written. So we're building things that we can sustainably operate, that we can continue to modernize over time as things change. Instead of just throw it away, start over with something new that has a whole new set of bugs we have to work through and so something battle tested.

Speaker B: So if you were writing a contract, what outcomes would you prioritize? Because the way I've seen it done when people write quote agile contracts is they emphasize delivery timelines, sprint cycles, sometimes Scrum or SAFE as like a methodology. Is that the right way to do it? Is there a better way?

Speaker A: Well, number one, I'm a notorious safecracker. If you ever want to talk about why I hate safe, I have many reasons. SAFE is terrible, so it needs to become a partnership where. So number one, I'm not a fan of turnkey. I'm going to write some requirements and hand it off to a vendor to go do that. The most successful partnerships I've ever seen was when we had contractors embedded in our teams. They were team members. I stood up on stage one time and somebody asked me how many contractors we had at Walmart. I said it doesn't matter. I said, you either treat them like team members and get rid of them because otherwise you're not going to get the outcomes you want that we are partners in this. And so how do you build right a contract around that? Now if you're just going to go turn to it the, you know, at minimum I'm going to want some sort of iteration on. I'm actually getting feedback. I'm seeing something move and this is the same Sort of interaction I. The business would expect from me at an enterprise is they don't want to see the finished thing they think they do until they wise up. What they want to see is motion. They want to see things changing over time and be able to give input back. I know, because I've changed those minds, because I used to think they wanted the real thing. But when I demonstrated to them, we can show you tiny steps forward and get feedback, they became addicted to that. They want that. They want to provide that feedback, at least if they're competent. If they're not, we should probably get rid of them.

Speaker B: So that's fair. You mentioned. Oh, go ahead.

Speaker A: No, I was just going to say that, you know, it's got to. The contract's got to focus on some sort of partnership and feedback loops. It's all about feedback. We can't get good outcomes if we just wait until the end.

Speaker B: That feedback can be scary for smart people. Tell me about how do you receive feedback that basically everything you thought you were building was wrong and, you know, either scrap it or completely change it.

Speaker A: I've been doing this for a long time, and I used to feel that way. I used to get really annoyed when I deliver something I was super proud of. And they'd say, that's terrible. Um, and the thing that helped me was I finally realized that everything I'm doing is probably wrong. And then I'm like, you know, but if I can become less wrong instead of trying to be right and tying my ego to what I did was right, if I tie my ego into becoming less wrong in the outcomes instead of what I've done, that massively accelerated my ability to learn. Because you're also. You're killing your ability to learn. Because if I'm right, I don't have anything to learn. You know, if I'm just becoming less wrong, I'm always, I've got to learn something. And it just. Then you start craving feedback. Especially when, you know, when we were doing. Because all this happened around the same time we were running the first continuous delivery pilot at Walmart where I went, oh, duh, I'm always wrong. Because I'm getting feedback all the time about how wrong I've been. And it's just, oh, I just internalized. I'm always wrong. But you start delivering stuff and getting feedback on that, it starts building a partnership where you start craving that feedback from your partner. And you'll see me all the time internally and externally. When somebody will criticize something I do, when I ask for it's like, hey, here's the thing, and they criticize it. They say, hey, you misspelled a word or this is wrong. It's like, hey, code review is love that you're giving me feedback and I appreciate that feedback as long as you're correct. If you're just being critical for the sake of criticism, we're probably going to have another discussion. But I crave feedback on everything I do.

Speaker B: One of the things I remember asking you, oh gosh, probably like almost a year ago now, was if you want feedback, how come you come across super salty on social media specifically? And I think I thought I was asking you something profound, but it was clearly like to me something you'd considered before. So I'd love it if you share like why you're okay being salty and also still really wanting that feedback.

Speaker A: So, you know, it depends on the kind of conversation I'm having. You know, if we're having a conversation or disagreeing over things where it's just difference of opinion on implementation, we're gonna have a conversation and try to figure out where that disagreement's coming in what you're seeing. Because I understand I haven't seen every context that exists. I've seen a lot of context, but there's a reason you're saying that. And I've had intense disagreements with people which have resulted in conversations where we both got to with taken off in a dm, um, honestly, where we both got to learn from each other about why we were saying the things we were saying and how we could apply those to help each other.

Speaker B: Right?

Speaker A: But then if I also see people pushing ideas either completely outdated or just flat out have never been good, ideas that I know from experience objectively harm teams and outcomes. And when I see people pushing ideas that they read in a book one time or took it a certification class that harm outcomes less than safer are good examples. So whatever. I forget what less stands for now, but these agile scaling frameworks where they have some things that are good, but they have some things that are just harmful and when people push those ideas, and I know, I've seen what it does to crush teams, I've seen high performing teams destroyed by these ideas is then I get irritated because you're harming my mission, which is to help every developer live a better life while we deliver better outcomes. And that's a personal mission I have that exceeds any job I own. M I'm on.

Speaker B: So LESS is large scale Scrum that you.

Speaker A: Large scale Scrum? Yeah.

Speaker B: I'm curious to like so. And this is a question I've gotten before because I've, I have railed against SAFE in the government. And the question I get back in return is how do I manage these complex interconnected systems if I don't have something like SAFE to tell me how to do it and how to manage interdependencies and shared resources and all of those things?

Speaker A: Yeah, you use engineering because I'm not even being flipped. I wrote an article for InfoQ called Agile Rehab where I talk about replacing process dogma with engineering based off of uh, what we did at Walmart to burn SAFE to the ground in our area. And we had this giant complex system with all these dependencies that were trying, people were trying to manage. Well, we changed our team structure and started changing the application architecture. Well, we leveraged Conway's law, started using a reverse Conway maneuver, though we didn't know it was called that at the time, to architect teams around our desired system architecture around communication paths so that we could decouple those teams and turn all the hard dependencies into soft dependencies where we could manage those through contracts and engineering and coding methods and handle it in the code and handle it with the architecture. Instead of having these hard dependencies between teams where you had to have release trains, you do things like self service platforms. So having to open a ticket to make a column change in a database or you don't have teams modified, you don't have a DevOps team out there controlling pipelines for you. You have the ability to build a pipeline. Your pipeline is part of your quality process for your application. That's you own that. But you have to have a platform that allows you to modify that pipeline while enforcing your organizational norms around security and compliance. It's so, so important. And so you engineer to decouple those things or you just keep doing things slow and have a lot of meetings and PI planning stuff and just spend a crap ton of money.

Speaker B: Can I double click really quick on Conway's Law? Because I remember when I learned about it for the first time, it totally blew my mind. But I was surprisingly far into my career when I heard about it for the first time.

Speaker A: Me too.

Speaker B: I wouldn't be surprised if some of our listeners have no idea what you mean when you say Conway's Law or the reverse Conway maneuver.

Speaker A: Yeah, so I mean, Melvin Conway in 1968, you know, pinned this observation that your system will mirror the communication structure of organization. Okay, well there's a lot of implications to that, but one of those that we, that People have demonstrated again and again was that, ah, okay, well now I've got this terrible system that was based off of how our communication structure ran. So for example, uh, in logistics we had about 400 developers who were uh, working on feature teams, which is a terrible anti pattern where a team goes and works on the next feature in the backlog on the system, which meant that we had spaghetti code. There was no architecture whatsoever. Well, how do we fix it? Well then we start saying, okay, well we're going to build a team where you leverage domain driven design. And so we're going to build a team around this business domain m, this business capability. They're going to own that capability and they're going to start pulling out the logic from the system that they own into their thing. So you start building, people say, well you've got this huge complex system. Yeah, so make it less complex. You just change it into smaller things that are not complex anymore. But ah, with Conway's law, that's where we started was we built the teams around these business capabilities. Then we told the team, okay, here's some basic architecture rules you must adhere to, now go and make your thing better. And they did. Until leadership changed, priorities changed and whole thing fell apart because the new leadership didn't really understand what was going on because we did a terrible job of documenting and communicating that as engineers.

Speaker B: That kind of leads to another question I have. If you're. I think there's two questions really. The first one is if you're an engineer and you see the waste and the pain of meetings after meetings after meetings to plan for something that ends up never going to plan.

Speaker A: Yeah.

Speaker B: How do you push for that change with your leadership and give them confidence in what you're doing without giving them a plan and vice versa. If you're at the top, how do you push that change?

Speaker A: You know, um, I have people tell me that I'm really good at communicating, that I'm a good writer, you know, that I'm engaging. I never used to be. I used to be that person sitting in the corner, heads down, who would never engage in anything. I was afraid of looking stupid. You know, I didn't have anything smart to say. But to get the kind of change you need to do this, I had to go and learn how to get people on my side. And it wasn't just me. I didn't go and lead this Walmart change. I mean this is a team effort, but you have to learn how to communicate. You have to go and find some leadership that might Be friendly and then learn how to talk to them, um, in their language. You don't talk to them in engineering language. You talk to them in business language. You know, if you're going to go talk to a vp, even if there's an IT vp, and you will talk to them about what the business goals we're trying to achieve, what the mission goals we're trying to achieve and how we can better implement this mission if we change things. Here's some examples from other areas. I'm always trying to find examples to show people they're kind of from other, you know, from. It's like if you're, you, uh, know, shipping freight or launching missiles, we need to find some examples for you so that you can point to them as evidence to help that conversation. But you have to get better at communicating. You have to understand how to transmit these ideas as more is just look at this nifty technology because they don't care. And so I had to learn how to do that. I had to go and learn how to talk to VPs, learn how to explain to them what we're trying to do, the value to them, and then try to get them on board and then go help the friendly ones.

Speaker B: Maybe a middle manager is a better example. Like I think I struggled as a, as a middle manager to. I would hear it's interesting some of the engineers would push back too on any semblance of anything other than the team's going to do whatever the team wants to do, which I think is a problem. You need feedback, you need all sorts of things. But on the up the chain I was resisting a, not resisting, I was complying as required with an agile transformation and what that they really meant by that was a safe implementation. How do you manage that from the middle of like on the one hand communicating up the chain that this is what we're doing, this is why it's important and this is why maybe we don't have the interdependencies that you think we do. Maybe we already have good practices that are delivering value and getting feedback, um, and then at the same time communicating to folks down the chain of like, hey, I do need some idea. For example, one of the biggest things I remember hearing is we're doing agile, so we can't give you any expectation of when we'll release this thing that you need. And I'm like, I don't think that's what it means. Like, I don't know. What's your advice for middle managers?

Speaker A: Uh, for a middle manager so number one, all of the biggest problem is it's all people. All of the problems we have is because of people. It's not technology. The technology is simple and getting people to think in different ways. And it's the same solution that I had for, you know, an engineer trying to get this change. You have to find allies, you know, who can help communicate that. So straight up, if you're a middle manager and you have no engineering background and you start trying to explain to me how to engineer, you're better, I'm just going to scoff, right? That's not a thing. You don't know. I've got all this stuff. You have no idea. Uh, thanks for playing. Right. But you know, if you're, if you demonstrate that you actually know what you're talking about technically and get more people on board. So like one of the jobs I had at Walmart was my team's job was to teach your team continuous delivery. We weren't agile coaches. We could teach you agile because if you understand how to deliver, you understand the process of how to get there. But we also were building our own stuff for the platform to demonstrate that we were software engineers so we would be taken seriously. But there's also community building. I mean if you're inside an organization, we worked really hard. My wife actually started a, uh, uh, developer community are built around continuous delivery as a marketing and knowledge sharing thing. Get the ideas out there, get people excited, get more people doing it so we have more examples to point to if we have naysayers. And so you have to build up the community and build up examples because nobody wants to, especially in government. Nobody really wants to put their head on the chopping block and be on the cutting edge. Unless they're mavericks. And God bless those mavericks because we wouldn't be anywhere without them. But you need to support them and use them as examples to help other people go, oh, well, maybe that's not so scary. And so it takes a lot of time. I'll tell you that from the time we started working on continuous delivery until they let us start deploying as rapidly as we wanted to was 18 months. And most of that, a lot of that was learning how to test better. But I'll say the last six months was just hearts and minds.

Speaker B: You talk about learning how to test better. Don't you just deliver your code to your test teams?

Speaker A: No, uh, when you start looking through, when you read the book Continuous Delivery and you think about, okay, what is it I have to do? I have to have the Current code always on a releasable state. Right. Which means it's fully tested. Then you also think, what's the job of a continuous delivery pipeline? Its job is not to deliver code. Its job is to validate your artifact to see if it's deliverable. Well, if I'm handing off to a test team, the pipeline doesn't have the ability to validate the artifact because it has to be able to validate from git. When I make a change to git, that change is either releasable or not. Which means the tests have to be there at the same time as the change. You can't add them later. And if you add, uh, can you

Speaker B: trust the developer to write those tests?

Speaker A: Well, but it's code. But the thing is.

Speaker B: But they're tests. Isn't that a tester's responsibility, Brian?

Speaker A: Absolutely not. If you have a tester who's writing tests, they're failing at their job, which is their job is to make sure we have a good quality process. Not to go. And it's their job is not implementation. Their job is to make sure we're executing correctly. Okay. It's a waste of human talent to put us, number one, it's a silo in your flow where you've got, you know, a few people doing things well. And also, let's say if you have an external testing team, you know those three wrongs I was talking about where we misunderstand. I have a development team and a testing team both. You've doubled the chance of misunderstanding, probably in two different directions, which causes a whole bunch of retry loops. I've got value stream maps I've done where I've demonstrated having a testing team, how destructive that is to your ability to deliver. But what you do is, number one, you have a team. You don't have individuals doing testing. You have a team deciding this is how we're going to be testing this. Next thing, you define what that quality standard is. The rest of it's just implementation, but the work is up, uh, front. Define deciding. This is how we know it's good. This is what we're going to test for the positive tests, negative tests, and then you practice testing. If you're doing continuous delivery, if you're doing DevOps. Uh, DevOps, and let me clarify, DevOps is the union of people, process and product to enable the continuous delivery of value to the end user. It is not infrastructure, it is the entire environment we work in to allow us to deliver value to the end user. Okay? That means we need to understand it's valuable, which means we're testing everything we do. So if you're working in a DevOps environment, testing is the main thing you focus on. Code is just implementation detail, but it's the idea. We're testing everything around us. How do we get faster feedback? How do we improve how we're doing things? How do we improve the ability to get, improve our ideas? How do we remove waste and drag from the system so we can get even faster feedback? It's always about, you know, efficiency of information.

Speaker B: What about that trope of the lazy developer who's not going to write good tests because they don't want to write good code? How do you answer that concern, first off, and secondly, like, how do you make sure that doesn't happen?

Speaker A: Um, you know, one of the things I'm a big proponent for is you build it, you run it. It goes back to the conversation about having a development team and an operations team on two different contracts. They have conflicting incentives. I build things. Uh, I mean, I write software today for defense unicorns and I guess who's responsible for the fort when it breaks?

Speaker B: Me.

Speaker A: So I want to make sure it doesn't break. I have incentives to make sure it's, uh, easy to modify, easy to fix. You know, I have all these. It's all about incentives. How are you incentivizing quality? If you're telling me it's somebody else's job to test and somebody else's job to operate, then sure, I can pound out code all day long. Cool. But it's all part of the quality process. And this is why, you know, when in DevOps you have teams, you don't have roles, you have teams, you don't have functional silos because it's just incentivized supportive behavior.

Speaker B: One other thing, so you mentioned earlier the sort of resistance of accountability within individuals who are in the government system. And I think when I've seen that, it's honestly more about. There's really no incentive to stick your neck out and take risk. Like it's. Yes, there's an incentive to not be, Make a mistake.

Speaker A: Yeah, I blame, I blame Eisenhower for that, by the way.

Speaker B: Oh, say more words.

Speaker A: So I'm also, I, I've never served in the military, but I love military history, specifically, you know, the Pacific War. And then later on getting into Korea and Vietnam in this, really trying to under and well, and especially after I started working on this sort of problem, trying to understand, understand the things that were put in place that caused different behaviors. David Hackworth's About Face was a really eye opening book for me about how the military really operates. The ticket punching culture that came in after Korea, we were massively downsizing after World War II. So what do you do if you make a mistake? You're gone. In World War II if you made a mistake they would move you someplace and figure out someplace else to go. In today's military patent would have been fired before anything he would before he was able to get to Europe which would have been a mistake. I mean he was a talented guy. He was just kind of mouthy in the wrong position at the wrong time, you know. But Eisenhower just created this culture, Curtis LeMay too, this culture of uh, you make a mistake we're just gonna have to fire you, goodbye. And so it caused people to freeze up and not want to make mistakes. So that's my perception anyway, so I blame Eisenhower.

Speaker B: I think that's really interesting. One of the ways I've seen this manifest is in this change review process and like decision by committee type decision, uh, making for software. How do you manage change, how do you manage environments, how do you manage engineering decisions without a committee, without a review board or uh, maybe that is the right answer.

Speaker A: Well I mean I have a committee, it's my team also I should have people looking and auditing my pipeline, my delivery and quality processes as not as this thing that you just built isn't correct. You should be looking at are we using the right methods to build things which is a different mindset. So you're not reviewing things at the end. You can't inspect, protect quality in you're continuously making sure that we're held to a high quality standard while we're doing things. Okay, so it's not a review board, that's people with expertise making sure that we're secure, that we're compliant, that we've got a good quality process. Although I consider security compliance to be quality aspects because it's not valuable if it's not secure and compliant either. Right. So it's switching from trying to inspect quality end to making it part of the process and turning a blocking activity into an ongoing continuous process of improvement.

Speaker B: So you just said something really interesting. It's not valuable if it's not secure and compliant. One of the biggest pieces of feedback against agile that I've heard is this idea that it's just not possible in high risk, high contact situations like tanks and drones and airplanes, that you have to use a different method because you have to hold it to a higher standard.

Speaker A: So, like Starlink.

Speaker B: Say more words.

Speaker A: Well, so we've got all these satellites, you know, orbiting, and, uh, absolutely, we've got to make sure that we spent months and months reviewing every single change, make sure it's. No, you have the ability to respond rapidly. The fact of the matter is that you can't predict every sort of security problem, or in the case of, you know, what we work on, threats, attacks, even in the commercial space, you can't predict all that. You have to be able to respond. And if you're not set up to be able to respond, you're going to respond poorly. You know, one of the things that I really want to stress in this blog post I'm writing is how we are delivering frequently to practice for when we have to. Whether or not you are. You, uh, want. Whether or not you actually, you know, need to deliver that often is irrelevant because when you do need to, you need to. And so, for example, Starlink, you know, when they released the satellites for Ukraine to use, Russia immediately attacked the satellites, uh, 48 hours later, they had a code change out there to fix it. Right. They have the ability to respond to what happens in production. It's operational readiness. It's not about. And this is why my claim is you can't be secure with a legacy process where we're going to go and inspect for all the security before we can deliver that. You have to be able to respond, to make things secure when you find that you're not. Yes, you do all the work to do everything you can while you're building something that's part of your quality process. You're building security in. You're not inspecting for security at the end, but then you have the ability to respond, to make it secure. When you find out you've got a problem,

Speaker B: that makes a lot of sense. What have I not asked you yet that you wish I would ask?

Speaker A: Uh, I think going back to something I said earlier about why I care so much, right. Why do I care how you're delivering Software? I'm on LinkedIn having these conversations, uh, about, you know, how do we do this better not. Because I'm paid to be on LinkedIn having conversation. I mean, that's, you know, I'm not 10 o' clock at night. That's not my job. But I care because of the very first experience I had with being able to deliver this way and every subsequent experience and the level of morale that was elevated by being able to deliver and get feedback and make it right, make it less wrong every single Day. I wrote a blog post while back. One of the things I really focus on is delivery metrics. That's something I've got a lot of experience with. How do you tweak metrics and display them in such a way to get delivered? Kind of influence behaviors. It's like a friend of mine calls it hacking the biggest undocumented API. But the. I wrote a blog post called the Only Metrics that Matters. And I'll share it to you. Share it with you later. But it's. I went out to Nellis. I have a friend of mine who's retired lieutenant colonel in the Air Force. He took us out to Nellis to go visit the Thunderbirds museum. And it was a Tuesday. It wasn't an air show. And we're watching the Thunderbirds prep to go on a practice flight. They're gonna launch two planes. They have the entire ground crew out there doing exactly the same thing they would do at an air show, practicing for the mission. They did everything exactly the same way they do at an air show to go launch two planes. You know, they had a guy they were evaluating on the ground crew there. I knew that because he was wearing camo instead of the standard blue ground crew outfit. And they treated him just exactly like a team member. While he was seeing what goes on and being evaluated to see if he was going to join the team, they had a plane go down. Thunderbird 1 had an issue where they popped a hatch, and they spent about 10 minutes trying to diagnose the issue. And then they just flipped him to another plane and launched him on Thunderbird 7. Right. Because they couldn't fix it in 10 minutes. What do they do? They just said, cool, you know, we know what to do. And it wasn't like someone, no one got fired. You know, nobody was stressing out. It was just routine because they practice for it. But the level of pride that I've experienced on teams are able to deliver this way mirrored the level of pride you saw in the Thunderbirds. Because so many people get ground into the dust and treated like children. Listen industry. And they get burned out, which. And you get really crappy results. And it just becomes this terrible loop of failure. But you know, what, if we fix the system of delivery, if we fix the organization we're in and we leverage people's pride, you get better outcomes.

Speaker B: So in that instance, then is the metric that matters, like pride of ownership?

Speaker A: Pride is the only metric that matters

Speaker B: others and not pride in the. Like, I won't admit I'm wrong sort of a way. But pride in like the outcome is something that I'm excited about.

Speaker A: Look what I did, look at how it helps somebody else. There's several things I've done in my career that have materially impacted people's lives. There's uh, one of the stories I tell is, you know, we, we went and implemented voice order filling in UK distribution centers and we got an email about one of the new someone who's promoted from floor sweeper to order filler because he was functionally illiterate and couldn't use labels to pick. But with voice order filling he got a promotion. The letter that went to Slack, that was written in crayon from um, a couple of kids thanking us, uh, for the changes we were able to make to touchless checkout rapidly during COVID to keep them safe. Right. That kind of pride,

Speaker B: that's really awesome. So I like to end with a lightning round of sorts of fun, easy, ideally questions. All right, we'll see how it goes. Maybe they aren't easy. Maybe I'm about to get some of the textbook. Uh, Finster Rage. We'll start with an easy, very non controversial question which is tabs or spaces?

Speaker A: Tabs or spaces? I use an ide. I don't actually know it formats. I have, I use auto format. You should never have that conversation. That's what automation's for.

Speaker B: I love that answer.

Speaker A: You file config on your project that your team agrees to and just use it and stop griping about it and

Speaker B: then no one has to fight about tabs or spaces ever again.

Speaker A: Yeah, just make it part of the automation.

Speaker B: What music do you like to listen to while you're like deep in work? Like, like doing deep work.

Speaker A: It varies. Usually if I'm really heads down coding, it's Rammstein. Uh, it has in the past been Bauhaus and even classical music on occasion. So it just depends on my mood.

Speaker B: Does pineapple belong on pizza?

Speaker A: I mean, sure, why not? You know, sweet and salty. I mean, how, uh, you can get arugula on pizza, you know, I don't know. People have different tastes. I'm not going to judge them.

Speaker B: If you could get everyone listening to this podcast to read one book, what book would it be?

Speaker A: It depends on their Persona. You know what? Modern Software Engineering by Dave Harley. I think that's a book that should be required reading for anybody in this industry. You shouldn't be able to graduate with any sort of computer degree without reading that book.

Speaker B: All right, last question from me and then I'd Love to turn it over to you for any closing thoughts, but if you had one piece of advice for someone starting out in this industry and define this industry however you choose, like, what would you advise them?

Speaker A: I'd advise them to find interesting problems. Don't get enamored with the technology. The technology changes. It doesn't. It's fundamentally irrelevant. Find interesting problems and then come up with good engineering solutions for them. I was talking to my old boss about continuous delivery pipelines in different contexts and how they're just engineering problems. And if you're going to air gap delivery, it's just an engineering problem. How do we make that easy? How do we make it so we can do that in small batches? It's engineer solutions. It's not some context for CD doesn't work. And so find as many different interesting problems you can find and work to solve them and don't just accept other people's answers for it. And you know, if anybody says you can't do that, they're just wrong. They just don't know how.

Speaker B: What closing thoughts do you have or final ideas that you want to share?

Speaker A: I just encourage anybody who thinks that I'm insane to just really look at, just really think through what your process for delivering software is and if that process works in an emergency. And in an emergency would you want to use, you know, like if you've got the, uh, it takes you three weeks to test, you're not going to do that in emergency, so you're just going to skip your quality process or would you write your quality processes to support an emergency and optimize for emergency and then only use your emergency process deliver software, which means you're going to have to optimize your entire organization to get that done. MinimumCD uh.org is a example of some problems to try that. It's a project I'm involved with which is just an example of problems to try to solve so that you can reach that goal. With the goal actually of improving your organization's morale.

Speaker B: No, I love it. Thank you so much for your time today, uh, and for sharing your insights. I really excited to have you on the podcast. I think yours is a voice that, that our listeners will really enjoy now.

Speaker A: Thank you so much for inviting me. I love sharing this stuff.

Speaker B: Thank you for listening to this episode of the Defense Unicorns podcast. If you enjoyed the show, please share it with your network and rate the podcast five stars wherever you listen. Tune in next time for the latest in DevSecOps.