Scaling Done Right 09: Deploy or Die

Speaker A: Hello and welcome to the Scaling Done Right podcast. This is episode nine and my name is Garyan Hermkes. This podcast is a companion to the book I have written with Louis Cantella, which is called Scaling Done Right how to Achieve Business Agility with Scrum at Scale and Make the Competition Irrelevant. Today we're going to look at a very interesting topic, which is deployment. And the chapter in the book is, I think, aptly named Deploy or Die. Deployment, for those who don't work in software, basically means delivering a product to a customer. It's not only the act of delivering it, but also getting it ready to deliver it. If you think of a, um, physical example, let's say, of an E Commerce warehouse, this becomes immediately obvious. So it's one thing to have all the items in store, have a worker, uh, pick the items and finish the package, right? And send it off to the outbound bay, to the shipping bay where it's supposed to be picked up. So far this is good, right? You have everything set up, you have a warehouse, it's difficult to hire the people, it's difficult to have the logistics systems, and you have the package ready to go. But now imagine that for in order to have it shipped to the customer, there are no systems in place. Meaning I, as a customer, I don't care if the package is ready in the warehouse. I need to hold it in my hand. So there's a lot of steps that have to take place for it to get to me. And so imagine that the package is ready and now somebody has to call the freight forwarders because there's no system in place, right? They have to call DHL or UPS or whoever it may be and get them to pick up that package. Similarly, if I live in a different country than the E Commerce store, what about customs? Are there any forms that need to be filled out? And how do they need to be filled out? Similarly, there might be regulatory restrictions. For example, lithium ion, um, batteries, which are often in laptops, that cannot be shipped via air freight, right? So the first step, the first sequence is very important and a lot of people get that, right? But the second one also, um, needs to be done, right? And it is very hard to imagine a logistics company only doing the first step because it doesn't bring any value until I have my package in my hand. But if you think about software development or knowledge work in general, that is exactly how it's going. So people produce something they think is ready, but it's really not. And if you're doing Scrum you already know that at the end of the sprint, you're supposed to have something that is potentially shippable, potentially meaning you don't have to ship it right now because it might not make sense, but it's supposed to be ready to go out the door. In reality, what happens is that after a couple of sprints, maybe a manager or stakeholder might say, yep, this is good to go. Please let's deploy to the customer so we can get some feedback. And more often than not, what happens in that situation is an excuse, an explanation for why that's not possible. What it should look like is basically a button that says deploy. You push on it and it's delivered to the customer. Amazon doesn't even have that button anymore. There's nobody sitting in the outbound bay pushing buttons. For every package that needs to go out. If it's flowing into that direction, it's going to be automatically transported onto the right truck and going to the right distribution system, such as DHLs. So think about what this does with a manager. He's putting a lot of investment into the team, he's trusting the team, he's seeing some results which he's liking, but then as soon as he actually wants to bring it to the customer, it doesn't work. So then maybe the Scrum master is taking care of this communication and he'll explain, well, yes, you know, um, we should do it, but actually we haven't built up the systems for deployment yet. Or well, we have to talk to IT Operations and the big corporations and that's a wholly different department. Maybe in a different country, we don't know who to talk to. Or maybe they are completely overworked so we can get a slot in six weeks. Or there's like a litany of excuses that happens in that moment. And I'm not saying that this is your fault. I've been in this situation numerous times and I'm sure I will be in that situation again. What we're trying to do is raise your awareness that this area of deployment should not be some afterthought, but should actually be one of your core focuses. Because similar to that example of the Amazon warehouse, the E Commerce warehouse, what good does it do you if you say we are being agile, we don't want to do two year projects and then find out that the project isn't even demanded in the marketplace. We want to have these Sprints, we want to have these shorter, uh, time frames where we build something, we get it out, we get feedback and we can readjust how are you going to do that if you don't deploy regularly? And the answer is you won't because you will be lying to yourself. You will act like you're actually doing short sprints, you're actually creating stuff, but you will never get the product out into the hand of the customer. And so it's very important to not just do this first part, but also have the second part that actually gets the product into customers hands. And there's a big, big psychological problem at play here. It's really insidious. So imagine that situation. You have worked for four sprints, everything looks fine. The manager comes and says, I would like you to go out if it's difficult to go out. What I'll do as the scrum master if I don't understand that we actually need to invest in the developments into the deployment systems right now, um, is I'll say, well, let's wait another four sprints because we know this is going to cost us a couple of days of the whole team to actually get the deployment done. You know, let's make what we deploy bigger so that cost for deployment is actually spread across more items that we finished. So let's really make it worth a while, right? Because deployment is hard, it takes time, so let's wait a little bit longer before we deploy. And that is very logical. That makes absolute sense. The problem with that is that the longer you wait, the more difficult the deployments will get. One of the aspects of that is that if we're talking about software, if you wait three months until deployment, the developers won't even remember what they did at the beginning of that time period. And they will have, if there's a problem, if there's a little glitch and a lot of time has passed, then they will need to basically redo the work, try to understand again what they did and it'll take more time. So if deployments are difficult, people tend to wait longer to make it worth their uh, while to actually accept that pain, to pay that price. But that uh, again will make it more difficult. And so suddenly you're in that downward spiral that is very well known from Waterfall project management. Because if you go that spiral until the end, if you follow it to the end, what are you going to get? You're going to get a deployment every two years that's going to be a catastrophe. Anyone who has ever lived through one of those deployments, basically a death march. Everything breaking not only in our software, but also all the other systems that are being touched and just a phase of chaos that everybody will talk about regularly and think back to as their private horror story. Right? And so what we want to do instead is we want to deploy as often as possible, to never build up this monster, this monstrosity of complexity that nobody understands anymore. If you deploy every two weeks because you have invested in the systems, you won't have that problem. The people will still be fresh on that topic and they can just deploy it and it'll be good. So this is something that is often not understood well, what impact this can have on your business. So it's one thing to say, well, we want to spare the developers of all that difficult work, the death marches, etc. But from a business perspective, this is actually what helps you win. And this is not. It's not very intuitive. So what if I told you that Amazon actually deploys their software several times a second, right? And so now you have to imagine that you compete with them and let's, let's have everything else equal. You have as much money, the same employees, and you have to compete with them, but you deploy every eight sprints of two weeks, if you're good, maybe even longer. And they just deploy whenever they're ready because they have that infrastructure. They don't even wait until the end of the sprint. They just, if it's done, it's done and it's out. And if you're using their website on one day and then on the next day, it's very likely that it's not the same website. You won't notice it because the changes are very small, but the website has changed somehow, somewhere. And so imagine you have everything equal, but Amazon is just deploying. They're just deploying. You know, uh, a story is done, so after two days it's going to get deployed. Over time, what you'll see is that Amazon will learn much more quicker than you do. They will have the chance to learn from the mistakes. They will see which features are interesting and invest in them more. And they'll see stuff that nobody picks up and not one of the users is actually using, so they know not to invest anymore into it. And this little effect at the beginning, over time, will overwhelm you because you just, you're not cycling as quickly as they are, you're not looping as quickly as they are. And so they, they'll always beat you to the punch just by a microsecond, but they will do it over time. And it's going to take a couple of months and you will Be you will not be able to catch up to them. So we have this perspective of protecting the developers from these death marches and not incurring that cost of deployments getting harder and harder. But because we're talking about business agility, right? This is a big part of it. If you cannot, uh, deploy this stuff quickly enough and learn enough, you don't have business agility. You can have as many scrum teams as you want. If you're not deploying, you don't have business agility. And Amazon and Facebook and Google and all the other death stars, they will eat your life just because they can deploy quicker. And the kicker to this whole topic is that the return on investment on this stuff is extremely high. It's exceedingly high. So even for a large corporation, the infrastructure setting it up is going to cost less than a million euros or dollars, which sounds like a lot, but it's really not. Because a lot of that technology is open source. And while there is, while there are specialized skills around it, it's not an arcane mystical science that only a handful of people can do on the planet. If you have competent developers and operations people, they can do it, they can set it up, they just have to do it. And the reason why, it's, one of the reasons why it's often not done, um, is because the business people, the managers don't understand this logic. They don't understand that this is a major competitive advantage that they can get for cheap. And one of the things that's behind that is also that people sometimes get nervous if we tell them, um, that we can, by implementing DevOps is one term for it, like continuous deployment and continuous integration and peer reviews and all that goes along with it. By doing that you're actually not only getting quicker, your quality is actually going up as well. And for managers there's often this dichotomy between those two. I can have speed or I can have quality. And, and now we're saying, well, you can have both and it's not even going to cost you a lot. And so that sometimes creates resistance in managers. But think about what we're doing. We're basically just automating stuff. And if you think of a, you know, if you think of trains way back when you would have um, the track switches and there would be a person living next to the track switch and like at the, at the right time standing out there with a little clock and waiting for the train to come through and then, ah, run uh, to that lever and pull it so the train would go into the right direction. Right? Think about that and compare it to an automated rail management system today. The automated system today is not only much quicker, it's also cheaper and it's also safer, um, because we are not depending, we are not relying for our lives on that person that's maybe been drinking all day long and that's, um, pulling the lever in the wrong way. So in all other areas of life, we accept that automation means more speed, more quality, and often a lower price. And it's the same in deployment. We have to invest into the deployment systems so that the customer is happy because they're getting products quickly. The developers are happy because they don't have to do death marches. But most importantly, because it enables us to do business agility. Because if we don't do it and our competitor does it, they will learn quicker and they will know where to invest their money into and they will just kill us over time. So deploy or die. Thank you for listening to that episode. If you would like to know more about this, you can find our book, Scaling Done Right wherever you might buy books. If you need help with your transformation or with Scrum at Scale, maybe in the form of training, mentoring, coaching, you can find me, garyon@teamflow.net and you can find luis@ruscareer.com thank you. And until next time,