Digging Into Security With Kat Cosgrove

Speaker A: Foreign.

Speaker B: It's time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices and operate your team and organization. For maximum DevOps awesomeness, I'm Matty Stratton. Today we are going to dig into a topic that everybody loves to talk about, which is security. You know, without security, we are insecure. But before we talk about my own insecurities, let's hear from our sponsors. Got robots writing your code? Great. But you're going to need a safe place to run it. Fly machines are fast launching hardware. Isolated VMs that spin up in milliseconds. Perfect for sandboxing even the sketchiest LLM generated code. Dynamic subdomain routing and zero cost scaling to idle. Yep, got those. Robots break things, Fly machines, restart cleanly and scale effortlessly. Fly IO is built for humans and loved by robots. Check out arrestedevops.com fly and make your robots happy. All right, joining me on the pod today, I have returning guest Cat Cosgrove. Kat, welcome Back to Arrested DevOps.

Speaker A: Howdy, Mattie. It's lovely to be back. It's been, it's been a hot minute since I've done one of these couple years. At least.

Speaker B: It has. I was trying to remember all the different episodes we've done and if you go to arrested DevOps.com you can figure that out. But I know we, we talked about deserted island DevOps in the past. We talked about the social network that has been renamed to the. The new name, so previously known as Twitter. There was a whole episode about that. But security.

Speaker A: Right.

Speaker B: You know, and I think specifically maybe we'll be thinking about vulnerabilities and how we respond to them and, you know, may container and cube space. But it's a broad topic, but we can cover it broadly or less broad as it might be. I don't know broads. I'm so good at this. I'm so good at this.

Speaker A: You're killing it. Uh, you're killing it. Yeah. Security is like a never not hot topic. I think the more of our lives and money and livelihoods and government functions are tied up in computers, the more anxious people are about security and the more dangerous vulnerabilities can. Can get. The, the days of the hacks and worms and trojans and vulnerabilities that we saw in the 90s, that like, that seems charming compared to how dangerous things are today. You know, it's not just like, I don't know, a chain email going around that, uh, crashes your computer anymore. It's something that shuts down the entire Planet nearly like with the, with the CrowdStrike outage, which was, you know, a bug, not an actual vulnerability. But things like Meltdown Inspector and Heartbleed seem terrifying today compared to like how goofy things were in the 90s. The names get more terrifying too.

Speaker B: Yeah, I was gonna say even earlier because if you think about kind of what's generally considered the first real worm was the rtm, Robert Morris's virus, which was kind of an accidental virus, which I want to say was in the 80s. But what's fascinating about it was at the time there was like what, 100 computers connected to the Internet or whatever it was. So it was relatively straightforward even then it wasn't, you know, so it's funny, you read the stories about it and it's like all these sysops are getting on conference calls to try to solve the problem because they're the only people. But it was only affected, affecting them. And you think about the much larger scale that that can come in. And also what was the impact, right? Because again, it was like research institutions and schools, which are important, but it's not that necessarily. People weren't able, you know, millions of people couldn't conduct their day to day lives which, you know, the whole, whole of the Internet and everything is held together with, you know, bailing wire and good intentions, you know, doesn't take much. So you talked also, you mentioned, you know, kind of those things and you know, the impact is much, much bigger now. And I would imagine it also becomes challenging because of dependencies and complexity. So when I start again thinking about in the 90s, in the time, you know, our systems weren't as complex, you know, fond of saying, like there was a time you could hold your entire stack in your head and you knew what all the things were. But we run into, and this was, you know, you know, we did, uh, an old episode with Pete Cheslock and Charity Majors called who Owns yous Availability? And this was after Left Pad. But it's, you know, the dependency upon dependency upon dependency. So when there's vulnerabilities in a package, you may not even be aware you're using that package or that thing because it's 17 layers below the thing that you're using. You're like, I just, you know, and that was kind of the thing, how much Left Pad's a great example of that.

Speaker A: It's really hard to like peel through the layers of, of something to find you, you, you have no idea. Like, think about the, like the Equifax hack, right? That was because of an outdated, wildly outdated version of Apache struts. And like yes, they should have updated that version of Apache struts. But uh, with hindsight being 2020 and all of us being like armchair computer science doctorates, we can imagine that we would not make that same mistake. That uh, that would be a thing that's easy to keep track of. But uh, it happened to Equifax and it could happen to you because you, you may not know that you have a wildly outdated package. Chilling out three dependencies deep like. And that's how we ended up with you know, things like software, bill of materials and uh, 85, 000 pieces of security tooling and every conference and their mother hard focusing on nothing but security for content for a couple of years because, because of stuff like that. It was, it was really, it destroyed a lot of people's lives. I have credit monitoring for the rest of my life because of that. So that's, that's, that's pretty cool. Um, thanks, I guess. But yeah, like back in the day, these kinds of attacks were like, they certainly were destructive. Right? Like the attacks were novel. They didn't happen as frequently. The people didn't have a good understanding of how these attacks happened. At first we were slower to respond to them. We didn't have really ways to prevent them. And now they're significantly scarier. We're more, we're more used to it happening. We're, we're more like defensive rather than just being reactive to, to an attack. But the amount of damage that can be done is absolutely wild these days versus, you know, 1989. Yeah.

Speaker B: And there's kind of two. Two pieces because I was just thinking about when you're talking about, you know, knowing what you have because that's how you can. And that's not, you know, s boms and things like that help with that. But that's still always going to be responsive. Right. And you're like, oh, there was a vulnerability. Now I can get to that. But we always want to be as proactive as we can, you know, which again we're not always going to be able to be. But I think about things like dependabot depend a bot are like they're going to let you know when there are known vulnerabilities in a thing that you have, which is actually a capability of being somewhat proactive. There may have been a zero day on it. You may already have been compromised because of it because it depends on how quickly that happens. But that's a pretty proactive way to connect to that. But then also how, how often do we just pull in so much stuff?

Speaker A: Like all the time, like constantly.

Speaker B: Which we need almost none of. Right?

Speaker A: Yeah. And big companies spend, uh, I mean any company really, with any software company, but especially big companies, they spend a, like an absolutely outrageous amount of time and money just mitigating CVEs. That's, that's pretty much. They'll have entire teams that just do that. And do they mitigate all of them? No, absolutely not. They get like the super high priority ones. They get the very highs and maybe the highs, but if it's going to take more effort than is worth it for the like, potential exploit, they're going to just leave it there. Which is maybe the decision that somebody at Equifax made about struts. That's, that's speculation and I have no idea what was going on internal to Equifax at the time. But yeah, most, most software companies do not mitigate the majority of CVEs in their dependencies. They just leave them chilling there because it's often just not really worth it versus, versus the risk. Whether that's because it's uh, a vulnerability that's particularly difficult to exploit or it's a vulnerability that doesn't allow somebody to do a ton of damage. Most CVEs just get left chilling there. So if you think that your software is secure, it's probably not. And nothing is immune. Kubernetes had a pretty hot 9.7, 9.8 a couple months ago at the time of recording vulnerability in Ingress NGINX, which affected 38% of cloud environments. So that was, that was super fun to deal with from a maintainer perspective.

Speaker B: So one of the things that I kind of wonder about and like there's lots of problems, right? I'm not looking for all the answers, but I'll, I'll run into this in a small place and I'm sure this, uh, my own projects and stuff and so I'm sure this is only exacerbated at scale is when you'll sit there and you have a dependency for this thing you're using and it's got high, you know, there's a CVE on one of its dependencies or something like that. And that's some shit that hasn't been updated in five years. It's some little package that somebody did whatever with.

Speaker A: Oh, we're on the XKCD comic now where it's just like one guy.

Speaker B: Yeah, yeah. Except that this guy the guy that's maintaining it doesn't even do anything with anymore and which is fine, it didn't happen. And then I guess like it's like what do you do? I guess you need to like at that point do you need to fork that thing? And then like now you have to maintain it yourself.

Speaker A: And yeah, private fork, you fix the problem yourself. It's open source baby. PR is welcome. Except when the guy who is the one admin for the repo goes AWOL and now the PR is going to sit there stale. So yeah, you, you fork it and fix it yourself and use your own forked dependency and then if you leave that thing public then God help you. Maybe other people see that you did that and now you're the new XKCD comic guy. And that sucks. Like I'm a big fan of open source. I think open source is the uh, best and most secure way to build software. But I do have to admit that it has some downsides. And that's one of them. That's one of them for sure. The ease with which people find vulnerabilities in open source software is also like, it's kind of a double edged sword. They get found more quickly so they get addressed more quickly. But also they get found more quickly so they get exploited more quickly. Quickly. Right. And that's, that's a, that's a pretty big downside. But as, as a community we do have the benefit of having a veritable army of very uh, dedicated weird nerds, myself included, willing to fix these things fast. Right. That, that is something that proprietary software can't have. They, you just, you just simply cannot move as quickly with a limited number of people and limited visibility into what's going on in your code base. So open source is better. Don't let the, the vulnerabilities and the dependabot alarms scare you off. But uh, you are going to get quite a lot of alerts and uh, most of them you're not going to be able to mitigate yourself. So that's, that's, that's fun. It is what it is. We've got a lot of new cool security tooling on the market, a lot of new cool observability and monitoring tools on the market that make it easier to tell when something might or is or has gone wrong so that you can, you can respond more quickly. But obviously we prefer to prevent these things from happening at all rather than just responding to a big, big piece of drama that is at uh, best publicly very embarrassing for you. And your, your company and at worst, destructive to your livelihood.

Speaker B: So you just, yeah, you referred to, you said, you know, there's a bunch of tools. What are some of the things that you feel like should be in everybody's kit? Like, and maybe there's multiple, you know, there might be like three tools that do this kind of thing. But if you're sitting there and saying, you have to have this, if you don't have this, hit pause on the podcast, go install this in your production environment and come back and listen to the rest of the show.

Speaker A: You absolutely do need some kind of observability and monitoring tool. Like you, you need to be using something like opentelemetry, something like Prometheus. You, you need, I mean, which one of those types of tools you use depends on what kind of environment you're working in. Obviously I'm a cloud native person, so I'm going to default to cloud native tooling. But you cannot just be like raw dogging production. You, you have to know what is going on in there at all times. You have to know when you're getting weird traffic from a weird location, hitting a weird endpoint or a lot like that kind of thing can be indicative of something very wrong going on, whether it is an attack or just like a bug, just something farting, a container falling over. You should not be blind to what's going on in your environments. Similarly, with alerts, you don't want to fire hose it. You need something intelligent enough to make those alerts useful rather than just bombarding your engineers or your DevOps team or whoever is getting your alerts with a whole bunch of nonsense that could actually be nothing. So it's got to be something intelligent enough to make some of those decisions for you. You do need some kind of dependency management situation. You need to know what your dependencies are and what your dependencies. Dependencies are. Cannot be just raw dogging. Those either. Can I say raw dogging on this podcast? I've said it three times.

Speaker B: You apparently have multiple times. So you are capable of doing it.

Speaker A: We're going with it because that CBS anchor said raw dogging on the air.

Speaker B: So my, my question is, is raw. Is the use of the term raw dogging sufficient to merit the explicit tag on this episode?

Speaker A: I think not. I think if you can say it on cbs, you can say it on a podcast.

Speaker B: But if we were to say fuck, then I'm gonna have to make this tag.

Speaker A: Well, you just said fuck.

Speaker B: So there you go. So there we go. And we've Said it twice because we uh, violated the PG13.

Speaker A: Yeah. Nah, it's not PG13 anymore.

Speaker B: Okay, well now we know that this episode will be March. That won't be the first. First time either. It won't.

Speaker A: I probably cussed in the last ones too. So.

Speaker B: I mean in the pre show recording with guests, Bridget would say we have made our piece with the explicit tag. So feel free to speak as you. As you would like to. Which amazingly I didn't actually say to you, which I normally would have.

Speaker A: So you know, I do usually ask. So that's. That's my bad before I go on a podcast. Anyway, don't raw dog your. Your production environment. Know what's going on. I think that most pieces of DevOps tooling do have a security angle. Your infrastructure as code tool has security implications and ah, you should be using an infrastructure as code tool. You should not be hand rolling your AWS deployments through clicking in the ui. That's outrageous and miserable. And I think you hate yourself if you do that. I try to not to talk about my own employers on podcasts, but I do work for a company now that solves exactly this problem. I work for Minimus. I'm their head of developer advocacy. So here's me advocating. I guess we build very small container images with most of them have zero CVEs. That is a super useful way to start your application environment with no CVEs. Obviously we can't do anything about vulnerabilities that you introduce through your own like maybe questionable code. My code is certainly questionable, but your base image will have one or two or none CBEs in them. Um, and they do get automatically rebuilt when there's an update, which is, which is pretty cool. So that's a good way to start from. From zero. You should be using some kind of like software bill of materials. You. You need to know we're all tired of hearing about SBOMs. I know, I know we are because it was like all we talked about at Kubecon for what feels like three years and it has finally cooled off. But they. They are still important. Especially as more and more of our lives are sucked up by computer. I, I find myself intentionally seeking out hobbies that do not require computer because I'm so tired of computer.

Speaker B: That's actually like really number one, like a really good and healthy thing. But it's funny, I remember having this conversation with Pete Cheslock years ago where we were just like at the bar and whatever and he. Something came up about him playing piano and how he was learning to play piano and he'd been playing piano for like a year. And I was like, I had no idea. You know, that's really cool. And he was like, well, I don't talk about it on social media. It's not computer related. But that was part of, part of why he did it was to say, hey, I want to do something that's not like involved in tech as a hobby, but also sort of peripherally was like, and that's also not part of how I talk on social. Then I'm not, I'm not talking about it or whatever. And I was like, oh, that's, you know, I talk about it like in person, but it's separating that and it's challenging, but it's a good idea. And I feel like I, I tried to do that. I think I am doing better because I ran. I had a self discovery thing. I think we've all been down this path. When you look at your hobbies and you're like, all of my hobbies are just a reframing of work.

Speaker A: Yeah.

Speaker B: You know, you're like, uh. And I think, I think I really became a parent of this years ago when I was, you know, like on the dating scene. And like when people like, what do you do? What's your hobbies? I'm like, well, I have a podcast. What's your podcast about? Tech? Well, I build this website, I run a conference, I do whatever.

Speaker A: Oh, Matt, that's embarrassing.

Speaker B: It was embarrassing.

Speaker A: Good Lord.

Speaker B: I don't know how, how I got past the first date with stuff then, but it worked out.

Speaker A: Yeah, she's much, she's very patient. She's a, she's a saint, I think. But like security is a security and computers are, are a burnout factory. So if you are listening to this and you don't have a hobby that doesn't involve computer, I am begging you to literally not, not figuratively. I'm not trying to be snarky. Literally go outside and touch grass. Like, please go touch some grass or moss, even if it's raining.

Speaker B: You can, you can dry your hands off. It's okay.

Speaker A: You can, you can. I went on an 11 mile hike last weekend in the rain. It was lovely. I recommend it. My thighs hurt afterwards because it was uphill for the first half. And by uphill I mean up mountain.

Speaker B: But, uh, real big hill up, uh,

Speaker A: up very big rocky hill. Yeah, yeah, touch, touch some grass. Especially if you are on somebody's security team on the mitigating CVE's treadmill which is like the much less fun version of the loot treadmill. In video games like Diablo, we like that kind of treadmill, right? That's good. Treadmill.

Speaker B: What? Treadmill.

Speaker A: Loot. Treadmill.

Speaker B: Loot. Treadmill. I thought I said loot.

Speaker A: No, no, no, no. Loot treadmill. Games like Diablo and Borderlands get called loot Treadmills because after a certain point, there's absolutely nothing to do in this game. Farm for more loot. And for whatever reason, that feels like super rewarding in both of those games. And it's like, somehow worth it to spend 30 hours grinding for a sword shaped like a cheeseburger in Diablo 3. I don't know how they managed to make that fun, but that is fun.

Speaker B: But doing it, uh, doing security looting or not security looting, maybe security looting is fun. I don't.

Speaker A: Oh, no, that's called crime, Matthew. That's. That's crime. You invented crime there. You invented hacking, which is, you know, which is fine, but don't get caught and don't commit more than one crime at a time.

Speaker B: I. It just made me think of the good place. But, uh, all I hear in my head is there's. There's a. A scene where Jason is working on trying to get his, like, dance troupe going. And he had told them all, he's like, no, we don't do crime. We're doing the dance troupe or whatever. And then after you know, how many iterations of failing at, he goes, all right, well, I guess it's not gonna work.

Speaker A: We're doing crime.

Speaker B: Yeah, go do crime.

Speaker A: Go do crime. You know, I. Look, I'm not going to advocate for crime, however, you know, live a little. Is that publishable it to say live a little?

Speaker B: I think that's safe. You're not implying necessarily that living requires the doing of legal things. You know, we are not providing legal advice on this podcast. And if you come to this podcast for legal advice on how to live your life, then God help you. You need to touch a lot more grass.

Speaker A: Honestly. You need to fully log off, actually. Especially if you know, like, anything at all about me. Do not take life advice from me.

Speaker B: Don't do that. I think if you are taking your life advice from Kat, then you need to Ron Swanson that and just throw your computer in the dumpster and pretend that they never existed.

Speaker A: I wish rage rooms were would come back. I do. There's not one here. I looked recently. There was one in Glasgow.

Speaker B: My friend Jamie, who is Glitter burrito on Blue sky. And I think other Places. So she, she's in Georgia somewhere, I think. But I remember she posted something relatively recently about being in a rage room. And it was actually very cool because you're like the pictures, it's like full, not hazmat suit, but like big protective suit with the goggles and the giant sledgehammer. Freaking shit, you know?

Speaker A: So you know how some conferences have like therapy dogs? Kubecon has done this before. I was just at rsa, unfortunately, God help me. And somebody had baby goats there. I think that there is a real opportunity for some conference or some vendor who wants to spend a lot of money on a big ass goddamn booth to do a miniature rage room. There's probably like some liability issues there, but I think it's worth exploring because surely I'm not the only one, uh, at a conference burnt out on computer who wants to take a baseball bat to Iraq. Like, you can buy these things for cheap.

Speaker B: That would be the M updated cloud native version of that scene in Office Space when they're going full execution mode on the printer.

Speaker A: Yeah, it's that, it's that. I want that, but I want a Dell Power Flex.

Speaker B: Exactly. Yeah. Uh, you need some switches. You need some. Because that's the thing. Because the only reason that we don't want to murder a printer right now is because we don't have to print anything.

Speaker A: You know what? Actually I have a printer, okay. And I've had to print something recently. And it was not that bad what

Speaker B: I have had to do with my printer because for two reasons. So one is the good thing about the I can't see the printer and you're like, you're literally right next to a computer. But for some reason my, my HP printer or whatever continually needs, uh, fundamentally, anytime I want to print, if it hasn't been in the last five minutes, I have to unplug the printer and plug it back in anyway. And so what I do is I just keep my printer unplugged all the time. Because the other thing is it has a status light on it that you cannot turn off. And this printer is in my bedroom, which means if my printer is plugged in, there's a blue glow throughout the entire room. So my combination was. Number one, I don't like that. Number two, chances are I'm going to have to unplug and plug it back in every time I want to use it. So you might as well make the default be that it's unplugged.

Speaker A: Sure.

Speaker B: Some metaphor for the Internet or working in tech in there. Somewhere we'll find it.

Speaker A: But, you know, I would actually like. I. I know that this is absurd. The podcasts, like, famously visual form of media, so you people can't see this. But behind me there is a big, enormous gaming PC with like, RGB LEDs on. Absolutely goddamn. Everything. My mouse, my keyboard, everything has RGB LEDs on it. But for the most part, I wish had less LEDs on it. I have two of those enormous ASUS ROG routers that look like, I don't know, uh, a satanic altar with, like nine antennas. And there are so many LEDs on that thing. And one of them is in my bedroom. So I can turn off the, like, fancy RGB LEDs on it, and I do, but the status lights stay on. So it just like, lives with a T shirt over it. Otherwise there's blinking at me while I'm trying to sleep, and I have. There's nowhere else it can go. It has to go in my bedroom because that's where the entry point for the fiber is. So if, uh, for some reason, like, Asus or whatever other hardware manufacturer or brother printers is listening to this, can we have a button to turn that off? Like, I don't want that turn. Turn that shit off. Because I don't want computer visible in my home, in anywhere other than the computer room. You know, like in the 90s, we had like, the computer room or the family room happened to have the computer in it. I'm bringing back computer room. Uh, this is, this is computer room, my living room and my bedroom. No computer.

Speaker B: I would, I would prefer that myself. I kind of had to make my bedroom also be computer room because I had to make computer room turn into one of my son's rooms. So that, that negated that. But even that, because I thought about that, because I'm gonna put up like a partition behind my, you know, so at least. Because I need the, like, mental block between my desk and my living area. But I was like, but if it's a bright light, it's gonna bleed through, you know, a room divider thing or anything. So this is all a metaphor for security. That's why we're talking about it. So it's fine. So if you're still listening to the show, I appreciate that. But, uh, that also kind of brings us around to, to, to an end. So, you know, if you'd like to see the show notes, we'll put some links to some of the things we talked about. You can go to arresteddevops.com diggingintosecurity for this. For those show notes, if you go to arresteddevops.com iTunes and leave us a review in the Apple Podcast Store that can help other people find the podcast. You can also listen to the show on Spotify, iHeartRadio, and Audible anywhere that fine and less fine podcasts, um, can be found. And I do think that I may be this close to finally updating that redirect so that it's not, slash, itunes anymore, because it hasn't been called itunes in forever. Oh, my God.

Speaker A: What's it called now?

Speaker B: Well, it's just Apple Podcasts or Apple Music or whatever. Like, so referring to it's the Apple Podcast Store, it's not itunes where the podcasts are. And coincidentally, I think what kind of made me decide I'm okay with eventually finally doing this is I've been updating the Hugo template theme that this podcast uses, amongst many others, and it all still internally refers to everything as itunes instead of Apple podcasts. And even, you know, and I was like, all right, fine, I'm doing a big version. We'll call it Apple Podcasts and itunes, because there's going to be people doing this who don't even know what itunes is. This. Anyway, that's how you can listen to us. We hope you listen to us more. Kat, thanks for joining me today. We'll continue these conversations. And this is arrested DevOps. And remember, there is always DevOps in the banana stand.