9.5 CVE, Zero Warning: Inside the Rails Active Storage Bug

Speaker A: Welcome to SaaS App Building B2B Web Applications, the podcast where we share real world stories, practical advice and tech insights for those building or thinking about starting a tech enabled business. I am your co host, Aaron Marchbanks

Speaker B: and I'm Justin Edwards. Each week we bring you the stories, strategies and insights you need to build your SaaS or tech enabled business smarter, not harder. Let's dive right in.

Speaker A: Hello, hello, hello, and welcome back to SaaS, that app for another fine episode where it's going to just be me and Justin. You are stuck with us again. Justin, how's it going, dude?

Speaker B: These poor people are not getting enough variety. And, uh, I like that we recorded something, not what, 30 minutes ago. And you have a different bow tie. I appreciate your dedication.

Speaker A: Absolutely. You know, it's important. And since we're going to be talking about security, went with the tiger theme for the tiger teams of old. Really dating myself in hackerland.

Speaker B: I'm like, I don't know what that means. Aaron, it's all right. I think my grandpa used to tell me stories about the tiger team, but he was pretty demented at that point. So we're just like, quiet, grandpa, go sit in your rocking chair.

Speaker A: Oh, my goodness. It's going to be a week of this. Yeah. Earlier this week, we're telling a story about Back to the Future. And the person we were on the winery with was like, now, I haven't seen that, but that's quite old, right? Like m. Oh my God.

Speaker B: What is it? Like the reverse of. Okay, Boomer, like, type down Millennial. I don't know, something.

Speaker A: Oh, man. Well, we are going to be talking a little bit about security, but very, very specifically, we're going to be talking about a recent vulnerability that popped its nasty head up in the land of Ruby and Rails. And so, uh, Justin gets to play the what in the world is happening? Role and I get to play the hey, what in the world is happening? Role. So first things first, Justin. Let's lay the foundation for, especially those who are maybe not down in the weeds or at the technical or get the information or follow those such things. But what we are talking about is a vulnerability in a piece of software. We will be talking a lot about, you know, how to protect yourself from that and why it's important to have patches and security and things along those lines. But tell us a little bit about. And the acronym here is cve, what that actually is, and this one in particular just kind of lay the foundation for us.

Speaker B: So CVE is the acronym certainly means something but basically it's a vulnerability. The V probably stands for vulnerability. But uh, uh, the idea is basically when you find a big defect in a software, you publish a, or a, uh, security vulnerability. You publish a CVE and there's a process that people who find vulnerabilities, uh, and repair them, uh, go through. And so there's a, there's an organization that deals with this. You basically reach out to them and you request a cve. The CVE has a code, it starts with the date. This is all kind of like official security stuff. CVE is assigned to criticality on a scale of 1 to 10. And I don't think they actually ever issue a 10. So 9.5 is about as high as they can get. Um, maybe there's been higher ones, but 9.5 is anything over 9 is pretty much everyone who works, works in software and tech, just stops their day and goes and takes care of that on all of their systems. Which is the experience we had a couple weeks ago when this came out.

Speaker A: So what was this one in terms of like that ranking?

Speaker B: Oh, it was a 9.5.

Speaker A: This was a 9.5. Yeah. Okay.

Speaker B: I think it's the highest one I've ever seen. I don't know if they've had one that's been higher. So yeah, this was a, Everybody stop what you're working on and let's get this patched, uh, situation.

Speaker A: And so the target here, uh, had to do with a, uh, flaw in Rails active storage in a particular library.

Speaker B: If I remember correctly, Ruby on Rails is used as a framework we use to build web applications. Tons and tons and tons of web applications are built in Ruby on Rails. Many, many applications use every day are Ruby on Rails Active storage has been the default way to handle file uploads and working with files and images in Rails for a hot minute. And when they go in and they put these CVE numbers on them, they consider a bunch of things. So one of them is like how easy it is to get to. So in this case, anybody who had access to a file upload, whether they were anonymous or whether they had a user account, it was unprivileged. So if you had users who could not log in, who could upload image, and you were running this vulnerable version of software, you were owned, you could be owned with an attacker. The other things that they consider is like, how common is this configuration? And in this case it was as bad as it gets because the default configuration that shipped with Rails and this particular jammer plugin, if it's not core at this point, the default configuration for how it's set up out of the box was vulnerable. So if you didn't do anything and you did all the right things that were using the standard tooling, you were in bad shape, uh, which they consider. So there's the number of applications that are out there that are likely to be vulnerable and then how hard it is to actually get into this thing and exploit it. So if we want to pass our passing series of tests, anybody who has access to fileable can do it, anonymous logged in or otherwise, tons and tons of applications. And the default shipping settings were vulnerable. So maximum number of applications, maximum amount of impact, and the next thing you look at is criticality, which is how cool. So there's an exploit. But all if you can do with an exploit to show a puppy animation, no one really cares all that much. They look at how bad the criticality is. In this particular case, the criticality was also as bad as they get because it was one step to dump any file on the server or container down to the user, and two steps to do remote code execution. Which means if you ran a Ruby on Rails application that had an anonymous file upload and somebody found you within a short amount of time, they could get access to all of your application secrets and also have your web application start doing things on their behalf using this vulnerability. So start combining those things and, uh, it's like super easy to exploit. Everybody who has access to viability can do it, and once you do it, you own everything. Really bad.

Speaker A: Was this for any upload or was it a particular kind of upload or a singular upload?

Speaker B: So the vulnerability was specifically with file uploads. And I believe it had to do with. There was a check that's important to run on certain file uploads, it's not important to run on others. And there was a way to basically circumvent that check. So you would take a file upload that had a certain file type that had a check associated with it, and then you would trick the software to not doing that important check. Ergo, the vulnerability that that check was trying to prevent existed. But we're already in the technical reads. I don't want to get too far into how they actually exploit it. But there was a vulnerable library version on the machine. That library published a new setting that made it more secure. And then the Ruby on Rails community updated their call to that underlying library to pass that flag to make it more secure. And so the fix was kind of twofold. You had to make sure that your underlying library version, which in this particular case is called libvips, was sufficiently upgraded that it had the support for that flag that turned off this vulnerable thing. And then you had to run a version of active Storage which was sufficiently patched to pass that flag, which didn't exist in a previous version of that. So the patch was two steps.

Speaker A: Okay, so backing up a step, this vulnerability gets released. Someone writes about it, notes it, People become aware of it pretty quickly afterwards. What is the. Basically the first step or steps. Yay, there was a patch. And yeah, that, that helps. But what are the steps in between? There's this vulnerability, and this one is particularly horrible. And we patch real quick.

Speaker B: This episode is brought to you by Delta Systems, which is what Aaron and I do when we're not talking through microphones to you, the people of the Internet. We've got a really, really great software team here, and we love to work with cool people on cool projects. So if that sounds like you and you've got a problem or you're in some kind of a jam, go to Deltasystems.com, grab a time with us. We can beat up on your problem together, and if there's a fit there, amazing, we'll help you out. So Deltasystems.com, grab an appointment, and, hey, maybe we can work together. There's a lot that goes into this. A lot of times when there's a really bad cve, the company that publishes or the organization that publishes them will hold it back and they will start reaching out to people who are vulnerable and basically letting people know about this ahead of time before they publish it publicly to the rest of the world. And the idea is just once the CV is published, everybody has access to it, and most of us are good, most of us are good, and some of us are not. So at that point, once the actual CVA publication comes, the race is on, so to speak. And people who, I don't know this to be true, but certainly there were a lot of people who were like, oh, there's lots of vulnerable applications out there. They're writing tools to scan for them and find these vulnerabilities, to actively exploit them. In this case, because the vulnerability is so bad and people don't have the patching posture that they should, there will be people using this exploit for a long time. So just to kind of sidestep your. Your question a little bit. So there is a process where we try to actually make the world more secure before we just publish what the vulnerability is to God and everybody. They publish officially, the race is on, so to speak. People start trying to exploit it because they know what it is, and then, um, they don't publish all the details about how to do the exploit. They just publish kind of enough details for us to patch it, but a smart person will be able to figure it out. So at that point you kind of do the first thing, which is to stop the vulnerability. So there's a good chance at this point you have not been exploited. And so you want to close the window during which you could be exploited. So what we did immediately is we patched, we upgraded these library versions and uh, underlying library and then active storage versions and published everywhere that we was using that. We scanned all of our projects to figure out everything that was vulnerable to this and then we put out publish patches for everybody to basically stop the bleed. So the world knows about this now we know about it, we're patched. So the people who just learned about it aren't going to be able to steal our stuff. And then we move into the second phase, which is kind of remediating anything, because if somebody exploited you, you determine what they got access to and then you start pulling all that stuff back. So in this particular case, we have these secrets files that ship with Rails Core, where all of you've got an email service that has a password and you talk to ChatGPT and Claude and you have your API keys for those and you have a capture and a Google Maps integration and all that stuff. Well, all the things are the server has to have basically username and password or an access token to access those features. And those are in this particular case, if somebody owned this, they got all that stuff, they could dump arbitrary files off the server, including the encryption key and the secret file. But that basically means, is that we have to assume that all of those are exploited, which means we have to roll all of them. And so that's kind of the second phase, once you close the gap is it's like, well, everything here was potentially stolen this. So now we're going to go update all of it. That process began too. We went into AWS and all of our AWS credentials open udundalist, OpenAI, Anthropic resend and sendgrid and mailchimp. I mean just every service that your software integrates with securely is now potentially exploited. That queue is potentially out there. So you go through and you just roll everything.

Speaker A: Got it. So is this stepping back from this particular situation, but these exploits in general that Might require this type of rolling of credentials across. Also presumably client structures and not M, just internal ones. How do you make the assessment or how do you decide that? Let's assume we have been compromised and there assume the worst case versus not

Speaker B: the security answer to that is you assume that you're breached and you do all the rolling. There's some practical limitations to this. I try not to be so dogmatic that all common sense falls out the window. So I'll give you two people that we considered as use cases or as samples. So we had one customer, they uh, did have a file upload. It was available to admin type users. They are in an early beta. And people who had access to do that file upload? There were three of them. All of them are personal friends of the founder. So the vulnerability existed. We patched it. There were three people who potentially could have exploited it, all of whom are personal friends of the founder. We did not take the time to roll secrets on that project. Was there a window of active exploitation? There was. And the most prudent thing to do would be to roll them if you're going to be 100% correct or anal retentive about it. But in this case I chalked it up to acceptable risk. These people who had access to do this aren't technical people, they aren't experts, they aren't security researchers and they're not trying to abuse their friend who they're baiting his software for him. Cool. So we didn't patch. On the other hand, I have a we'll call it consumer facing app that has tens of thousands of people in it. I think there's a file upload for both employees of the organization of which there are many hundreds and also consumers of the application for which there are many thousands. Non negotiable. We rolled that immediately and went through the process of the batch. We went through the process of rolling everything because there were just too many people who had access to the vulnerable feature for us to feel good about not doing that.

Speaker A: Yeah, no, no, that makes total sense. So I know that an aspect of that is in this rotation or rolling of credentials. There are other things that, that could adversely affect things that may be coded within the application, other systems that may be relying on integration points or things along those lines. So how is it that you make sure that while you're rotating these, you're not also breaking everything in your path?

Speaker B: Well, it's challenging and this is one of the areas we're following best practices all the way along is really helpful, because if you have a different username and password for your staging environment and for your production environment and for every application that you deploy, then suddenly you don't have issues where you have to roll credentials for one application and it breaks another one or another one or another one. Same thing for API keys. Like if we use Google Maps, we use a different API key for every single project. So when there's a vulnerability, it's just one API key. And even, let's just say every project that had those API keys were all breached and they all had to be rolled. It now lets us roll them one at a time where we can say, like, that one's vulnerable, replace it. And after doing the replacement, we don't have five sites that are offline while we're trying to go through and do that. But the short answer is that rolling these credentials, there's huge potential for downtime and integrations breaking and stuff like that, and requires a lot of care and it's not fun work. And that's why we're very thankful. These kinds of CVs don't come out. These things don't come out very often, which is great.

Speaker A: That actually is going to be my next question. You know, how frequently does something, maybe it's not even a 9.5, but maybe it's something that does necessitate both a patch and a rolling of credentials or some high level of adjustment beyond just simply applying a patch. How often does that come along?

Speaker B: I've only seen a handful in my career. This is probably the second, second or third worst one that I've seen. There's been some really concerning ones with operating systems that were really bad, but they were a lot harder to exploit than this, I would argue. So I'm old now, my career, I've been doing this for more than a decade, 15 years, something like that. I've seen maybe two or three really bad ones like this. And obviously, like, if you're in an older stack or in a community that's less security focused or something, it might be more common.

Speaker A: So obviously in today's AI landscape, and you read about it almost every day, is there a greater concern now with how frequently these may be occurring? Or maybe even worse, how bad, like how many nines and above we're going to start seeing because of AI kind of running around?

Speaker B: Yeah, that has been the big prediction of everyone. Um, if that is materializing, it's not doing so in explosive fashion. So it's very likely that AI is going to help us find vulnerabilities. And lock things down and secure our code. And there's notable headlines of it doing just that. But I guess I'm not going to cry wolf or predict the apocalypse. I think that we might find these things a little bit more regularly. But like so many things in life, in the world, the good guys outnumber the bad guys, and that's why the system keeps working.

Speaker A: I'm just about to say that you want to have the people who are using it for use the power for good. Great power, great responsibility and all that. That makes total sense. And I think it's also important just to call out the fact that this level of issue is something that's pretty infrequent. It's not one of those kind of things where you necessarily need to worry every time it's security release comes out. But at the same time, I, uh, want you to expand just a little bit on, on those best practices that you had mentioned, because we even talked about it in a recent podcast. It's one of those small, incremental but consistent things that you do that also helps to mitigate this even before it happens. And that could just be something as simple as patching. We've also talked about technical debt on this show, so give us some, some best practices there that would potentially preempt some of these issues from directly affecting us.

Speaker B: Two things that we use to talk about security. So one of them is we call it concentric circles of security. So every individual circle is stupid, but the net sum of them together is how you stop from having these catastrophic failures. The other kind of metaphor that we use is if you have four or five pieces of Swiss cheese and you're trying to draw a line all the way through all of them, every layer has holes in it. But to have 10 layers have all the holes line up in your piece of Swiss cheese doesn't tend to happen. And that's why the concentric circles, you can mix the metaphors. So there's a lot of stuff. So she's good. Up to date operating systems and starting kind of on the metal. If you're managing your own servers, you need to have a patching policy in place where you're going, logging onto your operating systems and keeping them up to date. If you're using a managed service of some kind, and you need to make sure that your settings are set up to keep you updated and you have some kind of cadence to make sure that you're just even security patches installed and all that monthly, quarterly, something like that, you adopt A policy and you just run it getting further, you need to make sure that your network controls are as tightened down as they can be. There's been many, many, many vulnerabilities that we have been vulnerable to that were unexploitable in environment because our server only talked to the world on the Internet port and the Internet port was running a uh, web server. And so yep, we had a vulnerable server online but nobody could talk to it provably. So that's where firewall rules come in. It's another concentric circle keeping your personal, using something like a VPN and not allowing access to everywhere. So I don't know if we were ever vulnerable to this but like SSH potentially could have a vulnerability. And if we find out that all of our Linux servers have an SSH vulnerability, I'm going to keep on resting. Great. Because we lock everything down to our VPN and so you got to be my VPN to exploit that SSH vulnerability and so on. I think for people uh, who are using modern web applications having some kind of a build related process that scans for dependencies, for vulnerabilities and dependencies in your libraries. Not that I love to say good things about GitHub, but GitHub has a service called Dependabot and it sits there and scans all the code that's on GitHub and lets people know when there's problems. We have some Rails tooling that looks at our gem file and says hey, this library dependency is vulnerable, you need to run an update. And that actually runs as we're pushing code up to the remote to be peer reviewed. We can't actually do that if there's a vulnerable library and that starts the conversation. It's no one thing. There are thousands of things to do and we try to do a little bit of all of them.

Speaker A: It's worth noting, and these are conversations that Jessa and I have with clients as well, that when you are building an application or you're rebuilding or refactoring an application or you're uh, a first to market or startup or whatever, software is a living breathing thing. It's never. We get to the point of launch and here's your application and now it's in its current state, will always run, it will always be secure and will always be safe. It's no different than having up to date antivirus libraries on your machine or anti malware things. There are things that just as good practice and good hygiene, to use Justin's terms, that you need to continue doing and it's important that you budget for those sorts of things. We used to get questions all the time of, uh, well, what should I expect in terms of ongoing costs after this application is built? And that's how we know that we've got a mindful client. They may not know exactly how good their question is that they're asking, but they are at least thinking long term. I have this application and I am recognizing that things are going to be maintained or upkept even if I am not building out new features. And so making sure that as an organization, especially a SaaS, that you are budgeting for these kind of baseline, really maintenance tasks and for people to keep an eye on it, even if it's not your own staff, is incredibly important in the life cycle of your application.

Speaker B: I always like to use homeownership, um, as a metaphor. Nobody ever is like, no, the price of the house is $400,000. What do you mean I have to keep working on it? You said if I bought it, I'd buy it at this price. While rips go bad and landscaping needs to happen and siding or whatever. If you live in the Midwest, you deal with snow related stuff.

Speaker A: Yeah, pick your metaphor. You got a car, you have to change the oil, put gas in it, occasionally your tires are going to get worn down. It's the same type of thing even with software.

Speaker B: If there's like one thing that I can be a champion for having build, build and publish related tooling to security scanning, I know Ruby on Rail is the best, but there's analogous tools for every stack, certainly. But every time that we push code up, there's a stat, uh, code check that looks for all sorts of common types of vulnerabilities. And we have the thing that scans the library for the library dependencies, for vulnerable dependencies and so on. And just having that to where you don't have to remember to do it, just have it be part of writing code, I think is the biggest one.

Speaker A: Absolutely. Well, that kind of brings us to our close, Justin. Anything that, uh, I'll take your question. Anything that, uh, I didn't ask you that I probably should have, or any notes that you want to make here.

Speaker B: If you are running a Ruby on Rails application that takes file uploads and no one on your team has talked to you about this, we will put a link to the CVE in the show, notes and all that. And you should go talk to your tech team about what to do. And if you're taking uploads from people who aren't known to you very closely personally. You should also consider rolling all of your credentials. So if this is news to you, I hope you're not hearing about it from me. And your tech team is taking care of you. But this is a big, bad one, and it's going to be a lot of people who are scanning the web for vulnerable versions of Ruby on Rails apps. So just give this one patch. It's important.

Speaker A: Absolutely. Justin, thanks so much, man. Really always appreciate your technical expertise. I'll, uh, lean on you heavily for that, my friend.

Speaker B: Hey, this is a fun one. I got to. I just got to answer questions, did no prep. It was wonderful.

Speaker A: Well, thanks as always, my friend. And thank you all out there. Appreciate you being a listener and a subscriber. And if you're not, think about doing so. We got a lot of great information that's out there and some new episodes coming your way soon. So thank you all from Sas, that app. We'll catch you next time.

Speaker B: Keep the maps sassy.

Speaker A: Thanks for cruising along with us on SaaS that app. We hope you grabbed some insights that were inspiring, actionable, or at least entertaining

Speaker B: if you enjoyed the show. Don't forget to subscribe and leave a review until next time.

Speaker A: Keep building, keep growing, and keep those apps sassy.