The Truth About The Recent Vulnerability Spike w/ Aaron Mitchell | Episode 198

Speaker A: Right now it's this tsunami of vulnerabilities that are getting reported. Everyone that gets, that comes in maintainer has to look at but we're also maintaining the older versions of those. Right. And so that puts a strain on our engineering team. Just to give you like a perspective on this, last year there were 17 vulnerabilities reported. It's in all of 20, 25, 17. In just the last two months we've seen over 40 of these vulnerabilities get reported. And each one of those we've got to go test, we've got to go validate, we've got to go make sure that we fix and then we've got to go make sure that all the builds that have to go out against those fixes work correctly. Right. Um, so uh, it's a lot of work to keep up with what AI is doing to uh, the tsunami of vulnerabilities.

Speaker B: Welcome to Software Leaders Uncensored. I'm Steve Chaplin and every episode I get the privilege of sitting down with tech leaders, engineering leaders, to understand exactly what's working, what's breaking inside scaling tech and software companies. After over 195 episodes, we've seen a lot of patterns forming and that's what this show is really about. In fact, we recently released our why Software Projects Fail report. That that was based on feedback from uh, our first 195 episodes. Today we have Aaron Mitchell, CEO at Herodoves, who is going to talk about the challenges he's dealing with in the open source industry and how AI is both helping them and hurting them. Um, Aaron, welcome buddy.

Speaker A: Hey Steve, great to be here. Thanks for having me on.

Speaker B: You bet. If you would tell our listeners about your company.

Speaker A: Yeah, I'm going to use the unsexy version of what we do. We're like the nursing home of the Internet.

Speaker B: All of uh, the description, all of

Speaker A: the code that you write to build these really cool applications, it's built on open source. And if I'm a big company, that open source eventually goes end of life. And it can be really difficult to maintain hyperspeed on my roadmap while also keeping up to date on the latest versions of these open source libraries. And so what we do is we come in all that old open source that you've, that uh, you're using, that is no longer supported, it's not secure, it's not compliant, we provide a drop in replacement of that. It takes five to 10 minutes to switch over to and then you're up and off to the races. Again, using our proprietary software. And yeah, and that's what we do. We do it for we. Hundreds of different libraries, most popular libraries across the whole Internet ecosystem.

Speaker B: Awesome. And let's dive in further. And your company, congrats on a major funding round you had last year. Uh, it sounds like you guys just have phenomenal traction in the market. Tell our listeners about the work model of your company. Um, and we talked pre show how you're fully remote. Talk about that and how you manage that.

Speaker A: Yeah, we, uh, yeah, we have a fully remote team. We're about a hundred employees right now, and we've grown really fast. I mean, I joined the company right as we were getting going. We were six employees. Now we're at a hundred. Uh, that was four years ago. So I think it's, uh, it's been really exciting. We actually use this tool, it's called Gathertown. If you haven't heard of it, you should check it out. But it's really good for remote teams. And uh, my kids come into my office and they think I'm playing a video game. But it's like a virtual office. You got your own desk that you sit at, that you can decorate it however you want. And then people just come up and, and when they come to your desk, your cameras both pop on and you start chatting. And so, uh, it's been a really cool tool for us to keep that in office feel. But yeah, we have people all over the world from Greece to Italy to Singapore, um, down Argentina and uh, yeah, fully distributed, fully remote company. It's been awesome.

Speaker B: Interesting. And so I haven't heard of that tool, Gather Town. I'm uh, and I also run, run a fully remote company, sonatafy Technology. And I'm always looking for better stuff to give more feel. And we've tried a bunch of tools in the past and always just been widely disappointed. So you gave a good one to try out.

Speaker A: I recommend it. I mean, we were just doing like the slack and zoom thing before we, we adopted Gathertown. And it, it really does make a big difference. It's. It's weird, you know, cause you're just like this little video game character walking around an office, but for some reason it just does it for you.

Speaker B: Well, and you know, being the CEO of a fully remote company, which I can relate. I mean, the leadership team has to work a lot harder to make sure there's a culture within the company. And you know, there's phenomenal benefits from working remotely, but to make sure you have that culture, you know, I think is 10 times the work in the remote world versus if you were in an office. What's your view?

Speaker A: Yeah, it's something that we're very intentional about is building that culture. And we actually just last week we just did a company wide off site down in the Dominican Republic at uh, a secrets resort down there. And uh, it was awesome. Yeah, it was so cool. I mean getting the face to face time, you can't really replace it. And so we try and be very intentional about allowing people to get face to face, have the meetings and you know, we do a bi monthly town hall where it's kind of, we lay it all on the table and say here's what's going on, how's everyone doing? And so yeah, we've tried to be intentional but you're right. I mean it's, it's. I sometimes miss going into an office and just feeling the energy of that in person. But um, we've been able to make this remote workforce work really, really well for us.

Speaker B: So you said you have uh, around a hundred employees and growing. How large is your product and engineering organization?

Speaker A: Yeah, I, I don't, I don't have the exact number in front of me, but we're between 40 and 50 engineers.

Speaker B: Okay.

Speaker A: Yeah.

Speaker B: And, and so you're worldwide, you're remote. You know, educate our listeners how you manage the kind of the time zone complexities and the cultural differences.

Speaker A: Yeah, I mean we, we really love all the cultural differences. It's fun, it keeps, it keeps it spicy. Hero devs, you know, so. But yeah, like uh, we, we run off of east coast, American east. And if you have some team members that are over in Europe, then we'll have standups earlier in the day. If you've got team members that are on the west coast or you know, in Australia or New Zealand, then we'll run later in the day usually. And um, but for the most part we've been able to make, make that work really well and people have been super gracious, you know, coming in that are working in sort of these off time zones to try and adapt to the, the time zones that we all work in as, as a company. So it's been, it's been really, really good.

Speaker B: So you guys maintain these massive open source libraries. You're the, the nursing home of the Internet to help companies stay in compliance, easily swap code in and out with that. You know, talk about, you know, the engineering challenges you're dealing with that are unique to your business.

Speaker A: Yeah, well, I mean I like to say a lot of People know how to drive a car, but there's a very few amount of people that actually know how to fix it. And the same is true for these big open source libraries. There's a lot of engineers that know how to use them to build these amazing applications. But uh, in a lot of cases you can count the number of people that know how to actually fix those libraries on one hand. Right. And so we've had to be.

Speaker B: And all of those work for you.

Speaker A: Yeah, that's exactly, exactly. Yeah, yeah, we're recruiting them all the time too. So if you're one of the holdouts, come join us. Right. But no, I mean, that's one of the reasons that we have to foster a really strong culture is because we need to be able to attract that talent. And yeah, we have dozens of open source maintainers that work at HeroDevs that are the original maintainers of some of the Internet's biggest projects.

Speaker B: So constantly making sure you're attracting that talent. Because I would agree, I would almost say the open source community could kind of almost be cult like and not in a bad way. But you know, uh, so how, how do you go about that to make sure you guys are constantly able to recruit that best talent who truly understands the open source space?

Speaker A: Yeah, I mean, I think it's a, it's a very special community. And I don't think I fully appreciated just how cool the open source community is until you're really in it. But uh, you, you have to, you have to weigh, you can't come in and be overly predatory. Right. Or overly capitalistic. This is a whole, it's an entire industry built on altruism and people giving freely of their time to make the world a better place, make the Internet a better place. And so, uh, I think with that in mind, like, we have done a lot to give back, both financially and our time back to those open source communities. We donate. A lot of our engineers time is spent working on the latest versions of these open source projects that we support. And we spun up a $20 million open source sustainability fund where anyone can come in and apply and receive funds from Herobevs to make sure that they're maintaining best practices as they think about the lifecycle of their, their project. And so yeah, uh, it really is just like a give value to get value in open source. And I think companies that try and come in and say we just want to make money on this thing, they uh, they get rejected. You know, they're the cancer of the open source community. And we don't like that.

Speaker B: So in the open source space, how, how do you manage compliance and regulation and industry specific requirements? There's, there's a lot out there to be able to manage within that. Tell us how you attack that.

Speaker A: It's, it's a lot. Um, and we're seeing more open source adoption than we've ever seen in the history of the world. Right now if you go look at Black Duck puts out their annual open source security and risk assessment and we've, it shows that the amount of open source that the average commercial code base has has tripled in just the last five years. Right. So companies are pulling in more open source components. Uh, it's getting, if you put your engineering manager hat on for a second and you think about, I'm running on average 1100 different open source components in my project, you think about each one of those has their own life cycle, right? Each one of those has a support date and an unsupported date on every version. And m managing that across a portfolio of 1100 different components is almost impossible. And so this is, this uh, is a problem that hero devs we're trying to solve right now for these, for these uh, managers and these engineering teams is giving you visibility into what are you running that's not supported, what are you running that's been abandoned by the maintainer? And how do we help you get secure and compliant and supported on those libraries?

Speaker B: So how do you do that for, I mean are there industries that you've seen generally don't use open source?

Speaker A: It's, it's, I was going to make

Speaker B: that assumption, but I wanted to ask the expert, so how in the world do you keep these repositories of open source code that you provide SLAs against? How could you do that for if every industry is using it?

Speaker A: Yeah, well Steve, the answer is it's really damn hard to do. And, and you know, I'm not gonna, I'm not gonna blow smoke up your ass and say that we've, we've got it all figured out over here but uh, it really comes down to how you prioritize it. Right? We look at what are the libraries that are critical for these enterprises to be running on and that they're using that, that that are standing up these really critical systems. And um, and then we start there and we say okay, now let's, it's really customer driven. We say what else are you running in your stack that you're having trouble keeping up to date? And we'll go and spin up engineering teams and resources to go and make sure that we can provide support for those libraries. So you kind of just, it's, it's sort of the snowball that you uh, once you get pushed down the hill, you uh, you just gotta figure out how to keep up with it. And uh, you gotta hire really, really good engineering talent. And AI is also unlocking a lot of new productivity gains and helping us stand up new projects faster, helping us resolve security issues faster. And so there's definitely some, some accretive uh, tailwinds there as well.

Speaker B: So as AI over the past couple of years has come mainstream, how has that both helped and hurt your business model and the open source community?

Speaker A: Yeah, I mean it's, it's definitely added a level of complexity to our business. There's some really strong tailwinds that uh, there are more CVEs, vulnerabilities that are getting reported against open source libraries than we've ever seen before. We thought, we thought this year was probably going to be like a 33% increase over last year. Now with Claude and, and Codex and all of these LLMs pounding these open source libraries for, to look for vulnerabilities, that number looks like it could as much as double. Like we might be approaching a hundred thousand CVEs that get reported this year, which is going to break the scale. And, and what it's doing, what it's doing Steve, is it's a, it's creating a lot of urgency at the enterprise level to say I've got to get secure on all these libraries because the vulnerabilities are running rampant. But what it's also doing is it's putting the strain on the maintainers of these libraries right where they're, they're having to go test and validate every report that comes in and then put the fix out. And that we're seeing a lot of burnout in open source among maintainers. There's a lot of maintainers that are like, I just can't keep up anymore. I have a full time job working somewhere else and I can't keep up with this volume of vulnerabilities that are getting reported. And so, ah, that's a big risk too for the industry if you rely on relying on open source. You have to make sure that you've got your bases covered so that you don't get caught, you know, with uh, an unmaintained library that uh, somebody abandoned because the volume was just too, too great.

Speaker B: Well, I, I love how you answered my last question. Of how in the world do you, you know, do it across every industry, have SLAs? And I really respect your answer that it's freaking hard and it takes a lot of work because if you were going to say it was easy and you guys have figured everything out and it's simple, my job as this podcast host would be to destroy you. And uh, I love the honesty and straightforwardness and clearly the traction your company has gotten, the investment your company has gotten proves that you guys are kicking butt and taking names. And I just had to say I have the utmost respect for how you answered that.

Speaker A: Yeah, well, you know, you can see it with the wrinkles on my face and my hairline's a lot higher than it used to be too. So it hasn't come without its stresses. But it feels like we're uh, we're, we're headed in the right direction. And at the end of the day it's all about like, we just, we just want to solve the problems that our customers are having and, and do that as best we can. And, and so, yeah, we, we've got a long road ahead of us of, of how to get to where we want to go, but so far so good.

Speaker B: What slows you down and you're continuing to uh, build more repositories, more open source components. What's kind of, what are the biggest hurdles that slow your team down?

Speaker A: Yeah, I mean right now it's this tsunami of vulnerabilities that are getting reported. Everyone that gets, that comes in maintainer has to look at, but we're also maintaining the older versions of those. Right. And so that puts a strain on our engineering team and um, I mean, just to give you like a perspective on this, so Spring, a very, very ubiquitous popular Java framework that's used for backend development. Last year we've, we've, there were 17 vulnerabilities reported. It's in all of 20, 25, 17. In just the last two months, we've seen over 40 of these vulnerabilities get reported. And each one of those we've got to go test, we've got to go validate, we've got to go make sure that we fix and then we've got to go make sure that all the builds that have to go out against those fixes work correctly. Right. So, uh, it's a lot of work to keep up with what AI is doing to uh, the tsunami of vulnerabilities.

Speaker B: Talk about your perspective on what AI generated code is doing in the market with, you know, And I'm talking the vibe coded by non senior engineers. And you know, they're, they're happy that, wow, I was able to create this. What vulnerabilities do you predict are, are going to hit companies hard from those applications? And I'm talking about the ones not built by the engineering team that a, uh, company puts in production.

Speaker A: Yeah, I was just joking. My CFO just built this like amazing dashboard that shows all of our metrics and revenue and renewals and all that stuff. And he just vibe coded this in Claude. My CFOs never written a line of code in his life. He couldn't even spell HTML right? So like, so I was joking. I was like, hey, do you know, like, do you know what open source Claude pulled in to build this, this amazing dashboard? He's like, hell no, I have no idea what, what, what it pulled in. Well, you multiply that across every employee at the company who's vibe coding these apps using cloud or Codex, and uh, you start to see like they're uh, accruing a mountain of tech debt and we haven't paid the bill on the tech debt yet.

Speaker B: Right.

Speaker A: Like we're still so early into this that everything's up to date, everything's working, but you look down the road a year or two and you know the, the bill collectors are going to start coming and we're going to see. No, like we, we actually have these systems that were vibe coded that we built a couple years ago. We don't even really know how they work. And they're also using all this old open source that we've got to figure out what to do with, it's gonna get ugly in my opinion. I, uh, I uh, think it's, it's gonna get really out of hand. And if you put yourself in the, in the seat of a CISO or a security leader and you think about like, well, if my CFO doesn't know what he's running, then definitely I don't know what he's running and I don't know what everyone else is running at the company either. That's a really scary security thought.

Speaker B: I think that is a great example of, you know, the buzz that software engineers aren't going to be needed and that's going to go away. Which I call bullshit. There's tons of efficiencies. There's great things. But the example you just gave of uh, the risk of non engineered trained people putting code out there and not knowing what's in it, the vulnerabilities and what it could do. And I like how you said it, that, you know, the industry really hasn't seen the tech debt, the vulnerabilities yet from those. But it's going to hit hard. I agree with you. And I think that's just a crisp argument of why software engineering is not going away.

Speaker A: Yeah, the job looks very different than it did a year ago, but uh, you still have to have software engineering talent on your team or you're going to architect your way into a nightmare. Right? A hairball of problems.

Speaker B: So yeah, uh, let's say there's a company out there, let's call them a manufacturing company and let's say they're a $500 million manufacturing company that has open source apps that were built by people that are no longer there may or may not have a focus on maintaining those. Like what, what do you see the tech debt issues that uh, companies like that are gonna face and how they fix it?

Speaker A: Yeah, I mean, I was actually just talking to one of our customers the other day and he was like, we bought this company and uh, there's a couple people running on this app and we make, you know, half a million dollars on it. But it's got all kinds of stuff in there that uh, we didn't know about and it's dead, like it's not maintained anymore. It's a big security risk for me, it's a big compliance risk and so I gotta, I gotta figure out what to do. Uh, right. And so they came to herodevs. We were able to get em taken care of. But that's a story we're gonna hear over and over again is my CFO built this application. I don't know how it runs, he doesn't know how to maintain it. We have no engineering oversight over it. What do I do? Right, and that's where I think we're going to start to see a shift from if I'm running open source, I've got so much of it in my environment that I just need a software vendor on the other side that can put SLAs behind it, that can make sure it's always maintained and secured. And I, uh, think we're going to start to see a movement towards that more and more as this starts to propagate across all different industries. Did you see the, did you see the black duck report? It actually asked a question of what percent of or the stat was the percentage of companies that are knowingly in violation of their stated AI policies. Did you see that stat?

Speaker B: I, uh, did not keep Going, do

Speaker A: you have any guesses as to what percent of companies are knowingly violating their AI security policies? It was 74%. Yeah. And, and they were asking security leaders. Right, so the security leaders knew we put out the AI policies. We know people aren't following these things. And, uh, it's just like the productivity and efficiency gains you can get from AI. It's too great to, uh, to wait around for, you know, the security team to approve that.

Speaker B: No. On that same vein, Wall Street Journal the other day this past week had an article how CISOs are quitting. Like they're too stressed. They. And you know, with the vulnerabilities that AI has, has, has brought to the table. And because of lazy or poor engineering that has gone into it, because it's all AI generated that they're like throwing their hands up, like, saying it's a lose, lose situation to be a ciso, um, in that type of environment. What's your take on that?

Speaker A: You know, I haven't seen that. You got to send me that article, though. That's, that's really interesting. Uh, yeah, I mean, I'm hearing it, we're hearing it from our customers. It's like, man, I thought, like, I just got a handle on this and then all of a sudden AI comes on and it's like, wild, wild west again. I have no way to, to know even how to start solving this issue because it used to just be the engineering team that I, I would go and, you know, like, hit on the head and be like, hey, get your, get your shit in order. Now it's the whole company. And none of these people are classically trained in engineering best practices and security best practices. And uh, yeah, I, frankly, I, I kind of commiserate with them. It's a, it's a tough job right now.

Speaker B: It is a challenge. And I am not a security expert, so I'm going to caveat before I make my comment. I probably know more than most, but I can't compete with a, uh, CISO in any means. And I'm not plug in, but I feel like there was a few years here where security was kind of ignored and not as much importance as it was. And that, that's just coming full circle right now to a head. Uh, do you. What's your take on that? Agree, Disagree.

Speaker A: Yeah, I would, I would have to agree with that. I would have to agree. I think it's, it's, it's all, it's always cyclical. Right. Like, you get there was like this concept of shadow it back, uh, I don't know. That was five, seven years ago, right? Where you had people coming in with their mobile phones, their laptops, pulling in sensitive information and doing who knows what with it. We've kind of come back around to that with now AI where it's like it just got a handle on how to manage all that and now it's like, okay, well everyone's a developer. I don't. What do I do now? I think it's probably 10 times worse than it used to be. But yeah, it's, it's. We're in unprecedented time, Steve.

Speaker B: No doubt. All uh, right, so let's just say you fell asleep for two years and you said I just want the company on, on cruise control and you're, you don't change engineering your processes. What's the first thing that would break in the next 12 to 24 months? Not saying you would do that, but

Speaker A: yeah, no, you're asking the tough questions. I like it. I would. I think that the, the, the, the way that people are solving the problems that we've traditionally solved is fundamentally changing. Right. It used to be that you were uh. I'm stuck on this old library and stuck means like it's going to take me months or even years sometimes to get to the latest in life version. Well, I think that timeline is significantly compressed. Right. It's not as hard for me to get to the latest version because I have these AI tools that are, that are really good at doing the grunt work of, of engineering of hands on keyboard and, and uh, helping you migrate. And so I'm compressing those timelines. If we just kind of sat back and said we're just going to cruise control, then I think our business, we would, we would see some really significant headwinds where companies are like, well we don't really need to buy your stuff anymore because we can just get to the latest version a lot faster. So. But what's, what's changing is that the problem is going from this very acute pain of I'm stuck on this one library. It's very critical to. Now I have thousands of libraries that all have their own life cycles and their own timelines and I just can't wrangle it. All right. And so I need broad coverage across all of it. I need to be able to say I'm always secure on whatever I'm running, I'm always supported on it. I'm always compliant with whatever compliance standard I'm behooved into. And uh, and that's uh. So we solved the problem differently. Right now it's about creating visibility for these leaders to say, here's everything that you're running and the end of life status of each of those things. And then here is a drop in replacement. If you're on an old library and you can't interrupt your roadmap to take a couple months to migrate now you can use one of our drop in replacements to, uh, solve that. Right? It puts the power back into your hands to control your roadmap and migrate on your own time when it makes sense, uh, for the business.

Speaker B: So last question here, Aaron. If you could give one piece of advice to tech leaders and engineering leaders, uh, out there that are about to go through a journey similar to yours, what would you tell them?

Speaker A: You know, I would say make sure you've got the right team around you. I, I've been really fortunate to have an amazing team of, of leaders and an amazing, uh, team of, of individual contributors at, uh, Herobevs that make rolling into work really fun every day. And you know, we're in times where it's, it is really uncertain. It's scary. Like, who knows what's gonna happen to humanity because of AI? Like, you know, those conversations, they're swirling in everybody's heads right now. And so I think you just got to make sure that the people you're working alongside you enjoy working with. And, and if you don't, then go find somewhere where you do work. You do like the people you work with.

Speaker B: Well, I, I definitely agree with that. Uh, if you don't like what you do and the people you work with, it's hard to be the best at it.

Speaker A: That's right.

Speaker B: Say the least. Thank you so much, Aaron. That's it for this episode of Software Leaders Uncensored. If you're leading a tech organization and any of these patterns sound familiar, make sure to hit subscribe. Also, feel free to inquire, uh, on our new why Software Projects Fail report based on the first 195 episodes we've done here on Software Leaders Uncensored. I'm Steve Chaplin. See you next time. That's a wrap. Another great episode of Software Leaders Uncensored. Thank you guys so much for listening. I really appreciate it. It's a lot of fun doing this show. I love talking with great tech leaders. As a side note, if you're interested, check out my new book, Fail Hard, Win Big. This is the story of how I built 30 companies, failed on 20, and turned 10 into multimillion dollar wins. The common thread with all of them was custom software. Also, if you need software development help out there, my company, Synodify Technology, we deliver world class software development engineers out of Latin America. We have a solid track record of helping companies launch better products and faster. Thanks again for listening. Appreciate it and appreciate you.