Quality at Scale: Engineering in an AI-Driven World.

Speaker A: Welcome to the SMC Journal Show. I'm Scott Moore, your host. Thank you so much for being with me today. This video, we need to talk, we need to talk about all the AI generated software that's going on out there and how do we keep up with not only it, but the quality of it because some of it's not so great. And how should we as software engineers or estets or quality engineers have. How do we shift our mindset around this new world that we're looking at? Well, people I want to talk to about that are the guys at speed scale. So we're going to be talking with them today about this very topic. It's going to be interesting. You're going to want to be listening to every word because there is so much right now that is changing within it and how do we keep up with it? Find out on this video today. Let's get right into the interview. Hey Matt, welcome to the SMC Journal Show. Great to have you on.

Speaker B: Great to be here. Thanks Scott.

Speaker A: Uh, we're talking about making quality scale at a time when we've got AI driving everything. And so Matt, what do you see as some of the biggest issues or problems, challenges that are facing development teams right now because of this situation that we see in front of us?

Speaker B: So I kind of think of there being three major areas that are affecting, uh, affecting, you know, engineering teams and testing in general. The first one is I think everybody's trying to figure out how to use the AI coding tools in a responsible way. So the way this kind of manifests is you see everyone saying I'm uh, using Cursor or I'm using, trying to figure out how to get value out of Microsoft Copilot or whatever it is. And I think that that's, there's just a general lot of confusion about how much can these tools actually do. Now we work at a startup and so we're at the leading edge of this and you know, the truth is going to be that, at least that's our truth that we're living right now is that there's a, uh, there's a lot of things are good at and a lot of things that AI is not good at and every organization is going to have to figure out how to fit it together. So that's step one. Now step two is once they figure out how to use these AI coding tools, they're going to run into a second problem which is that uh, their, their tool chains do not appropriately address the problems of this new world. So if you are a person who thinks of their job as writing test scripts. Uh, the next couple years are going to be very challenging because that style of approach of writing test scripts is probably not going to work in a rapid iteration world. So that's number two is that there's a turnover in tool chains and that may extend into things like observability or the CI pipeline might actually fundamentally change. We're seeing a lot of cracks open up. Things like GitHub Actions. They recently changed their business model, for instance where they charge for external runners. There's all this thrash going on in the tool chain space. So I think that that's number two and then I think number three that people are becoming aware of is that the process of software engineering is going to have to change. So when I started my journey in software development, it was very early. We didn't even have things like CI pipelines and all this stuff just didn't even exist. Uh, software engineering was not a professional discipline yet now we evolved over 20 years or so for it to become a professional discipline and now we have lots of understanding of how to do product requirements, documents and all these systems have arisen. I think a lot of that stuff's going to change. And I'll be specific in what I mean is, uh, front end engineering is still going to be very relevant. There's going to be a lot of people making front end code still, but uh, it's going to be driven through the AI lens and that changes what makes a good front end engineer. So instead of the process being around, uh, reviewing lines of code and looking for inconsistencies in a merge request, instead what's going to be happening is, uh, curating the context that is necessary for an AI to do things like that. Now humans are still very much in the loop. It's still very relevant, but that's a totally different way of working and it's going to be a huge process change. So I think the next, you know, what's coming here is a radical rebuild, rebuild of these things. It's going to be very exciting, but it's also going to be a little bit scary for a lot of folks.

Speaker A: Okay, so some of that actually I'm not too happy about because I'm one of those guys who used to write scripts, you know. So how should the software quality engineer mindset change as a result of all this AI driven stuff?

Speaker B: Emotionally, I think we need to shift from being afraid, right? And we need to start thinking about the possibilities and the opportunities that are available. So specifically when it comes to quality assurance, um, I think a lot of folks went through being they thought of their jobs differently, uh, with the original, when, when we all moved towards s debts or you know, software development, engineer and test. Right. That was kind of a different way of thinking about it. I think something similar is going to happen on the QA side. So fundamentally what we know so far is that um, legacy testing tools are going to have a very hard time adapting, but the problem of quality is going to get much larger I think, because you're getting these AIs generating tremendous amounts of code that is very hard to review and very hard to understand. And so I think specifically the QA engineer is going to, going to need to change what they do to think. Instead of thinking about code, uh, paths for instance, or code coverage or these other metrics that didn't really work, to be blunt, they never really work these ideas as far as assuring quality, they're going to instead start needing to think about things like deterministic gates. So a QA engineer is going to think of themselves more like a manager of pipelines where it's uh, about the automation they can bring to bear, specifically using LLMs and the tools that will grow up around them and instead and stop thinking themselves as test runners. So let me give you a specific example of what we're doing at speed scale. Um, the way we're adapting is we're saying, uh, we're not going to focus on getting LLMs to write better test cases. That's fine, I mean that's okay if you want to do that. But what is actually more valuable to us is to say, um, how can we create a QA bot that a QA engineer will still need to give feedback to? Because we're not anywhere near the world of fully autonomous agents. That's still a ways away. So, so you're going to have a QA engineer and they're going to be saying how do I train the bot to do something useful? So in our particular case that is, um, instead of running a single load test or test run, instead uh, design test cases using the AI, design test scenarios using the AI and then go and build the infrastructure to make that repeatable. That is going to be a job that has lots of future and there's going to be a lot of people that need that because we're going to get 10 times as much code, which means we're going to have to complete new adaptable systems that are self healing, you know, a little bit more. And so that's kind of the general thrust which we can get in more detail as we move along.

Speaker A: The amount of AI generated code is, is massive. How do you efficiently keep up with that?

Speaker B: Uh, I, I think there's an idea. So in, in the LLM training space they have a concept called an adversarial AI. Now for those of you that don't live in that space, what it basically means is that an AI gets stronger when you use another AI to test it or to fight against it. Okay. And so I think that what happens here is that um, we as quality, uh, assurance professionals or people that focus on the quality are going to need to think of our jobs a little bit differently and say, um, how can I describe an adversarial system that uh, an LLM can use to test another LLM? So specifically I'll give you something we have at speed scale is. So we go and we um, obviously develop a lot of microservices and distributed systems. Right. And so instead of saying, instead of putting my focus of attention on uh, reviewing the particular interaction model or flows, that's kind of a typical end to end test model is to do flows. Instead, what I start, we start focusing on is how do we break the problem into smaller and smaller chunks that can fit into the context window of an LLM. So what I mean by that is once, uh, all the code is produced, it's too big for an LLM to reason about or even to test anymore. These apps are getting so big. And I think everybody who's in the testing space realizes this is that these apps have gotten too big, there's too much going on, too many interactions, all these problems have just multiplied. And so instead we need to think about how we can shrink the problem down into subsystems that can be tested in an automated fashion. So instead of treating these things as huge systems, think of them as isolation tests or isolation, uh, bits and pieces. And that will fit into the context window in LLM better. And so we can use more automation, um, to do that.

Speaker A: So what are the biggest risks associated with integrating all this AI generated code? Because we've had these public stories of how it's done some good things, um, by utilizing it, but it's also done some bad things. And it's not just syntax errors, it's actually deleted databases and things like that. So what are some of the biggest risks? I mean, how do you proactively test for something like that?

Speaker B: Well, to be clear, I have humans that delete databases as well. Ah, so that's a problem that's existed for quite a while. It's not just an AI problem, but we don't want to admit that, right. Uh, now we've come up with. So if you think about, let's, let's define the problem very clearly. Okay. And I'll give you a specific example from one of our customers in uh, in uh, a semi technical organization like a, like a, they're fast food so to speak. So. But they're technically sophisticated but they're not like Google, right? Levels of uh, investment in technology. What they recognize is that they have to go and move faster and provide more innovation using these AI tools. So they hired a set of engineers who are AI native folks who are trying on the bleeding edge. They're using the tools like Cursor and Claude, and they started with the assumption that they will not be editing code directly. Now what those people, uh, what we've seen from it is that we're getting an explosion of garbage code as a result. Because the AI is producing just garbage, just enormous amounts of garbage, right. Filling landfills full of code. Right. And by the time it gets to a merge request it's too late because at that point it has, the amount of garbage being generated has exceeded the ability of the recycling plant to go and process, to turn it into something useful. Right. Because the systems we have now are human driven. And so um, the alternative to that is to uh, the alternative to that is to start earlier in the process. So what I believe will happen, uh, and we're seeing evidence of this in the early adopters, is that they're moving the idea of testing and quality into the software development cycle onto the developer's desktop. So to make that concrete, the previous model was you would use uh, some sort of HTTP or like test driver tool, you know, Postman's, Bruno's, uh, there's a million of them out there and they, they're fantastic. Many of them are wonderful. Right. And so then the, the maximum amount of testing the developer would do was go and run a few equivalent of unit tests like here and there. I believe that's not going to work much longer. Right. We're going to have to start solving the problem of giving engineers the ability to run this audit, these automated test suites on their, on their desktop in a way that is much more intelligent than what we do now. So right now that means standing up the femoral environments, all this machinery that has to go into the stuff, I think that that's going to have to fall away because we're not going to Be able to keep up with the explosion of garbage otherwise.

Speaker A: I see.

Speaker B: So, yeah, closer to develop shift left I think is what we call it in the previous era. It's the same concept. All these concepts come back around again and this is shift left again. But it's gonna have to really shift left.

Speaker A: So give me a practical developer feedback loop that implements very quickly so you can validate this messy code, uh, without becoming a bottleneck if you're quality. Because that's the biggest thing that I hear from developers is don't give me any friction. We need to get this stuff out. How do you do that under those conditions?

Speaker B: So let me introduce two concepts that are a solution and then the practical exact steps that we take. Uh, the first one is that velocity, um, in software development is largely driven by moving things from an outer loop process into an inner loop process. And what I mean by that is very simple is that if the developer can do it all themselves, then that's an inner loop process. If it requires the help of someone else or another system, it's an outer loop process. Testing right now is intrinsically an outer loop process, um, because you need staging environments and test scripts and all these other things, right? So the core of what you need to do is move things into an inner loop process where the developer could do it themselves. Now we tried things like backstage and these IDPs and they're all very useful in their way, but they don't completely solve the problem. Right? So concretely what we do, what, what we've done is um, we have used uh, the concept of just in time observability to facilitate testing. So if you are familiar with AI coding agents, we, we tend to use things like Claude and Cursor. Those are just our favorites right now. They change by the day, but you know, that's what we do. And then we have introduced, uh, using our own product speed scale, we have introduced the idea where the AI now has access to, to the production systems, including all of the details of every payload. Now what that allows us to do, and you can go look at our videos if you're interested in it, is it allows the developer to quickly say what is a curated golden standard of production that I can use locally just to test and see if my new thing works. Right. Drawn from the real production system. Which brings me to my second point, which is determinism. So the main weakness of the AI coding tools is that they are stochastic. Now if you don't have a math degree, what stochastic means is that they don't always reach a specific answer. It's not math. Well, I mean it's kind of math, but it's not what you're used to, like algebra when you were young, where, uh, everything has a defined answer. AI doesn't work that way. It produces a range of answers and picks one right. Most of them are wrong and some of them are right. So your job as the quality assurance engineer is to introduce determinism. And so what we do is we have our AI hooked up to our local development environments provide, but it's plugged into a production system to where the AI can ask questions about how the app actually works in the real world. That's a form of determinism. So I think, um, so it's an MCP server. If you don't know about mcp, you know, I'd go research that, um, I think MCP eventually probably gets mutated into something more like agent to agent. It's just a different API standard. It's not important, just learn about both of them. But, uh, what will happen is you're going to have a lot more tooling available directly to the LLM on the developer desktop. That introduces determinism with the objective of creating more inner loops. So those are some specific things that we do.

Speaker A: So how does quality assurance processes keep pace in this modern way of doing software development?

Speaker B: So I believe from direct experience that, um, QA changes. First we have to change our mindset and then we have to change the specific things we do. Now the shift in mindset is most of the way quality assurance and estets are, um, they quantify the value of their job is how many bugs did I find or more accurately, how many problems did I prevent from making it to my customer? So my customer experience has gone up because I have been a gate on these things. I think you have to shift from that. I think what we need to do to enable velocity and more deployments, which is what we all want, more quality deployments, is we need to shift towards thinking of the problem as, uh, we need to think of ourselves as tools enablers. So instead of saying, I get value by that gate and saying, I found it, it's great, gotcha, gotcha, I got that bug. Instead, we need to look at it more of. I provided a set of deterministic tools that software engineers can use to QA their own software. I think that's how this is going to have to go because the idea of send it to a QA team, we're still going to have End to end tests. Come on. We're still going to need some of these things, right? You need a final check. But, but the, the, the, the truly valuable QA person in the companies I work with is the person who says, I have provided a better tool like the one I just said with mcp. But there's a ton of these examples. Um, you know, it could be something that automatically generates test scripts, it could be a unit test maker, could be a lot of things. I mean, I don't know exactly all the ways the AI will shape, but that's what the QA needs to be, is you're a tools provider or a platform engineer. In a way, it's almost like, you know, QA engineer sort of melds with platform juice. Instead of the person who finds it, you're the person that enables the engineer or the AI to find it with better tools.

Speaker A: What are the things that we probably need to rethink or overhaul or get rid of?

Speaker B: Okay, so we're gonna have to balance two things at once, which humans are always bad at, including me. Um, the first one is we're going to have to continue to, uh, keep our quality systems running, such as they are, because they are what we have to prevent these issues from happening. But the second thing is I personally believe that we're going to have to introduce more of the idea of digital twins. So if you think about the way that software is developed, it's a million copies of the same thing over and over. But we test it like it's still one big monolith. We still test like we're testing a single service because that's what the user sees, but that's not actually what's going on behind the scenes. You're getting this fan out, going on. And it started with things like Kubernetes. Kubernetes. The core thing that Kubernetes does is it allows horizontal scale. Okay. And also a bunch of other stuff, but mainly horizontal scale. But it does that by replicating things. But we haven't really, uh, adapted any of our testing philosophies around that. Not really. Right. And so I think that what we're going to end up having to do is create various kinds of digital twins that create replicas of the real system. Like it looks like it's really happening in, uh, order for the AIs to be able to fan out, if that makes sense. So we're going to have to, we're going to have to keep the whole thing running. It's true. And, but I don't I wouldn't continue to invest too much more in that stuff because I think to the degree we figured those problems out, they're figured out. Right? Like, you know, we know how to write test scripts and there's all kinds of good stuff out there and fantastic tools and open source and all kinds of great stuff. Right. But we got to get into this, we got to get out of that, uh, investing in that mindset and instead investing in how do I create these little replication environments or whatever it's going to be, right, These deterministic steps, um, and invest in those tools. Now obviously my company, Speed Scale, we'd love you to try our stuff but there's, there's actually a lot of different ways to solve this. Our way we use something called Traffic Replay, but it's just one aspect, one way to do this stuff.

Speaker A: And I was going to ask you about that too. How do you get Traffic Replay as part of a. Just a quality practice and how do you get ingrained into that? How do you move into that?

Speaker B: So the simplest thing that we, we have observed is uh, we deploy something called an EBPF collector, which is like a data collector from a production system that doesn't impact production. So that's the key thing is it doesn't add late, it doesn't add significant latency or whatever to production. Now the way you get started with it sounds counterintuitive is you give developers access to those recordings, whether it's QA or production or whatever, as long as it's a safe recording. Right. And then you get, you enable the developer to replay on a local desktop. Now in the Speed Scale world that means things like proxymock, that's like our free CLI tool, you can just go run it and record and playback. So on the developer's desktop you can basically uh, you can treat your system as ah, a lab that you can keep running experiments in that are safe without breaking production. Right. So I think that uh, uh, that's one of the key ways to get started. Now when you get one step beyond that. If we have an EBPF collector recording from production now that gets supercharged because the developer can ask, actually the AI coding tool, I should say can ask unstructured questions that you didn't see coming. So it's because the production data is right there, they can go see what the actual production app does. So that's a great way to get started is to, in our case it's using something like proxy mock, which is automatic chord and playback on Local machine, it's free, just get started. But there are also other kinds of open source tools. If you're willing to do a lot of work and kind of put them together yourself, you can try that as well. Getting started. But that there's tools like Go Replay and others that are, um, you know, a lot more manual labor but they will get the job done with it. Um, I think that to be honest, a lot of this, this uh, this area is developing and there's still a lot more uh, innovation to happen here to figure out exactly what the AI tools need. But I guess I would say get on that train, uh, as soon as you can, whatever you choose, get on that train.

Speaker A: Yep. I think that's wise words. Um, some of this stuff, you know, I've, I've learned a lot just talking to you about this today and some of this stuff, we've, we've got a lot to uh, go back and read up on to be ready for this. So thank you so much for being on the show.

Speaker B: Thank you, Scott.

Speaker A: So what are you seeing at your company or what are you experiencing right now? Are you utilizing these tools? Have you already made the shift? Are you concerned about the shift that's happening? Has um, it already affected you in a negative or a positive way? Leave me a comment down below and let me know, uh, if you like this video and you like the kind of content I'm producing. It would be great if you like the video and subscribe to the channel. Really, really does help out if you need to contact me. I'm pretty easy to find on social media as well as through email. You can always send me an email. Let me know what you'd like to see in future episodes of this show. Heyscottmcjournal. Uh.com is my email. Again, be great if you subscribe to the channel. It helps me make sure that I'm doing the right content for the right people. So for this episode of the SMC Journal, this is Scott Moore saying thank you for watching and we'll see you next time.

Speaker B: Bye.

Speaker A: By.