Could AI End Human QA?

The original title for this episode was AI Killed the QA Star, which would make more sense if you knew the eighties lore of the very first music video played on MTV. Um, that's a music television TV channel. Back, back in the days, how you watched videos before the internet. That was in 1981 and it was called Video Killed the Radio Star.

But I decided that a Deep cut title was too obscure for this conversation. Yet the question still remains, could the increased velocity of shipping AI generated code cause businesses to leave human based QA behind, presumably, because we're not gonna hire any more of them, and we don't want to grow those teams and operations teams just because we created AI code. And would we start relying more on production observability to detect code issues that affect user experience? And that's the theory of today's guest, Andrew Tunall, the President and Chief product Officer at Embrace.

They're a mobile observability platform company that I first met at KubeCon London this year. Their pitch was that mobile apps were ready for the full observability stack and that we now have SDKs to let mobile dev teams integrate with the same tools that we platform engineers and DevOps people and operators have been building and enjoying for years now. I can tell you that we don't yet have exactly a full picture on how AI will affect those roles, but I can tell you that business management is being told that similar to software development, they can expect gains from using AI to assist or replace operators, testers, build engineers, QA and DevOps.

That's not true, or at least not yet. But it seems to be an expectation. And that's not gonna stop us from trying to integrate LLMs into our jobs more. So I wanted to hear from observability experts on how they think this is all going to shake out.

So I hope you enjoyed this conversation with Andrew of Embrace. Hello. Hey, Bret. How are you doing?

Andrew Tunal is the president and chief product officer of Embrace. And if you've not heard of Embrace, they are, I think your claim to me the first time talked was, you're the first. Mobile focused or mobile only observability company in the cloud native computing foundation. Is that a correct statement?

I don't know if in, probably in CNCF, right? Because like we're, we were certainly the first company that started going all in on OpenTelemetry as the means with which we published instrumentation. obviously CNCF has a bunch of observability vendors, some of which do, mobile and web run, but, completely focused on that. Yeah.

I would say we're the first. Yeah. I mean, we can just, we can say that until it's not true. the internet will judge us harshly, for whatever claims we make today.

Yeah. well, and that just means people are listening. Andrew. Okay, I told people on the internet that we were going to talk about So, the idea that you gave me that QA is at risk of falling behind because if we're pretty If we're producing more code, if we're shipping more code because of AI, even the pessimistic stuff about, you know, we've seen some studies in the last quarter around effective use of AI, it is anywhere between negative 20% And positive 30 percent in productivity, depending on the team and the organization.

So let's assume for a minute it's on the positive side, and the AI has helped the team produce more code, ship more releases, have, even in the perfect world that I see teams automating everything, there's still usually, and almost always, in fact, I will say in my experience, 100 percent of cases, there is a human at some point during the deployment process. Whether that's QA. Whether that's a PR reviewer, whether that's someone who spun up the test instances to run it in a staging environment.

There's always something. So tell me a little bit about where this idea came from and what you think might be a solution to that. Yeah. I'll start with the, the belief that I mean, AI is going to fundamentally alter the productivity of software engineering organizations.

I mean, I think the, CTOs I talk to out there are making a pretty big bet that it will. you know, there's today and then there's, okay, think about the, even just the pace that, that AI has evolved in, in the past, the past year and you know, what it'll look like given the investment that's flowing into it. But if you start with that, the claim that AI is going to kill QA is more about the fact that, we built, Our software development life cycle under the assumption that software was slow to build and relatively expensive to do so.

And so if those start to change, right, like a lot of the, systems and processes we put around, our software development life cycle, they probably need to change too. because ultimately, If you say, okay, we had 10 engineers and formerly we were going to have to double that number to go build the number of features we want. And suddenly those engineers become, more productive to go build the features and capabilities you want inside your apps. I find it really hard to believe that those organizations are going to make the same investment in QA organizations and automated testing, etc.

to keep pace with that level of development. The underlying hypothesis was like productivity allows them to build more software cheaper, right? And like cheaper rarely correlates with adding more humans to the loop. yeah, the promise.

Like I make this joke and it's probably. Not true in most cases, cause it's a little old, but I used to say things like, yeah, CIO magazine told them to deploy Kubernetes or, you know, like whatever, whatever executive VP level C, CIO, CTO, and you're at that chief level. So I'm making a joke about you, but, funny. but that's when you're not in the engineering ranks and you maybe have multiple levels of management, you get that sort of.

overview where you're reading and you're discussing things with other suits. You're reading the suits magazines, the CIO magazines, the uh, IT pro magazines and all that stuff. yeah, the point I'm making here is that, people are being told that AI is going to save the business money. I, I think I've said this to several podcasts already, but, you weren't here.

So I'm telling you that I was in a media event in London, KubeCon, and, they give me a media pass for some reason, because I make a podcast. So they act like I'm a journalist standing around all real journalists and pretending, but I was there and I watched an analyst who's, from my understanding, I don't actually know the company, but they sound like the Gartner of Europe or something like that. and the words came out of the mouth at an infrastructure conference, the person said, my clients are looking for humanless ops.

And I visibly, I think, chuckled in the room because I thought, well, that's great. That's rich that you're at a 12, 000 ops person. Conference telling us that your companies want none of this. these are all humans here doing this work.

the premise of your argument about QA is my exact same thoughts was nobody is budgeting for more QA or more operations personnel or more DevOps personnel, just because the engineers are able to produce more code with AI, in the app teams and the feature teams. And so we've all got to do better and we've got to figure out where AI can help us because if we don't. They're going to, they're just going to hire the person that says they can, even though maybe it's a little bit of a shit show.

my belief is that software organizations need to change. The way they work for an AI future, so that might be cultural changes, it might be role changes, it might be, the, you know, the word, the words like human in the loop get tossed around a lot when it's about engineers interacting with AI, and the question is, okay, what does that actually look like, right? Are we just reviewing AI's PRs and, you know, kind of blindly saying, yep, like they wrote the unit tests for something that works, or is it like we're actually doing critical thinking, critical systems thinking, unique thinking that allows us as owners of the business and our user success?

to design and build better software with AI as a tool. and it's not just QA, it's kind of like all along the software development life cycle, how do we put the right practices in place, and how do we build an organization that actually allows us, with this new AI driven future, you know, whether it's agents, doing work on our behalf, or whether it's us just with AI assisted coding, to build better software. And, yeah, I'm interested in what that looks like over the next couple of years.

That's kind of the premise of my, the new Adjetic DevOps podcast. And also as I'm building out this new GitHub Actions course, I'm realizing that I'm having to make the best practices. I'm having to figure out what is risky and what's not because no one has really figured this out yet in any great detail and, in fact, at, KubeCon London in April, which feels like a lifetime ago, there was only one talk around using AI anywhere in the DevOps and operations path. To benefit us.

It was all about how to run infrastructure for AI. And granted, KubeCon is an infrastructure conference for platform engineering builders and all that, so it makes sense. But the fact that really only one talk, and it was a team from Cisco, which I don't think of Cisco as like the bleeding edge AI company, but it was a team from Cisco, simply trying to get And workflow or maybe you would call it like an agentic workflow for PR review, which in my case, in that case, I'm presuming that humans are still writing their code and AI is reviewing the code.

I'm actually, I was just yesterday, I was on GitHub trying to figure out if I, if there was a way to make branch rules or, some sort of automation rule that if the AI, wrote the PR, the AI doesn't get to review the PR Yeah, yeah, right. we've got this coming at us from both angles. We've got AI in our IDEs. We've got, multiple companies, Devon and GitHub themselves.

They now have the GitHub copilot code agent. I have to make sure I get that term, right. GitHub copilot code agent, which will write the PR and the code to solve your issue. And then they have a PR code copilot reviewer.

Agent that will review the code. It's the same models, different context, but, it feels like that doesn't, that's not a human in the loop. So we're going to need these guardrails and these checks in order to make sure that code didn't end up in production with literally no human eyeballs ever in the path of that code being created, reviewed, tested, and shipped. cause we can do that now.

Like we did, we Yeah, totally, you're totally good. And I mean, you can easily perceive the mistakes that could happen too, right? I mean, before I, I took this role, at Embrace, I was at New Relic for four and a half years, and before that I was at AWS. And so, obviously I've spent a lot of time around CloudFormation templates, Terraform, etc.

You can see a world where, you know, AI builds your CloudFormation template for you and selects an EC2 instance type because the information it has about your workload is optimal for this EC2 instance type. But in the region you're running, that instance type's not freely available for you to autoscale to. And pretty soon, you go try to provision more instances, and poof, you hit your cap. Because, that instance type just doesn't have availability in Singapore.

And, us as humans and operators, we learn a lot about our operating environments, we learn about our workloads, we learn about the, the character, the peculiarities of them that don't make sense to a computer, but are, like, based on reality. over time, maybe AI gets really good at those things, right? But the question is, how do we build the culture into kind of be guiding our, our, you know, army of assistance to build software that really works for our users, instead of just trusting it to go do the right thing, because we view, you know, everything as having a true and pure result, which I don't think is true.

a lot of the tech we build is for people who build consumer mobile apps and websites. I mean, that is what the tech we build is for. And you can easily see, you know, some of our, our engineers have been playing around with using AI assisted coding to implement our SDK in a consumer mobile app that, and it works quite well, right? You can see situations where, an engineer gets asked by somebody, a product manager, a leader to say, Hey, you know, we got a note from marketing.

They want to implement this new attribution SDK that's going to go. build up a profile of our users and help us build more, you know, customer friendly experiences. You have the bots go do it, it tests the code, everything works just fine. And then, for some reason that SDK makes a uncached request out to US West 2 for users globally that, you know, for your users in Southeast Asia instead of adding an additional six and a half seconds to app startup because physics.

And, what do those users do? if you start an app and it's. sits there hanging for four, five, six seconds, and you don't have to use the app because it's you know, you're waiting for your boarding pass to come up and you're about to get on the plane. You probably perceive it as broken and abandoned.

And to me, that's like a reliability problem that requires systems thinking and cultural design around how you're engineering organization to avoid. one that I don't think is immediately solved with AI. right. observability is only getting more complex even, you know, it's a cat and mouse game.

I think in all, in all regards, and I think everything we do has yin yang I'm watching organizations that are full steam ahead, aggressively using ai, and. I'd love to see some stats. I don't know if you have empirical evidence or if you sort of anecdotal stuff from your clients, but where you see them accelerate with AI and then almost have a pulling back effect because they realize how easy it is for them to ship more bugs. and then sure, we could have the AI writing tests, but it can also write really horrible tests, or it can just delete tests because it doesn't like them because they keep failing.

it's that to me multiple times. In fact, I think, Oh, what's his name? Gene, not, it wasn't Gene Kim. There's a, a famous, it might've been the guy who created Extreme Programming.

I think I heard him on a podcast talking about he wished he could make certain, all of his test files have to be read only. Because he writes the tests, and then he expects the AI to make them pass, and the AI will eventually give up and then want to rewrite his tests. And he doesn't seem to be able to stop the AI from doing that, other than just denying, deny, or cancel, cancel, cancel. And there's a scenario where that can easily happen in the automation of CI and CD.

Where, you know, suddenly it decided that the tests failing were okay. And then it's going to push it to production anyway, or whatever craziness ensues. Do you see stuff happening on the ground there? That's did you read that article about, the, AI driven store, a fake store purveyor that Anthropic created named Claudius?

That, when it got pushed back from clo, the Anthropic employees around what it was stocking, its fake fridge. Its fake store with it called security on the employees to try to displace them. I mean, it's, I think what you're pointing to is like the moral compass of AI is not necessarily, it, like that's a very complex thing to solve, right? Is is this the right thing to do or not?

Because it's trying to solve the problem however it can solve the problem with whatever tools it has. Not whether the problem is the right one to solve. And right is, I mean, obviously, you know, even humans struggle with this one. Right.

I've just been labeling it all as taste. like the AI lacks taste Yeah, that's like in a certain team culture, if you have a bad team culture, deleting tests might be acceptable. I used to work with a team that would ignore linting. So I would implement linting.

I'd give them the tools to customize linting and then they would ignore it and accept the PR anyway. And did not care. They basically followed the linting rules of the file. Now they were in a monolithic repo with thousands and thousands of a 10 year old Ruby app.

But they would just ignore the linting. And I always saw that as a culture problem for them. That I as the consultant can't, I would always tell the boss that I was working for there that I can't solve your culture problems. I'm just a consultant here trying to help you with your DevOps.

You need a full time person leading teams to tell the rest of the team that this is important and that these things matter over the course of years as you change out engineers. You can't just go with, well, this file is tab. This file is, spaces. And this one follows, especially in places like Ruby and Python and whatnot, where there's multiple, sort of style guys and whatnot like that.

But I just call that a taste issue or culture issue. AI doesn't have culture or taste. It. It just follows the rules we give it.

we're not really good yet as an industry at defining rules. I think that's actually part of the course I'm creating is figuring out all the different places where we can put AI rules. We've seen them put in repos. We've seen them put in your UI and your IDEs.

Now I'm trying to figure out how do I put that into CI and how do I put that into even like the operations if you're going to try to put AIs somewhere in your, somewhere where they're going to help with troubleshooting or visibility or discovery of issues, like they also need to have. Taste, which I want to change all my rules files to taste files. I guess that's my platform. I'm standing.

Yeah. Cause it, Yeah, I mean, at the same time, you probably don't want to be reviewing, hundreds of PRs that an AI robot is going through, just changing casing of legacy code to, meet your, to the, Change obviously introduces the opportunity for failure, bugs, etc. And if it's, you know, it's somewhat arbitrary simply because you've given it a new rule that this is what it has to do. I mean, I had a former co worker many years ago, like 15, 20 years ago, who, was famous for just, going through periodically and making nothing but casing code changes because that's what he preferred.

And it was just this, endless stream of, trivialized changes on code we hadn't touched in months or years. That's it. that would inevitably lead to some sort of problem, right? And because, The toil of that.

I'm just like, yeah, that you're giving me heartburn just by telling me about that story. yeah, it's, well, I've gotten over the PTSD of that. It's a long time ago. So, okay.

So, what are you seeing on the ground? do you have some examples of how this is manifesting in apps? I mean, you've already given me a couple of things, but I'm just curious if you've got some more, I kind of alluded to it at the very beginning. obviously, in my role, I talk to a lot of senior executives about directionally where they're going with their engineering organizations, because I think, the software we build, it does a bunch of tactical things, but at a broader sense, it allows people to measure reliability as expressed through, you know, our customers staying engaged in your experience by virtue of the fact that they otherwise liking your software are having a better technical experience with the apps you build, that the front end and ultimately measuring that and then thinking through all of the underlying root causes is a cultural change in these organizations and how they think about reliability.

It's no longer just like are the API endpoints at the edge of our data center delivering their payload, you know, responding in a timely manner and error free. And then the user experience is well tested and, you know, we ship it to prod. And. That's enough.

it really isn't a shift in how people think about it, but when I talk to them, I mean, a lot of CTOs really are taking a bet that AI is going to be the way that productivity gains, I won't say make their business more efficient, but they do allow them to do more with less, right? You know, like it or not, especially over the past few years, I mean, in B2B, we're in the B2B SaaS business, there's been, you know, times have definitely been tougher than they were in the early, 2020s, for kind of everyone, the, I think there's a lot of pressure in consumer tech with tariffs and everything to do things more cost effectively, and first and foremost on the grounds, this is a change we are seeing, whether we like it or not, and, you know, we can argue about whether it's going to work and how long it'll take, but the fact is that, like you said, you mentioned that the CIO magazines, leadership is taking a bet this is going to happen, and I think as we start to talk about that with these executives, the question is like, is the existing set of tools I have to cope with that reality good enough?

And, yeah, I guess my underlying hypothesis is it probably isn't for most companies, right? if you think about the, the world of, you know, a web as an example, like a lot of, a lot of companies that are consumer facing tech companies will measure their core web vitals, in large part because it has SEO impact. And then they'll put a, exception handler in there, like Sentry that grabs, you know, kind of JavaScript errors and tries to give you some level of impact. around, you know, how many are impacting your users and whether it's high severity and then you kind of have to sort through them to figure out which ones really matter for you to solve.

so take, existing user frustration with, human efficiency of delivering code and the pace, the existing pace. users are already frustrated that Core Web Vitals are hard for them to quantify what that really means in terms of user impact and whether a user decides to stay on the site or not. so, yeah. and the fact that they're overwhelmed with the number of JavaScript errors that could be out there because it's really, I mean, you go to any site, go to developer tools.

look at the number of JavaScript errors you see, and then, you know, take your human experienced, idea of how you're interacting with the site, and chances are most of those don't impact you, right? They're just like, it's a, it's an analytics pixel that failed to load, or it's a library that's just barfing some error, but it's otherwise working fine. So take that and put it on steroids now where you have a bunch of AI assisted developers doubling or tripling the number of things they're doing or just driving more experimentation, right, which I think a lot of businesses have always wanted to do.

But again, software has been kind of slow and expensive to build. And so If it's slow and expensive to build, my thirst for delivering an experiment across three customer cohorts when I can only, deliver one, given my budget, just means that, I only have to test for one variation. We'll now triple that, or quadruple it, or, and, you know, multiply that by a number of, arbitrary user factors. It just gets more challenging, and I think we need to think about how we measure things differently.

once the team has the tools in place to manage multiple experiments at the same time, that just ramps up exponentially until the team can't handle it anymore. But if AI is giving them a chance to go further, then yeah, they're just, they're going to do it. I mean, yeah, I mean, you're going to get overwhelmed with support tickets and bad app reviews and whatever it is, which I think most people are. Most business leaders would be pretty upset if that's how they, are responding to reliability issues.

I was just gonna say, we've already had decades now of pressure at all levels of engineering to reduce personnel. you need to justify every new hire pretty significantly unless you're funded and you're just, you know, in an early stage startup and they're just growing to the point that they can burn all their cash. having been around 30 years in tech, I've watched operations get, you know, Merged into DevOps. I've watched DevOps teams get, which we didn't traditionally call them that.

We might have called them sysadmins or automation or build engineers or CI engineers, and they get merged into the teams themselves. And the teams have to take on that responsibility. I mean, We've got this weird culture that we go to this Kubernetes conference all the time and one of the biggest complaints is devs who don't want to be operators, but it's saddled on them because somehow the industry got the word DevOps confused with Something. And we all thought, Oh, that means the developers can do ops.

That's not what the word meant, but we've, we've worked, we've gotten to this world where I'm getting hired as a consultant to help teams deal with just the sheer amount of ridiculous expectations a single engineer is supposed to have, that not the knowledge you're supposed to have, the systems are supposed to be able to run while making features. And it's already, I feel like a decade ago, it felt unsustainable. So now here we are having to give some of that work to AI. When it's still doing random hallucinations on a daily basis, at least even in the best models.

I think I was just ranting yesterday on a podcast that SWE bench, which is like a engineering benchmark website for AI models and how well they solve GitHub issues, essentially, and the best models in the world. Can barely get two thirds of them. Correct. And that's, if you're paying the premium bucks and you've got the premium foundational models and you are on the bleeding edge stuff, which most teams are not because they have rules or limitations on which model they can use, or they can only use the ones in house, or they can only use a particular one.

and it's just, it's one of those things where I feel like we're being pushed at all sides. and at some point, it's amazing that any of this even works. It's amazing that apps actually load on phones. it's just, it just, it feels like garbage on top of garbage, on top of garbage, turtles all the way down, whatever you want to call it.

So where are you coming in here to help solve some of these problems? yeah, that's fair, yeah. I'll even add to that, All of that's even discounting the fact that new tools are coming that make it even simpler to push out software that like barely works without even the guided hands of a software engineer who has any professional experience writing that code. some of the vibe coding tools, like our product management team uses, largely for rapid prototyping.

And I can write OK Python. I used to write OK C sharp. I have never been particularly good at writing JavaScript. I can read it OK, but like when it comes to fixing a particular problem, quickly get out of my depth.

That's not to say, I couldn't be capable of doing it. It's just not what I do every day, nor do, I particularly have the energy when I'm, you know, done with a full workday to, go teach myself JavaScript. And, I'll build an app with one of the Vibe coding tools as a means of communicating how I expect something to work. So if you have any questions, feel free to reach out to me, and I'll be happy to answer them, and I'll be excited to hear back from you.

And for coming. and I'm like, ah, it's better to just delete the project and start all over again. And, you know, if you can make it work, that doesn't necessarily mean it'll work at scale. It doesn't necessarily mean that there aren't like a myriad of use cases you haven't tested for as you click through the app in the simulator.

and so, you know, I think the question is, okay, you know, given the fact that we have to accept more software is going to make its way into human beings hands, because we build software for human beings, it's going to get to more human beings. How do we build a, a reliability paradigm where we can measure whether or not it's working? And I think that stops focusing on, kind of, I guess to go back to the intentionally inflammatory title of today's discussion, it stops focusing on like a zero bug paradigm where I test things.

I test every possible pathway for my users. I, you know, have a set of requirements, again, around trivialized performance and stuff. And, trying to put up these kind of barriers to getting code into human hands. And I just accept the fact more code is going to get into human hands faster at a pace I can't possibly control.

And so, therefore, I have to put measurements in the real world. So, I use a lot of different tools around my app so that I can be as responsive as possible to resolving those issues when I find them, which is I guess my, you know, charity majors who co founded Honeycomb, she, her and I were talking a few months ago and big fan of stickers, she shipped me an entire envelope of stickers and they're all like, you know, ship fast and break things, right? I tested production, stuff like that.

And somehow I feel like, you know, in our world, because we build observability for front end and mobile experiences, like web and mobile experiences, I feel like that message just hadn't gotten through historically. part of it's because like release cycles on mobile were really slow. you had to wait days for an app to get out there. Part of it was software is expensive to build and slow to build and so getting feature flags out there where you can operate in production was hard.

Part of it was just the observability paradigm hadn't shifted, right? Like the, the paradigm of measure everything and then find root cause had not made its way to frontend. It was more like measure the known things you look for, like web exceptions, like core web vitals are on mobile, look at crashes, and that's about it. And the notion of, okay, measure whether, users are starting the app successfully, and when you see some unknown outcomes start to occur or users start to abandon, how can you then sift through the data to find the root cause?

Hadn't really migrated its way. And that's what we're trying to do. we're trying to bring that paradigm of how do you define the A, for lack of a better term, the APIs. The things that your humans interact with, with your apps, the things they do, you don't build an API in your app for human beings, you build a login screen, you build a checkout screen, a cart experience, a product catalog.

How do we take those things and measure the success of them and then try to, attribute them to underlying technical causes where your teams can have, better have those socio technical conversations? so that they can understand and then resolve them, probably using AI, right? as we grow. but like it, it allows better, system knowledge and the interplay between real human activities and the telemetry we're gathering.

Yeah. Have you been, I'm just curious, have you been playing with, having AI look at observability data, whether it's logs or metrics? have you had any experience with that? I'm asking that simply as a generic question, because the conversations I've had in the last few months, it sounds like AI is much better at reading logs than it is at reading metrics or dashboards or anything that sort of lacks context or, you know, it's not like we're putting in alt image messages for every single dashboard graph.

And that's probably all coming from the providers just because if if they're expecting AI to look at stuff. they're gonna have to give more context, but it sounds like it's not as easy as just giving AI access to all those systems and saying, yeah, go read the website for my dashboard, Grafana, and figure out what's the problem? I've seen it deployed in two ways, one of which I find really interesting and something we're actively working on, because I think it's just a high degree of utility and it's, You know, given the kind of state of LLMs, I think it's probably something that's relatively easy to get right, which is the, the notion of, when you come into a product like ours, or a Grafana, or, you know, a New Relic, you probably have an objective in mind, maybe you have a question you're trying to ask, what's the health of this service?

Or, I got page on a particular issue. I'm going to, I need to build a chart that shows me like the interplay between latency for this particular service and the success rate of, you know, some other type of thing, more like database calls or something like that. Today, like people broadly have to manually create those queries and it requires a lot of human knowledge around the query language or schema of your data. And I think there's a ton of opportunity for us to simply ask a human question of You know, show me a query of all active sessions on this mobile app for the latest iOS version and the number of, traces, like startup traces that took greater than a second and a half.

And have it just simply pull up your dashboard and query language and build the chart for you quite rapidly, which is a massive time savings, right? And. It also just makes our tech, which you're right, can get quite complex, more approachable by your average engineer. Which, you know, I'm a big believer that if every engineer in your organization understands how your systems work.

And the data around it, you're going to build a lot better software, especially as they use AI, right? Because now they, they understand how things work and they can better provide instructions to the robots. so I think that's really a useful, interesting way. And we've seen people start to roll that type of assistant act, functionality out.

The second way I've seen deployed, I see mixed results, which is I mean, an incident, go look at every potential signal that I can see related to this incident and try to tell me what's going on and get to the root cause, and more often than not, I find it's just a summarization of stuff that you, as an experienced user, probably would come to the exact conclusions on, I think there's utility there, certainly, it gets you a written summary quickly of what you see. But I do also worry that, it doesn't apply a high degree of critical thinking.

and you know, an example of where lacking context, I mean, it wouldn't be very smart, right, is you've probably seen it, every traffic chart, around service traffic, depending upon how it runs, tends to be pretty lumpy with time of day. Because Most companies don't have equivalent distribution of traffic across the globe. Not every country across the globe has an equivalent population. And so you'd tend to see these spikes of where, you know, you have a number of service requests spiking during daylight hours or the Monday of every week because people come into the office and suddenly start e commerce shopping.

And you see it taper throughout the week, or you taper into the evening. I think that's normal. You understand that as an operator of your services, because it's unique to your business. I think the AI would struggle, lacking context around, your business to understand, somewhat normal fluctuations, or the fact that, You know, a marketing dropped a campaign where there's no data inside your observability system to tell you that that campaign dropped.

It's not a release. see your AI is lacking context. It's lacking the historical context that the humans already have implicitly. Yeah, and I mean that context might be in a Slack channel where marketing said we just dropped an email and you know an email drop so expect an increased number of requests to this endpoint as you know people retrieve their, their special offer token or whatever that will allow them to use it in our checkout flow.

today like just giving, if we provided that scope to a, an AI model within our system we would lock that type of context. Yeah, if I'm not creating any of these apps, I'm sure they've already thought of all of this, but, the first thing that comes to mind is well, the hack for me would be give it access to our ops slack room, Right. we're probably all having those conversations of oh, What's going on here? And someone in someone's who had happened to get reached out to for marketing was like, well, you know, yesterday we did send out a, you know, a new coupon sale or whatever.

So, yeah, having it read all that stuff might be, necessary for it to understand the, because you're right. it's not like we have a dashboard in Grafana that's number of marketing email, you know, the email sent per day or the level of sale we're expecting, based on in America, it's the 4th of July sale, or, you know, some holiday in a certain region of the world. Or a social media influencer dropping like some sort of link to your product that suddenly, you know, that it's a new green doll that people attach to their designer handbags.

I don't know like anything about what the kids are into these days, but it seems kind of arbitrary and like I, I would struggle to predict that. Let me put it that way. they're into everything old. So it's into everything that I'm into.

Yes. all right. So if we're talking about this at a high level, we talked a little bit about before the show around how embrace is thinking about, observability and particularly on mobile, but you know, anything front end there. the tooling ecosystem for, engineers on web and mobile is pretty rich.

but they all tend to be just like hammers for a particular nail, right? It's you know, How do we give you a better craft reporter, better exception handler, how do we go measure X or Y? some of the stuff that we're thinking about, which is really how we define the objective of OB observability for, the coming digital age, right? Which is you know, as creators of user experiences.

I think my opinion is that we shouldn't just be measuring like crash rate on an app. We should be measuring, are users staying engaged with our experience? And when we see they are not, sometimes, crashes, I mean, the obvious, the answer is obviously they can't, right? Because the app explodes.

But, I think, you know, I was talking to a senior executive at a, a massive food delivery app. And it's listen, we know, anecdotally, there's more than just crashes that make our users throw their phone at the wall. you're trying to You're trying to order lunch at noon and something's really slow or you keep running into a, just a validation error because we shipped you an experiment thinking it worked and you can't order the item you want on the two for one promotion. you're enraged because you really want the, you know, the spicy dry fried chicken hangry.

and you want two of them because I want to eat the other one tonight. and you've already suckered me into that offer, you've convinced me I want it, and now I'm having trouble, completing my objective. And broadly speaking, the observability ecosystem on the front end really hasn't measured that, right? We've used all sorts of proxy measurements in, out of the data center because the reliability story has been really well told and evolved.

Over the past 10 to 15 years in the data center world, but it just really hasn't materially evolved in the front end. And so, a lot of that's like shifting the objective from how do I just measure counts of things I already know are bad to measuring what users engagement looks like and whether I can attribute that to change in my software or defects I've introduced. So, that's kind of the take. just about anybody who has ever built a consumer mobile app has Firebase Crashlytics in the app, which is a free service provided by Google.

It was a company a long time ago that. Crashlytics that got bought by Twitter and then got reacquired by Google. it basically gives you rough cut performance metrics and crash reporting. Right.

I would consider this like the foundational requirement of any level of, app quality, but to call this NSAID app quality, I think, you know, our opinion is that would be a misnomer. So we're going to go kind of go through what this looks like, right. Which is it's giving you things. You would expect to see, a number of events that are crashes, et cetera, and, you can do what you would expect here, crashes are bad, so I need to solve a crash, so I'm going to go into a crash, view stack traces, get the information I need to actually be able to resolve it.

and, you know, I think we see a lot of customers before we talk to them who are just like, well, I have crash reporting and I have QA, that's enough. there's a lot of products that have other features. So like Core Web Vital measurements on a page level, this is Sentry. It's a lot of data, but I don't really know what to do with this.

beyond, okay, it probably has some SEO impact. There's a, you know, Core, bad Core Web Vital or slow, you know, slow something for a render on this page. How do I actually go figure out root cause? But again, right, this is a single signal.

So this is kind of, you don't know that, whether or not the P75 core web vital here that is, considered scored badly by Google is actually causing your users to bounce. And I think that's important because I was reading this article the other day on this like notion of a performance plateau, like there's empirical science proving that like faster core web vitals, especially with like content paint and interaction to next paint. et cetera, improve bounce rate materially, like people are less likely to bounce if the page loads really fast.

But at some point, like if it's long enough, there's this massive long tail of people who just have a rotten experience and you kind of have to figure out, I can't make everyone globally have a great experience. Where's this plateau where, I know that I'm improving the experience for people who I'm likely to retain and improve their bounce rate versus I'm just, you know, going to live with this. And so we kind of have a different take, which is like we wanted to center our experience less on just individual signals, and more on like these flows, these tasks that users are performing in your app.

So if you think about the key flows, like I'm breaking these down into the types of activities that I actually built for my end users. And I want to say, okay, how many of them were successful versus how many ended in an error, like something went truly bad, right? You just could not proceed. Versus how many abandoned, and when they abandoned, why, right?

Did they abandon because they, they clicked on a product catalog screen, they saw some stuff that they didn't like? Or did they abandon because the product catalog was so slow to load, and images slow, like slow, so slow to hydrate? That they perceived it as broken, lost interest in the experience and ended up leaving. And so, the way you do that is you basically take the telemetry we're emitting from the app, the exhaust we collect by default, and you create these start and end events that allow you to then, we post process the data.

We go through all of these sessions we're collecting, which is basically a play by play of like linear events that users went through. And we hydrate the flow to tell you where people are dropping off. and so you can see like their actual completion rates over time, you know, obviously it's a test app, so there's not a ton of data there, but what gets really cool is we start to, we start to build out this notion of once you see. the, issues happen, well how can I now go look at all of the various attributes of those populations under the hood to try to specify which of the things are most likely to be attributed to the population suffering the issue?

So that could be an experiment. it could be a particular mobile, a particular version they're on. It could be, an OS version, right? You just shipped an experiment that isn't supported in older OSs, and those users start having a bad experience.

And then each of those gets you down to what we call, this, user play by play session timeline, where you basically get a full Recreation of every part of the exhaust stream that we're gathering from you interacting with the app or website just for reproduction purposes. once you've distilled here, you can say, okay, now let me look at that cohort of users. And so I can do pattern recognition, which I think is pretty Hmm. So for the audio audience that didn't, that didn't get to watch the video, what are some of the key sort of, if someone's in a mobile and front end team and this is actually going back to a conversation I had with one of your Embrace team members at KubeCon in London, what are some of the key changes or things that they need to be doing?

If I guess if I back up and say the premise here is that if I'm it's there's almost like two archetypes. What am I trying to say here? There's two archetypes that I'm thinking about. I'm thinking about me, the DevOps slash observability system maintainer.

I've probably set up. Elk or, you know, I've got the key names, the Loki's, the Prometheus, the, the Grafana's. I've got all these things that I've implemented. I've brought my engineering teams on board.

They like these tools. They tend to have, especially for mobile, they tend to have other tools that I don't deal with. They might have platform, like the, the Traditionally, right? We're obviously trying to change that.

But yeah, traditionally, they have five or six other tools that don't play into the ecosystem, observability ecosystem you have set up. Yeah. So, so we're on this journey to try to centralize, bring them into the observability world. you know, traditional mobile app developers might not even.

Be aware of what's going on in the cloud native observability space and we're bringing them on board here. Now suddenly they get even more code coming at them that's slightly less reliable or maybe presents some unusual problems that we didn't anticipate. So now, you know, we're in a world where suddenly what we have in observability isn't enough. yeah, you're a potential solution.

What are you looking at for behaviors that they need to change to things that people can take home with them? And I mean, I guess the way I think about it is, right, the way, the reason observability became so widely adopted in server side products was because in an effort to more easily maintain our software and to avoid widespread defects of high blast radius, we shifted from a paradigm of like monoliths deployed on bare metal to virtualization, which was, you know, various container schemes kind of that has right now most widely been around, Kubernetes and microservices because you could scale them independently and you could deploy them independently.

Right. And that complexity of the deployment scheme and the different apps and services interplaying with each other necessitated an x ray vision into your entire system where you could understand system wide impacts to your, the end of your world. And the end of your world, from the most part became your API surface layer, the things that served your web and mobile experiences. And, you know, there are businesses that just serve APIs.

Right, but broadly speaking, the brands we interact with as human beings serve us visual experiences that we interact with. right. It's the server team managing the server analytics, not so much the client device analytics that right. the world has gotten a lot more complicated in what the front end experience looks like.

And you could have a service that consistently responds and has a nominal increase in latency and is well within your alert thresholds, but where the SDK or library designed for your front end experience suddenly starts retrying a lot more frequently, delivering Perceived latency to your end user. And, and so I think the question is, could you uncover that incident? Because if users suffer perceived latency and therefore abandon, what metrics do you have to go measure whether or not users are performing the actions you care about them performing, whether that's attributable to system change?

In most instances, I don't think most observability systems have that. and then the second question is, right, so, and by the way, Bret, that's the underlying supposition that in a real observability scheme, mean time to detect is important, is as important, if not more so, than mean time to resolve. the existing tooling ecosystem for frontend and mobile has been set up to optimize mean time to resolve for known problems where I can basically just count the instance and then alert you.

So, And, you know, the lack of desire to be on call, like I've heard this stupid saying that there's no such thing as a front end emergency, which is like ridiculous. If I'm, you know, if I'm a major travel website and I run a thousand different experiments and a team in, Eastern Europe drops an experiment that affects users globally, 1 percent of users globally in the middle of my night. That, makes the calendar control broken and that some segment of that population can't book their flights.

That sounds a lot like a production emergency to me. I, that has material business impact in terms of revenue. Or the font color changes and the font's not readable yeah, I guess I am imploring the world to shift to a paradigm where they view like users willingness and ability to interact with your experiences as a reliability signal. And I think the underlying supposition is that this only becomes more acute of a problem as the number of features we ship.

I guess I'm starting with the belief that from what I hear, people are doubling down on this, right? They're saying we need to make software cheaper to build and faster to build. Because it is a competitive environment and if we don't do it, somebody else will. and as that world starts to expand, the acuity of the problem space only increases.

Yeah. I think your theory here, matches are well with some other like we're all kind of in this little bit of we've got a little bit of evidence. We hear some things and we've got theories. We don't have years of facts of exactly how AI is affecting a lot of these things, but other compatible theories I've heard recently on this, actually with three guests on the show.

that I'm just coming to mind. One of them is, because of the velocity change and because of the, it is this increase in velocity, it only is going to increase the desire for standardization in CI and deployment, which those of us in, if you've been living in this Kubernetes world, we've all been trying to approach that. Like we're all leaning into Argo CD as one of the, as the number one way on Kubernetes to deploy software. you know, we've got.

This GitOps idea of how to standardize change as we deliver in CI. It's still completely wild, wild west. You've got a thousand vendors all in a thousand ways you can create your pipelines. And hence the reason I need to make courses on it for people, because there's a lot of art still to it.

We don't have a checkbox of this is exactly how we do it. And in that world, the theory is. Right now that maybe AI is going to allow us to act where it's going to force us to standardize because we can't have a thousand different workflows or pipelines that are all slightly different for different parts of our software stack, because then when we get to production and we have started having problems, or if the AI is starting to take more control, it's just going to get worse because the AI Yeah, and at the end of the day, right, standardization is more for the mean than the outliers.

And I think a lot of people assume they're like, oh, we can do it right because we have hundreds of engineers working on like our, you know, our automation and stuff. It's do you know how many companies there are out there that are not technology companies? They're a warehouse company with technology. Right.

And like standardization allows them to more often than not build good technology, to measure correctly, to deploy things the right way, like we're all going to interact with it in some way, right, and as the pressure for more velocity and them building technology speeds up, the need for you know, a base layer of doing things, where it's consistently right, but only increases. And I think that's, you know, for the world it's a challenge, for us it's an opportunity. Yeah. Awesome.

I think that's a perfect place to wrap it up. we've been going for a while now. but, Andrew, io, right? That's, Yeah.

www. embrace. io. let's just bring those up.

We didn't show a lot of that stuff, but, the, in case people didn't know, I made a short a few months ago about Embrace or with one of the Embrace team members at KubeCon. which we had touched a little bit on this show, but we talked about that. Observability is here for your mobile apps now and your front end apps and that those people, those developers can now join the rest of us in this, world of modern metrics, collection and consolidation of logging tools and bringing it all together into ideally one, one single pane of glass.

if you're advanced enough to figure all that out, the tools are making it a little bit easier nowadays, but I still think there's a lot of. A lot of effort in terms of implementation engineering to get all this stuff to work the way we hope. But, it sounds like that you all are making it easier for those people, with your platform. We're definitely trying.

Yeah. I think it'd be a future state that would be pretty cool would be the ability for operators to look at a Grafana dashboard or a, you know, dashboard or Chronosphere or New Relic or whatever it is. see that, you know, they see a disen a, an engagement decrease on login on a web property, and immediately open an incident where they page in the front end team and the core team servicing the off APIs in a company and have them operating on data where the front end team can be like.

We're seeing a number of retries happening after we updated to the new version of the API that you serve for logging credentials. even though they're all 200s, they're all successful requests, what's going on? And that backend team says, well, it looks like our, you know, we were slow rolling it out and the P90 latency is actually 250 milliseconds longer. Why would that impact you?

And they say, well, the SDK retries after 500 milliseconds and our P50 latency before this was 300 milliseconds. So, 10 percent of our users or something are starting to retry and that's why we're seeing this. You know, the answer here is to increase. The resource provisioning for the auth service to get latency back down and or change our SDK to have a more permissive retry policy.

and, you know, have teams be able to collaborate around the right design of software for their end users and understand the problem from both perspectives, but be able to kick off that incident because they saw real people failing to disengage, not just some server side metrics, which I think would be pretty neat. Yeah. And I should mention you all, if I remember correctly in cloud native, you're a lead maintainers on the mobile observability SDK. Is that, am I getting that right?

I'm trying we have engineers who are provers, on Android and iOS. we have the, from what I'm aware of, the only production React native, OpenTelemetry SDK. we are also participants in a new browser SIG, which is a, a subset of the former JavaScript SDK. So our OpenTelemetry SDK for web properties is.

Basically a very slimmed down chunk of instrumentation that's only relevant for browser React implementations. so yeah, working to advance the kind of standards community in the cloud native environment for what, Instrumenting in real world runtimes where Swift, Kotlin, JavaScript are executed. Nice. Andrew, thanks so much for being here.

So we'll see you soon. Bye everybody.