Grafana’s Approach to AI-Native Observability

Speaker A: Advanced software systems have long been more complex than any single engineer can fully understand. Observability is the established solution to this problem. But with AI agents now generating code, deploying changes and operating autonomously, the challenge of understanding large software systems is entering a new dimension. Grafana is an open source observability platform and one of the most widely used in the world. The company builds tools that help teams collect, visualize and act on telemetry data across logs, metrics and traces. They are now extending that capability into the agentic era with AI powered investigation and monitoring tools. Anthony woods is a co founder of Grafana Labs. In this episode he joins Matt Merrill to discuss how AI generated code is straining software operations, why telemetry data volume has become as much a problem as a solution, how Grafana is adapting to a world where agents are the primary consumers of observability data and what keeps him up at night about where the industry is headed. Matt Merrill is a software engineering leader with over 20 years of experience building and scaling software teams across enterprise and product focused organizations. His background is in um, backend development, cloud architecture and distributed systems design. He he currently architects and delivers software products and leads a team of engineers at Dept Agency. You can learn more about his work@code.theothermatmm.com.

Speaker B: Hello, welcome to Software Engineering Daily. I am Matt, uh Merrill and I am here today with a Anthony woods, the founder of Grafana. Before we start I am going to let him introduce himself.

Speaker C: Thanks Matt. Thank you for having me on the show. Yeah, so I'm Anthony, I'm one of the co founders. There were three of us at Grafana Labs. I'm based in sunny Perth in Western Australia. Uh, it's a beautiful city, it's just far away from everything which is okay. I uh, spent a lot of time on a plane traveling around. Yeah, I have a background in tech. I started off as a uh, systems and network engineer, then kept building automation tooling until one day I found I was just writing code full time and becoming a software developer. And so that was a lot of fun being able to bring together all different parts of a system with tools we can build. And about 12 years ago Raj Torkel and I, we started a little company called Grafana Labs. Right. Based on uh, Grafana, the open source project that Tocal had created. And we took that love of open source and of helping people understand the data and telemetry that are getting out of systems and turned it into a

Speaker B: business that uh, is awesome. I am super excited to talk to you, because I am a backend nerd. I've done pretty much backend stuff my whole life. I've always been at the intersection of backend and DevOps and so super excited. So today I want to talk about software engineering operations in the age of AI. And I kind of want to go into how you're doing this at Grafana itself, because that's a company at scale and in this age. And also what you're seeing happening with customers, because I'm sure you're seeing a lot of it. And so I think where I'll start is when you look at how software teams operated five years ago versus today, what do you think are the most significant shifts you're seeing in software operations?

Speaker C: Yeah, I mean, obviously things are, uh, shifting and changing very quickly. That's one of the big changes is just the speed of change. Right. Both from a capability and tooling of what's available, but also just on the demands on engineering teams. Right. Like the pressure for them to continually deliver, uh, more and faster. You know, we've seen the big shift from building monoliths into, you know, microservices and the benefits that had for teams to have, you know, smaller teams with smaller scope so that they could move faster. Right. And that, that has really worked. But the consequence of that is now we've got these very complex distributed microservice architectures where no one actually knows how the whole thing works together. They just might know their own little piece. And we've certainly seen that both internally for how we build our software. You know, we find that that model works really well for velocity of development, but also adds that complexity. And so, you know, we combat the complexity internally. And the way we see our customers doing it is with better observability tools. It just means collecting more telemetry data to understand what's happening. We look at all of these small little microservices and they're really just little black boxes that are going pushed into your production. And observability is that tool that turns that black box into, uh, a glass box. So when it inevitably breaks, and they always do, you've got visibility to kind of see inside, see what happened, what went wrong. And that's becoming increasingly important now with the shift to AI, where a lot of the code is not being written by a human, but it's still going to break. And so you need to have that visibility to know what's going on inside. And this is another thing I think that we've seen 10 years ago. I remember maybe the conferences, maybe monitorama or something like that we would go to. There was this big push around, like measure all the things just because people wanted to capture more telemetry. And that really has come back to bite people with just like the costs. Right? Like we've got so much data now we're ingesting. It's expensive to ingest all that data and process it. But also we see so many of our customers who run into problems where now they just have too much data and when things go wrong, they don't even know where to start looking. They're just overwhelmed by this ocean of data that they've got. So like trying to help people trim that down and focus on um, what's the important data that you need to be able to solve the problem. Not being overwhelmed I think is really important. So that's a good problem to have. And then I think the other big thing we're really seeing across the world is open source. Right. And the advantages, especially on the operational side, we're seeing technology changes fast. And historically it was open source is a risk for a business because can you trust it? Do you know what's going to happen? Who's going to look after it? Whereas now businesses are looking at open sources as uh, a must have just because that's where innovation is happening. You look at things like Kubernetes, you look at what we're doing with Prometheus ecosystem look at opentelemetry is a great example. Organizations are realizing that just to future proof what they're doing from the operational side, they need to adopt open source technology, open source standards and these open ecosystems just to be able to get access to the innovation that's happening, but also move away from those kind of vendor lock ins that people desperately want to avoid today.

Speaker B: So I am admittedly not familiar with open telemetry. So could you say a little bit about what that is?

Speaker C: Yeah, definitely. So anytime you're building an application and you want to kind of instrument it, right, There's a lot of choices for how you go and do that. How do I, you know, when I'm writing my code, I want to emit logs, I want to collect traces or emit metrics. And so there's been a lot of different ways. A lot of vendors have provided proprietary kind of SDKs and things that you can use. And so open telemetry is the open standard, right. It's been around for a while, but now we're seeing where it's got the velocity, it's got the Input and people are actually using it and getting a lot of value out of it. And so it's a growing ecosystem. Um, and so it's really just a standard practice for how you instrument your applications and you collect that telemetry. So it has very good kind of semantic conventions. Right. So things look the same. So that way when you've got different teams working on different projects, building things slightly differently, the telemetry that's coming out of it should all look familiar. Right? So if I'm not, if I haven't worked in that team before, but I can go and have a look at their logs or look at their metrics or the traces, it should look familiar, like things should be named consistently where things are understandable. And just having that kind of consistent approach really helps. And obviously the huge advantage here is you don't want to have to be re instrumenting your applications every time you change your observability vendor. Right. It's a thing that you just want to do once and never have to think about again, but still have that ability to go and change your observability vendor over time or add additional tools or different capabilities. So having that open ecosystem, open standards and vendor neutral approach is very attractive for a lot of organizations.

Speaker B: The HTTP of logging, that's the way like hearing it. Yeah. That's awesome. I can't believe I'm not familiar with that, but that is very interesting. And I'm assuming Grafana supports this. Are you supporting the project itself too or.

Speaker C: Yeah, definitely, yeah. We're, I think the third contributor by number of commits that we make to the project. We're obviously very heavily involved in it, a lot of the products that we build. So within our Grafana cloud platform we have more opinionated solutions of how to do observability, how to do application observability or infrastructure observability. And a lot of that is built on the open telemetry ecosystem.

Speaker B: Gotcha. Cool. So what you said about open source and open standards to me also intersects with AI quite a bit, which is where I want to kind of spend a lot of our time. Right. And so one of the benefits, the many benefits that I see of open source is that it's well documented because it's out on the Internet. These models can, for better or worse, scrape all this information and support it very easily without some sort of vendor lock in or paywall or something like that. Are you seeing that happening already in this space or perhaps not? Is it not caught up yet?

Speaker C: No, definitely. I mean we're really excited. Grafana Labs with some of the AI capabilities we've been able to build into the product. So things like our, uh, Grafana Assistant, where we had a team, we were worried for a while. We saw AI coming. If you'd asked me 18 months ago, how's AI going to impact operation, I was like, I'm not seeing anything useful. But then we did see that big kind of shift with some of the new models that came out beginning of last year, just like, uh, the end of 2024. And so, you know, we thought we were behind and we were like, how would it catch up? And then we had a team of just two people during one of our quarterly hackathons who put together the MVP of what is Grafana Assistant. And it was just amazing at how well it worked. It was able to understand, you know, what people wanted to do within Grafana and knew how to use Grafana. And so we were like, how did you guys do this? Why does it work? And they kind of shrugged and they were like, I don't know, it just does. But as we thought more about it, the thing that we realized was we've spent more than a decade building this great open source ecosystem of, uh, we've got 25 million plus users around the world who love and use Grafana. But also they blog about it, right? They write tutorials, they write guides on how to use Grafana to solve certain things. Everyone's got public GitHub repos which has got their Grafana dashboards in it and they've got configuration for how they've employed it. They've got a whole bunch of public information and it's this information that the foundation models from Anthropic or Google or uh, OpenAI are trained on M. So out of the box, the models know our technology, they know what our users are trying to do and they know how to solve those problems for it, right? So we've been able to kind of leap ahead of our competition by not having to go and train expensive models, right? We can just use the foundation models and then just build a tighter integration and build more tools and uh, a more kind of integrated solution into our product rather than having to do all the expensive part of it. So that's a huge advantage that we've found. And great minds we had 12 years ago when we decided to focus on open source. It was a great plan that we'd put in place, but it really has helped. And we see this a lot where even simple things like asking Claude Code to instrument with OpenTelemetry, it knows how to do that. It can go and do that for you and add that to your code and make sure that it's collecting the telemetry that you need.

Speaker B: That's amazing. Uh, it almost seems like good karma. Right? Like, you've put out this good thing into the universe and now you're getting something back.

Speaker C: Yeah. I mean, that's it. And it's a big kind of value for our organization. Right. Where we just have this mandate that comes from Raj itself is like, we just do the right thing. Right. We do the right thing for our employees, we do the right thing for our customers, we do the right thing for our vendors. We just think that when you do that, it comes around. Right. And people will do the right thing for you. So, yeah, I'm m a big believer in doing the right thing, and it's worked out really well for us.

Speaker B: Used to have a saying at a company I work for is like, just do good work and the rest will follow. And we knew more of that. You don't have to answer this if you don't want, but are you afraid that somehow this might turn around and bite you in the butt and somehow replace the tools? I don't personally see how that might happen, but I'm curious if that's on your mind.

Speaker C: I mean, I'm generally, uh, a very optimistic person, so I think there's certainly risk there. Right. We can't just sit around and hope for the best. Right. We are definitely seeing a change with observability. And I think the main thing, you know, especially for us, which is scary for Grafana Labs, is, you know, we've built our reputation on being, you know, the company behind Grafana, the dashboarding and the visualization tool. Right. And it's something that users interact with. Right. And what we're finding more, both our internal use cases as well as our customers, is that more and more, it's not humans that are interacting with the data anymore. Right. It's agents. Right. That are going in, doing queries, looking at what's happening in your environment, trying to understand and kind of derive some kind of insights from it. And so we are seeing that shift, right, where there's less about the dashboard itself. And so we're definitely making changes within our product to support that. We announced last week at, uh, grafanacon, we've got our new GCX project, which is our, uh, kind of unified command line tool. Right. And it's designed for agents to be able to interact with our Grafana cloud service and all of your data and access it that way. But that said, I still believe that one of the things that's really important when it comes to AI giving you information is the trust but verify kind of piece of it, right? Where it's pretty smart, it's pretty clever at what it can do, but sometimes it gets it wrong. And so you need to be able to have that information that you can go and look at to say, hey, how did you come to this conclusion? How did you make this decision? Et cetera. And often the best way to remember that is still a graph, right? It's like, hey, and so we have that in Grafana, Insistent. When it'll do an investigation for you, it'll come back and explain to you, hey, uh, I went and looked at this data, I saw this trend or I saw this spike. And so that's why I've then gone down this rabbit hole to go and look for this problem. And so being able to just explain that with a graph is really easy, right? And that's that that the user is still going to want to be able to consume.

Speaker B: That makes sense. It makes a lot of sense that, you know, at the end of the day, mostly what you're looking for is the answer to a problem or a question that's causing a problem or a pattern or something like that. Are you doing things proactively? Is the tool looking at these patterns proactively at this point or.

Speaker C: Yeah, I mean, so we have, uh, a few different things that we do. So like the age of AI, I think we're in it, but there's a lot to it. I mean, everyone kind of now just associates it with LLMs and kind of generative AI. But there's still a lot of other AI practices that we use internally to be able to get, uh, better quality data, right? Because the LLMs are great when you can give them a lot of context and kind of quality data. But it's the classic, you know, junk in, junk out. So you want to make sure that they are getting kind of clean data coming in so that they can make the right decisions. And so we use a lot of kind of AI tools internally to understand your telemetry data. So one couple of things we do that for is we have something called, uh, a knowledge graph, right? Where we kind of look at all the telemetry and build a, uh, graph of all the different entities and how they relate to each other so that you can say, hey, I've got this application that's great. But I know that it's running on a pod, right? And that pod's running on a node, and that node is in a certain cluster in a certain region. And so you can kind of know the relationships. And that's great when we hand that off to the LLMs, because then suddenly when they do see a problem, hey, I'm seeing a latency spike for this application. They then can traverse the telemetry and understand, okay, well I can find what node this is running on. And it's like, oh, look, I can see that that got a resource contention problem. Oh, look, your latency spike is because you've got a noisy neighbor, uh, who's consuming all the resources. Right. And simple things like that. Sounds simple to us. But being able to understand all of that telemetry becomes really important and the speed at which things are changing. Right. Like in the old days you would just have like a CMD database and be like, oh, I can describe all my assets here. And it's a static thing. Right. And the reality is every time you have one of those, they're out of date by the time they get updated. And so we wanted something that was dynamic and just generated by the data that's actually being ingested in real time. So that way it's all always up to date and it's always a, uh, representation of what your environment actually looks like.

Speaker B: So if I'm hearing you right, like this is different layers and different applications of different machine learning, like AI models that feed into reasoning models that can help you make, identify those patterns and things like that. That's awesome.

Speaker C: Yeah, and I think that's really important. I think that's what we're seeing is just that the LLMs are very smart, but you have to give them the right data for them to be able to come to the correct conclusions. And so that's really what we want. We put a lot of effort and a lot of focus on is how do we make sure that we're curating the data set, uh, properly. And we can use a lot of machine learning kind of tools to do that as well as leveraging things like the open standards. Right. And open ecosystems. Right. Leveraging things like open telemetry means that we know the naming conventions and the semantic conventions so we know what the data is representing. Right. And that's kind of public information, so the LLMs can understand it as well.

Speaker B: That's awesome. So just to pivot slightly, I think one thing on my mind is observability has been about humans understanding what's happening in a system. But as we use LLMs more in the applications themselves, we start to lose a lot of control and understanding of what's happening there. So how is operations changing with that paradigm shift right now that you're seeing with your customers?

Speaker C: Yeah, we're definitely, you know, from our perspective, we see the shift to this kind of like agentic model of software design. Right? It's just a new design framework. We had mainframes, right. And then, you know, we had monoliths, and then we've moved to microservices and now we just see this agentic is just a new way of building software, right. It's got some different problems that are introduces of how we understand what's happening. But, uh, the tools we have for the most part get you a long way there, right? Like, as we think about how do we understand what these agents are doing, it's typical kind of application observability or apm, right? Like you want to be able to collect that telemetry to know what's happening. But then now we've got some new types of data and things that are coming out that we want to collect as well. Right. So a lot of things like the conversations to understand so we can go and write our evals and do interesting things. So, hey, uh, are you actually giving correct responses? And then also when things go wrong, trying to understand, well, how did it get there, right? What was the input that kind of caused it to kind of go off the rails? And so we're seeing definitely demand right across the market where people want tools that can just do this, right. That can just build into their agent framework. And so that's something we've had to do internally as well, right. As we were building out, uh, our agents and capabilities, right. We needed to kind of understand this. And as we like to do a Grafinal Labs, we just, you know, we're a company who just keeps scratching our own niche and solves our own problems. And then, you know, we share that with the world as products. And so we did the same thing, right? So, you know, we announced last week that we now have in Grafana Cloud our AI observability capability, which just does this, right? So it builds on top of OpenTelemetry to collect, you know, reward telemetry and then collects additional things, right? So the conversations and other data, that's really important and then builds, you know, an experience where you can go and have a look at the things that are important to you when you're building agents, which Right now is like reliability and quality as well as cost. Right. How many tokens am I burning for all these conversations? And which certain kind of workflows within my stack are the ones that are using the most tokens? Right. And that could be a good thing. Right. Often we care about token usage because we want to see adoption, we want to see people using the tool. We're in the kind of phase now, the growth phase certainly for AI, but there's going to come a time in the near future where suddenly people are going to be like, wait, let's think about how we can spend less on our token. So we want to kind of give that visibility. Uh, yeah, we want to give visibility to our users so they can understand what these systems are doing. Right. And be able to kind of dive in and when things do not go as planned, be able to kind of have the data they need to understand what went wrong so that we can make changes. Right. So that we can improve things over time.

Speaker A: Yeah, Most AI frameworks started with voice and bolted on video as an afterthought. Vision Agents by Stream was built Video first from day one. It's an open source Python framework that lets you build real time voice and video AI agents in minutes, not months. With 25 integrations for models like OpenAI, Gemini and Claude Sub, 500 millisecond latency on Stream's global edge network and support for YOLO, roboflow and custom CV models, you get a production ready stack without the infrastructure headache. Whether you're building coaching tools, multimodal assistants or real time security pipelines, Vision Agents handles the hard parts. Get started free at uh, VisionAgents AI here's something nobody tells you when you're becoming a senior developer. Getting better at writing code won't get you to the next level. The skills that make a great developer and the skills that make a great architect are fundamentally different. The Software Conductor is a new book written by SED host Lee Acheson that explores this gap. It explores the gap through the story of Aaron Blake, a senior developer who meets a symphony conductor and realizes the parallels are almost exact. A violinist makes sound, but a conductor makes music. A uh, developer writes code, but an architect creates the conditions for great software to exist. If you're ready to make that shift, this book is for you. The softwareconductor.com available@Amazon.com in mobile application security Good Enough is a risk Guard Square uses advanced multi layered code hardening techniques and automated runtime application self protection and mobile application security testing combined with real time Threat monitoring to deliver the highest level of mobile app security. Discover how Guard Square brings all these together to provide mobile app security for your Android and iOS apps without compromise at www.guardsquare.com One of the questions I

Speaker B: had was like, what you're seeing your customers, like what does it mean to monitor an AI feature in production? And I think you explained a bunch of that. But what I'm wondering too is what are those hooks? What are those hooks into? Something like OpenTelemetry? If I'm sitting there thinking about how I might prompt a mod to do something, to me, are people saying like, as part of what you're doing, log out your reasoning steps to open telemetry or something like that, where you have another agent that does that, or is it more deterministic coding before and after calling? Like, what are you seeing in terms of those? Are they integrated with the APIs themselves in some way too?

Speaker C: Yeah, so for a lot of the, there is like some good integration. You think about a lot of the tool calls, right? They're not really any different to any other kind of RPC call that you're making. Where you're going to want to measure is it actually getting a response? Right? Am I getting valid responses coming back? You're going to want to look at latency. How long are these things taking to respond? Because that's going to impact the user experience. And you talk about what does monitoring mean or what does observability mean? Why do we do it? The main reason for it is to make sure that we're meeting our customers needs or our users needs, uh, meeting their expectations. One of the ways we do that with observability and the simplest approach we have is around the concept of SLOs, so service level objectives. Rather than saying, hey, I'm going to go and monitor my CPU usage because do I care if I'm using 100% CPU? Not really. Because I'm paying for a CPU, shouldn't I be able to use 100% of it? What I actually care about is what is the quality of the service I'm giving my user. Am m I seeing a latency spike? That's something that I care about. And so being able to pull what are those core user experience things that I want to make sure my application is delivering on. So reliability is a simple one. But even latency, depending on what your business is, you're going to have your own objectives. And so those are the things that you're going to want to measure over time. And then as they change. That's when you know something's wrong. If your users are constantly, uh, getting 500 errors where you know something's gone wrong, right. Or if you're maybe something simple where you know, if you see traffic drop off right on your site, maybe that's uh, an indication that something's wrong and you want to have someone go and look at it, or you want to have an agent go and look at it and give you an idea of what it might be. But there's ways to kind of collect all this data today where you can a lot of logs, a lot of traces, et cetera, where you can bundle this data. One of the challenges with traces, for example, one of the challenges we have with traces with this agentic model is just the volume of data. So those conversations that you want to have usually exceed what a trace can recently contain within a single span that it can send. So we're having to use supplemental data stores to store that kind of information. And this is where we see then one of the nice things we like to be able to do in Grafana is we've built this ecosystem around integration and interoperability. We call our big tent philosophy, being able to bring data together from lots of different types of data sources and be able to kind of draw those correlations. And so we're able to do the same thing. We can go and build a new database. It could be a SQL style database that's storing all this information. But within Grafana we can just stitch that together and be able to have you kind of correlate between my traces, right. Where I've got all of my traditional kind of APM data, but then be able to also tie that to all of the chat history or all of the other kind of context information that you want to go and store and be able to then visualize that in one place.

Speaker B: Nice.

Speaker C: Mhm.

Speaker A: Cool.

Speaker B: So back to what you're doing inside of Grafana itself. So you mentioned that awesome hackathon assistant thing in terms of just day to day operations. Where are you seeing AI have the most operational wins for you? Maybe it's even outside of observability itself.

Speaker C: Yeah, I mean we're definitely very bullish on AI capabilities across the organization. Right. Like we have a mandate of like everyone, you know, try and use AI wherever you can. Right. So we can take advantage of it where possible. Obviously we're seeing huge adoption and use within software development. Right. Like it is great for building software, for getting code out there and just being able to innovate quickly. We're actually quite fortunate because we see kind of like the role of a software engineer is changing, right? It's less about the code and it's more about engineers becoming more product managers, right? Where it's really about kind of understanding the user problem and then being able to explain that to your agent, uh, and we'll go and build the code for you. And that's something that we're again, fortunate with at Grafana Labs, where we've always had this kind of bottoms up kind of culture where our engineering teams have been responsible for the roadmap, right? Like, we have a great product team and great product managers, but they're there to facilitate. So they bring the information in from our customers, they bring information for what's happening in the industry and we can see and then come and facilitate the conversations. But at the end of the day, we like our engineers to be responsible for the product roadmap, right? Because we want them to own it. Part of that is because we are building products for engineers, right? So they do have good kind of intuition and insights. That's the dream needed. Yeah. And so. And so we've always, always had engineering teams and had a big focus on giving them autonomy to go and make decisions about product and to think about what are the needs of our customers and how can we go and solve that. And so that's now helping a lot with the tools that we've got, where they are already experienced at being kind of mini product managers themselves. And so they're able to kind of leverage the tools quite effectively to go and build things and get new products shipped and delivered. The other area we see it is where it's interesting. It's obviously more on the operational side is investigations, right? Like understanding what has gone wrong. So again, I talk about the Grafana assistant started as mvp. One of the things that we like to do with our products, we're scratching our own itch. And before we obviously release products to our customers, you want to make sure there's good product market fit. And the way we do that is we just make it available to our internal teams. And if they start using it without anyone telling them to, then we know it's a good product. And we saw this with the assistant, right? Soon it was in there, people started using it. They get paid for something, they just send the assistant, hey, can you go and tell me what you think it is? Right? And it's very like, it's remarkably good at doing those investigations and going through and finding the problems. So that's been really powerful. So we leverage that a lot to go and understand it's not always right, but often even if it's not right, it's going to kind of reduce the surface area that you need to go and look at. Right. Because it will come with some suggestions or it might exclude some certain problems. Right. So it really does accelerate our team's ability to go and find that root cause and solve those problems. The other area that we're seeing more and more is, which is a little bit scarier, right. Is, you know, as we see AI shift from just the development side into more of the deployment side. Right. Like, you know, when do we start giving our agents keys to, uh, go and just, you know, you've written the code, why don't you just go and deploy it?

Speaker B: This is where I get uncomfortable. This is where I get uncomfortable. Yeah.

Speaker C: I mean, it's a scary prospect, but I don't think we're far away from it. I mean, obviously we see a lot of things in the news cycles, right, with, you know, agents going and deleting people's production databases. Right. And so people obviously a little bit scared about these kind of things, but me personally, I don't think we're that far away from it. But I'll caveat that a little bit. And I think one of the things that we see is that, that we don't inherently trust the AI to do the right thing. That's okay. We shouldn't, right. We should be kind of cautious because if there is an opportunity for it to do something wrong, it will find that opportunity eventually. But, uh, for me, I don't think that's any different to how we treat people. Right. The number of times I've said to someone, oh, we don't need to worry about the normal kind of process, it's just a small change. Uh, uh, it's not going to affect anything. I'll just deploy it in production. And then it's like, oh, oops, I didn't expect that was going to happen. And so we need to protect ourselves from ourselves already today. And so like, certainly at Grafana Labs, we have a whole bunch of tooling in place in our, uh, CI CD pipelines that put physical gates in place where it just prevents you from being able to do the wrong thing. Right. You can't go and do a change where the blast radius is global, Right. Like it has to be small blast radius, limited impact. So if something does go wrong, it's not going to impact all of our customers, all at once. We have things around cost management. Right. When we do a change, we have a CI check that runs and says this is going to increase spend by 10x. Right. And you can't do that. Right. And so we have got all these physical gates in place that prevent people from making mistakes. Right. And that also works then for agents. Right. Like once you build these gates in place where they're hard blocks that stop the agent from accidentally shooting itself in the foot, you get closer to this reality where you can start to let it do some things in your environment. And I don't think we're far away from that. But I think that having those controls in place is an organization maturity thing. It's taken us a long time to build that because we add new checks in place every time something goes wrong that we didn't expect were going to happen. And that's easy enough to do with people because we move much slower. The scary thing for an agent is that they can move very fast so they can make a lot of mistakes very quickly. And so we're going to have to make sure we've got the right kind of controls in place. And so that's going to be the, I think the thing that, that will allow us to give the agents a little bit more control to go and deploy things. They can write code, they can go and deploy it, they can observe it and see is it having desired effect, is it working correctly. If it's not, make some more changes, go and deploy it and just shorten that feedback loop so that they can base their changes based on direct feedback from users and what's happening inside the production environment. But it's a scary thing. But I also think it's exciting. I think it's an exciting challenge because we want to get to that place where we can free it up. Because right now as software engineers often, uh, our job is to balance the how do I deliver faster and balance that against how do I keep the system reliable and meeting my customers expectations. Right. And so we can write a lot of code right now, we can develop a lot of new features with the uh, AI, but we're still kind of bottlenecked on how quickly can we get that in production and validate that it's actually working. And so I think once we can get past the, where we can give agents a little bit more control then that whole cycle then accelerates a little bit faster and so we can then ship faster and we can get better insights and kind of keep iterating and going and it's a scary prospect and I think there's going to be a lot of pain along the way. But I do think it is coming. We just have to find uh, ways to do it in a controlled, safe as possible approach.

Speaker B: Yeah, there's definitely um, a risk analysis that needs to happen there and how you step slowly into that and if you have the right checks and balances. I can't wait for some of the stories that'll come out from this. It's going to be unbelievable.

Speaker C: I'll give you an example of what we're doing in Grafina lab. So one of the things we're automating is when we see we do a deployment and be small scoped into a specific subset of users and then we can go and look at that and make sure it works. And so we have some automated checks that we can leverage Grafana Assistant to go and say, hey, are there any problems? Are there new errors showing up, et cetera. And if it does find problems, we just have it automatically open a PR to rollback. Right. And so those kind of actions we're seeing kind of creep in. We still have a user look at that and approve that PR before it happens. But it' that's a simple things like, oh yeah, this is just a simple rollback. Can approve that and it will just roll back to the previous release and we can then go and fix it. And so we are seeing these things where we're slowly giving the agents a little bit of operational control because they can do things much faster than we can. But we still need people in the loop right now.

Speaker B: Yeah, self healing system. It also depends on where that change or what change it's making, what it's affecting. Like I don't know how I'd feel about a uh, agent deploying like a data migration on millions and millions of rows of data. But thinking of front end fix, something like that, something that could be easily rolled back. Hey, why not?

Speaker C: Exactly. Yeah. And I think that's where we're going to start. It's those safer operations. We're going to be more comfortable right. Keeping the AI tools. But uh, we'll definitely see there'll be a bunch of young startups where they're very risk tolerant. They're like, I just care about moving fast. Right. And they'll go nuts. Right. They'll just let it do whatever it wants. And some databases will get deleted. But that's okay.

Speaker B: Hopefully not important ones. So let's talk about the people using these tools. Right. So as AI handles more of this operational work, how do you see the role of SRE DevOps engineer, platforms, engineer changing? Do you think they're getting bigger, smaller?

Speaker C: I mean I think right now they're bigger, right. Just simply because the volume of new software getting deployed is just growing so quickly. Right. That said, I think they're getting left with, which is actually kind of for me personally. Right. Like they're getting left with the harder problems. Right. So a lot of the toil, the simpler work that the SREs used to do can be done by the AI now and be done by the agents now, which then leaves the SREs to focus on more high value kind of piece of information. So the hard problems, right? How do I go and diagnose this or you know, thinking more strategically around like how do I build a more reliable way of doing things, right. More kind of process orientated or you know, go and find, you know, how that AI is not supporting the needs of the business or you know, helping you kind of drive things forward. So right now I think there's just growing need for it. Right. We're shipping a lot of code, it's of questionable quality sometimes. So we do need more of that kind of SRE kind of practices to make sure we are thinking about reliability, make sure we understand, can we measure it? Are we making sure that as we're building cool features and new tech, is it delivering the value that we want it to have and meeting our customers expectations? And most importantly, when things go wrong, do we have the data to be able to understand what happened, what went wrong so we can go and fix it? So I think that's definitely the case as that changes over time. I think yeah, we're still going to need those practices. The thing that scares me the most, both from an SRA perspective as also uh, software development, is just that because all of the easy things are being kind of taken care of. That's kind of the work that you give your junior people or new grads where you just have to go and do it and you kind of, as you're doing it, you learn, right? You learn how these systems work, you learn what to do. But as we take that opportunity away, it's like how are we going to give people the opportunity to develop the skills that they need to become these experienced kind of senior SREs that we're still going to need, but they're not going to have that path. Right. And so that's the thing that scares me a little bit is what's going to happen in 10 years time when we realize, oh my God, we just don't have anyone who has the skills that we need. And so I don't know how we fix that. I think it's the same with software development, right? Like we still need really experienced software engineers, but the new grads and kind of juniors aren't getting the opportunity to develop their skills where they can have that experience, right? Where they've learned things the hard way and have gained that experience. Right. They've earned it by making a lot of mistakes along the way. But we're taking that opportunity away from people. And I think that's the scariest thing thing for me. When we think about the future, the

Speaker B: high level thinking that's needed just keeps getting more and more and more and it goes from reactive and detail oriented things to proactive high level thinking. And yeah, I'm in the software engineering game. We're seeing the same thing. I saw an interesting, I wish I knew who the author was, but I saw an interesting thing of one potential way out of this is to go back to like a apprentice, uh, journeyman master model, which I am a huge fan of software as craft. And I thought that that was really interesting where it's like the only way that you are going to get this experience is by sitting shotgun to an experienced person and feeling the pain with them and seeing their line of thinking. I think that's especially interesting with SREs because you're reacting to these potentially nasty problems and trying to unravel that and, and so thought that was a really interesting one.

Speaker C: I think it's a, it is an approach. I think the scary challenge is like, it's an expensive approach where you're not getting immediate value from. Right? And I think a lot of businesses and organizations are going to be like, I mean, sure, why don't we just let some other company be the ones that train all the people and we'll just keep using the agents, right? So I think it's going to be very difficult to find that. I think one area where we do see this opportunities is still open source, right? Like we, you know, we'd love to kind of engage with our community. We do get a lot of young people who come and contribute and that's our opportunity to kind of coach them. Right. And being open source, right. We're happy to invest that time to nurture those people so that we can help them develop their skills over time. So I will say there will be more of those like community driven, right. Kind of ecosystem. Because I do think this idea of training People up to be productive so that they can leave and go get a job at another company. Right. It's not going to be an attractive option for a lot of organizations.

Speaker B: It's a really good point. Yeah.

Speaker C: So I do think more of that, like as software engineers, as a community, I think it's our opportunity to find ways to kind of, of nurture those people coming through and give them opportunities. I think open source is a great way to do that. Right. There's big code bases, there's opportunity to write code, there's ways we can communicate and share ideas and help kind of train and coach people a little bit to give them the experience that they're going to need.

Speaker B: Well, also not all apprentices in the trades are paid either. Right. And so we've been spoiled in this tech industry, but high paid internships that lead you to a high paying job, the playing field may be leveled. But yeah, good points, good points.

Speaker C: But I think even like a lot of the people I talk to, colleagues I've got, even still people we hire today, I love to go to, we do our uh, onboarding in person and get people to go and talk to the new starters of what's happening. And a lot of them have similar stories, backgrounds to me.

Speaker A: Right.

Speaker C: Where they're very self taught. Right. And I think that's true within this industry where a lot of people love the technology, they're excited about it, they love the fact that things are changing all the time and there's always new things to do. And so, so you have to have this mindset of wanting to go and teach yourself and learn new skills. And so I think that's still going to happen. Right. But I think open source is a great avenue for that where it's an opportunity for you to go and do something off your own bat to develop those skills that you're going to need in the future. Because the speed at which things are changing in this industry. Right. You have to be committed to a life of learning because you blink. And there's a new JavaScript framework.

Speaker B: I hadn't planned on talking about this, but I feel like it could be fertile ground. So are you seeing open source projects struggle with the amount of code that agents are producing or what are you seeing in your area?

Speaker C: A little bit. Both from the volume as well as the quality. Yeah. So it's very easy to have an agent just go and build a feature. Right. But just because it works doesn't mean it's code that you want to maintain over time. And so that's I think where we're seeing the challenge is, I think this is still where there is that challenge with the AI assisted kind of code tools, where we put a lot of thought into kind of like the architectural design and guiding principles for how we build code. Right. And every organization is different, but you end up building your view of the world and how you want code to look. Right. So it's got consistency and so it's much easier for your teams to kind of like people to move around between your teams. Right. Because the code bases are all similar kind of layout. You're using similar kind of practices for how you go and build code and so things look a little bit more consistent. Whereas when you start getting the AI in, it doesn't care about those kind of things. Right. It's just going to find a solution that works and go and throw that code in. And so there's a lot more work that has to go on to, to think about and care about maintainability. Right. So that certainly affects open source, right? Where you don't just want to accept any pull request that adds any new feature. You want to think about is that feature, right. For this project, is that something that we can maintain over time? How does that impact kind of the rest of the ecosystem? And so that's where we're seeing the challenge, is that you'll get people that will just submit a pull request that'll be 20,000 lines of code that the AI has generated for them. And you're like, well, no, right. We're not going to review this. It's too big, it's too much. This is not how we build software. You need to kind of break this into smaller chunks and so that it can be reviewed and so it can be maintained over time and understood. And so I think that's the challenge that we're seeing is code is a collaborative business, um, especially for open source projects. And so the agents have to fit into that collaboration mindset, right. Of not trying to overwhelm people with too much change all at once because it's just not going to get accepted. Right. It's too hard to review, it's too hard to maintain over time.

Speaker B: And to your earlier point, these things might be learning moments for people that are less senior too.

Speaker C: Yeah, yeah, yeah, definitely. Yep.

Speaker B: So let's move towards the future a little bit. So what's the operational problem that you think AI and these tools are going to crack in the next two or three years that people are still doing manually right now, even right now with the existing Tools that we have.

Speaker C: Yeah, I mean I think right now we're at the stage where AI can be that first responder. Right. When something goes wrong in your environment. Right. So an alert gets fired because you know, you've got an SLO burn and uh, it's like, hey, like you know, error rates of spikes, something's wrong, you're going to page your on call person but at the same time you can go and spin off, certainly in Grafana Cloud you can go and spin off investigation, say hey, go and try and work out what's wrong. Right. By the time your on call engineer gets in front of their PC, right. There's probably already a response to say hey, this is maybe what the problem is. Right. It may have worked it out or it may have excluded a whole bunch of things. So having that I think is where we're going to see over the next few years and that's just going to get more reliable, more accurate over time. We're going to build more playbooks, we're going to build more skills for it. So that can do more interesting things. But I think that's where we're seeing it, is that first responder, as soon as you notice something wrong in the environment, it will go in and it will try and troubleshoot that and find that root cause for you, you. So that way instead of it taking an hour to find that root cause and kind of solve the problem, we're now talking in 10 minutes we can go and get things patched up and move on. We're still going to have the follow up process. Right. Which is so important for our uh, SRES is to go and actually don't just stem the bleeding, let's go and fix the problem. So that's still going to be important but again you can leverage some AI tools to kind of process more data and kind of do those kind of things. But I think the near term, so the next couple of years that will just be standard. Right. Like there'll just be an expectation that, that 60, 70% of the problems are kind of resolved before your on call person even gets in front of a computer.

Speaker B: Yeah. Because right now for resiliency in systems we rely on redundancy and things like that. But I think what you're saying is what we're starting to trend towards is actually in a limited sense but a self healing system, a dynamic self healing system, which is pretty amazing.

Speaker C: Yeah. And I think that's going to be really important just because of the speed at which software is being shipped. Right. If you want to move fast, you're going to inevitably break sooner and more frequently because you're trading speed for accuracy. And so the need for better observability just becomes more apparent. You're going to want to make sure you've got the data that you need so it can go and fix. But also we're just seeing that AIs, uh, are being able to use that observability data to understand what went wrong. And that fit feedback loop can be really fast. And that's just going to help us be able to ship things faster, but also maintain the level of reliability that our customers expect from us. And I think the other thing that we're seeing AI very good at is obviously processing huge volumes of data, right? Going and looking for those kind of trends, those hidden things that a user hasn't noticed. We do a lot of that internally where we can just say, hey, just go and query a huge volume of logs and find what's interesting. So we're seeing that use case change. We recently made changes to Loki, so a log aggregation system to just be back to do that smarter and better to fit with that use case of what AI, uh, agents want to go and do. So we've made a whole bunch of architectural changes and performance improvements so that it can query 10 times faster. More importantly, when you give it a query, it can reduce the volume of data that it needs to go and scan and look at to go and do that.

Speaker B: I'm curious at a high level what those architecture changes look like. That sounds like a very interesting problem.

Speaker C: Yeah, I mean, so one of the things I can. Yeah, yeah, we announced it last week, agrifinacon. So it's in open source. Right. So yeah, I can definitely talk about it. I mean, I'm allowed to talk about it whether I understand it as well as the people who have written it. That's a different question. So what we really think about is, you know, Loki was designed for cost efficiency, right? So how do we store huge volumes of logs as cheaply as possible? And the way we did that is like, just don't index them. Right. The use case we designed for was the developer use case where, where you just want to kind of grep, right? Basically through your logs. Grep AI. Look at the log message. I'm like, not that message. And you just keep excluding things or you might have a. Oh, um, I'm looking for a certain string process of elimination. Right. And so that's what we kind of optimize it for. But more and more we see people wanting to do more kind of analytics style queries or needle in a haystack queries more often where they're like, hey, I've got a transaction id And I knew it happened sometime over the last four days. Go and find it everywhere in the logs. And so. So Loki was designed to be cheap to ingest data, but we wear the cost on the query side, right, where we just brute force it. We'll just go and look at all the data and look what it's trying to do. And it can do that quite efficiently, but it still takes time, right? When you're talking about terabytes and terabytes of data that you want to go and process every second. So one of the big changes we've made is we kind of introduced kind of similar to light bloom filters, but it's a slightly different kind of approach where we can then know which portions of the data to go and look at, which ones we don't need to worry about. Right? So if you're doing that needle in the haystack query, we could very quickly say, hey, actually you don't need to go and look at the petabyte of data we've got. You actually only need to look at these 10 terabytes that we've got here. And it can do that very, very fast. So that's one of the ways. The other way is we've just put a lot of effort into like the query engine itself just to optimize it and just make it faster, being able to parallelize it as much as possible, because that's easy to do, right? With compute systems today, right, where we've got a huge volume of compute, where we can just go and spread out the workload and get things done as fast as possible. That works insanely well for us as a cloud provider, right, where we've got the benefits of kind of statistical multiplexing, right? Where we can, you know, we've got thousands of users all using it, right? We can provision a very large infrastructure estate and, you know, not every user's querying all at once, right? So we can say, hey, this one user's querying at this second they can go and use a thousand cores, and then the next second a different user uses 1000 cores. And so we can get really fast response times because we've got a bigger environment that we're able to kind of

Speaker B: build on top of efficiencies of scale.

Speaker C: Yeah, exactly, yeah.

Speaker B: Oh, man, I would love to just talk about that for an hour.

Speaker C: That Sounds interesting.

Speaker B: All right, so a couple of wrap up questions. This one I'm really curious about. So what's an operational scenario that keeps you up at night especially related to AI, something that you think the industry isn't taking seriously enough right now?

Speaker C: Well, that's a good one. I think one of the challenges for me is just a lot of the AI agents that are being built today are ah, just black boxes. Right. And we don't know what they're doing. Sometimes we don't want to know. So I think that's the scary thing is the speed at which these systems can operate and make changes and do things. Uh, because we've moved from the chat to this agentic model where it's like let the AI do things and not just respond to questions. So that scares me when you've got these black boxes that are just no one knows what they're actually doing. Right, Right. We have a mandate internally where our uh, engineering teams are leveraging AI wherever they can to build software. But at the end of the day, the person who merges that, they're responsible for that, they're accountable for that. And so I get scared about when that starts to erode and it's like, well, who's actually accountable for the things that are getting deployed into our production? I was having a conversation, someone actually just yesterday talking about kind of like this governance problem. Right. We're seeing more and more agent to agent communication. Right. What happens when you've got uh, an agent in your organization talking to an agent in another person's organization and they just doing things right and something goes wrong? Who's accountable for that? Right. Whose fault is that? Who's responsible for kind of remediating that and taking accountability? So that's the thing that scares me the most is around, I guess, two things. One is the black box aspect of it of like what are these things actually doing? And the other one is the accountability. Who's responsible when things go wrong? If we are giving all this autonomy

Speaker B: to our AI agents, some really interesting legal and contractual things that'll come out of that as well. Yeah, that's a really good point. Point. It always comes back to humans anyway. Right. You know?

Speaker C: Exactly.

Speaker B: Yeah. What's the throat to choke if something goes wrong?

Speaker C: Yeah. That'll be the most lucrative career in the future. Right. Where it's just AI, you know, scapegoat. Right.

Speaker B: And yeah, yeah, I'll take responsibility.

Speaker C: Blame me.

Speaker B: AI insurance.

Speaker C: O.

Speaker B: Okay. Hey, you want to, want to make a new startup? No, I'm Just kidding. So, all right, let's close with. I always like to leave the audience with something. So if you could give. Give one piece of advice to our audience about how to evolve their operations for an AI First World, what would it be?

Speaker C: Yeah, definitely. My strong opinion about this is don't go and build an AI solution. Make AI part of the solution, right? So we've got so many great products out there. You don't need to go and write a new product, right? That's AI first or something. Really where we see the value in AI is when it's behind the scenes, right. When it's not in your face, there's just things just seem to magically work and you don't think about it and you don't know why. You just know that it works and you got the response that you want. As we think about AI, I think that's, uh, the approach that we want to have is leverage its great capabilities, but it doesn't need to be the only thing that you're doing right. I think you think about when the Internet came along, it was a new and shiny thing, and everyone's like, great, it's now Internet enabled. Right now we don't care. Right. It's just an expectation that things should just work. And it happens to work because it's leveraging the Internet and the technology that we have. I think AI is. We're in that same phase right now. We're in that hype cycle where everyone wants to label everything with, oh, it's AI, it's AI, it's AI. We're really. The users are going to care less and less of that over time. What they actually just care about is, does it work right? Is it doing the thing that I want it to do and does it make me happy? So I think that's the focus, is just making sure that your AI kind of works with existing kind of workflows. It's coming to meet people where they are and help kind of accelerate what they're doing and not trying to force them to have to go and do things they don't want to do.

Speaker B: Well said. Well said. Well, thank you very much. Really appreciate your time.

Speaker C: No worse. Thank you.

Speaker B: M learned some very interesting things. Thanks for being here.

Speaker C: Thanks a lot, Matt. It's been a pleasure.