How do you turn AI coding chaos into a repeatable playbook?

Speaker A: Hi, and welcome to Leaders of Code. We are recording this episode at Snowflake Summit here in San Francisco. If this is your first time tuning in, Leaders of Code is a segment on the Stack Overflow podcast where we get senior engineering leaders in the room and ask them about the work they're doing, uh, how they go about building great teams and the biggest challenges that they find themselves staring down right now. Uh, my name is Ira May. I'm the B2B editor at Stack Overflow, and I'm here with Vivek Raghunathan, who is the SVP of Engineering at Snowflake. Welcome to the show.

Speaker B: Thank you for having me.

Speaker A: Yeah. How's the week been going for you so far?

Speaker B: It's been incredibly exciting. Uh, this is the time of the year where we get to show everyone what we've been up to over the last year. It's just an incredible experience. Seeing customers interact with and experience the things we've been building. This week gives me great energy going into the rest of the year knowing that we are on the right track. We are building things that will add incredible value to our customers, and they share our vision of the future. Awesome.

Speaker A: Uh, speaking about vision of the future, one thing that we've been hearing a ton about this week is about how engineering leadership is sort of shifting. Right. So with the cost of code generation, the cost of writing a line of code just sort of approaching zero, the bottleneck is less about writing code. Right. And it starts to shift to more kinds of work around managing and orchestrating teams, defining, intent, thinking about what you can do strategically to deliver that real business impact. So I wonder, for you and your role at Snowflake, have you really thought differently about what engineering leadership looks like in the light of all these changes?

Speaker B: I think the more interesting question in my mind is also how is the business of producing software changing by itself? Because at some level, uh, roles and what people do is an output metric to. How is the business of software resulting in software being produced? Right. We have taken a fairly methodical approach to doing this over the last 12, 18, 24 months, and I'm happy to go through it in great detail. The way I think of this is code gets birthed in an engineer's head, then on their workstation or in a cloud workspace of some form, and it goes through a bunch of stuff before it makes its way into main or master. Right, sure. Uh, I'll call that the inner loop of software. There's the outer loop of software, which is now that code in main has to get released into production. There's what I call the second outer loop of software, which is, uh, bugs are found in production and they make their way back into the systems. Either support tickets or incidents or some kind of anomaly detection in our systems. And how do fixes in response to those bugs make their way back into main? So I'd say that is like Outer Loop 2, if you will.

Speaker A: Yeah.

Speaker B: Uh, and then there is the question you asked, which is how's the roles, uh, and responsibilities look like in this new world? Uh, how is the act of building software itself changing? And then how do you structure your organizations in response to the fact that the act of building software is. So I break it up into those five tasks. Happy to dive into any one of them.

Speaker A: Yeah.

Speaker B: Um, but we're making like very methodical progress on each of those five vectors. Uh, almost building up to the last vector, which was the question.

Speaker A: Yeah, I'd love to just start at the beginning. If you could unpack it a little bit for us, that would be great.

Speaker B: I'm happy to do it. I think the inner loop is at some level the easiest because at its core coding agents make it easy for you to write more code and better code faster. Right, sure. We started very much with an approach to. Andy Grove once said this, and it's a very popular phrase that gets reused and thrown around quite a lot now. He said any platform shift, you need to let chaos reign m before you reign in the chaos. And so the approach we took initially was we're just going to let chaos reign. Habit formation is hard. Habit breaking is hard. We need to encourage people to use coding agents. Uh, we're just going to measure adoption. We're not going to measure lines of code. We're not going to measure PR certain. We're not going to measure any of these metrics that are easily gamble. We're just going to measure. Do you use this twice a day? 95% of you use it weekly. Um, I'm m going to encourage you to use it to write code, review code, understand code, write design docs, do everything right. We're going to give you every possible tool. We're not going to say no to any tool that you want. Right. And clearly that is an age of little more letting chaos reign than reigning in the chaos. So, uh, that's step one. Now you have chaos training. A bunch of people are using coding agents. 95% of our engineers use them on a weekly active basis. I think 97 are all of them equally effective doing it. That's the second question to ask.

Speaker A: Right.

Speaker B: It's the difference between using it to save yourself 20 minutes a day and using it to do 80% of your job. And I would posit that the difference between folks who are using coding agents and mastering coding agents or using it effectively is 14 AI design patterns. I use that word very deliberately. Design patterns. For those of us were in the software engineering industry, gang of four, you know the book with a whole bunch of like, you know, the factory pattern or the command pattern and so on and so forth. And that book created a language of almost how you think about becoming more effective writing software. Right. I think a similar thing will happen with how people are using coding agents. And so we have discovered, I'm going to say about 14 patterns right now. I stored them from a numbering of zero because zero indexed. And uh, they represent patterns that our most fearless explorers, our AI czars, if you will, the people who are at the cutting edge, have discovered as effective ways to use coding agents. So I'll give you an example of some of these patterns.

Speaker A: Yeah, I'd love to, uh, love to hear one.

Speaker B: Pattern one is, uh, it says plan in English and what it means is first plan, use plan mode in your coding agent. First figure out what the plan is in markdown and then write code. Right. So have a. And I'm happy to show that to you on my laptop when we. We have a XKCD comic with these 14 so we can easily disseminate knowledge in the organization. Uh, pattern four, I believe is fence your robots. And what it means is you can have a single agent. It's kind of slow. You can start a bunch of agents and have them all work together and then you have chaos. Or you can have git work trees that run each of them independently and fence m them a bit and they work on stuff. Uh, and that can dramatically improve the productivity of most of our engineers.

Speaker A: Okay, so order from chaos.

Speaker B: Order from chaos.

Speaker A: Yeah.

Speaker B: Pattern eight is what I call the TLA pattern or the TL of agents pattern. It is your orchestrator. The agent you're talking to, the master agent you're talking to is never holding a lot of context. It is delegating work. It is using an agent team of some form to actually do the work. And so it is always its brain is free to talk to you. Pattern 11 and 12 or 12 and 13 are new patterns. They are patterns around continual learning. There are patterns that recognize that you can mine memory.

Speaker A: Mhm.

Speaker B: And you can promote it into skills overnight.

Speaker A: Mhm.

Speaker B: And that will make the system get better as you use it. And if you do this in a m multiplayer way where everybody in the team is doing it, then you get to harness tribal knowledge. Right. Now, uh, each of these 14 patterns are patterns individual our best coding our best like AI forward engineers are using um, uh, so now you have these 14 patterns. Now you have a language to speak in terms of how you upskill or reskill the engineering teams if you will. And um, when you do that you can then progress to what I call stage three, which is rain in the chaos. And there you start taking some of these paved patterns and you say how do I get more people in the organization using them? Most interesting technique we have discovered is just a simple act of creating space and time for them. So, so we do these things called focus weeks. We do them pretty regularly and so week where everyone in the org just takes the time off to figure things out. It serves two purposes. There's maybe 95% of the organization is what I call exploiters. Uh, the term feels like very spicy but it actually means something very simple. It means they just want to know the paid paths and use them.

Speaker A: Right. Give me the good stuff.

Speaker B: Just give me the good stuff. I don't want to do all this learning. I just want to use this.

Speaker A: Sure.

Speaker B: My PhD is in RL so explore and exploit is very common. And so these are the exploiters. There's 5% of the organization and they need the time to exploit. They don't know the patterns. They're like I'm too busy. Right?

Speaker A: Mhm.

Speaker B: And then the 5% organization is what I call the fearless explorers. These are the people paving the paths. These are people creating these best practices. They will go find the time. They're doing it on the weekends, they're doing it in the evenings. They're doing it sometimes, uh, when they're out with their friends. They're busy hooking up a mobile app to code and doing stuff. And these users just. These engineers just need the time to explore. And so we give them this week to go in and I call it raising the floor and raising the bar. The first guys were raising the floor on the second set of people were raising the bar on. So that's the inner loop.

Speaker A: I like that.

Speaker B: Right?

Speaker A: Yeah.

Speaker B: What happens if you do this is like roughly speaking where we are. 97% of our users are of our engineers are weekly actives on coding agents and code is up about 1.5x in the last like year over year, maybe 3x up uh, last three years time to merge like things we know are characteristics of high performing teams. Like they review code fast. They're like a uh, music band really, just like riffing off of each other.

Speaker A: Yeah. You start to see them kind of compounding on them.

Speaker B: And so those kinds of patterns are all up into the right. They're all up like between 1 and 2x based on every metric. Yeah, that's just the inner loop, right?

Speaker A: Sure.

Speaker B: So I said there were five stages. That's just the inner loop. On the outer loop. We're using AI to do, to basically rethink every step of the outer loop. I think of three steps of the outer loop. Right. Can uh, we release code faster and better? Uh, the second is, uh, can we test uh, harder? Can we have a lot more tests, a lot better tests and a lot higher coverage tests? And the third is can we debug smarter on the first step? And this is very important for our customers, our customers, our enterprise customers. We are not a consumer startup. Our customers expect. Our customers expect, you know, no bugs to ever reach production.

Speaker A: Right.

Speaker B: Um, which as a software engineer is like hard to pull off.

Speaker A: Uh, yeah, it's a tall order for sure.

Speaker B: But this time last year it used to take us. So we go through extensive validation of our releases. We do like hundreds of thousands of tasks. We run every performance benchmark again and again to make sure nothing regresses. Individual queries on customers are part of our regression benchmark. They contribute things in. We cover 90% of everything we could ever possibly cover. That stuff used to take us an inordinate amount of time. Like this time last year, we were up at 15 days to bless a release. Lots of our peers will push releases very slow. I was at a consumer company before. We used to push like very regularly. Uh, and so we brought validation time down in the last 12 months to a day. And a lot of it has been better tooling, a lot of it has been better discipline. But a lot of it has also been can we use coding agents to automatically go through the. Oh, we found a bug. This is a blocker in the release. We go diagnose what's going on. Potentially put up a PR on GitHub automatically and let the person whose code it is go take a look and say, is this fix, fix what I saw. So doing that has gotten us down from like 15 days to a day. Lets us really safely release all the features Christian announced yesterday. Tests are up like 3 1/2x. Uh, LLMs make it really easy to write tests and so people have Taken advantage. One of the patterns I think is pattern two is basically use coding agents to first write the test for your feature and then write the code. Right. Do a new spin on test driven development from like a decade ago.

Speaker A: Yeah.

Speaker B: Uh, so doing that, the teams have been doing that of course. So Tessera OP3X which means while we're doing faster, people ask me if you're doing like quicker releases and faster releases, does that mean your quality is kind of regressing?

Speaker A: Right.

Speaker B: And the answer is like we're also releasing much more safely. There's far fewer escape hatches into production. Finally observing and debugging. Right. Mhm. Uh, we're completely rethinking how we do this. We have thousand engineers writing, I'm going to say like 7,000 skills at this point. And the core insight is uh, AI lets us completely transform the operational life cycle. There's a lot of ops IP stuck in engineers heads.

Speaker A: Yeah.

Speaker B: Tribal knowledge. When this alert goes off, do this. So we're able to take lots of what people would call runbooks which get outdated very quickly and instead build them as versionable CICD workflows of skills and that are part of the Coho coding agent. Uh, they're packaged into things called profiles which are units. Like oh, if you have an issue with streaming, this is the set of skills that you need to debug this issue.

Speaker A: Sure.

Speaker B: All customer issues have a blast radius skill which is like which customers are affected by this thing. And so when you do so that's one you get tribal knowledge encoded into repeatable workflows. You get complex deterministic workflows which are in people's heads now you can put them out into an LLM. Uh and the third thing you can do is like you know being on call is a real pain in the ass. Like most of my teams don't want to be on call.

Speaker A: Yeah, it's not a popular thing to do.

Speaker B: So it reduces the toil a lot. Like what we call KTLO is about 30% for us and I'm looking to reduce that to like 5%. And I think I have a path to getting there in like maybe not today, but like in a couple months. What we've discovered is the teams that work with production and coding agents, there's like a four step maturity model. First they write a bunch of skills. Mhm. They encode all their issues that never come in into a skill that cortex code can use. The second thing they do is they do event driven AI. So they hook it up to pagerduty or Slack or one of these things. So now they can, uh. The third thing they do is they start taking complex workflows and using an LLM to encode, like this multi step reasoning thing, Go call support, get them to do this with the customer, go do this other thing, ping this person on Slack. Uh, and the fourth thing they do, which is the thing I told you earlier, is continual learning. So as you discover new things in an incident or an on call investigation, you encode that back into, uh, the skills that the coding agent. So far, pretty good. The vision is we'll have primary on call, be an agent, and then have humans do secondary and tertiary or vice versa. You can have one of the on calls, the Triagers, be an agent before you, and that immediately reduces the actual workload. I want engineers to say, I like being on call because it's fun. Not like, oh, this is something.

Speaker A: I think they're excited to jump on stuff.

Speaker B: Not something like, I only want to be on call once in nine weeks because, you know, uh, it's the worst week of my quarter.

Speaker A: Right, right.

Speaker B: So that is roughly like, you know, how we think of the outlook.

Speaker A: Okay.

Speaker B: Um, product is still very much like work in progress. If you saw Snowflake Cocoa and Snowflake Cowork, we're experimenting with how to build products completely differently in those products. Finding small empowered teams work really well.

Speaker A: I was going to ask you about that because I know that you've talked about impact per engineer and wanting to maximize individual impact on a team. And you were talking about certain folks who can really push the envelope and really experiment with the agents. Where do you feel like is the, uh. I guess how do you go about increasing that impact per engineer, but also not allowing AI to sort of raise the bar for everybody in a way that just sort of puts everybody on the same.

Speaker B: Yeah, I mean, it's a great question. Right. I think the way I think of this is, uh, every big transformation has, uh, pioneers, people who are like Lewis and Clark. They will go and explore settlers, people who will come and follow the explorers or the pioneers into new territory.

Speaker A: Yeah. Kind of set up shop and sort of put the infrastructure in place.

Speaker B: And then it has the skeptics. Right. The resistors. Uh, they may not even be doing it explicitly, they may be doing it implicitly. Right. For me, it is very important for us to meet each of these constituencies where they are.

Speaker A: Mm.

Speaker B: I've had people walk into my office and say, you know, I went through my seven stages of acceptance. Uh, today and then you switch from being that implicit resistor to being a pioneer overnight.

Speaker A: Okay. Yeah.

Speaker B: So we had to meet everyone where they are. I think one of the realities of the moment is a lot of us got really good. I mean, if you spent your life getting really good at something.

Speaker A: Mhm.

Speaker B: And then you were, uh, encountered with a deity, uh, that is like about as good as you are or better.

Speaker A: Yeah.

Speaker B: Then there's just like some amount of going through those seven stages of grief, if you will.

Speaker A: You have to think about your value in a different way, I think.

Speaker B: And I think a lot of folks in the software industry are going through that. And I want to meet them where they are. Like, I want to take them along on this journey. I think you asked me what the role of leaders in this world is. And at this moment I would say the biggest thing a leader can do is lead from the front, is show people how to be on that journey. Like, uh, encounter their own emotions first, make sure they're at the place where they are pioneers, and then take their teams along. Now is the time for the charge of the Light Brigade, if you will. Not leading from the back.

Speaker A: Yeah, yeah. Well said.

Speaker B: And uh, so we find to your point, it's actually easy for us to identify the pioneers because they're all so. They're like kids in a candy store. They're like, hey, I discovered this thing on the weekend and I built all this stuff and um, I need one hour of your time to tell you about it right now. And I'm like, wait, is this urgent? Uh, and they're like, no, it's very urgent. And then they show you what they're doing. So they're fairly easy to self identify. They also show up in. They're suddenly magically 100 times as productive as they used to be. It's hard to miss them. Right?

Speaker A: For sure.

Speaker B: Uh, they're not always the people you Think were the 100x engineers before coding agents. There's a different set of skills that are being amplified. It is curiosity, it's adaptability, it's willingness to learn.

Speaker A: Yeah. Um, yeah.

Speaker B: And so they're easy to identify. I like to bring the, what I call the 95% of people who are exploiting all so long. Mhm. Um, so we think of this as less binary and more. It's a continuum. Like the more of those, like best practices. I told you, you're following, the further along that continuum you are. We internally joke about a, ah, Yegi scale named after Steve Yegi. It's like, are you yegi5 or are you yegi7? And I've almost told my teams we need to 5x the number of yegi7s in the organization and get more people. It's not like they'll magically. We'll magically hire them or something. We'll take people who are yagi3 and

Speaker A: take them closer to transform them. Yeah, yeah, I wonder, kind of. On a related note, one of the things that I've talked about with a bunch of the guests that we've had on recently is this sort of like the promise of agentic coding and via coding, running into the limitations of scale. And so just like, I mean from your perspective with the, the scale environment that Snowflake deals with is that. Have you been finding that there's a real like, disparity between.

Speaker B: Yeah, no, we're not like, I think if anything we are seeing that coding agent in the hands of the right people. M can, um, make amazing things happen. So you heard Christian talk yesterday about the interactive compiler? Yeah, he talked about a rewrite of our compiler. The compiler is the, you can call it the brains of any query engine. It's the thing that lots of blood, sweat and tears goes into. It's the thing that most of the secret sauce of the query engine goes into. Our compiler, um, is often the place where the hardest things that we do go in. So when you see efforts like our iceberg like support or our dynamic tables feature, those things all take. Or our unistore feature, those things all take time because work in the compiler is careful, painstaking, um, and sometimes the compiler is just slow. Uh, and this matters when you're dealing with things like interactive workloads. Workloads like some of your listeners might use Clickhouse for, or Trino for. Part of the reason an analytic engine like Snowflake doesn't automatically out of the box work great for those workloads is the, uh, compile time can be pretty high.

Speaker A: Mhm.

Speaker B: So if you have a short running query and the compile time is pretty high and you have a bunch of dead costs sitting at the front of the actual query execution. So our tech lead for the compiler effort said, hey, I'm going to take three people.

Speaker A: Hm.

Speaker B: I know all the things that I would do if I had to rewrite this compiler. I want to rewrite it using coding agents, me, a set of coding agents and three engineers. But these are all engineers that are domain experts. Uh, what they're seeing is things like 40x improvements.

Speaker A: Wow.

Speaker B: Now of course a lot of that is he knows exactly where the juice in the system is.

Speaker A: Does that surprise you though, to have that kind of like, uh, I mean

Speaker B: it surprised me a good way.

Speaker A: Yeah.

Speaker B: There are benchmarks that uh, we had no chance of ever measuring up to customer workloads. The customer was like, if you do this, I will move this over tomorrow. And we were like 3x away for years.

Speaker A: Boom, we did it.

Speaker B: We're there. Okay, so to your point, I think coding agents make the ambitious possible. So you can have infinite ambition. Right?

Speaker A: Mhm.

Speaker B: So that's at Google. Larry Page would say this, like, aim for the moon. It's always easier to work on harder problems than easier problems because no one else wants to work on them.

Speaker A: That's right.

Speaker B: So you have the field to yourself.

Speaker A: You have it to yourself. Yeah, exactly.

Speaker B: Uh, uh, and I think coding agents take ambitious people who want the field for themselves and suddenly say, you can do this quickly. You don't have to do this. It's not a two year project. You can go try ambitious things, whether it's on the product side or the technical side.

Speaker A: Right.

Speaker B: Completely by yourself.

Speaker A: Yeah. I mean we've had a lot of sense.

Speaker B: The product we just announced came out of someone hacking over Christmas break and saying, oh really? I'm tired of like creating these semantic models myself. I'm just going to build a system that does this for me.

Speaker A: Yeah.

Speaker B: And one thing led to another and we're like, wait, he's onto something, right?

Speaker A: Frustration plus the agentic.

Speaker B: So I think technical ambition, product ambition, more ambitious the better. Uh, the moment is made for people who are, who are not afraid. We're very excited to be part of the journey that is just rethinking how data and AI can meet synergistically to create great outcomes. We see our mission as, uh, AI that makes data go faster and data that makes AI go faster. And we think if we can create that loop, our customers will see a lot of value. That, uh, is the thing. I wake up in the morning jazzed by. That's the thing. My teams wake up in the morning jazzed by. We're eager for the world to see it as well.

Speaker A: You've been listening to leaders of code. My name is Ira May. I'm the B2B editor at Stackover Flow. And I just want to thank Vivek Raghunathan for joining us again today to talk about, uh, all things Snowflake Summit. Thank you so much for being here.

Speaker B: Thank you again, Sam.