AI's unglamorous wins for developer productivity with Tara Hernandez from MongoDB

Speaker A: We're enhancing the tooling and by how we are using the tooling, we're driving changes to the communication and process that helps the team overall accelerate. But it is slow. This does not happen quickly. And I think a lot of other companies are seeing that too. You think, oh, I'll fire all my junior engineers and have all these agents and all of a sudden we're a billion dollar ARR. And I'm like, uh huh. Notice people aren't saying that as much anymore. Foreign

Speaker B: I'm Rebecca Murphy and this is Engineering Unblocked. Engineering Unblocked is brought to you by Swarmio, the engineering intelligence platform that's trusted by some of the best software companies in the world, including startups like Superhuman scale ups like Miro and companies from the Fortune 500. On, um, today's show, I'm talking to Tara Hernandez. She's the VP of Developer Productivity at MongoDB. Ah. And I have a lot of questions about how one ends up with that title. So Tara, welcome.

Speaker A: Hello.

Speaker B: You are the first VP of Dev Prod that I have met. I've met directors and senior directors, but I've never seen it hit the VP level. So, uh, yeah, tell me, tell me, what do you, what do you do?

Speaker A: Well, funny story, I have never met one either. I like to tell this story. I got this cold email from Mark Porter, who was the CTO of MongoDB at the time. And uh, I was at Google and, and I was like, no way that some CTO sent me a cold recruit email to talk to me about a VP role. And so I was doing all of the things they train you to do. You know, I was mousing over any links and checked the mail headers and I finally did the LinkedIn thing. Cause I saw we had a bunch of people in common. I'm like, do you know this guy? You know, would he do this? And I, uh, was talking to Brad Porter, no relation. And he goes, oh yeah, Mark, Brad and I worked together at Netscape a million years ago. And he goes, oh, Mark would totally do this and you should absolutely talk to him. You know, would get on like hair on fire. Which was a funny thing for a bald guy to say, but there you are. And uh, yeah, so I talked to Mark and then he described the job. I'm like, are you sure this is a VP role? I actually asked him that. He goes, yeah, yeah, yeah, it's going to grow. And it did. Um, and so I got, that was my first VP job. I had been up to senior director at that point. Um, and this was my first VP role and I uh, started out overseeing the CI ecosystem. Specifically, um, MongoDB has a proprietary Evergreen. Proprietary CI system called Evergreen, don't judge us. It predates a lot of commercial, uh, uh, solutions and it is purpose built to build and test a distributed documentdb. We run about 3 to 400,000 compute hours worth of tests for this thing a day. So our orchestration is dialed.

Speaker B: Yes.

Speaker A: Yeah. Um, but now uh, sometimes I refer to myself as the VP of kitchen sickness. I've got build systems, test frameworks, performance tooling and analytics security infrastructure, uh, in now AI, own AI, which is a funny thing as an, as a confirmed AI skeptic. Um, it's been an interesting journey. So. Yeah. So developer productivity at MongoDB is everything that's not SRE and is not product development.

Speaker B: Seems true. Uh, like eventually that sounds about correct. So you weren't raising your hand and saying, hey, I'm looking for a VP role in developer productivity. He's looking for a VP of developer productivity. Like how do you get there? What was happening before you showed up?

Speaker A: Yeah, MongoDB, uh, the history was there was the core database and its ecosystem, all of the drivers and client libraries and tools and whatnot. Then uh, I forget exactly when. Maybe 2013, 14, somewhere in there, I should probably know this more accurately. Um, they started to have a SaaS service atlas and so there was a separate engineering team that spun up. So there was the core team and the cloud team. Roughly. Um, and there's a lot of differences in approach because of the distribution. The core server team was really more thinking about the on prem ness and a lot of the, you know, the rituals and processes were around that. And the cloud team, we need to have cloud native control plane and uh, all these new things. And so uh, there was a lot of, I don't want to say it was outright duplication, but there's a lot of similarities in function that was sort of tailored to those teams. But after time, this is very common at startups. Right. You have this team over there or Apple, as far as I know, still does this. Every team does it differently and then they pay a bunch of people to try and unify it. All the people I used to know at Apple aren't there anymore, so I'm not sure if they still do that. But anyway, uh, Mark and his team decided they wanted to break out some of the tools that were in common or increasingly becoming in common and have that be a separate organization. And at the time, I guess Organizationally, they decided that the VP role made more sense. And then subsequently, uh, as happens, they're like, okay, well you're doing this, we should probably have you do that too. And what about this? And so the size of my remit has tripled, I think almost quadrupled in. It'll be almost four years. It'll be four years in May.

Speaker B: That's a very clear remit, um, to start with, of like take these two things and make them one thing where it makes sense to make them one thing. Right.

Speaker A: Yeah.

Speaker B: And outcome is also like pretty clear. There's not a lot of this is what I, what I. Well, I don't know. You can tell me I'm wrong, but this is the kind of thing that sounds like what I would call just work.

Speaker A: Yeah, I mean my, I've done the same thing my whole career at different levels, but, uh, in the course of my career it's been called build engineering, release engineering, DevOps, though I hate it when anybody calls a team DevOps, um, infrastructure. Uh, uh, I think there's one I'm missing. It's like, it's all. At the end of the day, it's how do you take a group of people and get them moving in the same direction procedurally, culturally, in the most efficient way possible for the ultimate end goal, which is ideally the relative rapidity of quality software releasing. And, uh, what does it take to do that? Um, when I went to Pixar, um, Pixar was a movie studio, still is a movie studio. It was a movie studio that had a software development team and that. And back in the late 80s and early 90s and then into the, let's see, when did I start there? 2002. The early aughts. Right. That was. They were just starting to realize, okay, we had more academic approach. Um, the managers in what was called the Studio Tools organization were all PhDs with many patents. You know, um, uh, people like Rob cook and Tony DeRose and others. And they're just, you know, these people have written books, right. And the hierarchy felt more like sort of academia. And they would have a version of the proprietary software for every movie. And that made. That worked for the first couple of movies. Right. And then they realized, well, if, okay, we want to make movies faster than every four to five years we want to. It's like, well, now we have to be able to do things in parallel. And so. I don't know how I like to say this. I don't. People might dispute this characterization. I like to say I taught Pixar, how to develop, to do software development. I think that's an overstatement. Let me just, let's just acknowledge that. But you know, using branching, using version control, it was, that was not 100% right. Um, how to do version libraries, like having a proper build system, having a CI system, uh, having a bug tracking system that didn't require you to write SQL statements. These are things that I helped with them.

Speaker B: So this is early 2000s, right? Auths we use that word. Um, this is early 2000s. It was reasonably normal then for companies to have not figured this out. Especially like Pixar is in the vicinity of movies. Movies have been made since the what, 18, late 1800s. Made like I can't remember when they started movies. But like so, and we see this with our customers too. We have customers who are, you know, hundred year old manufacturing companies, um, trying to figure out like, okay, how do we do, how do we do software? How do we bring software to these, to these practices? Was that your remit at that time to like teach them those things or were you kind of trying to open their eyes to something that they just didn't know? They didn't know. A little both.

Speaker A: A little both. I was hired, um, so the reason I got the job at Pixar was my VP at the previous company, Blue Martini, had been at SGI and uh, knew the hiring manager at Pixar. So a lot of people at sgi, a lot of them went to Netscape, um, and a lot of them went to Pixar. And so, um, the hiring manager is this woman named Marianne Gallagher. And I think last I knew she was at VMware. Really great woman. And her boss was a VP by the name of Rob Cook who was running the studio Tools organization. And so when they hired me, they said help us manage the software such that we can do more than one version of software at the same time. That was the remit that I was given. And the other part of it was that I was the first sort of pro infrastructure person previously. And this is very common at many companies, not just movie studios, that you'll either delegate like the last person hired or the lowest person on the totem pole or someone who's, you know, that's not, you know, infrastructure is not real engineering work. Like you just go take care of that, right? Um, somewhere along the line they decided, oh well, maybe there's people who actually do this is what they do. The ironic thing, by the way, as an aside, was my specialty in information, computer and information sciences. At UC Santa Cruz was in graphics and animation. I never used it, not once. By this point, I couldn't have done a Z transform if you put a gun to my head. But it was kind of cool to meet people who had written some of the textbooks I had used. Uh, but anyway, I was hired to just. That was my job. My job was not to work on the Pixar proprietary software, to understand anything about graphics and animation. My job was to work on the infrastructure that this 90ish person, core studio Tools team, and then another couple of hundred people who would write, like, special effects plugins and things like that to make sure that they had a good environment. And I ended up building a team out at Pixar. Um, I think at its peak there was 13 of us. Um, and the one time I was paid to write C code was at Pixar. I wrote a version library. So, uh, yeah, you know, and the job both never changes, but then always has funny little nuances, right? Because you could take, and I give this example between when I was at Netscape and then I went to an early E commerce startup called Blue Martini. I ended up pulling a lot. I think every senior engineer that had worked for me at Netscape ended up coming to Blue Martini eventually. And we used a lot of the developer tools that we developed at Netscape, um, Tinderbox and Bugzilla and Bonsai, which was an early, uh, HTML code introspection tool. A lot of what we did, uh, at Blue Martini, even though I was the same and my engineers were the same, the tools were the same, the things that we did were differently because the engineers were different, the products were different, the requirements were different. Right. And so, um, one of the things that was an early lesson, which is development, the act of software development is, um, I view it as having three major pillars. There's the tools pillar, that's obvious. What do you use for your compilers and your CI system and your testing, and how do you deploy and where do you deploy? Right. Um, but then there's also how you talk to each other, right? How do you convey. And not just like in the be nice to each other kind of sense, but in the, you know, how do you communicate, how do you do documentation? Right. How do you take a group of people and have them know things? And then there's the processes that bring them all together. How do you have, you know, do you have coding standards? How do you do code review, how do you do bug tracking? Right. And those three things, and I call that, you know, the first one is kind of like uh, culture, environment and then process and policies and then tools. Three pillars. Technology is only one of three. The other two could have some technology involved in it, but it really has more to do with uh, that going back to that idea of a group of people, right? And that group of people is, is your set of requirements, like how do you make this group of people work? Right? Because other groups of people might want to work in a different way. And there is no one right way. There is one. Maybe there's one right way for this group of people. Maybe. And not always.

Speaker B: So given that. And I talk a lot about how, like how size, age and culture determine kind of the, even the solution space available to you in a, in a productivity uh, situation. A team that's releasing, you know, twice a year has different goals than a team that's releasing as often as they want to. Um, and the shrink wrap software which like literally, you're not literally sending out shrink draft boxes anymore. But like I still think of a versioned on prem software as leveling. I might have had one of those at one point. You never know. Uh, was it the, was it a flop gear or Was it a CD at that point?

Speaker A: This is Netscape Navigator version 3. This was the last version that went out on a 3 1/2 inch floppy. Version 4 was the first one that was CD only.

Speaker B: Nice. Hey, I loaded computer uh, programs off cassette tapes.

Speaker A: So I'm like Commodore, we're dating ourselves.

Speaker B: Indeed. Um, but you know, the dating ourselves I think is interesting because like throughout this there is, you know, this, this theme. I mean I don't know how much you were talking about productivity when you're um, writing floppies at Borland. Um, but at the same time like you're writing floppy's at Borland. And that's really annoying I assume. And at some point you raise your hand and say like could we make this suckless? Um, or may, like maybe, maybe not. What is productivity? What has productivity meant to you throughout this experience?

Speaker A: Yeah, it's a really great question, right? When you think about like my first job was Borland. I worked on the languages team which was the compiler team. And at the time the Borland compiler was the number one compiler in the industry. And uh, actually a lot of my peers at UC Santa Cruz, for their first job they tried to get into Borland tech support, you think really Borland tech support at the time that was elite class developers. They would help you debug your code. I don't know if you remember that because this is pre World Wide Web, so you couldn't just go to Stack overflow and look something up. So, and if you paid extra, I think they would help you debug your code on somebody else's compiler. But for Borland compiler, you got a certain amount of tech support, what we consider professional services now. But uh, you know, it's interesting. The Borland, I think was the last version that I shipped, I think was Borland version 4. It was something like 33 and a half inch floppies and 55 and a quarter inch floppies plus paper books. So the shrink rack box, I swear weighed like 30 pounds. And then the next job I went to was Netscape. Okay, so now this is like the beginning of the early public Internet, the Internet for normal people, um, for grandma, as we used to say. Uh, and now we still had shrink wrap software. As you saw. You would go down to Fry's or CompuServe or whatever, but you could also, if you had a modem, you could download it. And so a big part of what my team started to do was to, um, it was to push bits. And we realized, well, if we're just pushing to the FTP server one at a time, uh, and using completely insecure, uh, mechanisms to do so, but now we could push bits out, right? And people could log onto their modems and make the funny little noise and download directly and not have to go to the store, right? And fast forward, you know, a little bit into the teens, I guess. And now I forget exactly when AWS launched, but by, you know, the late aughts, early, early teens now you, you know, you don't need to run software. Well, client server started before then, but like, you could even, you know, rent services by the second and run stuff randomly without even having a computer. Right? And so it was, it was amazing how, you know, my, my VP who gave me the recommendation to, to go to Pixar, he used to keep punch cards in, uh, his pocket. He used them for, for notes, you know, and, but he had used those in when he worked, I think at IBM. And so from punch cards to where we're at now, which is, um, you know, your, your smartphone, your Pixel or your iPhone or whatever, you know, has more compute power that than probably the entire United States of 30 years ago.

Speaker B: Well, I think about what I have seen and I think what my parents have seen, and then I'm like, I've got nothing.

Speaker A: Yeah, right, fair, Totally fair. So we think about what, what did productivity mean before we started talking about Productivity, I think some of the same principles applied. It just was a different context. Right. You're still talking about a set of tools. It was just the set of tools I had to care about in the beginning was shell command line utilities, revision control systems, installer engines. I had to write a lot of install code, um, and then started to have some network remote things that came into play. But the other part of it was, is there a bug tracking system? At Netscape, we wrote our first. I like to say I wrote CI before Martin Fowler invented the term. Right? We did. My team, our first TI system, we didn't call it that, but it was, you know, on Windows it was a dos, it was a batch script. On Mac it was, um, applescript. Ah. And then our UNIX guy was lucky because he actually was able to write a BASH script on his SGI machine. We were all very jealous about that, but that's where it started. And then everything turned into Perl, right? Perl became the language of infrastructure, right? And then when I was at Pixar, we finally gave up on Perl. I remember I was trying to rewrite the build system in as object oriented Perl and I realized, why am I doing this to myself? And Python had finally come along enough and had enough momentum that there was, you know, I was like, all right, I guess it's time for me to learn Python. And I was able to rewrite the Perl build system in like half the time with half the code. I'm like, okay, well, obviously that was the right decision. You know, these days I look at all this, like gems and M's and all of this, you know, and the public NPM repo and maven like, God, I miss cpan. But, you know, it was still about how did we talk to each other, what are the documentation processes, how do we track our issues, how fast are we able to get code in and testable, even if we're not shipping it, right? And what are the tools that we use to do that? I think the World Wide Web unleashed infrastructure as a concern that mattered. I think before the World Wide Web infrastructure was for the people who couldn't be really in real engineers. But you know, my team was writing, maybe it wasn't code that people would look at and put on a wall frame, like, this is an example of great Perl code. But it worked, right? And now speed is. And we talk about Internet speed, right? That started in the late 90s. And then now speed becomes more important than quality because we're increasingly not to, not to like shame the entire industry. But there's a different conversation you have about quality when you're doing physical media versus when you're publishing something digitally, right? Because when you make a mistake you just republish. When you make a mistake with physical media you just cost your company millions of dollars. So it changed the conversation, right? And then all of a sudden now it's like, okay, now how do we do stuff at stale. If I had an ounce of entrepreneurship, I would have predated Circle CI with a Tinderbox solution maybe and made millions. But um, I'm not an entrepreneur so other uh, people did that, right? But you think about like GitHub and uh, Atlassian. I mean that's all about developer productivity, right? At the end of the day it's about how do we communicate with each other, what are the bug tracking systems. That company would not have needed to exist prior to the Internet being to this average person. And so it really changed things, right? We talked about inflection points. It's like okay, from floppy to optical media to Internet based. We also talk about the other big transition coming out of Netscape was from companies who made their money selling hardware of which the operating system was just the way to use the hardware, to Linux. And uh, then the birth of the Linux foundation and then that changed everything. And then from Linux to cloud and now we have from cloud to AI. And so we see these really interesting inflection points. But from my perspective the three pillars of what we now call developer productivity still persist.

Speaker B: Still, still pretty much the same. Yeah. So I've. You've walked into my AI trap.

Speaker A: I'm. Darn shoot.

Speaker B: You said the word first though. You said. I did, I did, yeah. Um, but one of the things I'm m really meditating. I don't know, I don't meditate but like professionally meditating. Ruminating. There we go. One of the things I'm ruminating on is a lot of the reason that we, we have spent in the past so much money on improving the productivity of software engineers was because they were expensive engineers in cloud, right? Like those are your two big expenses. And so it was really urgent to you know, get the most out of every very expensive engineer that you're, you're hiring. Does our productivity conversation like move elsewhere in the age of AI or is it still firmly focused on the developer?

Speaker A: All right, I mean this might be a hot take, but hot uh, takes.

Speaker B: Appreciate it.

Speaker A: From my perspective, it really doesn't. It really doesn't. And let me tell you why. I think that. So my team was asked, I don't know, 18 months ago. Uh, the CEO Dave Itacharya, the time, and my boss, Jim Scharf, were like, okay, Tara, go figure out this AI thing. How can we have AI, uh, help us? Right? Everywhere you look on LinkedIn, and I live in San Francisco, outside of San Francisco, and you go down San Francisco, all the billboards, it's like, I hate all of them. Go figure this out. You realize that I hate AI. Everything about, uh, AI, to me, it's like a ginormous Ponzi scheme. This is what I'm saying, 18 months ago. And I'll get to the punchline eventually. So we started investigating it, as I'm sure a lot of people found. If you get behind the hype and you actually start doing the evaluations and you assess things, um, you realize, oh, this is very nascent technology, right? We got, you know, okay, here's a set of coding scenarios, and we're going to put a number of vendor. I don't want to get sued, so I won't say who they were, but there was, you know, 18 months ago, there was a number of vendors, and we went through all of them, and the code quality across the board was pretty abysmal. They all failed.

Speaker B: This is 18 months ago?

Speaker A: Yeah, 18 months ago. So. Which is, you know, might as well eternity. Yes. Um, so I ended up going with one and then continuing investigations. Because what's also started happening as it. As. As will happen is that everybody and their mother is now launching an AI startup in some form, you know, where Stanford is here, and they practically mint them in a. In a factory and like, pop, pop them out. And so it's like, oh, here comes this vendor and that vendor, so let's try them out. We might. One of my, uh, directors actually wrote a new procurement policy because having too many of the same type of tool causes all kinds of flags to go for finance, which is reasonable. So we're like, well, here's our proposal. And we were able to get that bought in, uh, to make sure that we weren't being crazy. And we started to see some improvements, right? And then you can see how fast those improvements started to happen. If you think about when Copilot came out, for example, as kind of like the inaugural, you know, it's like, ooh. And now you think about what Copilot was two years ago, and it's like sticks and rocks, right? And now we've got all these other things and agents and MCP instances And is an IDE plugin or is it just an ide or is it a command line? Because actually command line, and I say command line will always live forever because humans are who they are. And so it kind of depends on what you wanted. And we started to do some real research. This is the other thing and why I'm, I think it's still funny, um, that I'm considered kind of one of the AI people at MongoDB and in a larger industry is the number of vendors who provide any kind of telemetry. If they provide it at all. You can't trust it because not to malign them, but they're going to try to give you the stuff that makes their product look the best. But in most cases, it's not actually telling me anything meaningful. So then you have companies that will ingest the data on your behalf. And now you're paying for the AI technology. And now you're paying for the AI technology company that's measuring the AI. And I'm looking at the numbers going, well, this isn't going to work. So I go back to my data analysts and like, help me out here, scientist. So we figured out how to ingest it and we started doing some experiments and the first measurement that we did, and again, I don't want to get sued, so I won't say the tool, but we figured out that we had scientific proof that we had pre and, um, post users of AI and we were able to measure that at, at, on average, they saved an hour a week on the amount of time they spent coding and had zero impact on the amount of overall product velocity. Zero. So we saved an hour a week per engineer. Okay, that's a little bit of money. But we did the math. It wasn't more than it was. The amount we saved was less than the amount we were spending for them to save that hour. So the roi, therefore, scientifically no good. Right. So now we're starting to look at other tools and changing the game. And another thing that we found out, again, we put telemetry on everything. And I've seen people talk about this. Coding is not actually the problem. Right. We've now proven it. Not just us.

Speaker B: Posted that a couple of times.

Speaker A: Yeah, many people are now posting this. Coding is not the problem. Code review, release, deployment. Like, uh, you know, uh, what's your operational set of steps? Right. Um, that's where you start to run into friction. Or if you look at, if you still use the DevOps, uh, you know, the DORA key metrics, like did it do anything to your time to remediate, time, uh, to discover and time to remediate. Maybe it did, uh, the one, maybe not the other. Hard to say, right? Still nascent, but, um, it's harder to do that. And the reason that it's harder to do that and I think a lot of people are realizing, oh shoot, what's the security posture of these vendors? In order for it to be useful, you have to hand it the case to the kingdom. Well, if you don't want to do that, it limits to the amount of value you get. Right. So that's another challenge that we've run into. So, okay, so coding speed was not necessarily a win until you had the breakthrough which was the more senior people discovering agents and agentic programming and what are they calling now? Spec coding. Right. So now actually we're cooking with gas and we're able to show that for the right set of people you can do a lot of really interesting work there. But ultimately it's still constrained by the amount that, that senior staff engineers are the folks that often we see the most benefit. You're still constrained by the amount that they can either delegate or hold in their head because ultimately human still has to be responsible. Right. But as we also have them kind of leading the way from an agentic programming coding standard, that type of thing, we're now enriching our documentation and now enterprise. The search function is now helping the lower tier. Right. So now we're creating an ecosystem, but it still goes back to the same three pillars. We're enhancing the tooling. And by how we are using the tooling, we're driving changes to the communication and process that helps the team overall accelerate. But it is slow. This does not happen quickly. And I think a lot of other companies are seeing that too. You think, oh, I'll fire all my junior engineers and have all these agents and all of a sudden we're a billion dollar ARR. And I'm like, huh, Notice they don't. People aren't saying that as much anymore.

Speaker B: Not so much. This is another rumination of mine is, you know, the cost of AI is substantial. Um, absolutely. And is probably, I would suspect, going to increase over time. And even if it doesn't increase, then our usage of it is going to increase. What if this comes to a point of like, you have to say which engineers get access to AI?

Speaker A: Oh, well, we have that already. So there are many reasons why you might want to regulate it. It might have to do with software licensing. Like we don't Want if you're working on something where it would be, you know, eyebrow raise. If you're saying you're doing a lot of AI coding, okay. Those engineers don't get to use it. Right. If you say, all right, Finance, um, has said, I'm going to make up a number like you, you get, you know, $10,000 a month, then you're like, okay, well, that can actually be exhausted pretty quickly with, with the right model, uh, usage. So now you have to prioritize. Which projects are we going to prioritize or are we going to peanut butter it? Like you could make that choice.

Speaker B: Right?

Speaker A: And it's, I think that, I think part of the excitement there is that it is a dynamic discussion they should be having on an ongoing basis. Right. There's no one and done, especially in the AI space because everything's changing all the time. Right. Um, we were talking to one company that they got acquired. Well, that changed the conversation. Right. And I expect that's going to continue to happen until, you know, we kind of see things start to settle a little bit. Assuming that happens. But, but to your point, the current cost model, nobody can monetize. Well, it's going to be hard to sustain that. You're going to have to figure something out. The chips, people are realizing the power math doesn't work, so now they need to invest in different types of chip development to lower power consumption. I think there's going to be the unexpected, or at least not largely. Most people would find it unexpected. Where AI is going to drive us as an industry, I think, because as we find what are the actual friction topics that we're running into and how do we address them, that's going to take things in a new direction. But if you remember when we started, Everybody was about CodeGen, everybody was about CodeGen, and that was 18 months ago. Now you almost never hear about it as an interesting topic. It's CodeGen plus at a minimum. Or it's something else completely different.

Speaker B: That brings me back to my question of, uh, like, what does a developer productivity organization focus on in an AI world? I have so many more questions too, on AI, but let's, let's talk about that one first.

Speaker A: Well, so here's one that we've talked about, which is a developer productivity team is an engineering team, but it also provides services. So if you, uh, live in Slack or whatever your chat of choice is, if you're living in a chat world, which these days people tend to, you're a team that people, other teams come to you to ask questions, can you have something answer questions for you? Especially if it's something that was probably what we would call generously an RTFM moment. Right. So, uh, so one of the first things that we were successful at is we prototyped which was a Slack bot and we fed it everything in that legal allowed us to feed it, which thankfully was a lot more. Because in infrastructure where you don't have customer data or financials or anything else that they would be worried about. So we were able to show here's what the data ingestion is going to look like and we, you know, it's kind of Stack overflowish. In fact I think Stack overflow, if they had solved this, might still be a concern. So it's like, all right, people come in and uh, to our Slack channel ask dev prod. Uh, and we'd ask dev prod. And so it's like, all right, how well can the bot intercept those and reduce the load, the support load on the engineer who would otherwise get that question. Right. Okay, now we see that the accuracy rate is 50%. Okay, now we need to figure out how to improve the accuracy rate. Right. Okay, now that we've improved the accuracy rate, what else can we do? Oh, perhaps we can intercept a ticket, see that it's not able to be answered in Slack. Now we can generate an uh, issue ticket and make sure that it is by, have an agent make sure it's formatted correctly. Now have another agent see that there's an issue ticket and go try to solve it and then direct, direct a PR to the on call engineer to go and carefully review the code and see whether or not it would solve their problem. Right. And so these are, you know, I would consider these uh, for us to be experiments. We are not at the point I feel where a conservative company and a company that like us, it's like we are not an early stage startup. You know, we have to make sure that we are mindful of risk. But I think we also are a company that likes to experiment and likes these new things and is looking for ways where we can improve our overall productivity as a topic, not as a team, uh, through ways like that. And I think one of the things that's really fun about a team like mine is we can afford to experiment more because our customers are internal. And the worst thing that happens is we make them mad. We don't cause a uh, business deal to go south. And so that's another kind of implicit benefit. Especially why I like working at a developer tools company is that we can be customer zero. That proprietary CI system I was alluding to before, it's built on top of Atlas. So we stress test the heck out of our own product in ways that is, is amazing and wonderful. So, and my team feels proud when we find a stop ship bug. They all, everybody walks down with their chest out.

Speaker B: Like I think that is kind of what I'm alluding to is this, um, maybe there's less to do in the inner loop and more to do in the outer loop now, um, and more to do that, that is more ergonomics than um, tooling. Because I think of that like we at Stripe, we had the same. It wasn't even ask dev prod, it was just dev prod people asked. Anyway, um, and it was somebody's job to staff that for a whole week to staff that dev prod channel and deal with anything that came up. And it's. So it's very obvious how you could free up like some portion of a person at least by, by building that sort of thing. And I am, um, I'm personally skeptical of like, like you said, you, you get more developer output, but not necessarily more product outcomes. If you just focus on this, um, you know, cogen side of things. I think it's much more interesting like looking for those sorts of opportunities are probably higher leverage.

Speaker A: Right? And so you think about. So another example, um, in our, in our CI ecosystem, like I said, Evergreen is purpose built to test a, uh, distributed document database. All right, well, a distributed document database with all these shards and replication all, it generates a ton of logs. Now we're very big fans of OpenTelemetry Framework around here and we have our traces, but we still have logs and even structured, you know, we can have terabytes. Right? So there was another thing where it's like, oh, so now we have an agent that lives in our log viewer that tries to help identify and all of you know, the potentially terabytes worth of logs. Can we understand where the problem is and actually can we recognize patterns? Right? So that's shortening the debugging time, right? Something not, you know, maybe not terribly sexy if you think about all the stuff that people like to write about on LinkedIn or whatever. But for Joe Blow and Jane Doe in the trenches, uh, they love stuff like that because that's making their day incrementally better.

Speaker B: Well, that's what I love about working in a dev fraud organization, period, is that you can see the people that you're impacting, you can talk to them. So Easily.

Speaker A: Yeah, it's both the good news and the bad news that you know, you get that instant gratification or that instant karma if it's on the negative.

Speaker B: Yes, one or the other. But they will find you.

Speaker A: Maybe.

Speaker B: One other thing I'm curious about is any of your work expanding beyond developers to look at other parts of the sdlc, whether that's, you know, go to market support or customer support or uh, marketing or any of the above?

Speaker A: Oh yes, um, there's not that my team has been involved with ah, necessarily but that um, you know, our education department has done all kinds of uh, really amazing things in order to enable our customers around, you know, how we do documentation and support around our client libraries and, and things like that. Um, even ensuring that the model, including assessing, you know, all the models are the models providing uh, correct documentation for our customers. They go look up, well, how to talk to MongoDB through these drivers, you know, so they've taken on trying to figure out how to scan the Internet's models to whether or not there's good stuff out there that maybe we don't even control. But we, you know, we can try and go help get addressed by working with whoever the model owner is. I know that the technical services and professional services organization are always looking for ways like how do we improve the turnaround time for customer satisfaction. Right. And a lot of it I think is where we're seeing AI benefit again. We talk about it's not necessarily the sexy. You can go do a unicorn with like two people and, and a big enough EC2 instance to host your agents or lambda or whatever your you know, orchestration of choices, but how you can identify like the thousand death by a thousand cuts of friction that collectively actually probably results in a much bigger impact. It's just not as flashy.

Speaker B: Speaking of those thousand different things. So at Stripe, I ran unsuccessfully, um, I'll be honest, a program called Papercuts where um, developers could click a button in various places. They could click a button and basically say like this is not going well. And it would open up a ticket, uh, that would get reviewed. And some of it was one thing that we learned was that human to human communication was one of the most uh, expensive things that we were doing. The amount of time that you had to talk to another human just to like make progress was really costly to us. And so that not a technical uh, issue necessarily, um, maybe a documentation issue, maybe an organizational issue, um, how do you get attention to those sorts of paper cuts internally, um, just within your Remit of the engineering organization.

Speaker A: There's a couple of different ways. One of the things that I've started to incorporate is I don't think I invented this. I won't take credit for this. I'm pretty sure someone else invented this. But the concept of in developer SLOs or development SLOs, right? Um, we actually measure how, you know, if somebody brings a question to Slack, how fast do we respond to it and what's our time to time to reply and time to resolution. Kind of like what you do in production, except it's a people incident. Right. Someone has a question. So we measure that and then we figure out how do we improve that. You know, do we have the right on call rotations? You know, if someone acts in the Slack channel, does it trigger the right thing? Or now does the bot provide the right answer in a timely fashion? Um, but it's really, it's, you know, going back to, uh, this will feel kind of silly, but, you know, I remember reading John Doerr's book, right. Measure what matters, right? So if the mechanics of how people operate with each other is important, figure out how to measure it. Right? So, uh, we talk about onboarding costs. For example. Onboarding costs can be for a new employee. It could also be for an existing employee trying to use, uh, an API that was developed by a different team or they switch teams or you can imagine any number of scenarios. Or it could be the cost of onboarding your manager onto what you're doing. We actually have been talking a lot about the DORA key metrics as a framework to guide, uh, and up level a lot of this, this type of thing. So how long does it take to get all the reviews needed for uh, a technical scoping document? How long does it take for the engineer to get their coding done? Turns out really fast. We don't need AI. How long does it take to get a code review? How many code reviews did it have to go through? Um, and again, not to penalize if the answer is a lot, but to understand, oh, this team's code review response time is terrible because, uh, they have one person who is responsible for reviewing, uh, every code review. We didn't realize that. Nobody knew that. We hadn't recognized that. Right. So it's to, you know, it's too, it's. And it's also that I think this is a really important nuance using metrics from an enablement standpoint. Right. How long does it take the CI to do compilation? How long does it take the test to complete? Okay, well, the Build team needs to go figure out why this compilation is taking way too long. Right? Go fix that. Oh, now it's slow. It's faster now. How many tests are we running? Do we actually need to run all those tests? Well, how do we know? I think if I were to sum up Developer productivity, it's the constant search for and providing of critical information in whatever form that takes. And all the tools that we do, all the documentation, improvements that we have, all the processes that we have. It's information flow at the end of the day and it can be measured in many cases.

Speaker B: I think that's great to be, to be thinking about. Um, you know, we talk. Our three pillars at Swarmia are business outcomes, Developer Productivity, and Developer Experience. I'm pretty sure I could map those to yours pretty easily. But like investing in that Developer Experience, um, I think is, is, uh, what's interesting. Like there was this moment in the late 2010s where developer experience was like all the rage because you want to hire, grow, retain. Right. And that was the, that was your whole job as a leader. Let's hire grow retainers. And now that, like, not so much with the hiring, um, and maybe not so much with the retaining, um, I think that develop. Right. Um, but I think that with that change, the Developer Experience piece kind of like lost the attention that it was getting. And I think it's not, it doesn't exist. Developer Experience is not about free beer and free lunches.

Speaker A: Right? It absolutely is not. And I think that companies that dismiss that, I would argue it's even more important now. Right. Um, because not all engineers are equal nor fungible as much as, you know, some. The temptation might be to think of them in that way. The best people will never have problems getting jobs, no matter what circumstance. Like, I'm old. You're. I think you're about as old as me. Right? So we remember, we remember the.com bus, we remember the 2008 bus. We were, you know, now we're going the put the COVID in and uh, the best people, it's very rare for them not to be able to immediately turn around and get another job. Right. And so having the right kind of environment to incentivize people is even more critical because now they have more options. And you know, engineers there was, I don't know if you saw, I had a little thing someone was saying, oh, uh, you know, developers spend too much time worrying about things like kubernetes when they should be focused on the products for their company. And um, you Know, they shouldn't be worrying about, we could have solved this if everybody had just used Cloudfront or not Cloudfront, um, whatever. The previous to Kubernetes, Orchestration, maybe it was Cloudfront, I forget. In any case, it's like, sure, that's true, right? But the fact of the matter is, the thing that makes, uh, a good developer, a good developer is curiosity. Right. When Google came out with Kubernetes and then made a deal with the Linux foundation and started the cncf, don't you think that's the reason that there's nine bazillion Kubernetes offshoot projects? Because everybody was curious about it. Right. I called that a great brand of marketing. Is it the best orchestration tool ever? Probably not. There's arguments that could be made, but everybody was curious about it. And because of how they did it, everybody could participate in it. And that's what developers want. They want to be curious, they want to be supported in their curiosity and they want to know that they can be creative in their own nerdy little ways. That's what makes a developer. And to say, well, that was silly, they shouldn't have done that, Kubernetes shouldn't have won this, that and the other thing, I'm like, well, maybe. But one could argue nobody paid Linus Torvalds to go invent Linux. He did that because he was tired of having to try and pay UNIX licensing bees to whomever AT and T or whatever. Right? Like from, from irritation or curiosity comes innovation in the tech world. And because of how the tech world is digital, not physical, that curiosity can be represented in ways that are so much more profound.

Speaker B: Yeah. And I think it is, on the one hand, there are lots of people in the industry looking for jobs. And so that is true. I think that, um, leaders are maybe taking advantage of that sometimes. Um, and exactly what you said, if you are good, you make a few phone calls and you have another job. And I think we are definitely still there.

Speaker A: I think so. I mean, you know, I don't want to disparage. I know there are many people who have struggled, right? And it's not because they aren't good. I don't want to say that. Um, but maybe they don't, you know, they don't have the benefit of the reputation or the right network or their particular skill set is not as desired. And so, you know, they're having to go figure things out. But I also, you know, I have to wonder. I think the thing that still puts me in the AI Skeptic category and why I will continue to wear that badge, even though I am less skeptical than I used to be and I'm responsible for its success within my company is that I think it kind of the inaugural sort of themes that came out with AI were handled poorly by many people. Right? Oh, you don't need all of these people. Like the, the things that I saw in San Francisco is like, you can lay out, you know, you don't need customer service anymore because now we have these AI agents. Right. And I'm like, it feels ethically questionable to celebrate a world where we lay off thousands of people from their jobs,

Speaker B: absent another way for them to live.

Speaker A: Right. And, and then you and it. I think it's exacerbated by people saying, oh well, eventually nobody will have to work and we'll have, you know, universal income. And I'm like, in this country, you think that's actually going to be a thing? You guys can't even agree on single payer healthcare. And with that, there is clearly also evidence that outside of the tech industry, the appetite for AI is low. Right. There was that great thing about like Microsoft finally admits, nobody wants Copilot within, within Windows. Right? Nobody wants it because human interaction is fundamental to how we are. Which is why I love what I do. Because what I do is focus mostly on the interaction of humans. Right? With the technology being an implementation detail, it's not, not important. But the humans to me are more important. And I think that is the premise of developer productivity, is developers being productive.

Speaker B: Well, that seems like a great note to end on. Almost like you just made a little stump speech.

Speaker A: Oh, gosh,

Speaker B: you can call it there. Thank, um, you so much. I had so much more to talk to you about. We can do another hour, um, on all of this stuff.

Speaker A: Well, I keep telling myself I need to start blogging again and maybe, ah, I'll get around to it. But I think we're at another inflection point as an industry and it's always interesting to see how they turn out. And they never turn out the way I think they are. But I don't know.

Speaker B: Yeah, I mean I have my own, my own predictions, but, um, they haven't happened yet. We'll see. Give another year, we'll see what happens. Um, but I am, I'm, I'm not a skeptic. I am a, you know, I am a realistic optimist. How's that?

Speaker A: All right, I can, I'll buy that.

Speaker B: Buy that. Yeah, that's, that's what I'LL that's what I'll say. I'm a realist. Like, this has changed how we do work.

Speaker A: Oh, like 100%. It absolutely has.

Speaker B: But Cogen as the, like, secret magic bullet, like, I don't buy that at all. But some of the other stuff that you're talking about, I buy that a ton. Um, and I think that's where AI is going to have so much more value in companies that are mature enough to see that value. Um, and those tend to be companies that were already invested in developer productivity, coincidentally.

Speaker A: Funny how that works out.

Speaker B: Anyway, well, thank you so much. It has been great to. Great to chat and, um, I hope we can continue the conversation once you start blogging again. I'll comment.

Speaker A: All right, I'll let you know.

Speaker B: All right, and that's the show. Engineering Unblocked is brought to you by Swarmia, the engineering intelligence platform that's trusted by some of the best software companies in the world. See you next time.