Code Review Is a Taste Problem | David Poll ⁨@GitHub⁩

Speaker A: I mean one of the things that I've been chatting with folks at GitHub about, uh, a bit is like, you know, are we headed toward a world where uh, there's like 20 times as many PRs that don't get merged, uh, and maybe that's okay.

Speaker B: Hello everyone and welcome to HangardX Podcast. This is a podcast about developer experience and how engineering teams solve productivity challenges at scale. I'm your host, Ankit, co founder of Aviator. At Aviator, we are building verification platform for AI generated code that helps you create better guardrails. Today my guest is David joining from GitHub and I'm super excited to have David here. He is also a longtime ex Googler and uh, has been talking and sharing a lot of opinions about pull request and code reviews. I feel like David is also in one of the most interesting places, to put it one way, uh, of uh, today's software development by managing some of the pull requests, pull request, uh, experiences at GitHub. So with that David, I'm very excited for this conversation, but before we get started, please introduce yourself for the audience.

Speaker A: Yeah, I'm really looking forward to it too. Uh, my name is David Paul. I am a senior director of engineering at GitHub. Um, I'm responsible for the code and review organization which has uh, as part of it, pull requests and the pull request experience and then Copilot, uh, Code Review, which is the AI code review tool that's the part of, part of GitHub. I've spent my career though building stuff for developers. I love building stuff for developers. I, you know, came by way of uh, Google for a decade where I spent a lot of time on Firebase and the Android developer experience and the Assistant assistant developer experience and the cloud developer experience and all of that stuff. And prior to that there was a startup parse that was an amazing, an amazing place to be and uh, I still miss it has a warm place in my heart. But uh, I don't know, I've been doing it for a long time now and, and uh, I have opinions.

Speaker B: Amazing. All right. And for audience, this is actually like one of the articles that David wrote recently about like code review is not about catching bugs, which is also. I was telling David that most of the things that he writes, ah, also resonate with a lot of things that I talk about. So it might be hard to find topics to argue about. But regardless, I think this will be an interesting conversation. Maybe I would like to start from uh, Google. I feel like both of us have worked at Google we know kind of like Google had a pretty strong engineering culture. Uh, there was a lot of emphasis on code reviews. So maybe just starting from there. Right. Like I remember still it took my first review over three weeks.

Speaker A: Yeah, I mean it's uh, Google has an intense culture around this stuff and actually I left uh, last November and so I haven't gotten to see how it has evolved since. Although I hear it is changing. You know it has always been the case that, that code reviews were about more than just is the code correct? Uh I think it often becomes like the thing people assume people are doing in code reviews. But in practice it never has been. It is one small aspect and so much of the code review process is fundamentally built around applying judgment and saying hey is this the right thing for my product? Happens um, to be expressed as code. But most uh, of the time there's lots of people using code reviews for all sorts of other business processes as well. Uh, I think Google took it very seriously. I think it's evolving and out of necessity is evolving. Uh, but I think the like fundamental, we still want to apply judgment about what it is that we're building. Uh, that ah, that thing's as durable as anything is in this industry.

Speaker B: Right. So I think there's like a lot of things to unpack here. Right. Uh, one is about the collaboration part which is like how do you actually even share your ideas? And I think there's a second part which is about code correctness. So can you share a little bit about when you say it's not about catching bugs, what is your perspective on it? And uh, maybe we can reform this.

Speaker A: Uh, I have a follow up blog post actually to this, uh, that's maybe worth taking a look at too about software, uh, uh, not being a single player game. Uh, there are definitely aspects of um, code review that are about collaboration. There's definitely aspects of code review that are about uh, uh correctness. I actually think that there's a whole bunch of it that is uh, much more uh, taste and applying organizational processes and uh, there's a whole slew of things that happen at this point. One of the things that I think is, is changing pretty dramatically is uh, as it's becoming cheaper to write and rewrite code, uh a lot of the things that we used to do that we used to front load, uh, in order to make the same types of decisions before code existed, uh, are kind of collapsing into the same space. Uh so PRDs and uh, architecture document documents and you name it, there's, there's A hundred of them that you can probably come up with. Uh, we used to do all these things in advance because it was very, very expensive. Uh, if we didn't have uh, this understanding up front. Now we can actually do it in the context of real code, uh, where you can actually look at the, the real genuine artifact and massage it until it ends up in a place that uh, that um, everybody can be happy about and where as a team you can decide this is what we want. So I think there's some collapse of this stuff and it's likely to uh, load this up even more. But it's fundamentally that kind of decision making. But I look at even the history of pull requests and so on. You have the old mailing list with Linus taking uh, uh, emails that said uh, please include this uh, uh in your software. He wasn't just reviewing for correctness. Uh, so much of what was happening there was just applying taste. Does this belong in the software? I don't think that's changed. I think uh, that's really the question we're still trying to answer.

Speaker B: Very interesting. So I would like to double click on this a little bit. Right. So there is a concept which is, I feel like it kind of like got some movement but like it's also dissipating a little bit which is spec driven development. Like if you're Talking about these PRDs and sort of like ADRs, all of these things, it feels like they're sort of similar to how we think about specifications. Um, is there a space for specs in this AI driven software development? Do you see it being helpful, not helpful in certain specific scenarios or cultures?

Speaker A: I think the answer is yes, with a whole bunch of caveats where it's like, I think it's evolving and exactly what it's going to look like is going to change. Specifying software is something that uh, doesn't go away because it's a big part of just being able to express what it is that you're trying to get to where exactly it happens. I think we're all kind of uh, bouncing back and forth on right now like does it happen way up front and you just generate off of it and you might regenerate the entire application off of it, uh, as the spec evolves and so on. There's that kind uh, of more extreme uh, version of it. There's also the like eh, uh, let the agent uh, extract the spec back out uh, from what it built, uh, at the end of it. That's another more extreme uh, uh, path I suspect it will be a tool that we use in the long run to, to help provide guardrails, uh, for um, uh, what it is that we want to build and you won't be able to say oh, I wrote that, uh, or the LLM wrote that, uh, you'll say, well, through the process of getting to where we want to go, uh, it became, uh, it came into existence.

Speaker B: Interesting. So I mean uh, this is also interesting because in some ways my mental model is the challenge with specs is somewhat similar to what the challenge with waterfall model was. You can't identify everything before you go and implement because sometimes you kind of like have to make certain judgment calls when you actually go implement because specifications being complete is really hard. Right. And I think the second part of that is also something that you were bringing up. This comes to code review part. Right. I think there were some discussions which at some point in software industry or like even today. Right. Happens through documents. Like you kind of like write PRDs, you write like technical implementations, you get that reviewed. You worked in API team. I suspect there's like a lot of like, you know, documents that you have to review agree on whether this makes sense or not. Makes sense. Which is also part of like, you know, your article you were actually talking about like does this belong in the product?

Speaker A: Yep.

Speaker B: So what does that mean from uh, evolving agenda coding software? Because now cost of software building is so low, does it make sense to have those conversations during a code review process?

Speaker A: I think there's a, there's a journey that we're all on at the moment for exactly where this is going to land. But I think it was always the case that you could do it at various points, uh, in uh, the development lifecycle. Actually when I was at Google doing this for Firebase, I was responsible for um, Firebase's API Council for years and years, uh, where we kind of reviewed every single API and cli, command and rest API and every, you know, database schema and everything like that to kind of produce the voice of Firebase, uh, and Firebase's APIs. Um, we did all of that in documents ideally before code was written. Uh, I'm not sure I would do that that way today, to be totally honest. But that is how we did it at the time. By the time I was in the cloud SDK world, that organization, even before I got there, was doing review of APIs, which is this is not, you know, reviewing the, the code for whether it's correct, but the shape of the API and whether it fits the Standards that we have and so on that that team was doing it within cls, uh, within, within changelists at ah, Google and like the, the gate was actually whether you could merge your uh, merge your CL exactly where it lands. I don't know if I had to to kind of project. I think there's a world where uh, we are both uh, creating context and rules and so on to help uh, and uh, to help actually check in an automated way at the time that code gets built, uh, with rigor that would have been extraordinarily difficult for people to do. Where it's like well I can keep the entire set of rules in my head and make sure that I actually applied them to every single API that got created. Like that was a hard thing to do as it was uh, having an LLM help with that is great. But also then saying okay well but you also reviewed and did you know, came up with these different uh, trade offs or made these different decisions. Can we extract those things back into rules that can then be uh, used uh, and part of the institutional memory uh, of the code base going forward, uh, so that we can do this more effectively and get more effective as time goes on and like that, that loop, the more that that can take place in the same place where I'm actually iterating and talking to people and collaborating and so on the uh, tighter it will get and I think the more effective it will get. So my, my guess is that it's actually going to trend toward happening uh, in the process as opposed to before or after in the process of uh, building.

Speaker B: Very interesting. So one of the things which I try myself a few times is I will ask agent to produce two different implementations, right Kind of like based on two different choices and see how it feels. In some ways this is also the decision making which we are kind of actually now doing with the code versus specifications. Do you see sort of like a world where kind of people try out or review different versions of it. Maybe there's different realities, like alternate realities kind of things that a product exists in. Do you have any thoughts on that?

Speaker A: I mean one of the things that I've been chatting with folks at GitHub about uh, a bit is like, you know, are we headed toward a world where uh, there's like 20 times as many PRs that don't get merged? Uh, and maybe that's okay, like maybe that's part of how people build software going forward because there's going to be a lot of like well we tried it, uh, these ones we accepted those ones we rejected. That doesn't mean that we weren't being productive in the process. I wouldn't be that surprised to see a lot more of that uh, going forward. I think there's a lot of tools that we're starting to build. Not, not we specifically, but we as an industry are starting to build uh, uh, around doing this. There's all sorts of interesting questions about the granularity that we should be doing that at, uh, and so on that I think is shifting dramatically as well. Uh, but why not? Like I even, you know, I'm a, I'm an engineering manager. I don't like code day to day in my job anymore. I do lots of it for, for uh, uh, you know, personal projects. But uh, I have also, you know, quickly thrown together a draft TR just to like express, hey, this is a thing that I, that I'd uh, like I'm trying to help the team see or understand. Uh, you know, please don't work with it. Don't actually use that as the basis of what you're doing. But also like now you can see it in context, the real deal, uh, and we can have a conversation about it. I think there's quite a lot of that that's happening more. I think it happened before too, but I think it's becoming cheaper and easier to do it. So uh, we're seeing more going back

Speaker B: to kind of like our experience at Google and sort of like where the world is today. I also have a lot of friends who are sort of like principal engineers in different companies and it's very hard for them to move away from looking at the code just because they're like, oh, this is sort of like we own this. It's our responsibility. We want to make sure there is the right quality controls, uh, everything. Do you have any thoughts or like, have you seen practices where we can start thinking code reviews about like you know, these architectural decisions, discussions versus line by line code and like how do we actually build that kind of confidence into the review process?

Speaker A: I mean I think this is the million dollar question right now. Like everybody's trying to figure, figure this out. I know that we're just ah, a bunch of things about um, uh, what is it, uh, Uncle Bob, uh, on Twitter and stuff like that, uh, about reading the code. Uh, I think that uh, it is shifting. I think the question is how far can these tools get in being able to create that kind of confidence? What do we need to give them in order to create that kind of confidence more and more My own mental model of this is like, uh, it's not do I read or not read the code, it's do I get uh, does my attention get directed to where it's actually useful? Uh, and uh, can I be pointed at the parts that are meaningful for me to go and look at and apply my judgment? Uh, uh, and expertise, uh, and the things that can be, with high confidence removed from my attention set, uh, can fall away more and more over time. And that gives you now a sliding scale of like uh, you know, do I read it or not? No, it's uh, how much and what do I pay attention to? Uh, you know, different organizations are going to find themselves at different part at different places on that line. Uh, depends on the sensitivity of what you're building. It depends on uh, you know, the consequences of the change and uh, you know, what happens if something goes awry and all of those things. That's uh, probably a more nuanced answer than ah, uh, is tweet worthy. But it's, but I think it's probably the reality.

Speaker B: No, that's true. I mean I think there's also a big human part of it as well. Right. Like if you've been in industry like me and you, like who've been in the industry for last 20 years, it's hard to just stop reading the code. Like this is how we have been trained, this is how things have been done. But like, if you think about it pre Google era, like code review wasn't that big a, uh, thing. People did ship software without it. Right. And I think part of it one is trust. Can there uh, be enough trust built into the AI systems? Because today I feel like people are building enough trust on the coding side. I feel like actually many times even authors are not spending enough time reviewing their own code while authors as humans review the code which is generated by AI, which I think they should still be doing. But I think that kind of like also leaves the onus on the senior developers in the company because generally it's like, you know, if you have junior developers, they're probably not understanding the nuances, the challenges sort of like, you know, the issues with the architectures.

Speaker A: So I'm going to challenge something you said a little bit. Uh, let's do it. We maybe found a place where we're a little different. Uh, you mentioned that, uh, uh, you know, we had an industry that used to ship without code reviews and there's an extent to which that's true, but also the size and sophistication and Complexity of the software that we now build and ship has increased dramatically. And I actually think the practice of software engineer, of software engineering is how we got there. Uh, and I think what's happening now is uh, a shift in how much and where our attention can be. Uh, but uh, you know, a place where, where um, uh, we have to find the new practices that will allow us to build the even bigger things that we can, that are, that are actually accessible now, um, with the new tools that we have. I actually think that there's a, there's a fairly forward pointing arrow with all of this that is a, like how do we build up our practices, uh, to be able to handle more and more sophisticated things. The flip side of that is the uh, there are a lot of software projects that don't require that. I often use this analogy. Uh, like Excel spreadsheets ages ago gave people the ability to build uh, software really, uh, in a way that's, that was accessible to far more people. And nobody's sitting around doing code reviews on your Excel spreadsheet. Like there is a new wave of what's accessible to people to build. Do we need to do that with all of them? Probably not in the same way that that was true of everybody buddy's Excel spreadsheet. But if you want to build something that's dramatically more sophisticated, uh, and at a larger scale, I think we're going to need uh, at least some practices that create um, uh, systems that we can have confidence in working together and uh, building up and being durable and so on.

Speaker B: I mean, yes and no. I think this is probably worth debating on because in some ways I feel it's a lot easier to build software today than it was 20 years ago. And um, big part of this is also like, you know, the cloud systems that existed it, that didn't exist back then. Right. Like you have these infinitely scalable databases that you can just rely on or at least pass on the responsibility to those cloud providers to manage. Right. I think the complexity, I worked on some of those. So I think yes, there are things which have gotten easier, which obviously goes to the point which I would agree that it has given us the opportunity to build more complex systems.

Speaker A: I agree. I think that it has given us the ability to build more complex systems. But I think that Frontier is always going to be ahead of us, which is to say, like we are always, ah, the easier it gets to build complex systems, the more complex the systems, uh, and sophisticated systems that we try to build as organizations to have a larger impact, uh, we'll get, maybe there's an end to that, but I don't, I don't know that's going to happen. And actually I think a big part of what's, what will enable it is both getting the primitives right for the technologies that we build. On top of. You talked about the cloud native databases and things like that. Like a big chunk of all of that was making so much more accessible, uh, than was possible before. Um, it was possible, but then what was, uh, possible for small teams to build before? Uh, now you don't need to coordinate whole teams to uh, to go build those things and so you can spend your effort on things that are more valuable and bigger and so on built off top these very sophisticated systems. And so I don't think, I think the ability to build things is getting easier. The sophistication of what we're building is expanding. Uh, uh, because it's built on something that's already very sophisticated. Uh, and the question is, you know, what does it take for people to work together to build something bigger? Uh, and those are the questions that we're going to ultimately have to.

Speaker B: I think part of this is also like just building that trust, right. If you think about it like even the cloud movement took several years to build up, whereas like the AI things are happening on a monthly basis, not even nuts, right? So like how quickly, like for instance, you know, even if you think about companies like Netflix who built on top of like aws, I think they were like publicly about that, right. Initially their cost wasn't that high, but kind of like slowly built up. They kind of built a lot of infrastructure around it. But with AI, almost everyone, I think the adoption has just skyrocketed, which means it's like everyone relying on it, which means now our tooling is now further lagging behind. Right. To actually support that kind of ecosystem or the transition that happened in the software industry.

Speaker A: I mean, I think, I think that's true. I do think it is changing so rapidly that uh, it's a little hard to know exactly where it's going to land and we're all kind of inventing it as we go. And that's part of what's really fun about it right now for being honest. Uh, but it's also like, uh, I think it's why there's such a wide variety of opinions about, uh, you know, how much you can trust it and whether you should read every line of code and, and all of that stuff. Everybody's experience, uh, like the things have accelerated and so uh, everything's kind of spaghettified, stretched uh, uh, out into a place where like the difference between being you know, slightly ahead or slightly behind has grown dramatically. You know, there's going to be an interesting problem for those of us building developer tools to say how do I like actually serve the full audience or do I serve the full audience uh, of these folks? Uh, uh, you know I think those same questions existed even before, uh, uh, even before uh, but they were uh, more easy to categorize as like yeah, uh, this is someone doing low code, no code development versus someone who's you know, doing systems development versus someone who's doing applications development. Uh, and like you could kind of target uh, but now it's like on an individual basis like where is this engineer on their journey? So that's a much harder thing to do.

Speaker B: Actually there's also brings up to an interesting point which sort of you mentioned in the passing, the software being built today by individual developers is very different than the software that Enterprises built. Right. Like I feel like the entire processes, even the review part, the collaboration part is completely different. Um, how do you think about sort of like that differentiation, especially from a GitHub lens who actually has to support everyone? Um, um, what does, what does that look like? Do you see sort of like there is a sort of like more separation happening between Enterprises versus maybe even open source projects or side projects.

Speaker A: There are definitely some different needs for folks in those uh, in those settings because the kinds of contributions that you get are meaningfully different, uh, and so on. Right. It's, it's different to get a contribution uh, from someone that I've never met before on the Internet versus uh, you know, my coworker at my company who's you know, passed my hiring bar and worked in my uh, in my project for however many years and so on. So there are differences. Um, and some of them are like in, in both cases there's new stuff to go figure out. Um, I think uh, there is yet another category as you were sort of, sort of mentioning of like individual developers who may not be building and maintaining open source software but are building things that are useful uh, within a small scope and that's different from enterprise development, at least as it's often described, where it's you know, uh, building uh, the these uh, large products for large companies, uh, and so on. Um, I actually think there's maybe more overlap uh, there than uh, might otherwise be apparent. When I started my career, um, I was actually a PM at Microsoft at the very beginning of my career working on Silverlight. If you don't remember that thing from

Speaker B: I worked at Adobe, so I remember Silverlight.

Speaker A: Oh yeah, I was struck. It was kind of a formative experience for me. Uh, I was struck uh, when I went to a developer conference by the line out the door of developers who were looking at the like talks at a conference for how to build line of business applications, which seems like one of the drier topics that you might get. Uh, uh, in that setting. Uh, but everybody wanted to do it because it's. This is actually where a humongous portion of software development happens and it's often the like little onesie twosie things that are just, they're part of how we do our business but they're not the business. Um, uh, and I think that there's a lot of that kind of development that happens at Enterprises in addition to the like here's the core product that we're building and how we build it out. Um, so there's uh, uh, there's a lot of stuff uh to figure out and there's a whole spectrum of solutions that you can build for folks there. Um, I think a huge part of what I'm thinking about right now is how you do like progressive uh, enhancement of the experience so that uh, no matter how much or how little of the stuff you're ready to bite off, uh, you get value from it. Um, uh, it's not an easy thing to figure out but it is definitely a thing for us to figure out.

Speaker B: So uh, given that this is about code reviews, let's uh, at least think about if we are not doing line by line review, what are like the core values we get out of code reviews?

Speaker A: I mean I think there's a couple of things. One is uh, it's a place to collaborate and this is where teams come together to make decisions about what belongs in the product. Uh, uh, you know, is that code review at that point? Uh, it's review uh, of some, for some sort. And I, this is part of why I'm like, I don't, I don't even know if I like the moniker anymore. Um, uh, but I think that's, that's a big part of what's going to happen. I think there's also knowledge sharing, uh, you know, helping build some shared context both for people and for agents, uh uh, in the changes that are happening to a code base, uh, but also to the underlying business needs and so on. Um, I do think that there's verification and uh. Is it correct? Will it work. What's the risk? Uh, all of that stuff, uh, that's going to be a part of it. Um, but I think, you know, fundamentally, uh, uh, the question is like, how do people come to uh, a conclusion that they're confident this is the right thing for their product and there's lots and lots of questions to answer there and they're different for every project and different for every organization. Uh, uh, and it's kind of collapsing toward that rapidly.

Speaker B: Amazing. So I actually also did a talk which was like, you know, code reviews are not about code reviews. And I feel like this is also sort of like we are getting there. I do believe that this will take a while. I think there is a definitely along with processes, there's a big part of trust that has to be built into the systems. And I think, uh, uh, it's an exciting time to be.

Speaker A: Yeah, I'm having so much fun. It's so much fun and so much is happening, uh, very, very rapidly. Uh, and I'm with you.

Speaker B: Okay. On that note, David, this was a blast. Thank you so much for your time. This was really an amazing convers. And yes, this is again very close to our heart as well. Um, yeah, thank you for making this time. And for folks who are listening for the first time, this is our regular podcast. If you're also interested in developer experience work in this space, you can also join us at HangerDX Community where we run off the record sessions. It's DX Community and David, this was a blast. Thank you so much.

Speaker A: Thank you so much for having me. Take care.

Speaker B: All right, bye.