Ryan Lopopolo: OpenAI's Framework for Shipping Code at 70 PRs/Week

We wanted Codex to be able to connect to our Electron app. Chrome DevTools protocol MCP was how we started out. Eventually, I kind of took a peek at the code and it was entirely different. Still, Codex was connecting to the Electron app over Chrome DevTools protocol, but it was actually a local TypeScript daemon that was being spun up that was providing a little CLI interface instead of the MCP because we found that there were only two or three tool calls that we actually needed, just more context efficient as faster.

And this had happened underneath me without me knowing my workflow was not interrupted. That same dependency edge and entity relationship existed, but I almost didn't have to know how it was implemented. Did you find that cool or worrying? I found it shocking.

The AI Native Dev is a podcast with developers and engineering leads at the cutting edge of AI and agentic coding. Join your host, Guy Fajani, and meet Simon Maple every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev. Hey everyone, hope you're enjoying the episode so far.

Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development. Whether that's talking about the latest tools, the most efficient workflows, or defining best practices. But for whatever reason, many of you have yet to subscribe to the channel. If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favour and hit that subscribe button.

It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you. Alright, back to the episode. Hello and welcome to another episode of the AI Native Dev. And we're here live at uh AI Native DevCon 2026 in London.

And joining me is Ryan Lapopolo, who is a member of technical staff at OpenAI. So Ryan, how are you? Doing great. Excited to be here.

Uh the conf is popping. It's really fun. Uh I go on stage later today, and I'm just excited to get after it. Absolutely looking forward.

And tell us a little bit about the session that you're giving later. Yeah, so uh I'm gonna be talking about uh all things harness engineering, which is sort of, I guess, kind of like some of the discourse that I helped define earlier in the year here, where uh talk about what it means to kind of set up agents to do highly complex, autonomous, software engineering-like work in your code bases with high confidence and uh almost headless nature if you can get to it. Yeah. A lot of fancy context management techniques in order to get these agents to learn from their own mistakes, produce high quality code, and go for a very long time without human intervention.

Absolutely. And we're gonna talk about harness engineering, and we'll talk about some of the methodologies that you've used internally at OpenAI and talk through maybe some of the bits that have been enjoyable, some of the bits that have been scary, and it's gonna be great fun. So, Ryan, why don't we kick off with harness engineering? For the layman, what is harness engineering?

Yeah, so we have these wonderful technology and these coding agents and these fantastic models that are very smart and able to produce a ton of code. Uh and harness engineering is this idea that to produce code that is acceptable, to produce code that we trust, that we can build confidence in business systems around, there are a bunch of little decisions that go into the process of producing PRs, producing systems that we have to make along the way. And in order to get the agent to know and understand those, we have to write it down.

We have to make sure that like the content of those choices, all those non-functional requirements that go into making good code get surfaced to the agent at the right time. So it's a combination of getting context into the repository in a place where these coding agents can see it, surfacing it to the agent at the right time, and making sure that all the code that we produce kind of like follows the golden thread of what it means for us to produce high-quality software. Reviewer agents, tests, lints, uh big refactoring loops, all sort of ways to get the agent to close the loop and make sure that uh it proves to me and the rest of the team that the code it has produced is acceptable and can't be merged.

So we've had prompt engineering, context engineering, and harness engineering now. What would you say are the kind of like the main differences or overlap between context engineering and harness engineering? I think really the two only levers you have when interacting with agents is which context you provide and which tools you provide. And harness engineering is kind of a combination of both with the goal of making sure the context that we give the agent is correct so that it can work its reasoning magic to produce good code.

And one thing that I think is unique about harness engineering is we're actually using the tool calls themselves to prompt inject the agent to manage its context window. One thing that's pretty neat here is the way that we structure tests and lints and well-formed tests in the code base is actually very different than how I would structure those messages for a human, right? I would have an exhaustive list of failures that I give the human and have them wade through the logs in order to figure out what's wrong.

Whereas with agents, you know, due to the nature of how they invoke tools, we want those messages to be somewhat compressed, but also semantically meaningful, right? Rather than giving sort of a mechanical ESLint failure, for example, I can give a prose error message that says, you screwed it up in this way in this file. You know, please open it and follow the runbook in file XYZ to go fix it. Because we've seen you make this mistake before, and we know that we can trust you to do the right thing.

Yeah. And that that that level of integration, that deeper integration, really unlocks that next step, right? Um now interesting, I'm looking I'm like looking at some of the uh some of your background here. You built Stripe's efficiency engineering team, led developer productivity at BREX for 350 engineers.

There's a bunch of kind of like infrastructure and DX background there. Would you say that helped you or hindered you from the point of view of harness engineering? Uh definitely helped. It kind of uh got me used to this idea of working through others.

Uh, you know, when I'm trying to improve productivity for an engineering organization of 350 engineers, I can't be hands-on in the weeds of every PR that's being produced. And I can't work directly on the underlying systems. I kind of have to steer from the background, make sure that the right things are happening by default, the right systems are in place in order to have high trust in the organization to do good work. And that's exactly what it means in order to be high functioning with these agents.

In order to move beyond sort of coding agents as pair programming assistants to go massively parallel where I've got the 15 TMUX windows open, I need to be able to take my hands off the wheel to a certain extent. So, this way of operating, this sort of systems thinking mentality kind of comes pretty naturally to me because that's kind of the expertise I've had to build up over the course of my career. Yeah, yeah, interesting. We'll actually touch a little bit about, I guess, the the part of that which is, well, how much is how much trust do you need then to have those 15 open?

But we'll cover that in just a second. I'd love to, you know, when you have those 15 open, there's obviously a huge amount of agent-written code and and various different uh roles in the in the in the workflow in the pipeline. Um now a really interesting constraint that you imposed on yourself and the team was no human written code. That's right.

And um I guess, first of all, how how fearful were you at the time when you said, like, no one's to write any code here in this project? How confident were you that that was the right course? It was definitely a bit of a radical idea at the time. Um kind of when I started out uh building in this way was uh even before we had proper reasoning models, GPT 5 came out in August of 2025, and I started working in this hands-off way in June.

Uh so this is you know 03 era Codex Mini had just came out, the very first versions of Codex CLI. And uh to be honest, back then the models were way less capable. Uh it was very painful in order to operate hands-off like this. Um and sort of by necessity, working with the model, working with the coding agent, I kind of had to stick myself in as a very chunky tool uh for the model to delegate to.

If it got stuck or if it needed to do something, the only thing it could do at the beginning was ask Ryan to do something for it. Uh eventually I get tired of being asked the same thing all the time. Initially, back then it was um installing dependencies with cargo. This is just not something that uh that Codex was able to do reliably back then.

So I would type the commands in my terminal and you know, give the agent some dependencies to work with. And sort of automating that was the first step. And it kind of got me in this mode of paying very close attention to where I'm spending my time, figuring out ways to build up a tool call that removes the need for me to spend that time. So then I can kind of figure out the other ways that I can help the agent get more and more chunky tools to give it more capability.

And to start with zero like that, you kind of viscerally experience the failure of the agent. Uh I think to kind of have a good sense of the tool calls as they are making, the mistakes the agent is making sort of in the process of writing the code is really, really important in order to build confidence in it. Yeah. Because to see what mistakes it's making, I'm also able to see the parts that it does really good at and can start to retool the way I go about my week to lean into the stuff Codex is really good at.

It's really good at following instructions. It's really good at writing tests and calling them. So collapsing a lot of the engineering process into these like very well-paved roads for the agent helps me build confidence in it. It's funny how you say it's it's really good at following instructions and really and really good at writing tests.

And I'm thinking, wow, the complete opposite of most developers. Is that is that maybe that's wrong to say? Um, but so that's really interesting. And I guess when you think about your team, like this is a decision that you made, but when you think about your team, how how comfortable was every how on board was everyone in joining this?

I got very lucky that as we grew the team, we grew it uh all with new hires to the company. So they kind of get dropped into this insane environment and being like, oh, I guess this is just the way we do things now. Let me just go whole hog at it. Uh it usually took about two weeks for folks to ramp up to this way of operating some growing pains very early on in growing the team.

Um, probably a good amount of dog fooding as well. You were able to go back to the team. I mean, the the product we were building too also helped a bunch. We were building sort of a data analyst agent, uh, a data analyst agent using codecs inside in order to write SQL and call data bricks and all these sort of things.

Um means a lot of the techniques that we were using to write the code were techniques we needed to deploy into this agent to have it be effective. A very cool virtuous cycle there. Uh took about two weeks for folks to onboard. Initially, very painful in terms of figuring out how to uh coach folks to not produce slop.

A big part of maintaining high velocity with these tools is just not permitting slop to enter the code base. Uh, but once we sort of had figured it out, by the time we got our fifth, sixth, seventh hire onto the team, uh within those two weeks, we were actually seeing like velocity and PR throughput go up 5, 10, 15%, uh, which is again not the common path when you onboard quickly to a team, like the the pre-singularity time, right? Uh and the reason for that is because as we accrued more and more context, more and more capability into the code base, sort of everyone invoking Codex as the sole entry point to the code base was getting the best parts of everybody by default.

Uh rather than having to spend one, two, three months absorbing the best practices of the team, they're already there. Yeah. Uh which means new hires to the team were able to very quickly supply their best judgment and context, uh, which means now everybody is being more effective very, very quickly. You're effectively onboarding the new hire through the agent already.

So brilliant. Um so there's a really nice phrase, I'm sure you've heard. I've said it a million times as well. Uh you you can go fast.

You can only go fast when you have brakes. That's right. Um now going in in this scenario, going fast is using, you know, codecs and AI agents to generate that code for us at a faster velocity than we would be able to manually write. The brakes, of course, are those tests, those reviews that that you know validate the code that show and highlight where we need to actually spend time and fix and adjust uh before we go live.

Another interesting thing uh that is the zero human review. Now, this is almost you're you're putting AI to be the brakes now, which is which is a little bit more worrying. Um in terms of um, I guess, you know, an alarming, an alarming uh proposal, how comfortable were the engineers when you said, right, we're now gonna do zero human review on this code. Uh we're gonna we want you know agents to be able to do that review.

Yeah, so sort of like the idea of post-merge code review, if at all, sort of thing is uh, you know kind of like not a new thing, right? When I started my career back in the early 2010s, like this was this was a normal part of what it meant to write software is kind of have a high trust environment with your team, bias to action, bias to throughput, and you know, have some representative sample of the code that you are looking at, but you know, otherwise have lots of synchronous communication on as a team to make sure that everyone has the same mental model of the system.

Yeah. And it's very cool to kind of like have this come back into vogue with agents because uh if the entry point to the code base is always the agent, it's really not required for every engineer on the team to have a full and complete mental model of the system, so long as that they are able to trust that everyone else on the team is enabling codecs to have that level of expertise. Um there is no review at all in the system, right? Like uh to kind of provide a little bit more nuance here, we do have some cases where we require like two-party traditional review pre-merge.

And most of the time, that is centered around things like highly complex plans, sort of phase milestones that would take place over the course of an actual week, these sorts of things. Uh, and the reason for that is these plans, these milestones, these exec plans, function as the prompt you're gonna give the agents. And if you are relying on a text document to drive implementation, being crisp around what words are in there and whether or not they fully under or misspecify the task is quite important.

Because if you misspecify the task up front, you are gonna get garbage. Uh so that's kind of where we focus the bulk of the human review. Aaron Powell, Jr. And so the review here, like it's it's different to how we would normally think of a code review, right?

Yeah. It's like first of all, a code review, you know, we will argue about details and namings and things like you know, but almost more style guide style comments. Um or rather, that's what sometimes happens. Unfortunately, yes.

Whereas now what we're looking at is actually a much more I guess I guess there's probably a couple of reasons. One, because it's actually more interesting to actually you know review at that higher level anyway, because they're more meaningful uh input from from our point of view, where we need to kind of like much more think about what's the greatest value I can provide here. And it is that more meaningful decision-making at the at the top. But also, I guess we're probably more abstracted away from the code.

So there is an amount that we just have to trust, where actually it'll it's almost a waste of our time to go too deep into the implementation because you know we we need that to be abstracted away and we need mechanisms of where yeah, testing just occurs to make sure that that succeeds. That's right. Uh this is sort of again that sort of like group tech lead sort of mentality, right? Like when I led developer productivity at BREX, right, like I don't have my hands in the weeds of every bit of Terraform every team is writing, but I do care at a high level about like guiding principles, right?

I want the infrastructure to be reproducible. I want it to provision quickly. I want us to have modules in place that solve common developer workflows. But like how you structure your underlying modules, whether you're using for each in an idiomatic way, I don't really care too much about that.

As long as the code is like locally coherent and actually solves the business problems that I'm hoping to do. And ideally, that's a decision that maybe even change over time, depending on how something should be structured. So it's it's not something we should always be so connected to uh at all times. Anyway, uh I think we mentioned kind of like trust a little bit.

Um I can't remember what the the magic ratio of trust is, where it doesn't matter how many times something works, it's it's how frequently something doesn't work, is the key which breaks the trust. Um but I guess at what stage were you and the team like truly thinking, oh actually, do you know what I completely trust this? Not not you know, I guess from my point of view, and I'll talk I'll talk about myself here when I say laziness, but there is a there there is a time when I'm using agentic development where I'm just kind of like assuming uh and just being lazy about not you know reviewing things in enough depth.

But in terms of like true trust, as in, yeah, I'm gonna I'm gonna hand this off to the agent and believe that this is gonna be high quality that I can actually uh you know push to production. At what moment would you say in the in the during the process, would you say you were truly trusting of what was being delivered? I think there were two sort of phases here. One was uh do I trust the agent to be able to produce code well?

And the magic moment uh uh for that was sort of working on what was essentially a very early version of what the Codex app is today. Uh talking with a teammate around man, I wish I had voice input here. And then being able to just give a prompt and half an hour later see that in the real world in the app was fantastic. You kind of got this feeling of confidence that it really is down to my capacity for scheduling prompts to go into this magic machine to see my ideas come to life.

Uh, so that was sort of like the first phase of getting into this mindset of our role as engineers should be to figure out ways to feed the machine to schedule work in order to kind of have our vision come to fruition. The next sort of phase was is this code trustworthy? Is it high quality? And this sort of came to us as a team sort of out of necessity.

When we onboarded our third engineer to the team, uh we significantly increased our PR throughput, but at the same time were unable to keep pace with reviews and making sure that slop didn't enter the code base. We kind of started to kind of pump the brakes a bit, like you mentioned, uh, and take one day a week Fridays to do garbage collection, where a lot of long synchronous stand-ups in this way of working, right, in order to socialize that mental model. So we would kind of compile a list over the course of the week of things that we didn't quite like, right?

We want to bias toward throughput, bias toward merging, but still we want to make sure we maintain our quality bar. So every Friday was trying to eliminate that slop, but also finding ways to systematically eliminate it going forward, right? We never wanted to give the same feedback twice in the same way that you want a new engineer you're onboarding to the team to actually internalize what it means to do a good job. So that kind of stacked into sort of these programmatic guardrails, and then a very long outer loop of sort of these automated CI jobs that are looking for slop, identifying slop, kind of compiling a list of golden principles for what it means for us to write good code, finding divergences from it in the code base, proposing PRs.

And at that point, we were able to see oh, the agents are able to write code the way we expect. We just kind of need to put the right loops and systems in place for it to kind of close that feedback cycle for itself. And presumably that was given to it by context or skills or something like that. Uh this is before skills were.

There was a time before skills. I know, I know. Only only what, eight months ago at this point. Um we kind of did a very cheap hack.

Um we kind of had just a GitHub issue where engineers and agents would leave comments around sort of underlying principles of good software development, right? How to properly make a React snapshot test, uh, what it means to write reliable network code, uh, how to think about structuring a lint. Uh, and you know, you end up with this big issue with 100, 150 comments on it. And that was sort of the seed for what it meant for the agent to go sort of spider through the code base, stack rank a set of violations, propose PRs.

And then one interesting thing there is we had like an on call whose job it was to supervise all the agent. Produced stuff, all the headless stuff. Because we we didn't quite trust it yet. We needed eyes on it in order to build that confidence.

Uh human with thumbs up, thumbs down, merge or not, leave review feedback, which means the next time that anti-slopification loop spins up, it looks at all the PRs it had produced, it ingests that human feedback, compares it with the session logs from the last time it ran, says, which mistakes did I make, which priorities did I miss? Did I produce misaligned code and why? Let me figure out how not to do that going forward. Produce some markdown, save it as an artifact in the GitHub run.

So the next time I run, I have this extra context that I have learned from the actual human feedback in the system. And is that the correct way? Is that the way you would recommend uh others set up their agents to almost like iterate through? So it's almost like build the code and then do these almost like audits or reviews in post and then raise PRs where you're almost uh you know you're outside of the guardrails?

Or or would you uh today look at adding that into the uh agent at code creation time? It's a mix of both. It's a mix of both. Yeah.

We we do think sort of these asynchronous loops are a big part of what it means to be effective. Uh early versions of codex security via Ardvark also worked in the same way, right? Where I'm getting a security review of my code after it has ended up in main. It is maybe finding a way in which it is vulnerable, producing a patch, we put up a PR, we review it, and it merges.

And I think this sort of flow is really, really nice because uh it allows mistakes to actually make it to main, which means they can be learned from. Yeah. Uh anytime we are able to come up with a correction, one, we want to fix it, but also repeating patterns sort of thing, which we identify with agents and loops and distillation and all of that. That's the stuff we want to pull forward earlier into the pipeline.

Yeah. And when you when you do push with mistakes or things that you want to correct, um are you less fearful of uh uh pushing those mistakes to production these days because you know how fast those can be reviewed and fixed by an agent? Uh it's definitely a spectrum, right? Um I will say that for this project, we didn't have continuous deployment, right?

We're building a native app. We are manually cutting a release branch and doing smoke testing. That smoke testing is initially all by humans, where we look at where we're spending the time and try and get agents to do some of it, right? Uh but uh it is important, I think, to have some level of human supervision over the release process in this world.

Yeah. But if I were to kind of like put my group tech lead hat on again, right? Yeah, uh, some mistakes are are able to be made. Like the consequences of them are low.

And uh much like I want engineers who are becoming more senior to learn from their own mistakes, I also kind of want to see mistakes that the agents make so that we, as their supervisors, can figure out how to get them to learn as well. Because it it feels like in this kind of scenario, debt and and fix is cheap because it's about, well, we'll run an audit and we'll so long as you have those guardrails and rules in place to say these these are these are some of our best practices, I can I can put you know something in which isn't ideal and no it's not gonna take six months for a human to come and review this code.

It can be done, you know, in a in a a more regular time frame. Yeah, like with this sort of like new world where the production of code is so cheap, uh I actually think that you can, at least working with agents, flip sort of like the DevOps shift left thing onto its head a bit, where the cheapest thing I can do is literally produce a change to my prompt. Whether that is like one of those golden principles that I'm yeeting into that GitHub issue or, you know, a prompt that I socialize with my teammates, right?

That's the cheapest thing I can do in order to fix the code that is produced. And, you know, if we find that like the mistakes are a bit more uh systematic, uh, I can kind of think about pulling that further left. I can add some docs, or I can operationalize a review agent that is going to judge every PR according to those docs. Or I can do more shifting left by making deterministic tests that suss out this behavior and on and on earlier in the process.

But the cheapest amount of effort I can spend to shake out that misbehavior is kind of where I want to start. Because these things are magic reasoners that most of the time they do pretty well with very little effort on my part. It's super interesting. And I'd almost love to jump ahead a couple of questions.

Uh, because I I kind of want to ask the question when you what the way the way you talk about that, whereby you know you make minor changes over here to a prompt, and then you know, implementations and things can just change. And it kind of makes me feel that when we talk about the thing that persists, it's less about the code and it's more about our prompts and our context and our and our you know functional requirements. And it brings me to um something called uh Symphony, which is a ghost library uh because it's spec.

It's all about specs and context versus versus uh implementation or code. Um and I guess when we think about when we when we think about that, the code just becomes a throwaway artifact. The implementation is just it's there today, it might change tomorrow. But the thing that persists, the thing that we care about, and we should put most time in, like you say, lifting up those reviews more to the to the functional requirements and things like that, is the specification and the prompts.

Um is this a is this a future? Is this a is this like when we when we think about the relationship between the specs and the code, how much is that changing today? Yeah, so it's kind of funny where you think traditionally in spec-driven development, right, you start with a spec and then you kind of refine it as the code is produced, sort of thing, to make it more concrete, to suss out those hidden decisions and ambiguity in the spec. And for things like Symfony, for all these lints that we're producing, uh for things like documentation on critical user journeys or product features or click flows and screens, I've actually found it is much easier to produce the code first, to have that code sort of like present a straw man for what the world could look like, engage with that as a team to refine it, and then distill the spec out of that.

Uh, because sort of produced artifacts, whether they are code, a spreadsheet, a Google Doc, right, these things are very information dense in terms of the decisions that were made to go into producing something that we have accepted as good. So for Symfony, the thing that we shipped is very much a spec. But the thing that we started with was kind of a vibed implementation of Symfony in TypeScript in our mono repo, where when we thought it was good and it was solving the problems that we wanted it to, we worked on producing a spec out of that to share with the world.

And it was this very, very cool process where we had this three-phase pipeline. We started with Symfony as it was, and we gave it to Codex to say, produce a spec markdown file that is able to reproduce this system. We end up with a spec. Give that to a new agent that says, without access to the original, that says, You have a spec, please implement this system.

Uh you know, it goes for a while with goal in order to produce this thing. When it says it's done, we give that system, the spec, and the original implementation to a third agent who's gonna act as the judge to say we are attempting to replicate the original version of Symfony. This was the spec that was given to an agent to produce it, and this is what they came up with. Where are their misalignments in the derived artifact versus the original system?

And propose changes to the spec in order to make the next attempt better. That's like a very token-intensive process, but you know, working at the token factory, token factory. I suspect you get a discount number. But what you end up with is this really refined spec that is reliably able to produce the system as it is written while still providing space and ambiguity for consumers of this ghost library to adapt it to their actual business context, their actual repository and tool chain and issue tracker and all these other sorts of things, uh, which I think is pretty cool because it means that we end up specifying in very tight, high detail the parts that actually matter, the business logic critical parts, while still leaving the flexibility for folks to evolve and change it as they see fit.

Yeah. And what I love about that is the process that you just described, and I think we need to learn from everything that we've done in software engineering for many, many years going into agentic development. But the process that you just described is a very pure prototype. It's us rapidly applicate uh rapidly developing our application.

So we make the learnings we need to be able to say, oh, this worked, that didn't. Actually, I'd prefer it to be like this, or let's change the UI, which ultimately breaks a number of flows. So we we make those learnings. But the difference is because development is cheap now, uh, we are able to say, I can just throw this away.

I'm not tempted to use that, but I'm gonna try and distill everything that I have learned and the functional requirements, etc., into that spec. And I love that iterate iterable kind of like that iterations through to make to really, you know, I've seen implementation to spec gaps, and this is this is really good to be able to really fill that in to make sure that the spec captures everything that we need. Yeah, it's not like this process is new, right?

Like if I think about other areas in my career where I've seen this, right? Like partnering with data science, they want to cobble together a model in a Jupyter notebook to prove out the idea first. And once they get something that they are confident in, then it sort of gets handed off to engineering in order to productionize it, to actually deploy it into the MapReduce pipeline to operate at scale sort of thing. Uh if we're thinking about you know doing rapid prototype uh of a design like in Figma and then handing it off in order to productionize also the same sort of context, right?

But now, because, like you said, production of code is cheap, we can actually do that in our production systems. Yeah. One neat way that we made that idea real in this app we are building is in dev, uh electron app, we had a uh window screen that the agent could launch, uh, which would basically give it the native rendering canvas that we're shipping, as well as our full design system and component library, which means it's actually able to prototype new screens directly in the surface that we're deploying, produce screenshots of them, give them to our designer, and now it's a super, super tight loop on does our vision match the reality?

Are we able to produce this experience in the surface that we're gonna ship? Which is just wild and super, super cool. Yeah. I'd love to talk about capabilities.

Um, and I'll take a slight deviation here. And I'd love to talk about codecs and um the industry perception of codecs. On socials, in community discussions, and all kinds of things, particularly over the last month. I'm hearing people rave about codecs and moving from you know other well-known uh agents, Claude and things like that, over to over to Codex.

And they are they're stating, oh, all of a sudden I'm I'm I'm more productive, more productive on codecs, or I'm actually it's actually more accurate in terms of doing these various things. Internally, I guess you're probably maybe seeing this and like you know, it has has there been a has there been a moment or has there been a uh a feature or something whereby you know codex leveled up for it because of a number of things, or is it just you know the gradual hard work of a of of a group of uh of teams?

Yeah, so uh the the the product velocity on codex is fantastically quick. We just recently celebrated a milestone of having five million uh weekly active users. Wow. So congratulations.

Uh Thibaut just did uh weekly resets, you know, go fast, go up, go burn those links down. Um I will say that there is a very tight and virtuous cycle between the development of Codex itself and our research team, which means there is this fantastic flywheel on every point revision of the uh GPT-5 series where uh the capability just improves in leaps and bounds. I would say uh big lifts uh from 5.2 to 5.

3 background tool calling, parallel tool calling uh means that Codex was able to do more work all the time in order to make more progress more quickly on more complex changes. Uh the unification of uh the normal GPT-5 model and the codex model in 5.4 means that you are not only getting a superpowered code producing agent, but one that is also incredibly generally intelligent, a good writer, which means I'm able to start using Codex for other parts of the software development loop other than producing code.

The slide deck that I'm gonna be presenting later today is 100% written by Codex uh using the app connectors in Google Sheets, which uh is not something I would have imagined to be possible last year. Uh and uh some super powered stuff that came out in uh 5.5 with uh computer use, the browser embedded in the app, these things make it so much easier in order to close the loop, where a ton of the jank that we had to put in place in the 5.1, 5.

2 era just doesn't need to exist because we are giving more chunky, powerful tools because there is this virtuous loop between the harness and the app that we are deploying and sort of our research environment and what it means to get the model to be effective at using these things. Just a constant march of capability improving. And one thing that I love about Codex is it'll just do the full job. It won't give me much nonsense about it.

I can treat it like I would another member of the team. Yeah. Right? I'm not shoulder surfing all seven of my teammates in order to bop them on the head if they do a bad thing.

I give them a task, I check in maybe occasionally at stand up, and I trust that I will get a produced PR or several that does the job. And I have the same level of trust in Codecs in its ability to be autonomous and complete. And you mentioned codecs and then you mentioned models and model versions. What would you say moves the needle most?

Is it updates to the agent or updates to the model? Uh it's a mix of both, but I have to say the new model releases I think are the most exciting uh bit to me. The the most sort of powerful levers we have in terms of increasing capability are uh continuing to train the model. And uh this is a thing that goes faster and faster over time.

We're super excited uh every time a new model comes out. Uh getting to test them internally is also fun to be able to kind of see the leaps and bounds of new progress we are able to make. Uh each revision of the five series from 5.2 onward, I think 5.

2, I think is Singularity Era here with these coding agents, uh, has seen increases in PR throughput uh time and time again. You know. Whereas at the beginning of sort of the 5.2 era, we were looking at maybe three and a half PRs per engineer per week on my team.

Uh now with 5.5, we're looking at 70, uh, which is just fantastic. Like that that's more than than linear scaling sort of thing. It's um it's truly amazing.

And is that people changing or is that capabilities doing a better job? A bit of a mix of both. A mix of both, I would say there as the model capability improves in leaps and downs like this, there is certainly less effort that needs to go into the enablement step. Yeah.

The harness engineering gets easier. The same techniques um we would use, but the need to build chunky tools in order to attach the agent to the outside world is just things that we need to do less and less. Yeah. Uh, to give you an example, in order to get Codex to drive our electron app, before computer use was a thing, we needed to boot up a graphical Ubuntu in a Docker container with a virtual display driver and X server, have engineers install X quartz on their Mac in order to proxy over to that headless host uh and like wire up FFmpeg to this gross contraption in order to record videos of what the app was doing, how the agent had changed it.

And this is like entirely obsoleted and like a single toggle in the app to turn on computer use, like so, so much easier. Uh, and I expect this to only get better and better over time. Yeah, it kind of reminds me of uh thinking back to you know when I when I've been in in the industry 25 years now, but when I when I think back to wow, people actually used to write like bytecode or machine code or binary, and it's like this is the same thing, right? It's like wow, people used to actually do that and connect.

I don't need to write TCL in order to test my user UIs anymore. That is definitely an improvement. So let's jump back into harness engineering then. Um and let's specifically look at uh a good use case, the Frontier project.

Um tell us a little bit or walk me through what the harness actually looks like for that. Umcluding like how I guess what gets updated, what's in it, but also ownership. Who owns that and who changes that? Yeah, so uh a little bit of context, uh OpenAI Frontier is sort of our enterprise platform for agents, and it covers everything from how to build agents with the API, agents SDK, the codex harness, uh, to how you would sort of observe and govern agents running around your enterprise.

Uh and it's very, very cool to kind of see all of the harness aligning on codex for all the things that we do, which means there's a single highly leveraged uh interface between the product we are building and research, which means uh sort of all the post-training improvements that we do on the model accrue leverage to every product that we build. And right now there's very, very clear interface in order to inject new capability into codecs via plugins. Uh these things are skills and scripts.

Same thing as always I'm talking about context and tools. Yeah. Uh very, very cool that we have ended up sort of in this like IOCTL sort of extensibility, uh, where really the biggest levers we as agent builders, as product builders, can give the model give codex is to give it chunkier and chunkier tools that connect to the actual business problems that we're trying to solve and to do context shaping and management for it. Another exciting part of Frontier that me and my team have been working on is sort of like the organizational context management pieces to be able to uh figure out what work is actually done in your enterprise or what the data ontology is of your data warehouse and how that is actually used in order to answer metrics questions and these things.

Have been doing some really, really interesting experimentation on what it means to potentially give each agent in your enterprise sort of a sidecar that is continuously managing context for it. Uh, this idea that all agents are coding agents, so we should give them things that are native to that, little Git repositories that they can grap and search over. Uh and all of that is to say the techniques that we're using to write the code are things that are going to translate really naturally to building agents in the enterprise.

And all of that comes from using that base codex harness in order to build these agents. Yeah, yeah. And I guess when we think about, you know, a minute ago we were just talking about bytecode and you know machine code and things like that. The code that gets created, or let's actually let's go back then.

The code that we used to write years and years ago when we did assembly and so forth, that it's really machine readable, less than less than human-readable. Then we kind of moved into the next next generation of programming languages, and actually we were writing more for us as humans. Sometimes we actually took uh the decision to make to write more maintainable, human-maintainable code versus most performant code because it mattered, it was less likely that we would add bugs and things like that when we make changes.

Um today, and I'll ask two questions. Today, and in maybe a couple of years' time, I hate asking questions for a couple of years' time when we can barely barely think about this week, but today in a couple of years' time, is it more important that code is legible to agents or humans? I would say if I had to pick only one thing uh that was legible to me, it would be the reference documentation of the system, the interfaces, and the rendering of all of that into mermaid entity relationship diagrams, system diagrams, and sequence diagrams.

And that's today. That's today. If I could only pick one, right? Uh still, of course, like I am peering down into the machine code, right?

I am looking at the diffs as the agent is producing them, but I am able to do that less and less as I gain trust in it. Uh and I am able to do that, uh I'm able to ignore the code as it is produced for more and more complex tasks. Um Not all of them, but for increasingly many of them. The things where I find that me and the team provide the most outsized value are on defining the interfaces, defining what the components of the system are, how each component should be structured, and how the code relates to each other.

Things like wiring up all the system dependencies together and using like Scala cake pattern style dependency wiring, I find to be very useful to pay attention to because it actually means that I am able to draw the lines on that system diagram in my own head and in the reference documentation. Now, you know, sort of the concrete implementation of those things, like to be honest, I might not even know what language they are in. Very, very funny story. Um to think about those system diagram, the lines.

Looking at the local development process, we wanted Codex to be able to connect to our Electron app. Chrome DevTools protocol MCP was how we started out. Eventually, I kind of took a peek at the code and it was entirely different. Uh still, Codex was connecting to the Electron app over uh Chrome DevTools protocol, but it was actually a local TypeScript daemon that was being spun up that was providing a little CLI interface instead of the MCP because we found that there were only two or three tool calls that we actually needed.

This was more context efficient, it was faster. And this had happened underneath me without me knowing my workflow was not interrupted. That same dependency edge and entity relationship existed, but I almost didn't have to know how it was implemented. Uh Did you find that cool or worrying?

I found it shocking. I'd find it cool. I'd shocking, but also I found it great, right? Yeah.

It basically means I live that high trust relationship in a very, very real way. Yeah. Someone on my team imagined a way the world could be better, went after it, and did it in such a way that disrupted literally no one on the team. Uh and because it's the agents that are writing code, they have no opinions around the tools or their structure, as long as we give them things that work and permit them to do their job.

Yeah. Super interesting. And I think it's it's definitely a it it feels to me like humans are going to be the slowest, the slowest here in this in adapting to this kind of level of uh level of trust. Yeah, it's a consistent challenge.

My attention, my ability to focus across all these threads, it's it's uh still something we are contending with. So when do you feel like you know, in your usage with these product product uh projects, when do you what what are the circumstances that you see today where you actually want to go in and look at the code? Sort of if we were to kind of put a sort of like two-axis graph together here of sort of uh low and high ambiguity and low and high complexity, in this sort of high high bucket here, I think that's largely two shapes of projects or tickets or epics, whatever you want to call them.

Things where I am literally starting from zero to do a net new thing, where maybe the shape of the interfaces, where it lives, what the experience is, I don't know yet. Yeah. Or sort of the most difficult refactors that I want to do, where I either need to break interfaces or redefine them, maybe go backwards and delete code in order to achieve this thing, but I don't know what the end shape that I want is. Uh, these are the places where I pay the most attention.

This is honestly where I want to be, right? Because this is what it means to kind of like peer six months into the future to unblock my team. Um it's also you know, you know, code code is cheap to produce here, right? Like I find myself operating in this space differently than I did a year ago, where I am willing to vibe out a 50,000 line of diff PR and throw it away only to learn the areas where the agent is likely to fail, uh, where it is going to struggle putting those interfaces into place that I like.

And usually I'll actually push up those PRs, trash them immediately, and then crank out 15 PRs out of that to kind of like do the prep work or do the staging in order to prepare myself for the actual cutover that I want to do. Yeah, yeah. Let's let's uh let's jump to a quote that you uh that you made a while back, which is you called it borderline negligent not to use a billion tokens a day. Um it's a strong claim.

Who who are you aiming to shake the ladder of the idea that the amount of intelligence we are able to extract from the models is in a way linear with token consumption. I think at the time was quite controversial, but we see it being increasingly true, right? Uh this is ultimately why test time compute exists, right? In order to get the models to be smarter and have more and more rich side effects out into the world, we want to move workflows to high token consumption use cases.

And I think in order to get to a billion tokens a day, you must be thinking in a grander world than pair programming with the model. You must find ways to be in parallel. You must find ways to put these asynchronous loops into place. Uh, you must find ways to build agents that are able to side effect onto organizations and teams rather than just yourself.

And in order to do that, we have kind of had to invent a bunch of patterns on what it means to do that, to have automations that are seeded from the repository context that everyone in the team can use rather than sort of single-player mode sort of things. And it's definitely not the case that everyone today can say, I'm going to consume a billion tokens a day, and poof, magic, uh, it happens. There is work that has to be done in order to make that safe, in order to make the output aligned, to make sure those tokens are burned in ways that are productive, that make it to main and improve the end experience that I am shipping to users, these sorts of things.

And I think at this point in the industry with this technology, we don't really have a durable, scalable, locked-in sense of what those patterns are. So, you know, to kind of poke aggressively like that and get to a billion tokens a day is really to kind of challenge people to figure out what it is that's gonna work. Yeah. Uh sort of feels like we have just invented the concept of CI CD, right?

And like people are kind of scrambling to figure out what that means. And only 15 years later did this become a thing that's like fairly standardized, that is table stakes. And that's I think what needs to happen right now uh with agents and producing software. We need to figure out those patterns.

So let's continue this flow then. Harness engineering. If we if we follow that to its logical conclusion. Um engineers and a well, harness engineering is is part of every project, every team.

Uh developers on that team are using a billion tokens uh every day. Woohoo! Um OpenAI, very happy with that. Um what does an engineering team look like in that scenario?

I have found it very valuable to have a diverse team in terms of perspective, experience, and expertise. Um I know myself, I probably biased more toward backend and infrastructure uh sort of work. You probably uh see in my sort of profile here, uh, which means that when it was just me on the team, I was producing quite terrible React code. We had we had some components that were 6,000 lines long, uh a ton of really bad re-renderings, uh these use effect hooks that were like four overlapping closures, like very hard to reason about.

And to bring folks onto the team that actually were able to say, Ryan, this is garbage, uh, really helped. Uh and to be able to kind of like have these full stack teams, I find really, really useful. Uh the other sort of thing is I think we no longer need to focus so much as software engineers on developing code production as a skill. Really, what I want to see uh engineers sort of nurture in their own careers is this sort of systems thinking mentality.

What it means to set up teams for success, how far can I peer into the future to solve problems, to increase the throughput at which we are producing code and shipping to customers? Um probably for me, my time has been split 30-30-30 between sort of the the hardest refactors and zero-to-one product ideation, talking to customers and prioritizing scheduling and staffing work. Uh whereas before I was probably spending closer to 50 or 70% of my time producing code. So I am able to step back a bit and focus on these other highly cross-functional, high-priority work streams in order to unblock teams of agents from going and doing the code production piece.

And the most important piece there, talking to users and understanding the requirements and actually mapping those back to the functional requirements of the app versus thinking too deeply about how an app should be built in the implementation and not thinking enough about okay, what does the actual user want rather than me building this uh the this this back end to be absolutely perfect in its design and so forth. Yeah, and one of the things that's part of that is deciding what not to build.

Yes. Very, very common uh trap I see people falling into these days with agents. And particularly because, yeah, coding's cheap. We can build everything, but we need to kind of like stop and ask ourselves, should we build this?

Because I guess it's still us as humans that are consuming uh in the majority, in the most part. Um well, I suppose that there will be these middleware things where uh I guess other agents and other apps can consume. But when we think about humans and users actually using this, we need to develop at the at the pace that they're comfortable consuming uh as well. So it's super interesting, super interesting balance.

Ryan, this has been uh absolutely fascinating and really enlightening to chat with you. So a massive thank you for spending the time to chat with us today, and really looking forward to your uh session later today, which is going to be uh live uh on the main stage and also recorded for for uh uh listeners uh to watch back in their own time as well. So, Brian, massive thank you, appreciate it. Amazing, thank you for having me.

This is such a fun conversation, and I will catch you later at the conference and on the live stream. Amazing. Thanks very much for tuning in and uh hope to catch you on the next episode. Bye for now.

Bye folks. The AI Native Dev is brought to you by Tesla, the package manager for skills and content. Your hosts are Guy Pajani and me, Simon Maple. Our producer is Tom Dowler.

The AI Native Dev is not just a podcast, it's a community. And we host monthly meetups at the Tesla offices in central London. Visit Tesla.io forward slash community.

To learn more, I hope to see you there.