Why Developers Hit a Wall at 4 AI Agents

Simon: Hello and welcome to another episode of the AI Native Dev. My name is Simon Maple, your host. And joining me today is Nicholas Arcolano, who is the head of AI and research at Jellyfish. Nick and the Jellyfish team unveiled a huge amount of data which describes how agentic coding is done at organizations - some really interesting findings which show developers creating and merging twice as many pull requests as they were without AI coding tools.

We're going to be looking at the barrier, how developers hit a wall when using multiple coding agents in parallel, and what it is we need to do to get beyond that. And finally, 2026 - it's the year of the CFO. What do our engineering leaders need to bring to the conversation to make sure engineering teams are seen as effective and productive? Nicholas, welcome.

How are you? Nick: I'm great. I'm happy to be here. This is exciting.

Simon: Tell us a little bit about Jellyfish for those who haven't heard of them before. Nick: So Jellyfish, we're in the AI observability space. In particular we are focusing on understanding AI transformation at the organizational level. We're tracking what tools our customers are using and pulling in information not only about the agents they're using and how they're using them, but also how those things connect to outcomes - the code, the quality of that code, and ultimately what their business outcomes are.

Simon: And with all that, you've been reporting fairly regularly on a lot of this data. I should call you Doctor Nick, right? Because you have a PhD as well. Tell us a little bit about the background behind that.

Nick: Yeah, my Slack avatar is the Doctor Nick from The Simpsons - some folks around the office call me that. My background: I was in national research in the US for years before data science was a thing. My degree is actually in applied mathematics. I looked at doing inference and signal processing on massive networks for things like cybersecurity defense, and realized I could start calling myself a data scientist.

And now that title is kind of passé - it's AI engineer that's exciting. Simon: I'm looking through your CV - Harvard University, statistical signal processing and graph theory, then MIT Lincoln Laboratory, then Runkeeper, True Motion as director of data science, and now Jellyfish as head of AI and research. We have the right person to talk us through AI and data. Tell us about the data you collect and share every month.

Nick: The data has just been phenomenal - the growth of it, what we've been able to collect. Jellyfish has been around almost a decade, and for a long time we've been pulling in things like git signals, commits and pull requests, comments, and data from Linear and ADO and Jira in terms of what tasks people are doing. About two years ago, when Copilot really started becoming ascendant, we started pulling in data from those APIs to understand just that - basic usage. And that's evolved to where now it's just a wealth of APIs, hooks, open telemetry.

We're able to not only understand whether people are using these tools, but what the token spend looks like, what models they're using. And just recently we've started getting data at the agent turn level - things like planning and tool calls. That's the most exciting frontier of this data. The scale of it, being able to see across the entire industry what's actually happening out in the wild, helps ground the difference between what you're reading on X and what's actually happening at real companies.

Simon: That's what I love about this kind of data. It's not sentiment data, not a very small subset - we're talking about 40 million data points assessed. From all of that, what are the headlines that really stand out for you? Nick: One of the biggest ones is how fast it's changed.

This time last year, people were debating if AI was even useful - or at least that was true outside maybe the bubble of us who had already bought into the utility of it. We were talking to customers who weren't convinced about how fast they should even move, given that AI "just didn't work yet." And we know what happened at the end of last year - the models that came out in the fall really advanced engineering. So we find ourselves now with this massive data set, and we've really cemented the understanding that there are real raw coding gains, and we see them out in the wild.

People using Cursor and Copilot - those things are increasingly agentic. And without changing other material things about your workflow, 2x is about what you can do in terms of just raw code throughput. Simon: And that 2x - is that merged pull requests or raised pull requests? Nick: Merged pull requests.

Simon: Are there types of projects where you see those gains increase faster, or types where AI maybe doesn't help as much? Nick: Yeah, that's a fascinating question and we do see that out in the wild. Some of it matches what you'd expect - smaller codebases, newer codebases, language differences. The languages that are more AI friendly - Python, TypeScript, things that are heavy in configuration, markdown, YAML - those things move faster.

Things that move slower are older codebases, big messy distributed codebases. We have results that show you essentially get little to no gains due to AI as your codebase becomes very distributed, where there's a lot of human work involved in mapping together the context of how all this code relates. You know, we've all been in the situation where you've got the senior engineer who knows where the bodies are buried - you have to change code in ten places across these repositories.

AI just isn't operationalized yet for those big, messy, sprawling codebases. The agents don't know what to do. It's still heavily human in the loop. Simon: From the pull request point of view, there was some fascinating data about merge rates of AI-generated PRs versus human PRs.

Tell us about that. Nick: So the average for Q1 for humans was about 80% of pull requests that were opened ultimately got merged - the other 20% either stay open or get closed without being merged. What we see with AI-generated PRs is 60/40 instead of 80/20. You're talking about double the amount of PRs that are kind of dying on the vine.

We're digging into the reasons for that. Anecdotally, some of it is workflow differences. A customer request comes in, you just tell an agent to go fix it - it's just there - and then you decide later if you want that fix. Some people do things two, three, five different ways when they're not sure of the architecture they want, and then pick one.

That's intentional throwaway work. And then there's the version we hear a lot about where people see something on the backlog, vibe code a fix, and then people realize it's more complicated than they thought. A senior engineer steps in and says "no, we didn't fix this for a reason - there are deep technical issues buried under this seemingly simple thing." All of these compound to give you that doubling of the rate.

Simon: Is this a flaw in our workflow - where GitHub just isn't cut out for agentic development, and we need a layer that expects multiple versions of a fix? Nick: I completely agree with that. The whole world of software is littered with tools that aren't being used for the purposes they were designed for. I think history will tell whether my kids doing agentic engineering ask "why do we do all this weird git stuff?"

The question of what a pull request looks like - the core unit of software engineering - is fascinating, because right now it's so tied to value and shipping. And we see the whole SDLC collapsing in new ways that doesn't necessarily make sense anymore in a truly agentic world. Simon: Let's move on to AI adoption. You've said in the report that AI adoption has reached a median of 71% of developer time.

That seems like a lot. Tell us about how that data is generated. Nick: The 71% - that's weekly active users across the entire developer base we're tracking, about 250,000 developers in total. The median is 71%, and P90 is 90% - so the 90th percentile is people using it essentially all the time except on vacation.

For some of us it seems like "why are we even talking about this, all of us use this all the time." But we see such a broad swath of companies. All of our customers understand they need to be doing AI coding and agentic coding. The basic adoption barriers are still very real for a lot of companies - most have gotten over the hurdle of security enablement and getting licenses, but some are still just measuring whether people even use these tools, and if they do, whether they're just checking in performatively or actually integrating them into the workflow.

Simon: What's the typical background of these 250,000 developers? Nick: Our customers are paying us to track their AI transformation and how it maps to business outcomes. They tend to be companies that either build and sell software, or software is part of how they do business - maybe an apparel company that still has 200 engineers because they have operations and a digital presence. You need Jellyfish when you get to a couple dozen engineers.

We have customers up through tens of thousands of engineers. What we don't see are the five-person AI native startups that just got started - they haven't run into the problems where they need this level of observability. In terms of tools, they're using Copilot, Cursor, and Claude Code. Cursor and Claude Code are dominant - Claude Code has just been ascendant over the past six months.

And they're using those models both through those providers and through Bedrock. Simon: Tell me about depth of use - how much people are actually relying on agentic development versus just being a weekly active user. Nick: There are a couple of lenses. The very basic one is whether someone is even touching these tools on a regular basis.

Then we look at whether people have a reliable habit - using them repeatedly, day after day, week after week. One benchmark we use is whether you're using AI tools three or more times a week. That persistent, frequent active usage is what correlates with the companies that are able to drive up productivity and see those 2x gains. The level beyond that is autonomy and agentic workflows - what percentage of PRs are being generated in an autonomous fashion, where an agent takes a spec and does all the work, opens the pull request with minimal human interaction.

We've seen fascinating growth there, but really different stories. The elite companies - the 90th percentile - crossed 20% last month and it looks like it's closer to 30% this month. Basically exponential growth. They were at maybe 2% about a year ago.

The median company is just past 2% now. So it's really a story of leaders running away with it while everyone else is still figuring out how to get these workflows repeatable and scaled. Simon: That leads nicely into what you refer to as the agentic barrier. Tell us what it is and when developers reach it.

Nick: This is work we did in my team - a colleague of mine, Tomas Pardiñas, has gone deep in understanding these workflows. What we see is most people working interactively with 1 to 2 agents. The most experienced engineers, if they have to do any interactivity with these agents, get stuck at 4 agents max - very few people get to five. It kind of jives with what any of us who work in Claude Code every day know: human attention is limited.

You can only babysit so many agents and you inevitably end up forgetting about one. You miss the push notification, forget to go back to that terminal window. And at that level of four concurrent agents, you end up spending 80% of your time just focused on one. Interactivity just has its limits.

To break beyond that barrier, you have to get to a fully autonomous mode - you have to be able to hand off work in its entirety. If you have to nudge the agent along, total concurrency has a hard limit. Simon: That's almost like instead of the human being the orchestrator, you're enabling another agent to do the orchestration. Do you feel there's a better way to work with multiple agents, or is this a technology issue?

Nick: There's clearly technology issues. Lots of us agree that the tools we're using to manage code, manage agents, inspect what's going on - they're limited. All the players in this space would love to invent the new interface, the new IDE for agentic development. That would be a trillion dollar invention.

But I also have my own skepticism about what I'm trying to understand from dissecting this data. There are governors - a speed of light associated with how fast you can make business decisions, get feedback from the market, enable your go to market, change your product. I hear product leaders talking about starting to deal with user fatigue from products changing too fast. You don't even have time to get feedback on whether features are better because people don't have time to use them, and they're becoming increasingly frustrated with how fast things change.

Simon: It's a great point. We need to consistently remind ourselves that our users are still our users. Are there other ways autonomy is actually the wrong call, and how does that adjust team sizes? Nick: We see a lot of customers experimenting with smaller team sizes and the ratio of product folks to builders.

You can just do so much more with an engineer now. A good product leader - someone with good user and business sensibility - is just worth their weight in gold, or tokens, or whatever the most valuable commodity of the moment is. There's a real question I find myself asking when I sit down to do something: how much should I brainstorm with the agent and figure out exactly what I want, and then hand it off in its entirety? Or how much should I work through it interactively and have the agent help me understand what I want?

Engineering so much depends on sometimes high-stakes decisions about architecture, tradeoffs in the gray area. Those are really the meat of true engineering and true product development. And if you outsource too much of that to the agent, do you lose the opportunity to have those insights because you're not in there? Simon: I'd love to talk about one of the differences from 2025 to 2026.

You mentioned 2026 is the year the CFO gets involved. In 2025, people were encouraged to try everything. Now people are more cautious about spend. Do you see a difference in terms of the variety of tools being used?

Nick: We definitely have seen this, and we advise people at this point - so many of the tools have gotten so good that the risk is getting analysis paralysis trying to find the exact right agent or workflow. Should I do spec-driven development or not? Should I use Codex or Claude Code? For the vast majority of cases, picking a horse and riding it, and just getting good at that - building that muscle - is the right move.

The big difference is the move to scaling this year. There was budget and urgency to just explore and try things out. Now as companies try to scale, agents have gotten so much more complex and capable, but also so much more token-hungry. The models are beefier, the agents are doing more turns, more code exploration.

Token consumption has just skyrocketed. The tension engineering leaders are feeling is that they still very much believe adoption needs to be driven and supported - you've got to be burning enough fuel to achieve escape velocity - but they also have to show their receipts. Engineering leaders have to explain where all the tokens are going. Simon: If the CFO says "we've spent all this money, show us the efficiency and productivity gains" - what should an engineering manager produce back?

Nick: It's a real mess. There are good metrics - you can look at raw token spend, which is sort of a basic signal like your heart rate. Having a heart rate is a good signal you're alive. It stirs more sophisticated questions: what's the token spend per PR, per feature shipped, token spend per outcome.

That gives you a sense of what you're actually spending. The reason I say it's a mess is that what businesses really care about is things like revenue - things that actually change the trajectory of the business. And companies generally haven't adapted to think about what it means to essentially have infinite capacity. We've entered a world where you can spend infinite money to get things built faster - you can hire infinite robot contractors, so to speak.

But companies aren't designed to reason in an infinite capacity world. They're designed to think: these are the headcount we have, these are the things we can build, what maximizes our business? That feedback loop of what would actually matter - that's still being built. Simon: I always tell my team to focus on outcomes, not output.

The 2x in pull request merge rate - for me that's output. The outcome is: is it good code, is it quality code, is it being patched, is the second PR fixing the first? Nick: Those are important outcomes for the engineering machine, and lots of folks are concerned about quality. Right now there aren't huge smoking guns.

I think the main reason we haven't seen quality go off the rails is that engineering teams are responsible stewards - they're not merging bad code. You see the difference in merge rates. You do see some upticks in reverts, but it can be hard to analyze because we've also seen a growth in "fixing forward." As teams get faster, they may not even file bugs or revert code - they just apply a fix forward.

So it can look like just more building when you're actually fixing a problem. But those are still engineering outputs - not necessarily business outputs. Did you build product, or did you just tinker with things nobody else cares about but engineering? And even if you build new product - if I could wave a magic wand and build five times the products tomorrow for Jellyfish, could we sell five times the products?

Probably not. We'd need a whole new type of AI enablement in our go-to-market to accelerate that team to where they could capture all that value. That's what I mean by: we're in a world where we invented these jet engines and we're still putting them into the cars we used to have. We haven't designed the rest of the vehicle to accommodate the one thing we've massively accelerated, which is code generation.

Simon: As we talk about becoming more effective and scaling across an organization - tell us about the AI enabler role, the platform or developer experience team that helps roll out AI adoption. Nick: It's hugely important. We see this uncanny valley where small teams can move fast because they're small and have younger codebases. But we actually see very large companies that have the types of teams you're describing - they're able to make big investments.

The most successful ones are moving with purpose and deliberation to enable folks. They understand you need to invest in the tools, in context engineering, in training. This just doesn't happen for free. Where people struggle is in the middle - they may not even have a head of developer experience, let alone a whole team to develop tooling.

And when you leave every team to figure it out on their own, it can be really challenging. What success looks like in my experience: putting dedicated resources to it, making it someone's or multiple people's full-time jobs to figure out this transformation. Making clear investments with money, with training, with time. Going slow to go fast.

And picking some things and not getting analysis paralysis stressing about the fact that tomorrow new things will come out that make what you just did suboptimal - just building those muscles. It comes down to continuous learning and continuous evolution. If you can't build that muscle in your org to just evolve continuously, we're all on this treadmill for the duration. It's just going to keep accelerating.

Simon: What would you say is the biggest misconception engineering leads have today around AI? Nick: One big misconception is assuming there's some silver bullet - if I just give these tools to people, they'll magically work. Part of that misconception is that we've kind of done the easy part already. We gave a lot of people fancier IDEs and there were real gains associated with that - fancier autocomplete, they bootstrapped a lot of coding.

But by and large, the gains from getting a Copilot or Cursor in the hands of developers while leaving everything else in the process the same - that's the easy part. The misconception is not understanding that to get to the next level, to get to what you and I believe AI native development is really going to look like - those are big cultural changes, big skill changes, big architecture changes. Much bigger rocks to move and requiring much bigger investments. The payoff is going to be massive, but some people aren't prepared for how different that is from what we've done so far.

Simon: It's like questioning everything. Don't assume that just because you used it the last ten years just fine, it's the right way of doing it with AI. Though there are probably a lot of things we should very intentionally keep - AI sometimes makes it easy to drop best practices and good hygiene processes of software development. Code reviews, good practices in your workflow - a lot of that we need to maintain.

Nick: Absolutely. We've had this argument internally at Jellyfish that a lot of folks have had: we've done the two-person code review thing - if I open a pull request, a different person needs to approve it, I can't approve my own pull request for production. If an agent wrote the code, is that a different person? Can I review code that a robot wrote that I never read?

It logically kind of makes sense - but trying to decide whether the value of two different eyes on the code is worth the bottleneck that reviews are becoming. And I love the way you phrased "what things do we want to hold on to?" One of my pet peeves about the current conversation is this obsession with agent coherence - how long an agent can run on its own. It's such a weird metric.

When you translate that to the real world: is the best engineer on your team the one who goes off the longest into the cave without any questions? We tend to complain about those folks who don't know when to come up for air. Feedback and communication is a core part of a really well-functioning engineering team. Having agents go off on their own for a very long time is an anti-pattern.

It's a dysfunction. Simon: Nick, where can people go to keep up with all the great work you're doing with your reports? Nick: If you go to Jellyfish Engineering Trends, we're publishing updates to our benchmarks monthly - adoption, impact, quality, growth in engineering. We're adding new stuff every month.

We just added some token insights and we're going to add more in the next cut. You just can't keep up with all the exciting things we can see in this data. Not enough hours in the day or tokens in the world. Simon: Amazing.

Thank you very much, Nick. Nick: Pleasure. Thank you so much. This was fun.

Simon: Absolutely. And thanks everyone for listening. Be sure to tune in to the next episode. Bye for now.