The B2B Podcast Index
Index
All categories
MarketingSalesSaaSFinanceHROpsLeadershipCustomer SuccessAI & DataProductStartups & FoundersRevOpsEngineering & DevTools
MethodologySubmit
Best of:MarketingSalesSaaSFinanceHROpsLeadershipCustomer SuccessAI & DataProductStartups & FoundersRevOpsEngineering & DevTools
An independent project byFame
SearchBest episodesGuestsInsightsMethodologySubmit a podcast
Index/AI & Data/Using AI at Work
Using AI at Work artwork

Why AI Pilots Fail: How to Escape AI Pilot Purgatory and Scale Enterprise AI with Ronnie Kwesi Coleman

Using AI at Work · 2026-09-14 · 45 min

0:00--:--

Key moments - from our scoring

Substance score

72 / 100

Five dimensions, 20 points each

Insight Density16 / 20
Originality14 / 20
Guest Caliber15 / 20
Specificity & Evidence13 / 20
Conversational Craft14 / 20

Enterprise AI pilots often stall in what Ronnie Coleman calls 'pilot purgatory' - they show promise but never scale into production. Coleman, a three-time founder focused on regulated enterprise AI, identifies five deadly sins that doom pilots: (1) assuming messy organizational data can be cleaned and that 20 years of expert judgment lives in databases when it actually lives in people's heads, (2) launching pilots without securing budget commitment for commercialization, (3) designing pilots too broadly with too many stakeholders rather than focusing on one problem with one expert, (4) treating integrations with existing tools (CRMs, project management, databases) as afterthoughts when they're essential to AI agent productivity, and (5) optimizing for 'people faster' productivity theater rather than measurable business outcomes like risk avoidance, revenue growth, or cost reduction. Coleman argues AI agents must work alongside experts - shadowing them, bringing forward edge cases, and freeing them from repetitive work - not replacing them. His company uses a model where agents handle rote, tedious tasks after learning from subject matter experts, leaving judgment calls and high-risk decisions with humans. The radiologist analogy illustrates this: AI doesn't replace radiologists; it lets them review 10 critical scans instead of 1,000, freeing them for patient care and strategic thinking.

Key takeaways

  • →Start AI pilots by identifying business problems first, not technology - leverage your subject matter experts who own those problems, then design AI to augment them, not replace them.
  • →Secure C-level budget commitment before launching pilots; without a clear path to commercialization and funding, successful pilots die in handoff between experiment and production.
  • →Design pilots narrowly (one business unit, one problem, one expert) but architect them to scale across the enterprise; avoid death-by-committee reviews that dilute focus and clarity.
  • →Build AI agents as junior employees that shadow experts and surface exceptions - not robots that do 100% of the work - so domain specialists spend time on judgment calls and high-risk decisions, not tedious reviews.
  • →Measure success by business outcomes (risk avoided, revenue growth, cost reduction, operational efficiency) not vanity metrics like 'people are faster'; tie ROI to CFO expectations and compare against similar pilots at peer companies.

Guests

Ronnie Kwesi Coleman

Topics in this episode

AI agentsSubject matter expertsPilot purgatoryenterprise ai scalingExpert judgment captureBusiness outcome ROIAI integrationsCompliance and risk in AIRadiologist analogy for AI augmentationFive deadly sins of AI pilots

Questions this episode answers

What are the five deadly sins that trap AI pilots in purgatory?

Ignoring the expert judgment that lives in people's heads, not securing budget authority before launch, designing pilots with too many stakeholders instead of one expert and problem, treating tool integrations as an afterthought, and optimizing for productivity theater ('people faster') instead of measurable business outcomes.

How should companies select which problems to pilot AI on?

Start by asking subject matter experts 'What business problems do we need to solve?' without mentioning AI; pick problems where experts have 20+ years of context, then design AI to augment - not replace - those experts by automating the repetitive, rote parts of their job.

How do you build AI agents that don't require constant oversight?

Design agents as junior employees that shadow experts, perform rote work, and bring exceptions or high-risk items back for human review; if you must review everything the agent does, fire the agent; structure the system so experts only review spot-checks or mission-critical decisions.

Why is integrating AI with existing tools critical to scaling pilots?

AI agents need to live in your organization's ecosystem (CRM, project management, databases) the way employees do; if agents require employees to learn new tools or work in isolation, pilots won't scale beyond the experiment - integrations are not an afterthought but essential to production success.

What business outcomes should leaders measure instead of 'productivity gains'?

Measure risk avoidance (e.g., reduced compliance failures or recall risk), revenue growth enabled by AI, operational cost savings (e.g., reduced contractor spend), and efficiency gains tied to CFO expectations; benchmark against peer companies using similar AI solutions to quantify ROI.

What our scoring noted

Our reviewer’s read on each dimension, with quotes from the episode.

Insight Density

16 / 20

Coleman delivers substantial, non-obvious frameworks throughout - particularly the 'five deadly sins' of pilot purgatory and the distinction between replacing jobs vs. augmenting experts. However, the episode includes moderate throat-clearing, repetition of concepts (expertise in heads, job shadowing analogy), and some padding around platform diversification that dilutes density.

The biggest problem is the judgment, the context that AI needs lives in people's heads. So John and Susie cannot be downloaded into AI.
If you need to give feedback on everything they do, you should fire the agent. Now, if you only need to spot check or for the things that are high risk, those are the things you need to review, then great.

Originality

14 / 20

Coleman presents relatively fresh frameworks (five deadly sins, job-shadowing-as-model for AI agents, radiologist analogy) but relies heavily on well-circulated ideas (problem-first thinking, ROI measurement, integration importance). The radiologist comparison is effective but not novel in AI discourse. Limited contrarian positioning beyond divergence from 'single brain' concept.

Your early question, it's how to think about the business outcome. Yeah. And this is the hardest thing for leaders at all levels to think about how do I put a number of ROI to success?
I disagree about the single brain stuff. I think that's a lot of it is marketing uh like AGI kind of futuristic talk.

Guest Caliber

15 / 20

Coleman brings legitimate operating experience - three-time founder, enterprise AI/regulated industry focus, direct client work at Fortune 100 companies. His credibility is grounded in practice rather than theory. However, he presents himself as YPO member and consultant rather than operating executive at scale, and references previous marketplace business rather than deep trenches in AI operations personally.

What qualifies me, I think BattleScars a little bit, uh, as you mentioned, it's my third tech company, but it's mainly been focused on enterprises and regulated enterprises.
I've been in this since you know mobile was hot back in 2013.

Specificity & Evidence

13 / 20

Coleman references specific Fortune companies (Nestle, Kenview, Danone, Meta), real-world examples (friend's billion-dollar company losing value, radiologist workflow), and concrete context-window numbers (million tokens = three Harry Potter books). However, he avoids quantifying most claims - no specific ROI numbers shared, no actual pilot success rates disclosed, and case studies mentioned without detail or data.

I give you case studies from my other customers, like, okay, Nestle did this or Kenview did this, or Danone did this. Here's how we saved X.
a friend of mine that built a billion-dollar company on Meta's ads because they were cheap early on, those ads 10x in price, his company went down to nothing over a year

Conversational Craft

14 / 20

Host Nagel asks solid opening questions and occasionally probes (e.g., on capturing expert judgment, trust levels), but rarely presses back on claims or explores contradictions. Nagel validates frequently ('So really good stuff') rather than challenge. Coleman largely controls the narrative without rigorous follow-up on specifics like the Gartner/Wharton studies mentioned or the actual five deadly sins prioritization beyond assertion.

So outside of that, um, how can a company capture this expert judgment without creating another like massive documentation project
So for the listener, especially if you're uh introducing AI into the organization, you're leading it, that's a really good distinction there

Conversation analysis

Computed from the transcript - who did the talking, and the words that came up most.

Most-used words

context27organization21jobs20agents19level19pilot13trust13start12data12ronnie11expert11review11three10judgment10first10problem10

Episode notes

Most AI pilots do not fail because the technology is weak. They fail because the organization was never prepared to turn the experiment into a real operating workflow. In this episode Chris sits down with Ronnie Kwesi Coleman, CEO of Puntt AI and a three-time founder focused on the future of work. They unpack the mistakes that keep enterprise AI stuck in AI pilot purgatory, why expert judgment matters more than simply giving AI more data, how AI agents should learn alongside experienced employees, and why productivity without measurable business outcomes and AI ROI is the wrong target. They also explore the shift from creation to review as a new AI bottleneck, the limitations of the "single company brain" idea, and the risks of building an enterprise AI strategy around one provider. Listen to learn how leaders can move AI pilots from experimentation into production while preserving the judgment, integrations, and strategic flexibility required for scaling AI across the organization.

Full transcript

45 min

Transcribed and scored by The B2B Podcast Index.

The danger of building all of your workflows and everything on one platform is that inevitably there are going to be price hikes and closed loops, and that means that you're gonna be fucked. How should a company go about picking pilots? Forget AI and what I know about it. Start with where are our problems?

Who knows those problems better than anyone? And that's where you want to start. Friend of mine that built a billion-dollar company on Meta's ads because they were cheap early on, those ads 10x in price. His company went down to nothing over a year.

There's trillions at stake. Trillions. So they're going to need to recoup that. So think long term.

Short term, it's easier, I agree. But have a strategy to diversify. If you're a leader and you're thinking about AI, think that the winners over the long term are not going to buy software, they're going to hire agents. Everybody, and welcome to another episode of Using AI at Work.

This is Chris Nagel. I'm the host of the show. And today our guest is Ronnie Coleman. And Ronnie is the CEO of Punt AI, three-time founder, YPO member, speaker, who's really focused on the future of work.

And we're going to be talking about one of his key concepts about this pilot purgatory, which I see regularly at all strata of business as we work with companies. But his, I guess the core lane for Ronnie is moving enterprise AI from experimentation into full production, where brand standards, expert judgment, uh compliance, and the cross-functional workflows are going to matter. So and why I think this matters today for our folks, our listeners, is because many companies have launched AI pilots, but they can't turn them into dependable, uh production-ready workflows.

And what we're going to dig into today is why that's happening. Why do those efforts fall? And how do you go from that hit or miss with your pilots to uh a higher consistency of AI is working in the organization. So Ryan, before we get started, anything I guess that you think that the audience should know about what qualifies you to talk about pilot purgatory?

Yeah, thanks, Chris, for the introduction and excited to be here. What qualifies me, I think BattleScars a little bit, uh, as you mentioned, it's my third tech company, but it's mainly been focused on enterprises and regulated enterprises. I've been in this since you know mobile was hot back in 2013. That was when I built my first company.

So I've seen kind of the the gamut of what you know the SaaS transformation was, but I think there are things that are happening in in AI specifically, and we've made some mistakes when we were trying to scale pilots at some of the largest companies in the world that we work with. Um that I think you know I can share some insight on that would help some people avoid it. And I don't think it's just relegated to like the large enterprises that we focus on, but everybody can learn from it.

What are some of the signs that even before that, how should a company go about picking a pilot or pilots? It's really critical that first of all you have your leader, like really gung-ho about AI. I mean, it goes without saying, right? But what happens is sometimes you have a leader that has a sense of, okay, well, AI is cool.

I have an AI strategy, they tell you three bullet points that are cool talking points, but they're not putting any money, they're not putting any like time, effort, energy into the organization and transforming it with AI. It's all talking points. So, first of all, you have to know the kind of organization you're in based off of your leadership. And now, if you are a director level, VP level in a company that has great leadership that's thinking about AI, they're gonna rely on the subject matter experts within the organization.

So the good ones are not just gonna come top down and say, hey, just to use this. They're gonna come to you as a leader and say, All right, where do you think we could have AI in our organization? And so this is where it needs to start. It doesn't start from the CIO or the like chief CTO or even the CI CEO.

Once they have context, it's always the subject matter experts that need to think about first question they have to ask themselves is well, in a perfect world, forget AI and what I know about it. Yeah. But what business problems do I need to solve? Just start there, list them, forget the technology, don't think about because I think a lot of people, when I ask them about what big what are their biggest pains, what are the business problems they need to solve, they start telling me about how AI can help them.

And one of my mentors, who's the the founder of Waze, he has a book and it's a mantra I live by. It's called fall in love with the problem, not the solution. And so if they're thinking about where to pilot AI, it's forget AI. Start with where are our problems?

And who knows those problems better than anyone? And that's where you want to start. Great. Yeah.

So not top-down by any stretch, but and maybe even not exclusively bottom-up, but some sort of a hybrid between the two. Yeah, it has to be a marriage. It has to be a marriage for sure. So uh what would be some success metrics that you think leaders should establish before they're ever even starting to uh you know work on the pilot?

I think before success metrics, uh if you want, like I'll I'll give you kind of the the context of some of the I call them the five deadly sins that create pilot purgatory. Yeah, let's do it. Metrics are one, but generally there are five specific things that create problems. And so the first thing I was saying is you want to get a leader that has expertise.

So for instance, one one thing that you can use in your organization to test one of these uh sins is if you have AI and you ask it a question, let's say about something very specific in your organization, and it gives you an answer, how do you validate it? So first they're just it's something esoteric. You ask it, it's connected to say your data or some context in your organization. Well, how do you validate whether the answer is right?

Yeah. Usually it's Susie in regulatory or John in RD. It's some person that's been there for 20 years that has this context. And so the mistake that people make, number one, is they think that if I go to an organization, it doesn't matter the size, if you're more than 100 people and you've been around for more than five years, you have messy data.

And so people go in and the thing they say is give me your data, give me your rules, and I'll create a pilot that can do the jobs of your employees. And that's a complete fallacy. That is 100% wrong. You're never gonna get clean data out of the box.

And even if you do, the biggest problem is the judgment, the context that AI needs lives in people's heads. So John and Susie cannot be downloaded into AI. Well, at least not yet. I'm sure uh Elon's probably working on that somewhere, but you but you can't download 20 years of judgment, right?

If I asked you, like, tell me about your business, tell me about all the things, you don't have a neat rule book that you've categorized every decision you've made and the way that you've no, it's you're an expert, you're a leader because of all this like intuition and judgment. And so nobody actually has that data neatly packaged up. So that's problem number one. Yeah.

Right? So expertise is in the heads of a few people, and you can't download those people. So sometimes people say, well, we we don't want AI to replace the jobs, we want it to work with the people. Yeah, that's true, but it's not because of some altruistic reason, it's just that it's absolutely necessary for AI to work, it has to work in harmony with your best people.

That that's it. So that that's number one. The second, and this goes to kind of what I was saying, or your question about like who needs to be involved, you need the like sea level or at least some budget authority to give money. Right.

So if a pilot works, right? You say, okay, well, where do I start? Somebody says, okay, I have $20,000 or $50,000 to try something. But then if it works, then what happens?

Somebody goes, Oh, I need the budget. They say we don't have budget for this, or it's gonna take a year. Well, now all the work you've done and everything you've put in just goes to waste. So you need to understand that, well, what's the commercialization of this beyond the pilot?

How much would it cost? And it doesn't, you don't have to have a million bucks, but you have to have some way to pay for the the AI solution, right? And then the third thing, and and this is a little tricky because it's important to start in one place. So it's important to start small in one business unit.

Yes. So with one specific problem with one set of experts. It's critical. Like it's really hard to prove any pilot, any POC, anything of value when there's a committee of 20 or 50 people judging it with their own context, with 17 different ways that they want.

It sounds like you've you've seen this problem. Yep. Yeah. So it's it's a very common problem where it's too many cooks in the kitchen, right?

So you want to start with one problem, one expert, somebody that has money, and says, okay, before we couldn't do this, and after we're doing this, right? But here's the rub. It still needs to have the context of the enterprise if it's successful. So there needs to be an expansion path.

It can't just live in that tiny little world of that small thing. The product needs to have enough value to go outside of your little experiment and have some kind of commercial application across your organization, right? At least across departments and users, right? It can't just be this tiny thing.

So that's the third thing. And then for any organization that has complex tooling, so you have your project management tools, you have, you know, like I don't know, it depends on the design tools, you have databases. Every organization, even the simplistic CRMs, you have 10, 20, 30, 40 tools. And the larger you get, the more tools you have.

And again, people think, okay, I'm gonna get a data lake and just put all the data in one place. Yeah. Well, the reality is that's not how things work. Like you're still going to use different tools.

And what people do is they say, okay, let's use the AI here, let's ignore the tools, and then just see what happens. So essentially, what they're doing is they're putting integrations for later. We'll do the integrations later, or worse, the AI product does not support integrations. So they're saying go live in a context.

So the idea of AI, right, is that your these AI agents are supposed to be like employees, right? Right. They're supposed to come into your organization and help. So imagine you hired, you know, an employee and they came to your company with their suite of tools, and they were like, Here, you have to learn my tools in order to work with me.

It's like, no, you'd fire that person, right? They have to come into your organization, learn your tools, learn your context, pull from the context, and be able to work in real time. So the integrations and connection into an ecosystem are not an afterthought, right? It shouldn't slow the pilot, but they're critical to be part of success if you want that to go beyond the purgatory.

Yeah. And the last thing, the fifth, the fifth sin is, and this is probably gonna get get some people, it's the people faster trap. So 90% of the people that I would talk to, if I asked them yes, how are you using AI and what value value does it have, oh, it makes my people faster. It makes me more productive.

If you've ever sat down to use AI, I think well, people mistake sometimes for productivity is just doing shit, just like busy work. And so because they see AI creating a bunch of stuff, they think, oh, I'm super productive. Yeah. Yeah.

They're sitting there on Facebook scrolling while the AI writes a bunch of stuff. They don't even read it, they pass it on to their friend, their friend doesn't read it, they summarize it with AI, and then and then the organization is everybody's high-fiving, saying they're using AI, it's super productive. Of course, you know, there are sets of people that are extremely productive, and I think it depends on the context. And so instead of just falling into that productivity trap, this is that your early question, it's how to think about the business outcome.

Yeah. And this is the hardest thing for leaders at all levels to think about how do I put a number of ROI to success? Yeah. And I'll tell you what, it's not, it's not simple.

This is what we spend, and I spend a lot of time with like C-level leaders at Fortune One and Hundred companies doing. It's really sitting down with them and giving them case studies from my other customers, like, okay, Nestle did this or Kenview did this, or Danone did this. Here's how we saved X. This is what the CFO would look like, and this is what they're expecting.

And that takes a lot of work on the company that's selling their product, not just on the internal organization. So if you're gonna buy a product, ask them what is the ROI? Like, how have you helped other organizations make money? And how can we prove that in this 90-day or 60-day period?

So at the end of that, I can go to my CFO and say, look, this is the business value to our company. Yeah, yeah, it's not just about cutting jobs. Like, that's where, again, people always think productivity, they think about no, it's like you need the people. Think about what is the business outcome.

Again, there's usually risk, we're avoiding risk. How much money could we have risked? Here's how the AI helped, this is what it did. There's growth.

How is it helping me grow? Can I is there measurable, even a small percentage of growth that I achieved, right? Then there's the operational side. Okay, did I cut contractor spend by X percent within this time and still achieve the results?

But it needs to be a business outcome. If you just tell me my your people are faster, then you're you're missing the point. And this is why a lot of people think AI is hype, because they're just throwing tokens, people are playing around with it, it's super fun, but they're not getting any business outcomes. Yeah, yeah, yeah.

I'm actually going to use uh a number of those in the conversations I have with executives. So let me ask you then, Ronnie. Yeah, I'm sure you're familiar with uh the sensational report that came out from that showing that MIT indicated 95% of uh enterprise pilots were failing. This is like December, I guess.

You you're familiar with that, right? Yeah, it's it's what we I mean, uh I think the pilot purgatory came from one of their their guys, and yeah. You're talking about these things, I'm like, that was the problem here, here, here. So you're right on with uh the findings that MIT had.

So yeah, yeah, they had more, they had more, and then we spent like Gartner had a another study and Wharton did a study after that. Yep. And so, and then we've done our own research because you know, there's this consultant level kind of insights of kind of high level, and then there's we're a startup working with these companies, and like our kind of everything depends on expanding within these companies. And so I think like out of all of those lists, I would say those five are the most critical, starting with one and five.

I think one is is like a thing that people miss that like the context lives in people's heads, and then five, it's there's a really hard problem, which is kind of equating it to business outcomes. I think those two, if those two are solved with the product, then there's 90% success. It's like a or 80%, it's like a Pareto principle. Yeah, you know, but of course, um, but yeah, so so those are the things that we we've learned through our own studies and kind of battle wounds.

Number one seems obvious now that you mention it, but I guess it's not something that that people normally think about. So let's maybe dig in on that a little bit because companies have this expectation that AI is going to perform like the employee who's been, you know, spending 20 years learning the business. But because the models don't automatically inherit the judgment or the exceptions or the the understanding of the social dynamics around these decisions and those sorts of things, there's the issue there.

So outside of that, um, how can a company capture this expert judgment without creating another like massive documentation project of cloning that individual to like at the at the granular level? Yeah, so I'll use a I'll use our own company as an example and and what we've done. So I said we work with some you know Fortune 100, like really large companies and and some mid-market enterprises as well. And when we go in, the first thing we say is that our software is not aimed to replace your experts, it is aimed to replace redundant, repetitive, automatable jobs.

And so if we look at a lot of some people's jobs can be replaced at 90%, because all they're doing is taking data, doing something that's basically an agentic loop that does the same three things all the time, and they're usually better served to something else. But I'll tell you why I know this because so I my last company we built a marketplace for talent, and it was contractors globally. And a lot of these Fortune 500 companies would come to us to hire people as contractors to do jobs that were repetitive, menial, meaningless, right?

Because they didn't want their internal teams to do these jobs because they the jobs sucked. And so they would hire outside contractors and we would provide it. So when AI was growing, I was like, oh, this is a perfect opportunity. I can build a platform and agents to do all those jobs and free people from all this meaningless work, right?

So a lot of the the ways that we do it is every product we build is like a mid-level employee or junior employee that has a very specific job, and their job is to shadow the expert. So it's here are the rules, here are the things it needs to do, here's the context. Every time it goes in, it does the work. It goes to the senior person, says, Hey, I've done this.

Am I on track? Am I off track? My own track, my off track. Now, the nuance is in how much do you bring to the expert where it's not tedious and laborious for them?

And that's that's where the design and kind of thinking through works because nobody wants to sit there correcting a junior level employee, right? If you hire somebody, you want them to be self-sustaining over time. That you still want feedback to give feedback on their work, but you don't want to give feedback on every line of work they do, otherwise you would fire them, right? And it's the same concept with agents.

If if you need to give feedback on everything they do, you should fire the agent. Now, if you only need to spot check or for the things that are high risk, those are the things you need to review, then great. And so that's you have to design the product, and that's where the nuance comes in. So the decisions that remain with experienced employees are the ones that are like mission critical, high risk, high compliance stuff.

Yeah. And it's, I mean, there are some compliance tasks that are not super risky, they're repetitive, right? So there's some like one of the agents that we have is like a claim specialist. And the claim specialist's job is to just look at the repository from the expert specialists, learn, yeah, kind of, okay, I need to look through a bunch of data, and then bring it to two or three examples.

And then the two or three, that's where the expert person goes in. They don't have to look through a hundred or look through weeks or months of work, or wait until their backlog fills up. They just have to look through two or three things. So the expert still does the review because of course you don't want anything going out into the world that can get you fined without the expert looking.

But again, do you need 20 people or 50 people doing that job? No. Do you need somebody, an ex one of your top people that could spend time with judgment and thinking strategically to sit there and look line by line? No.

So then let me ask how do you know when the AI has enough context to be trusted? Again, these are it, it's nuanced. It depends on the problem we're solving. So how much context?

So one of the things that we do, and this is like the least risky thing, is like review like videos and artwork and marketing materials for like brand guidelines. Yeah. Right. So we look at that's like one of the things, because there's a human that used to sit there, creative director, and art director creative that would sit there and just make sure things are on brand.

It's a tedious job, but it's not a risky job. Like nobody's gonna get fired because you know the the shade of blue is wrong, right? Your CIO is not gonna wake up and yeah, the font is slightly off in a TikTok ad. Nobody really gives a shit.

Yeah. And so that's like a hundred percent automatable through AI, and then once in a while, some you know, somebody can go like, yes, yes. So the trust level there is like you can trust that in two weeks. You can look, okay, put a hundred things into it.

The creative director looks at it, it's like, yes, no, yes, no, yes, no, this is right, this is right, this is right. Great. Like, that's it. Like, it's not overly complicated.

Now, we work with like highly regulated enterprises, like global enterprises, and a lot of the things we do are riskier. So the trust level is not so much about like how much the agent can be trusted, it depends on the context. So, what if we review artwork? So, for some of the large brands, there's artwork and an artwork recall.

So, if you put something on a label, the food that you have, and it's wrong, the recall can cost the company 10, 30 million, right? Yeah, if if there's a recall on some like medicine, like we work, we work with consumer health companies and and pharma companies, that could be in the hundreds of millions, up to a billion. So you never want to like the question is like, where are you gonna trust your agents? Well, you're gonna trust them as much as they can review large amounts of data.

So the analogy I give is in medicine, right? So AI is used by doctors, and one of the places it's used with the radiologists. And before, when AI first came out, people thought radiology is dead, AI is gonna do the job. Well, who the hell wants AI to be their radiologists?

Nobody in the world would want that. It's super risky, and that's not what AI is meant for. So, but what it did was instead of the AI, the radiologist field going down, they're hiring more radiologists now than ever. And and the reason is radiology went from I needed to spend my time looking at a thousand x-rays a week to I need to look at 10 and spend the rest of my time talking to patients, helping them, understanding their their kind of their needs, where a doctor is doing what they were trained to do.

They can get ahead of things, they can help, they can think through problems, right? They're not just sitting there doing tedious manual tests. And so the question is well, it's not so much that do I trust the AI or not. It's like, do I trust it enough to bring me the right information?

Not do I trust it to do my job, but do I trust it to sift through a large amount of data and tell me if it can pattern recognize the way that I would pattern recognize the rote parts. So you can trust it doing the rote job. You can't trust it generally to make the judgment calls at the highest level that a doctor or a regulatory professional, a lawyer would make. So for the listener, especially if you're uh introducing AI into the organization, you're leading it, that's a really good distinction there on what okay, we a job is made up of projects, made up of tasks.

Is it kind of how I heard it described? At the task level, that's not so much where you need to be worried about, where you need to be thinking about supporting your people is giving them less noise to deal with, let AI handle that, and they're getting more signal that would that AI probably wouldn't do nearly as well, wouldn't decide nearly as well as somebody who has touched that button a million times or has sat in that seat for 20 years. So really good stuff. Um now tell me about Punt.

You guys are focused on the internet. One other one other thing on that is the AI should be built so it learns from that expert, right? So there's the judgment, but that would that's a longer process, and it's the contextual layer of the AI that it learns. And so it's again, it's like how long will it take you to promote a junior person to a mid-level person?

Not in 90 days. It's never happened. I mean, unless they're insane. But like usually you're thinking at least a year, two years, and it's the same with agents.

That's why the analogy of how long will it trust kind of a junior level person to get to the next trust level, it takes a while. And they have to learn from the experts, and then they have to make the right decisions over and over again. And that makes perfect sense. So so tell me what you guys are up to at Punt.

It's primarily focused on supporting the, like you were mentioning earlier, like making sure that there's consistency across the branding and marketing in the companies. No, that's like one of the jobs our agents do. So uh at the core, you know, we we have a platform of agents. And like I said, from my career of building marketplaces for jobs, we have built agents that do the jobs that block growth.

So when we go into a company, we understand what are the tedious manual jobs that are slowing your company's growth. Right. And then how do we how do we give you the right agents to do those jobs so your company can grow, focus on the strategic goals, the the things, because every organization has a hundred to two hundred, three hundred, maybe a thousand little tedious jobs that everybody hates. They slow everybody down, but they're incredibly important because that's the process, right?

And so one of those jobs, like I said, is it was like reviewing marketing materials at scale. And so if you think about what AI has done, and this was one of the catalysts, I was like, well, AI is helping any brand grow content at scale. And so if you're growing your content at scale and your known brand in the world, what's gonna happen? Right?

That means somebody else is gonna have to review everything. And so your bottleneck is gonna shift from creating to review. I see. And this is already happening in engineering, sure.

Yeah, it happens in engineering too. It's like engineers can create all this code, but that puts more pressure on senior engineers to review all of that code, and then the people that just accept without reviewing end up creating a Frankenstein code base, and your product sucks, right? And so we we are like by creating these jobs, the review job was one of those jobs that our agents do. And so brand and marketing is one, but the more important review job is you know, can it do can it do the job of a regulatory specialist?

Right. So the regulatory specialists, like they're looking at labels and artwork. They're looking at it at product facts, warnings, required copy, market rules, regulations, like all of this, again, a lot of it is wrote and consistent. Then there's like promotional specialists that do like regulatory affairs.

So they're looking at what regulation has changed in the world, and how do I go find out if this is relevant to my company and then pull that into like a thousand or ten thousand pieces of artwork and check that against the web, approved claims. So there are all these jobs that are super tedious, and again, nobody loves it. There's also like a product uh information specialist job, and their job is just to take data from one place, clean it up, and then move it to another. And then there, these are these are hundreds of people in a company.

It's not, and a lot of them are super smart, like some of them with advanced degrees, but that's what the company needs, and so that's what they do. Yeah. Um, but they could they could spend it doing a lot more uh impactful stuff. You've referenced a couple times a concept that I've been following since I first discovered it at the end of February, was and that's the intelligence layer that Jack Dorsey talks about in his uh From Hierarchy to Intelligence.

It's also another uh YPO or in Beverly Hills, Eric Si U calls it single brain. And I'm hearing it called context layer and company brain and all these types of things. So you referenced earlier that there's there's individuals that are doing things that AI would be much better at, and sometimes their entire job is something that AI would be much better at, such as um these roles that we talked about here. Is that is that what you're referencing when you when you say that?

Is that concept of if once we have that, I don't know, that that robust uh uh context environment that the agents can lean on that the humans that were doing the rote activities are now ideally going to elevate into something that would require more decision making, more judgment, more discretion. Uh well, shout out to Eric. Eric is a buddy of mine, uh and like what he's doing. But yeah, I I disagree about the single brain stuff.

I think that's a lot of it is marketing uh like AGI kind of futuristic talk. It's like it's gonna have this one brain that's gonna have all of this context. Right now, there is no evidence that that is like gonna happen or on track to happen. A lot of it is like this is why if you look at agents, what has happened, it's not that they've gotten just like yes, context windows have grown.

So you know, nerd out a little bit. But context windows are basically the knowledge that the agent can pull from. And a million is about three Harry Potter books, right? And the average organization has maybe a whole library full of context.

So think about the difference between three Harry Potter books, which seems like a lot, to everything in a whole library, right? So, first of all, then you then you're saying, all right, yeah, the agents are gonna pull from all of that context. Okay, fine. But if we look at how agents have evolved, they haven't focused on just giving them more context windows.

The thing that Claude did well, Anthropic did well, is like they did this concept of skills. And skills are taking a lot less context, giving them a very specific three-point job, and saying, just do this job with this tool, and then you can go in and go somewhere else, and just like a human. Like a human doesn't have all the world's knowledge. Humans have a bit of context about something, and then they know how to go learn the thing if they need to.

Yep. Right. And so that's what all the evidence in the science is showing. It's like you're gonna have all these agents and all this work that no single one is going to have all this context perfect, but they're gonna be able to know how to use tools and go find the information in the right place, right?

And that's the better way of engineering, anyway, because there's no single point of failure. Yep. Yep. I like that.

I like that a lot. So do you have a book coming out or anything where you you really dive into how can people find out more about these um five deadly sins of pilot purgatory? And just in general, like the things that you're thinking about as you're running this business and working with clients. Where can people go to find out more about uh your positions on things?

Yeah, I mean, right now most of my conversations are either in private, but in public, it's it's mainly on LinkedIn. So people can reach out to me. It's it's just Ronnie Quasi Coleman, uh, Pontei. If you Google that, I'll come up.

Just add me on LinkedIn, send me a message. Usually I'm sharing some thoughts as I learn them at a high level. Try to make it as simple as possible and not not overly jargony or or technical. Um, and then yeah, folks can always email me if they want to talk.

It's uh Ronnie R O N N I E at puntp-un t t dot AI. Yeah, and for the listener, we'll have um all these links in the show notes, but I can tell you, yes, there's a lot of uh I don't know, AI experts, quote unquote, or AI consultants out there. What I'm always looking for is did you try it? And then what happened?

Not like, hey, this would be a great idea. Great, sure, but did you try it? Did you do it? What did the clients say?

How did they react? Where did it fail? And those types of questions, there's a limited number of people that you can get that information from. There's just not that many of us that are out there doing what Ronnie's doing.

So I would encourage you, like I'm looking forward to reading the back catalog of posts from what Ronnie's done, because the insights that he's getting, we don't we don't work at the enterprise level, but people are people, behaviors are are consistent across employees. And that's the kind of type of thing that if I were you as a listener, I would be looking for some predictive intelligence. What can I expect if we do this in our organization? What can I expect as we continue to scale this out in our organization?

And people like Ronnie have been there and are seeing what's happening and are sharing that information. So I appreciate that. And Ronnie, thank you so much for taking the time to be on the show today. And um, any, I guess, closing remarks or last thoughts?

Yeah. I think there just two quick thoughts. I think one is kind of a danger that I hear from smaller companies specifically. So I work at the enterprise level, but when I talk to my YPO friends, maybe hundred-person organizations, something smaller, and they want to grow, uh, a lot of them are saying, I'm building everything on anthropic, Claude, I'm building everything on OpenAI.

And going through, like being a student of platforms, the danger of building all of your workflows and everything on one platform is that inevitably there are going to be price hikes and closed loops, and that means that you're gonna be fucked. Like there's a friend of mine that built a billion-dollar company on Meta's ads and because they were cheap early on, those those ads 10x in price, his company went down to nothing over a year, and it was a billion-dollar company. It it's so understanding that if your entire infrastructure and everything you're doing is built and given to one corporation, regardless of what they say on the news, like nobody's doing this for altruism.

Like they're burning tons of. They're trying to make money, trillions. So they're going to need to recoup that. So think long term.

Short term, it's easier, I agree, but have a strategy to diversify, build, build the top layer, but also have the bottom layer. And then the last thing I would say is reframe your thinking. If you're a leader and you're thinking about AI, think that the winners won't buy software. Right?

The winners over the long term are not going to buy software, they're going to hire agents. And if you can think about the future as not I'm just buying another software tool, but how do I hire agents to do jobs that I would have hired a junior young person to do over time that works with my software, those are to be the people that would win long term. Nuggets. Thanks for that.

That's awesome. Okay, everybody, thank you so much for joining us on the episode of Using A Network. And I just want to let you know if you've got somebody up here, um, somebody else that's geeking out about AI, and you think this episode would help them kind of get a better understanding of how they need to be approaching things, handling things and their business to be that AI leader, please forward this along. And if you've enjoyed the show, we'd love a uh we'd love a review on whatever platform that you're listening on.

So thank you so much, everybody. We'll be back next week with another uh uh fantastic episode with a fascinating guest, just like Ronnie. Thanks, everybody. Thanks for tuning in to using AI at work.

Don't forget to subscribe for more conversations about how to use AI at work. And a special thank you to our sponsor, Chief AI Officer, for empowering businesses with AI education and training. Visit their website for free AI readiness assessment and AI strategy guide to help you get started using AI at work. That's www.

chiefaiofficer.com. Follow us on Twitter at the handle usingAI at work, and visit www.usingai at work.

com for free resources to help you harness AI in your role.

Related episodes across the Index

Other episodes covering the same guests and topics, from across The B2B Podcast Index.

  • More Agents Than Employees: How Zapier Disrupted Itself Before AI CouldTalking AI · on AI agents90 / 100
  • PMs Are Fintech: How Column is Rethinking Property Management BankingThe Profitable Property Management Podcast · on AI agents89 / 100
  • Inside an AI-powered personal dashboard | Matthew TiemannPossible · on AI agents83 / 100
  • 90% of AI prototypes never reach production (w/ Temporal's Samar Abbas) | AI BasicsThis Week in Startups · on AI agents81 / 100
  • From Billable Hours to Business Outcomes: How AI Is Rewriting the Rules of IT ConsultingEvolving the Enterprise · on AI agents78 / 100
  • Ep 86 | Konstantin KleemannMastering CS: Candid Leader Insights · on AI agents72 / 100

More from Using AI at Work

All episodes →
  • How to Implement AI in Your Business: From AI Use Cases to Real ROI | Dr. Markus Schmidberger92 / 100
  • 111: Using AI in the Workplace for Smarter Hiring and Pay Decisions with Cary Sparrow80 / 100
  • 109: Using AI Meeting Notes to Turn Conversations Into Business Outcomes with Artem Koren65 / 100
  • 108: Using AI Detection Tools to Fight AI Slop and Preserve Authenticity Online with Max Spero69 / 100
  • 107: Using Generative Engine Optimization (GEO) to Win the AI Search Race with Cole Casperson71 / 100
Explore the best B2B AI & Data podcasts →
All Using AI at Work episodes →