Stop Measuring Your AI ROI: A Better Framework for the Age of Agents

Adam Brotman In a world where you're trying to figure out is the ROI there, like keep in mind the first thing you want to ask yourself before we get into the details of like what do we mean by ROI? Is it there? Is just what are we talking about here? Are we talking about like API costs?

Are we talking about just like how many people in your company have a seat license for like a co-pilot, a chat GPT, or a Claude subscription on an enterprise subscription? Are we talking about one of those two things versus an API? And is that what we mean by The I, the investment in ROI? Or do we mean like I'm bringing in consultants and they're gonna help me with a pilot?

That's part of the I. Is it my time and energy that I'm focused on a project that could have been focused anywhere else? That's part of the I. So I do think it's important for us to get everyone to sort of level set on what we mean by it.

And then let's talk for a second about, you know, is the ROI there? Andy This is AI First with Adam and Andy, the show that takes you straight to the front lines of AI innovation and business. I'm Andy Sack, and alongside my co host, Adam Brottman, each episode we bring you candid conversations with business leaders transforming their businesses with AI. No fluff, just real talk, actionable use cases, and insights for you.

Welcome everyone. Today we have a conversation about AI, agents, and ROI. And we hope to give you some pointers on how to think about ROI. we did an episode like this previously, a little over a year ago.

We thought with the introduction of agents, we have a couple of customer stories that we might share with you. And ⁓ We thought it was worth updating because we've gotten a lot of feedback and the market has evolved. So I think the thing that has triggered this is there's been conversation at Forum3 really driven by the press about this trend of token maxing and CFOs getting upset or alarmed about the bills that they're getting from. in particular the large language models, anthropic and open AI, but also, it's relevant to Microsoft and Copilot.

let's start with the media coverage of token maxing and CFOs freaking out about the bills and what your initial reaction is, Adam. Adam Brotman Well, I think it's really important for you and I to slow down in this conversation and just kind of level set for everybody about what we mean by like ROI and cost and agents and all of that. Because in this AI space it all gets kind of well, no, that's okay. It it just gets jumbled together.

And I think that's a key point we want to make when we prep for this session. Andy We don't want to jump into token maxing and and CFOs freaking out. Okay. Adam Brotman We we wanna make the point that like Andy I'm personally kind of excited about token maxing.

Adam Brotman what you did is what everybody's experiencing, which is that like they're just watching YouTube or TV or reading their email or their social media, and they're just hearing stories about like, you know, Uber blows through its entire annual AI token budget in Q1, CFOs beware, or you're an actual CFO, and all of a sudden it's like, geez, our anthropic bill used to be Or an open AI bill as a hundred person company, used to be like, I don't know, a couple hundred bucks a month, thous a couple thousand bucks a month, and now all of a sudden it's like eight thousand dollars a month.

Like what wait, is that gonna keep growing? Is that gonna go from two thousand to eight thousand to twenty thousand? This is out of control, and is there ROI there? So that's what's happening.

But I think it's important everyone to kind of take a deep breath and slow down and say, first of all, Andy mentioned agents. it doesn't matter whether it's agent or chatbot or whatever the definitions are. Just keep in mind AI has progressed so much in the last year since we did an ROI episode that we're talking now about ⁓ much more powerful systems, whether they're agentic or not, from the AI company. Andy And their base plans have not materially changed price.

Adam Brotman Correct. And let's talk about the difference between a base plan and an API plan, if you're a developer at coding and you're using AI to code or you're building a platform that pulls AI into it, a lot of times you're gonna use the API model, which there's no all you can eat subscription. It's like the more you use, the more you pay. So those bills can get really big, like really fast.

And that's where you hear the stories about like we blew our annual budget in a quarter. It's from mostly developers, coders that are using it, or people that are like now newfound vibe coders, like non-technical people that are using the APIs, and they're just constantly running stuff on Claude Code or Claude Cowork, but usually Claude Code or Cursor or whatever. And they're just running up these big bills because They're just like the more you use, the more you pay. For most knowledge workers in corporate America right now, they're not using the APIs per se.

They're using some business account or enterprise account. But let's just say ChatGPT business or claude team. And every seat that's using AI is either using kind of a $25 a month plan or up to like a hundred dollar a month plan. And I'm generalizing that they might be higher or lower numbers.

And so In a world where you're trying to figure out is the ROI there, like keep in mind the first thing you want to ask yourself before we get into the details of like what do we mean by ROI? Is it there? Is just what are we talking about here? Are we talking about like API costs?

Are we talking about just like how many people in your company have a seat license for like a co-pilot, a chat GPT, or a Claude subscription on an enterprise subscription? Are we talking about one of those two things versus an API? And is that what we mean by The I, the investment in ROI? Or do we mean like I'm bringing in consultants and they're gonna help me with a pilot?

That's part of the I. Is it my time and energy that I'm focused on a project that could have been focused anywhere else? That's part of the I. So I do think it's important for us to get everyone to sort of level set on what we mean by it.

And then let's talk for a second about, you know, is the ROI there? Andy Yeah, so I mean the point I think you're making is ⁓ you did a nice job of framing up the cost side, which is base plan API time consultants. I think the one thing that summary of costs did not come is that we think that the framing of your AI bills should not be seen in the same lens as SaaS software and technology. It's actually much more useful to think of it as it in relation to Your HR salary and staff costs, because you're getting significant, we think, at least 20 to 30% automatic productivity lift when people use it the staff.

and if you just think about a 20 to 30% return, you're not paying for that per employee when you're paying $20, $25 a month. Adam Brotman point, Andy. Andy ⁓ that was one main comment. you you had you had something you wanted to say about it?

About the Adam Brotman Well, I just wanted to add on to your great point about like the that don't just think of the costs framework wise as like how am I paying for this in terms of tokens or subscriptions versus how I'm paying for other technology software. It's better to think about in terms of overall well, I pay for intelligence In people. I hire people, I have teams that have a cost there, and I have an ROI analysis I do there. Like this is actually more analogous to that than it is to you know paying for some SaaS software.

And so I think that's a really good point you're making. And a great way to think about that, let's just make it really tangible. If you're running a hundred-person company and you're using Claude and you're on the What's called Claude Team, and it's like their business account, and it's secure, and it's like you can administrate it, and there's like an administrator that's looking at all hundred people that you've given access to Claude to in this team business account. And I think there's two options typically in team.

There's like a twenty-five dollar a month option per head, or you could you could set somebody up to a hundred dollar a month or something like that. And you get to decide and by the way, if either one goes over their limits, you're allowed to approve whether or not they they're allowed to go over and you can see how much they're spending over, whatever. So you have control over it. so you can see it.

You know, are they at twenty-five, are they at a hundred? Are they going over? How much are they going over as an administrator? So let's say you have a hundred person company.

It probably makes sense to start with You know who your five super users are. So you set them to a hundred. Everyone else is at twenty-five. And you got your monthly bills at like three thousand dollars a month, okay?

Or thirty-six thousand dollars a year or something like that. I'm making up numbers, but I'm in the ballpark. But then all of a sudden, if everyone starts coming to you going, I keep hitting my limits. Can you move me to the 100?

And you can see they're using it and they're going over, and it's like 50 bucks a month, over, 20 bucks a month, over. Yep, I it makes more sense to slide them up to the hundred. Because they're using it, they're telling you they're getting value from it, they're telling you they're more efficient, they're getting better answers, they're driving more revenue, et cetera. Fine.

Let's just say you put all hundred people up to $100. Okay, now you're at $10,000 a month or $120,000 a year, but you know because of the way you did it that everybody was getting a ton of value out of it. At least they say they are, but it's not like blind faith. You haven't measured a perfect ROI, but you're like, wow.

Everybody, I saw them all increasing their usage, hitting limits. I slid them all up. Now I'm at 120,000 a year. That's a lot of money.

But it's like kind of one FTE. So it's like one person. And so you've just taken your whole company and demonstrated that they are all using it, getting value out of it. And that's not a perfect ROI analysis, but like, my God, it's a lot better than just everybody gets a seat license to something I don't even know they're using it or they're not using it or whatever.

Like this up. Far cry from that. And to your point, for the cost of one person, I can get a hundred person company all using the heck out of an AI system, telling me they're getting a ton out of it. Like there's no way you can tell me that that's not worth one person.

Right. So it's not a precise ROI calculation, but it's the mentality that we like to recommend to your point, how people think about AI, ROI. how people are using it. Think of it more of like, you know, what it's doing to your staff and think about it more as intelligence as a service and an ROI analysis there versus like some you know tech cost you're spending in an ROI on.

Andy Yeah, so I think, I mean, we could go on. There's more to say on cost, particularly around time and prioritization. But I think on the cost side, we covered it. Let's move the ROI and let's start talking about the R in the ROI.

And I think what we've seen and what we've observed in conversations with the market is that, and we've said this previously on other episodes. Is that business leaders typically go to cost savings and they typically go to existing workflows. And particularly when it comes to R, one of the points, and this is like a relatively new point, is that where you choose to aim the agentic and AI machine is really, really critical. And that means how you as a business leader, where you focus your time on deploying AI.

and as well as whether it's a cost savings or more of a either revenue producing or moat producing differentiation competitive moat producing ⁓ customer experience where you aim that as a business leader really impacts the ROI the ability to measure the ROI whether it's actually measurable and perceived or just intuitive. So let me give a concrete example. It's super easy if you're a B2B company, to have a before and after measurable ROI if you focus deploying agents, AI agents in your sales process.

that's both like cost savings, but it's focused on being more efficient and it's a process That's what we would call a measurable return on investment. Most instances in deploying technology, the conversation around being really the press conversation around AI and the return on investment on AI, it's really like you don't bring that much rigor to like all your deployments at technology, you tend to like have an intuitive understanding that it's going to make you more efficient.

And so the sales example is one that is measurable. Most instances it's not. And I would say that you should focus your aim on instances that are growth oriented, not cost savings. I said a lot there, but pick up on anything that you want, Adam, and Adam Brotman No, I I'll just round out what you're saying because I agree.

Like in other words, it kind of goes to your point about don't think of this as like a typical technology implementation. Think of it more like a people initiative. How you know, when you run your company, you go, I'm looking for, you know, some unlock of revenue growth or cost savings or improved customer experience or whatever. And you're like, You sit there as a company thinking, if I could achieve, you know, the following capability set or whatever, I'll unlock some either revenue growth or cost savings.

And so we don't typically do an ROI analysis on that as business leaders. We're like, hey, I have a general goal I'm trying to achieve. And so I've got people on my team and those people use tools called technology. And they talk to one another and they figure stuff out and they accomplish things, they manage things, they introduce new products and services and capabilities for the company.

And AI is just an incredible enabler of that, whether it's agentic or not, like if you can sort of weave that into what you're doing to unlock something, because it's smarter, faster. cheaper, all the above, it's usually intertwined with the people on your staff using it as a tool, that's how we like to think about AI. to your point, your best ROI is when you do that, when you actually integrate it into what your people are doing, but they're just able to become output managers of the agents and they're able to sort of integrate it into And whatever they're doing to unlock something to your point about to aim it at an unlock.

if you're able to accomplish that better, faster, or even at all because of AI, then that's your ROI. we as business leaders don't normally think that way because we're just like, I got people, I've got fixed costs in my staff, and I and they're trying to achieve something, somehow because it's AI is like technology, we want to apply this like direct deterministic software version of ROI to it. When that can be the case, like you're pointing out about the software, the sales, excuse me, example.

most of the time it's the ability for you to just do your job better as leaders and as managers using this tool and then unlocking something and saying, okay, great. Now, probably you could take, you know. What you spent on the AI, the time your people spent, you could try to run some ROI analysis and A-B tested against what would happen if you didn't do that. But that's not how we typically measure ROI.

And I just think people are getting, just to kind of bring this full circle for this episode, people are getting so hung up on the, my God, the the cost went from 3,000 a month to 8,000 a month, or I reading about these crazy token max encoders that are using the API and running out of their budget. What you know, what do I do? There's no more subsidy in the industry. Is this all gonna come crashing on me and it's just so expensive?

And the answer is like, hold on, like if you manage your cost, if you manage your model use, if you are educated and administrative it right, then yes, you're gonna go from zero AI cost to some number, but if you're doing it right, it means your business is run better. And that's your ROI on all of that, but it's not a pure classic ROI analysis and stop. measuring it like pilot by pilot by pilot or like token use by token use. Think of it in a more holistic way.

Andy So I'll close out. I'll be typical Andy, I'll be more provocative and advice giving and direct, which is if you want some tips about how to think about ROI, stop measuring it for the time being when it comes to AI and Agentic Why? I think business leaders in general, from what I've seen talking to people, is there So eager to measure ROI that they spend a lot of time doing that and it impedes the exact experimentation and usage to try and actually understand where the learning that you need to get so that you can actually aim the agentic and the AI machine properly.

So I would say stop measuring your ROI. That's unsolicited advice number one. number two, I would say I think early in this episode, the framing that we had around cost, which is as cost goes up, think about it relative to human salary and the productivity gains that you get from your employees being more productive. I thought Adam's summary of the hundred person company and if it ends up costing you and you go max everyone because everyone's max using my hunch is this that you won't have a clear ROI It won't all be measurable my strong hunch is that the ROI on that is pretty significant.

You'll feel comfortable and you'll sleep at night even without measuring it. ⁓ the last thing is it seems that this wave it resembles history, I remember when the cloud came out and there was all this talk about ROI of cloud. There's a lot of excitement about cloud, and yet people are like trying to prove out the ROI. Every tech psycho goes through this, it's mostly the journalists building it up.

I would say if you're listening to this podcast, You know, you yourself should be trying to token max for like a week and see the gain and the learning that you have. So with that, hopefully this little revisit on ROI was useful to all of you. Adams, thank you again for as always being my partner in crime on this. And thanks to the audience for listening to AI First with Adam and Andy.

For more resources on how to become AI First, you can visit our website, forum3.com. Download case studies, research briefings, and executive summaries, as well as join our email list. We also invite you to connect with our AI First Community, a curated hub and network for leaders turning AI hype into action.

We truly believe you can't overinvest in your AI learning. Onward.