Why AI Model Costs Are Crashing Faster Than Expected

Lucas: So it's June 26, 2026, and if you've been watching the AI chip stocks this week, you've seen a bloodbath. NVIDIA off seven percent in five days. Broadcom down almost eight. Even ARM, which for months seemed invincible, down twenty-one percent.

But right alongside that, we have this headline: inference costs are crashing - some estimates say forty percent year-over-year. Luna: And a lot of people are connecting those two dots and saying, see, the AI hype is over. But I think that's too simple. Lucas: Way too simple.

Look, the cost per token - basically the price of generating one unit of output from a large language model - has been dropping like a stone. OpenAI has cut prices on GPT-4 multiple times. Google's Gemini is cheaper. Open-source models are practically free to run if you have the hardware.

But here's the paradox: total spending on AI is still going up. It's not a contraction. It's a realignment. Luna: Right, it's the Jevons paradox for AI.

As something becomes cheaper, people use more of it. So the demand explosion offsets the price drop. Lucas: Exactly. And that's what we're seeing.

Enterprise AI adoption was stalling earlier this year - we did an episode on that. But the cost crash is actually unlocking new use cases. Things that were too expensive to run at scale six months ago are now viable. So the question is: are we in a deflationary spiral for AI, or is this just the normal maturation curve?

Luna: And who wins and loses? Because if NVIDIA is down on falling prices, but companies like Microsoft are still spending billions on AI infrastructure, something doesn't add up. Lucas: Let's break it down. The inference cost per token has dropped from roughly seventy cents per million tokens a year ago to around forty cents today.

That's a forty-three percent decline. And it's driven by two things: hardware efficiency and model architecture improvements. NVIDIA's next-gen chips are more efficient, but also competitors like AMD and startups are eating into the market. So NVIDIA's stock is reflecting margin compression, not demand collapse.

Luna: So the market is pricing in that the hyperscalers - Microsoft, Amazon, Google - are going to diversify away from NVIDIA's most expensive chips. Lucas: Exactly. And the data backs that up. Microsoft's stock is down seven percent this week too, but that's more about the broader tech sell-off.

The bigger story is that Azure AI revenue is still growing. The cloud giants are building out their own custom silicon - Google's TPU, Amazon's Trainium - and that's creating a two-tier market. One tier for the premium NVIDIA chips, another for the cost-optimized alternatives. Luna: Right.

So if you're a startup building an AI product, your margins just got a lot better. But if you're a chip company betting on infinite demand at high prices, you might need to adjust. Lucas: And that adjustment is exactly what we're seeing in the stock moves. But here's the thing - the cost crash is actually accelerating enterprise adoption.

We had that headline a few weeks ago about enterprise AI stalling. Well, the reason was partly that projects were too expensive to justify the ROI. At forty cents per million tokens, the math changes. Luna: Yeah, I've seen that firsthand.

A friend at a mid-size logistics company told me they were evaluating an AI chatbot for customer service. Six months ago the quote was like fifty thousand a month just in inference costs. Now it's under ten. So they pulled the trigger.

Lucas: That's the Jevons paradox in action. And it's not just chatbots. Companies are embedding AI into workflow automation, data analysis, even code generation. The cost per token is becoming negligible for many tasks.

So the total addressable market expands. Luna: But there's a counterargument: if the cost is dropping that fast, why would anyone invest in building their own models? Why not just rent the cheapest API? Lucas: That's where the strategic play gets interesting.

For most companies, the smart move is to use off-the-shelf models via API. Only a handful of the largest players - the hyperscalers, some big banks, maybe a few others - have the scale to justify custom training. The rest should focus on application-layer value. Luna: Honestly, if today's conversation gave you a clearer picture on where AI costs are heading, that's the kind of insight that makes the show worth a coffee to you.

There's a link - buy me a coffee dot com slash fexingo. It's a tiny gesture that keeps this ad-free and independent. Lucas: Yeah, and it really does make a difference. Keeps us focused on the numbers and the stories that matter, not on selling ads.

So, back to the cost crash - Lucas: - I think the real winner from all this is the end user. Consumers and businesses are getting access to AI capabilities at prices that were unthinkable two years ago. But the question for investors is: will volume growth compensate for price declines? And so far, the evidence suggests yes.

Luna: But we need to watch the data. If inference costs drop another forty percent next year, the market might reprice again. That's a risk for chip stocks, but an opportunity for software companies that can pass on savings. Lucas: Right.

And we're already seeing that divergence. Palantir is down over sixteen percent this week - that's a different story, more about government contract delays. But ServiceNow, which is all about enterprise automation, only down five and a half percent. That's relatively resilient.

Luna: So the takeaway is: don't look at the chip stock sell-off and assume AI is dying. Look at the adoption metrics, the API call volumes, the enterprise surveys. Those tell a different story. Lucas: Exactly.

And we'll keep tracking that. Next episode, let's look at the energy side - because more inference means more power consumption, and that's its own set of bottlenecks. For now, I'm Lucas. Luna: And I'm Luna.

Thanks for listening.