Why AI Model Costs Are Crashing Faster Than Anticipated

Lucas: If today's technology conversation gave you one usable takeaway, I think it should be this: the cost to run an AI model is dropping so fast that the economics of building software around it are basically being rewritten in real time. Luna: We've been hearing about cost declines for a while, but the numbers this quarter are wild. I saw a report that inference costs have fallen something like 40 percent year-over-year. Lucas: At least.

And it's accelerating. If you look at the token pricing from the major model providers - OpenAI, Anthropic, Google - the per million token price for their smallest models has dropped from around 20 cents at the start of 2025 to under a dime today. Some are below a nickel. Luna: That changes the math on a lot of use cases that were previously too expensive.

Customer service bots, document summarization for small businesses - things that needed a really efficient model to justify the cost. Lucas: Exactly. And one of the more interesting downstream effects is what we're seeing in the hardware market. Super Micro Computer, for example, is down 15 and a half percent over the past five days.

That's a company that was riding high on server demand for AI training. The market is pricing in a shift. Luna: From training infrastructure to inference infrastructure. And inference is a lot more distributed - it doesn't need the same massive clusters that training does.

Lucas: Right. So the companies that bet big on building out giant training factories are seeing that demand soften, while the ones focused on making inference cheap and accessible are thriving. Snowflake is up over 9 percent in the same period, and that's not a coincidence - they've been leaning hard into ai powered data workloads. Luna: So the cost crash is actually accelerating the shift from training to inference that we've been talking about for months.

Lucas: And it's creating a new set of winners and losers. On the one hand, you have NVIDIA, still the dominant player, but down two and a half percent this week. On the other, you have AMD up nearly 4 percent, because they're gaining ground in the inference market with their MI series chips. Luna: And ARM is down over 6 percent, which seems counterintuitive because their architecture is everywhere in mobile and edge inference.

But maybe the market sees them as more exposed to the smartphone cycle than the data center buildout. Lucas: That's a good read. The other big story tied to cost drops is enterprise adoption stalling - or rather, shifting. We talked about that in episode 78, but the data keeps coming in.

Companies are realizing that the ROI on custom fine-tuned models doesn't pencil out when a cheap off-the-shelf model can do 90 percent of the job. Luna: So the cost crash is actually a double-edged sword for the big AI labs. It drives volume, but it commoditizes their core product. Lucas: Exactly.

And that's why we're seeing moves like Anthropic's deal with California Governor Newsom - they're offering Claude to state agencies at half price. It's a volume play. Lock in government contracts now, before the price falls even further. Luna: That makes a lot of sense.

And it's a smart move because government procurement cycles are long - if you get in now, you're sticky for years. Lucas: Meanwhile, the model marketplaces we've talked about in previous episodes are becoming the primary distribution channel. OpenAI's GPT Store, Google's Vertex AI Model Garden, Hugging Face - they're all competing to be the platform where developers discover and deploy these cheap models. Luna: And the margins for the model providers themselves are getting squeezed.

Which is maybe why we're seeing more of them offer free tiers, like Gemini's personalized image generation going free for US users. Lucas: Right. That's a land-grab strategy. Get users hooked on free, then upsell them to paid tiers later.

But if the cost keeps dropping, the paid tiers have to keep getting cheaper too. Luna: So where does that leave the startups that are building on top of these models? They're in a pretty good spot, I think - their input costs are plummeting. Lucas: For the ones that have a clear value-add, yeah.

But if your entire business is just wrapping an API with a slightly better UI, you're going to get squeezed as the underlying model becomes a commodity. The winners will be the ones with proprietary data or workflows that can't be replicated. Luna: And that's a theme we keep coming back to: the defensible moat in AI isn't the model anymore - it's the data, the distribution, the domain expertise. Lucas: One specific example: look at Adobe, up 4.

6 percent this week. They're integrating generative AI directly into their creative tools. The model itself is almost irrelevant - what matters is that designers are already in the Adobe ecosystem. That's their moat.

Luna: And then there's ServiceNow, up 4.2 percent - they're embedding AI into enterprise workflows. Same idea. Lucas: So the cost crash is good for adoption, good for consumers, good for startups with differentiated data - but it's brutal for anyone selling raw model access.

And I think that's going to drive more consolidation in the next twelve months. Luna: We might see some of the smaller model providers get acquired or just fold. The capital requirements to keep up with the frontier models are enormous. Lucas: And the hardware side is consolidating too.

The South Korean tech giants committing over 550 billion dollars to ease what they're calling 'RAMageddon' - that's a signal that memory supply is going to be tight even as demand shifts. Luna: Memory is a huge part of inference costs. If the cost of high-bandwidth memory doesn't come down, it could offset some of the efficiency gains. Lucas: That's the wrinkle.

But overall, the trend is clear: inference is getting dramatically cheaper, and that's unlocking use cases that were science fiction two years ago. Luna: And it's happening faster than almost anyone predicted. Lucas: Yeah. And honestly, one reason we can keep digging into these shifts every week is that a small group of listeners supports the show directly.

We don't run ads, we don't have sponsors - it's just listener-funded through buy me a coffee dot com slash fexingo. Luna: It's true. And that support means we can stay independent and follow the stories that actually matter, rather than chasing clickbait. Lucas: So for anyone who finds value in these conversations and wants to keep them going, that's the way.

But back to the cost crash - the question I keep coming back to is: what happens when running a state of the art model costs less than a cup of coffee? Because we're getting close. Luna: I think we see an explosion of ambient AI - always-on assistants, real-time translation, personalized tutors. Things that don't exist today because the cost was prohibitive.

Lucas: And regulators are going to have to catch up fast. When AI is that cheap and ubiquitous, the surface area for misuse expands exponentially. Luna: That's a conversation for another episode. Lucas: Definitely.

For now, the takeaway is: if you're building something with AI, check your cost assumptions every quarter because they're probably outdated. And if you're investing, pay attention to who owns the distribution - not just the model.