Do AI Tokenomics Matter More Than Model Benchmarks? with Chris Potts - #776

Speaker A: This episode is brought to you by Blizzy, the autonomous software development platform built for enterprise scale. Today's code bases have grown beyond human comprehension. Millions of lines, decades of tech debt and complexity existing tools just can't fathom. With Blitzi, thousands of specialized agents reverse engineer the code base, mapping the architecture, dependencies and business logic with that context. The platform then autonomously executes entire epics, writing, validating and testing the code for every project. The result? Fortune 500 enterprises are able to modernize legacy systems and ship new features five times faster. Want to try Blitzi on your code today? Unlock 1 million lines of reverse engineering and 25k lines of code generation by visiting blitzi.comsandbox We've started to enter a phase of AI that's not just about making models smarter, it's also about making them economically sustainable. As reasoning models consume more tokens, context windows continue to grow and agents become embedded in more products and workflows, the economics of these systems are becoming impossible to ignore. That's given rise to a new conversation around tokenomics. How we think about the costs, incentives and trade offs shaping the next generation of AI. One person who's been thinking deeply about this is Stanford professor and BigSpin co founder Chris Potts. His recent work argues that measuring AI progress requires looking beyond model benchmarks to ask a different what are our tokens actually buying us? Here's Chris explaining how he thinks about tokenomics.

Speaker B: Another interesting moment to be in, as we're all being made aware of the true costs of all this AI usage. The analogy here is like it used to cost me $20 to take a, uh, ride share to the airport, Uber or Lyft, and now it costs 90, but it's more like 20 to like 500 or something, right? I think what's happening is that the big providers are testing the waters on charging us the true costs plus whatever profit they need to make as they all try to gear up for IPOs and so forth. And in turn, that is very quickly leading people to ask questions like what is the return on investment for all these tokens that we have purchased? And it's a very tricky area to be in because what does it mean to think about value in this context?

Speaker A: I'm, um, Sam Charrington and this is the TWIM AI podcast. For over a decade I've been exploring the ideas and innovations shaping the future of AI through conversations like this one that help you understand what's real, what's next, and what matters. Let's jump in. I want to say, thanks for coming on. I've been looking forward to this conversation. And I think where I'd love to start us off is to really dig into your background and how it kind of got you to where you are now.

Speaker B: Yeah, my background is in linguistics, um, linguistics proper. Not even natural language processing. I did my PhD on, among many other things, swears. What swears are like, why we swear, what information they encode, what kind of taboos exist around them, and so forth. And that was actually the trigger that got me into nlp, because I wanted a lot of data of people swearing. I wanted to know what the context was like, what their intentions were. So I turned to corpora. And from there you start using NLP toolkits to add structure to those corpora. And then after a few years, maybe you're writing your own tools for doing that work. And then when you look back after 18 years or whatever it's been, you're just an AI person or an NLP person. But that is the true story. And I feel like if I had to, I could trace the lineage of every one of my current projects back to my fascination with why we care when someone drops an F bomb.

Speaker A: So are you an F bomb dropper, or did you come at it from the perspective of trying to understand these others?

Speaker B: I think very infrequently in my life, um, on. For my linguistics class, Semantics and Pragmatics, which is about linguistic meaning. On the final day, we always do a class on swearing, and I review the history and we kind of tie all the course themes together. My handouts for that are full of swears, but I only swear once in the lecture, I present the result that people remember things better if the utterance contains a swear, because it has a kind of emotional resonance, very primitive reaction. And so in that moment, I pick some fact from the course, some trivial thing, and I restate it with a swear. And then I say, all of you will remember this for eternity. But other than that, I'm very shy about it in the class.

Speaker A: That's funny. I'm sure there is loads of research on this, but I, uh, grew up in New York City, and as a New Yorker, I think that swearing is just kind of part of my natural language and way of communicating. And I married a Midwestern girl, and she doesn't tolerate it at all. She doesn't do it. She doesn't tolerate it. She won't tolerate it for me. And it made for. We've been married for 30 years, so I adapt quickly, apparently. But uh, it, you know, for a long time it took a lot of restraint to like, change that way of communicating, particularly when I'm communicating about something that I'm excited about or emotional about or, you know, want to convey the importance of. It's a really interesting topic. And uh, um. Well, we're not going to turn the podcast into a podcast about swearing, but I imagine there's enough research there that we could if we wanted to.

Speaker B: It's a fascinating area. Yeah. Because it gets right to the heart of the culture that we've constructed and how it relates to our usage and everything else about us. Yeah. It's fascinating that we have swears. When the old swears lose their power, we invent new ones. We pretend like nobody should use them, but as you say, people use them all the time. And it feels like an important part of being a language user that we've got them available to us. Yes. Endless string of questions.

Speaker A: I'd love to hear your take on kind of a linguist in the age of modern AI, you know, transformers, statistical models, you know, this is a, uh, NLP used to be kind of coming from a linguistic perspective and now the entire field is shifted to a, uh, statistical perspective. And I'd love to hear your reflections on being on the other side of that transition as well as maybe more importantly ways that you think that kind of the traditional foundational linguistics is still important to the way we think about AI, uh, today.

Speaker B: These questions are on my mind all the time. Yeah. Because I operate at the intersection of all these different fields and I will say useful to distinguish in this context linguistics, you know, and people in my department at Stanford study language and social identity, historical linguistics, the structure of language. And they're just doing scientific investigation of language as a human phenomenon. And they are not technologists and they're not trying to inform technology. So their project is interestingly impacted by technological developments for NLP people who are of course participating directly in the engineering project. They're affected in a very different way by the rise of Gen AI and the kind of homogeneous nature of the solutions that people now adopt in that space. So for the linguists, I feel like this is the most exciting moment that anyone could have dreamed of. I feel incredibly privileged to be alive in this moment where humans encounter for the very first time non human creatures that use our language very fluently. I think it's weirding us all out. But from the point of view of understanding the human capacity for language, what a gift. Because you can ask about the mechanisms, um, which are different from humans, but obviously sufficient for achieving a certain kind of behavioral performance. Um, we can think about them as investigative tools. I mean we train them on the Internet. They're basically incredibly powerful distributional learners. And we can learn a lot from them about the true structure of language by just looking at the kinds of things that they learn. And it really gets at the heart of core questions in linguistics about how much of language learning is innate and the nature of our capacity and whether it's statistical or symbolic, all those things come flooding in in a completely fresh way. And so whatever your reaction to language, uh, models is, it should be a significant one, right? This should be causing you to rethink key questions and that's all you could hope for as a scientist. That you have new angles, new perspectives, new questions reopened. That's been incredible for nlp. I think it's a more uncertain prospect because pre, uh, the arrival of like pre trained models, which for me would be like the Elmo model back in 2017, 2018. Before that there was still a lot of statistical work, of course, and we were in the deep learning era. But you could still for example, do a PhD that was entirely about some specific phenomenon and maybe some very specific tweak to a model. So you could say, I'm going to work on summarization and I've got a new idea about how to do that. Well, using deep learning models, that could be your Ph.D. and what we started to see 2018, um, 2019, 2020, especially with the arrival of GPT3, that that was a very uncertain prospect because you might wake up one morning to find that you had been completely scooped, that with essentially no effort, one of these large pre training runs had done better than you at the thing that you'd worked so hard on. And that caused an interesting, probably overall productive, but interesting and challenging crisis for people, especially students who are trying to figure out what to do next with their PhD research. But I think all of us felt a, uh, kind of real uncertainty in that moment.

Speaker A: Yeah, I remember the anxiety of that time and I always felt it was kind of expressed as, you know, is research and NLP fundamentally like scale limited or do you need a certain degree of scale that only a handful of organizations have to do foundational research? And is everyone else going to be relegated to like poking the, uh, poking the beast and seeing what it does? And I'm curious, do you feel like that was an anxiety that's passed or is it still very present? Has it panned out quite like that? Uh, how do you, you know, how's it been resolved for you?

Speaker B: Also fascinating. Not resolved. It's something I discuss a lot with my collaborators and with my students. We're all trying to figure this out in this moment. I will say one concrete thing we did was orient a lot of our research toward interpretability. Uh, just the project of understanding how these models end up being so good at such hard tasks. And the reason we did that is it's relatively inexpensive. And it's also an area where clearly you would be explicitly hoping that models would get better because then there would be more to explain versus if you were doing that summarization project, you might quietly be hoping that there wasn't going to be so much progress so that you could make the progress. Like, let's hope the next model isn't good at summarization. I want to be the star of that show that's, as I said, very uncertain. But if you're doing mechan, you're like, let's get the new model released, because now we're going to have even more structure to find, even more to explain. And that felt like a very productive choice. I, uh, don't want to leave out the fact that it's also cheaper to do this research, and that is significant. And then I would say that right now a lot of us are in a moment of thinking we should do stuff that is weird and creative and out of the mainstream. We should be thinking about trying to achieve the next big thing, because competing with these massively resourced, incredibly creative and talented teams is just not a winning game. So let's play a different game and hope that that's, as they say, where the puck is going, not where it is.

Speaker A: And, uh, what are some examples of that kind of thinking?

Speaker B: We've been thinking a lot about architectures because I have a lot of complaints about current architectures. And I would say the other main theme right now for us in my group is thinking about data. You know, data have strange and wondrous properties. I think we don't understand how data affect models. And that has all sorts of implications for security and safety and also the nature of the learning that these models do. It really, data is fundamental. It's all data driven learning. And so telling the full causal story from data to final model state feels like it will just be significant for lots of questions. But I wouldn't want to leave out the architecture one, because I feel like the architecture everyone has arrived at, uh, these stacked transformers that we make very deep and very large are tremendously inefficient you would hope they were using all that depth and all that representational power to learn modular recursive functions for things and all sorts of exciting stuff. It is not what we find. And that seems like a real opportunity to just level up and do better. And maybe we could get massively more capable models with half the depth and a quarter of the representational width and that would be transformative for the economics of AI, uh, in addition to leading to all sorts of exciting things for capabilities.

Speaker A: It's funny and maybe a bit validating for me to hear you say that, because whenever I articulate a thought in that direction, uh, particularly with folks that are coming from the Frontier Labs, or um, essentially the Frontier Labs, I get back this kind of feeling that, yeah, you're just not bitter lesson pilled enough. Like structures. That's old school thinking. You're just trying to train some features, just collect a lot of data, throw it at the model and that's all you need.

Speaker B: Okay, but here's my response to them. Let's say rewind to 2017. We've got the transformer. It's got absolute positional encodings and it's got a particular structure for its uh, MLP layer, which is pretty narrow and pretty dense and a certain structure to its activations and its uh, layer norms. That's 2017. The bitter lesson pilled thing to do would be to scale that up. But just consider for example, how much it would cost to use the N squared attention and the absolute positional encodings, but have a context window of 1 million. This is the bitter lesson pilled thing, right? Just keep scaling. But it would be absurd. It would cost trillions of dollars to produce models that we all interact with right now. What did people do instead? They thought hard about locality and they thought about how uh, like positional encoding should be, favoring local relationships. They completely rethought the MLP so that it's now wide and sparse. Everyone did careful work on the uh, activation functions to make sure there weren't weird outliers so that they could quantize in a good way and so forth and so on. All of this analysis work built on intuitions about data and learning led to the model that we have now, which is like a ship of theseus compared to the 2017 transformer. The only thing that survives is attention and the feed forward layer. And I claim for you that none of that stuff is bitter lesson pilled. That was all analysis work that was meant to save based on priors and the Data and priors about how they knew learning would happen. So I go back at them. You're not bitter Lesson pilled enough, apparently. Although this is a reductio, I think.

Speaker A: Oh, I love this. That's such a great response. Uh, I think it also really calls out the relationship between data, mech, interp and efficiency, like core themes that you've been focused on and how they, you know, interrelate and support one another.

Speaker B: Yeah, absolutely. And this relates to one of my hot takes. You know, it's very fashionable, especially among interp researchers. But I think in general for people to say we don't understand how these models work, it is also very mysterious to us. But the truth is that people in the field have very deep intuitions about how these models work and that is the causal factor in us making so much progress. Because they could think analytically. What would the structure of positional encodings and attention be so that I could do this at million context scale? You can only achieve that kind of thing based on deep analysis and insight, not by just guessing. And so when people say, oh, we don't uh, understand, I say, I think you understand much better than you're letting on. I think you understand at least as well as my car mechanic understands how my car works. There are mysteries, but you can take a lot of action and be very effective improving things.

Speaker A: Why do you think they say that? Why do you think they say that they don't understand the models? There's gotta be some payback there.

Speaker B: It's probably a paradox of expertise, right? So the more you do know, the more you feel like there are also mysteries. And it's hard to step back from that and be objective and say, yeah, well we did make a phenomenal amount of progress and that can't be just because of happenstance. That was because we know a lot. But all you see as an expert is all the things that are still to be explained. Um, partly. Also it's just a narrative in the field and it does stretch back to days when I think we had very little understanding of how these models worked. And possibly because a lot of them weren't that good, there was very little to explain. And so that's just been slow to catch up with how much progress we have made in understanding the kind of intuitive human level mechanisms that these models are operating with.

Speaker A: I also wanted to ask you about dspy. I forgot about this as we were talking earlier, but you were involved in dspy, which, um. Well, I'll let you talk about it, but I'm curious, uh, how it connects into your research and like uh, some of these pillars that we've talked about.

Speaker B: Oh, there's lots of wonderful strands and what a meta strand I could offer you because we were talking about being strategic with research. This does stem from Omar Khattab, my student. He's the visionary behind DSPY and still its lead. And he just had the intuition early on that we should rethink what it means to make a scientific contribution. Previously we thought in terms of papers as the beginning and the end of all of this kind of thing that you would contribute. We should instead, he said, think about projects and about empowering people. And so for him the paper is one part of a broader contribution that might actually be centered on an open source or open weights release that would allow people to do big things. And that's where you find impact and that's the nature of a contribution going forward. And DSPI is a kind of embodiment of that. Although he made a similar investment with the uh, Colbert retrieval model and then people built on what he did and then you really saw it take off where open source contributions made it easier and easier to use that technology, leading to more and more impact. And of course DSPY is another wonderful example because in investing in this community and in the open source resource itself, he built a huge following. There are lots of startups, mine included, where the core tech stack for the LLMs is built on dspy. And that has made life so much easier. And then of course it was a platform for him and for us to really think in an innovative way about prompt optimization and agentic workflows and all of those things.

Speaker A: Yeah, I was thinking not too long ago the degree to which model strength, uh, as a correlate to model size I suppose, and capability has kind of overcome the need for an explicit framework like dspy. Dspy, yeah.

Speaker B: There are kind of two levels to that. One would be just the engineering side where dspy, uh, is great four years ago because it's kind of hard to construct the code around one of these systems in a way that's modular and reproducible and so forth. Because pecking out something where you've got a prompt string in the middle of your code with some slots in it, it's very error prone and it leads to bad system designs. And DSPY solved that. And you could think that the need for that is diminishing somewhat because now we all specify these systems in English and have the coding agents do them.

Speaker A: And even before that there were, you know, Another hundred frameworks that solved that particular part of the puzzle.

Speaker B: Oh yeah, there's always competition and I think at that level of just thinking about programming interfaces and APIs, they can all learn from each other. And so like, you know, DSPY learned a lot from Pytorch in terms of layer wise design and the of modularity that introduced. And then of course you would hope that everyone kind of slurps up all these interesting innovations and it leads to everyone being better. There's lots of evidence of that. At the level of interfaces. I would maintain for you that even if we have agents actually writing the code for these systems, it's great for us and for them if they write it in something that actually expresses these systems as modular components so that we can audit them, so that they can change them. It just feels like good engineering practices for any agent to think in a modular way. And that's what DSPY encodes. The other side is like the prompt optimization side and a uh, belief people have that the need to be careful with your prompts is diminishing over time. I understand that narrative, but people should also, for example, just run like a simple annotation study where they use a few different models or the same model a few times on slightly different data. They will be blown away by the amount of variation that still exists. To be charitable, let's say that these LLMs disagree about fundamental facts about how to label certain texts or what kind of response to give. We all kind of slip past this because we feel like, hey, they're smart and they're good and they're getting better, but if you quantify it, it's pretty disturbing. And the next step from that is to think about having all those agents optimize a prompt so that their behavior is at least consistent. And then you're right back at that DSPY vision.

Speaker A: It's interesting that you say that because I don't feel like that necessarily aligns with my recent experience. And in particular, one thing that I've noticed that's been surprising is how well aligned. I guess maybe that's not the right word, but how similar the responses I get to uh, a uh, query across different models. So for example, these are often kind of what I would call like a casual prompt, a casual query, something that I might, you know, type uh, into Google and it will now generate uh, an LLM response for me and it's kind of AI mode, um, and I'll take the same thing and put it into ChatGPT and maybe Claude and it surprises Me that the, the responses are often very, very similar. Like, you know, very similar structure, very similar facts, very similar citations. And uh, you know, I, stepping back, like there are lots of ways that they could answer or approach these different questions, but it seems like, you know, the models or the training or the system prompts or something is all kind of converged on something that makes the models express themselves, you know, very similarly. Which, you know, seems to be at odds with uh, you know, the, the last thing you said about the need to optimize prompts or the impact of the individual prompt.

Speaker B: I'm open minded, but for example, like we just did a, uh, we did, we did a thing recently. We were writing a grant and we needed a title. And you want to be strategic with these titles so we come up with a whole bunch of them ourselves and then we all disagree on what would be the best. So let's find out what the agents think. So ask a few anthropic models and a few, um, GPT models, which of these five titles, which is the best? So you get a different answer from all of them, along with a detailed rationale about why. Obviously of course, the choice that the model has made in that moment is the best one. This is great because then we can think about which one of these arguments is most persuasive. But if you were hoping for consistency at an subjective labeling task, which this is one, uh, you can see right there that you're going to have a real problem unless you give very specific criteria and then you're kind of also constructing a prompt for them and you might want to manage them differently. There is a real, I don't have evidence for this yet, but we have an intuition at Bigspin in the research we've done that you get a kind of paradox that uh, the more requirements you add, actually the more variation you'll see because the different models will key into different subparts of the requirements. And since they do it very concertedly, you can actually get systematically biased behavior from something that you thought was a very good specification. And that again calls for this idea that what you need to do is figure out what the labels ought to look like and then have some automatic optimization process get the model there. And that's what things like JEPA and mepro were for.

Speaker A: A topic that I really wanted to, a topic that I would really like to dig into with you based on, uh, our previous conversation was the idea of tokenomics. It's something that people are talking about a lot recently. I, um, think, you know, Folks that use Claude code, for example, have like a very visceral experience with anthropic changing the terms around usage. But it's happening under the covers with all of these large providers. And so I think way more now than, you know, six months ago, like we're all a little antsy with the relationship we have with these big model providers and the value that we get. You recently wrote an article about this. Talk a little bit about a,

Speaker B: how

Speaker A: it ties into kind of your broader research, uh, but also some of the things that you, uh, found when you started to dig into this area.

Speaker B: Yeah. Another interesting moment to be in as we're all being made aware of the true costs of all this AI usage. I saw a tweet from Ed Zitron, just a screenshot from someone who was noticing that copilot was telling them that their bill last month was $500. And if they keep up the way they are with copilot's new billing, it will be $11,000 in the next month.

Speaker A: Wow. Wow.

Speaker B: Which is real sticker shock. And the analogy here is like it used to cost me $20 to take a ride share to the airport, Uber or Lyft and now it costs 90. I use that as, but it's more like 20 to like 500 or something. Right, right.

Speaker A: Um, if only the slope will be as shallow as Uber.

Speaker B: Right? That's right. We start to wish for those easier stories. Yes. And so what will happen? I mean, I think what's happening is that the big providers are testing the waters on charging us the true costs plus whatever profit they need to make as they all try to gear up for IPOs and so forth. And in turn that is very quickly leading people to ask questions like what is the return on investment for all these tokens that we have purchased? And it's a very tricky area to be in because what does it mean to think about value in this context? Even if we focus in on people who are doing just coding with coding agents, can we agree on what it means to add value? Maybe we have a few measures in mind like making a pull request or uh, committed lines of code that last in the repo for a while or documentation touched or skill files created. But we might also worry that that's not capturing the value from many kinds of sessions we have which are more open ended and about discovery. So that's the first question is just solving this value issue. Right. Let's just agree on what it would mean to add value for a, uh, for a coding agent.

Speaker A: Uh, I like this line of inquiry because to me it's the response to this thing that drives me crazy, which is, oh, big, uh, tech company, CEO. This year 95% of our code will be generated by AI. It's like, yeah, hey, what does that really mean at that level? Like what's, what's, what are the details beneath there? But uh, is that a good thing or a bad thing?

Speaker B: That's a, oh, another dimension. Right, which is, is that code a liability or an asset?

Speaker A: Right, right, right. And this idea of like uh, you articulated as kind of code longevity in the code base, that's an interesting way to think about it. There's probably a lot of interesting ways to think about it that very few are thinking about right now.

Speaker B: And all of these fall victim to the standard thing that once you make it a metric, it's no longer useful to you. Like if we said, oh, it's completion of projects, right. Well then everyone would just have many projects that they completed. But they could all be liabilities and add very little value. So. But one framework we could offer that we did in the research you alluded to is let's think about this like economists might. So we might have like a consumer price index. And the first step will be what's the basket of goods that we're going to consider in that standard land? It would be like the price of eggs and the cost of rent and other kinds of tangible goods. What are engineering goods that we might track?

Speaker A: Eggs might be a summary or a pull request, a ah, bug fix or something like that.

Speaker B: Or we could think broadly because we both use these coding agents, um, requirement discovery. Right. Knowledge accumulation. These are things that we don't currently track, of course, even as engineers, but might be behind our intuition that these coding agents are making us productive even if it's not reflected in the PR counts or whatever. Right. I mean in a sophisticated approach you might say, I don't want more PRs because this is just a certain kind of, um, busy work that doesn't relate to the actual goals I have. What are the actual goals? It's completing valuable projects and so forth. If I could do it with fewer PRs, um, but I had, you know, really robust code, I'd be possibly happy with that. So we gotta figure out what the basket of goods is, but then we could start to track it relative to token usage and that would be the consumer price Index. So for any time period we could just say, I've got my tokens spent and I've got my goods produced. Tokens divided by goods produced is a pretty rough measure of um, the purchasing power of the tokens in those time periods. Then you would do the standard consumer price index thing of making what they call a hedonic adjustment. So you could just say maybe quality is improving over time, so you'd pick some measure for that and make an adjustment to the line. And when we did that study we did code survival. So um, the number of lines of code that survives more than four days in the repository, we made an adjustment upward because that rate is going up.

Speaker A: That's surprisingly short. Four days, four day survival.

Speaker B: So again all this is around measurement and I'm happy to just be starting this discourse because we can see it's important to the economics of AI and it seems like the work isn't being done at a high enough rate for us to get a clear picture. So we could make it longer and maybe the adjustment would be different. I think currently for the data we have, which is this switchat benchmark, which was released by researchers at Stanford, it's about 6,000 real coding sessions, all the metadata, everything you'd want. What we See with Opus 4.6 Usage in the time period we have, which is February to mid April of this year, uh, a decline in the purchasing power of tokens. That CPI is going down. And again I just want to open the question, is it because we have the wrong basket of goods or is it because we're actually getting less value from these tokens? The, the, the one thing I can say that's kind of definitely a causal factor here is that in February of this year most of the tokens went to producing code which relates to the outcomes we just talked about. By mid April it was quite split between co generation thinking and also explanation to the user. And so that split now is going to have an effect on the things we're measuring and that might be cause for reflection. There's value in those explanations that's not reflected in PRs but might be reflected in something like knowledge discovery.

Speaker A: And I see that coming up within the same time frame. It's become very common to now talk about the token efficiency of a new ah, model that's been released with uh, the implication being uh, tokens of you know, internal use, tokens, thinking tokens versus you know, per token of output I guess is maybe a way to think about it.

Speaker B: Another fascinating dimension and this actually relates all the way back to the theme of efficiency for these architectures. So here's a claim I'll make for you based on my Read of the literature on inference time scaling, what's sometimes called test time scaling, which is just having the models generate lots of tokens at the moment that you ask them a question. So those scaling trends, everything we're seeing now is completely in line with those predictions, which is you get pretty good gains for a while with the more tokens you spend on a log scale. So this is jumping up quite a lot. But you do see it reflected in performance improvements. But it flattens out over time and it's not like this curve skyrockets. It's sobering. You gotta spend a lot of tokens for small gains in performance. We all knew this, uh, we all knew this and we're just seeing it now play out. And when people talk about token efficiency and worry about this, I think what they're seeing is just the real lesson of what we already projected from inference time scaling.

Speaker A: And this is independent of the, the approach to inference time scaling you're taking. Whether it's uh, you know, multiple parallel, uh, you know, multiple parallel inferences or some kind of oracle or you know, any number of other schemes, it's just fundamental to inference time scaling.

Speaker B: That's a great question. Right. I think we know that it's independent of some of those things like the parallel work versus having it do lots of long chains. But some of the other factors you mentioned, I think we just don't know. And that's why I said it relates back to the question of efficiency for these architectures. Uh, if we made a fundamental change to how the models work, maybe these trade offs would be very different. I mean after all, so all of this stuff is a kind of patch job on the fact that there's no recursion in the depth, it's a fixed depth. And so the only recursion we can get, the open ended notion of computation is by generation. But if we had models that could be recursive, maybe fewer tokens for larger gains. I think we don't know. Yeah, I mean in the end we're going to spend the cost on compute or tokens. So this might not affect our bills in the end. But it is a fascinating question, what are the true scaling laws and what's possible in this space? And you're right to push back. We talk about these things like they were like Platonic ideals of laws. Scaling law invokes that. Right. But even for the scaling laws for pre training, there's lots to discover there and many of the stories of progress are actually transcending the scaling law. And we See better improvements than those laws predicted because everyone worked so hard behind the scenes to do very innovative things. Which maybe relates to our bitter lesson

Speaker A: discussion, any particular example come to mind,

Speaker B: uh, of that data usage and the nature of the data really matters and overtraining the models really matters, which is kind of pushing up against the standard of scaling law presentation. And now I'm just going to speculate, I should check on this. But things like mixture of experts might have really flipped the script on what it means to count parameters and in turn how these laws relate. And then I think maybe even also stuff like the context window and so forth. This is another thing to check. But I just speculate that we've seen larger gains from pre training than you would have predicted by those early scaling laws papers suggesting that there is some innovative thing that was happening on top of pure scaling.

Speaker A: Thinking about the concept of a market basket, one kind of pushback that comes uh, up for me is in the uh, you know, the real economy, you know, eggs is different than milk is different than uh, you know, beef, et cetera, et cetera. And they're all influenced by different factors, uh, you know, production for example. Whereas what you've done with uh, this kind of CPI basket with tokens is kind of like, more like analogies. Like here's the typical bundle of work and you know, what it requires from a consumption perspective. But the tokens aren't fundamentally different. Like they're the same tokens. It's just like how much it takes to do this versus how much it takes to do that versus how much it takes to do that. Um, you know, tell me what I'm, what I'm missing there and you know, what does uh, kind of characterizing these products, you know, give you in your analysis?

Speaker B: Yeah, fascinating to think about. One thing I could insert there is the tokens are different at the level of being used for co generation or skill, file writing or explanation or thinking. Right. Those are different kinds of tokens that probably do feel tangibly different to us. So is that an element in your think?

Speaker A: I think I was thinking from our conversation that you had 10 different, almost like tasks, like 10 different types of tasks from the domain of code generation, which, you know, if they were all kind of largely code generation, you know, that is the part that had some dissonance for me. But if you're talking about like if your, your basket is like creative writing versus you know, a few code generation things that are kind of and different, uh, versus, you know, summarization versus editorial, commenting, feedback, Those, you know, may be more fundamental.

Speaker B: Yes. So I think this is very significant when we have done some research on this as well, at the level of what kinds of session types exist and in turn what kinds of users are there. So you might notice of your own behavior. I guess this is reflected in your comment that sometimes you want a quick check in on a question, sometimes you want a quick, um, PR to get fired off, sometimes you want to be in a mode of deep collaboration, sometimes you're partnering with the AI, sometimes you're delegating the work and so forth and so on. And the outcome measures that we choose should be sensitive to this. We shouldn't penalize the agent. If your chat interaction with it, uh, about some scientific question didn't lead to a pr, it was never on the table in the first place. Whereas if you're trying to get some work delegated, that's actually a coding task and all it does is chat with you, that would feel quite unproductive. So we need to bring that in and that would be a higher level discovery process of what people are trying to do and so forth.

Speaker A: In thinking about the notion of value, is this something that you're anticipating? Like, it strikes me that that's an entire research thread that, you know, one could go into. I don't know if that's a linguistics or linguist or an economist or a computer scientist. You know, probably inter, interdisciplinary, like most interesting questions. But is that, you know, is that something that you're working on or um, was it something that you put out there for someone to take up and run with?

Speaker B: I am not sure I can tell you the, the lineage of this idea is that we founded this startup, BigSpin, because we would like to see more people benefit from AI. Whether you love it or hate it, it's here. And I would like the benefits to be more evenly distributed. And I can tell that that will mean bringing on board many more people than currently benefit from AI. Uh, right now I would say that it's mostly experts deriving real value. And a lot of the world is currently even trying to figure out what this is all about as a tool or an entity in their lives. So we would like to have more access and more productivity. And that implies making the user experiences much better, um, figuring out what interactional patterns lead to success for people, meeting them where they are in a kind of adaptive way. The whole list of things that you might worry about if you were a product manager who had some deployed AI product. And I think by that route and from that perspective, we just ended up worrying about our own token usage increasing and wondering whether there's real value there. And it just happened to collide actually just like three weeks ago with this emerging narrative on the back, I think of all these rumors about IPOs, about what the return on investment was and then all these CEOs came out and said, oh, our spend was enormous and we want to scale back and we're walking back our claims from a few months ago. And that is just a fascinating thing to witness in the, in the narrative here.

Speaker A: I'm wondering, are you also, does this research also attempt to project forward? In theory you could um, you know, create a model for, you know, anthropics cost and spend, you know, based on, you know, publicly available data and some presumptions and give us a sense for how close we are to paying full freight for our tokens versus if we're only paying 10% for our tokens. You could then project what that cost might look like, uh, over time as we're paying more and more of the full cost.

Speaker B: Yeah, I don't have again fascinating questions. I don't have resolving answers. I am glad I am not tasked in some organization with projecting spend on all of this stuff because I think it would be basically impossible for the time period that I was describing for our little um, CPI experiment, Anthropic changed the default reasoning on the model at least two times. So we see like they launched it with default reasoning high. We have a mysterious sudden rise in the token usage which we cannot explain. Then there's a new baseline. They lowered it to medium as the default. They patched a bunch of bugs that were related to context management and then turned it back up to high. And all of these things have an effect on the total token output as you can imagine. They also changed the default context window, which meant people could swallow up much more, uh, stuff at any given moment. So imagine trying to predict what token spend is going to be like when you have all these exogenous events in addition to changes that we don't even know about. And questions about where the value actually lies, very difficult. And then, you know, the true cost of a token, the estimates vary wildly. For every dollar we spend, it could be as low as 2 and as high as 20. And I think this is just because it's hard to factor in things like R and D and future build out and depreciation and all of that stuff. I think at the current moment we just don't know. But there couldn't be a More significant question for the global economy basically than where the value is and who's going to pay and how much.

Speaker A: Yeah. In your article you coined the term tokenflation to describe at least the recent behavior of token, uh, economics. I imagine you see that continuing seems to be continuing.

Speaker B: Yeah, yeah, that's certainly the picture that we get from the cpi. Uh, a picture of tokenflation. Yes. Your token is not buying you what it once did. According to everything we can think to measure here and even adjusting for models getting better. Right. That's critical there because if it was just a story of models thinking more and being more robust and we were all getting exponentially better outcomes from this, then the spend would look completely rational. But that's not the picture that we see. And so we have to do some hard thinking about what's going to happen and how to improve the situation.

Speaker A: Uh, let's dig into that a little bit more. How would you articulate what you're seeing? The models are getting quote unquote better. Um, there's a set of open questions about uh, are the reported ways that models are better actually reflective of some intrinsic betterness? And that question brings, uh, is often about benchmarking and learning, uh, the benchmarks overfitting, that kind of thing. Uh, and then there's the kind of question of chattiness and the volume of thought that it requires a given model generation to produce an answer. What are other factors that you see?

Speaker B: Yeah, we could pick that apart as well. And this relates to, ah, a line I've had consistently which is that we should think in terms of systems, not in terms of models. So in the data that we've got Sonnet opus and Sonnet 4.5 versus 4.6, those two generation changes, those are real model changes I assume. I think they did something very substantive at the level of the weights. Um, and everybody immediately saw that that led to like a 5x increase in token usage. And this was related to the introduction of adaptive thinking. Now fix that. That's the level shift that we already took and maybe we're seeing improvements there that are worthwhile, it's hard to say. But let's assume there was a level up in improvement then for the period that we did our CPI experiment for, that's a fixed model, Opus 4.6. So all the code improvements that we saw in the data relate to the product. This has to relate to things like them turning the knobs on the adaptive thinking, changing things about the system prompt, changing things at the level of the product. And that's where the improvements were. And so that shows you that even for a fixed model we could get very different outcomes for these things because they really are sophisticated engineered systems at this point.

Speaker A: Yeah. And so was the product in this case specifically Claude code or.

Speaker B: Oh yeah. And so we don't. There's tons of stuff there. Yes, I believe we know that these are all Claude code sessions that we kept in our data. Sweetchat is broader than that and involves a couple of other coding agents. But I think I can say that all our data are cloud code Sessions using Opus 4.6.

Speaker A: Have you seen any evidence that changes via uh, API usage experience, uh, similarly dramatic variation, uh, in performance.

Speaker B: Oh, fascinating to kind of control for a lot of that product level stuff. All the prompts that are hidden from us, uh, all of those affordances, I don't know. But that's a nice thing to think about because it gives us a, uh, m. More things that we can control for and more things that are knowable.

Speaker A: So kind of in parallel to the model evolution, there's also evolution of the user. You've alluded to this a little bit about kind of your concept is that most AI users now are experts. Uh, talk a little bit about the role of expertise. I think this is also kind of echoing back to our conversation about DSPY and like prompt optimization. You know, you've done some research into how folks are using these models and the role of, you know, AI fluency. Tell us about that research.

Speaker B: Oh yeah, first I should say so, the distribution of users across expertise levels. I'd. So I guess the nuanced picture I'd offer is that the people deriving a lot of value from AI in the current moment tend to be experts. It must be the case that most users of AI are beginners just because the numbers are so large and expertise can't be that widely distributed yet. And that's a very interesting thing because I think probably most things are getting designed for those experts implicitly or explicitly. But for the whole economic picture to work out, many more people need to derive value from this via one avenue or another. And so that does shine a light on this expertise thing as a real factor. And the headline result there actually builds on something that anthropic did they have this AI fluency index. And their core observation in that work is that experts display an augmentative style. They iterate with the AI, they push back, they complain, they change their requirements. It's a really a collaborative mode. Whereas novices, low fluency users delegate. So they Trust in the AI. They let it do its thing. They accept the responses uncritically. And our contribution is to just show that this is a causal factor in success with these products. Right now. Experts can do harder things more reliably as a result of all that friction. They introduce all that pushback, whereas novice users, they accept, but they end up accepting the wrong thing. And they're not able to level up from the basic tasks that they think to start with. That's obviously significant. And it feels so tantalizing because pushing back is a natural human behavior. I feel like we could encourage everyone in the world to do this. We probably need to get them out of the mode of thinking. It's a super intelligence. You should just trust it. That has been the narrative for a while. What we're seeing in the current moment and possibly for the foreseeable future, is that you got to complain, collaborate, introduce yourself, push back. All that stuff that I think we do, that we take that for granted, right?

Speaker A: Yeah, yeah. And so, uh, from a methodology perspective, how did you approach exploring this?

Speaker B: We built on the work that Anthropic did, um, which they set up a nice framework with some independent research who were doing this kind of usability stuff. And we just have an annotation protocol. We can talk in detail if you want about this. But at Bigspin, we have lots of these best practices around having language models, essentially collaborate on annotation projects to kind of triangulate on the truth and factor out their individual biases. So we do that stuff and we apply all these fluency markers. And then separately we do a thing of estimating task complexity and looking for signs of visible and invisible failures. And so it's the connection between the fluency markers and the, uh, task complexity success metrics that was our contribution there. And that's where you can see high fluency users are the ones doing harder tasks. Paradoxically, there's more signs of failure for them, uh, because they complain, they push back, they're trying harder things, but as part of all that friction, they're successful with harder things as well.

Speaker A: And if you were to try to apply this insight from the perspective of someone in an organization that's trying to help or guide their organization to be more successful with AI, uh, like what do you think are the key lessons of, uh, this fluency work?

Speaker B: If it's an org that's just starting out, it wants people to figure out how this could be part of the organization's mission, it would just be that pushback message. And you could do an experiment where you interact with it about something where you're a world expert, we're all an expert in something. Engage in a discourse with one of the best models about something you're an expert in and see how often you feel you have to push back. And the same could be a kind of a lesson. You say aha, uh, for other spheres where I don't know the answer, it might be just as errorful. That could be a good visceral thing if the org is very far along. I think the main thing to do right now is to have a team of these LLMs interacting to improve things. For example, at Bigspin, I didn't set this up. Our founding engineer is very future forward on agents. And um, he's incredible at this. And when we do PRs now, the first round of review is the agents all interacting, collaborating, disagreeing. They do the first round of comments, they do the first round of code updates. Only after they've resolved things do we look at a pr. So the final human stage should be very high value and the agents did all that work. But when you have one agent do it, they often just reinforce themselves and you don't get good outcomes. It's that team of rivals thing that is transformative.

Speaker A: You know, I think it's interesting because, you know, on the one hand, like, of course that makes sense, but on the other hand it, it, there's something, you know, it also implies that you shouldn't be using these things in areas where you don't have enough expertise to evaluate the answer yet. That's where you most need the assistance, the support. Um, so.

Speaker B: And again. And um, this is a little bit worrisome about the overall narrative around AI. The place where we can get around this is with software development. Because let's say that I'm trying to accomplish something in a language that I don't know how to code in. I can have the agent do work for me because probably in the end I can run the program and look at the results. And that's what mattered to me, is that I run the results and I see. And if I don't see what I want in it, I can complain and we can iterate that verification step. That doesn't imply I have comprehensive knowledge, it just implies that I know what I want to see in the end is so critical. And I think this is a causal factor in models being so good at coding because it's like the ultimate verifiable domain for them. But as soon as we leave that and go even into something like the legal realm where the requirements are strict, but they're not codified in code and they have ambiguity about them. This whole picture falls apart and you then are back at what you just said, which is this awful kind of paradox is like, yeah, use AI, but in the end, unless you're expert enough to evaluate every single one of its responses, you might be in real trouble. I don't know how to get out of this, because the verification step is like, we go to trial, but this is very consequential.

Speaker A: That's expensive. Yeah, that's funny. I mean, it does make me think a little bit about, you know, some of the types of errors that we're trying to avoid are factuality. And, you know, there is a temptation to say, well, let's just throw more tokens at it. Like, I'll have a critic model that, uh, evaluates everything that the, you know, is generated by the primary model. But then you go back to my observation that these models tend to correlate, uh, and their responses as well. Um, yeah, it's super interesting.

Speaker B: That's a good point. Yeah. From my picture, we want real diversity of perspectives. This is just like red teaming for humans. This is most successful when you have a really diverse team of people who think creatively and differently. And if every one of the members of that team is thinking in a homogeneous way, they miss all of the crucial things. Same exact issue. If all the code review agents are biased in the same way, they will miss exactly the same class of bugs, and then we're all sunk. Yeah, I don't know how you'd encourage this diversity in the ecosystem. We're probably, as you say, converging towards some kind of one model. Um, but I think for my picture, we need diversity. Yeah, we got to keep those open weights models going or something, because they're the weird players in the space.

Speaker A: For sure.

Speaker B: For sure.

Speaker A: So we've talked about, uh, efficiency, interpretability, tokenomics, uh, fluency. Yeah, you're involved in a lot of different research directions. Excellent, excellent.

Speaker B: Where.

Speaker A: What's next for you? Where do you see either. Where do you see this all going? Kind of externally, but also like, where is your research going?

Speaker B: Yeah. Ah, this is great. Um, and as I said before, we're trying to think in weird and creative ways about what the future could hold. And I encourage my students to do this. And they're smart, so they say, all right, Chris, I'll think along those lines. But what's your answer to this question? So I do have an answer, and it's really shooting for the moon here, which would be what about the architectural innovation that would upend the whole story around the stack transformer and the way we need to do data center build out to even get incremental gains in performance that could be upended and it would come from some very innovative thing around maybe recursive use of the building blocks that we've got. So architectures, we should think, and when people say oh no, we don't need more architectures, the transformer is good enough, that's where we should push back as academics doing something more clever and more scrappy that could change the world. And the other one is thinking in the interp space much more about data. And that's just because I want to tell the true story of how we go from data to model capabilities. But it also checks a box for me on connecting interpretability to safety. It has been hard for me to connect those two things. We have found some ways to do it, but it's not a slam dunk as a narrative, even though it's the dominant narrative. But I will say that when we get into things like data poisoning from innocuous examples, this is probably a growing societal concern. There is evidence that with very few examples planted in a pre training data set, you can have a significant influence on the outlook and preferences and quirks of the final model. So can we detect those examples? What's the nature of those attacks? How well hidden could they be? What's the smallest number of examples and why does it happen? These are all going to be very pressing questions. And so again it's just a data oriented question that's very alive for me in the current moment.

Speaker A: On the architecture front, are there, is there research that you're seeing or doing that is, you know, as yet under the radar that you think is, you know, promising and, or underappreciated?

Speaker B: I think, uh, you had my student Julie Colini on uh, and she is an advocate for byte level models, essentially tokenizer free models. I think that's a big part of the future. It's a critical thing if you want to have truly multilingual models that are also equitable in terms of how many tokens they charge us for. Getting back to that earlier theme, but also Julie's perspective is that this is speculative but I think there's something to this, that it's a kind of inference time scaling because you do more compute at test time, um, because you have more tokens and therefore more opportunities to build on interesting things. So that could be a big part of the future. And the other one would be recursive architectures, as I said. But if you want to go all the way out, you could think, why do we always assume we're going to do gradient based learning? There are lots of alternatives to that and nobody is exploring them because everyone takes it as a truism. We're all in our very narrow row here without even really realizing it. Who knows what's outside in this garden. It's very risky as a research bet because only one in a thousand of these ideas will pay off. But what's the point of being an academic researcher if you're not going to take that kind of risk? That's what we're positioned to do.

Speaker A: Well, Chris, thanks so much for jumping on and sharing a bit about what you're working on. It's, uh, very cool stuff.

Speaker B: Thank you. What a wonderful conversation to get. Gave me lots of new things to think about.

Speaker A: Awesome, awesome. Thanks so much.