Hidden Layers · 2025-07-16 · 28 min
Key moments - from our scoring
Substance score
66 / 100
Five dimensions, 20 points each
This monthly decoded episode covers major AI developments across research, video generation, and copyright law. Ron Green and his Kung Fu AI colleagues dissect Apple's flawed 'Illusion of Understanding' paper, which tested reasoning on Tower of Hanoi puzzles but was hamstrung by insufficient context windows and punitive evaluation metrics - ironically critiqued by OpenAI's O3 model itself. Michael Wharton explores Meta's Fast and Simplex technique, which modifies transformer attention mechanisms to shift scaling law exponents, potentially reducing token requirements for training. The team then surveys a 'Cambrian explosion' of video generation models including Black Forest Labs' Flux (now open-sourcing Flux Pro), Vue 3, Runway's Gen-3, and emerging tools like MultiTalk for multi-speaker video and Blender Fusion for 3D-grounded editing - though physics consistency remains a challenge. On the legal front, federal judges in Anthropic and Meta copyright cases ruled that training LLMs on copyrighted material constitutes fair use, though plaintiffs face high evidentiary burdens proving market harm. The episode concludes with CMU research showing recurrent neural networks handle longer sequences better by preserving hidden state across training batches.
The paper had a fundamental context window limit: 15 disks require 32,767 moves to list out, and 20 disks require over a million moves, exceeding the 64,000 token allowance, making failure inevitable regardless of reasoning ability.
According to the federal court rulings, using copyrighted works to train LLMs is transformative because models learn patterns to generate new text rather than replicate the originals, analogous to how students learn from books without violating copyright.
It adds a K-prime projection to create 3D attention tensors comparing vectors with other pairwise comparisons, changing the scaling exponent and reducing tokens needed to achieve the same training loss.
They struggle with physics-constrained tasks, such as accurately simulating ball trajectories, water dynamics, and other real-world physics phenomena in generated videos.
By not resetting the hidden state between training mini-batches, models explore a more diverse state space and better generalize to inference sequences longer than training sequences.
Our reviewer’s read on each dimension, with quotes from the episode.
The episode delivers multiple substantive technical ideas (Tower of Hanoi evaluation flaws, scaling law exponents, state space model improvements, video generation progress, copyright fair use rulings) with concrete explanations. However, several segments rely on summaries without deep drilling - e.g., video models are listed with one-sentence descriptions rather than explored in depth, and some discussions lack follow-up rigor. The copyright section is well-structured but somewhat expository rather than analytically novel.
everything I just gave you. All of those critiques and more were generated by OpenAI's O3 model when given the paper
what they did was they took an old, I would say under explored technique that came out in 2019 that's sort of uh, a new flavor on transformers and tested how that scaled relative to our modern dot product attention. And it seems like it actually changes the exponent on the token budget part of the scaling law
The hosts surface some genuinely interesting critiques of the Apple reasoning paper (context window limits, scoring bias against abstraction) and the O3 evaluation is a nice meta-observation. However, much of the content is reporting on published research and industry announcements rather than generating original frameworks or contrarian takes. The copyright analysis follows standard fair use doctrine without pushing beyond existing legal commentary (even citing Ben Thompson). Video model coverage is largely descriptive enumeration.
the evaluation was literally publish uh punishing abstract reasoning the very thing they're trying to measure
there are just basic, simple things right now with an AI that we haven't tried because there are so many of them to try. Right. Like, that's the reason I'm so optimistic that we're going to see so much progress in the near future
The co-hosts Dr. Zzai (distinguished engineer) and Michael Wharton (VP of Engineering) both appear to be practitioners with credible technical depth based on their specific knowledge (scaling laws, state space models, recurrent neural networks, architecture details). However, they are company-affiliated (Kung Fu AI colleagues), not independent operators from the companies being discussed, and there is no external guest with direct experience shipping the featured products (Apple, Meta, Anthropic, Black Forest Labs). The panel is competent but somewhat closed-circuit.
my co founder and distinguished engineer, Dr. Zzai and VP of Engineering, Michael Wharton
I would say it's kind of an unfair comparison because they compared the compute requirement of their modified windowed technique to traditional dot product attention, which I would say is not fair because you can still do the same windowing with traditional dot product attention
The episode provides concrete details on several fronts: Tower of Hanoi disk counts (15, 20 disks = 32,767 and 1M+ moves), context window limits (64K tokens), scaling model parameters (1B, 2B, 3.5B), video generation model names (Flux, Vue 3, Sora, Seed Dance Pro, Hiloa 2, MultiTalk, Wonder Play), sequence length scaling (4K trained to 128K inferred, 256K tested), and post-training budget (0.01-0.02%). Fair use ruling details are explained. However, critical numbers are missing: no specific model sizes for comparison, no timing data, vague references to 'a couple weeks ago' and 'just recently' for announcements, and video quality metrics are purely qualitative ('impressive', 'great'). Copyright case citations lack judge names or docket numbers.
once you get to 15 disks there are 32,767 uh, moves. But once you're at 20 disk it's over a million moves
they looked at 1 billion, 2 billion and 3 and a half billion active parameter mixture of extra expert models
The host (Ron Green) asks clarifying questions and seeks deeper explanation ('Hold on, I don't understand that. So you would go into a little bit more color there.', 'is it um, because you go from quadratic to cubic'). However, follow-ups are often brief and sometimes affirming rather than probing. There are few moments of productive disagreement or sharp pushback. Michael's cautionary note on the Meta paper's comparison ('I think it's just a little funny the way that they make the claims') is acknowledged but not pressed. The copyright discussion lacks legal expertise to challenge the framing, and video model coverage accepts the guest's enumeration without comparative analysis or skepticism. Conversational pacing is smooth but somewhat deferential.
Hold on, I don't understand that. So you would go into a little bit more color there
is it um, because you go from quadratic to cubic with this higher dimensional tensor. But do you remove like instead of having key query value matrices, do you not need all of those now?
Computed from the transcript - who did the talking, and the words that came up most.
In this episode of Hidden Layers: Decoded, Ron Green, Dr. ZZ Si, and Michael Wharton unpack July’s biggest AI developments - from flawed reasoning tests to surprising training breakthroughs. Apple’s “Illusion of Thinking” paper draws sharp critiques - from both humans and language models. Meta revives a forgotten 2019 attention mechanism to reshape scaling laws. Video generation tools from BlackForest Labs and others hit new levels of realism and interactivity. Federal courts weigh in on Anthropic and Meta’s use of copyrighted training data. A one-line tweak in training recurrent models dramatically boosts performance on long sequences. Cloudflare announces it will block AI scrapers by default - though it might be too late. From Transformer alternatives to copyright battles, this episode dives into the fast-moving intersection of AI research, engineering, and regulation.
Transcribed and scored by The B2B Podcast Index.
Speaker A: Foreign
Speaker B: welcome to Data and Layers, where we explore the people and technology behind artificial intelligence. I'm your host, Ron Green. We're back with our monthly decoded episode where we cover the most important news and announcements in artificial intelligence. There's news from Apple, Black Forest Labs, Anthropic, Google and more. To help me cover it all, I'm joined by two of my amazing Kung Fu AI colleagues. As always, my co founder and distinguished engineer, Dr. Zzai and VP of Engineering, Michael Wharton. ZZ Michael, welcome back.
Speaker A: Thanks for having us.
Speaker C: Thanks.
Speaker B: Okay, I want to kick off with uh, the Illusion of Thinking paper that came out of the Apple, uh, AI Research Group. This came out about a month ago. Um, it essentially used um, the problem, the Tauru Habnoi problem, uh, which is like an old puzzle and I'll give you a description of that in a second to try to measure um, the reason ability of these models. And it came back with a pretty negative result for whatever reason. When I heard about the paper, I was pretty skeptical right off the bat. Like it just didn't seem like it was accurate. Um, and there's been some pretty interesting critiques of it. So let me set things up. So the paper is called the Illusion of Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity. So the Tower of Hanoi problem, briefly is a problem where you have, you have little discs and you have uh, three basically um, pegs and you have to, each disk is smaller than the one, um, uh, uh, below it. And you have to move the disks from one peg to the other. But you can only move one disk at a time and you can never have a bigger disk on top of a smaller disk. Okay, so the challenge here is that as you add disks, the problem becomes pretty complicated. It grows exponentially the number of steps you need to take. There are very simple recursive computer science algorithms to solve this problem that'll work on any size, any number of disks. But what the researchers did here was they said you cannot use an algorithm. You have to literally describe the steps one by one to see how it could scale from a reasoning perspective. They found that um, once in the number of disks got beyond a certain size, the model's reasoning abilities essentially completely collapsed and it got 0% of the um, the puzzles correct. Seems a little odd, right? So um, uh, I, I, I read the paper, went and did a bunch of research on sort of public criticism and I found some interesting things I want to share. So the first one was, turns out that because it's got this exponential growth once you get to 15 disks in the normal, the human puzzle will have 3 or 4 disks if you see them in the real world. But once you get to 15 disks there are 32,767 uh, moves. But once you're at 20 disk it's over a million moves. So you'd have to list out a million moves where they were asking it to solve those level of problem, that level of disk complexity. Turns out Even with the 64,000 token allowance they gave it, there wasn't enough context window. I mean literally it couldn't spit out the right answer like liter, um, and so the collapse was guaranteed there wasn't enough context window which is just kind of crazy. Anything above 14 was guaranteed to have collapse fundamental problem right there. Second one was the way they did the scoring format. Um, if the model output like a concise algorithm and said oh it's a recursive ah, solution. You move n minus one disk here and you basically solve it this way they would mark that entire solution wrong. They would say no, you have to lift the steps out. Interestingly I think you can say that the evaluation was literally publish uh punishing abstract reasoning the very thing they're trying to measure. Right. They're trying to measure reasoning and they're sort of punishing this, this higher level abstraction.
Speaker A: Well why, why the narrow focus on the Tower of Hanoi problem? Also because I feel like these reasoning benchmarks are meant to be diversified tests of reasoning and that's a great question.
Speaker B: This I think it's, I think that right there alone maybe set off my um, my suspicion um initially because I was like that is just such a, you uh, know, such a very narrow um, task and it's in its um. Once you understand the trick and it's recursive, I mean it's actually not that, that complex. So I don't really know. I know there were some other things they were trying trying but that was, that was the main focus and I think they thought it was something that they could scale the compl. So you could start small, it succeeds and then you would see could it continue to reason as the complexity increased. And then the last thing, there were many of these but many of the critiques but the last one I'll say was they used a temperature of one for sampling. That means that they had maximum variance M. Right. And um, they essentially you know quote entangled luck with skill. Right. And a better approach would have been use either deterministic decoding or a ah, large sampling number to kind of average thing. Okay, so I won't go through the rest of the critiques, but this is the best part about this, and this is probably the reason I'm mentioning this here. Today is everything I just gave you. All of those critiques and more were generated by OpenAI's O3 model when given the paper. So there's this. I just think it's amazing. They're literally the paper that purports to find reasoning shortcomings. There were all these reasoning shortcomings found within the paper. Buy a reasoning model. So I mean to me it's just like amazingly. Hello.
Speaker A: I love it.
Speaker C: I love it.
Speaker A: That's amazing.
Speaker C: Yeah.
Speaker B: Ah, I really do too. All right, Mike, I'm gonna throw to you.
Speaker A: Uh, cool. Well, I was gonna talk about this paper called Fast and Simplex. It's from Meta and I think they just poached some people from Apple like all over the place. Yeah, that's a whole lot of sense. There'd be a brain drain.
Speaker B: The whole super intelligence group that they're
Speaker A: standing up, they're poaching people, Alexander Wang stuff. Yeah. Um, but anyway, yeah, so this is a, this is from the Meta lab and the whole idea here is to focus on the scaling law problem and to just cut to the chase. Basically what they did was they took an old, I would say under explored technique that came out in 2019 that's sort of uh, a new flavor on transformers and tested how that scaled relative to our modern dot product attention. And it seems like it actually changes the exponent on the token budget part of the scaling law and it makes the training process more token efficient.
Speaker B: Okay, so it changes the slope of the curve, that log loss curve.
Speaker A: Yep.
Speaker B: Okay. Yeah.
Speaker C: So you need less data to achieve the same accuracy.
Speaker A: That's exactly right. And yeah, for folks listening, I know we're all probably pretty familiar with power laws, but basically the idea is that to achieve a given training loss, it's a function of both the model size, the number of parameters and the amount of tokens that you train the model on. And there's actually a compute optimal combination of those two that depends on, you know, whatever your starting point is. That was in the uh, that chinchilla paper a couple years ago.
Speaker B: Yeah, legendary.
Speaker A: Very, very cool paper. Um, but so there's this growing body of research that shows that most of the changes that people tweak with modern architectures, training techniques, optimizer variants and things like that, they only really shift the curve. So unless you can actually fundamentally change the shape, then the payoff of increasing scale is pretty much the same even if you come up with some clever optimizer or something. Um, but what they did was, you know, traditional dot product attention, you have queries keys, you combine those, you get your attention matrix and you combine that with a value matrix. Well, instead of getting attention matrix, they basically just add another projected set of keys. They call it K prime and they combine it to create an attention tensor. So it's three dimensions now.
Speaker B: Mhm.
Speaker A: And what that allows you to do is to capture a lot more complexity. So you know, with pairwise comparisons you get from attention, it's vectors compared to vectors. Now in this formulation you're comparing vectors with other pairwise comparisons of vectors. Okay, that makes sense.
Speaker B: Yes, absolutely.
Speaker A: All. Yeah, it's all. All good so far.
Speaker B: All good so far.
Speaker A: Okay, cool. Um, so that's kind of scary when it comes to scaling.
Speaker B: We're going the wrong direction there already.
Speaker A: Yeah, like state space models, you know, they're trying to move us in this frontier to stuff like that's more simple.
Speaker B: Right.
Speaker A: But these models are going more cubic. So they have to of course invent all these techniques to make it compute efficient, like sliding windows and all sorts of stuff. The cool thing is, you know, I would say it's still pretty underexplored, but they looked at 1 billion, 2 billion and 3 and a half billion active parameter mixture of extra expert models. Um, and if they empirically fit this scaling curve to those and they use this modified technique, they do show that they can improve the scaling exponent.
Speaker B: And is the amount of compute that is being done, is it um, because you go from quadratic to cubic with this higher dimensional tensor. But do you remove like instead of having key query value matrices, do you not need all of those now? So you save compute or is it just. No, there's more compute but you get more bang for your buck for that computer.
Speaker A: There's still um, you know, I actually say. Would say it's kind of an unfair comparison because they compared the compute requirement of their modified windowed technique to traditional dot product attention, which I would say is not fair because you can still do the same windowing with traditional dot product attention. But they uh, I mean, I guess the compute budget with it is just it there. They make a claim that it's improved over this quadratic.
Speaker B: Okay.
Speaker A: Scaling. But I, I think it's just a little funny the way that they. Okay. Make the claims. I'm still kind of thinking through it.
Speaker B: Okay, okay.
Speaker A: All right. Um, super interesting.
Speaker B: Yeah.
Speaker C: If you can change the exponent then
Speaker A: you know that's not only payoff is huge.
Speaker B: Right.
Speaker A: If it's actually.
Speaker C: Yeah.
Speaker B: Even if, even if it's a little bit and we've got to figure out something that's not quadratic net net long term.
Speaker A: So. Yeah, it's not going to work.
Speaker B: Yeah, exactly.
Speaker C: Yeah. Super excited about the data, ah, data efficient techniques for training.
Speaker A: Yeah, me too.
Speaker B: Yeah, 100%. Zezi, what's your first topic?
Speaker C: So I've been watching uh, video generation models and image generation models for a long time. I think you know since uh the SARA announcement came out, I think it's over a year ago now, um, there has been continuous um, progress in a field, um, in both commercial and also open source communities releasing new video generation models. But I think recently I really saw kind uh, of like a uh, Cambrian explosion of such models. And I uh, want to run through a few uh, either papers or announcements, uh, maybe one short sentence for each just to uh, give you a sense of what's happening in the last few weeks. So uh, the first one is Black Forest Lab. They have the Flux models. They have several versions of the models and one of them is called Flux 1 context. It's kind of similar to GPT4O type, uh, of capability where you can do pretty fine grand control of image editing. So uh, that model used to be not open source, it was proprietary. But they are kind enough to open source it just recently.
Speaker B: Oh wonderful.
Speaker C: And its quality is great.
Speaker B: And I know they have a lot of their Flux models come in different sizes and the smaller ones there are multiple are open source.
Speaker C: Right.
Speaker B: Or at least they're open use the weights are. The weights are available.
Speaker C: Yeah, exactly. I think most of their models are open source. They have some you know, higher tier models that are proprietary. But this context just went uh, from proprietary into open source, which is very exciting. And then in a commercial uh, APIs we also Vue 3 and uh, it's very um, impressive uh, with uh, audio plus video, uh in sync, there's Hiloa 2, there's seed dance pro and clean, uh version 2.0 and all of them I test them and they're very impressive. Compared to just a couple months ago,
Speaker B: uh, what's the biggest shortcoming you still see with these models?
Speaker C: That's a great question. So although they are very impressive however, uh, when people are testing it to generate sports videos, for example tennis ball trajectories, playing tennis ball, you can still see uh, uh, phenomena of the ball that's not uh, complying with physics, um, people have been using such, uh, models to generate, uh, um, uh, water diving of cats.
Speaker B: Yeah. Oh yeah, and I saw the dog on the diving.
Speaker C: All the fun videos, super fun to play with. Um, yeah. And then, uh, the third one is multi talk. So that model is really interesting. It just came out I think last week. So it allows you to upload one image and two audio tracks. Um, so that if that image had two person, then you can generate a video with two person talking. So like, if you want to generate a podcast, uh, with two persons, now you have a tool to play with.
Speaker B: Oh, nice.
Speaker A: That sounds scary.
Speaker C: I know.
Speaker A: It's like totally lowers the barrier to entry to, you know, make videos of people.
Speaker B: Well, when they get to three, we'll, we can save a bunch of time.
Speaker A: I know. Yeah, we can totally automate this. This sounds great.
Speaker C: Here goes my weekend. Um, and this one is open source too. And then uh, the fourth one is called Blender Fusion. It's interesting because it's not only uh, video, uh. Oh, oh, sorry. So that one is actually 3D grounded visual editing. So you can use Blender and it uses Blender as a verifier when generating.
Speaker B: Hold on, I don't understand that. So you would go into a little bit more color there.
Speaker C: Yeah. Um, so I think this one, you know, it generates uh, visual contents but um, uh, not just using Transformer, uh, model and you know, uh, using the model weights, but also has a tool calling using Blender to kind of as a verifier to bring the kind of grounding into the generation.
Speaker B: Okay, that makes a lot of sense.
Speaker C: Process. Yeah, I think that um, takes us to the promising direction of uh, grounding the generation to have more physics kind of uh, reality. Um, and then the last one is called Wonder Play. And uh, this is also a fun one. So it allows you to apply a 3D action, uh, on the virtual world. So like for example, you can say, drop this rubber, ah, duck onto the water and see the effects. Uh, so then it will generate you upload an image, uh, of a rubber duck and then it will uh, see
Speaker B: the effects and you'll see the ripples in the water and everything. Have you guys seen the uh, I think the Star Wars Stormtrooper video blog videos yet?
Speaker A: Not yet.
Speaker C: What is that about?
Speaker B: So I think because they're wearing masks, they're wearing helmets and you don't have to worry about lip syncing things like that, they're really, really good. So they're essentially um, like video blogs from the first person Perspective of stormtroopers and they're holding like the camera out,
Speaker A: you know, that's so funny.
Speaker B: And they're like, you know, I'm on Hoth and they're talking about how rough their day was. They're really, really well done. I mean, check it out.
Speaker A: That sounds great.
Speaker B: It was the first, it's really, truly the first time. I think I watched like a 5 minute AI video and was really impressed and like with the quality of the content and the humor. So loved it. A whole bunch of um. Okay, so all right, next topic. This is a bit of a big shift, but I want to talk about copyright law because we've been talking for a long time about when are we going to have legal rulings on uh, all this data that's going in, copyright data that's going into training language models and is that considered fair use? That's the term in America for um, sort of the standard for when you use copyright material in some way. And so to set the table, I did a little bit of research to make sure, ah, you know, everything makes sense here. So fair use is the concept for copyright law, but it's judged on an individual basis. Meaning there's not a broad standard. That's a doctrine. It's not a doctrine. It's a set of rules. And the courts look at four different things. They look at the purpose and character of the use. So was it transformative? You can't take somebody's copyright material, material and just regenerate it, you know, as is, and then say that's fair use. You have to have some sort of transformative use to it. Merely, you can't m. Merely copy the original number two, they look at the nature of the copyright use. Was it factual information or was it more likely, um, or was or was the published work more likely to be fair use if it's in the creative domain. So meaning if um, it's just factual information, it's, it's harder to just, you know, demand a copyright on that. The amount and substantiality of the portion used. If, if somebody published a five second video and use four seconds of it, it's kind of hard to argue that you are not just using all of it. And there's this much transformation. And then lastly, this is pretty interesting, the effect on the Amar on the market. Right. Uh, is this going to somehow negatively affect the market of that copyright holder? So there were two rulings recently. They came just days apart, two weeks ago. So Anthropic was sued and a federal judge allsup Ruled actually mostly in favor of Anthropic. And they said that Anthropic, all of the copyright books that they purchased, yeah,
Speaker A: they like literally purchased them, um, cut out the pages, scanned them. I read this the other day, it's crazy.
Speaker B: You nailed it. They literally purchased, um, just an enormous number of books, ripped out every page, scanned them, ocr, scanned them for training. The judge ruled that was fair use and because he felt it passed all four of those, uh, measures we just talked about, including transformative use. Um, interestingly, Anthropic also pirated a bunch of books, I think roughly like 7 million books. The judge says this ruling does not apply to those and they will have to go to that case on their use will still uh, be heard. But the really interesting thing about this is that there's a couple of quotes here from this judge that I want to read because I think it's really interesting. Get your head around where he's coming from. So, um, when you're training an LLM with the copyrighted work, the question is fundamentally transformative. And the judge says, quote, the purpose and character of using copyrighted works to train large language models to generate new texts was quintessentially transformative. Like any reader aspiring to be a writer, Anthropics large language models trained upon works not to race ahead and replicate or supplant them, but to turn a hard corner and create something different. If this training process reasonably requires making copies within the LLM or otherwise, those copies were engaged in a transformative use. And that is basically his ruling. He essentially said anthropic put in a book, bunch of safeguards on the generative side, on the inference side to help to prevent it from just replicating content verbatim and reproducing copyright works. And so he essentially said, yeah, this is a transformative use. Um, the authors argued that the um, that the work was um, essentially going to affect their ability from a market perspective, from a competitive spec perspective. And the judge. I've got to read this. This is just awesome. The jud judge essentially said, um, that is like arguing that school children learning to read, uh, your book is anti competitive because they could grow up and write works of fiction and you'd have to compete with them. And he just basically dismissed that out of hand. Okay, so just the very next day a different federal judge ruled on the same question, but this time with meta. Interestingly, he basically came out with the same ruling that um, he considered these the copyright work fair use. But this is crazy. He all but said I'm ruling for the defendant here because the plaintiff did a really bad job of arguing their case and he all but begged them to come back and make a better case. And foundationally he said that in the market competition front, that the, uh, I think it was the Authors Guild that was in the lawsuit, they were saying that this is going to hurt their market of competitive capabilities. And the judge said that's speculative. You have to come back with examples of that. And he went so far as to say that he essentially disliked this ruling, but that judges are forced to rule on the cases argued, um, and from the evidence, not what they think should have been argued. So, um, it's very, very likely that, uh, the plaintiff will come back and um, resubstantiate their case. But interestingly, um, the podcaster, uh, Ben Thompson, he has a great take on this, which is they're in a really tough place, uh, from a market competitive perspective because you essentially have to show that purchases that would have been made of that author's work didn't happen. So you have to kind of prove a negative in a way. And, uh, that's a really, really high bar. So the first two examples of fair use case that we, that we've gotten at the federal level essentially ruled on the side of the AI, um, companies with a bunch of caveats. So I think, I'm not sure we got a lot of clarity, um, about where this is actually going to go long term.
Speaker A: Doesn't sound like it.
Speaker B: No, it really doesn't. It really doesn't. Okay, Michael, I'll throw back to you.
Speaker A: All right. Um, man, that's great. I'm gonna have to keep thinking about that. Yeah. Uh, so there's a new paper. I'll keep this brief. Luckily it's pretty easy to summarize. There's a paper from cmu, I think this is at icml, in addition to the one I talked about earlier. But it talks about this idea of state passing. And the idea is. Well, I guess the problem they're trying to solve is that with recurrent neural networks there is a strong performance drop off whenever you increase input sequence lengths beyond what it was trained on. Okay, so if you train on a model on 4000 token input sequence lengths and then you try to do 128,000 sequence length inferences.
Speaker B: And this is, this is for recurrent neural networks.
Speaker A: For recurrent neural networks, that's right.
Speaker B: Okay.
Speaker A: Um, so all they did. This sounds so stupid simple because it is. They just didn't reset the hidden state between mini batches during training. Ooh, so they're totally unrelated to each other. It's not. That makes no sense jump in context. And their, their theory behind it is that there's a bunch of. That the state space is very underexplored if you just keep resetting it, but you improve the diversity of the, the uh, state space that it explores during training by this.
Speaker B: Wow.
Speaker A: The crazier part is that it only takes like point. I uh, think they said it was 0.01 to 0.02% of your training budget to do this as a post training step to elicit this kind of behavior where that performance cliff doesn't happen anymore.
Speaker B: That is absolutely unintuitive to me because it feels to me like. It feels to me like you would be, um, maybe the analogy I would use is you, you translate a sentence into. From English to French and then you start off your next translation with that French sentence already kind of in place. A little bit like you, um, it's a little apples to oranges, but that just doesn't make intuitive sense to me.
Speaker A: I mean I had the same thought whenever you. I think it was last episode you were talking about the um, you know, using these verifiable domains, but you give it the wrong answer and it still improves model performance.
Speaker B: Right.
Speaker A: You know, there's a bunch of weird wacky stuff going on, but I mean the, the performance results undeniable. They showed they were looking at Mamba, as you might expect.
Speaker B: Okay.
Speaker A: And there's um.
Speaker B: You know, okay, that, that, that answers my question. I was like, why are they using a recurrent neural network in the first place?
Speaker A: State space models.
Speaker B: Yeah, state space models. Okay.
Speaker A: Yeah, that's what they're focused on. And they showed that, you know, with this model, the performance drop off happens at that, you know, input sequence length, average input sequence length. Okay. And then the long tail, uh, like if you scale it up to, I think they looked up to 256,000 token input sequences drops down to like, like uh, 75% drop off of performance. Now with doing this post training technique with literally just 500 steps for the same exact model, it settles out to about a 40% drop instead. And that, that settling is very gradual. There's, I think it's figure six in the paper. If you want the, the one slide takeaway, you just watch it kind of just barely drop off versus that steep cliff from before.
Speaker B: Interesting.
Speaker A: It's a stupid, like you take out a line of code to do this. You don't add code.
Speaker B: So I, that was the other thing I was going to say. Uh, I literally, I was wondering, was this a mistake? Did they. Did they actually forget to reset the context and notice?
Speaker A: That's a brilliant thought. Yeah.
Speaker B: Um, to me, that's also an example, which I say this all the time is like, there are just basic, simple things right now with an AI that we haven't tried because there are so many of them to try. Right. Like, that's the reason I'm so optimistic that we're going to see so much progress in the near future, because little, tiny, simple things like that can have an impact.
Speaker A: Yeah, no doubt.
Speaker C: And it doesn't have to be, uh, to have a theoretical, uh, ground grounding or found. It can be just smart engineering. It could be a wild west.
Speaker A: Anything goes.
Speaker B: Yeah, exactly. Okay, I've got one more thing to bring up, which is, did you guys see the Cloudflare announcement last week that Cloudflare is going to, um, by default, now block AI companies from being, uh, able to access and index and search content, um, uh, without being paid, which is kind of interesting. One, Cloudflare has, I think they, you know, back 20% of Internet traffic or something like that, so that's not inconsequential. Um, two, I think this is a great move for Cloudflare from sort of a business perspective. It's another monetization stream. And three, feels about, like, five years too late. I mean, hasn't the entire Internet already been sucked into these models?
Speaker A: Um, you think so?
Speaker C: Yeah.
Speaker A: 15 trillion tokens if you include this or whatever it is.
Speaker B: I think it's a great idea. Maybe just a little late to the table, so we'll see. I don't know.
Speaker A: Um.
Speaker B: All right, guys, that was a great, great time. Thank you so much. Um, and I'll see you next time.
Speaker C: Yeah, it's a lot of fun. Thank you.
Speaker B: Thank you for listening to Hidden Layers. This series is hosted by Kung Fu AI, a management consulting and engineering firm focused exclusively on artificial intelligence. If you have any questions or thoughts about today's episode, or if you know someone we should feature, please visit us at Kung Fu AI.
Other episodes covering the same guests and topics, from across The B2B Podcast Index.
AI Is Designing the Next Cancer Fighter | EP.5383 / 100
Anthropic Code Leak: A Rare Look Inside Frontier AI | EP.5282 / 100
The "AI Bubble" Bubble | EP.5174 / 100
Did AI Kill Programming? | EP. 5072 / 100
Your AI Is Too Big, Too Expensive, and Probably Wrong | EP. 4978 / 100
