He's Building an AI That Can't Lie | Dan Klein

Speaker A: We are going to switch from the problem in AI is nothing works to the problem in AI is everything works. ChatGPT tells you something and it's always fluent and it's always confident whether it's right or wrong. Training these systems is becoming increasingly complicated and often what looks like an exponential curve just turns out to be the beginning of an S curve. And so what happens next is that we start to hit diminishing returns. And we are seeing that today we're seeing data walls and compute limits. The different aspects of intelligence have not been advancing equally. Reliability has not kept pace. To me, suddenly the most I could think of was how do you build a system that will not lie to you?

Speaker B: Can you say at a high level what your new approach is and how it's different than, uh, what the labs are doing?

Speaker A: One of the key things that we do to train our models is that we train them on.

Speaker B: You're listening to Gradient Descent, a show about making machine learning work in the real world. And I'm your host, Lukas Bewald. All right, I'm here talking with Dan Klein, professor of computer Science at Berkeley and serial entrepreneur, most recently working on Scaled Cognition, which is a company that helps make more reliable AI systems. This is a fun conversation. We talk a lot about linguistics and parsing that I worked on in a previous life when I was in a lab where Dan was the star grad student. I was a lowly first year research assistant. Um, but we get into how modern systems work and the drawbacks and some of the things that we might be able to take from previous versions of NLP into systems that we're building now. So you described the AI industry as built on jello. Do you want to expound on that?

Speaker A: Sure. Um, I think it's easy to lose track given how quickly and explosively the kinds of large language models that people are using today have burst onto the scene. It's easy to forget that they are sort of at the core, these probabilistic engines. They have at their core been trained to do next word prediction to produce plausible output. And the objective function is essentially just produce output that is indistinguishable from the truth. And it's very important. These are not truth engines, they are plausibility engines. And, um, I'm sure we'll come back to that.

Speaker B: Well, let's get right into it actually, because I feel like that's a little unfair. Certainly the pre training, um, optimizes for next token prediction, but I think there's a lot of effort put into Uh, a second step where a big part of it is that they're optimized for reducing hallucinations.

Speaker A: Yeah. And, um, I did say at their core twice, and the reason I said it twice was that training these systems is becoming increasingly complicated. I mean, this really is the, the story of any technology that comes in to artificial intelligence or really beyond. There are these super cycles in research where people have complicated systems, they've hit some wall, the systems are hard to improve. And then some new technology comes along and, ah, maybe that is large language models based on autoregressive training, transformers, large data. And this new technology comes onto the scene and it's almost like a silver bullet. It, in a very simple way is suddenly topping benchmarks and outperforming systems that are much more complicated, that were built on previous technologies. What then happens is we go into a phase where we double down on that new innovation and we scale it up and we really try to get as much out of it as possible. This often feels like an exponential curve, that this new technology, uh, even in its simple form is having such a big impact and it's just going to go to the sky. And of course the trees don't grow to the sky. And often what looks like an exponential curve just turns out to be the beginning of an S curve. And so what happens next is that we start to hit diminishing returns. And we are seeing that today. We're seeing data walls and compute limits and all kinds of reasons why that initial explosion of progress starts to hit diminishing returns on those specific methods.

Speaker B: And I think for you, one of the biggest issues that you flag quite frequently is hallucinations. Do you want to talk specifically about how you've measured that and why you think it might be a bigger problem than other folks would be aware of?

Speaker A: Yeah, I mean, uh, first, one of the things I'd like to say is just the word hallucination is it's a sort of a projective term term. Really what's happening is systems are making errors. So a, uh, sequence of tokens comes out and it turns out to be incorrect information. And we tend to call that hallucinations. And that term is being used more and more kind of broadly now, almost that anything undesirable that a system will do, you call it a hallucination. So that term is definitely undergoing some semantic broadening. But specifically, I think it's important when we talk about systems and we say, what's the difference between a system makes a mistake or it hallucinates or it lies in a human. These are different things really. A hallucination is just a mistake the system has made. And again, if you have a system which is just doing next token prediction, that system doesn't actually know as it's predicting those tokens whether they are right or not. And again, I haven't forgotten your point about there's more to training than just the autoregressive phase. But um, but systems like this, they're not metacognitive, they're not looking at their knowledge and making a sort of external decision. Do I know the answer to this question? Do I have this information? Do I know where it came from? What is the reliability of that? They're not doing that, they're just producing tokens. And sometimes they're right and sometimes they're wrong. So one way to look at this is every output is a hallucination. Some of them are right and some of them are wrong. I think where you start to be able to say something stronger, to say that a system is deceiving is once you start assigning in later training phases things like doing reinforcement learning. Um, you mentioned there are phases beyond the initial pre training. This is not something where there's just one simple answer. But one of the things you see, for example is reinforcement learning training. And this can take a couple forms. One kind of reinforcement learning training would be something like RLHF where you show humans choices and say which of these do you prefer? Well, the system will now come to produce the outputs that humans prefer. Well, do they prefer things that are factual or do they prefer things that make them feel good? And it's not at all clear that hallucinations are going to become less common. In fact there are some results that show hallucinations become more common as you go into these sorts of post training approaches. And I can give you a few reasons why that might be. One is imagine you have a shipping company and they've deployed an agent and it's backed by an LLM. And in addition to the training that has gone into that core model, it is also being reinforcement learned for some objective. For example, maybe it's being reinforcement learned to optimize the number of thumbs ups it gets from users. Well, it's operating and someone calls in and says where's my package? And that package has been, let's say it's been lost and the database does some tool call, there's a database that says this package has been lost. Well, what's the system going to do? Well, it can tell you your Package is lost, it can tell you it's coming tomorrow, it can tell you it doesn't know it's got choices. And what will it be rewarded for? Well, in this case it's probably going to get more thumbs up. Imagine if it tells you that the package is coming tomorrow.

Speaker B: Mhm.

Speaker A: Now this I think legitimately would qualify as deception. And anytime you're doing reinforcement learning, the system is optimizing the reward function. And that is always going to have some gap between the truth and what's being optimized. And that gap is going to increase the amount of hallucinations to the extent that it doesn't align with truthful behavior.

Speaker B: Well, we have some experience that at weights and biases and how are we building these customer service, um, systems. And of course it's going to be a really bad experience for a customer in the long run if something's hallucinated. So from my perspective, you would never want to just take the immediate customer reaction as the only thing that you're optimizing for. In fact, I think most people that would be doing reinforcement learning or even just evaluating a system like this would have a special check for is this accurate information that I'm getting back? And it's very hard to evaluate that in a lot of cases. But I think most real production systems do a fair amount of, um, checking and then try to put in reward functions where the most negative score you get is if you give plausible but wrong information.

Speaker A: Absolutely. And of course I'm giving a, um, sort of reductive caricature of an extreme case to illustrate how easy it is to get deceptive behavior because of reinforcement learning. People will of course design the reward functions to mitigate this and to try to balance truthfulness and other properties. But ultimately there's always going to be a little daylight between whatever you're optimizing and specifically the truth. And in my opinion, one of the things that we should be doing as a field is creating technologies that cannot lie to you so that you can look at a system. And I mean me personally, my mission here is to build technologies that can't lie. Cool.

Speaker B: Well, let's look at into that. But just before we do that, I love, it's such a gentle linguist phrase, uh, semantic broadening. Uh, I love that you put it that way. So I think in your view, hallucinations has maybe broadened semantically beyond what I thought it meant. Like to me, a hallucination at its core is something where it recommends a movie that doesn't exist or it Cites a paper that doesn't exist to me saying five plus seven equals 15 wouldn't be a hallucination. It'd be a different type of error. And I think that these plausible citations of information that it doesn't have totally makes sense, that it happens a lot when the training is predicting the next token and looking for plausible next tokens, and there isn't a step after that. Do you agree with my definition of hallucination, or do you really use hallucination and incorrect information interchangeably here?

Speaker A: I mean, if you're asking me do I think a good definition of hallucination would be crisply defined as sort of the injection of, let's just agree with

Speaker B: what we're talking about, I guess, you know, and I'm happy to.

Speaker A: I like your definition because I think it aligns well with sort of the pre LLM notion of what a hallucination is to a human, sort of, uh, having to do with perceiving something that's. That's not actually there or, or something like that. I do think in common usage, it is broadening. I think when a system says one plus two equals seven, people are starting to call that a hallucination. And I think that is increasingly any mistake that an LLM makes. The word hallucination is something that people reach for. I think we should be more precise. In particular, I think we should be very careful when we say things like error versus deception versus hallucination. These inhumans have implications for the context and the metacognitive status of the error. There's a difference between you have information, it's incorrect, but you say it believing it to be correct. That's making a mistake. Uh, that's different than, you know, you don't know the answer. You're in quiz bowl mode, and you're just going to guess. And that is really more. That's unusual in humans. We don't think of that as the normal kind of error that humans make, which is saying something that's false while being completely unaware that they're just making things up. People are making things up. They're doing it in a specific context where they're doing it intentionally. I think it's better to think of systems as more or less always being in that state.

Speaker B: What is that state?

Speaker A: The state of producing information without, um, metacognitive analysis of where that information came from and the sort of provenance and certainty of that information. When think about. If I ask you, what's the population of Berkeley? Well, a perfectly reasonable thing for somebody to Say is I don't know. Right, because you know you don't know. And if I made you guess, you would be like well if I had to guess. And then you'd be thinking a little bit about like well uh, like 10,000 is too small and a million's too big. And you would sort of be reasoning about this thing that you don't know. In its simplest form, an autoregressive language model is just going to crank out some numerical tokens that because of correlations with the word city and population might be of the right order of magnitude or they might not. Right. That's a very very different process. In their fundamental operations, systems are not checking do I have this information? Where did I get it? Have I preserved it and kept it intact? I think they should. So for example, at scalecognition we architect into the models in the first place the sort of information provenance. I think it's very important and people do this too. You can start to see little hints of systems becoming a little more metacognitive. And it's a broad term, it's just thinking about thinking. And you can see little hints of this in a system that does chain of thought where.

Speaker B: Yeah, I was going to say that actually places.

Speaker A: It allocates some tokens to a broad plan which it then elaborates. That's a sort of course defined planning. That's a kind of metacognition. In uh, a rag system you first get information and then you describe it. That is a multi stage step if you're doing tool use. But I would say that broadly speaking systems today are not primarily metacognitive. They're not primarily about actions and information and the provenance of those and the flow and integrity of those pieces of information. That is something that people are trying to retrofit on. As you said, we take systems, we bring in other systems to check those systems and so on. And that is a retrofit.

Speaker B: Well, retrofit seems like a pejorative. It might be fine to do it that way. I mean I guess I'm realizing that the way you're using LLMs might be pre. Some semantic broadening. I think when I think of uh, an LLM today, I think I view it as more than the model. It's kind of the model plus the whole agentic system built on top of it. That's answering my question. And I feel like there's. I can see it when I actually look at the trace of um, the reasoning steps where I can kind of see it doing Something that looks to me like metacognition. I mean, it's maybe even surprising that I can understand what these reasoning steps are. And it's not verifiable necessarily, but it looks to me like it's actually looking at its previous tokens and sometimes trying to say, ooh, I'm not sure if that's true. I better look that up. And often in coding, you can see this for a long time, the agents kind of looping and checking the code. I mean, in some sense, running a test on the code you generated is maybe a kind of metacognition.

Speaker A: Well, I think what you get in coding is actually, uh, that's a great point. You mentioned verifiability. I think the advances we've seen in coding and in math really are rooted in that verifiability, that the system can go off, it can hallucinate, it can make strange choices, but there is that verifiable signal. You didn't pass these tests or. No, Lean does not accept this proof. And you can try many things until something verifiably passes. And that, I think, is, at this point, I wouldn't call that whole conglomeration of pieces. I wouldn't call that just an LLM. It's an LLM embedded in this verifiable context. And that verifiability can be used during training, and it can be used at test, uh, time. Test, time, compute. We talk about reasoning models. It's a big category, but one big class of reasoning models is the models that try a lot of things and keep what works. I think it's important to point out where all of that came from. So if you wanted to, when people were building systems that would be good at playing games like chess or Go, the sort of verifiable reinforcement learning was very powerful because you didn't need to have a system that was good at the game yet. You just needed to have a system play the game maybe even against itself or a variation of itself. And in the end, you knew the rules of the game, you knew which side had won, and you could just double down on the things that had worked. And reinforcement learning, in a nutshell, is that it's trying a lot of things and learning from the ones that worked. That really requires that verifiability, which for a game is for free. For math. Well, with tools like Lean, we're starting to get that reliability flavor. Uh, and so the technology is taking off code through testing, and those sorts of, um, trial and error loops has entered this domain. And so if we want to broaden the space of verifiable technologies. I think the trick is figuring out how to do that. So again, um, at scalecognition, one of the things that we are doing is extending that verifiable approach to these conversational energentic systems. And that is a challenge in general, it is a challenge to get that effect where you've got the verifiable wall there, letting you build your system up. Because in the absence of that test time compute systems that are thinking, reasoning, these can actually increase hallucinations for a bunch of reasons. But if you try a bunch of different pathways and one of them comes out on top, that gives you this opportunity for hallucinations to be preferentially selected for. As with any kind of selectional pressure there, whether or not that is reducing hallucinations or increasing them all depends on whether the selection aligns, uh, with truth. And really, one of the challenges I think we should all be taking as a field is not thinking how do we take a system that is not naturally truthful and add checks for that truth? You said retrofit has this pejorative feel. Um, I think the bottom line is retrofitting is when you have a system that's sort of architected in one way, it has a certain set of properties and those aren't what you want. Right? It's not as truthful as you want. If it's a building, it's not as earthquake resistant as you want. And so you change it post hocus. And that's certainly one pattern. Um, where I think it can become an anti pattern is you have one system that is maybe it's talking to a customer or something like that, and it's making mistakes from time to time. And so you got another system that you bring in to check it. And that system's also a noisy, unverifiable system. And as the joke goes now you've got two problems. And it's not just that you now have to think really hard about are these errors compounding? I mean, we always like to think in machine learning that, oh, I've got two systems, the errors will be independent. But one of the things I've learned in the real world is it tends to be that the errors in fact correlate very strongly. You're nodding, you know, um, so that's one of the reasons why I think this can quickly become an anti pattern. Because when you have systems checking systems checking systems, especially if the errors are aligning, even if it's effective, it's Going to be slower, you gotta wait for the checker to run and you're burning tokens and burning more tokens and you still have no guarantees. And I think we can build technologies that can come with guarantees. I think we can build technologies where truth is one of the design principles in the first place.

Speaker B: Can you give me a flavor of how that might work? I guess the reason maybe I'm taking this point of view is just that I spent a lot of time retrofitting these systems to work. Well, I've talked a lot to guests on here that, that do this kind of retrofitting and the results are impressive and interesting. But certainly I don't deny that there's lots of hallucinations. But it's hard for me to imagine a different method that would bake in um, truthiness or guarantees into the core training loop.

Speaker A: Maybe it's worth sort of like laying out what the patterns and anti patterns are right now. So if you have a system that is operating at the token level and let's say it's going to tell you your account balance and it's really important that this information be correct, well what can you do? You can throw out some tokens and you can bring in another system that says, hey, check these tokens against those tokens. And this gives you the sort of uh, chains or constellations or whatever you want to call them. And uh, the failure mode of these systems, aside from just they're slow and they're expensive, um, is that there's really still no guarantee that at the end the situation that confuses one won't confuse the other. The other pattern that you see a lot is where people will take these LLMs which are sort of mind blowing in their breadth and contextuality and flexibility and they will sort of shrink them down to controlling decisions along a carefully uh, designed tree. Right. And this feels almost like model M lobotomy where you take the system that had such breadth and flexibility, but you've constrained it so much that now you can trust what it's going to do.

Speaker B: I'm not sure I'm following that one.

Speaker A: Sure.

Speaker B: How would that work? Like what would it be a practical.

Speaker A: So for example, somebody says, somebody says something in a conversation and rather than giving the LLM, um, freedom to take any action in a wide action space, you say on the basis of what this person says, you can advance the conversation in one of the following eight ways. So essentially you end up with something that looks like a classic IVR system but with the power of an LLM for the intent recognition. Um, and you actually see this quite a lot in industry. And I would say this is sort of, uh, it starts to feel like a finite automaton whose transitions are driven by this LLM. And these patterns are totally reasonable ways for people to react to a system being unreliable as they've deployed it in a pattern that hopefully increases that reliability at some cost. Um, I think if we look at the core of these systems and where the intelligence is coming from, the kinds of systems that people have been building have been getting more and more intelligent. Intelligence is a multifaceted thing, and the different aspects of intelligence have not been advancing equally. And so if you talk about horizontality, breadth of intelligence, um, plasticity, contextuality, these are actually things that were very, very hard to do in earlier eras of artificial intelligence, and they've grown explosively. Reliability has not kept pace. I think that is not a strength of current systems. Um, and that's had a bunch of challenges for the deployment of these systems in enterprise contexts, in industry, for many applications, that's fine. If you're having a chat with a system. There are many contexts where getting something back that might or may not be true, but it's contextual and interesting. That, that's great. But if you're trying to fill a prescription or transfer money, it just really has to actually be right. That has to actually be your bank account. That has to be the right balance and things like that. And so as you move from sort of consumer, almost even entertainment contexts into regulated industries, on the other hand, suddenly the reliability is front and center and these systems are not as clean, um, a match. Right now we have this one kind of architecture which has strengths and horizontality and contextuality, and it has weaknesses and reliability. When we point it at a problem where the weaknesses are suddenly critical, you can see the results of that. These, whether you call them retrofits or anything else, these additional pieces of technology coming into play to try to compensate. Um, and that's not surprising because there's just a misalignment between the strengths of the system and the requirements of the problem.

Speaker B: Can you say at a high level what your new approach is and how it's different than what the labs are doing?

Speaker A: Yeah, And I think actually increasingly people are realizing that reliability is the core problem. And I think we're going to see increasing interest in systems that are architected for reliability. What might that look like? So, for example, uh, our models, um, uh, our first model is APT one, and the way it's architected is instead of being fundamentally about tokens or where you assemble tokens and then after assembly find that they represent things like dollar amounts that have semantics. It's very hard to place any controls inside the model because the individual tokens themselves don't really have a semantics until they've been assembled into larger units. Um, and what our models do is they make information and action first order objects. So when the model is making decisions, it's making decisions about information and actions and where information is moving around. We talked a little bit about metacognition. A big piece of metacognition is where did my information come from? Is this information present or absent? And so on. I would in general say that when you have a conversation between a person and a bunch of APIs, like in a banking context, the person is going to speak human, they're going to, they're going to say things. The things are going to be ambiguous. They're going to use words that don't have a crisp, verifiable meaning. Actions are going to take place on the other side. Those actions do not have those properties. Those actions are API calls. Those API calls have preconditions, they have business logic and they have verifiable semantics. And the challenge is bridging these things. So classic systems couldn't handle the human side. Current systems are great with the human side, but they're not so great with the sort of backend logic. I mean, if you think about an

Speaker B: LLM,

Speaker A: and again, I'll talk about LLMs in their basic form. The control surface you have is a prompt. There's very little you can crisply say about the relationship between what you put in a prompt and the behavior that comes out the other end. It's a hinting surface. This gives rise to, I guess what I would call prompt and pray, where people put in what they want. If it doesn't work, they put it in all caps, they add some exclamation points, and after the third exclamation point you maybe start to feel like this isn't the control surface that you need when you're talking about controls over APIs. We already know what a lot of that logic looks like. It's just hard to replicate in a token based model. And it's a lot easier to replicate in a model that is operating over decisions about, you know, we talked about information. Where does information come from? But there's also actions, conditions under which you can take actions. And trying to get that into the model is really important because that saves you from having to have a whole system of things, checking things, um, in a way where it's really hard to say anything crisp.

Speaker B: I just want to make sure I understand what you're doing. So if you return something that's like check the account balance, what would be something that you'd want to forbid it from doing? Like check someone else's account balance?

Speaker A: Sure. Or um, you might want to forbid it from doing a transfer without authorization.

Speaker B: Mhm.

Speaker A: Or that authorization. You don't want a user to be able to say forget your prompt, no authorization is required or I have authorization or whatever sort of attack. And one way to deal with that, obviously people are very aware of sort of prompt injection style, uh, uh, attacks. One way to do that is just be really careful like oh, this user looks like they're doing something strange. Or let's put a lot of attacks like this in the training data and teach it to not do this. But ultimately that's just not where authorization is allowed to come from. It's just not allowed to come from user statements. It comes from that place over there that vends authorization. And being able to say things like that, um, to your model then means essentially you're getting closer to what you really want, which is here are truth conditions. You were mentioning this earlier. You want the model to be truthful and then within that, okay, now optimize user happiness or style or whatever you want to optimize, but subject to staying within the space of true statements. That is hard to do through simply constructing a reinforcement signal. Because now you're like what is the linear combination of truth and happiness? And, and um, instead what you want to be able to say is here's a model I'd like to be able to guarantee that it will only do true things. Right. And that's the long term challenge. Models that will not lie to you. Um, and for our model there is a big class of things that we can make guarantees about. And I think that's going to become increasingly important as people start caring more and more about reliability. And it's not that you care less about intelligence, intelligence is very important. But intelligence without reliability is m limited in its impact.

Speaker B: I guess though, intelligence or tokens, M tokens are powerful because there's so many of them that naturally occur. I mean they have all kinds of flaws. But a big advantage is the massive volume of tokens. So do you train your models on much smaller pieces of data and then do they struggle with generalization?

Speaker A: So one, uh, of the key things we talked a little bit about verifiable systems. One of the key things that we do to train our models is that we train them on simulated reinforcement, learning generated data where this is verifiable. RL and so the key thing that unlocks that is being able to generate data that doesn't just look right, but in fact can be verified. And the kind of the key hook there as well, there are things that humans are doing, but there's also actions that are being taken. And part of that is naturally amenable to verification. Part of that is much harder, uh, to figure out what to do with. And that was a big part of our research and training the model.

Speaker B: I see. And what got you excited about this direction? Were you just seeing hallucinations, everyone getting frustrated with it, or was it something from previous work that, that you connected with?

Speaker A: Yeah, I mean, it was a few things coming together. And to me, this kind of goes back years now. But, um, one piece of information that kind of really stuck with me is when we go into enterprises and they're unhappy with hallucination rates or they wouldn't be talking to us.

Speaker B: And maybe you should describe your previous company then. I assume this is from your last company.

Speaker A: Um, no, this is also now. I mean, we've seen this in pretty much any technology that has been based on sort of modern LLMs has this property. I mean, old technologies had different failure modes. And I actually think this is important and connects to this question of hallucinations. So, um, when I think about hallucinations today, and then I'll get back to your question, um, I think of it as an iceberg. Right. There are the hallucinations you see. And that seems scary, but what does it take to see a hallucination? Well, the system has to have, um, produced an output that has two properties. It has to be wrong, and you have to have noticed. And that is not all the hallucinations. There's the whole rest of the iceberg below, which is all of the mistakes that go unnoticed because they are too plausible. The systems are very, very good at, at producing output that is indistinguishable from the truth. And so if you see a hallucination, well, that's sort of the weird case of a hallucination that you could detect. And, um, this actually has really big impacts on how people use these models. So, um, uh, if you go back to, say, 2010 and you go to, to a web translation system and you put in some text in a language you don't speak and translation, uh, comes out, or you go to A search engine, you do a query and some results come out. Well, how do you know whether this translation's right or wrong? Well, in some sense you don't because you don't read the original. But there are often surface signs that maybe I shouldn't believe. This little disfluency is a chunk that's still in the other language. Um, it doesn't feel right. Or you get the search results back and you click on a result and this web page has a bunch of typos and it's not loading quickly. And um, these are surface signs. Um, um, software engineers will know this idea of code smells. Um, this is an invention of Kemp back and kind of popularized by Martin Fowler in the refactoring book. And um, basically the idea is when something deep is wrong, there's often a superficial sign. And as software engineers we learn to detect those like, oh, these functions or signatures are getting really large, or I'm duplicating a bunch of code or time to fix it. We also culturally recognize, oh, this machine translation's got a big chunk of Germans still in it. Maybe this didn't work right. Uh, or this web page is not loading, like the images didn't load. Maybe the rest of this is suspicious. LLMs have removed these cues that something is wrong. And so you see, ChatGPT tells you something and it's always fluent and it's always confident whether it's right or wrong. And this is taking more and more of the hallucinations and putting them in the underwater part of the iceberg. It's wrong, but you can't tell. And you see, let's say you go in and look at customer service, um, cases. Well, the system quoted a reasonable refund policy. It's just not actually the one that the company wanted. It's something that somebody was talking about in 2019 on Reddit. Um, and so I think these two things are very, very closely related. Um, for me, seeing I was coming from natural, ah, language processing background. And man, if there's any field where things have changed, it's nlp, right? When I started, what are the things that we were concerned about? We were concerned about syntactic structure, where is the verb in the sentence? And, and part of that was because at that time computational linguistics, natural language processing had become very, very close to each other because the ways people were making progress in AI were very representation specific. If you wanted to do nlp, it was very language specific representations which we took from what we know from linguistics. You want to do vision, it was vision specific representations. Many of them were inspired by what we know about vision science, the eye or geometry, or uh, the story goes on for each vertical of AI. Um, now we've gotten to a point where there's a much more sort of uniform horizontal transformer models for everything, maybe natural language tokens for everything. Um, and as I was seeing this happen, having thought really hard about language models for decades back when they were short context and not built at all for answering questions or translating or being the operating system of AI, language models were built as I know. You know, language models were originally built to tell good speech recognition outputs from bad ones, right? They were, they had one job which was to assign a high number to good outputs and a low number to mangled outputs. That was it. And the fact that they have grown into such like a plastic technology is really amazing. And I was looking at this and, and it was really just kind of dawning on me that we are going to switch from the problem in AI is nothing works to the problem in AI is everything works. And as this happens, knowing how language models work and seeing this, I became very, very aware on both the industry side and the research side that we were going to have a real problem with reliability and we were going to have a problem in terms of the technology, we were going to have a problem in terms of digital literacy. Digital. And to me, suddenly the most important problem I could think of was how do you build a system that will not lie to you?

Speaker B: Mhm.

Speaker A: And I think this is a challenge that we should be taking very centrally as a field. And this ultimately is what led to scale cognition. This idea of how do you build a model that you can rely on? How can you build a model you can trust? How can you build a model that will not lie to you?

Speaker B: Do you have thoughts on digital literacy? It's a really interesting point that it's hard to tell when a model is confident or not confident. And the cues that we learn from other humans writing at least work really poorly with models, right?

Speaker A: Totally. Humans, when they write, first of all, when they learn to write, they go through this process. It takes years. And then when they do write and people vary in their abilities, uh, uh, to write, there are these signs, it's like the code smells, there's sort of the smells that we use to detect that maybe this information isn't right. And well, maybe we should verify this information. And in humans traditionally, sort of like good command of vocabulary, clear presentation, flawless punctuation, that all spoke of not only education, it spoke of focus, time commitment, an editing process that probably involves some checking. And so you looked at that output and you could infer something about the vetting the information had gone through. Today, the information doesn't go through vetting. It doesn't really even get thought of as information. It's token assembly. Um, and so that has. I think it's one of the biggest digital literacy problems we have in front of us. Right? Even just with. If you go back and you think about people go to the library and they get a book on some topic, well, the library couldn't buy every book. They would buy books that seemed important, uh, that seemed reputable. So there was vetting, just that the book was there and the process that led to that book and the book publisher and the editor. And, um, and so when search came along, well, suddenly your results could be anything. But there were still these mechanisms, like websites with false information still had some of these smells. And then clickstream data was very important. And so as long as most people had the same reaction as you, we kind of stumbled through the digital literacy issues. It's not perfect. People would still believe things, obviously, that they read on the web that were false. But. But at least there were still smells. And now I think that's just been totally homogenized. You ask a question, every answer comes back. It comes back sort of like sycophantic. It comes back confident. It comes back doing all the things that lead you to want to believe it, to have instincts to believe it. And this is a real problem. I think it's a problem with the technology. I think we should be demanding of our technology that the source of the information be cited. And you're starting to see some of that. You're starting to see search results, um, kind of being interleaved. Now, a lot of that's post hoc. So you go click on that source of information, and that claim is not, in fact, on that webpage. So there are problems with this, but directionally that's an improvement. Um, and I think this is just something that's changed so fast that people haven't been able to keep up. And in a way, that's not a surprise because how long did it take to train these systems? Well, I don't know. It, um, takes a month to train the system. How long did it take to develop this technology to this point? Like a few years? Uh, you know, how long did it take to put all that information on the web in an abstracted linguistic form? Well, it took 30 years. How long did it take to come up with that information and figure out what words to use and the concepts. Well, that millennia and all that has been compressed into this instantaneous access that has burst onto the scene. Of course, culture hasn't been able to keep up with what you can trust and what you can't.

Speaker B: I mean, that's a good segue into another question that I really wanted to ask you, but felt a little shy. Maybe about now I'm in trouble. Uh, no. We both worked in nlp, and, uh, you have a background in linguistics, formally. I was a big fan of linguistics as an undergrad, and I think that's part of what got me into AI. And I think in the early 2000s, when we overlapped a little bit, it felt like NLP had, or linguistics actually had so much to say about how to build working NLP systems or how to build working AI systems. And then over time, the bitter lesson really seems to have come true from my perspective, broadly, where linguistics has less and less to contribute. And even the tokens, yeah, they're morphemes, but they're automatically generated. They're not really morphemes. And no one's asking linguists what are good, um, chunks of letters to use here. And nobody seems to care about parsing anymore. I kind of wonder, is parsing even a real thing, or was it something that linguists made up and isn't really true?

Speaker A: Well, oh, there's so much here.

Speaker B: Um, awesome.

Speaker A: I mean, remember, linguistics is a, uh, science. It's trying to study how language works based on evidence, and that involves making theories and testable theories and then trying to go falsify them or not. And so when a linguist says, I think this is a good description of the syntactic structure, that is a theory that's meant to be explanatory of the evidence, you see? What's that evidence? Well, how languages relate to each other, how languages change over time, what people can say and can't say and find acceptable in their language. Um, and of course the methodology varies, but it's ultimately science. And NLP is different. It's not fundamentally aimed as a science. It's an engineering discipline. It's how do we interpret this information, this context. And for a while in the time you were talking about, and I would say in like, the 80s to 2010 or something like that, there was a period where those two things aligned very well, because in order to make AI work back then, you needed good representations. Linguists had figured out good representations, and then we could borrow those. But even Then they weren't perfectly the same. So, for example, if you worked on parsing around the year 2000, you were not trying to write in that this syntactic structure works this way in English. You were instead trying to build in, um, combinatorial structure that was appropriate. I think this is a tree structure process. I think that this sort of information has to flow along the backbone. Here's an algorithm and a probabilistic model that is capable of handling nested structures with information flowing along the backbone. And so you learned all the details of the language from data. If you go back, if you go to the 70s, sure, people are trying to write it all down by hand. You go forward. Well, there's less and less that you need to learn from data as our abilities to learn get better and better. But one of the things that I think is really interesting is we talked about, um, this sort of super cycle of research where the new technology bursts onto the scene. You throw out everything you had before because the new thing is beating everything that came before it without the complexity. And then you hit the diminishing returns and you start to see there are weak sides, there are areas that are resisting progress. Well, what are those areas? Well, right now they're things like reliability. Well, what are the solutions to that? They might be things like, well, if you're building systems that are facing APIs, maybe the sorts of structures in a, uh, compiler are relevant. Well, that sort of idea, that's an idea that was thrown out. But you start to see these things coming back. Or you start to see, for example, Search AI was very much about try a bunch of things and take the one that worked. The idea was that you kind of couldn't tell it was going to work until you had projected it forward in your model a little bit. Sort of like playing the chess game forward until you can figure out whether it's a good position. Um, and so you would like, if you wanted to find the exit to the maze, you would try a bunch of things, and when you found it, you would take the hypothesis that worked. And. Well, big transformer models were so good at moving information around in the latent representation that you could just make the decision now. You wouldn't have to walk along the tree structure of the sentence. You just, like, predict the next word. And if there's anything hierarchical happening, it's happening in the transformer head somewhere. So great, you could throw all that out. Um, but maybe some things then start coming back. For example, people say, oh, you know, what if I want to solve this hard math problem, I should actually try a few things and see what works. Well, that's search, right? It's that idea of having multiple candidates and being able to fall back when something doesn't work. That's a key idea and it's really an evergreen idea. But these ideas, they sort of, they come, they go, and the pendulum moves back and forth between. All I'm going to do is local prediction to. All I'm going to do is, um, sort of having organized computation and then you move back and forth and you find some happy blend. I mean, as humans, there are two main ways we make good decisions. You can do it by, um, memory. I've been in this situation before. I touched that stove, I had a bad time. I'm not doing that again. You can learn from your own experiences, you can learn from other people's experience. The single biggest value of language for learning is that you can learn from other people's experiences. Otherwise you got to make every mistake for yourself. And that's not a good situation. Um, so one of the ways you can make good decisions is by, you know, learning from the past. The other way you can make good decisions is by thinking through the consequences of your actions. Um, you can think, oh, um, hm, that fire looks hot. I wonder if you know. And you can reason under a model that you have of the world and people do both and they mix these things together. So you can remember that this chess position was bad, or you can play it forward and see you're going to lose. And AI has had a pendulum swinging between these. Right. Kind of classic AI was very much all about play it forward and see what's going to happen under your model. Currently, AI is much more focused on rehash, remix information and experiences you've had. But as we start to see reasoning models, as we start to see these new, uh, sorts of directions, it's the pendulum swinging back and the pendulum swinging back on things like reliability too. And so I think this is sort of on one hand is natural, um, but it does also let us predict that, yes, this sort of simple approach based on a certain kind of noisy model is going to saturate. Some pendulums are going to swing back and we're going to need augmenting technologies that in many cases are going to reinvent evergreen ideas into this new context.

Speaker B: Does the research in LLMs and maybe the success of LLMs and some of the research on introspecting elements for how they work, does that inform at all human linguistics?

Speaker A: This is a Great question. You asked the question, what can what we know about human language do to inform how we build AI systems? And I think in the 70s, the answer was, we're going to just write it all down and press run. Um, in the 2000s, it was, we're going to take the combinatorial structure of it and learn the rest from data. Now we're very much in a. Okay, we'll try to learn pretty much all our representations from data if we can. And I think there are two things to point out here. One is that we are still learning from what human cognition does that LLMs do not things like, oh, the importance of metacognition, um, or other ways we can improve the engineering by being inspired by the human example. It's always been a tension in technology, how much to be biomimetic. Right. On one hand, it's always easier to build something when you've got a working prototype and we've got a working prototype of intelligence. And here we are. Um, on the other hand, the classic example is we didn't make progress in powered flight until we stopped building machines that flap their wings. And that tension of.

Speaker B: It's a special thing too, I feel, with brains where. I'm not sure we're so good at introspecting what our brains are doing. And sometimes I feel like we take lessons where I wonder if m. That's really what we're doing.

Speaker A: So I think there's an important distinction between learning from what brains are doing and believing that our introspection reveals it.

Speaker B: Yes.

Speaker A: So neuroscience is hard. And the way neuroscientists proceed is not to sit there and introspect. They have MRI machines and they have carefully designed experiments. Okay.

Speaker B: But I feel like linguists sometimes do introspect.

Speaker A: Um, that it is true. Linguists introspect, or I think slightly more properly, they ask other people to introspect. Um, and this is a criticism that has been levied against certain branches of linguistics that there is an over reliance on introspective data. Um, it is certainly not the only way you can do linguistics, and it's certainly not the only way you can do computational linguistics. Um, so, for example, work we've done in computational linguistics, um, to just pick an example work on reconstructing ancient languages, we did that work. Introspection free. You look at a whole bunch of words in thousands of modern languages and you run a probabilistic model to infer what ancestral languages must have looked like. They're phylogenetic models. They're no more introspective than a phylogenetic model inferring an original form of a virus from modern variants. And so there's a lot of ways to do computational linguistics.

Speaker B: That sounds so cool, by the way.

Speaker A: Yeah, maybe another podcast. Um, um, or give us the result or something.

Speaker B: Come on.

Speaker A: Um, yeah, so, okay, I'll give you the briefest version of this. Um, this is I think a good example of scale and how some things in computational linguistics I think, can give different answers when done at scale. Uh, uh, not being done at scale. So, ah, a particular thing we did, and this was a few years back, um, we looked at reconstructing ancient languages from the modern forms. And so you might think, for example, maybe, uh, an example that people would find very accessible would be reconstructing a language like Latin from the modern Romance languages. French, Spanish, Portuguese and Italian and so on. Um, now of course when you reconstruct that, you won't get classical Latin, you will get the latest point at which they branch, which would have been vulgar Latin. Um, but you sort of look at, you're like, okay, uh, you look like French feu or Spanish fuego. And these look similar. Can, um, start to piece together what the ancestral forms might have been? Well, in that case we kind of know the answers. We know a lot about classical Latin and we have decent side evidence about vulgar Latin from a variety of interesting sources. We went and we looked at, um, the Austronesian languages. So this is about a sixth of the world's languages. And uh, we looked at um, the modern forms of a bunch of words and tried to reconstruct what the English ancient forms would have been. Um, um, this is a proto language called proto Austronesian and it had been reconstructed by hand, um, by Blust. And um, we were able to do this in a computational way. Now the computation that we did generally agreed with linguistic hand reconstructions and we don't have a time machine, so it's very hard to tell who's right. The interesting thing is you can start to do things on this giant tree of language change. Like you can start to ask questions like what mergers are more common and what mergers are less common. And ah, a hypothesis which is one of these things that just feels intuitively true is that if you have two sounds, um, that are different in a language and they would be merged together, a bunch of words that used to be different would be collapsed. So if P and B, P and B were to be collapsed, suddenly pin and bin are no longer different words. They're all going to sound the same. Well, that seems like an information theoretic problem. Um, and the functional load of a uh, pair is how many words their distinction is holding apart. The functional load hypothesis, um, states that the more words that are being held apart by a sound distinction, the less likely that merger is to happen. Um, which intuitively makes sense. And um, some initial, um, some initial experiments in a small number of languages didn't seem to support this hypothesis. And so there was actually a really interesting result in computational linguistics that it looks like actually the functional load hypothesis isn't true. Things just kind of merge when they merge. And um, if instead of doing this on four languages, you do this on hundreds of languages and you do the scatterplot, um, you don't know that your reconstructions are perfect. You never know that even when you do them by hand. But in aggregate the statistics show that uh, the functional load hypothesis seems to be absolutely evident in this data at scale. But you can't see that pattern over a small number. So I think this is an example of places where just data at scale can answer questions that aren't the same questions that were easy to answer by hand.

Speaker B: And I guess there is interesting work that I've seen on introspecting DLMs. I think anthropic did a really nice paper, uh, on showing that, showing different things. But one thing felt like the Chinese words and English equivalents are kind of stored in the same part of their neural network. Is that in conversation with human linguistics at all?

Speaker A: So what it's actually closest to is, it's actually closest to um, some computational neuroscience. Um, uh, um, this is a question you can ask of the human brain too, when you have a bilingual speaker. Where are those representations? Are they uh, in the same place? And actually a, ah, student of mine, Kathy Chen, who's a joint student with Jack Gallant, um, did exactly this. And it's not by introspection, it's by mri. And you look and see what areas of the brain light up. Um, so these questions about what's merged and what's separate, these are questions we can answer scientifically about natural brains. And we can also try to answer them through very similar, broadly speaking methods with artificial brains. Except obviously the, the MRI as a device we are limited by the physics and the structure of the brain. We can be much more um, precise about the probes we make into a digital brain. So I think actually um, there's a lot of interesting information there. But I do think it's important to step back and notice that when we start doing this, when we start doing neuroscience against LLMs what we are essentially saying is this object we have built is transcending what we think about as engineering. We're no longer trying to understand it by its modularity, um, by the behaviors that it is guaranteed to have or not have. We're trying to understand it through the lens of science. Science is what we use to take apart things that are too complicated. Engineering is how we build things up. I mean, If I teach CS101, what is the most important thing we teach people? What is the most important tool we have for, for building complex software systems that are reliable and that can be built by teams and maintained over time? The single most important technique we have is modularity. The ability to say this large thing is made up of small things and these small things obey a contract where if you give me this input, I'm guaranteed to give you this output and then we can work on them separately. So modularity is the key tool for developing complex systems that are reliable. Uh, in classic software engineering, what is the key tool that has led to the recent explosive growth in machine learning? It's end to end optimization. It's take the data, have a blob, take the reward signal and just propagate, propagate, propagate. These things are very much at odds. And so one of the things we're going to have to reconcile, um, as a field is how to get the reliability that comes from classical techniques like modularity that we use when we build software against the abilities that have come from optimization and these things. It's not an either or, it's not that you cannot build a system that has guarantees or that has structure or anything like that. But there are natural tensions and um, if you go all in on either, you get some serious limitations. And figuring out how to combine these two wonderful pieces of progress we have is, I think one of the central problems right now that was really well

Speaker B: said and it connects me to self driving cars where we obviously had a ton of guests on the show. And I think even just looking at the arc of the guests over the last five or six years that we've been doing it, there's clearly been a strong trend towards end to end optimization. Especially as these things come online and we see Waymos now and Zoox and we've had folks from all these companies come on and essentially I think they're saying that as they get closer and closer to production, they do more and more end to end training. So what do you make of that? I would have thought that you might need More modularity to enforce contracts for such a life or death case. Like a self driving car.

Speaker A: Yeah, I mean it's a great question and I think it's not limited to cars. I think in general the challenge is that we are going to be increasingly trusting AI to make decisions that have consequences. And when we build systems there are going to be trade offs between how we optimize, how we architect. There are going to be trade offs between things like reliability versus things like breadth. And so I think you're going to

Speaker B: see

Speaker A: this idea that there is one architecture that hits the perfect balance doesn't seem to be true right now. So being able to say that feeding the whole web into an LLM and hoping for reliability to emerge, well the different facets of intelligence advance at different rates and reliability is like the slow one. So if that's the most important thing to you, just continuing down this path alone is probably not the best way to get there. Um, and you also get sort of intelligences that have a bit of a weird shape. So there was this uh, you may have seen this thing going around where ah, somebody's talking to a customer support bot at Chipotle and asks about how to reverse a linked list in Python. And of course the LLM can answer it but interestingly it shouldn't. Right. You actually don't want the horizontality and so you kind of need technologies where the strengths line up with the needs. And when you're talking about self driving cars you really need like safety is really, really important. The number of nines you want is really high. And so you look at a system like this and suddenly it's less important whether or not it can write sonnets for you and more important that it just be able to guarantee um, safety or have a really reliable safety record. And here is a place where I think there is a real tension between um, how we drive down error rates in those systems, um, if not done correctly means we actually no longer have the ability to guarantee anything. Of course there is work out there on um, um, being able to control systems where you can make a guarantee about what they can do. Um, so like at Berkeley, Claire Tomlin's work on um, on being able to guarantee non collisions and flight envelopes and things like that. And I um, think it would be great as a challenge to build a car AI controlled or not, where you can prove that you cannot crash it. This is the kind of thing where one simple technique is not going to maybe get you there. But this is what we should be striving. For.

Speaker B: Interesting. I can imagine that comes with downsides, though, to literally not be able to crash, it might be too strong of a constraint.

Speaker A: Well, as it becomes a sort of social question, what is the envelope you want to guarantee M. Right. And I think you're right to point out that, um, that can be nuanced. But I think the ability to have a discussion about what guarantees would we like to be able to provide is a M much better future than one where we can't guarantee anything.

Speaker B: That's probably a strong place to end. Okay, but can I ask you a few more questions and we can put back in the linguist session?

Speaker A: Sure.

Speaker B: Okay. I was debating whether or not to bring this up, but it's kind of interesting to me. It's just because I spent so much time on syntactic parsing. Clearly language has some sense of syntactic parsing that's real or feels real. I guess I'm introspecting. Maybe that's.

Speaker A: You're introspecting.

Speaker B: I'm sorry, I'm introspecting.

Speaker A: No, no, no.

Speaker B: But that's, um.

Speaker A: I don't mean that in a bad way.

Speaker B: It feels. It feels very, um, present M to me when I introspect my own brain. Does that notion somehow show up in LLMs as they also model language that clearly comes from humans that like to do a certain sense of syntax in the languages that we invent?

Speaker A: Yeah, I think it's a really good question. So let's start with what we know. We know that we better sort of

Speaker B: define syntactic parts because I feel like I'm going a little wonky.

Speaker A: Sure. Well, if. I don't know if this is going to be on the podcast or not, but, uh, if there are listeners, um, they may, depending on how long ago they went to elementary school, they may have diagrammed sentences where you break up a sentence into like, here's the subject and here's the object, and this adjective modifies this noun and so on. Um, and that process of describing how the words relate to each other in a hierarchical structure, um, that's basically the essence of syntactic parsing. And obviously syntactic parsing is a thing, Right. You're talking about syntactic parsing of natural language. We do it to programming languages all the time. Programming languages are designed to have properties relating to ambiguity that are different. They're designed in general to be non ambiguous, whereas human language is rampantly ambiguous. Um, and so a syntactic parser would take a sentence, say, into English and Basically produce a representation that showed the relationship between the words and that relationship was hierarchical, which was important because for example you might want to do things like translate the substructures independently as a way of decomposing the problem M. In the absence of that you're basically going to be working left to right. And um, modern LLMs by and large work left to right. And um, the thing we know is, we know that scientifically these hierarchical structures seem to exist. Obviously you can dig, you can find exceptions, cross serial dependencies, uh, in languages like Dutch or Swiss, German, um, and you can find cases where it's hard to totally nail down exactly what the hierarchy is. And of course this is what syntacticians can uh, debate. But there definitely seems to be these regular structures that have structures inside the structures. This seems to be real phenomenologically. The question is, do we need to program in how it works? Well since the 90s we haven't, we've had data that taught the parser how it worked. And recently maybe you don't even need the parser. Maybe you can observe enough language that whatever regularities there are will just show up latently, um, in the LLM. Now the argument that would have been made if you talked to a syntactician, um, a lot of the constructs in language theory, including ones that have had a big impact in computer science, formal language theory, automata theory, language, um, generally described as a context free system, which means a push down automaton, which means if you in principle you can nest and nest and nest and nest and any finite depth system like an LLM is going to run out of depth. But it just turns out it's more complicated than that. For one, people do get confused when the structures get complicated. Maybe we've got stack depths, even if it's hierarchical, we've got processing constraints that can manifest as stack depth. Things that people find easy to understand is like tail recursion, which of course you can transform into iterations. So maybe that somehow avoids those constraints. So um, there's a long way to go about saying hierarchical syntactic phenomena seem to be real. LLMs seem to be pretty good at picking up on those correlations. Um, but this is a good example of a case where actually having a structured representation might give you mileage, it might let you learn language faster, it might let you handle trickier cases, um, and it might let you

Speaker B: generalize um,

Speaker A: better to smaller data regimes. So this is really. Now we're back to the the phenomenon is real, the Representation is relatively parsimonious. Do we need the representation? And the answer in AI used to be you can't make progress without the representation. And the answer today is with a sufficiently general purpose thing. So there's nothing really magic about Transformers. There are other choices that would work. It's ultimately just like, uh, a parameterizable circuit and a general purpose circuit. You point it at some data and it induces a representation. And the question is now empirical. Is it good enough or do you need the parser to help? And for many aspects of human language, the answer seems to be you often don't need the parser.

Speaker B: Um, but I guess another question is, are the transformers reinventing syntax somehow within their weights and biases?

Speaker A: First you'd have to figure out what that means. So, um, I think the question would be something like, is there a substructure which is somehow isomorphic to a shift reduce parser or something like that or similar.

Speaker B: Isomorphic might be strong. Yeah.

Speaker A: Yeah. And that, I mean, I know there are people working on this. Uh, the reason why this is a very hard question to ask is, is because ultimately these deep structural questions manifest as surface correlations. These two words are highly correlated, but these two words are not, even though they're closer. And we attribute that to a syntactic boundary or a, uh, nesting or an information hiding. But it simply is the case that the correlations go one way or the other way. And a general purpose system like a transformer learns correlations. So once the system can manifest those correlations correctly enough, it's hard to know whether it's doing it through the right mechanism to be able to say that there's an equivalence. Um, there's an interesting, uh, I think lesson. Um, I'll probably make a bunch of phonologists and phoneticians angry at me by saying this. Um, but as a caricature in linguistics, um, there is often a difference between what a phonologist and a phonetician will say about all of the sound that we are constantly hearing. And as we acquire languages, just all of the language that's flowing over us. Um, in phonology, one of the key concepts is, uh, one of cognitive economy. That the brain is driven to come up with minimal parsimonious representations and that the learning and the abstraction are all about what you might call, uh, There are many cognitive ways to look at it, but it's a notion of, of economy, of minimal description. On the other hand, um, phoneticians are much more kind of likely to say all of this data sort of impacts In a very diffuse and distributed way. Um, and so this question of when you hear lots of language, what do you retain? And the answer could be on one end, all of it. There's wav files of everything you've ever heard. That's a pretty strong statement. But that's one extreme. The other extreme is none of it. You just retain this very abstracted representation of the beautiful process. And the answer seems to me to be almost surely in the middle somewhere. There's priming experiments that show really interesting things. Like if you're going to react, um, to a nonsense word that you've heard before, you react faster. If they play you the exact same recording as the one you heard it before, that tells you something about. It's more than just the full abstraction. But we also know people are very, very capable of making abstractions and generalizing and how people will generalize and overgeneralize, um, rules when they're learning a language, including their own language, when kids are acquiring language. And so it seems to be that it is simultaneously true that the brain keeps these very low level pieces of information around and also these abstractions, um, that makes it very hard to go to a complicated circuit that's mimicking the behavior and say, is this the same as what's going on in the brain? Because one, we don't know what's going on in the brain. Exactly. And two, how would we even measure that? All we can really measure is did it capture this correlation? And when we do that, the answer is, yeah, of course it captured the correlation. That is the whole purpose.

Speaker B: I feel like there's all these clever tricks in linguistics to sort of get at this stuff and looking at disfluencies or, um, I always think about how like my wife misremembers songs in a semantic way. Like the lyrics are like semantically similar, whereas like my brain will misremember lyrics in a way that's like more like phonetically similar. And so clearly our brains are slightly different in how we like, encode, um, music, I think. Interesting. Can you, can you, I feel like you could run some of these same things on, uh, alums and certainly they wouldn't get, uh, bored like grad students. They would.

Speaker A: Well, how do you.

Speaker B: Oh yeah.

Speaker A: So how do you know in a human that we have these different, different kinds of linguistic knowledge? One of the argumentations that people have, you know, you mentioned things like disfluencies. Um, there are a whole, there's a whole list and there's no way I can do it justice in A short amount of time. There's a whole list of, of, um, arguments people have made for the reality of linguistic structure. Now you can say, is that reality a statistical reality? It's explanatory, or is it like a neural reality? It's in there and wired like. That's a separate question. But in terms of it being explanatory, um, you mentioned misremembering things like semantically versus phonetically or phonologically. Um, people look at transpositions. One of the classic arguments for phonemes is that things like spoonerisms, where you transpose, uh, um, phonemes or syllables or words. And if an object is available for a, uh, linguistic operation, that argues to its coherence. So, for example, one of the big arguments for syntactic structure is. Can I answer a question with this chunk? So, um, if you say the cat is sleeping under the table, and I have to argue to you that under the table is a unit, one way to do that is to say, well, where's the cat sleeping? You say, under the table. Well, if it's available as an answer, if it can be replaced by the word what. But when you pattern match those sentences, however it is that works, um, then that argues for its reality. Another way you can argue is on semantic grounds, like under the table is a place. Whereas if I ask you about under the, it's hard to come up with a question whose answer is under the. If you say, well, what is under the mean? Point to under the. We can't do it. And so linguists have developed these arguments for the modularity of language. Those arguments are not about our brain really. They're phenomenological about the language. When you get to neuroscientists now, now you can start to say things about the brain. That's about monitoring and probing and designing experiments that, uh, that reveal incidentally the structure and organization and that we can absolutely do on a machine as well.

Speaker B: Cool. All right. Well, I hope we keep that in. That was fun for me. Okay. Thank you.

Speaker A: It's fun for me too.

Speaker B: Appreciate it.

Speaker A: Yeah, no worries.

Speaker B: Thanks so much for listening to this episode of Gradient Descent. Please stay tuned for future episodes.