Why AI Hallucinates (and Why It Might Never Stop) | EP. 46

Welcome to Hidden Lawyers. We're exploring the people and technology behind artificial intelligence. I'm your host, Ron Green. So here's the puzzle.

Large language models can explain medical exams, string theory, and even debug your code. But ask them a simple fact like somebody's birthday, and they'll just make something up. Not tentatively, not with caveats, but with total confidence. Why does that happen?

Researchers call it hallucination, and it's one of the most interesting unsolved problems in AI right now. Some groups are trying to catch hallucinations in real time, almost like putting a fact checker inside the model. Others are asking deeper questions, like maybe the way we train and score these systems actually pushes them to bluff, rather than admit when they don't know. If that's true, then hallucination isn't a bug at all.

It's what we've accidentally optimized them to do. To make sense of all this from the technical fixes to the deeper incentives, I'm joined by two the sharpest minds in the industry, my co-founder and distinguished machine learning engineer, ZZSI, and our vice president of engineering, Michael Wharton. All right guys, ready to talk about hallucinations? Yeah.

Oh yeah. Okay, wait for it. So first off, hallucinations are obviously still a big problem, but I feel like I'm definitely seeing them less now than even six months ago. How about you?

What do you think, ZZ? Yeah, I definitely feel like the language models are getting better, and the early kind of simple errors, like counting ours, install, bury, and other similar tests are now passing with the frontier models. So definitely, I feel that's true. What about you, Michael?

Yeah, I think that's the case. And even still, I feel like there are ways to use them now, especially for personal productivity tasks, where you can pretty quickly identify if there's something that's a problem, because I mean, I'm just in the habit now. I use clawed most of the time. And at the end of every prompt, I'll just be like, please search the validator response.

And I'll go check. And the fact that they can do tool calling and go validate with searches. Exactly. See, all right, but I almost feel like that's cheating a little bit.

It is cheating. But it's the dominant use case that I have. I completely understand. So probably in my, my, I use clawed for coding, and I probably use chat GPT five, mostly for personal stuff.

And I found that, well, I took a couple things. One is, this is strange, but I've felt over the last couple of years, I've gotten good at almost detecting hallucinations. Do you feel that way too? Like when it says something, and I feel like, that's, that can't be right.

And I'll check. And the reason I feel like hallucinations are down is increasingly when I think the models are wrong about something mostly factual. It'll come back and go, no, I double checked. That was correct.

So definitely getting better at that. But then the other area is that there's just a distance difference between sort of work hallucinations and personal, you know, request hallucinations. And most of the time, you guys are using fear using anthropics, clawed model. You're using it for coding.

I would imagine most of the time. That's the time, man. ZZ, what percentage code do you think you are writing these days? I use it every day.

And I feel like I am already used to hallucinations. It still happens every day. But I think we're just trend to deal with it and, you know, expect that will happen. Yeah.

So an example is, I'm, I'm sure if you use one of the coding agents. And this is not a particular to any providers, like, you know, I think cloud code has the problem, codex has the problem, which is the GPT1, you know, Gemini also have similar problems. So it will hear, you are absolutely right. And then I will say, okay, here is what I changed.

With my change, you're going to get this, this, and this, and you know, passing 100% off the tests. You're going to get this functionality working 100%. And I was like, no, it's not. Because I checked, and you know, I would run the code and it's far from working.

So that, that definitely happens. And you can get in that death loop, too, where you get, you're consistently asking you like, oh, no, that's wrong. Try again. No, that's no, wrong.

Try again. Exactly. And then it's almost, I've gotten to that loop before it will happen like five times. And at some point, it's better to just scrap it and start over with the better.

Yeah, yeah, exactly. My, my worst, like hallucinations cycle I've been in recently, this is like in the last couple of months, was I was working on a presentation and I was kind of word smithing it. And this was with, uh, CHGPD. And it was, and it said, you know, would you like me to create some images for that?

No, I didn't really want it. But I was like, sure, yeah, I'll take a look at all the images it generated with the, the links were broken. And it was like, you, do you want to slide? I was like, well, yeah, create a slide.

Got done. The links were broken. Like it was just making fake links. Had it done none of the actual work.

Yeah, I love it. Yeah. So, okay. So I, we're making progress on this front.

But I, I genuinely believe hallucinations is one of the most interesting and most important unsolved problems. Um, I'm going to come back. I've asked some questions about whether you think, I'm going to ask you guys later whether you think we could ever solve this problem entirely. Oh, Michael, you read a paper.

You're going to talk to us about called why language models hallucinate. This is a new paper from OpenAI. Just came out this month. I took a look at it.

It's really, really interesting. Can you take us through it? Yeah, absolutely. So yeah, I mean, at the end of the day, it's not, um, I mean, there's some dense parts of the paper that get into some pretty rigorous math and stuff.

But at the end of the day, it's really not that complicated in terms of outcomes. There are two really key points. One is that, you know, for I guess years at this point, we've been just thrown around that claim, hallucinations are inevitable. They added some mathematical rigor behind that claim that I think is just useful to know about to just verify that this is going to be a persistent problem that we'll have to deal with.

Um, and then they also made some recommendations on how we do evaluations. Very practical. I would argue kind of like obvious and hindsight, but I'll go through that in a second. So in terms of the sort of inevitability of these class, uh, these hallucinations, you know, it's actually kind of a hard problem because when you define what's correct, what's hallucinated, what's not, they're these degenerate cases that are still technically correct, but that you, um, you know, kind of muddy the waters in terms of the analysis.

Like if you were to create a simple computer program that just said, I don't know to every single prompt, theoretically, that's a correct or at least like a valid response. No hallucination. Yeah, there's no hallucination there, but it's also total garbage in terms of utility. Right.

Um, the same thing if you just memorize your training corpus complete wholesale. Yeah. Um, you're not really learning any underlying distribution, but I guess it's correct. But what they did was they made the observation that you can frame any language model as a binary classifier in terms of just, is it valid?

Like you get a, you supply prompt, do you get an answer and you ask, is it valid? That's a binary classification problem. And there's a benefit in terms of just decades of research and statistics and learning theory that helps us make a lot of big claims as long as it's a binary classification problem. Uh, there's a bunch of math underlying there and, uh, really the, the claim here is that those types of errors that you would have with binary classification are a subset of that broader group that include hallucinations and those we can mathematically guarantee under reasonable assumptions that you have with language models today.

Is that generally track unpacked out a little bit more? Yeah. So based, some of those assumptions are things like, um, you're using cross entropy to train it and that it's reasonably calibrated, which notably that's a really crazy assumption with some of these post train models because we know they're famously poorly calibrated. Right.

But their analysis was pretty focused on pre-trained models, which are generally pretty well calibrated. Right. Right. But yeah, so a lot of those assumptions were, you know, pretty important to at least include.

And then if you just assume that, yeah, actually, an interesting part is that even if you're training data set is completely error-free, which you would imagine to be a pretty key contributor to hallucinations and errors. Assume that those are not even the case. These types of hallucinations are still guaranteed. Mm-hmm.

That to me was one of the most interesting parts about the paper. Yeah. Yeah, just because of the nature of how binary classifiers work. Right.

Exactly. So, all right. So let's go at another level. There were some of the things in the paper that really blew my mind that I grew with you.

They kind of felt obvious in hindsight. This idea that models are being graded judged. You know, their loss function is trying to determine how accurate they are on their output, right, and their ability to predict the next token, which essentially does not reward you in any way for admitting you don't know because even if you have a one in a hundred chance just by guessing, that's better than a zero percent chance of saying you don't know. So, one of the things that the paper, I think, again, this seems pretty obvious, is there needs to be a modification to the way we train the models so that the loss function can separate, guessing, and not encourage guessing, right?

And one of the examples they give is essentially penalizing models based upon the basically the inverse of their confidence level in the tokens that they're generating, which seems, I think, at a high level seems pretty straightforward to me. The other part of this is that, you know, there's always, I think there's always going to be a problem with hallucination when you're dealing with sort of rare information, you know, information that was maybe only the example they give is like somebody's birthday that appears in the corpus maybe once, right?

Or maybe not even at all, you know, how you handle those situations. And then, of course, that dovetails with the whole theory around how much do or the whole thought that hallucination is not necessarily bad thing because it's sort of tied in with the imagination and creativity and that you want to have those components. I'm less sold on that, but it's an interesting thought experiment as well. Yeah, especially that, I mean, I think that evolution or sorry, the evaluation piece really to just kind of hammer that home.

The thing that again, yeah, it's just obvious and hindsight is like most of these really rigorous benchmarks are multiple choice. Like even the medical benchmarks, the software coding benchmarks in a lot of cases are multiple choice. And if we're just rewarding confident guessing, like it's no wonder that these models are really poorly calibrated. So if you penalize them for answering a question and getting it wrong, but you reward for getting it right, then I mean, that makes no claims about how we're supposed to make these models better.

But it does at least align the evaluations more with the behavior that we're trying to elicit, which seems, yeah, great. I'm just wondering what that right. And I think it's also a reason that we're seeing coding and math performance with these models lead the way because they're verifiable, right? You're training with verifiable domains.

And I think it's easiest to drive out the hallucinations and achieve human or superhuman abilities. Yeah, 100%. Yeah, I'm curious whether the paper mentions like using more of reinforcement learning as a way to reduce hallucination. Yeah, they did talk about, I guess they made no claims on exactly what sort of algorithm, but they did talk about the changes that happen between you have a pre-trained model and then use reinforcement learning to post-training.

I don't think that they dug into that as much as they probably should, but they did imply that there's like a host of techniques and it's relatively under-explored to try to minimize these hallucinations with algorithmic tricks and techniques. But the thing that I don't know about how you guys feel about this, I don't care if we get rid of them completely. I think if we can just get them to the point where the models have utility, it's almost like I think about it kind of like medicine where the goal is not to just remove all error.

That's kind of ridiculous, but you can imagine you'll cross this threshold where the error rate is below a specialist or like a human level and then at that point it's just negligence not to use it. Yeah, who cares if we can get rid of these things completely? Let's just suppress them as much as we can and move on. I generally don't think it will be possible to get it to zero anyway because I don't care how good these AI systems get, even if we have artificial superintelligence, they're not going to be omniscient.

They're not going to know everything. They're not going to be error-free, right? I generally think that that is sort of a false error and there's no way to achieve systems like that, no matter how sophisticated they get. It's just a part of the universe we live in, am I opinion?

Yeah, I totally agree. I also think that it seems to be inherent. I mean, the hallucination seems to be inherent due to the way it's trained because, well, let's consider the pre-training and not too think too much about our AIO yet. I feel like in the AIOM world, there's no notion of reality versus not reality.

The whole world to the AIOM is the web corpus that it has seen. And all it's asked to do is to, you know, when we're using it, when we're chatting with it, is to, hey, here's a new document and this document is a new conversation. Just help fill out what could be the possible words spoken here based on what you have already seen in a web corpus. And so, I mean, there's no notion of reality.

There's no notion of grounding. So, yeah, I mean, we cannot ask the AIOM to do things that it's not like 24. It's totally different. That reminds me of that quote.

I don't know if it's even been attributed, but the quote that all language model outputs are hallucinations. It's just that some of them turn out to be true. Yeah, right. Yeah.

I totally buy it. Yeah. Yeah. All right.

So, Zeezy, you're you're going to talk about a paper that kind of goes into the details of maybe not fixing hallucinations, but modifying them, monitoring them in real time, which is really, really interesting. So, your paper you're going to talk about is called Real Time Detection of Elucinated Entities in Long Form Content. Yeah. Yeah.

I think it's a really interesting paper. You know, of course, as humans, when we use the AIOM, we care about the hallucination. We care about, you know, whether it's true according to the reality. So, this paper is talking about a method to kind of detect and mark the risk score of the generated text in real time.

So, before this paper there, there have already been a lot of methods to mitigate and detect hallucinations. But most of them are post hoc. So, like, you know, after you chat with the AIOM, it generates an answer. You can use the AIOM or another AIOM to independently do a web search or do a retrieval in your internal documents and then using that as a prompt to compare against the answer from the original AIOM.

And that's a relatively easy task because once your prompt includes all the information there, then, you know, comparing them is a relatively easy task. So, you can use that to do the check post hoc. But this paper is interesting in that, you know, it can do it in real time. Like, it's not a post hoc thing.

It's basically treating the problem as a sequence labeling problem. So, when you, when the AIOM is generally talking by token, you can use this paper's method to annotate each token with a risk score. And they have a pretty cool demo to show, you know, there's a streaming text generated from the AIOM and then some words will be marked as red indicating high risk and some of our, you know, if it's not hot, then that's relatively safe. Yeah, the demo was super cool.

Is it, is it using an external model that was trained for this specific purpose or is it like aware of the weights or, yeah, okay, that's a great, that's a great question. It's not entirely an external model. That's such a great question. I want to dive into the details and I wanted to discuss and see what you think of the method and how we can apply to maybe our future clients.

So, first, it collects the data using a kind of the benchmark prompt set. I think it's called a long fact. Yeah. And they expanded that original long fact prompt set to long fact plus plus, which has, I think, roughly 20,000 prompts covering multiple domains, like healthcare questions, legal questions, questions about movies, you know, stuff like that.

And then they would collect a training dataset by, you know, just using AIOM to generate the answer. And then they will use AIOM as a judge to do a post hoc validation by doing web search. And that process kind of generates this ground-choose data, which is not perfect and I'll talk about this later. But still fairly high quality kind of label dataset for which spans in a text is hallucinated or not.

They actually has three labels, I think. It's either supported, unsupported, or something like not sure. So that's the output from the AIOM as a judge. So that's the kind of label that you wanted to predict.

Now, they will train another model, yeah, they will train a model to based on not the output tokens, I think that part is interesting. They're basing on the internal kind of the hidden activation. They picked one of the layers that's later, I think, later in the network. I don't think they did a very thorough kind of sweep on in which later to think they just picked something like 95% into the network.

But yeah, they use that as a feature. It's a token label feature. So for each token, it tries to retrieve before that tokens output. So it's like, hey, get the thought vector.

And then using that as a signal and they train two types of models. One is called the linear probe. And the linear probe is very simple, it's just that latent feature, hidden feature, and add a linear layer. And then to predict whether it's hallucination or not.

And then the other method is Laura, where it's a more communicative way. So basically, they will train this Laura, which basically will update the model weights, right? That's what I thought was really interesting is the linear probe is just trying to learn like a linear center box classifier. But the Laura Pro, they actually were allowing it to modify the weights a little bit to improve the predictability of it.

And they said it did not, in any way, perfect the model, affect the model performance or degradate the model performance. That sounds almost like at least the former case sound like a polygraph or something like you're picking up on biometric signals to try to predict if there's a liar. Yeah, you're putting a probe in the model's brain to try to detect if it is lying or hallucinating. Yeah, that's amazing.

So yeah, the result is super interesting. Yeah, both the linear probe and the Laura method, I think I'll perform the baseline like, you know, post hoc kind of validation without web search. Because you know, if you use the web search, and then, you know, of course, that's going to be better because that's the teacher. Right.

But then, you know, the paper, the paper's method outperforms all the baseline post hoc way of detecting hallucination. And the Laura works better than a linear probe. Which makes sense. Yeah, yeah.

It's a more complicated model. It's more accurate. It also changes the model's behavior. It doesn't degrade the quality.

But it changes the behavior subtly. I think they will find this interesting. So after their Laura training with the same target of, you know, predicting hallucination or not. They observed that the model became more cautious.

Great. Which is really interesting because it's not trying to do that. It's only trying to predict hallucinate or not. But it's not trying not any generation task.

That's one of my favorite parts of the paper. It would still answer the question. But if it answered it with the hallucination, if it answered it, it was more likely to add something like, but this information cannot be verified. Interesting.

So it's almost like adding some layer of awareness to the performance. Exactly. It's almost like it's almost like backpropping through that Laura mechanism, that Laura probe gave it some additional awareness of when it's hallucinating that it could leverage. That almost sounds like I would put my money on that new model being better calibrated, too.

Sounds like it would have to be. It sounds like it would have to be. Yeah, I totally agree with that. That's great.

The only thing that I think we have to bring up is as good as it was, it still is nothing that you could put in production. It's nothing that would allow you to use it, I think, in sort of high critical domains like clinical situations, where you had the model suggesting dosages or something like that. I believe, if I remember right, the paper said that it could catch about 70% of the hallucinations with a false positive rate of about 10%, meaning it would classify true statements, not hallucinations, about 10% as hallucinations.

So a lot of work to be done, but it's a really interesting approach. So first, I've seen anything like this working inside the model and not not working once the tokens are fully generated. Yeah, totally agree. And I very appreciate this paper's approach for mentioning limitation.

It explicitly said, it's not ready for production. I wish more papers and more research papers. Yeah, I totally agree with that. I'm glad you mentioned that.

That really stuck with me as well. Too many papers come out and they want to hide the dirty laundry. They did a fantastic job of putting sunlight everywhere, even in the shortcomings. Yeah.

I want to ask you guys on the hallucination front. We talked about this just a little bit earlier, but to kind of wrap up, what is your timeline on hallucinations? Meaning, when do you think as an industry, we'll get to a point where I think we, let me back up. Do we all agree that we think hallucinations are here to stay?

I don't know what you think is easy. Do you think it'll I feel like it's here to stay because the model doesn't even know hallucination or not right during training, right? Michael, you believe the same. We can approximately figure it out, but I think at the end of the day, yes, an approximation.

I agree too. And the way I would think about it is the word hallucination to me is maybe problematic. I would, I think about it more like, will we get to a point where the models don't make errors? No.

So if you frame it like that, I think it's kind of easily to say, no, we're not going to achieve a world where a models are omniscient and they never make mistakes. Okay, but given that caveat, what is your prediction? When do you think we'll get to the point where we at least live in a world where the most advanced AI's make mistakes hallucinate at such a low rate that it's not a topic of discussion anymore. In fact, it might even make the news what it happens.

I have a maybe a hot day. I mean, I think assuming we give time for the public to catch up, I think we're kind of already there technologically because I mean, you mentioned that the Oracle that was used to train this data set in your paper was just a language model that had access to the internet had access to some knowledge base of facts, which is not perfect. It's actually. Yeah, it's still a problem.

It's still there, but I think as humans, we do the same thing like we've all but offloaded a huge chunk of our cognition to search engines in the last 10 plus years. That's a great point. And people that use Google more often than others are probably less likely to make nonfactual claims. Yes.

I think that if you're willing to hook up an agent or like hook up some sort of tool that has access to that knowledge base, I mean, I think we have the fundamental technological technology. You could have it now if you were willing to wait for that sort of post prediction and pay the cost and all that in terms of your actual question when we'll get there with language models. I think it's, I don't know, a year or two. I don't think it's too long.

All right, so you're pretty optimistic. ZZ, what do you think? I don't have a crystal ball. My bet is maybe longer, and this is informed by my kind of experience with the current coding agents, but of course they're changing fast too.

I feel like it's maybe the complexity of the reducing LOM hallucination to a really, really low level that's negligible is maybe similar to self-driving cars kind of our comfort level of using self-driving car on the road. That's a good comparison. There are probably many corner cases and when we get into deeper domains of either building software or automating internal processes for healthcare for legal professionals, there's probably a scare of training data for those. So sometimes even with web search, you still cannot be 100% sure that there's no hallucination, and this is also mentioned in the paper too.

So even the training data is only 80%, 90% correct. So that's why all these kind of labelling platforms like Mark or they're seeing crazy growth and evaluation. Okay, like if you had to say how long tell how many years until you think hallucinations are so uncommon, again, that they would make the news. You're really trying to.

I'm just going to pin you down. I give my hands. Ten years. Okay, that's perfect.

I think I'm right in the middle of between you. I agree. I agree to you, Michael, that I think that if you're willing to wait, we could probably get the right answer at any problem now. But I'm probably a little bit more on the five-year time horizon before I think that the tools that we baked in to the systems that are used every day and enabled will make hallucinations seem like we'll laugh about 24 or 23 and what we used to see.

On the other hand, I do think that LOMs and all these agentic tools can be incredibly useful already without getting to that low hallucination level. Yeah, absolutely. Yeah, not to take away from the incredible capabilities they have today. Especially that we have tools to monitor it and to mitigate the risk.

I think that's a really good point, ZZ. I mean, I don't think that we've learned the extent to which we can effectively use these tools yet and still so much headroom for us to just, I mean, I don't even think that we're using coding tools. Code assistance as well as we could. I completely agree.

I think I underestimated it, but I'm beginning to only appreciate how important how important the prompting is. There are things you think these models can't do, but if you spend time in the prompt, you can eke it out. Do you want to hear too kind of funny tips that I have about using clock code? Absolutely.

So whenever I see you're actually right or some bold claims about, hey, your apps is now at the top of the world. I do one of the two things. One, I would say be honest. Review your change again.

And then surprisingly, it will have an opposite answer. It will list like 20 mistakes and things to change. I believe it. So that's one trick.

The other trick is I would say, well, my friend said, this and this. And then, you know, the clock code won't agree with the same level of enthusiasm. So that is, there's a paper that came out of, I think it's Wharton School business recently where they used a bunch of like social manipulation tactics to try to, you know, basically jailbreak these models. We're just too mean to the models.

It's shocking to me how much you can take proven methodologies to manipulate people and apply those to models. And it actually seems to work just because it was trained on a corpus of language that was generated by humans. That's exactly right. It makes perfect sense.

I love it. All right. Well, this was fantastic guys. I appreciate it.

Thanks for having us. Thank you so much. Thank you for listening to Hidden Layers. This series is hosted by Kung Fu AI, a management consulting and engineering firm focused exclusively on artificial intelligence.

If you have any questions or thoughts about today's episode or if you know someone we should feature, please visit us at kungfu.ai.