The Bitter Lesson: The history of reinforcement learning

Speaker A: Hi, I'm Adam Gordon Bell, and this is Code Recursive. And today I have here again, uh, Don.

Speaker B: Hi, I'm Don McKay and, um, I'm here again.

Speaker A: So I've been texting you again.

Speaker B: Yes. Yeah. And it's always late, too. It's never like, you know, it's 10 o' clock at night or something.

Speaker A: I know, sorry.

Speaker B: I mean, when you're like an old man like me, like, that's late, it's too much.

Speaker A: Yeah. You're like, I eat at 4, I go to bed at 8. Um, yeah. Yeah. So I've been. I've been trying to understand, I guess, like, machine learning. And that's why I sent you some things. So, uh, yeah. What did I send you?

Speaker B: First you told me you didn't have whiskey anymore. And I was like, that's a crime. And then you said, okay, I've been continuing going deep on AI and ML stuff. I'm trying to understand it at a level of like, could I write a program that's just normal ifs and loops and whatever, but yet it can learn? And I think I found the key. It's this paper. Reward is Enough. Silver Singh Precup Sutton, 2021. I want to explain it to you. Sort of simple and simple. My neat quotes.

Speaker A: And, um. Yeah, because, like, okay, machine learning is fancy and cool and like, we're software developers, we should be able to understand it. But it feels, like, magical.

Speaker B: Yeah, it's like existential. Right? You're like, oh, I don't know. I don't know how it works. So. Yeah. Oh, okay. Uh, here's the important line. Intelligence and its associated abilities could be understood as subserving the maximization of reward.

Speaker A: What do you think? Subserving is such a weird word.

Speaker B: Yeah. You haven't heard of subservient individuals?

Speaker A: Well, I guess. Okay, maybe in that context it seems

Speaker B: like he's saying that as long as you have a reward structure, you can create intelligence, which I think is a little bit reductive, in my opinion.

Speaker A: I mean, it's incredibly reductive. Right?

Speaker B: Yeah.

Speaker A: He's saying it all reduces.

Speaker B: It all reduces down to m the maximization of reward. Right. And furthermore, saying intelligence and its associated ability, Associated abilities can, can, uh, be understood by subservient reward, which means that everything about our intelligence is just serving some kind of reward structure. And I think that that's kind of an. I don't know. Yeah, it's, it's, it's very reductive. Once you, once you reduce something to that to that level, you. You. You lose the fidelity of. Of the statement. Right.

Speaker A: So it reduces everything. But as a cool title, right? Because that sentence is very complex. But the. The title of the paper, right? Reward is Enough. You kind of get what he's saying, right?

Speaker B: If you wanted to create, uh, an artificial intelligence, reward is enough. That's all. That's the only structure that you require.

Speaker A: Everything. Yeah. That's all you need is just that one thing, right. Like, I've reduced the entirety of, uh, intelligence, of intellect, of culture, of. I mean, I guess he didn't say culture, but, uh, everything to do with what makes something intelligent and smart and able to take action in the world is just reward, Right. Press the lever, get a piece of cheese, basically. Okay, so, yeah, Sutton, he wrote this book on reinforcement learning, and, like, his. Him and his advisor, like, kind of invented this field. Yeah. So I just want to pull it apart. Right. That's what I want to talk about today. I was like, this is a great thing for our kind of stack trace format, which. Which I asserted before and maybe still needs a better name that, like, we take something and kind of, like, back up the stack frames until, like, you get.

Speaker B: Right.

Speaker A: Uh, we have this really grand statement like, can we pull it apart? Can we get to somewhere?

Speaker B: Null pointer exception.

Speaker A: Yeah. Turns out it was all intelligence is a null pointer exception. Gone the bonkers. Okay, so it's 1988, and Richard Sutton is finishing his PhD at the University of Massachusetts. And his advisor is named, uh, Andrew Barto. And Barto, uh, he actually comes from B.F. skinner was his background as a, uh, academic. Do you know B.F. skinner?

Speaker B: I do not.

Speaker A: So interesting enough, right? B.F. skinner, he taught pigeons things where they would, like, press a lever and they would get, like, a little piece of thing.

Speaker B: Okay.

Speaker A: He's the person who, like, operationalized how to, like, train animals to do things based on reward. And, like, his theory was also that all of human behavior is based on, you know, going after rewards. Like, the same way your dog wants a treat. It's kind of a weird background. Like, Sutton, uh, is like, a computer scientist, but he's working under this guy whose background is, like, this behaviorist and who, like, taught pigeons, basically. And side note, Skinner is super interesting. Probably not a topic for the podcast, but Skinner in the. In World War II, he built things for the U.S. military. Uh, any guesses what they would be?

Speaker B: Uh, homing pigeons.

Speaker A: Yeah, so he built, um, a missile guidance system where they just put the pigeons.

Speaker B: They put the pigeons in the missile. I did, uh, actually see See, uh, like a video about that.

Speaker A: Yeah, it's insane, right? Like, he. He had very reductive view, as you would say. Uh, like he reduced everything to like. Well, yeah, let's just. We need to hit this target. Why don't we just train a pigeon? Yeah, they.

Speaker B: They were good at it.

Speaker A: And like, he also had, um, or like one of his students built. I wrote this down. This guy who worked for him named Tom Verhave, uh, had a pigeon product or project at the company. Uh. E. Is it Elly. Is that how you say it? The drug company? E. Lily.

Speaker B: I think it's. I think it's Eli Lilly.

Speaker A: He had this project at this company where basically they just had pigeons do qa. So like all the. The parts are going down the aisle and they just have pigeons there who like, are trained, but look for the. The broken ones and pack them out of there. So they tried to. Basically they replaced human workers with pigeons and uh, like they got the whole thing work. And then they canceled it because they thought it would be like a PR disaster. Although I did hear a story before that they canceled it because it was very demoralizing. Like, imagine you're at your job and they like, they fire the guy like further down the chain from you, like further down the.

Speaker B: And just replace him with.

Speaker A: Replace him with a pigeon, like he. His door. So it didn't take off, but same idea, right? You just give a simple reward. The pigeon learns like, oh, this is a defective part or this is a defective pill and knocks it off. Anyway. This is behaviorism. Anyways, that's where this Bardo guy came from, and Sutton's working under him. But the whole computer science thing at this time in 1998 was very different, right? It was AI, but a different approach to AI. Expert systems. So expert systems, they sometimes call it good old fashioned AI expert systems. Like, was this idea, uh, of AI was like different than today? It was if you could interview a doctor and figure out how he assess a patient, and you can just like write down all those rules and then ask people about their health. And then it's like a doctor in a box because it's like a. Just like a flowchart that you can travel through. I mean, it seems kind of basic right now, but they had this idea, uh, you know, if we can get all the rules for all of intelligence written down, right? Here's all the doctor rules, here's all whatever we can replace, we can have intelligence just in sort of if statement and whatever. So Sutton's not that Right. He works for this behaviorist. Different approach, different. And he publishes a paper called Learning to Predict by the Methods of Temporal Differences. And it's a super. It's a super cool idea. I'm going to show you because this is how I started to understand, um, machine learning. Right. So this is. Yeah, this is tic tac toe, right? And this is how he explains the thing, right? So if we. If we play tic tac toe, you go first. So you're. You're X. You can pick which position you want. But this is all the possible states of the entirety of the game in this tree.

Speaker B: It's a, uh, it's. It's just a giant chart with a bunch of, um, X's and O grids and what the current state of them would be very similar to, like chess, uh, moves, kind of like mapping.

Speaker A: It's kind of like a tree, right? Or a flowchart. So it's like his idea is, okay, there's tic tac toe. He figured it out. There's like. I think it's like 725 possible states that you can be in, right? And then, like, at the very end, you have a result. Right? So, like, if we played this game where you get three X's across and I do nothing to stop you because I don't understand the game, then he. He gives that a value.

Speaker B: So, uh, yeah, he's assigning values to win conditions so that, you know, that's what you want to get. That's the. That's the ideal state.

Speaker A: It's like, there's no magic. It's like an array of 725 values the game. And, like, you put an X and I put an O, and so it's a tree going down, and at the end, I win. Right? So here's this trick. This is. This is like the whole of the thing. And then all he does is this. Right? But this one must be negative one, because Adam went. And then. So this one, he goes backwards and he's like, okay, this one is like negative 0.1, right? And then this one is negative, uh, 0.01 and so on going backwards and saying, like, well, those must have all been pretty good because you won. They're not as good as the part that you won, but they're. They're kind of good. So it's basically like walking backwards. It's like how people develop superstitions in a way, right?

Speaker B: Uh, so you trace back the path to how you got there, and you assign values according to how Much you want to incentivize that path based on the end result or decentivize the, the path based on a, uh, losing condition. So at the end of the day, when every possible move has a value, you'll know how to respond according to always going towards the one that will give you more of a score.

Speaker A: Exactly. This is this whole paper, I think you just nailed it, right?

Speaker B: Yeah. So whatever choice you make, it will then evaluate all of the other choices that it's learned and then it will have a score and it will go with the one with the highest score.

Speaker A: So if I go play. So that was play one game, right. If we go play 10 games, like every time it goes through the tree, it decides if it wins. So it's, it's like, hey, play a whole bunch of tic tac toe games and see where you won and where you lost. You don't even have to know how the game works. You don't even need to know what's good and what's bad.

Speaker B: You just need to be able to play it many times so you can start building it like a, like a model of scores.

Speaker A: Yeah. And if you do like enough times, you start to know what the values are. Like what's a good move and what's a bad move. This is his whole thing, right? Uh, this is like his early paper. And I love it because it just explains this simple version of machine learning.

Speaker B: Right.

Speaker A: It's learning to play the game, um, just by doing this working backwards step. And like there's no crazy stuff, right. It's just a bunch of array values and you're like updating. It's like if, if at the end of the day you get told you did good or bad, then you just like figure out during the day what were the things and work backwards.

Speaker B: Yeah, that's, that's amazing.

Speaker A: And like, that's his, uh, that's, that's part of his like, reward is enough. Right. Like all you need to do is

Speaker B: assign a score to desirable or undesirable outcomes.

Speaker A: So what happens next?

Speaker B: Ah, he moves up to chess.

Speaker A: Not yet, but if we're going to get there, um, four years later, 19, uh, 92, um, somebody picks up his paper, uh, and it's this guy at IBM, uh, Watson Lab, and he does backgammon, which was like, kind of an interesting. Backgammon. People are super into backgammon. But like, I don't think it has the allure of like chess. Right.

Speaker B: It doesn't have the pedigree of chess.

Speaker A: You know, backgammon involves gambling often, too. Like you're betting on it. Like, I don't know, like old British men play it, I guess. I don't know who.

Speaker B: I know my mom played backgammon all the time.

Speaker A: But it is like a complicated game, right? And it. And it has some interesting. So the guy who decides to work on it, his name is, uh, Gerald Hasaro. He's at IBM Watson. And he just takes that rule, right. Which Sutton calls the temporal difference update. Um, but there's a problem, uh, with backgammon versus tic tac toe. Maybe a problem versus every single game in the world versus tic tac toe. Do you know what it might be? They're just bigger. Like, tic tac toe is so small.

Speaker B: Oh, you mean like the possible outcomes?

Speaker A: The possible outcomes is just like, astronomically larger. Right. So like tic tac toe. It's great for his paper because there's like less than 800 states, but, um. Yeah. Do you want to guess on the backgammon number?

Speaker B: I have no idea.

Speaker A: So this says it's a hundred quintillion possible states, which are, uh, so that would be possible game states. Right? It's. If you played all possible games, if you needed to make this array to hold all the values of every possible

Speaker B: position in every possible game.

Speaker A: Yeah. So it would be. No, that's exactly what it is. It would be as many grains of sand, uh, on the beach avert. So you can't even enumerate it. Right. You can't make an array or a hash table.

Speaker B: You'll need a lot of memory.

Speaker A: Yeah. It's just not possible. And then, like, to play that game forward and backward and update those all. It would take too long. Right. So he does something different. Um, he just. He uses a neural network. So you can imagine that we have, um, like in our tic tac toe, we have like, a function, say, that takes in the like, nine values, like your state of like, X's and O's, and then it returns that number. He does that, but he doesn't actually, you know, come up with that much memory. He just puts this neural network in the middle that somehow returns a value. Right. And that way he can shrink the problem. So the network that he makes is tiny. Three layers. Uh, in a neural network. A neural network has, like, kind of floats in it and then some conditions that make it either return, you know, a high value or a low value. I don't super know how it works, but he's using way less values than there are Possible states, right? And the idea there is it's kind of like compression, right? If you need to store all the values of the game state, but instead of having just a bazillion values, you say, like, oh, we only have this many, then it has to figure out, like, what the duplicate parts are. It has to, like, pattern match. It's kind of like a JPEG compression, right? It's, oh, all these games are similar. We can update this little spot at once.

Speaker B: It starts to see winning patterns from losing ones.

Speaker A: It just doesn't have that much memory, much like we don't. So it has to find ways like, oh, this is similar to that other time. Maybe this is like that. Uh, but he does the same thing, right? Have this thing play backgammon against itself, and then when it gets a good result or a bad result, you know, update the value in the neural network. Uh, so what happens? Like, how can something actually get better when it's just playing against itself? You know what I mean? Like, it doesn't know backgammon. Like, how is it going to get good at it if it's.

Speaker B: Well, you get to know all the moves, right? If you haven't played it before and you play against yourself, but you know the rules, like, you have to at least know what the rules are to the game. And when you play against yourself, you start to recognize what to do in response to certain moves.

Speaker A: So this is a side project bit. He plays and he has this thing like, it's 1992. Computers are still pretty slow, but he's, uh, got off in the corner playing backgammon against itself at this time. Like IBM. Watson is this prestige lab of the AI era, like, for this expert systems of, like, writing down all the important rules. Like, this is the preeminent place in the world. But this guy's, like, doing his other thing, right? He's taking the Sutton guy's idea, having it run. Well, they're all busy off working on more important expert projects. I think it's. You can guess what happens, like, after

Speaker B: they let it play with it, like, play against itself for a number of days, months. Like, how long did they spend.

Speaker A: Hundreds of thousands of games against itself.

Speaker B: Hundreds of thousands of games. Um, imagine they would have moved on to something else.

Speaker A: Well, I mean, maybe they did, but I mean, he's a researcher, so he wanted to write a paper about it and whatever. But anyway, so he brings in. He brings in backgammon players, he brings in expert, um, and they play it, uh, and they lose. So this thing has Gotten better at backgammon than the real world players. And backgammon does have chance in it. So it's not totally definitive, given enough time. Because, I mean, this is. That guy's point reward is enough. Like, you get to the end and, oh, I lost this round against myself. Well, what did the. My, you know, what was the difference? And, um, how can I. Like, it's the same. It's just a bigger scale of that tic tac toe thing. Right. It also just did interesting things. Much like chess, there's this whole theory behind backgammon and, like, when you should do this and when you should do that and what you should do in these situations. And it does different things. It hasn't read those books. Like, it's just the computer. Computer programmer.

Speaker B: It only knows the data that it's acquired by playing itself.

Speaker A: Yeah. And so it does things playing them that they're like. And then it beats them. And they're like, oh, wait.

Speaker B: So it beats them because it's unconventional. Rather than trying to imitate humans, it develops its own sense of positional judgment by learning from experience and playing against itself.

Speaker A: And that's why it was so surprising to these backgammon folk, right? They're like, wow, this thing, uh, plays unusually and it's beating us, right? So they brought in this guy, Bill, uh, Robert D, two time world champion, top three player alive, who had written books on backgammon. And, uh, he's like. He says, like, yeah, this thing is better than we are. And we were wrong about things because it beat me in these ways. And so he's written all these books about how to play backgammon. And so he starts updating his book, right? He's like, playing against this thing, learning new moves. He's like, oh, yeah, we gotta change that. Cause it just beat me this way. So in its unique experience of, like, not being exposed, it's almost like, as you said, not being exposed to humans has actually benefited it. Right?

Speaker B: Yeah.

Speaker A: It's learned extra things.

Speaker B: It didn't. It didn't learn, um, the game framed in any kind of way. It just kind of learned it through its own experience.

Speaker A: Yeah. So that was cool. They beat backgammon. And then, like, you would think the next thing that happens is this takes the world by storm. Uh, but didn't, uh. They just forgot about this idea.

Speaker B: They just forgot.

Speaker A: Yeah. Guess for how long.

Speaker B: I'm guessing they forgot because, like, nobody plays backgammon. So it didn't grab headlines to, like, your. Your layperson. Um, but for how Long. I. I mean, if it's. It's a successful project, I wouldn't imagine it would stay dormant for that long.

Speaker A: Yeah. So, uh, 20 years. 20.

Speaker B: Okay. That was way out.

Speaker A: I mean, they had a much bigger project they were working on, which was, uh, Deep Blue was like beating the chess expert and using these expert systems. And like, the researchers who were building this amazing chess thing, they were like, I don't know. That's just backgammon. It's weird. Like, of course it works with that method, like, because it's. I don't know. Who cares? Like, yeah, it's. It's like saying some. It's like a result they didn't care about. Right. They're like, we're not interested in this way of solving things, and we're not interested in this game.

Speaker B: So they, uh, they just dropped it for 20 years. Yeah. Like, those researchers went on to other projects.

Speaker A: Yeah. I mean, I'm sure some people kicked this idea around, but this wasn't like the hotness, Right. It's like, there's probably somebody out there still really in to like, uh, Macromedia cold fusion. But, like, that's not. That's not the thing. Right. Like, that maybe it had some cool ideas, but, like, who knows? Um, and so DeepMind, they. They published this paper in Nature. Yeah. This is from their paper.

Speaker B: Perhaps the best known success story of reinforcement learning is TD Gammon, a backgammon playing program which learned entirely by reinforcement learning and self play and achieved a superhuman level of play. However, early attempts to follow up on TD Gammon were less successful. So it's not that they were successful and then they got shut down. It's that they were less successful when they tried to expand, expand beyond backgammon. And they're like, this is a dead end.

Speaker A: I guess so. And everybody gave up on it. So Sutton himself, in 2017, uh, gave a talk where he was kind of saying, like, wtf?

Speaker B: Yeah. Temporal difference learning. Temporal difference learning. Temporal difference learning. It's a method for learning to predict. It's basically the center, the core of many methods. You know about Q learning? Sarsa, TD Lamba, Deep, Uh, Q networks. TD Gammon, the world champion backgammon player using deep root reinforcement learning from 25 years ago. Oh, my God. 25 years ago. Deep reinforcement learning, 1992.

Speaker A: I don't think you gave it very good emphasis, but, like, he's pissed off, right? It's like, he's the guy. You get where he's coming from, right? He's like, I had this thing Yeah.

Speaker B: I mean, if he's.

Speaker A: If he.

Speaker B: He's probably pissed because he's like, I did this stuff like, 25 years ago and nobody gave a crap. I mean, how often does that happened? Like, on a very much smaller scale to a lot of. A lot of programmers in the field, right?

Speaker A: Like, uh, listen, our database, like, if we follow this trend, it's going to get too big. There'll be no instant size that can hold it. And, like, it'll destroy our business. And I've worked on a solution. It's just like, if you give me three days, like, it'll save us in 18 months.

Speaker B: And they're like, three days, though now. You could be working on this other thing for three days. Yeah. Uh, forget about that.

Speaker A: And then 18 months later, they're like, so we got this issue.

Speaker B: Can you, like, whip that up like, right now?

Speaker A: And that's. That's when this guy's like, temporal difference learning. Like, I'm telling you.

Speaker B: I've been telling you this for 25.

Speaker A: Yeah. So then. Because what's happening at the same time, right, Is this Deep Blue thing. Deep Blue. Do you remember the deep Blue story?

Speaker B: I do, yeah.

Speaker A: I remember what happened with Deep Blue. Give me the. The summary.

Speaker B: That was the one that, uh, played chess, right? Yeah. And it started, uh, winning against grandmasters.

Speaker A: It was this big project to beat the world champion at chess as, like, publicity for. For IBM. And it was like, part like, oh, they had these very big, uh, supercomputers. But a big part of it was this expert systems, right?

Speaker B: Do you play chess?

Speaker A: My dad always wanted me to play chess. And he, like, taught me. Um, but I would just lose to him. And then I think once I left, like, after university, I learned. And then occasionally, like, when I'd go home and visit him, I would play chess against him. At one time, I beat him, but I think he made it, let me win. But that's like my ex. Like, he, like, from. From my perspective, like, chess has a lot of ranges. From my perspective, he was just way better than me. But chess has so many levels, right? That I'm sure, like, uh, there's tons of people like a grandmaster, like, nobody can beat them. When I learned chess, there was this idea of you. Certain pieces of certain values, right? So it's like you're sitting there and you're playing chess, and it's like, my dad's gonna do this move where my pawn takes his pawn, and then I could take it back with my knight, but then his other pawn could take my Knight, right? And, you know, like, oh, the knight's more valuable. It has a value of three. And, like, the pawn only has a value of one. So, like, I shouldn't do that because then, um, like, you're basically, like, adding up the value of each side. And so, like, a big part of playing chess is, like, this idea. It's like, you have these pieces, and you know what's valuable. And it's like, okay, if I make this move, then what'll he do? And what are the values? So the. The interesting thing is, like, the. Like, they've looked into this, right? Grandmasters actually don't play that many moves, um, ahead. So they might play, like, three or four, right? But the thing is that the m. The moves that they consider are always just the best move. Just interesting.

Speaker B: Yeah, like, they have names for all those, like, different opens.

Speaker A: Yeah. And they have the open books, um, which is like. Yeah, all these people who've played forward, like, the first whatever, like, four or five moves on a chessboard, like, they've all been exhaustively done, and they know, like, which one is better and which one isn't. And so Deep Blue used this strategy, right? So it had this giant open book of all, you know, the good ways to move forward. It had this giant book of end games. Like, if it's just down to these couple pieces, how exactly do you win it, right? And then instead of what I was saying of, like, I just add up the pieces and determine the value. I mean, I guess not instead of, like, it had something like that, but, like, IBM was throwing all this money at. They brought in all these grandmasters, and they came up with, like, the most accurate way that they could come up with to, like, assess at this point in this game, right? What, what value? Like, how good am I doing? Right? And so what I described as, like, whatever, three or four rules for adding a point. Like, guess what theirs was.

Speaker B: Uh, 20.

Speaker A: It was 8,000.

Speaker B: Okay?

Speaker A: Like 8,000 rules.

Speaker B: I don't know why you ask these people these, like, arbitrary questions, like, different answers.

Speaker A: Well, no, because it's good, because you said 20. And I think the thing about 8,000 is it's absurd, right? But this was this, uh, expert, uh, system idea, right?

Speaker B: Which 8,000 moves in advance?

Speaker A: Well, so the 8,000 isn't how many moves in advance. That's just how they figure out for any given game position, how. How valuable it is, right? It's like you get plus one if you have this piece there. And, like, like, just 8, 000 different rules to Say, like the equivalent of our number on tic tac toe. That's like, this is a 0.7 scoring system. It's the scoring system for evaluating any specific chess move, right? And then, like, as you said, right, like, you know, we play this and we think through, like, okay, if I move here, you move there, right? And then. Or that state will use all those rules to figure out if this is advantage. But then the other thing. Yes. Was like, Deep Blue was just like, I can look through a bazillion of these, right? So I have my 8,000 rules, and then I'll just play it out really quick, right? This was in 1997. Computers weren't as fast, but, uh, it would play 200 million positions per second. And, like, it was this big boon for, for IBM, but also, like, for AI, they're like, ah, we hit the pinnacle, right? But in the meantime, like Sutton, the. The tic tac toe guy, right? So he's working on a tech.

Speaker B: He's.

Speaker A: Yeah, he just keeps saying that whenever you talk to him. He's like, temporal. What is it? Temporal?

Speaker B: It was temporal difference learning. He's just screaming about that in the background.

Speaker A: So he writes a textbook, right? Him and Barto, who's like the behaviorist with the chickens or the pigeons.

Speaker B: Pigeons.

Speaker A: Pigeons. So Reinforcement Learning, an introduction comes out in 1998. And, uh, now we would say it's a very influential book. People use it and it's important and whatever. But then it was like, what? Whatever. Like, IBM is the thing and Deep Blue and nobody cared, right? Everybody's working on something else, right? It's still like, he's like the cold fusion guy, like, doing his Macromedia thing. Everybody's like, dude, what, what are you doing? Um, but there's. There's somebody paying attention, right? The thing that IBM really figured out was this searching forward, right? It's like we're moving really fast through this game and we can figure out, you know, chase down the tree of possible moves and find the best one. Um, where the Sutton guy was like, you know, just play it to the end and then we'll figure out who won and work backwards. Uh, but. So somebody, uh, looks into this in 2015. Dennis Habiz, and he. He founded the company DeepMind. So this is years later in 2024. Um, he said this.

Speaker B: We bet on generality and learning. So those were always at the core of any techniques we would use. That's why we triangulated on reinforcement learning and search and deep learning as three types of algorithms. That would scale be very general and not require a lot of handcrafted human priors. Oh, ah, okay.

Speaker A: I mean, he's an expert in the field and I'm some guy, but. Yeah. So DeepMind is this company, and they're gonna turn like, they're like, we think we can reinvigorate this. Right? Like, we think this is important. Um, um. And so they come up with these Deep Q networks. Deep Q network is just the same as the Sutton rule. It's this idea of, like, working backwards and applying these values. Right. And so it's the same ingredients. There is Dutton's Update rule, and then there's a neural network. And then there's just a, uh, programming method, right. To, like, play games and put together data so that it can, um, learn network with like three layers, very small. But in the meantime, right. While all this stuff was happening, uh, we talked about it in the LLM one actually. Like, these dudes like Hinton and stuff at U of T, they figured out how to make really big neural networks and learn all this stuff. And so they decide to do Atari. Basically, we want to build an AI that can play Atari game. Um, so they. They have 49 Atari games. And. But they have this problem, right, where you think of, like, if we do, um, Tic tac toe, it's like our state, you know, into our neural network or array is easy, right? It's like we have nine fields, like, either filled out or not. Right. But how do you send in, like, where we are in a video game, like, into something, Right? Like, it's hard to make a function call that's like the current state of my video game.

Speaker B: Yeah. Because there's too many factors.

Speaker A: Yeah. But just do the simplest thing.

Speaker B: Well, I mean, like, you could do something like X, Y, Z, um, for like, coordinates or.

Speaker A: Yeah, so they did. They took whatever the resolution of an Atari screen is and they blocked all the pixels. So not they wouldn't send in every pixel value, but they would send in like, say, like a five by five block. Every five by five block. And they put it to grayscale. They took out the colors because, like, that doesn't actually matter usually for.

Speaker B: Okay. So they, like, reduced the resolution and then sent the pixels in.

Speaker A: It's like asking it to learn. It would just like. But it can't see the game. It can just see, like, you know, 0,0111. Like, uh. So the software they built, they made it play like a whole bunch of Atari games. And so it gets its reward thing, right? Is Just the score. And what its input is is, like, all of the pixels, and then Atari has a joystick. So then it's just, like, it can decide. It gets in all the pixels, and then based on his joystick, it can just, like, side. Right, I'm gonna push this way or push that way. So this is their. This is, like. This was pretty famous, I think, when it came out. So this is one of their videos.

Speaker B: It's Block Breaker. Like a. Like a very, uh, early Atari version of Block Breaker. So you've got your, uh. You've got your paddle at the bottom. You've got, uh, you've got your. Your ball in the middle, and you've got, um, rows of blocks at the top, each of them a different color.

Speaker A: And so this is after it's played a hundred times. Right. So it's played a hundred times and learned some stuff. And it's just trying to.

Speaker B: Yeah, so the. The goal of it is always just to prevent the. The ball from slipping past you. So you've just got to intercept it.

Speaker A: Yeah, but it has to learn that, because all it really knows is the score. It got something. It's basically randomly moving the stick around. Okay, here's 200. It's more like I would play could occasionally it, like, randomly moves the stick. Okay, now we're at 400.

Speaker B: 400. It's getting a little bit more, um, accurate.

Speaker A: It's catching it. It's getting it every time.

Speaker B: It's moving it to intercept.

Speaker A: Yeah. Okay. So it feels like it knows what it's doing. Like it's learned and. Okay, then the. What is the. The overlay here says at this point, the agent finds and exploits the best strategy of tunneling and then hitting the ball behind the wall. It's learned something. After 600, it's.

Speaker B: It's learned a strategy of trying to get in behind all the blocks, because then. So it must be, um. So, yeah, because it's counting the score. Right. So it's like, if I can do this, then the score will go up more without my intervention.

Speaker A: Yeah. So it's figured out this idea of, like, drilling a hole through all the blocks, and then. Then it doesn't have to do anything. Right. The ball just bounces around back there and clears things. That was, like, a huge deal. Like, there was a paper in Nature. And, um, it wasn't just that game. Right after that, DeepMind gets acquired by Google, which is nice, because Google, much like IBM, has just, like, a bazillion computers. The thing about this type of training Is you need something that can, like, play the games over and over.

Speaker B: Right. You need the hardware.

Speaker A: You need the hardware. And so that is when they decide to tackle. Right. Uh, because I feel like if they were, like, cool, we have this method. Everybody said it was no good, but we made it work. And, like, we're going to tackle chess, nobody would care. Like, chess has already been vanquished with this other tech.

Speaker B: Nobody wants to hear about that anymore. That's old news.

Speaker A: Yeah, we've moved on from that. So chess is already won. So they decided to do Go. And the thing about GO is it's incredibly hard. So at that point, there was no good AIs at, uh, playing Go, or very few. The reason that Go's interesting is because of, like, this kind of exponential explosion when you play chess. Like, there's maximum, like, 35 moves you can make each game. And as we said with Go, there's much more. 250 possible moves at any point. Which means if you're trying to, like, play forward and build that tree, it's just astronomical. It's a very large board. You have a lot of pieces that you can place in lots of places, and then it takes a long time to play, and it's not till the end that it kind of all resolves. And it's like, did you win or lose?

Speaker B: Human players probably haven't even played moved those moves, right? Because there's so many.

Speaker A: But humans are good at pattern recognition, and somehow, you know, they learn to play this game and there are champions, and. And they're good at it. But, like, it doesn't seem to fall to our normal techniques of, like, oh, let's map this all out. These reinforcement learning guys came in who had been listening to Sutton with, like, his temporal difference. Yeah, what is it again? Temporal difference learning. Temporal difference learning. They listen to him, right? And so they built their machine. But there was this guy, Remy Coulomb. He was an AI expert, and he had built the best GO program in the world. I think I have a quote from him.

Speaker B: So the quote is, I think maybe 10 years. But I do not like to make predictions.

Speaker A: So he said it would be 10 years. Um, but then 22 months later, they. They had beat this, right? AlphaGo beats the best Go players. That's awesome. But, like, what is it? Right. Because I'm a software developer.

Speaker B: Yeah. You explain, like, the problems with, like, we can't. We can't play the game to the end. Like, we don't know what a good state is. So, like, how does it make the decisions then on what? How it quantify vibe.

Speaker A: Exactly. This is the question. Right. And so I built, I built a version of wordle. You know wordle?

Speaker B: Yes, I think everybody knows wordle.

Speaker A: Yeah, I never really played it, but it's my example. I built like alpha wordle, basically.

Speaker B: So if you haven't played the game, it's a, uh, is a five letter word that has been chosen that you don't know what it is and you're trying to guess it. And you can put in a word and it will evaluate your guess and tell you each letter, whether that letter is in the word. And in the right spot, it's in the word but in the wrong spot or it's not in the word at all. And given that feedback, you have to then pick a new word and you only get a finite amount of guesses. I think in this one there's like six. And then after six guesses, if you haven't figured out the word, then you lose the game.

Speaker A: Here's a, here's a wordle game. We're looking at, um, let's guess. So I don't know, what should we get? So I'm going to pick the word slate. When you play wordle, you can do the same thing as in chess. You can say like, okay, if I play this move, there's not an opponent, but I can get back a score. Right. And so like if I get back and none of them match, then what? I've eliminated not very many positions, but

Speaker B: some because I mean there's a finite amount of. It's that it's quantifiable now.

Speaker A: Yeah. So you can kind of play it forward in the same way you would with chess. Right. So we're looking at a wordle game. Right. And when I pick, I'm going to pick slate as my guess. And it was completely wrong. So it scored them.

Speaker B: And none of those letters are in the word.

Speaker A: Yeah. And so there's only a finite number of wordle words. Like in their set, I guess there's 2,000, 315. So like I, I played slate and nothing matched. But actually that eliminates a lot. So this is saying in my little app I made here before this, there was 2315 possible words.

Speaker B: Yeah. Because you eliminated A and E. Right. Which is contained in a lot of English words.

Speaker A: Yeah. And then I can play this forward. Right. So, so say to, to go back. To go. Right. At any given state in the game, I can tell how good I'm doing just based on how many Possible words are left. Um, and so that's like the one component that's hard and go, right. The other component is the playing forward. So now that we have the slate position, I can pick a new word. Like let's say if I pick this birch. Uh, but before I select it, right? So for any given word I can do the same as in chess, right. I can search forward. So I know, like if I played the word birch.

Speaker B: Oh, I see. Then you just be like, oh, if I get this result, then yeah, that's because the, um, the result here has a finite amount of possibilities.

Speaker A: Yeah.

Speaker B: Yeah.

Speaker A: Okay. So this is kind of, this is my dumbed down version of how the AlphaGo works, right? So it has this neural network, right? And in the neural network, the, the weights in the neural network are basically from playing the game through all the way to the end, the. The same as tic tac toe. And it figured out what won and so it updated things, right? And so for the game, for, for its playing, it thinks this word arose is like its best guess. Probably it played that word at some point in one. And so it's like, oh, this is awesome. Right? So this is its best plays. Um, and then it has this value which it calls position strength. So for Wordle, it's saying my position strength is 3, 6, 4, 4, which is basically, it thinks that it will win in about.

Speaker B: Yeah, about three. Yeah, about three guesses. Or, or four. Three or four.

Speaker A: So this is, uh, these are like the two important parts of the machine learning, right? It's like it needs to know how good it's doing. This is kind of like the chess points, right? And then this is kind of like what move to make. And so it thinks that we should do arose. How do we do arose? Uh, but then the other important thing is we can, we can play it forward in our head. So before we submit our move, we can kind of. Okay, if I pick that, what are the possible options? Right? So it thought this arose was best based on its original playing, but then when it plays forward from that position, um, it decides like, no, irate is better starting word. Like, I'm going to pick irate. So basically AlphaGo, they've combined these two ideas, right? So Deep blue was this one idea, sort of let's play forward into the future and figure out the values, right? Like anytime it's our position, we can kind of spread out and try to go quickly through all these rules and figure out what the best thing is with this, which is like, then once we get to the end of the game. You know, we'll learn from that and we'll update these, which is kind of like Sutton's like tic tac toe. Does that make sense? It's like they came up with this idea that we'll have a neural network that tells us how good each position is. But how will it know that? Because we said, oh, and go. It's super hard to know actually if this is a good position or not. And so they said like, well, we'll just, we'll do the Sutton trick. We'll play a game of Go. And when we get to the end, we'll say like, hey, if we won, all the places that that happened are good.

Speaker B: So is this what sets them off? And he's, he's like

Speaker A: learning, but like that would take forever, right? Like the to. To do that because there's just so many Go games.

Speaker B: So you have to constrain how many games you play maybe.

Speaker A: Well, so you, you combine it with this other idea, which is the search, right. So when you're playing your game forward, right? So I played 10 games, right. And I have uh. So I think like these 10 random moves I did are good, right? And then I'm going to play my 11th game and before deciding what move to do, I'll play forward.

Speaker B: Yeah, you play it quote in your head.

Speaker A: But we're doing this two stage learning, right? It's like when we get all the way to the end, we say all these things must be good. And then when we're playing in the future, we don't wait until the end to decide if it's a good move, we play it forward. And if it looks like any of the other ones that we did before, then that also must be good.

Speaker B: Like uh, if a, uh, like if a piece of it matches one of the winning scenarios and you're like, oh,

Speaker A: this is a good one, this must be good, right? We're learning faster. It's like we don't have to play all the way to the end of the go. So it's this double layer thing. It took me a long time to understand this. Yeah, I mean, obviously it hasn't learned everything, right? Obviously it's playing like things that are not great. But it can play this situation forward, right. And you can keep playing it, right? It can keep learning and it will get better. And in fact, like, you can do this whole thing.

Speaker B: Yeah, you're like increasing the resolution of the, um, prediction.

Speaker A: Wordle's not that hard. You can actually just figure out the Perfect wordle move by walking through every game. Like, it's somewhat like tic tac toe and then it has a bottom. So now it's running a whole bunch of games and it's learning more. Key one.

Speaker B: Yeah, you got it in the fourth guess.

Speaker A: Uh, not bad. Not bad. Alpha wordle. This is their idea. Right. And it took me a while to understand this. It's like these two things. Like, one is playing against yourself, or in the case of wordle, there's not another competitor. It's just like, they're just playing. But it's still the same idea. Right. It's like thinking through the moves and updating it. And then this idea of searching forward. So they use, um, it's called Monte Carlo tree search. Like, but basically they're randomly playing forward in this tree of words. And which moves they decide to play forward in are basically the ones they think are like, oh, this seems like a good move. Let's play it forward, couple moves and see if it is.

Speaker B: It's like limited foresight with some randomness,

Speaker A: because it's like you don't just want to play the moves you think are good forward and see if they are. Because, like, you start off knowing very little and you're probably wrong. So you have to kind of like,

Speaker B: you have to fail a lot.

Speaker A: Yeah. So they do this and they train, and then, yeah, they're going to play this big game against this Lee Sedol, who's the champion. Um, yeah. So they win the first game against him. At this time, it's inconceivable that they win. Echo. Right. Like, obviously we know that they won, but I was part of this thing called the Human Judgment Project at the time, and it's like they tried to train experts, bet on outcomes.

Speaker B: Yeah, I remember you talking about that. It was because we, um, we both read that book about super predicting.

Speaker A: Yeah, super predicting. Right. So I tried to join that guy's project and like, you could bet on these things. And this was one I bet. Like, I was like, there's no way. Like, they've determined it will be like a million years before they're good at this. So I bet, like, no, Google will lose. And then they won the first game. Um, like, they crushed them. And then during the second game, I don't know what's happening. Like, they're playing at the moves and I don't super understand things well. Um, but AlphaGo is playing white and Lisa Doll is playing black. And it's pretty early in the game. The position's pretty open. And the commentators, because it's like a live broadcast thing, uh, they say that Lisa Dole is. Is doing quite well, maybe even a bit ahead, but it's still anyone's game. And it's only game two, so they don't really know. So then AlphaGo plays a shoulder hit on the fifth line, which I guess

Speaker B: is like, oh, my God, a shoulder

Speaker A: hit on the fifth line.

Speaker B: No.

Speaker A: Yeah. And so people don't really know what to do. Yeah. So it doesn't mean anything to me. Like, I don't know what a shoulder hit on the first line, but go is like a 19 by 19 board and people put their stones down around, uh.

Speaker B: Yeah.

Speaker A: And the. The basic rule you learn, right. Like, it's just like a basic rule is in the opening, in the middle, you play on the third and fourth lines. You don't play in the fifth line. And so AlphaGo played on the fifth line, which is away from the center of the action. And I guess that's for the end game.

Speaker B: Yeah. I mean, we'd have to look up Go terminology, but like, if, if. If it was doing that, there would have to be a, um. There would be a factor there that would throw off the human competitor because they've never played against somebody who did that.

Speaker A: The same as the backgammon when the.

Speaker B: Yeah. It's like, what's going on?

Speaker A: What are you doing? So Lee Sedol, who's like, he's the best at this, but. And so he only plays other, like, amazing. The best people. Um. Yeah. What do you think he does?

Speaker B: I mean, I think that he would probably, like, he doesn't have a strategy for that because he hasn't played against anybody who's made those moves. So it would. He would have to train. There's no counter for it.

Speaker A: It's funny because we know nothing about Go. So it's like. But, uh, I don't understand the rules, but I assume they each have so much time to make a move. And what Lee Stoll does is he, uh. He just gets up and leaves.

Speaker B: He just left.

Speaker A: Yeah.

Speaker B: He didn't finish the game.

Speaker A: He just walked. Like it's a televised thing.

Speaker B: And he's like, like, why did he get out?

Speaker A: I mean, I'm sure he was just shocked. He didn't know what, you know, Wouldn't

Speaker B: you just keep playing? Like, if I was playing chess against somebody and, you know, they made a weird, unconventional move, I just like. Well, I mean, I guess. And then I would just be playing with my own strategy. Yeah.

Speaker A: And it's a live broadcast thing. And he left for 15 minutes. He just walks off. And it's not like they have, uh, alphago. Why don't you entertain us? Like, no, that's a computer, right? Um, so the commentators on the live broadcast, they don't really know what to say. Like, they're basically like, uh, I think this is a mistake. I think AlphaGo is off its rails. What should we do? Um, and so an important thing, right, is AlphaGo has these two systems, right? It has the one that tells it the move it should make, right? And then the other one that we saw that kind of like plays forward, what's going to happen, right? And so this, this weird move that it made, uh, the thing that said, like, what you should do next, basically said you should never do that, right? But then they played forward the rules, and all of a sudden it looked really good. And so it was a rule. It was like, you, like, it knew the same as we did, or Lisa told knew. Like, this is not a conventional move, but then it played forward and it was like, hm, interesting.

Speaker B: This is a good strategy.

Speaker A: Okay? And here is the architect of AlphaGo, who is actually David Silver. He was running the team at DeepMind. Here's what he said was happening at that moment.

Speaker B: The professional commentators almost unanimously said that not a single human player would have chosen move 37. And then we found out that AlphaGo said that there was a 1 in 10,000 probability that a human would have played that move. So it went beyond its human guide.

Speaker A: Then Lee Sedol comes back and he, he sits down, I guess he isn't gone. Uh, and they start playing and like, yeah, the Go games are long. Like, this was considered early games, but it's move 37, right? But over the next 50 moves, everybody starts seeing what AlphaGo's doing, right? Like, this move starts to make sense and it starts to be like a dominant factor. And they're like, oh, I get it now, right? It starts crystallizing everybody's mind who understands Go.

Speaker B: Like, so is that like a move now? Like, now people like, play to the outside or what?

Speaker A: So he loses the game, right? And yes, I mean, I don't know for this specific game, but I know that much, like Backgammon, the rules shifted. People start being able to play against these machines that have new techniques and then they learn from them, right? It's like we were based on our own strategies that we learned over time when we had these biases. But the computer's out playing random games and Discovers new things that we didn't know. Right. AlphaGo wins the match. 4 1. So at least it all won one of them. Right?

Speaker B: 4 1's pretty decisive though.

Speaker A: Yeah. And um, that was March, uh, 2016. But people aren't willing to give it up. Right. And so this fact comes out that, okay, AlphaGo did actually have some human input into it. So there is this online thing called the Kaito Go server. It's been around since the 90s and people play Go on it. Yeah. Um, how many was it millions of games of humans playing that it had trained off. So people were like, well you said the machine has beat us, but maybe it's just Remembering, you know, user 5639 played move 37 on a Tuesday in 1993 and it's just remembering it. Right. You see the same things with LLMs, right? They came up with a strategy to overcome this, uh, which was, uh, let's do this again, let's build this again. But we won't include the human, like throw it out. Right. We'll start again. So um, it's like the same machine as wordle, right? It's like if in my wordle I originally played forward and I played a whole bunch of games and then I start training it and they're like oh, maybe it learned from me. So then they do this again. They call it AlphaGo Zero. It's a good name like Zero because it's like zero human knowledge. So now they play their Alpha zero against the one that beat him and it. And it crushes it. So they built an even better one. And so this, the whole thing, um, playing all these games forward, uh, it only took three days because they're Google and they can just run so much in parallel and play all these millions of games at once. Which is, it's kind of crushing, right? Because you could dedicate your life to being good at Go. And we're like, well we took this thing, didn't even know what go was.

Speaker B: That pigeon.

Speaker A: Yeah. No, it's just like we're giving it and it's like dawn in a box. So it's like running at the super fast speed in parallel and it learns. Here is how DeepMind announced it in the Nature cover in October 2017.

Speaker B: A long standing goal of artificial intelligence is an algorithm that learns Tabula Rasa, a superhuman proficiency in challenging domains. Starting Tabula Rasa, the new program AlphaGo Zero achieved superhuman performance, winning 1000 against the previously published champion. Defeating AlphaGo. Tabula Rasa was also a Video game that Richard Garriott tried to the guy who made Ultima Online. And it didn't really get off the ground and then got scrapped.

Speaker A: Do you know what it means?

Speaker B: Plain slate, I think.

Speaker A: Yeah, yeah, yeah. Blank slate.

Speaker B: Blank slate.

Speaker A: But, yeah, it's like, because it knew nothing, right? It. Like, it started from absolutely nothing. Um, okay, so they removed the human games and it worked. But then they're like, hey, we should remove more things, right? Like, what more can we remove? What do you think?

Speaker B: What more can they remove?

Speaker A: So they did the Atari move. They basically removed the rules of Go. It doesn't know how to play the game much. M. Like my Wordle guy.

Speaker B: Uh, it doesn't know that you can't play the same letter that you've already discounted.

Speaker A: So they're like, what if it just. All it can figure out, like, it just gets whether it won or lost a score.

Speaker B: Yeah.

Speaker A: And then it's like, figures it out the same way the Atari thing did.

Speaker B: Yeah. So it just needs a win condition.

Speaker A: Yeah. And then once they built that version, they had it play, uh, a whole bunch of games. So they train it on Go. Um, they train it on Shogi, which is some sort of Chinese chess, like, game. Uh, they train it on chess. And so then they have it, like, once they built their chess one, uh, they have a play against Stockfish. And Stockfish is like, the most powerful chess thing at that point that, like, Grandmaster's learn against. And it's got all this custom stuff. And then, like, yeah, ah, it crushes Stockfish. And, like, they're making a point, Right? Like, they're, like, doing the Sutton thing. I mean, I think they're also drumming up publicity, but they're like, this thing knows nothing. All it knows is whether it won or lost the game. And you can build the most complicated thing in the world. And you give me some time on Google's compute things in a couple days, like, we'll crush you. They're making a very clear point, um, that they can just learn it. All right. Yeah. So that one was called mu mu zero. Like, mu because it doesn't know any rules. Like, they've removed even more. But. But meantime, since all this has happened, Sutton comes out and, uh, he writes kind of a. A manifesto. I feel like it's. It's always good when you can say, I'm writing a manifesto.

Speaker B: Sometimes they're good, sometimes they're bad.

Speaker A: Like, he could go Unabomber.

Speaker B: Yeah, it's. It's risky.

Speaker A: Um, so he writes this short essay March 2019 that he posts on his personal website, and it's three pages long, and he calls it the bitter lesson. Um, and so for Deep Blue, right, The, the problem was that they had to have that evaluation, um, evaluation function, right? They needed to be able to say, like, how good is this chess position? You will hear people talk about the, the people who are big onto AI right now and who are building LLMs and all that. You'll hear them talk about the bitter lesson all the time. And they'll say things like, uh, are we sufficiently bitter lesson pill? Basically, like, you know in the Matrix where it's like, take the two pills. It's like, have we sufficiently taken the bitter lesson pill? Right? Like the bitter lesson that this button guy wrote has become like the important rule that all these people believe in, right? And so here's how his three pages that becomes this big thing on his website opens.

Speaker B: The biggest lesson that we can read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective. And by a large margin, general methods,

Speaker A: right, that leverage computation. That's his whole point. Uh, uh, I mean, it's a, it's a larger essay, but he's saying, none of your clever methods, right? We don't need 8, 000 rules for how to evaluate a chess thing. We just do this tick tock toe thing. Just let it go like it has a reward, right? Like, let it go, right? Um, um, but he means like, large when he says like a large margin, he means like extremely large. Like whatever you got, however clever you can be, like building, um, a complicated system, it will be beat if we can just figure out a way to get a reward signal and just turn the crank. It's basically saying, stop being clever.

Speaker B: Yeah, because like, computation has reached such a level that if you just can provide it with a reward condition, then it can play out massive amounts of scenarios and, uh, create its own kind of guide, right? So you don't need to actually provide it with any rules. Your, your rules might actually be constrained,

Speaker A: move 37 or whatever it was. Or the backgammon thing that people are like, that's not the way you do it. Um, but they were right because, like, Deep Blue had all these rules, but then, you know, Stockfish had less and was stronger. And then AlphaZero had like, no rules and like crushed it because it's just, it can learn all the rules itself, right? It's just simple algorithms, but they're these learning, like meta algorithms. Um, they win, right? And so he Was just, he was salty about this. He, he calls them sore losers. Uh, and he wins a Turing award. Um, and then he starts using this method everywhere, right? He uses it for, to get good at speech recognition. So there was like 30 years of building amazing speech recognition systems, you know, learning phonetic rules, and then people just come up with this deep learning method, just like, hey, we learned how speech works by just throwing lots of compute at it. And then, uh, right in the middle, in a paragraph about go, he says this thing that kind of summarizes this whole episode, right? This thing that I'm trying to get at. So read, read the next quote.

Speaker B: When a simpler search based approach with special hardware and software proved vastly more effective, these human knowledge based chess researchers were not good losers. He said that brute force search may have won this time, but it was not a general strategy. And anyway, it was not how people played chess. So learning by self play and learning in general is like search in that it enables massive computation to be brought to bear. Search and lear learning, uh, are two of the most important classes of techniques for utilizing massive amounts of computation in AI research.

Speaker A: Yeah, because he's saying like a program can make its own data. Like, uh, we talked about this when we did the LLM episode. Like, oh, they're like we're running out of data. But like here it makes its own data. You just like let it run as long as it can figure out whether it's one or lot.

Speaker B: But you still have to, yeah, you still have to provide it with good and what's bad, right? Like what's a positive and what's a negative condition so that it knows how to evaluate.

Speaker A: And that idea, right, that came from Sutton. Like they were taking his tic tac toe idea. I mean, you have to squint and say like, well, maybe math. Like they're like, well, math could be like a tic tac toe thing, right? We'll just randomly guess and we'll learn backwards. But this is his idea, right? He's finally, he's in, uh, his brilliance has been recognized. There's like 20 years of objections that this of his idea of self play, right? But the bitter lesson, the reason he called it the bitter lesson, um, actually you have the quote. Tell me why it's called the bitter Lesson.

Speaker B: We have to learn the bitter lesson, that building and how we think we think does not work in the long

Speaker A: run, which is super confusing. Building and how we think we think. But what he's saying is like, don't try to like reflect.

Speaker B: Don't ascribe your human conceptual models onto the AI because it doesn't think the way that a human brain does, or

Speaker A: it doesn't need to. Right? You don't need to record the way the best chess player plays chess. You can just let it figure it out on its own, right? Like, don't give it how we think we think. Just let it run. And it's bitter because. Because, well, you want to. You know, you're the greatest AI person or software developer, and you build this complicated system and all these rules, and it's getting better and better. And he's saying, like, no, actually don't be clever. Just throw compute and reward at it, and it will be guaranteed better than whatever you can come up with. Like, it's a bitter pill to swallow to say, like, whatever cleverness you can come up with, a human will not be as good as us. Just figuring out where the reward signal is and giving it unlimited compute. Right? He's like. It's kind of saying, like, let the machine win. Like, the machine is better at this than you will ever be. Stop trying to teach it things. Just let it learn. So then, October, uh, 2021, in the artificial Intelligence Journal, he publishes this thing that I texted you, right? Reward is Enough. Um, and so the people on this one is Sutton, right? David Silver is the other person on it. He was the guy, the lead architect of AlphaGo and AlphaZero and Alpha, um, MU0, I think. Um, but, yeah, the last guy is Sutton, and it's the same Sutton that's all the way through. And in Reward is Enough, they. They make this claim, like, even larger. Do you want to read that?

Speaker B: Yeah. Reward is enough to drive behavior that exhibits abilities studied in natural and artificial intelligence, including knowledge, learning, perception, social intelligence, language generalization, and imitation.

Speaker A: Now he's. Now he's definitively saying, like, reward is not just enough for beating somebody at chess. Reward is enough for knowledge, for learning, for perception, for social intelligence, for language, for imitation, for. He's basically saying that's it. Like, all of, um, all of intelligence is contained in beating something, a reward and seeing what it learns.

Speaker B: Yeah. I mean, again, it's. It's still reductive because we do undertake some endeavors not for rewards. Like, I don't think there's a tangible reward to figuring out what dark energy is, but it's something we haven't discovered. How would you ascribe a win condition to that?

Speaker A: Uh, and Skinner. So Skinner, you know, was the. Skinner was a psychologist, and his methods got left behind a Lot. Although they were powerful because, you know, you think of Freud and, like, wondering why people do things. Like, Skinner never wondered about that. He was like, no, you press the button, you get a piece of grain, right?

Speaker B: It's enough for the. A pigeon. Yeah.

Speaker A: But he had nothing to say about individuals and how they worked because people to him were like a black box. Right? You give a reward and then the pigeon does the thing.

Speaker B: And people aren't pigeons, Right. They're a little bit more complex.

Speaker A: But the interesting thing is these AIs are black boxes as well, right? Like, we don't know what's going inside the neural net. Nobody care. So it's very interesting how it actually follows from this guy. He's like, I don't. I don't care what the pigeon thinks about it. It's like I trained it to do the bomb guiding. It's like, I don't care what the neural network does now. It's just better at chess than you will. And. I don't know, it's many years later and he won the Turing Award and he. He wrote this, um, and it's affected the whole world. But he went on this podcast about AI recently, 20, 25, he went on the, uh, uh, Dwarkesh Patel podcast.

Speaker B: Yeah, I know.

Speaker A: Yeah. And so he interviewed him because he's like, oh, my God, look at AI so big. And you finally got your comeuppance, you know, what do you think? Like, we've all finally learned your bitter lesson and we're building these things. Will you read it?

Speaker B: Large language models are about mimicking people, doing what people say you should do. They're not about figuring out what to do. They have the ability to predict what a person would say. They don't have the ability to predict what will happen.

Speaker A: So they thought they learned all his bitter lesson, but he's saying, like, no, man. You built an LLM by consuming all of human knowledge. That's not AlphaGo Zero. Right. You actually just built something that's pretending to be human. Like, his way would be, like, it starts with nothing. By his thinking. Then we would be at the big blue era of AI. It's really good at doing all these things, but it's encoded all this human stuff. And the next step should be like, can we get rid of all the human stuff? Because right now the LLM isn't going to do the move. 37. Right. It's trained on. On human knowledge.

Speaker B: Yeah. Um, it's also trained on, like, what it thinks you want. Like, I think that he's on the right for like a, like a general artificial intelligence. Because he's talking more about, like, don't teach it the content. Teach it how to discover the content to learn.

Speaker A: Teach it how to learn. This guy who worked with him and who worked on the Alphago team, Julian Schnitzer, uh, he's at anthropic working on LLMs, building large language models. And he said, uh, after that statement, he said, like, oh, no, you're actually wrong, Sutton, because as we discussed the other time, these LLMs are, they have these RL loops in them now where they're like, teaching them reward based on math and based on whatever. So he said, no, actually, Sutton, uh, you're wrong. These LLMs are doing exactly what you're saying. I guess they're built on human knowledge, but we're still, we're giving them this reward loop and they're learning. Okay, but now I'm gonna go dark, uh, on this whole thing, right? Um, because, like, I feel like this goes further than all of that, right? Because he's saying, you know, reward is enough. Well, enough for what? Saying, well, it's enough for everything.

Speaker B: Like, I think that there's room for nuance there. I mean, most of the things we do for reward, some things we don't have a discernible reward, right? Uh, like we eat to stay alive, we work to make money so we can be alive. Right? But there's, like, pursuits that don't have any kind of discernible reward or reason why someone would do them.

Speaker A: True. But, uh, I feel like what he's saying is like, if you can come up with the scoreboard, which is basically what the reward is, right? It's like you need a way to score something, then the computers will. Will beat you, right? And so it was easy to come up with a score for chess, and now the computers b U h, it used to be this imagenet. It was in the last episode. It was like identifying animals in images, and computers couldn't do it. But then they, they built this big data set and they were like, it has to guess and say, like, is this a cat? Is this a dog? Once they had that scoreboard, computers just learned and got better at it, and so they're way better at, ah, identifying things in images than humans are. But now there's like, SWE bench, right? It's like a benchmark for programming tasks. It's like, you pull down this task and it's like an open source bug, and you have to solve it, and then it scores you on whether you're good or bad. Problem is that's a benchmark that now a computer can do its reward game on. And so, like, the larger, bitter lesson is, like, we're in trouble if there's a reward signal, like the SWE bench, like, a, uh, computer will be able to dominate us at it. The dark version of reward is enough, is that reward is enough for a computer to crush you at what you do. And that's the bitter pill. So for 70 years, the job was to be clever, right? Like, I'm the best plumber at solving a problem, or I'm the best software developer. But I feel like when I try to extend his analogy, it's like Sutton's whole life says that, um, these things are actually clever without us now that we've built them. Like, the advantage of being clever is kind of escaping us that what. What's valuable isn't what you know anymore. Like, it feels like the world's changing. I don't know. It feels bleak, right?

Speaker B: I mean, it could. It could be, yeah. I mean, rewards are also dynamic. They change a lot. Um, sometimes, you know, like, uh, what is today's reward might be different tomorrow by being able to redefine it. I think the thing is, like, when we. We work on something, we define the rewards. It comes from within our. Our own kind of minds. Um, it has yet to determine its own rewards.

Speaker A: So that was the show. I know. I. Exhausted, Don. Um, so you walked in not knowing, actually. So I walked in not knowing what subserving meant. Like, I had just never heard that term. Oh, yeah, subservient. I had heard, but I think we figured it all out. Whether we agree with or not, I think is a little bit undecided, but. Yeah. Thank you, Don, for letting me go through this all with you. Here's what I would say about Sutton. I think that he's right. If you can come up with these rewards in this box system, the computer will win. But as humans, we just come up with new boxes. And so this guy who did the backgammon right after the computer beat him at backgammon, like, he put out a new book, right? Or you invent a new game. And the humans, you know, the way it is now, like, we're setting the limits of what these boxes are, right? And so I think, like, you just have to keep learning and growing. If you try to compete head on head at a limited game with the. With the AIs that we have right now, you will be beat.

Speaker B: Yeah. Don't worry about it when it can create its own box.

Speaker A: Yeah, yeah, you just make a new box, right? I'm not going to try to write SQL, the most performant SQL anymore. Like, I'll just ask cloud code. It'll be better than me, but I'll. I'll. I'm building the things.

Speaker B: Yeah.

Speaker A: Yeah. So until next time. Uh, yeah. Stay curious. Learn new things. Uh, thank you so much for listening.