Ep 90: AI Pioneer Jürgen Schmidhuber on the State of AI Today

Speaker A: It sounds like you think a crash is coming.

Speaker B: There will be one of these stock market crashes.

Speaker A: What about like robotics?

Speaker B: Robot hardware is really inferior compared to human bodies and there's no human made technology that compares to this cad.

Speaker A: It seems a lot more than a hardware problem.

Speaker B: You can't have AGI without hardware like that. You can't have AGI just behind the screen.

Speaker A: Are we kind of close to that?

Speaker B: It will come, but it will take, uh.

Speaker A: Jurgen Schmidheber has been cited as the father of AI by the New York Times, Forbes, and more. He is behind some of the most important advances in the field that really power the AI revolution today. And it was a real privilege on unsupervised learning to get to sit down with him and talk about everything that's top of mind in the ecosystem today. We talked about what's missing in models today and what he thinks is required to get artificial scientists that can push them forward. We talked about why he thinks today's capex boom is massively overdone and why he's very optimistic on AI technology, but deeply pessimistic on the model companies and how he doesn't think recursive self improvement will, will actually be a moat for the companies. We also talked about AI safety and why he's notably less worried than a lot of others in the field. It's just an awesome opportunity to get to sit down with a legend in the field and ask him all these questions. I think folks will really enjoy hearing his perspective. Without further ado, here's Jurgen. Well, thanks so much for, uh, for coming on the podcast. Really excited about this.

Speaker B: It's my pleasure, Jacob.

Speaker A: Well, I feel like there, there's so many different things I want to talk about today. You've obviously been a pioneer of, of a ton of different parts of this current AI moment. You're. I figured where I'd start at the highest, um, level was, as I understand it, you've had this goal for a long time, which was to build an AI smarter than yourself. How close are we to that right now?

Speaker B: From a cosmic perspective, we are as close as we, uh, were in the 1970s when I first formulated that wish. So we are very close. But is it going to be a couple of years or a couple of decades? I'm not totally sure about that because true AI is not just the AI behind the stream, which is working very well and which is passing the Turing test. Now, true AI is also, you know, real robots, real machinery outside of the screen, in the real world, in the physical world, uh, and that's not working as well. So the hardware in the real world has lots of limitations so that human bodies don't have and we still have a way to go to be able to compete with human bodies and physical AI.

Speaker A: Have there been any current, you know, AI results over the last few years that have surprised you?

Speaker B: Not really. Not to the extent that it surprised uh, people who had no contact to you know, neural networks and artificial neural networks uh, in the previous decades. And suddenly there was a, a chatgpt moment and suddenly people um, started being interested in that thing which they had never seen before. So they, they didn't know that there was a long history uh, of large language models and an even longer history of basic um, insights and algorithms for training these large language models that goes back to the previous millennium. Um so for a guy who was in the, in the center of all of that, it was much predict that than for someone who had totally different interests.

Speaker A: Well, you know I definitely want to hit on um, recursive self improvement in meta learning because I feel like you've, you know, it's been a huge focus of yours for a while. It obviously seems to be a main focus of a lot of the major labs these days and you pioneered a bunch of the research here. How do you kind of articulate the, you know, where we are today on the, on the path of getting to RSI and kind of what still needs to be solved.

Speaker B: So in 1987 this was about uh, using meta evolution, evolutionary uh programming through, for evolving better programs that learn to do better kinds of evolution, meta evolution I call it. And it was very Darwinistic in many ways. And so over time you had better and better uh, learning algorithms, learning to combine code from previous programs in, in better and better ways, uh, to solve problems better and better. And then um, in. Well yeah, and, and in 1994 we had um, reinforcement learning, um techniques, self referential, um, machines that uh, were able to, that basically were using a universal uh, programming language to generate arbitrary self modifications of the code that was running the machine interacting with some environment. And then in uh, 2003 that was a uh, mathematically optimal way of um, generating self improvements by a machine that has some software and it's interacting with an environment. And this environment um, you know, sometimes punishes you or provides a reward and you want to maximize the sum of all the rewards in your life until the end of your life and you want to minimize the sum of the pain signals. And then there's an initial software in that machine and this machine then um, can in principle write programs that modify the initial software. But before it modifies itself like that, it first has to prove, which means there's a proof search in the software. It has to prove that um, the modification that will be caused by the execution of this particular program is useful in the sense that it will lead to more expected reward than um, the alternative which would be not to execute this program. Um, um, and then it has to um, generate a formal proof which was called the Godel machine. It is less practical than certain other things that we did. Like uh, you know, um, neural networks that change their own weight matrix by running a learning algorithm on the network itself. That is what we started in 1992 and that's currently more or less the most popular kind of self modification and

Speaker A: self reference today as you kind of alluded to. Like one of the limitations is you have you know, humans defining the start and end of these uh, of these trials versus you know, uh, kind ah, of a mathematical way to determine hey, is this going to be helpful beforehand? And I guess obviously the trade off of that being that it's very compute intensive, right to do, to do some of these proofs. I guess as you think forward to what the path to RSI might be, do you think it will involve this kind of uh, ability to do these proofs beforehand or is it like the modification of weights or uh. Yeah I guess. What percent likelihood would you think that the answer lies in one of those?

Speaker B: Yeah, so I would say uh, most of the current um, self improving systems are scaled back, uh, versions, you know, toned down versions of the girdle machine of 2003, the mathematically optimal thing. And they are more like what we had earlier, you know, neural networks where um, you have the weights and the program of a neural network is basically the weight matrix m outside program. And then you have um, certain types of neural networks that are general purpose computers, recurrent networks for example. They are general purpose computers because on a recurrent neural network you can implement the processing unit of your Apple laptop, something like that. So if you have certain instructions that allow uh, you to modify um, the weights itself then um, you can basically run arbitrary learning algorithms on the network which has to see the errors or negative reward signals that are coming in. So that has to be part of the input, which is essential. And that's all what we did in the early 90s and back then compute was so expensive, 10 million times more expensive than today, that we could do only little tiny toy experiments. But today you can really show nicely that methods uh, like that can learn to generalize and learn new tasks much faster than if you don't have this meta learning capacity. And um, and this is currently um, the most popular way of doing uh, recursive self improvement. However, one has to admit that it is limited because there uh, the learning algorithm is invented through gradient descent. So everything is differentiable and then the whole network allows through gradient descent to generate weight changes that are um, better than what's caused by grain descent. Um, that uh, has the limitations of grain descent. So it's not like the optimal good machine, but it works really nicely in practice.

Speaker A: I uh, think a big question people have around recursive self improvement is that it is in retrospect is it going to feel like hey, it was this gradual and boring improvement or is there some like huge discontinuity around the horizon where suddenly you know, models take off in capabilities. What's kind of your gut instinct around that when we get there and they're kind of looking back?

Speaker B: Yeah. Well from a cosmic perspective it will look just like a uh, you know, like a stick. There was no self improvement and no real AI and certainly there was um, AI. But then from a cosmic perspective this is also true for all of civilization. So civilization roughly 13,000 years ago and before that there was no artificial intelligence. And then only 13,000 years later there was artificial intelligence. So and, and, and basically the first guy who had agriculture 13,000 years ago and domestication of the animals, there was almost the same guy who had the first AI and um, and, and the 13,000 years of civilization, they are just one millionth of world history, which is about 13.8 billion years. So it's just a, a flash in world history. In hindsight it will look like that civilization was almost occurred almost at the same time where AI occurred. Because over these past 13,000 years more and more stuff was automated and more and more of agriculture was automated and more and more of human labor was automated. And at some point thinking started to become automated a couple of hundreds of years ago. So the first calculators and then the calculators become, became less expensive and then they became faster. And now you can calculate much more than 100 years ago for the same price. And then um, suddenly it was there. Now um, from, from this global perspective it's really like um, there was nothing and suddenly there was a lot. From the perspective of a guy who is living through that age, um, it looks like a lot.

Speaker A: I think one of the hopes I guess of RSI Is that uh, you know it may end up compute intensive over time. Right. To actually continue making some of these uh, developments. I guess we'll, we'll see. But that uh, that, that I think is, would be, would be one of the hopes there, right?

Speaker B: Yeah, absolutely. And um, whenever you are uh, talking about intelligence you basically are talking about laziness. So an intelligent being wants to be lazy, it wants to achieve whatever it does, but with, with the least possible effort, with the least possible energy consumption. So all of um, our self um, improving systems, they, they have this extra reward for, for being efficient. In other words they get um, they have an extra cost for every time they wake up a neuron and use energy to wake up that neuron or use energy to do other things. So all the computational costs and um, the other energy costs have to be taken into account uh, in the function, in the objective function that our self improving systems are optimizing. Uh, and to the extent that they are good at that, they will do the same thing with less and less resources, computational resources, other resources. So um, a natural consequence of intelligent behavior is that whatever is being done is done more and more efficiently.

Speaker A: As you reflect on uh, maybe uh, the broader work that's going on at the AI labs. I mean obviously there's tons of money and compute going into uh, a bunch of the research questions that the labs are going after. And I'm wondering if you were, if you were kind of running one of these labs or you know, I'm sure you talk to folks there or think about advising them like what do you think they're maybe uh, that you advise them to do differently today from what they're doing.

Speaker B: So um, today you use large pre trained systems, large language models that already have read um, a lot of papers about coding of course. And then um, now a modern way of using um, a pre trained model like that is you let it code something and uh, you only improve the way it codes, um, or recodes its own code. Uh, so that is now a pretty obvious way of doing it. And uh, quite a few labs are interested in exactly that. And of course you have to have safeguards. It doesn't really write its own code in a way that is totally stupid and makes uh, things worse and uh, better rather than better. But um, there are ways of dealing with that.

Speaker A: No, ah, that makes, that makes sense. I mean, I guess. Is there, is there anything you'd be kind of doing differently there?

Speaker B: It's a reasonable approach. Um, it's not the most general approach in the sense that um, if you rely on anything that is pre trained uh, on human generated data, then you are ignoring many of the other possibilities. So look at our current large language models. They are super biased towards humans. Why? Because they are trained on all the data on the World Wide Web. All the data on the World Wide web is there for the only reason that at least one guy or one person at some point thought this is interesting from a human perspective. And then all this material that at least some guy thought is interesting is used to train the systems. And they ah, are for that reason super biased towards human language, towards videos that humans find interesting, towards behavior that humans find interesting, and so on. So they are very aligned in a certain uh, way with humans. Maybe they are more aligned with certain humans than with others. Nevertheless, uh, there's a tremendous human bias. Now of course, if you have an artificial scientist who is living in some unknown environment and tries to build a model of the world by just predicting the consequences of its actions and then use the model of the world for planning, um, such, a, such an agent will um, have to create through its own actions the data ah, that trains the world model. So suddenly um, you, you have something like an artificial scientist who through its own actions generates the data on which the um, world model is being trained. And, and this is much more like what humans do, what babies do. Babies don't learn by downloading the web or something. No, they learn by pre predicting the consequences of their actions. So if they move their fingers like that, then the video changes which comes in through the cameras and they learn to predict these um, these changes. And that's how they learn about the physics of the world and about how the world works and about how they, their fingers work and everything. And they, they are being trained on a lot of data that is not on the World Wide Web, you know, that is collected through their own experiments. And, and it's exactly the kind of data that the baby needs to better understand what it can do and that it needs um, to maximize its own reward. So um, now you see that all of the data that is collected on the World Wide Web, that seems to be a lot, but it's just a tiny, tiny, tiny fraction of all the possible data that you could collect out there through your own experiments. So the future of um, um AI is going to lie in such systems that through their own actions, through artificial curiosity as I called it in 1990, collect all the data that ah, is used to train the world models. And these world models, they will not depend on human language. And they will be very focused in many ways on this particular robot that is collecting the data. And yes, of course it is going to communicate with other robots living in different environments, is going to incorporate that knowledge such that it can better generalize and so on. But you know, um, suddenly you have a situation where the systems are going to be much less human biased.

Speaker A: It was interesting. I mean a lot of what you just described, obviously there's companies that are trying to build, you know, automated labs for material science discovery or for biology or even in robotics. There's all these folks doing teleoperated data. But obviously today it's kind of humans deciding what the data that should be gathered is or designing those kind of experiments and trying to feed it back. And I guess the dream is to have you know, a, ah, fully AI driven loop of that, of that discovery. Right?

Speaker B: That's true. Yeah, yeah. 20 years ago I, I wrote this paper about um, the formal theory of fun and creativity which, which is about exactly that. So what should an, what should an artificial scientist or any scientist or any artist or any comedian do when um, when its main needs, you know, like eating three times a day are satisfied and um, and it has extra time to do stuff? What do you do? Well, you listen to music, or maybe you compose your own music, or you generate art or you are a scientist who not only is uh, trying to solve problems given to you from, by, by other people. No, you also try to invent your own new problems, ask your own questions, not just answer um, questions given to you by somebody else, but invent your own new good questions. That's what scientists do. Um, and all of science is about two things. Not only, um, taking an existing question and um, investing a lot of time into solving it and finding an answer. No inventing the good questions. And uh, the basic um, principle is very simple. So an artificial scientist, or any scientist, in my point of view, a human scientist as well, is driven by one simple thing which is try to find through your actions, your own self invented experiments. Data, um, that has the property that in the data uh, there is some regularity, some pattern that you didn't know, but can quickly learn, that you didn't know, but you can quickly learn it because it's at the limit of what you already almost understand. Near the horizon of what you don't know and what you know. Uh, and then you generate this data through your own actions, through your experiments, sequences of actions which are experiments. And the data comes in and if there is something interesting in the data, what does that mean. It means that there's a pattern in there which you didn't know. All patterns mean there's some way of compressing those things in space and time. We are not talking about time right here to make things not too complicated, but, um, to compress the patterns in a way that you didn't know before. Which means that before you understood the regularity and the pattern which is coming in, you needed so many internal bits and hidden units and so on to encode it. And afterwards, after you learn to see the pattern, after you learn to compress it, you need only so many. And the difference between before and after, that's the fun that you have. You know, that's uh, just a real number which says how much internal joy does the scientist now have from recognizing, from realizing, oh, there is uh, regularity that I didn't know in the data which is coming in. It tells me something about gravity. And then this becomes the reward of the controller who's generating the actions and that lead to the data, which means now the control is motivated uh, to, you know, generate more and more experiments that lead to insights about the wild that uh, it didn't have before. And everything that it understands becomes boring. And then, uh, it wants to create more complicated experiments to, you know, after it has grabbed the low hanging fruits, more um, complicated expands and maybe the, the same, the baby which first learned about gravity when it was a little being, maybe a year old or something, maybe 20 years later, the same baby is working uh, at the particle Collider at the CERN and is part of the team that discovers the Higgs boson or something. And the only difference is that the experiments are more expensive.

Speaker A: And I guess is that really the blocker? As you think about what stands between us and getting to this AI scientist, obviously the ability to stand up experiments and do them, uh, both at an affordable price point, but also kind of feedback into models is something that a lot of people are working on, but certainly remains a blocker. And it also seems like there's all sorts of algorithmic problems to be solved, uh, to actually create, uh, this setup that you said. How do you think about what needs to be solved to make this AI scientist a reality? And then how soon do you think we'll get there?

Speaker B: Yeah, well, we have simple AI scientists. We have had them for a long time. It's just um, that maybe they haven't seen their ChatGPT moment yet, you know, ChatGPT. When was that? Uh, 22, 23.

Speaker A: Yeah, end of 22.

Speaker B: It was based on stuff that was really old. We don't have the same um, kind of chatgpt moment yet for these artificial scientists. But the artificial scientists to a certain extent they already exist and they are being used and lots of special applications. For example in chemistry you have lots of pairs of inputs and outputs and you train your neural network to predict, to predict these um, these new substances given the previous inputs. And then over time if you show it millions of experiments, it learns to become an artificial chemist, an intuitive chemist. So it's not a uh, chemist who understands all these reactions from first principles, from valence electrons or whatever? No, it just becomes um, uh, an intuitive chemist that better understand, understands what can be done in chemistry. And then you um, have certain objectives. Um, for example you want to uh, create a material that is um, that is um, twice as efficient m against a certain maybe, maybe an insecticide or something that is twice as efficient as the most, the best thing known so far. And then you can say okay, let's have um, a desired output, uh, like that which uh, encodes that it should be twice as efficient as the best thing I know. And then you can work the whole chemist backwards and can um, look at the input time, can say how much should I change my experiment which is visible at the input side to get this thing that I would like to see, um, to fulfill my wish. And then um, the chemist will basically give you a suggestion and do you

Speaker A: think within a decade we'll figure out AI chemistry.

Speaker B: So at Kaust we have an interesting project where the goal is to um, use certain patented uh, structures, metal um um MOFs they're called. And um, the, the goal is to extract um, carbon dioxide from thin air which is important for improving the climate. And at the moment all of that is very expensive. And the goal is to make it so cheap that um, that you really can make a dent and maybe um, maybe improve the global warming situation. Um, so that is one of the potential applications. Um, and there are lots of um, applications and all kinds of um, chemistry, uh uh, fields.

Speaker A: What about robotics? How do you characterize where we are? What's been happening there? Um, everyone has to stream of an at home robot. Uh, uh, are we kind of close to that?

Speaker B: I remember again in the 70s when I told my mom about the future of AI and how AI is going to colonize the entire universe and she said just um, build me a robot that cleans my uh, kitchen.

Speaker A: The age old dream.

Speaker B: Yeah, exactly. Um, and back then that didn't work and it still doesn't work because um, robot hardware is really inferior compared to human bodies. And there's no handmade technology, no human made technology that compares to this hand. This hand is full of sensors, millions of little sensors and little cables that connect it to the control center. And um, and I wouldn't even know where to put all these cables in an artificial hand. And the, the crazy thing is you, you harm it, you cut it and it starts healing itself. This is super advanced technology. We have nothing like that. And man made tech. Um, and that's the reason why the robots of the movies are ah, all played by humans. Because humans are much better robots than um, the robots.

Speaker A: But it seems a lot more than a hardware problem, right? I mean even with the hardware we have, if we had good models, I'm sure we could do way more. Right?

Speaker B: But you know, you can't have AGI without hardware like that. You can't have AGI just behind the screen. Yeah, you can have a superhuman chess player behind the screen screen and something that passes the Turing test. But if it does master the real world, it's not an AGI, you know, it's just a um, maybe a fancy text editor behind the screen or something that has an idea of how this, your world should move or whatever. But it's not the real thing. And um, if you want to master the real world through a uh, real AGI, a physical AGI, so much more has to be done. So our robots have to become much better than the limited stuff that we have today. How much longer will that take? We can easily predict how um, compute per dollar is going to evolve. It's probably going to um, stay like um, what you have seen for decades now, factor of 10 every five years, roughly like that. Which means also that the guys who are investing $1,000 billion into GPUs for data centers today, within the next five years they are going to lose $900 billion. There's no business model that um, can recuperate at last, which already is an indication of the coming crash maybe. But um, in, in robots and, and robot technology, how long is it going to take to, to get something that is compatible to this hand, which can do both, um, strong grips and very delicate uh, you know, finger movements that ah, just manipulate tiny little things in a, in a way that is inf. Um, what, what kind of robots can do that is at least for me, much harder to predict. It will come, but it will take uh, maybe it will not take just a few years, it will take another few decades.

Speaker A: Maybe Well I do want to switch over to the business side because you said obviously something very intriguing there, and I've heard you say this before, that like you know, uh, this massive investment in AI data center build out, you know, ultimately with the improvement in compute performance, you're investing a ton up front in something that's going to be, you know, kind of outdated and, and worth far less five uh, years from now. The like counterargument or what folks would say to that is that there's just going to be like infinite need for compute and that ultimately like we're going to be you know, bottlenecked in our ability to, to produce enough compute such that like even if you have hardware that's legacy and way less efficient, it's actually going to still, people will still want to use it because there's just going to be such demand to run inference for uh, for all of these different, interesting, at least today digital and in the future physical use cases. So even if it's less efficient or not the best compute, it's like every chip on the planet will need to be used in some way. What's kind of your reaction to that?

Speaker B: Yeah, maybe there will be more and more demand for compute, but somebody has to pay for it, right? And if it turns out that those guys who are currently paying that they are um, losing m a lot of money, you know, at the moment, um, a couple of um, companies um, are investing hundreds of billions per year. Uh so in total maybe a trillion per year or something like that into GPU is for data centers. Um, and these companies which used to be nimble software companies, uh, and had a little team improving some shitty operating system and then rolling it out for billions of people who had their own phones and their own computers to run the operating system, suddenly they are providing clouds and data centers and suddenly they have to become like utilities, like electricity companies and they have to invest in nuclear power plants and gas turbines and whatever. And certainly the cash flow, the free cash flow of these companies goes down from 100 billion down to 10 billion or maybe minus 10 billion, uh, like for some of these companies. So at the moment um, while everybody is trying to get market share, um, these, these um, services are, are really not um, efficient in the sense that uh, the guys who are providing the services, they are losing a lot of money. It doesn't show up in the price earning ratio because the free cash flow which really should be considered is not showing up there. But uh, all these companies are getting less and less efficient. So at some point they will have to stop. You know, now they're taking on debt to finance even more data centers. And this will be possible only to a certain extent. And then these companies will become less and less valuable so they, because they misinvested uh, their money. That means that at some point, form or later, uh, if you can't pay for all these services, if the entire economy is not set up to pay for all these services because COMPUTE isn't cheap enough yet and because people want to compensate for it by buying even more computer computers, um, if you have a situation like that which, which doesn't lead to um, to a profitable um, economy then it's going to disappear um, due to the laws of supply and demand.

Speaker A: I mean I guess there's kind of like two, two questions embedded in that. Right? One is on, on the inference itself. You know, are there enough use cases out there that are valuable enough for companies to, to want to pay uh, you know, well, well above and beyond uh, the kind of cost of these data center build outs and certainly with AI coding it seems like there's you know, as that is as the first market with a ton of fit, it seems like there's been at least uh, some use cases there that uh, that certainly meet that bar, you know, certainly. On the training side I think there's a big question, and I know you've talked about this before and I think our listeners would love your thoughts um, on whether these you know, closed source model providers who obviously are spending a ton on training, whether there's a business there. Right. Or whether you know, over time open source models just catch up and uh, ultimately you know, I think there's, there's two sides of that argument but would love for you to share just how you think about uh, for the closed source model providers. Like is there, is there a business staying, you know, maybe three, six months ahead of the open source models?

Speaker B: Yeah. As you say the open source models are uh, one of the major reasons why the big companies cannot simply raise the prices because they are so close to behind and somebody, some commercial um, large language model is, breaks another benchmark, uh, record here. But a few months later there's an open source model that catches up with that. Which means um, there's enormous pressure on pricing which means um, all these expensive uh, data centers and GPUs, um, where these companies invested in and, and took on debt to pay for that they are not being profitable at the moment. You know, so a stupid way would be just wait a little bit, just wait for five years and compute is going to be 10 times cheaper. You will be able to do the same thing for one tenth of the price. Or wait 10 years and you will be able to do the same thing for 1% of the price. But then of course um, the big companies say oh, but if I do that, if I wait until the others do it, then I, I will lose maybe my market share or whatever. Or um, I miss out on the opportunity to be the first to have a true AGI or something like that. And in that case um, once I have a true AGI that can solve everything, then I can take over the world and it will, and the investment will be nothing compared to the value of that. But um, but I think all of these um, these outlooks are over optimistic and um, um, probably we will see that the current way of um, investing a lot in computers that aren't fast enough yet is going to backfire.

Speaker A: Um, I think related to our earlier conversation, I think there's this belief that hey, if you can be the first to recursive self improvement or the first to have models that make it easier to develop the next models, that it becomes this self perpetuating cycle that makes it really hard to catch the folks uh, that are in the lead. What do you think of that?

Speaker B: Not really. Because many um, of the guys who are interested in self improvement, they are open source guys. Uh, and everybody's cooking with the same water. Everybody's cooking with the same water. And almost all of the important algorithms in AI they were not invented at big companies or whatever. No they are, they were invented at little labs and um, with, without much funding and especially uh, the requested self improvement uh, stuff and the basic ideas for that all ah, come from little labs, you know, academic labs and uh, any company that wants to somehow grab that and make it its own doesn't have a moat because there are all these other PhD students out there who are super excited about um, uh, because of self improvement and um, and there's no way to you know, keep a little advantage there, uh, um, for long enough to make a lot of money. So I think um, I think it will be very, very difficult for the, for the huge companies to be very profitable in this business.

Speaker A: Yeah. Because basically like these ideas, you know, uh, permeate throughout the broader ecosystem, you know. And so as a result, uh, you know, when one lab gets to some level of rsi, it'll be kind of out in the open as well. And um, unless there's some massive compute need required for it or some Kind of data advantage to distribution. It'll just be kind of available, you know, more broadly.

Speaker B: Nobody knows exactly. Maybe um, you are lucky and you stumble across um, a really good new way of um, conducting incremental self improvement, um, using the um, enormous resources that your particular company has for now and then achieve uh, the AGI that uh, you need to take over the world stock market and um, finish uh, the whole game. Of course there's a remote little chance of that happening but uh, all the indications point the other direction and instead AI is getting cheaper and cheaper and um, there are more and more people who are, you know, involved in developing new AIs and many of them are poor PhD students somewhere, um, haven't even founded their companies yet and so on. And then um, uh, you really will have this AI for all situation where you know, uh, everything that seems um, impressive today, 30 years from now, uh, will seem trivial because you can do a million times as much for the same price. And um, and it will be, you know, it will be just like with, with the smartphones.

Speaker A: Yeah, but it sounds like you think a crash is coming.

Speaker B: Well, uh, not a crash in the sense of a civilization crash, but um, there would be one of these stock market crashes because at the moment I think there's a lot of misallocation. Of course the stock market is going to learn from these misallocations and it's going to pursue different um, approaches. But um. And it's not going to be the end of civilization or something. No, but it's going to be um, a renormalization of what we currently have because at the moment we have just super expensive companies which don't really have much to offer.

Speaker A: I feel like you've kind of publicly been less worried about AI safety today than maybe some of the uh, other folks in the community. And I'm wondering, these past years, has anything changed your mind at all or are you kind of still in the same camp?

Speaker B: I Remember in the 2000 and tens we had lots of these uh, safety conferences and alignment conferences and um, people writing letters, um, that certain types of recursive self improvement should be forbidden. Uh, of course I never signed any of these letters. Um, um, but back then it was already a big deal, at least for the machine learning um, specialists. And how can we prevent uh, AIs from, from becoming dangerous Now? I think all of these uh, approaches were misguided in many ways and it's kind of obvious that, well, uh, first of all this whole alignment business, uh, means that there is somebody who Decides what is the nature of this alignment, how do I align these AIs to the um, needs of humans. But if you have 10 different humans in one room, they all will have different needs and they will have different opinions about what is good for humans and what is not good. And um, and, and so this whole perspective is dominated by the idea that you give one objective function to this AI which is supposed to optimize, you know, and that's then the alignment um, that you create through that. On the other hand uh, for you know since, since 1990 or so we have built these artificial scientists which are not really aligned much with anything because they all the time, they invent their own new uh, objective functions, they invent their own new problems. All the time they um, they, they change their own objective functions. And really starting in 1990 we had uh, systems like that. So this whole premise of uh, having systems that don't change their objectives didn't make any sense to me. It was very naive in my point of view. So I didn't sign those letters. I just couldn't. Um, and today um, we have these extreme cases where people um, are using AI to fight against each other. If you look at the um, war between Ukraine and Russia, it's about AI based drones on this side fighting against AI based drones on the other side. And there's no alignment whatsoever. And of course you cannot expect in our world, you know, a universal government that is going to align all these AIs because all these different secret services and militaries, they all have different objectives which are often totally conflicting with each other. So um, so it seems to me that, that um, that all these um, efforts of the 2000 and tens were kind of naive. Now on the other hand I think that um, and, and if you want to get really smart AIs you have to give them the opportunity to set themselves their own goals. You know like little artificial scientists that, that um, now ask a new question about what happens if I do this and um, how does the world respond if I do that. So only if you give them the freedom to ask their own questions and to solve their own self invented problems, only then they will become really smart. On the other hand they will become less predictable. That is of course um, the issue that is being addressed here. And um, and, and now people are asking uh, but, but in a, in a world like that where you have all kinds of AIs that set themselves their own goals like they have done in my lab for many decades, uh, isn't that super dangerous and so on. I think it's not going to be more dangerous than what you have now with humans. Humans also um, set themselves their own goals all the time and, and come up with new questions to, to ask. And um, and yes it's true that the kid, my kids are unpredictable. I'm not sure what they are going to do uh, in the future. But um, at least I can contribute as a, as a parent uh, to um, to making them useful members of society. You know, and whenever they come up with, with bad experiments, like for example, let me take this magnifying glass and, and focus the sunlight on this little ant here, that I'm going to punish them and say that's bad, you shouldn't do that. And um, and that's how I taught my kids to become reasonable members um, of society. And the same thing we are going to do with our robots and machines. And in the long run once they are much smarter than we are, uh, I think we have a certain, we can expect a certain kind of protection through the fact that they are scientists because artificial scientists will be super interested by life and by their own origins and civilization and by everything that led to their current state. And they will, they will be fascinated by life and they will be greatly motivated to protect the source of interesting patterns and, and um, uh, and they will be motivated to uh, protect it rather than destroy it. So from, from this high level perspective I think there's a very strong reason why you shouldn't be too afraid of uh, you know, Arnold Schwarzenegger Terminator scenarios.

Speaker A: We always like to end our interviews with uh, with a bunch of like you know, quick fire questions where we shove a bunch of things into the end. Um, and so maybe to start actually I'd love just give our listeners context on. You've obviously done uh, so many interesting things over your career. How are you kind of splitting your time today and thinking about like what's most interesting to work on going forward?

Speaker B: I'm a conservative guy and uh, I think the most interesting way of going forward is still the one that I pursued in the 70s and 80s when I tried to build this general purpose AI that learns to become smarter than myself such that I can retire. I'm still working on the same thing.

Speaker A: You've obviously have this network of a ton of uh, amazing folks that have come through and worked under you and then gone on to do great things uh, as well. In addition to you, what have you learned about what makes a really good researcher?

Speaker B: Many of the best um, PhD students, um, that I have had the pleasure to work with. They usually focus on a, uh, particular question. So yeah, there's the general big problem that we want to solve, but then there's this particular little thing that currently doesn't work. And then you have to look at the details. And how do these weights change on this neural network which is trained by this particular learning algorithm? And why doesn't um, the network do what you want it to do and you have to debug a little tiny thing and suddenly you see the devil is in the detail and there is a little detail which you have overlooked so far. And if you fix that one, then suddenly everything, uh, everything works out and suddenly you have a breakthrough and then the same PhD student can come up with one research paper at a major conference after another just because it's such a rich source of, um, additional improvements that um, was triggered by this little devil in the detail, uh, solution.

Speaker A: Do you think the transformer will persist as like the dominant architecture over the next five, ten years?

Speaker B: I guess some sort of transformer. But uh, I think it's going to be um, a little bit more like the efficient transformer that we had in 1991, the Linear Transformer. Today it's called the unnormalized linear transformer. I called it the fast wave controller. And, and, and what is the interesting thing about it? Uh, it scales linearly, which means that, uh, if you have 1,000 times more more text, then you need 1,000 times more compute. While the modern quadratic transformer of 2017, if you have 1,000 times more, um, text, you need 1 million times more compute. So everybody's worried about that. It's one of the reasons why where these data centers are now so super expensive, you have to do so much compute. And everybody wants to bring down this complexity into um, to a more reasonable, um, linear complexity or maybe log linear complexity. And there are quite a few, um, transformer variants that are going in this direction. Or there is stuff, ah, that Seppheit is doing, the xlscm, which is a, uh, which is a thing that combines aspects of the old linear transformers. So you want to get the complexity down and it's related to what you asked before. Um, intelligence is about doing the same thing with less effort. Yeah, we want to reduce the effort.

Speaker A: Well, I love that. Um, look, this has been a fascinating conversation. I'm sure there's a ton of threads that folks will want to pull, uh, on in addition to what we've discussed here. So I want to make sure to leave the last word to you uh, where can folks go to learn more about your work and really anything you want to point our listeners to. Uh, the mic is yours.

Speaker B: Where can you learn? Um, what I think is uh, interesting. Look at my blog. It's easy to Google. If you Google for Jurgen.

Speaker A: Yeah, we'll link to it too.

Speaker B: You will find it. And um, there are overview pages with links to the original papers on meta learning, on artificial curiosity, on the formal theory of fun and creativity. Also in the history of our field, the field of machine learning is uh, about the science of credit assignment and apply that to the field itself. And who invented convolutional neural networks and who invented deep learning. All that stuff is uh, nicely listed and explained there. And it also explains what I consider the most important stuff, uh, and including what I just mentioned. Uh, um, the important thing is physical AI outside the screen. So not behind the screen, but in the real world. And once we have a robot. And for hundreds of years people have talked about um, self replicating machines, but nobody had any idea how to get that. But now we see an opening, uh, for the first time, we now can have, or maybe we will soon have robots, uh, that can learn through imitation or reinforcement learning to operate the existing machines, all the existing machines that already um, are part of our civilization. Once you have a machine that can operate all the machines that um, humans currently are uh, operating, then you have a new kind of life, you know, then you have a, uh, a way of um, implementing this ultimate scaling machine because you can have robots that make more of themselves. And I've uh, said that ah, for decades and now it's getting closer to reality. You don't need super smart robots for doing that, just smart enough to learn to operate all the existing machines. And a collection of machines like that can make more of itself and something like that can also improve itself, not only make replicas of itself, because all the concepts of machine learning that we already path or AI behind the screen, you know, in the virtual world. All these concepts we are going to apply to um, self improving robot societies and something like that is not only going to work in the biosphere, but also on the moon or on Mercury, where there's a lot of material for um, building infrastructure and bigger AIs and more AIs and more robots and huge uh, spacecraft and all kinds of stuff that you need to colonize the solar system.

Speaker A: Well, I think that's the perfect note to end on, um, an incredibly exciting vision for the, for the future. And seriously, thank you so much for taking the time to chat through everything here. It's such a privilege, uh, to get a chance to talk and I know our listeners really enjoy it too.

Speaker B: It was my pleasure. Thank you, Jacob.

Speaker A: I'm, um, Jacob Efron and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project in addition to my day job as an investor at redpoint. But our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends. It's really what ultimately makes this whole thing work. And so please consider doing that. And thank you so much for your support and listening. We'll see you next episode.