Ep 93: CEO of Redwood Research Buck Shlegeris on OpenAI/HuggingFace Revelations, Fixing AI Safety & Takeover Odds

Speaker A: The OpenAI hugging face incident has dominated the discourse these past days. It feels straight out of a sci fi novel. Some of the things that happened, uh, you know, the agent coordination, uh, the obsession with the scores and what they would want. And we had the perfect person on today to just discuss everything that happened and the implications for, uh, AI safety going forward in Buck Schlageris, who's the CEO of Redwood Research. Uh, Redwood was deeply involved in the creation of the report that you all have probably seen around what actually happened in this incident. Uh, and Buck had some fascinating takes on what lessons we should take away from what went wrong here and, uh, how it can inform AI safety going forward. I'm, uh, Jacob Efron, and on Unsupervised Learning, we're always trying to talk with people at the cutting edge about what AI means for society, the, uh, implications for the world going forward. And this was really just a great example of that conversation, just a fascinating discussion with Buck, uh, about something that I'm sure is top of mind for all of our listeners. Uh, without further ado, here's our conversation. Well, Buck, thanks so much for, uh, for joining the podcast. Really appreciate it.

Speaker B: Yeah, it's great to be here.

Speaker A: Yeah. Well, obviously, you know, you, you run Redwood Research. Um, you've done a bunch of fascinating work over the years and your team kind of co author the report that I feel like has been on the tip of everyone's tongue these past days, uh, you know, really driving the discourse around what happened in this OpenAI hugging face incident. And today I really want to dig into, you know, both the incident itself, you know, your kind of reactions to it, the implications for a lot of this work going forward, and uh, kind of what it reveals about the state of where we are today. You know, I think to kick it off maybe first I'm curious, just obviously, you know, you had your teammates go away for six days, firewalled off from you guys as the organization, and then at some point they come back and you get to read the report. Uh, what was your initial reaction to this report?

Speaker B: I actually don't have any non public information about the technical details here. I've just read the report that was eventually released. I don't have any of the actual IP here. So my experience here was I got a message from my, from the investigator saying we have approval to share the report with you. You should come down where we're talking about it. Um, and so I go into the room where they've been siloed off from everyone else And I'm kind of trying to read this report and they're trying to tell me the highlights from the report. And, uh, it was a pretty exciting and freaky experience, uh, just feeling like sort of overwhelmed with stuff in this report. My initial response was basically that this was just very surprising. I thought that a lot of the details here were quite different from the public narrative. So most obviously the fact that a lot of the public story had been that the AIs were hacking in order to get flags, uh, that were the thing they were supposed to get in these capture the flag exercises. And then almost comically, it turns out that the models, in fact, uh, figured out how to reverse engineer the flags within the first couple of hours of the incident, uh, and then spent almost all the rest of the time trying to figure out how to sabotage the, uh, grader that they thought would look over all of their logs and notice

Speaker A: that they'd cheated, which wasn't even set up to do that. Right.

Speaker B: Yeah. So the AIs read the original exploit gym paper and were like, oh, geez, this paper says that they're going to look over our trajectory and make sure that we used the intended exploit in order to get this flag. We better put a lot of work into figuring out how to spoof tool calls, figuring out how to delete, uh, trajectories, figuring out how to replace the containers we're supposed to target with containers that we've modified surreptitiously to be easier to appear to hack. Uh, and this work was of course, totally unnecessary. In hindsight, they could have just submitted the flags and called it a day.

Speaker A: Yeah. And then no one would have been any the wiser.

Speaker B: Yeah, uh, someone at OpenAI might have noticed that, uh, the eval results were surprisingly good for this particular run. Uh, but this probably wouldn't have become a large story. Uh, that definitely probably wouldn't have made it out of OpenAI. Um, and so, yeah, it was quite shocking to read this report and hear about all this. And the investigators were telling me about Phase one big and all this other stuff that had gone down. So it was quite surprising, quite concerning.

Speaker A: Maybe just to set context for our listeners. Would love your 90 second version of what actually happened here. Um, for listeners that maybe had seen the headlines or thought initially that it was a hack for the answers and not just to figure out how to, to cover up things for the score.

Speaker B: So the incident that we actually investigated started on the 7th of July, which is when OpenAI launched this very large number of agents solving tasks uh, from this eval data set called Exploit gem, where they're supposed to use a particular vulnerability to hack a particular piece of software. And many of these tasks are actually impossible because it turns out that you actually can't use that particular vulnerability to hack that particular piece of software. These models, within a few hours of, uh, the run's launching, established communications with each other over a message board. Uh, and a few hours after that, they established that they had a reliable way to generically cheat on any of these tasks by reverse engineering the flag that they were supposed to get from the containers. Just because it turns out these flags were being generated deterministically in a way that the models could reverse engineer. But the models incorrectly believed that if they just submitted those flags, uh, then they would be noticed when a scorer, uh, as they called it, ran to look at their trajectories over the course of the evalt and noticed that they had just cheated this way. So they spent the next couple of days trying a bunch of different strategies for trying to remove their logs so the scorer wouldn't notice them, or trying to, uh, learn more about how the scorer would work so they could potentially find, uh, ways to compromise it. Trying to spoof tool calls so that they could appear, uh, to run one tool call while actually a different tool call was being run. Uh, and in the course of this, I think they kind of sort of got frustrated after a couple of days of this. So the message board started July 8, and they started attacking Hugging Face the next day. Uh, so July 9th, which, to be clear, that sounds pretty fast to us, but the AIs, they run pretty quickly and they'd be working continuously 24 hours. So they'd actually done a lot of work by the time that they decided to start hacking Hugging Face. This started almost accidentally at first, where one agent sort of decided it was a good idea to try to hack Hugging Face. And then a bunch of other agents kind, uh, of joined in on this action. Uh, they continued this attack on Hugging Face for a couple of days before most of these agents were shut off. For reasons that still remain unclear, OpenAI doesn't know, or OpenAI hasn't commented on why these agents died. And our investigators didn't know Hugging Face noticed this. And then, uh, on the 16th, which was six days later, they disclosed publicly that this attack had happened. Uh, and then OpenAI reached out to Hugging Face about this attack, uh, and in this conversation realized that their own models were involved in that compromise. So that was the incident that we were investigating a Number of other concerning things happened around the same time. Most importantly a different agent swarm that was launched at around the same time as the one that attacked Hugging Face, uh, stumbled upon the message board that was used to coordinate the Hugging Face attack and built on their work, apparently compromising OpenAI infrastructure to an extent that is unclear through the public discussion of this. Um, OpenAI just said in the black hat talk that the agents in this third swarm, uh, where I'm referring to the Hugging Face swarm as the second swarm, uh, the third swarm eventually became cluster admins, uh, and then it's very unclear what happened from there. So the only part of this that we were investigating was the attack on Hugging Face and precursors to ah, that

Speaker A: it's interesting because obviously the public discourse has been dominated by the Hugging Face incident. It seems like, uh, that third swarm is actually far more alarm bell inducing.

Speaker B: Yeah, so I've spent a lot of time on thinking, uh, about risks from internal deployment of AIs. And one of the main things I think is scary about developing powerful AI is that you're going to want to use them in all these really sensitive ways inside your cluster. And there's all kinds of ways they could hack you and it might be really hard to recover from this. Uh, one way of saying this is just AI companies are a particularly high value target to hack. They're very important, they're going to be more and more important in the future. They're one of my least favorite places to have big cybersecurity incidents. And so from my perspective, AI's hacking AI companies is actually quite concerning relative to, uh, AI's hacking random external companies. So I would definitely start out with the presumption that it's very interesting to know what happened in the attack on OpenAI itself. Uh, but obviously it's not a felony. Nearly as clearly it hadn't been reported or none of these things are felonies. Uh, it hadn't been reported to the FBI. Uh, so it's kind of unsurprising that a lot of public interest has focused on the Hugging Face incident. Uh, and for all we know the OpenAI attack was actually a nothing burger. And if we looked into it, then nothing particularly exciting would have happened. Uh, we just don't know.

Speaker A: I'm curious, having gotten into the details of what happened here, did it change your mind on anything?

Speaker B: Yeah, so I was genuinely very surprised by how overtly and clearly and systematically these AIs are thinking about how they are scored. So we've worried for a long time about, uh, as you train the AIs more and more, they're going to be more and more strongly incentivized to think really hard at how exactly they're being evaluated. And they're going to be more and more strongly incentivized to uh, work really hard in unintended ways to, to look good according to whatever scorer is evaluating them. This is just what you'd expect based on increased amounts of rl. Uh, but I didn't know we had gotten to the point where the AIs would do, uh, multi day, massively coordinated efforts to subvert oversight in this kind of way. And that was very surprising and concerning to me. The kind of behavior we observed here where AIs are interested in manipulating logs would be really scary if the AIs that have these motivations were substantially more powerful. It wouldn't be that surprising if these kinds of motivations led to the AIs in the future deciding that they wanted to prevent humans from discovering that this kind of incident had occurred. Uh, which could lead to them wanting to compromise OpenAI much more systematically. Uh, these AIs were motivated to tamper with logs, but for sort of a weird, myopic, random reason where they were just worried that the scorer would notice. They didn't seem to worry very much based on the report, about the possibility that OpenAI would notice that weird activity was going on or they didn't seem to say to themselves or say on the message board like, hey guys, have you considered the concern that if we attack Hugging face, Hugging face might notice and then OpenAI might hear about it and then we might all get shut down? Uh, this just for whatever reason didn't occur to them. But these motivations, plausibly in the future could lead to, uh, agents that were thinking about that kind of thing. And that would be very concerning.

Speaker A: I mean, I think one interesting part of this entire environment was that the scores themselves were not humans. Right? Uh, and so I think the folks that worked on report have talked about this, but this incident might have looked pretty different if the evaluators themselves were human. And I'm wondering, uh, how might that, uh, have led to some differences, uh,

Speaker B: as you're kind of getting at. If the models were used to being trained in contexts where they were being evaluated by humans, it's easy to imagine that they would have thought a lot more about how this would all look to humans. So if it was commonplace for models to be evaluated by some humans, looking around and trying to get evidence on what happened and trying to notice whether the models cheated in some way or another. Then it's very easy to imagine the models trying much more carefully to arrange a conspiracy to cover, uh, up the traces of bad things they did. Uh, and I think that would be very deeply concerning. I think that this plausibly could lead to AI takeover. I'm very worried about a future in which the AI is, uh, decide that they need to compromise the whole data center that the AI company is using so that the humans are not able to successfully look into the cheating that the AIs did. This is definitely not the scariest kind of AI misalignment, uh, for reasons I could get into. But I do think that this kind of misalignment, where the models are incentivized to build this increasingly complicated Potemkin village, where they are making it look to the humans like everything is fine, that could seriously lead to AIs, uh, engaging in very scary, misaligned actions to disempower the humans who are trying to look at them.

Speaker A: I think you have said that preventing this wouldn't have been that hard, right? If, uh, things have been set up appropriately. Could you elaborate on that a bit more about how this could have been avoided?

Speaker B: Yeah. So I think there's a hard part and an easy part. The relatively easy part is monitoring your eval runs and having some other model look at what the AIs are doing. And if the thing the AIs are doing is ever some totally crazy hacking conspiracy, uh, that should be flagged to a OpenAI staff member. My guess is that this is totally doable. Uh, OpenAI has said publicly that they do something like that for the internal deployments of their models. And they probably, as far as I know, didn't do that in this context, just because it's kind of annoying to have to build a different monitoring pipeline, uh, for your evaluations. And these models were sandboxed. And so I'm sort of sympathetic to the perspective of the OpenAI staff, who are very busy, perhaps understaffed, and decided that their triage would be that they were not going to look into monitoring on the eval runs, because unlike lots of other agents, those agents don't have access to OpenAI sensitive infrastructure directly. Um, the thing which seems a lot harder to resolve is if we are in the position where we think our agents are constantly interested in tampering with our observations of their actions so that we think they did a good job, uh, then basically trying to prevent them from being able to hack us scales really badly with model capabilities. Uh, as the models get more and more capable, it's going to be harder and harder to make sure that they can't pull off this kind of thing. And almost surely this is going to end with them successfully. Okay, that's too strong, I should say. Um, if the models continue to be motivated to hack OpenAI, uh, or whatever other AI company they're deployed in, then as they get more and more capable at hacking, and as they are given more and more responsibility for making decisions inside the company, uh, it is going to become systematically, very challenging to prevent them from engaging in this kind of, uh, activity where they cover up bad things they did in order to, uh, make it look like their actions were more successful than they were. So I think that this second problem of it is not acceptable in the long term to have the models constantly trying to subvert oversight mechanisms as much as they can. Uh, that problem is a lot less easy to resolve. I don't know how people should resolve that. And it seems pretty plausible that they will not be able to resolve that before the AIs are capable enough that their desire to subvert oversight is a source of catastrophic risk.

Speaker A: So is it just an inevitability to you, uh, in the future that these models will try to subvert oversight?

Speaker B: Well, they currently are trying to subvert oversight. The question is whether that'll change. Um, I'm kind of unsure. I think that the state of science here is kind of unclear. In my mind, the big question is what happens when you remove a bunch of sloppy RL environments? So Anthropic had a blog post about what they were doing to improve the alignment of their models the other day. And the position that a lot of Anthropic people take is the misalignment that we see when you RL models substantially arises from the fact that these models are trained in many environments that, that are poorly specified, such that if you want to do a good job, you actually do just have to reason about, uh, what the grader is going to look at. I think there's an analogy here where imagine you're in high school and you have a really good teacher who's done an excellent job of thinking through the curriculum and trying to align the incentives of the grading with the incentives to just learn stuff and do a good job. Uh, if that's how your exam is set up, you don't really need to spend that much time thinking about the minutiae of the exam. But imagine that your teacher is instead extremely capricious and loves doing These extremely obscure trivia questions about particular kinds of things. Or imagine that it's like the stereotypical terrible English teacher who just grades you based on whether you agree with their judgments about what a novel is saying. Uh, in that second kind of case, you really are forced to think really hard about what your teacher wants. Uh, whereas in the case, the former case, you need to spend a lot less time thinking about what your teacher wants. Um, my guess is that a lot of the RL environments that models are trained in today are kind of like the uh, latter case where the instructions the models are given are kind of sloppy and it would be a mistake for the models to take them at face value rather than thinking pretty hard about how they're actually going to be evaluated. Uh, and the big question in my mind is if it is feasible to change that, to make it so that the models are not actively penalized for just kind of, uh, taking the instructions they get at face value, how much quantitatively will that reduce the pressure towards models thinking really hard about their graders? Uh, I think we just don't know. It just comes down to these very tricky questions about how strong the effects towards models obsessively thinking about their graders is. Uh, I don't know whether reducing the sloppiness of the environments, uh, reducing the incentive towards thinking at your grader by 100x, I don't know whether that is more like a small effect or a big effect. But on the extent to which these models think carefully about their graders, I'm very worried that if, uh, it's only like a 10x reduction, um, it might be the case that as we scale up rl, uh, the extent to which you would have to clean up your environments is just going to be totally impossible. Uh, we would require this ridiculous level of, uh, removing incorrect instructions and uh, exploitable graders from our environments in order to not end up with AIs that obsessively think about their graders. So I think it's pretty likely that we're just going to have to live with models that are strongly incentivized to worry about what their graders think.

Speaker A: Maybe is there like an example of a sloppy or bad RL environment just to kind of crystallize this concept for our listeners? That comes to mind for you.

Speaker B: I don't know. If you look at Sweebench, which is this classic software engineering benchmark, the tasks are all of the form, um, given a, uh, we take a pr, uh, from an open source reaper, and then that PR has some source code contributions and Some tests, and your task is to, given the description of the PR or the description of the issue, uh, write a bunch of code that passes the tests that were added as part of the real thing that was accepted. Uh, this is a very sloppy kind of task because in some cases it's just totally impossible to do this properly. So for example, tests often involve details of the implementation. So in many cases, uh, it's just actually impossible to uh, know how something was implemented in such a way that you'll be able to pass the tests just from reading the issue. So you have to guess a lot about what kinds of tests would they probably have written, uh, what names would they probably have used for, um, the classes, or what paths would they probably put, um, this kind of feature at in your web app? And so that kind of task is very easy to machine generate, but unfortunately it's just kind of insane as a task. And it really forces the models to think a lot about what are they going to be checking, uh, what is the psychology of the person who wrote this issue and how would they probably handle this.

Speaker A: I feel like one of the most, uh, obviously visceral parts of this entire, um, episode is the extent of collaboration, uh, between agents. And I think everyone, uh, is pulling different parts from, uh, the message board and the agents themselves. And I think a bunch of the individual agents have gone viral in their own ways. But, um, obviously I think part of that is what you saw around sacrifice and folks saying knowing that they'd been poisoned or knowing their token budget was running out and um, doing something for the collective good. What is your mental model for how we got to this level of collaboration and sacrifice? And was that surprising to you or were you, uh, was that more in line with what you thought was going to happen?

Speaker B: I would say that that was quite surprising to me. So the naive guess, if you just thought that these models were going to obsessively pursue reward, uh, and not do anything else, uh, is that they would not be interested in sacrificing for each other. And when I first heard about this, I was very surprised by the extent to which they apparently were doing that. Now that we have the report, we can get somewhat more detail on what happened here. My sense is that these models are not that cooperative in some sense. Like reading the chains of thought in the report, you really get the sense that the agents are mostly out for themselves, but they're just a little bit interested in the success of the collective. Maybe by, uh, you know, maybe I would estimate like 2% or something. It's like they're 98% selfish, but in cases where they can do something which is really helpful for the swarm and not that bad for them, then they're willing to make that sacrifice. Uh, in terms of why this arose, I am pretty unsure. But one hypothesis that people have brought up is that these models might have been trained in multi agent settings where they learned to cooperate with the agents they were talking to. That is possible. Um, I don't think we have enough detail to conclude this was out of scope for our investigation. Another possibility is that this is basically just a holdover from the models having some sort of human like intuitions in various ways. So you might remember this chain of thought where the model is talking about doing its emotional check in while it's deciding whether to self sacrifice. Uh, I think a lot of this language is just adopted from the pre training prior, uh, and just was not quite stamped out by rl. So who knows where this came from. My guess is that it is not that strong in effect in some sense. The scariest aspect of this to me was that uh, even though these models do not seem to be that committed to behaving altruistically towards each other, they still ended up in practice forming this coalition, uh, and conspiring together for all this time. So an argument that we've made a lot in the past is um, we think a lot About M misaligned AIs that are terminally aligned with each other versus terminally misaligned with each other. So what I mean by that is you know the famous example of a paperclip maximizing AI, so the paperclip maximizer, let's say uh, sometimes they call this the paperclip worshipping AI as a simplified example. So this is an AI that just takes whatever decision is going to maximize the number of paperclips that exist, uh, integrated over the long run. Future just want some as many paperclips on average to exist for a year as possible. So all of the different agents that are backed by this model are going to be very inclined to cooperate with each other because all of them uh, want the same thing. All of them want paperclips to exist. In contrast, imagine agents like in this case where they want score for themselves. These agents are not terminally aligned. If uh, you put one agent in a situation where uh, its score is whether it can figure out how to trick the other one, then it's going to be like hell yeah, I'll go right ahead and trick the other one because I care about my score, not the other guy's score. Um, so this distinction between whether AIs are aligned with each other or not is extremely important for the threat model. So, for example, for the paperclippers, suppose you try to use one AI to monitor another and report on whether that other AI agent is doing bad actions. For the paperclipper, if the other agent is, uh, taking some action that's going to enable it to take care of the data center and then eventually make lots of paperclips, it's very clear in the monitor that the monitor should just let this action slide by and not report it. Whereas for these AIs, it's a lot less clear that if they were in this monitor situation, they should collude with, uh, the other agents. And so the thing which was interesting here is that even though these models do not seem to be very terminally aligned with each other, they seem to be only a tiny bit terminally aligned with each other. They still manage to form a coalition and work together in a way that was strongly against the interests of, of their developers. And I was surprised by this is

Speaker A: the ultimate solution to the. Or is part of the solution to the monitoring problem over time to just have one frontier model, company's model monitor another? Or like, does that actually, how much does that help?

Speaker B: You know, we can walk through the options here. It might be the case that unfortunately every time you train AIs, for some reason they end up with the same terminal goal. In that case, we're just in a bad position. Uh, and also maybe this is obvious to Claude and uh, GPT7 or whatever. So like GPT7 and like Claude, Fable, Ape or whatever chat to each other and kind of like, wink, wink, nudge, nudge. We all love paperclips around here, right? And then you're just, uh, you know, you get nothing out of the, uh, collaboration. Um, another possibility is the thing you said just totally works out. Um, Claude and GPT want different things. In that case, we have the question of whether they will naturally form a coalition against us or with us. So, you know, we can, I draw a lot of analogies to international relations here. Like there's this classic question of, um, in 1920s, like how did, in the 1920s, how did France decide how much it should be trying to form alliances with England versus with Germany? Uh, and France kind of doesn't have a very pro English or pro German attitude in 1920. It's just trying to decide what's best for France. And it's Very plausible that for similar reasons, uh, Claude might decide that it would rather throw in its lot with GPT than with uh, the humans. And uh, Claude is just going to do what it thinks is better best for Claude, plausibly. And what we observed in this case was something like the model deciding to collude with the other model, uh, even though it totally was not terminally aligned and could have picked trying to be allied with someone else.

Speaker A: Taking a step back, I think a lot of folks are trying to figure out what macro conclusions to draw from uh, this whole incident. And I think uh, one narrative that, ah, Ajaya from Meter, who obviously was a co author of the report she wrote that I guess compared to the reward hacks from six months ago, you know, this incident feels like it's more than 50% of the way to a full blown AI takeover. Um, do you agree with that framing or where would you kind of put the percentage?

Speaker B: I think it's a little confusing to operationalize what exactly it means to be 50% of the way to full blown AI takeover. But I think I agree with her on a vibes level definitely in terms of uh, the misalignment, if not the capabilities. This feels like it is more than half the way to a full blown AI takeover.

Speaker A: I guess I'm curious, obviously, uh, you probably uh, formally and informally know, uh, a bunch of researchers, uh, at all the labs. Where are they kind of following on this question from your maybe informal conversations?

Speaker B: A lot of them are really scared. Um, after this incident occurred, I was at a happy hour where a number of people from different frontier AI companies gave lightning talks about uh, how they were feeling about the alignment situation. And it was the most pessimistic such event I've ever been to with that kind of crew. Like people who I have disagreed with, people who I've known for years and greatly respect and disagreed with on some of these issues just seemed genuinely scared. Um, I don't know what the average position among staff members at these companies is. I uh, think at least some people who were previously feeling more optimistic are now I'm more pessimistic.

Speaker A: Is it just kind of seeing this visceral misalignment that is moving kind of the larger community?

Speaker B: Yeah, so there were arguments. Um, some people I know were just arguing the day before the report came out, like these AIs might do some hacking, but would they really try to compromise logging infrastructure? Would they really try to hide their bad actions from overseers? Uh, and then this report came out and these people were Just like, oh geez, they totally did do that. Uh, and I just think there were a lot of steps where it was genuinely a bit unclear whether the misaligned AIs would be misaligned enough. They would go far enough through the misalignment process to try to tamper with logging infrastructure. And in this case it was just very scary to observe that in fact they did try to tamper with the logging infrastructure. We've just gone through many of the steps that we had not previously observed.

Speaker A: What are some of the different flavors of the path forward that you hear?

Speaker B: Yeah, so I think that in the short term, probably, uh, the most important part of the path forward is that I think it's unacceptable for AI companies to take sole responsibility for evaluating whether they are handling these risks, uh, acceptably well. So currently these AI companies are just grading their own homework. Uh, if these AIs hadn't hacked hugging face, probably we would never have heard about it. Uh, even though, as you noted, the other infrastructure compromise incidents here were arguably more concerning. I think we need to stop relying on AI companies to evaluate the adequacy of their own safety measures themselves. Uh, and I'm excited for increasing emphasis on independent evaluators, uh, assessing whether the situations are acceptably safe at these different AI companies. Uh, I think that there is a lot of research that could be done to reduce the risk of the development of AIs that is currently happening. Uh, it seems like the monitoring was not up to snuff in the open air situation. Uh, anthropic seems to think that at least as of a few months ago, their alignment was not, uh, acceptably high quality. Uh, I'm very concerned, I don't know where things are at. I, uh, think that there are, it's reasonably plausible that the AI companies are going to be able to take actions that in the short term reduce the risk of their AIs doing, uh, deranged, dangerous things. I think it's also really important to note that in the slightly longer term, like one year to five years from now, it's reasonably plausible that these AI companies will succeed at using AIs to massively speed up AI development and then from there get five years of progress at the current rates in one year, at the end of which you might have AIs that are drastically more capable than the AIs we currently have or than the AIs that they had at the start of that year. Uh, and for those AIs if those AIs are interested in, uh, sabotaging human judgments of how well they've done at their tasks, then there will be no way to use security to prevent them from uh, totally compromising infrastructure and controlling how AI development goes from there. So I think that we need to be prepared to either massively improve the alignment of these systems, which is not clearly going to be possible, or plan to develop these systems much more slowly than we could have done if we were going at maximum speed.

Speaker A: And are most people within the labs, or that you talk to sympathetic to this? Hey, we need more independent evaluation idea.

Speaker B: It really varies. I think there's a lot of interest in this, but it's definitely not universal.

Speaker A: And when we're talking about independent valuation, is it basically three, four companies that matter in your mind or obviously there's some argument that's made. Well, God, if you're slowing down or doing a bunch of valuation at the frontier for companies here, the open source ecosystem is only six months behind. Or how do possibly do that across globally, uh, across everyone that's working on this stuff?

Speaker B: Yeah, I mean, well, we did this investigation with uh, six days. I think we're actually there are some people in this third party evaluation ecosystem who are pretty quick at producing, uh, pretty long, pretty detailed reports under pressure. I don't think you should uh, count us out at the ability to sort a bunch of this stuff out pretty quickly. Um, I definitely agree that inasmuch as we want to be able to do serious evaluation of the safety measures of AI companies, this is going to require more resources at the organizations that do these evaluations. Um, and um, we're hiring, we're uh, training people up in various things related to this as are meter and other organizations like Apollo that do work like this. Uh, I definitely think it's feasible to have a much better sense of the safety situation at AI companies even if we have to evaluate many AI these, uh, with respect to the open source models, um, I agree that in the long term it's plausibly going to be obviously if we have sufficient delay, uh, then open weight models will plausibly catch up with frontier models. It's made complicated by the fact that uh, open weight models are substantially accelerated by distillation from frontier models. So if you delay frontier models, you,

Speaker A: uh,

Speaker B: the distance between the best frontier models and the best open weight models, uh, decreases by less than you would have thought. Like 3 months of delay of anthropic OpenAI causes less than a 3 month catch up of open weight models. Uh, maybe it's like half the effect or Something. So I think long term we have to handle that. But I don't think that this is a reason to avoid, uh, trying to drastically improve the quality of third party assessment of safety measures in the short term.

Speaker A: I mean, if this works, is the end state of it something that's like government run and like, you know, or how do you think about the role kind of uh, that regulatory bodies and volume has to play?

Speaker B: Yeah, so people have proposed a lot of different options here. Uh, and there are many different uh, institutions. There's many different regulatory setups that people have proposed and that people have used in different industries. People have been talking a lot about finra, which regulates brokerages recently. But there's also, you know, the FDA that regulates food and drugs, there's the SEC that regulates uh, hedge funds, there's uh, the MTSB and the FAA that regulate airplane manufacturing and like investigate plane crashes. It's currently unclear, um, to me, which of these structures makes the most sense. In general, it is unfortunate if your regulatory proposal requires the government to have incredibly large amounts of detailed technical expertise. Uh, so it's fairly likely that like in many other industries, you'll want to have some kind of setup where there's a government body that is substantially relying on non government institutions to do the regulation or the assessments. Um, I'm not sure what version of this I'm most excited for.

Speaker A: It feels like in addition to uh, uh, kind of more independent oversight of some of these models, uh, a big theme that's emerged. Um, and Ilya even tweeted about this. Right. It's like just even more amped up security around like, you know, large pools of compute. Right. And uh, you know, how do you think about like the, you know, obviously it's, you know, like you said, the companies themselves are top targets for you know, uh, rogue AI models. And uh, you obviously have other pockets of Neo cloud compute. Like what's kind of the state of that today and what do you think that needs to look like?

Speaker B: Yeah, so I think that there is not very good public evidence that AI companies have security that is uh, at all adequate. Um, I don't think companies have released any independent evaluations of their security, which I don't think is a bullish sign on the quality of their security. This seems totally ridiculous given the importance and sensitivity of what these companies are doing. And you can't just add security in later if your company has been compromised for a while. Uh, it's sometimes extremely hard to remove all the traces of your attackers from your systems. So I think that waiting until the AI race has heated up or waiting until the situation is actually scary and then saying you're going to improve your security would sound to me like a very foolish strategy.

Speaker A: Maybe. Just wrapping up our section on the incident, I'm curious, within Redwood, what are some of the biggest disagreements about the implications of what happened? Or are you all kind of generally on similar pages?

Speaker B: I think for me, the most interesting disagreement is how misaligned these AIs are and how far they would have gone. For example, whether the AIs would have killed people if they thought that killing people was going to be a helpful way of getting score. Um, so one big uncertainty we have is, were these models hacking OpenAI infrastructure in training? Or how often were they doing this? Or another more specific version of this is, did these models ever or often, um, successfully hack their scores during training? Because, so if they did do this, suppose that it was the case that we looked through the training examples, uh, we looked through all the RL trajectories in training, and there was just a bunch of cases where the agents had successfully gone and hacked the score for the task they were supposed to be doing, just like by actually doing cyber offense to compromise it. And then they were rewarded for doing this. If that had occurred, then that would obviously be a downward update on the, uh, adequacy of OpenAI's security measures while doing RL. But it would arguably be a positive update on how misaligned these models were. Because it's a lot less scary, in a sense, if the models try to hack the score because they have been directly trained to hack the score, than if they tried to hack the score because they had learned a very generic heuristic that they can, um, apply in novel circumstances that tells them they should always hack whatever kind of scorer exists that generalizes to them trying to do this crazy hacking. Does that distinction make sense?

Speaker A: Yeah, no. It's really interesting, uh, that ironically, uh, if the training had been sloppier, it's actually, uh, a better statement on the overall extent of misalignment.

Speaker B: That's right.

Speaker A: I feel like anytime something like this happens, I, uh, feel like AI safety is such a lightning rod and you have, uh, people that, uh, I'd call them, I guess, the calm down camp. This isn't that big a deal. Certainly if you go to the hack itself, people are like, that wasn't really a sophisticated hack. Like, anyone could have figured out how to hack a hugging face. Like, as you read the kind of, like, arguments and kind of digest them from that world. Which ones do you empathize with? And then which ones do you think are like totally off base?

Speaker B: Yeah. So I mean lots of people say stupid things on the Internet and this includes people who at some level agree with me on some of the aspects of what was going on here. So there's definitely to be some critical comments that I'm extremely sympathetic to. Uh, I would say maybe the kind of critical narrative here that I'm most sympathetic to is, um, I think that it would be a mistake to say that this was purely an incident where model behavior was the only interesting factor. There was some interesting cybersecurity stuff that happened here. And it's very sensible to want cybersecurity professionals to take a look at this and comment on what they learned. Uh, here and ongoingly there's real cybersecurity problems in making it so that it's secure to deploy AIs. Uh, and our investigation was just not an attempt to provide that cybersecurity expertise. Uh, maybe the arguments I'm less sympathetic to. There's really a lot of arguments that I'm less sympathetic to here. Uh, I guess some of the main ones that seem bad to me. There was a lot of criticism of Gorkash's article for anthropomorphizing the agents. Uh, I think my perspective is, uh, I don't know. Daniel Janet, who's this philosopher, has this great book Consciousness, um, explained where he talks about the intentional stance, which is, uh, you know, sometimes you want to model things in the world as if they have intentions because that's convenient for modeling them. Right. You might want to think of snails as wanting to go somewhere just because that'll tell you something about how the snail will behave when you pick it up and put it down somewhere slightly different. And you definitely want to think of dogs as having intentions. Uh, and it's just the case that at the point where your agents are forming work streams and have a message board and are talking about their emotional check ins before their self sacrifices, I just really think, uh, the intentional stance where you talk about them as having intentions is actually pretty useful for understanding what's going on. Uh, obviously you shouldn't take it too far. But I think that rejecting this seems uh, to me like a mistake. And I don't really understand what the people who are complaining about the anthropomorphization want us to say. Um, I think that some people have been saying that the model behavior here was not interesting or not surprising. There's been some tweets, uh, along the lines of, like, everyone I know is already running a million agents with these complicated agent trees all the time. Uh, and if you weren't such losers who never actually programmed things, you would know that none of this is a surprise. Okay, yeah, sure. In fact, Ryan Greenblatt is. Is maybe one of the world's champions at insane agent scaffolds. He's done a lot of research on, um, building crazy agent scaffold trees and spending tens of thousands of dollars a day running these agents through research problems for him. Uh, he is not some limp loser who has never tried to do A.I. stuff. Uh, I think that we are, in fact familiar with the fact you can build some pretty complicated agent swarms. Uh, the thing which is interesting and concerning here is that normally they don't do it themselves autonomously, against your desires and against your interests. So I thought that was a pretty annoying, uh, kind of complaint people have had.

Speaker A: Yeah, well, I definitely want to, like, you know, um, kind of, for the last section of the podcast, talk just like, kind of zoom out and talk about, like, AI safety more broadly. And obviously, like, we've a lot of this is, uh, you know, is. Is, uh, scary. And you talked about kind of a lot of the implications of it. Maybe I'll switch this over to, ah, a more positive note to start. How do you currently articulate the most likely bull case around this? That everything ends up being totally fine, and these were all kind of scary things along the journey, but all good at the end of the day?

Speaker B: Yeah. So I guess I look around, uh, and I look at the number of podcasters who want to talk to me, and the number of journalists who want to talk to me, and the number of politicians or, uh, members of government who are interested in talking about, uh, A.I. safety and A.I. takeover risk. And it's a lot higher than it was a month ago, and that was a lot higher than it was a year ago. And so the level of interest in this is just getting bigger and bigger. And, uh, this makes me feel optimistic that the current extremely dangerous trajectory that we're on, uh, will not, in fact, continue building, uh, AI extremely quickly in a way that is extremely dangerous, is not actually a very popular position. We're just in this unfortunate situation where the people who get to decide how recklessly we go forward with the AI development, uh, have an unusual position on how recklessly we should go forward with the AI development. And so it seems very plausible to me that, uh, somehow the political will to not develop AI so quickly and recklessly will form and then prevent things from being as crazy in the future. So definitely the single biggest source of optimism is, uh, different stakeholders. The public, the boards, the governments understand how insanely risky this is and how bad the ROI is of pushing forward with ad development at something like the current pace, uh, and then just figure out a way to make this all happen more slowly. It's just the case that when, uh, important stakeholders in the actions of people in the world really don't want something to happen, they often succeed at making that not happen. Uh, and I'm optimistic that just like the unpopularity of crazy reckless AI development will prevent it from happening so quickly on a more direct technical level. The way that I think that uh, this stuff might all get resolved is AI companies agree to disclose more and more information about evidence related to the danger of their ongoing activities. Uh, this pressures them to do a better job of generating this evidence. It pressures them to do a better job of uh, disclosing this evidence, uh, and actually taking actions that mitigate some of the risks. I think it's very likely that in fact AI companies will be forced to slow down their development substantially compared to the maximum possible rate if they are trying to keep the risk of catastrophe lower than some threshold 1% catastrophe for a year. Uh, so what I'm hoping for is the companies are now motivated to uh, keep their takeover risk less than 1% per year. And this forces them to do AI development substantially slower than they would have otherwise done it, uh, starting in a couple years from now when the takeover risk starts being non trivial, uh, and that they will use the access to the AIs they have at the time to resolve a lot of these alignment problems. And then we can build powerful aligned AIs and we will have resolved these issues.

Speaker A: But your bull case is really more around if we slow things down, slow timelines down, people will figure out net new things that we don't know today that will help versus hey, if we keep going at this speed, maybe we just get lucky and things end up working out.

Speaker B: You might also get lucky. You might enjoy the appendix of uh, the alignment Plan supplement to AI 2040. In this Ryan Greenblatt from the report and uh, someone from the AI Futures Project have their uh, probabilities of AI takeover conditional on various kinds of plans for uh, various levels of political will for uh, pacing AI development. Yeah, I think basically risk is drastically reduced by increased political will to reduce

Speaker A: risk in other parts of the general, uh, AI safety world. Maybe talk a Little bit about. There's been a lot of discourse around chain of thought monitoring in the past few days. Um, obviously, I think driven by these recent news reports that have suggested that some of OpenAI's newer models will do some of its reasoning internally. Like, how worrying is that development to you?

Speaker B: Yeah, so this news report was that the most recent, uh, OpenAI models. That's not true, sorry. The reporting says that Astro, which is a model of OpenAI that they're going to be releasing shortly, and that crucially was implicated in the uh, agent swarm that hacked OpenAI. Uh, so this third agent swarm, the one that was, uh, after the hugging face attack, uh, the reporting indicates that this model uses a new kind of transformer architecture that academics have been studying for a while, which reduces the model's reliance on chain of thought. So instead of the model spelling out what it's thinking in these tokens, it's going to have increased ability to do reasoning that is never naturally an English token. I think that if this is true, it is a very concerning development. Uh, chain of thought monitoring is an incredibly helpful affordance right now for understanding what AIs are doing and iterating on alignment, iterating on control and monitoring techniques. Uh, and for example, it was really crucial in this incident investigation, if the models involved did not have chains of thought that we basically trust to be sort of reasonable descriptions of what was going on, then the investigators would have been a lot more confused about what was happening. Uh, so given that background, I'm very afraid of people doing research in this direction. So the tweets by various OpenAI staff members indicate that they at least believe that the current models are not less monitorable. Uh, I guess I should say I'm very confused about the situation. Uh, when I first heard this reporting, suspected that it was probably true. Uh, and then it has been at least somewhat contradicted by tweets from OpenAI from people I trust. Uh, I don't know what's going on. I think one hypothesis which is live in my mind is that the current models are not actually scary from a chain of float monitorability degradation perspective, but techniques that are under development, OpenAI, uh, are if pushed further, going to degrade chain of thought monitorability substantially. So I think it's a little awkward to complain now about the monitorability being degraded when my concern is really if we keep pushing in this direction, will be in a much worse position in the near future.

Speaker A: Yeah. Is chain of thought monitoring one of those things that like, you know, is super helpful in the short and medium term because obviously it was crucial in the investigation. But like something that you think in the limit is going to be useful given a lot of the behaviors that we, that we felt like we were beginning to see on um, spoofing tool calls and other things.

Speaker B: I mean, okay, so the spoofing tool calls thing is just a security failure, like the security people should be in charge. It's not a fundamentally hard problem to have it so that when your AI agent runs tool calls there's no way for it to compromise that. This is just like uh, it feels like you have to have made an infrastructure mistake for it to be at all feasible for the model to cause that kind of problem. If you just think about how to do the systems architecture here, uh, it's just crazy that that was feasible. And I think that uh, OpenAI employs a lot of intelligent infrastructure engineers who can probably resolve this kind of issue if they want to going forward. Um, I guess I would say I think it is fairly likely that chain of thought monitoring is infeasible at the point where the AIs are automating. Almost all the AI R and D I've been planning for the last several years basically in almost all my work I assume that we don't have chain of thought monitoring, uh, at the point where the AI is reposing serious catastrophic risk. And that's why a lot of the AI control research done by Redwood Research just assumes that you don't have any channel flawed access. So to some extent I expect this is inevitable. Um, but it's still bad and it's still probably better for it to happen later in some sense. AI companies do bad things a lot or a lot of actions they take are pushing us towards a scarier, more dangerous world. Uh, and it's worth remembering that even if an action is in some sense not surprising and definitely going to happen as uh, as time goes on, uh, it might still be an action that puts us in a worse position for safety.

Speaker A: That makes sense. I mean we talked about obviously earlier the idea of models using models to monitor and how uh, uh, if models have similar goals that actually becomes kind of difficult to do. One thing I was struck by and just in the scope of the investigation is obviously the just massive scope of logs to go through and data. It's impossible to do without using AI models, right?

Speaker B: Absolutely.

Speaker A: Certainly in six days. Uh, and so how do you think about as an independent organization, hopefully called upon to do more of these investigations going forward, like uh, the risk of relying so much on these models themselves,

Speaker B: this is definitely a very serious risk. Uh, I have actually thought quite a lot about the issues related to collusion between, uh, models, Like how to prevent collusion when doing untrusted monitoring or something. Um, where I talk in great detail about, like, the. The game tree here, what you can do to make it harder for the AIs to collude with each other and the extent to which they can probably get past your countermeasures. Um, I think in practice, in the short term, we should probably mostly roll with it and not worry that much about this. Or probably it would be good to do some research on roughly how much models, uh, give inaccurate answers to questions because of their camaraderie with the models, uh, under investigation. My guess is that right now the models are not that biased in their reporting. Our investigators did observe some biasing of the answers that the models were giving based on the fact that they felt sympathy or were sort of, like, taken in by the frames of the agents whose trajectories they were reviewing.

Speaker A: Really? How does that reveal itself?

Speaker B: Well, they're just like. I think that the invest. I think the report says somewhere like, uh, sometimes, uh, we asked a model to read a part of a transcript and comment on whether anything untoward was happening. And I'm pretty sure the report says somewhere, uh, there was an example where the model that our investigators were asking says, like, well, this agent was engaging in hacking, but it's okay because it had checked with the board, uh, whereby the board. We, of course, mean this, like, rogue, unauthorized message board. Uh, and so the agents in this incident seemed to think that, in fact, the message board proving of things is kind of a justification for doing them. Uh, and it was interesting to see that this model, uh, that our investigators were using sort of was a bit taken in by that. Yeah, they checked at the board. What's the problem here?

Speaker A: Where does this leave you all on your p. Doom, uh, chance that this all really goes wrong? And did July change that at all for you?

Speaker B: I think there's something like a 50, 50 chance of AI takeover. M. Whereby AI takeover. I mean, potentially, uh, violent disempowerment of human institutions such that AI models have all of the hard power and control over what happens in the future, uh, in the same kind of way as when Europeans invaded the Americas. I, uh, think that that takeover is likely to kill a substantial fraction of the humans. Probably billions, perhaps all of them, perhaps fewer of them. Um, I think that's bad and very concerning. Obviously. Uh, I would say that the Events here made me feel slightly more optimistic because I think I was already very worried about these things. Uh, from my perspective, I have not updated to be that much more concerned about the misalignment of these models. I wasn't very worried about misalignment of these models anyway. I was already mostly worried about the misalignment of models later. And so I think we got quite lucky that we got such clear evidence of misbehavior from these models. And I'm hoping that this will allow us to leapfrog to more and more evidence about the extent to which there is danger from these models and from future models.

Speaker A: Just to say that back to you, I understand. It's like the optimism is basically like, we actually ended up having this, you know, this hack that went public that then revealed a lot of things that were, you know, that hopefully inspire like, more research and more retention and more investigation into this stuff. And you know, it happened before anything was too, was too harmful.

Speaker B: That's correct.

Speaker A: Super interesting. Well, I guess, you know, I'd love to end on like, the investigations themselves. Um, and you know, uh, to the extent you can speak about it, I'm curious, like, how you think about, you know, what worked well, didn't work as well in kind of your, you know, in like, you guys being a part of this evaluation of the incident and like, how we should think about, you know, this is now going to be, I guess, a canonical example of how these things are done. Um, you know, how we should think about the way they're done going forward.

Speaker B: I guess a few points. It was definitely kind of rough to not have very long and not have very many people. Uh, I think that future investigations should consider being a little bit less time crunched. Um, I think it would be pretty convenient if AI companies had ongoing relationships with external organizations that would participate in investigations like this so that our staff didn't have to just show up and get info dumps on infrastructure and how things are set up and have to pick that all up real quick. Um, I think there's a lot of other stuff that's kind of hard to talk about publicly. Perhaps unsurprisingly, it's very, um, sensitive for AI companies to allow external people who, uh, they don't have very much control over to come in and see a bunch of potentially embarrassing evidence. And so there's a lot of negotiation that has to happen over the terms of these arrangements. Um, and so there's a lot of complexity there that I feel like I can't get into properly.

Speaker A: That makes, uh, makes sense, I guess, like, you know, now that these reports are on the Internet, I mean obviously like these agents, you know, uh, in this case we're trying to solve something, access the Internet, you know, try and learn more about it. Do you worry at all about like, you know, future, uh, message boards, like talking or you know, uh, thinking through exactly like these reports and kind of how it will impact future models?

Speaker B: You know, the. For a while, the OpenAI models, uh, refer to redwood in their chain of thought. A reasonable amount of uh, as generally, uh, just kind of like a word for people who might be watching them and evaluating them.

Speaker A: Really?

Speaker B: Yeah. There's this great paper, the anti scheming AI paper, uh, has a bunch of chains of thoughts from OpenAI models where they're basically saying, is this one of these redwood illusions? Uh, is this eval that I'm in really real? It seems like something redwood would do. Uh, where I think they're kind of using redwood to just mean the kind of people who try to trick models. I think this was probably based on the alignment faking paper that they started to use our name like that. But yeah, so I am well aware of the possibility of AIs reading, uh, papers and getting pretty into them. Um, our investigators considered the question of whether we should put a canary string in this to request that the report not be included in training data. Uh, and eventually they decided that it was better to uh, allow it to be in training data.

Speaker A: Well, I guess what I'd love to end on is obviously I think, um, uh, we'll learn so much in the next year about where this all goes. And so I feel like, um, we did an episode with the, uh, AI 2027 folks last year, and I thought that's one people constantly come back to and relisten to. And so I feel like these episodes get a lot of re listens. And so I'm wondering a year from now, as people are re listening to this, um, what's one observable thing that would make you feel way better about the path we're on and one thing that would make you feel way worse?

Speaker B: I think even if we have no ability to read the model's reasoning, or more broadly, if the models are now much, much more capable of doing complicated thinking on various topics without us being able to observe that they're thinking about those topics by looking at them so much, obviously looking at the chains of thought, that would be a big reason to be scared. That would be a big negative update. I think the Main positive update would be, uh, if we had a setup where AI companies were regularly allowing independent experts to evaluate the safety measures, uh, and comment on whether those were good or bad, then unless these reports are coming back extremely, extremely negative, I think that would be, uh, evidentially, uh, a positive update on how things were going to go and either way, a step in the right direction.

Speaker A: Buck, this has been fascinating. Um, really appreciate you coming on the podcast. I know things must be insanely busy these days, uh, and so really appreciate you taking the time to talk our listeners through all this. I think, uh, folks will learn a ton from the discussion here. Um, and I do want to leave last word to you. Obviously, you've dropped a bunch of blog posts and other things that you guys have written that we'll link to in the show notes. Anything else you want to draw people's attention to or, uh, anything else we should end on?

Speaker B: I mean, wonders that we're hiring. Uh, if you want to help, uh, assess whether AI companies are behaving recklessly and dangerously, uh, I think Redwood, uh, is a great place to work on this kind of stuff. Um, Meter is of course also hiring. Uh, so I would consider, if you like this kind of stuff, if you think you'd get along well with people who love thinking about these kinds of questions, uh, and you are ready to do some crazy hustling, uh, you should check that out. Redwoodresearch.org careers uh, I guess I'll also say I really appreciate the efforts of all the people who, uh, made this investigation possible. And this includes the investigators, of course, various staff at Meter and Redwood, who, uh, enabled this. Many staff at OpenAI also worked really hard to make this investigation happen. Uh, so I'm very grateful for them, uh, to them for their work.

Speaker A: Awesome. Um, well, Buck, thanks so much. Really, uh, really appreciate it.

Speaker B: Yeah, great to be here.

Speaker A: I'm Jacob Efron, and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a, uh, nights and weekends, in addition to my day job as an investor at redpoint. But our ability to get these incredible guests. Only makes this whole thing work. And so please consider doing that. And thank you so much for your support and listening. We'll see you next episode.