Unsupervised Learning · 2026-07-09 · 1h 15m
Key moments - from our scoring
Substance score
45 / 100
Five dimensions, 20 points each
This episode features a recorded debate between the host (who previously created a critical video about Dario Amodei, CEO of Anthropic) and Daniel, who defends a more nuanced view of AI safety restrictions. The conversation centers on whether widely releasing advanced AI capabilities - specifically models like Mythic with hacking and vulnerability-finding abilities - would constitute genuine infrastructure catastrophe or manageable disruption. The host argues that releasing such capabilities to the entire public, including adversaries, would be devastating given that critical infrastructure runs on fundamentally broken, vulnerable software. Daniel counters that the solution cannot involve illiberal gatekeeping by companies like Anthropic, and that the benefits of open-source models (cheaper, more accessible AI from China and elsewhere) outweigh theoretical risks. Their disagreement hinges on competing assumptions: the host sees Amodei as well-intentioned but proposing measures catastrophically mismatched to actual risk, while Daniel acknowledges the risk but rejects the proposed solution. The debate also touches on Chinese AI strategy, distillation, geopolitics, and whether companies should be trusted to decide what capabilities the public can access.
Amodei believes AI has a 25% chance of going extremely poorly for society and proposes that Anthropic and a few trusted US labs maintain control over the most capable models, limiting public and competitor access (especially blocking distillation and open-weight releases) to manage this risk.
The host argues that while Amodei's concerns may be sincere, his proposed measures (restricting access, blocking competitors, limiting distillation) are wildly disproportionate to the actual risk and amount to paternalistic gatekeeping that ultimately benefits Anthropic's market position.
The host argues that releasing models like Mythic would be harmful because critical infrastructure is vulnerable, but advocates ripping the band-aid off quickly rather than restricting access indefinitely - society would adapt and fix problems as they emerge.
Daniel agrees the risks are real and serious, but argues that restricting capabilities through corporate gatekeeping and policy favoring a few US labs is worse than accepting the disruption from open access, especially when competitors like China release capable models anyway.
Daniel points out that models like GLM 5.2 (a distilled Opus clone) are already available cheaply and openly, undermining Anthropic's rationale for restrictions; attempts to block this would simply deny Americans the benefits of cheaper, open AI while other nations release them anyway.
Our reviewer’s read on each dimension, with quotes from the episode.
The conversation covers well-trodden ground in AI safety debates - extinction risk, open source vs. closed models, and government regulation - without introducing substantively novel claims. While both participants articulate their positions clearly, the core disagreement hinges on empirical beliefs about AI risk that neither party advances with new evidence or frameworks. Significant portions involve restating positions rather than building new insights.
if we accept the worlds of like runaway AI, that might kill everyone and all of these very much extreme positions that like, if you believe all of this is true, uh, it leads you to conclusions that are still different from what Dario is doing
the belief that he has about the risks, right? And then the measures he wants to take in order to address those risks are like completely out of weight from each other
The episode reiterates standard positions from both the AI safety alignment community (recursive self-improvement, extinction risk, controlled release) and the open-source/competition advocates (regulatory capture, benefits of distributed access). The framing as a personal moral debate between two individuals discussing Dario Amodei is somewhat novel in format, but the underlying arguments - risk vs. libertarian access, necessity of regulation, Chinese competition - are well-established in existing discourse. No counterintuitive frameworks or first-principles reframing emerges.
Nick Bostrom
Less Wrong ideology
The host interviews another independent podcaster/commentator with cybersecurity and AI safety exposure, but neither participant holds significant operational authority in AI development, policy, or deployment. While the guest has studied at Oxford and HK, and claims cybersecurity expertise, there is no demonstration of shipping products at scale, building companies, or making material decisions in AI governance. The conversation features two informed amateurs debating rather than practitioners with skin in the game at scale.
I was at Oxford, which is where Nick Bostrom was
I happen to study China quite a bit. I've read like dozens of books about it
The transcript relies heavily on paraphrased positions from Dario Amodei's essays and interviews rather than specific metrics, timelines, or financial data. References to models (Mythos, GLM 5.2, Opus) and pricing comparisons (one-fifth of Opus cost) provide some concreteness, but most claims about Dario's beliefs, intentions, and proposals are asserted without direct quotation or documented evidence. The wet lab scenario and cybersecurity examples remain largely hypothetical.
I can pay one fourth of the opus pricing to get an opus. Basically an opus clone like GLM 5.2 is an opus clone, okay. And it costs one fifth of opus
Glasswing is like clearly an opus clone
The host pursues genuine disagreements and asks substantive follow-up questions that probe the guest's claims about risk, competition, and Chinese strategy. However, many exchanges become circular restatements of core positions without sharp pivots or productive pressure testing. The host occasionally concedes or sympathizes rather than pushing back with counterevidence. The conversation covers ground thoroughly but lacks the incisive questioning that would force either party to update or defend specifics rigorously.
Do you agree that it would be bad if the public suddenly received like, um. Let's say mythos or further level
What would a person who agreed with what I just said, what would they do relative to a person who is saying, I want to control everything
Computed from the transcript - who did the talking, and the words that came up most.
A debate with Zack Korman on the morality of Dario Amodei specifically, and Anthropic in general. Zack's version of the video: Become a Member: See omnystudio.com/listener for privacy information.
Transcribed and scored by The B2B Podcast Index.
WEBVTT - Debating the Morality of Dario Amodei Daniel. Hey. How are you? Good.
How are you doing? Not too bad. Great. Thanks for doing this, by the way.
Totally. Yeah. Yeah. It was.
It was. You were the one who said we should do it on camera, so we're doing it on camera. Uh, I think there's a lot of people excited about it, so, uh. I'm happy.
I think we have, uh, so the chat is apparently a little bit messed up in terms of like time delay and stuff. Uh, but we'll just kind of ignore the chat for a bit and just assume there's an audience who cares. And yeah, just do this between us. Uh, sound.
Okay. Sounds perfect. Great. Uh, nice to meet you.
By the way, you and I have never actually talked on a call before. Like, we've only interacted on the internet, so. Good to meet you. Yeah, good to meet you.
Cool. Okay, so what I thought we'd do is just. I kind of want to start off by just giving like, a little bit of a summary for people of what, what the interaction was. Uh, and then I'll just let you kind of give your take on what your opinion was, and then we'll just go and have a discussion from there.
Cool. Um, or maybe you can give the background of like what your opinion was that I responded to. Yeah, yeah. So I'm going to start from where I, yeah.
So okay. So for anyone who is like just new to this or okay, so what has happened is I made a video, uh, of 18 minutes of me just trashing Dario Amodei. Okay. Who was the CEO of anthropic?
Uh, just saying. Basically I think he's a bad person. Okay. Or, um, bad person is maybe not the word I use, but it was, it was along those lines.
And I basically said, the problem is, is that what? And I walked through in this video, uh, Dario's, uh, position on AI safety. And for anyone who doesn't, um, understand some of the context, this can only give a little more information. I am a little bit fortunate in that I happen to be when I was doing my master's, I was at Oxford, which is where Nick Bostrom was.
And Nick Bostrom is a person who was like very early in or like a big name in the AI safety debates. So I went and I heard him speak at some event. I forget what it was exactly. Uh, and so, and then that made me read up on all of the AI safety discussions.
And this was like 2016. So I'm like a little bit like more well versed in AI safety as, um, than your average person would be just because I just happen to be at the place where they were talking about it, like where it was kind of was a big idea at the time. They have like a whole thing called like the future of humanity. Um, and so I make this video where I basically walk through what Dario believes and then walk through what Dario wants.
And then I say basically, uh, no, uh, fuck that. We're not doing that. Uh, everything Dario wants is bad. And I think that we should all reject it, and we should all just completely reject Dario completely because he's trying to lead society down a path that is bad.
And specifically my argument comes down to this, which is that Dario believes that AI has a 25% chance of going extremely poorly for society. That's his position on it. Okay. And in that world, we want to he wants to make sure it goes well.
And his basically, his proposal is like, help me regulate all of my competitors. And, um, let me continue to run AI so that he gets put in a position where he runs the safety of AI and he can keep you all safe. Uh, make sure that you're all safe by not giving you access to anything dangerous. Um, only he gets access to that.
And so I basically called that out and made fun of it for 17 straight minutes. Uh, and then Daniel, you wrote a post and then a number. We went back and forth a bit. Do you want to explain what you said, what your position is on this?
Yeah, I don't really remember what I said, but I but what, what you said just now reminds me of like what my position is. So, um, so I guess at a high level, my position on this, um, is that what you're saying? Sounds perfectly logical. And I, I actually think we probably agree 80 to 95% on like pretty much everything here.
Um, and I kind of just have a way of thinking about this, which is a little bit different. Um, so I essentially think that if we take certain things into account about what we want to happen in the world and what we don't want to happen in the world, the question then becomes, okay, what do we do about it? Okay. So, so let's just step through these because I think we probably actually agree.
Um, do we agree that we don't want like a government, um, authoritarian government who has like AGI, ASI or whatever that just like subjugates the population. Yeah. Subjugation of the population is bad. So I think.
Yeah, that's bad. Yeah. Um, so we don't want that from the government, but we also don't want in your mind like a Dario type or like a, you know, super rich, ultra rich, like oligarchy, who's on the corporate side doing the same thing. Right.
Both, both like government or corporate. It doesn't matter. Like either way, if it ends up with us subjugated, that's bad. So 1,000% agree with you there.
So I agree with what you're kind of like attacking. I think you're attacking the wrong person in the wrong entity. Um all right. So let's keep going.
Do you agree that it would be bad if the public suddenly received like, um. Let's say mythos or further level, um, ability to, um, create bioweapons, um, planned terrorist attacks, uh, create end to end like vulnerability research all the way to exploitation. So I think you have to break that. So if we're talking the cyber capabilities.
No, I don't think that'd be bad. I think that'd be good. I think like from a cyber perspective, if the whole public got access to like mythos plus plus, let's call it like mythos seven, and it's like the greatest hacker of all time. Like I still think that's net positive for the world because like cybersecurity is not magic.
So like, there are still ways to protect, like, it's not like, you know, a mythos seven can not just hack anyone, even if like, there has to be some issue to hack. Right? Um, so I'm not really, I don't. Okay, well, maybe this is our, maybe this is our first level of disagreement.
Um, uh, let's see. Okay, I guess we can go tactically down into here. So it's not my main point, but but maybe this is part of the main point. So you're saying that it would not be bad for Society.
Because keep in mind all attackers who have been attacking us this whole time. They would also have this capability, right? Sure. Because the whole public does.
Everyone has it. Um, you're saying this would not be a problem? No, no, I mean, it would not. A problem is a bit of an understatement.
Like, obviously it would be inconvenient and it would be like a little painful. Um, but like, is it a thing that I would go to bed worrying about? Like, no, not really and I don't. Okay.
I think that's what's true of a lot of people in cybersecurity, as we're pointing out that like, I kind of asked that question of like, how bad would it be if mythos level capabilities were available to everyone? And the answer was, like most people in cybersecurity that I, of course, biased by the people who follow me, but like it's a pretty big pool. Uh, we're like, yeah, it'd be a little painful in the short run. I think that there's no reason to take actions.
And I think this is sort of comes back to the original position I have, which is there's no reason to take actions based on those types of fears. If the actions themselves require us to do things that are like wholly illiberal. Right? In a generic sense.
Liberal, right. Okay. Okay. So a couple of things there.
Um, so I, I think we're very far apart on how bad this would be. Um, so let me ask you this. Do you not believe that most of our software and, and mythos is essentially like showing us this? And it's not just mythos, like a good AI stack on a previous level model can actually do the same thing.
I've been doing this forever. Um, that it can reveal the fact that our entire infrastructure, basically everything that our society, by the way, your video's frozen for me. I don't know if it's frozen for other people, it's probably probably just my computer. Um, but just in case it's not, um, basic, basically my argument is, um, and, and just like you were saying, people agree with you about this.
There's lots of people agree with me about this. Um, our infrastructure is basically a giant soup sandwich. It is a complete mess. Um, Linux is massively still vulnerable.
Windows is massively still vulnerable. All the critical infrastructure runs on software that is full of holes, which we simply haven't taken the time to look for. And if we did take the time to look for it, we would find serious issues. So if the whole world, including all the bad guys, suddenly get the ability to uncover those, that is not an inconvenience.
That is seriously bad. Especially if you do it all at once and you just release the model if this suddenly becomes good. But your your take on it was that this is not just mythos thing. It's also true of things like, you know, if you're running a lot of stacks with like, okay, other models that you can use to do, but those, those do exist.
And it's not like nothing bad has happened yet. No, no. But they require extraordinary. Sorry.
They require extraordinary skill to create and manage. And only a few people have them, right. Me and my buddies, we have them lots. Lots of other hacker buddies.
Like we have these stacks. And so it is a you know, the top of the top has these stacks and has been able to do this type of stuff. And the models like improve them. But the more you go mythos mythos plus plus, it's like a single prompt.
I want you to achieve this. Believe me. Yeah, yeah. Well, I don't think we, I don't think we disagree on the premise here.
I just think that basically what happens as a result of it is if you were to do something like, let's say, hold these models off for release to try to like solve some problem in advance, like try to gate access, whatever, like the solution to that might be, it's just going to be so much worse than just saying like, let's just rip the band aid off. Some problems will happen. We'll fix the problems like no, nothing is going to like the world will go on in very normal ways. Like I think that if you release those capabilities into the world, the world will go on and it will just be like, yeah, some bad stuff happened along the way and we fixed it and we're good.
Okay. And so like, I don't think that there's any major reason to be like in a panic about that besides, like, you know, compared to the types of especially the types of proposals we're talking about, which would be like, okay, well, now we need to limit what open rate models can be put into the public, right? We need like, these are extremely draconian measures to prevent something I think is a shrug, you know? Okay, well, a shrug, okay, a shrug.
Okay. All of our financial systems are based on this software. All of our economic systems are based on those financial systems. Um, our critical infrastructure for running, uh, water, energy, everything, everything is based on running that same exact shitty software, which it sounds like we agree.
We know it's shitty and we are now releasing into open source models or or openly releasing mythos from closed source models, whatever. We're just giving it to the entire world. And you believe that is a shrug. I believe it is like, yeah, effectively, I think that basically we will have.
Well, I think we found I think we found the disagreement. Uh, I think that is just wrong. Okay. But like, but that I think gets actually to like the deeper point, which is like, okay, so imagine that you're sitting there going like, okay, I believe that like all of these terrible, that mythos is this amazing model and that if we release it to the public, like lots of terrible, terrible things happen, right?
The, the follow on steps and actions from that are like, okay, so what do you do? Like what is, what's your proposed plan to that? Totally. 100% agree.
So let me just say this. I want to I want to make this very sort of agreement based. I agree with you that if this were a minor situation, okay, if this were a minor situation and it wasn't that big of a deal, and then you have someone like anthropic or someone like Sam or Dario or whoever, and they're like, mm, I want to control everything. And like, we can't let this get released.
And they're clearly doing this for many reasons. Then I see that the same way it seems like you're seeing it, which is these guys are just trying to control things like this is nasty. And I guess what I would make a 17 minute video bashing them just like you did. The difference is I don't believe that is what's happening.
I believe what's happening is they believe, just like I do, and tons of people in the military do, and tons of people who, in my opinion, who are close to this are like, no, that would be bad. That would be seriously bad. Well, to clarify, I believe Dario believes that too. So of course, one of the things that makes me diverge from some of the people who just believe it's regulatory capture play is that I fully believe Dario believes like, so I don't I don't for any moment suggest that Dario is doing it all just to exploit something for his own personal gain.
I think Dario is is deeply versed in the world of like, you know, less wrong ideology of, you know, less wrong being a blog for anyone who doesn't know. It's not a. Rationalist. Movement.
Yeah. You know, e world. Like I think he's so deep into that he believes it fully. So I don't accuse him of not believing, uh, what I think is that basically, um, the belief that he has about the risks, right?
And then the measures he wants to take in order to address those risks are like completely out of weight from each other. And for one of the reasons we found we disagree is that, of course, I don't think that releasing mythos publicly would have been the end of the world. But at the second time. On the other hand, I also think the measures he's proposing can lead to some extraordinarily terrible things, right?
You know, the measures he's proposing is basically to say, I want to make sure that China does not have capability to develop models. Okay. Good models at least, I guess. Do you?
I guess the question is, do you agree with his proposal there? Like, should we be blocking distillation and should we be blocking open weight models? Like what do you what do you think there? Right.
So this is completely consistent with the belief that I have, which I believe is the same one that he has that mass release at extremely high pace of all this stuff is very bad. So we're trying to like control the flow and the speed to like get society ready. Um, so I don't believe we should give this to the CCP immediately. I don't believe we should give all these capabilities to the public immediately.
I think it needs to be done slowly, um, and carefully and thoughtfully. Um. I don't believe that he is. People will characterize him as being like, oh, he hates open source.
He hates all this stuff. I think what he's saying, and I think is clear from all his interviews, what he's saying is like, we got to be careful. Okay. So so let me ask you this.
What would a person who agreed with what I just said, what would they do relative to a person who is saying, I want to control everything and I don't want to give to anyone? No, they would do, of course, exactly what Dario's doing because I don't believe that it's about I don't believe that Dario's doing it because he wants to control it for his own good. I believe Dario is doing it, and I know a lot of people in cybersecurity disagree with me, but I believe that Dario is doing it because he wants to control it.
For your good and my good and all of our goods. Right. Um, Dario believes this is for the betterment of everyone for the reasons you're stating, like. Right.
The problem I have with it is that the what he's basically proposing is that we trust him and that we take a hit on our side. So, I mean, if you think about it this way, if we had done what anthropic had asked for us and has been asking for for years and years and years and all of their policy proposals and stuff, especially for the last year, uh, we would have to pay opus level pricing for an opus level model. Okay. Whereas as of a couple of days ago, now I can pay one fourth of the opus pricing to get an opus.
Basically an opus clone like GLM 5.2 is an opus clone, okay. And it costs one fifth of opus 4.8.
And that is like a huge net benefit to society and to every American, like basically to every American company, to every, you know, Western company, to every company. Uh, that's huge. That's a hugely beneficial thing that we would have not allowed had we followed anthropic policy proposal. Uh, is he saying we should limit things like GLM 5.
2? He is very clear that we need to limit distillation. He wants the US government to make an attempt to limit distillation more. Okay, so he's like, that's one of their policy proposals from like six months ago.
Well, isn't distillation just isn't distillation? Isn't he trying to limit like the CCP's ability to distill from our top models? Yes. And that's what GLM did, obviously.
GLM is like clearly an opus clone. Like, I think it would be very hard to believe that they trained GLM without hitting, without basically training it on Opus. Like they, they behave identically. Yeah.
I feel like this is a little bit of a diversion, but I'm happy to go down. So basically I think it's, it's blatantly okay. Do you agree that the CCP's China's overall strategy of just blatantly, blatantly just stealing from everyone on the planet to better their country is like not a good thing? Well, well, if they're releasing it to the public, like as an open weight model the way that they have been.
As long as that is there. For the for the purpose of damaging the United States, that is, the strategic goal is to. Yeah. But how how did they damage the United States for releasing GLM 5.
2? Like that's just it just doesn't I don't. See no, no, the overall Chinese AI strategy. Um, and to just to get my bias here.
So I'm like ex-military, uh, I wasn't like super elite or anything like that. But my bias is that like, I see, I see things in terms of like, what, what happens if the adversaries win? What happens if we win? What does it mean to want somebody to win?
Why does it matter? And ultimately, like, I'm a Star Trek liberal. So like, I think we should all just get along anyway. But in the meantime, the CCP is stealing everything, trying to win because they just want to win everything.
So the strategy for China is to completely underpin US dominance via AI, by harvesting everything through distillation and making it open source and getting it into open source as fast as possible. But that's but you think about it from my perspective as just like an American citizen, which by the way, I know a lot of people on the internet think I'm not. But like American citizen here, I. Was trying to figure that out.
I'm like, you do an amazing American accent. Yeah. For an Oxford guy. Yeah, I'm American.
Just happened to move around too much. Um, you know, uh, the thing is, as like, an American running like a company and, like, living my life and all of that, it feels a very much like China's helping me, okay? They're giving me great models at low prices. Right?
Um, open weight stopping these. Okay. And then the question and then the question becomes, if you want me to believe that it's all like, this is a plan from China to hurt Americans and all of that. It's like you have to be willing to give something quite, quite strong there.
Otherwise you're asking like people like me to give up all of the benefits that we get from this in exchange for like, some global foreign policy that really doesn't seem to affect us much at all. Like, I don't notice these things in our normal life. I notice like the fact that, okay, I now can't even use this model or I can't do that. Like for my daily life and for my work.
Chinese models giving me like open weight models is like just positive for most Americans. Yeah. And they're subsidizing it. So it's cheaper too.
It's like very nice of them. So, so basically it's true. Wait, but that's not true because from the perspective of inference. So if you meant inference there, that's not true because like Google Cloud's pricing is, is like $1 per input token, which would be compared to like $4 per input token for opus.
And of course, Google Cloud's like, that's just Google cloud. It's not China subsidizing that. Of course, you could be saying like that they capitalizing the cost into the. No, I'm saying like.
They really are. I'm talking about like EC2, like a whole bunch of Chinese models that are available that people can use as alternatives to US models. They are super cheap, and you're literally taking all of the stuff in your harness and all your cool ideas, and it's going off to China. I'm sure they're doing nothing bad with it, right?
But that's not true for like, but that's not true. If you're running it on like Google Cloud. So if you run any of these. Oh, sure sure, sure.
Hosted. Right. And they're still cheaper. So the point is it's not about stealing our data because they're not getting the data.
It's not about that. They're forcing us to use their AI APIs because they're not. And that's actually one of the things is that that's very hard for me to accept, I think very hard for a lot of people to accept, which is like Anthropic's plan is to say, uh, we and maybe a few other labs in America will have the best models. You will have to go through us for your own safety.
Um, we will block the companies that were willing to give it to you cheaper and better because for your own safety. And, um, and no, we don't have any intention or plan of giving you cheaper models, right. We don't have any intention or plan of giving you better models that will be able that will serve your actual purposes. So you see why that's such a hard sell.
Like politically. I hear you, I hear you. That sounds it sounds awesome. Sounds amazing.
Um, it sounds amazing. It really does. Like I'm not being facetious there. Um, the, the issue is that, okay, let me continue to propose this, this training strategy, um, the, the concept of, um, the US companies having a stranglehold, like super strong control, the best ice that just keeps getting better and better.
And China is kind of behind that is not okay for China. So they decided, I don't know when like a year and a half ago, three years, who knows to go the open source model, just really everything public, public, public. Um, now, so I happen to study China quite a bit. I've read like dozens of books about it.
They, they just clearly state what their strategy is, which is to win, and for the U.S. to become a minor power with them being a major power like this is not conjecture. Like, they just write about it.
They talk about it openly because they think no one is is actually reading anything. They're writing because it's in Chinese. So it's very clear what they're actually trying to do. Right.
Um, and the fact that they're giving out, oh, we're just free and open. It's not free and open in China. It's not free and open in Hong Kong, which they took over. Um, it's not like they're going to give us all these models and then we're going to be like, yeah, so what are alternatives to Chinese government?
And they're like, yeah, that's a perfectly, you know, natural question you're allowed to ask. It is allowed. We are allowed to ask it in the US. We're not allowed to ask it in China.
So like, I don't think I mean, you're you're you're kind of jumping ahead to a world in which I agree with you. Okay. Both of you and I agree in the Star Trek world. What are you talking about?
All this should be open source. I 1,000% agree with you. This should all be open source. We shouldn't have these crazy governments like me.
And you are probably Star Trek liberals. I'm talking about how we get from here to there. And I don't want an extra step where CCP runs all of this. Um, now, now, let me ask you this.
You're saying that the actions anthropic is taking is like really, really bad. Wasn't his action to go and get all of his competitors and get them all on board and do this shared community release of like, we're going to give this to hundreds of companies, they're going to come back, we're going to roll it out slowly for the purpose of doing the thing that you said, which is you kind of like, do the controlled burns in the, uh, in the forest. You burn things down, you clean things up, then you start releasing it.
And now I would agree with you, once you've done that, maybe it's okay now we can release more, but you don't just drop it. Um, because here's the thing, dude, they don't they don't hate money. They like money. They would have made so much more money if they just released mythos.
Of course. And, and, and believe me, it's because I believe that they're believers. Like, I think it comes down back to that now on the China point, like it's important to understand. Like, so I actually, I studied at the University of Hong Kong law school for a year, uh, the year prior to Hong Kong, basically getting completely, you know, done, uh, and that the protests that kicked off actually kicked off from the University of Hong Kong law school.
So I was like at the epicenter of this point. Um, and I have tons of friends who are, uh, from mainland China, including one who by all accounts, I will, you know, can pretty openly say he is like clearly a spy, uh, for China. He works for in the US for a tech company. Um, and I happen to know his background.
Right. Well enough to know that he would be a spy for China. Um, and so, but the fact of the matter is, and like, no, of course I don't think that's great. Um, you know, at the end of the day, I'm sitting here going, yeah, but they are giving us models that.
No. Yes. If I asked about Tiananmen Square, it would deny that. Right.
Like it will deny that request. Okay. But I don't typically have to ask my AI models about Tiananmen Square. Right?
Like it's not it's not a thing that comes up when I'm doing like any type of cybersecurity work, right? Those types of like, can they influence it with pro-Chinese bias is such like an abstract concept of how you might influence a population in comparison to like, what if only Dario controlled it? And now when you mentioned it like, okay, but that it was that he went and got all of his competitors together to this is just what we call like collusion, like it's illegal for a reason and normal circumstances that would be a crime.
Uh, we're willing to accept it because we pass it off through the safety exception. But in normal circumstances, putting together, like all of the leaders of all of the major companies of a Get competitors and saying, how about we cooperate on this would just would just be illegal because that would be a cartel, right? So we don't want cooperation. I do not want cooperation between these labs.
I want them to compete. Okay. But they're competing. You're familiar with like the Malek, like you're chasing this thing that you know is going to kill you and you're you're aggressively chasing a thing.
Everyone knows if we get there, it's going to kill us, but we can't let the other person win. So we accelerate like that. That's that's what they live in. That's what they believe.
Right? Well, no, but that's what they would do if they weren't collaborating in some sort of way towards a positive outcome. Right. So so they literally said, and Sam has also said this earlier, um, I've a whole, whole nother talk show on Sam, but, um, they're not just saying, let's collaborate.
They're saying, let's collaborate with this thing, check this out. This concept of government is the idea that the people are involved. So it's them, all of them getting together with the government to say, hey, what is the best thing that we should do? That is literally the consortium that that he is arguing for and, and saying he's saying, look, we need to be regulated.
People need to be involved. We want all of our competitors to be involved. We're trying to control the speed that this comes out. And that's literally what he's proposed and.
Done, because I don't want them to cooperate on the speed with which it comes out, because that means that they can both say, hey, because what I thought when they didn't, by the way, in the chat, there is a guy named Daniel that I worked with as a developer for years ago. That's why I laughed because like, yo, what's up Daniel? Uh, just like it's been like eight years since I talked to that guy. Uh, but anyways, um, back on the point if we go back to that though, is that if you think that like, uh, when mythos came out and they said, we're not going to release mythos to anyone, I was like, watch it.
Because what happens is, of course, if Sam Altman releases his version of mythos and like, then of course it kind of forces Dario to release mythos, Right. So the game like the game here of competition forces them into the world that like I want to live in, which is that like, we just get the best models and we get the best prices and we get it the fastest. That's the world that I want. And so the, the collusion they're trying to do is to avoid that world, but it's coming at a cost and the cost is not on them.
The cost is on us. They're already trillionaires. Right. It's like the permanent underclass that loses from this.
It's all the people who are not trillionaires and do not control the AI, you know, or are not friends with anthropic, because of course. We still have. We still have. I'm sorry.
Yeah, but but they but but opus being much more expensive than what we got from the labs that he doesn't want to have it now, but also importantly, um, you know, you mentioned what I was about to say there, uh, I forget whatever that was it. Okay. So I, I think, I think I'm hearing something strange in the argument. The reason I reacted so viscerally to your video is because you were saying Daria was a bad person.
Now you're telling me I agree with you. He's a good person. But the output, the output that he is asking for would be bad for us. No, no, no.
I don't think he's a good. I don't think he's a good person just because he genuinely believes. I mean, like I said, I think in the video there are people who have believed in like ethnic cleansing and they, like, genuinely believe in it. Uh, but it's like still evil.
Right. And I think it's the same as that to he genuinely believes he's keeping us safe and that it's better for everyone to keep us safe and that he's doing that for safety. Um, the problem is, of course, is that he is willing by doing that, he is willing to basically create a society in which very few people are benefiting because at least they didn't die. Okay.
I think, uh, what. Are you talking. He still wants to release all these models. He still wants to release them.
No, no, no, he wants to release the current level of models. He's very clear on that. The next. and the next generations of models we will never receive.
Okay, he is very clear on the fact that he believes. What are you talking about? They're in development. They're going to come out.
What? What are you talking about? It's his quotes. He literally says, in the future, you know, we might have to treat this like nuclear weapons, not as a safety risk, but rather as a risk to society and to humanity, and that we would need tighter controls.
You know, basically the point being then it is not a public good. It becomes, you know, not not a public good is the wrong term there. It is not a good that he would be willing to release to the public ever. I think Dario would not deny that.
I think. But if you look at the context. Of that, Dario does not want to release all future models if they become sufficiently smart. He's talking about in the case where it becomes certain amount of dangerous.
And like you shouldn't just drop it on the world, which is my point. Okay. Which I think we disagree about. Yeah.
But we disagree about the potential impact of that. He believes, as I do, that the potential impact of dropping. Let me give you a better example. Okay, let's call this mythos six or mythos seven or whatever.
Okay. Um, give me I don't even know if I should say this prompted. Um, yeah. Okay.
Imagine the creation of a wet lab combined with doing the research for creating, uh, uh, different, uh, pathogens that combine the worst of the worst. And the problem with this is that the knowledge has been there. Like if you could jailbreak or whatever potentially been there, but as you start to move into AGI, ASI maybe mythos 6728 who knows when it's going to happen? It starts to be like, oh, actually, if you make a cut here in this particular virus and you add this, um, okay, this will kill everyone.
This will effectively do what your goal, you did a slash goal. And guess what? I made it for you. The problem has always been, okay, that knowledge is fine.
Kind of like with nuclear. That's fine. The knowledge is there. But you don't have a lab.
Well, guess what? I'm going to give you all the stuff to go and make this in a lab. And I'm going to help you automate the process. And now we've actually built it.
This little canister here now has the thing you believe that is safe to provide that level of knowledge to the world, the entire world. Give it to CCP, give it to Joe next door. Importantly, here, this this kind of moves into the world of what we consider like, I think Nick Bostrom refers to it as like this idea that you can make nukes like in your microwave, right? If there is some piece of technology that like we discover that would, whether that's through AI or something else that would allow like every random person to go make like civilization ending technology at home easily, right?
That would be catastrophically bad for the existence of humanity. That would be like probably civilization ending if it got out. Like that's what we're talking about there. The problem is, of course, I don't believe this.
Okay. I think that we've not seen sufficient evidence that AI will get to a level that will allow that to be possible. And with that being the case, what happens is, is that we're starting to be asked to accept, again, like fairly liberal positions of like, we will allow for the AI labs to control larger and larger. I hear you get the best models.
All in favor of this theoretical possibility. And now I'm sitting here going, well, let's say that Dario says the counter to this is, well, Zach, you don't have the access to the information like I. Dario do. Dario.
I'm Dario and I can see the model does it. The problem is of course I don't want Dario to be able to see this either. Because if it is at that level. Of course I think it should be like completely expropriated, you know, expropriated out of his hands and like the government should just roll in.
But he doesn't want that either, which is kind of a good signal that this is not that. Right. This is not telling people how to. What do you mean?
He is literally saying we should be regulated. This stuff is super dangerous. Regulation is not what you would do if you believed that you had civilization ending technology in your hands. And I think that.
He believes it's it's about to come. He believes it's starting like it's very clear from his statements. Do you think that Dario. It's a very interesting question, is if I believed.
So this is one area where I do think it becomes very clear where Dario and I disagree. The strongest, besides just about what we believe comes in the future. If I believed that that was coming in the future, there would be one and only one appropriate answer, which is to hand it over to the government and say, look, guys, this is now a matter of your this is on, you. Know, now we're now we're getting to it.
No, no, no, but not as a matter of regulation, but as just full on nationalization, because most people would accept that you should not allow civilization ending technology to be developed in a private lab, uh, and making trillions of dollars. But Dario referred to that as communism. Okay. So Dario very openly said, this is communist.
Okay. So say that again. Say that again. What did he say was communist?
Uh, nationalization. So when he said that some people argue now that because of the risks that it should be nationalized, he said, guys, you're yo yoing between, uh, full, um, basically full libertarianism and full communism. And so is of course, the implication being that nationalization was the communist position. He did not he if if, if Dario came out tomorrow and said, I think that the US government should nationalize all AI labs, I'd say I'm sorry to Dario because at least he has.
I would say he's wrong because of course, I don't believe in the pre. The premise that this is going to become make a wet lab at home, kill everyone technology. But I would at least say he's drawn the correct conclusions from the series of things he believes the problem. The biggest problem I have with Dario is that he wants to believe that, uh, this is civilization ending technology and that he should be able to make $1 trillion off of it.
And that's like super weird. Like, why would anyone think that? What do you mean? So, so, so I think our positions are extremely close here, Extremely close.
I think this is a framing issue. Okay. So first of all, the government has no ability to do anything. They they have no AI expertise.
They can't make anything. We can't nationalize crap. Um, guess what an actual nationalization looks like. An actual nationalization.
Looks like the labs get together and they're like, hey, we're inviting the government in to be a huge part of this because this is super important, which happens to be the exact thing that he's asking for. No one. So if I went and asked my parents who don't know about AI or my brother who's a lawyer, or any one of my friends who aren't in technology to explain that premise to me, which is that this technology is being developed and it might kill everyone in the world, right? It might cause basically mass, mass, mass extinction, which is the belief that he has.
Right. And then he said, okay, but they don't believe that this should be nationalized and they just believe it should be regulated. All of the people, all of those normal non-tech people would go, what? That's absurd.
Obviously, we don't want to see people, uh, having any sort of. You know, you're basically it's a negative externality that you're basically saying, what if we killed everyone and made $1 trillion in the meantime? I mean, like, what are you talking about? Of course you want to support that.
What does nationalization mean? What I'm arguing to you is that there is a smooth spectrum between the government having control and oversight and transparency into all U.S. companies who are pushing towards the same goal.
And there's like some sort of internal consortium there with the government actually having veto power and control. And and actually the labs are the ones doing the work because the government doesn't there is no nationalization there. Like, there's not a government thing that is just like anthropic and just like Google. And it's just as good.
It would be anthropic. You would expropriate it, it would be yours. Now you own 100% of the shares. Game over.
Right? It would be anthropic. Yeah. Okay.
So imagine you're Dario. Do you think that's a good idea right now, given our administration? Of course. Dario does not believe this.
Okay. Do you believe it's a good idea? Okay. Let me ask you this.
Do you think it's a great idea to hand all the power of of anthropic and Sam? Sam and Dario should go over and meet with Hegseth and Trump and say, hey, this is all yours. Um, we just got a piece out of this game because you guys are super you guys are super talented in this, and it's all yours. Take it.
Wait, I, I absolutely don't believe that because of course, I don't believe the things that Dario believes about the world. But if I was sitting in Dario's shoes believing the things Dario believes, yes, I would, because I think that even though we might say like, oh, well, we don't think that the Trump administration would do a good job or whatever you might believe. Uh, at the end of the day, a public process is like, that is the best system we have, what they're saying. And the reason that I have such objections to what Dario's position is, is to say, you can't trust this government which was elected.
You can trust Dario, who wasn't elected. As long as we get a light level of regulation from the government. No, I think that absolutely, if you were to move to this fake extreme world, which I do not believe in, you would believe in in just full nationalization. No one should be profiting off of taking a 25% risk of everyone dying.
Okay. So like, that's a very basic premise, which is like, if you believe there's a 25% chance of extinction risk, which admittedly is not what Dario said. It said 25% chance of very, very, very poorly. He didn't call it extinction risk.
But if you believe 25% chance of going very, very, very poorly, I don't believe that we should be driving that as a for profit profit enterprise. That would be basically just saying if if it goes well, we become your new trillionaire overlords. If it goes poorly, you all die and who cares, right? That's that's just not an acceptable state of the world.
I hear you. So so I think you're talking about worlds that don't exist. Zach. That's what I think you're talking about.
100% agree because I'm talking about the world's Dario has put forward. No no no. I think you're talking about worlds that don't exist. Okay.
I believe that this progress is is happening. Um, I believe Dario when he says they were surprised by mythos. Do you agree with that? Of course.
Yes. Okay. So he was surprised. He's like, Holy crap, this is way better than I thought it was going to be.
This is happening faster than I thought it was going to happen. Okay. Do you agree that most AI experts are worried about the future and quickly being able to have this kind of capability? Yeah.
Most AI experts, yes. But that's that's a narrow pool of technology. No, no. But what what I'm what I'm saying to you, Zach, is that there is a consensus.
Everyone knows. Every book written knows. Holy crap. We are going into something very strange.
Uh, we're at the point where things like auto research can just make AI better, right? And it's slowly happening. Like, this is such an inevitable thing. Yes.
Okay. Okay. But yeah. Go on.
So so it's it's inevitable. We agree on that if it's inevitable and. Well I. Don't.
And hand in handing it. Yes. Yeah. Handing it to.
Well I don't I don't think anthropic. If they shut down tomorrow they literally turned everything off and went home. That would not. It would be a blip.
It would be a blip. Of. Course. And overall.
Progress. Anthropic should shut down. I'm you know. Right, right.
So this is why I'm saying that that we're not arguing in a realistic way. If we're saying this is inevitable and anthropic shutting down tomorrow would would only be a blip in the progress. Then the reality we're dealing with is this is happening and we need to try to do the best we possibly can. Yes.
Okay. And one, one thing that is not the best we can possibly do. One option is handing it over to the current US administration, and walking away and going home with your billions and starting a farm. That would be super cowardly.
But but okay, but let's get back to the point. What I believe the best thing to do is that they just keep on building these models in a competitive way. Where OpenAI builds their models and anthropic builds their models, and maybe Google decides to build models again, and China builds their models. And we do not do anything as the US government to regulate China so that the American people can get the best models at the best prices.
Right. That's what I believe. Okay. Uh, fundamentally, because if we accept the worlds of like runaway AI, that might kill everyone and all of these very much extreme positions that like, if you believe all of this is true, uh, it leads you to conclusions that are still different from what Dario is doing.
Okay. Which would be like, you should believe in nationalization at that point. Like if you are publicly minded, right? Uh, well.
But right now, he wouldn't want to give it to them. He wouldn't want to give it to them. Who gave? But who gave Dario the choice to make that call?
If it is what he says it is, Dario has no right. Okay. It doesn't matter that I don't I don't want you know, it's like it doesn't matter. It's like at the end of the day.
He has no right. To say yes. He has no right to do what I don't understand what he shouldn't be allowed to do. To decide to say, if you, like, believe all of these things and you're actually sitting in the shoes of saying it's going to kill everyone, or it might kill everyone, whatever it may be, unless somebody really takes leadership and Dario decides that Dario should be the person to take that leadership instead of, for example, the public, like the US government, right?
Like Dario, Dario is taking a pro-u.s. position, which is the US should get there first. Yeah, but but Dario is not doing that through any sort of public apparatus.
Dario is doing that through a private. Yes he is. Yes he is. He's a private company.
He's attacking Jensen uh Wong publicly against everyone else, saying we should not be shipping our best GPUs to China. I know, but the point is, is that he's taking a pro-American position that also benefits. It's not because he benefits. No, it's not because he benefits.
It's because he believes. It's because he believes that it will make him remain in power so that he can lead the world to the best path of the future. Like I think he does believe that he would be the best person to lead us to not dying. Okay.
I think he does believe that. The problem is, is that. He believes he's one of the best people. One.
Exactly. The point is, I find the whole premise like kind of repulsive, which is that like, if I were sitting like, I believe just to say it from this angle, like I believe very much in myself, right? I believe like I am a very good guy and all of that. But if I was sitting in the shoes of saying like, okay, I have this great technology, blah, blah, blah, and if I were saying like, and I trust myself more than the public system, I would still hand it over.
Okay. And I would hand it over because I believe that there's like a process issue here, which is that like, no one picked me to decide that, okay, even if I pick me. Dario was saying pick me and this can. If you think about this future state of the world, imagine that.
Like you believe in this recursive take off, like, you know, self-improvement, you know, at each point. Dario could keep saying, well, I'm actually the best person to be the dictator and I'm actually the best person to be in charge. And I'm actually the best person to be. Right.
We don't want that. I want to make sure that that doesn't happen. I want to make sure that if this fast take off thing happens and that self-improvement happens, it is not controlled by somebody who thinks that they should be in charge of it. Yeah.
So this is fascinating that we're talking about this. I'm reading the culture series right now. Um, I just got done with the games one and, uh, whole bunch of conversation about this. It's, it's crazy.
So it got me to reading, uh, the Machiavelli stuff because, um, that just came out on Dwarkesh podcast and that got me to reading, um, Hobbes Leviathan, which is like, which goes back to Plato, which is philosopher king. And, um, what if someone believes that they're the right person or whatever, and they just think they're the best and you want to give them all the power? So I see the thing that you're getting at, which is like you're saying, it's kind of gross when someone is like, I am the good guy, I'm going to do my best.
Right. Yeah. Do my best individually without, you know, obviously everyone should. No no no no no.
But see, here's, here's here's my bigger point. Okay. I think there is a really gross narrative which says that if somebody believes something in a positive future, they believe that they can help in some way. They believe they can contribute and they they go into the arena and they fight for what they believe in, that people like us should sit on the side and we should be cynical.
We should assume the worst of them. We should attack them. And I think that is super wrong. I think it's super fucked up.
But I don't assume the worst of Dario. I assume the best. Dario. Dario is terrible for it.
No no no. You are assuming the worst of Dario that your video is assuming the worst of Dario. Which thing do you think that I believe about Dario. That's not true because I, I thought in my view that I expressed every opinion Dario has the way Dario has them, right.
Like I, I tried to um you know, I'm not making any assumptions about his intention that is different. I tried to say this is his intention is what I really believe is true. If, if I don't think there's anything in that video from that perspective that Dario would say was wrong, at least besides the part where I call him shitty. Right?
Um, no, no, but but what? You're what you're saying. I'm sorry. Go ahead.
The next piece of that, though, is to say just because somebody is like you're saying, okay, they're out there, they're trying to make the world better. They're trying to, you know, like I said, there are people who've committed like, like genocide under those terms, right? They genuinely believe that like, you know, you know something, some group is after them or whatever. And then and they're out in the arena trying to tell people not to, you know, trust this ethnic minority group.
And then people are critical of them. And we say that they, you know, you see, it doesn't it doesn't change anything. It's like it doesn't change the in conclusion, which is that like, in the end, what Dario wants is bad for all of us. And he wants it kind of knowing that he kind of knows it's bad for all of us because he doesn't believe it's as bad as what could happen, which would be like extinction or like genocide or things like that.
Right. Okay. So it sounds like you're arguing against E. Oh, yeah.
Oh, 100%. Yeah. Well, so, so what I haven't studied is how close he is to e my read on him is like he doesn't care at all about that shit. He literally is going in a different direction.
Um, I hear, okay, let me just I'm going to steal your position. Um, he thinks he understands the world. He thinks he's trying to do the right thing and he's wrong in some sort of way, but he is mostly good and he just is like Jesus like syndrome, where it's like, I'm, I'm going to save the world or whatever, and he's wrong. And if too many people give him the power, um, he'll end up doing bad things for us because he doesn't want us to have all the toys or whatever.
Not, not quite what I would say. I would say it's not just because that would not make him bad. That wouldn't make him a bad person. Okay, so being wrong doesn't make someone bad, right?
Um, what it is, is that he is wrong in a specific way, which leads him to a conclusion which is, uh, on its own grounds, immoral, which is basically to say that he should be trusted with an amount of power that no one entrusted him with, and that he can seize that power for himself unilaterally. Okay. Because he happens to control AI because he happens to be the CEO of anthropic. But he's not doing that.
But he is, because he's trying to maintain private control over AI while believing in this self recursive self improvement world. Like that's just not how I would play it if I were in his shoes. Now, of course, I don't believe any of this, so I would not I'm not asking Dario to hand over to the government. I'm asking Dario to just compete.
Just play the game so we can keep this ball moving. You know, keep the game rolling. But he is competing and he's inviting collaboration from his competitors and inviting the government. So what I'm asking is like of people listening to this, what I want is for you to basically not vote for politicians that would believe in supporting the types of policies that Dario wants.
Okay. I think we should not support Dario's policies are bad for most people. Uh, unless you believe in extinction risk. If you believe in extinction risk, okay, fine.
We can play a different game. But if you don't believe in extinction risk, how could not? How could you not believe in extinction risk if most AI experts believe in it? Well, okay, so that's like there have been plenty of historical examples of like people believing, like groups of people believing things that aren't true.
Right? No, no, but but, but I, I gave you an example of the wet lab and the the advancements of AI. So so let me just pitch this to you. Um, it will not be long before open source models and closed source models will be able to do that end to end and lab, uh, wet lab thing and the end to end cyber, um, zero days that that will not be long.
Do you agree with that? No, no, no, no, I, I don't I don't necessarily have a strong opinion of course end in zero days. Yeah. Sure.
That's not that's not that's already kind of there. Um okay. Do I believe that the extinction risk I mean zero day doesn't cause extinction risk by and large. Okay.
Can I, can I have a quick technical sort of breakout here? Um, the, the reason that mythos advanced, in my opinion, based on being, uh, somewhat of an AI expert compared to a true like AI researcher, I'm basically a four year old, but, um, My understanding of this, okay, is that mythos and all the good models, the way the good models are getting better is because they're better at pursuing goals long term. There is no difference between cyber and building a wet lab when you generalize.
This is all the same shit. And the the speed in which these are improving. Go ahead. But there's like other parts of this, like, of course, I am not a biology expert, but this is one of the problems is that like, neither are any of these AI experts either.
And so what happens is, is that like, and I can give a really great example of this. When I was at the University of Hong Kong, one of the professors there happens to be one of the leading experts in nuclear non-proliferation. Uh, he was cited in the I c, J blah, blah, blah. Okay.
And I remember I tried to get him involved in AI safety stuff because of course, it has a lot of parallels to nuclear non-proliferation. And he said that when he tried to discuss it with his AI, you know, people from the AI part of HKU, basically they said, you don't know what you're talking about on nuclear non-proliferation. So he said, okay, well, I'm now. Right.
And so what happens is, is that like the AI safety people can genuinely convince themselves, they have logic out all of the way in which these wet labs might be created and stuff. But like AI experts are not an expert in wet lab creation and like biological weapons and things of that nature. We haven't had some sort of discourse on this. And then we also don't know what happens as a result.
Like what if AI also produces some sort of like biology protection mechanism, right? Like we don't know. And so it's like, these are risks that I think are just like, very like, uh, they, they're kind of like, we don't really know that's true. And yet we're being asked to accept things now.
And I think so we have to kind of ramp up. But like, I just want to say it from this angle, if we want to believe that, like if Dario really believes this, there's also this piece of advice I have, which is to say, currently AI safety ism is losing the political, um, war, right? Like I think that the position that I'm expressing in one reason that some people like yourself might get frustrated is because like, my argumentation is actually quite popular and it's quite easy for like people to accept and latch on to, whereas to accept the safety as an argument, you have to like kind of go a lot deeper and believe a bunch of things.
If you believe the safety ism position, you need to switch the political framing of it to give people something positive other than just, we'll protect you from the risk that, you know, you don't believe in in the first place. I hear you. Right now, they're not doing that right now. They're basically saying you give up everything for your own safety so that we can control you.
No one's gonna accept that. It's just not going to work. So, like, their plan doesn't work. Yeah.
So so the thing is, Zach. Yes, your argument is very popular. Your argument is basically you see these guys like they're, oh, man, they're just making all this stuff and they're trying to control us. And, um, they've got all the best stuff and we should have all the best stuff too.
And Dario was like, guys, this is fucking bad. This is really fucking bad. We have to be very careful here. Let's get the government involved.
Let's get everyone involved. We need to regulate this. This is like I'm still going to do it. Like I still have a company and I still think I have a good path going forward.
But we need to be super careful about this. They were the first people to release their own dirty laundry. The guy has consistently been releasing all, all the sort of flaws in their shit. And describing this consistently put his company on the line multiple times, um, financially to, to stand up for, for ethical red lines and ultimately, like the stuff is going to happen and he is not trying to say is so this, I disagree with you.
He's trying to produce some negative thing in the future. What he's trying to do is saying this is bad, it's happening. Everyone's going there. It's going to happen no matter what.
Let's be careful and release slowly and make sure that the US wins before CCP. That's the argument. But also, when we say the US wins, like let's keep in mind this is not the, you know, unlike the CCP, which is like, okay, obviously can control any Chinese company. Like there were people were already suggesting like, what if anthropic just moved to Europe?
Right? It's not like, you know, also at another example, it's like when the Department of War said, we want to use your models in this way. Dario kind of refused. So it's not like it's controlled by the US government to the same degree.
So saying America wins is a bit unfair because like, it's not an American model. It's just a model made by an American company, which of course is like a fairly floating concept. I mean, you could, you know, but. Let's make an American consortium to try to do this.
So we win before China. Yes. But he's not giving. So this is the issue.
He's not giving anything. He's asking us to believe in all those risks, which I don't believe in. And he's asking us to believe in that and do things that will hurt us, like all of the people who are not him. Right.
And his group. Uh, he's not giving us anything. And so you sit there going, ah, I'd rather the Chinese model than believing that this risk, you know, this is how most. I mean, isn't this like, I've not watched this Star Wars thing, but isn't this the whole like the Chancellor took over to protect the people from some made up risk or some shit like this happens all the time.
Like Napoleon did it. Exactly right. That's a common story, Zach. The the emperor was an evil person.
Like just straight up pure evil. Like wanted to do evil things for evil people. You met Sam Altman, like, I mean, come on. Like, people talk about this all the time.
That's a different talk show. Sam Altman is not Dario. They are the opposite. I to some degree.
Yeah, sure. I concede that. Of course I believe that there's a difference there. Um, so so here's, here's my thing, Zach.
I think the situation is I believe Dario is a moral actor. I don't think it's perfect. Like I'm super pissed about my my prices. I wish I had more stuff in my subscription.
Like I've been pissed off at anthropic plenty of times. The issue is, you're not explaining to me how a moral actor which you agree that that Dario is, he's behaving as a moral actor. You think he's ultimately going to lead somehow to an immoral thing? I'm saying I can't see any difference between how I would behave and how Dario would behave, given my model of the world.
Yeah, but I would hope that if you were in Dario's position and actually sitting there, you would turn around and go, oh, God, and believe the things Dario believes. You would go, oh, God. Actually, probably it shouldn't be Daniel deciding it should be the government. Okay.
But importantly, which he referred to as communism. Uh, but importantly, I don't believe any of these things. And so therefore, I don't believe in a world where Americans are made worse off or not just Americans like the whole, you know, I live in Norway. I don't want my Norwegian family made worse off.
I don't like I don't want anyone made worse off to protect from a risk that I have not seen playing out or seen proven in any meaningful sense. Well, I do not think we should be willing to accept that trade off. And if that is the trade off that we're being asked to accept, how about if Dario believes this is the end of the world? He makes some concessions on his side to give us things back, as opposed to just saying we'll block the model that would do, you know, give you cheaper prices and things of that nature and not just prices, but also better capabilities.
We will block that one and you get nothing. I mean, he's got to come if he wants to save the world. That's a trade he's got to be willing to make, surely. But he's not doing it.
Dude, it's only been a few months. What are you talking about? We're talking. We're talking about.
Okay, potentially. Okay. This is we just disagree on this. I just think you're wrong about the actual threats to the world.
But I'm going to ask you to empathize and just accept it for the sake of argument, right? For the sake of argument, if we are correct, me and Dario and these other AI experts or whatever. If we are right about that, then he is behaving as a as a good actor. No, you should if you.
That's my point is that if he if I was sitting in his shoes and I believed that thing. What. Would you do? He.
I would basically say this needs to be nationalized because this is not something that should be privately owned. Right. But nationalization doesn't. Nationalization doesn't solve anything.
It doesn't solve anything. That's. But that's the point. That's not for us to decide.
That's that's that's my point is that that's the thing. It is up to us to decide. If not us in the sense of Dario. I mean, it's up for us in the sense of like American people who can vote and stuff like that.
But it's not us up to Dario to be making that call. And so. You have. To.
You have to look at the actual thing that would happen if he did that, if he walked off the. Field. That's not up to him. That's not that's my point is that at the end of the day, there's a public system that needs to be played.
He is making. That's my point is he is he is acting like somebody picked him to make that call and no one did. Okay. If you found out tomorrow that a company in Alabama was making nuclear weapons in their garage, right?
You would immediately say, go see, like, send the troops in. Cease that shit, Zach. All of us are are supposed to be making that call. Every single one of us starting with just happens to have a company.
No, but as a it's in a. Different than me. I would make the same decision. Anyone in that position should make the same decision.
No, you should make the decision to hand over to the public system. That's like my position. And just leave and not participate anymore. Participate?
Sure. But like it's not up to you to decide it then. So like to basically say this becomes a thing that is controlled entirely by the government. Okay.
You can still go to work every day, help make things safer, whatever you want to. Do. You think if he thought things were this bad, he should completely give it to the government and just hand over full control? The same we would accept.
But this is the same thing we would accept if someone was making nuclear weapons, which is what he compares it to. If you found out that a company in Alabama was making nuclear weapons in their garage, you would say, this is unacceptable and we should go take it. Of course, like you would believe that. I don't think anyone would think.
No, but but but tactically but tactically the way it would work. That's the problem. The tactics are not up for us. Right.
That's that's that's something that should be handed to a public. But that nationalization in that model, why wouldn't it be? Okay. Okay.
So let's say I'm in charge of the government now. I'm in charge of the nationalization program. My first step. Get all these AI companies in here.
All the US American AI companies. Get them all in here. This is now nationalized. We're using these resources of these companies to move forward.
Yeah. That's of course what you would. That's how nationalization would work of course. But it's.
Very. How different is that from what he is proposing? I'm arguing it's private. Ownership that he's going to IPO.
Like who cares about ownership? Who cares? Who cares about shares and ownership debate? That's the whole debate.
It's like, should you be able to build nuclear weapons and have private ownership? That's not the debate. The debate is whether or not the government is involved in controlling. I don't want the government, and I don't want the FDA for nuclear weapons.
Right. Like, it's like it's a silly. You're asking for complete nationalization. No I'm not.
I'm asking for competition. It's just if you believe the crazy things Dario believes, I would be asking for nationalization. I believe that people should be that all of these models need to compete so that we don't run into a world where all of. I think the biggest risk for AI is that it gets controlled by a few small companies, and that's why I want everyone to be competing openly.
I want us to make it so that distillation is like a right. You get to distill models, okay? Because I want there to be a ton of labs all competing in parallel. And that is because I don't believe in like the existential risk here.
Okay. I believe the biggest risk is Dario's company becoming king, right? That's horrible for society. It would be horrible to have $150 trillion company and everyone else be unemployed.
That's what I don't want. If that were he were trying to do, he would not have to. It's just that could happen. No, but I'm saying he's explicitly like doing the opposite.
No, I mean, his company's gone from whatever to one point whatever trillion. I mean. No, but he's the one that started glasswing. He's the one that did all of this.
But yeah, but glass, this was the other point I forgot to make, which is like Glasswing didn't even have any app set companies in it. Why didn't they give it to them is because they're going to try to compete with apps. So obviously that was like there were aspects of Glasswing that were not purely altruistic, right? Like, why would you have not given it to an application security company that has more access to code than they do?
I mean, just because they don't want to compete. They gave it to the people with the actual infrastructure that they were worried about. There are plenty, like there are plenty of application security companies that would have the ability to use mythos to make a large lot of code safer. Right.
You could have done that. They chose not to. Yeah. I mean, that's an implementation thing, right?
But I mean. But it's an implement. It's kind of the point here in the comments that Glasswing is like becomes when it's an implementation that doesn't reflect the full details of safety ism, you start to go like, it's not really that wasn't Glasswing was not really a safety thing. Okay.
Glasswing large. How could it be anything? Okay. Okay.
If someone who believes they're behaving morally comes up and says, hey, I don't want to release this immediately. I need to give it to people who have infrastructure so they can defend. Yes. Like, how is that not a moral action?
That part's normal. That's normal. Okay. And then what happens?
They go, I need to give it to these people. Um, can you come up with a list? And then the marketing department comes up with a list and they say, oh, we should probably not include the appsec people, because. Now you're back to assuming the worst of Sam or of Dario.
No, no, no, Dario is not even in the room for that discussion. Right. It's like that's what happens because it's consistent with safety ism. He's not released it to the world.
So Dario's not going to have a problem with this, right? But like, so that's the one thing to understand is that I argue that Dario genuinely believes in safety, blah, blah, blah, but it doesn't mean that every action Dario takes is only about safety. It just means it's consistent with safety. So Glasswing is consistent with safety because he didn't release it to the world.
Right. But it is not only about safety, okay? There was choices made in there that were not about safety, the choice to use fire. Well, he's also running.
He's also running a company. He's also trying to make money. Sure, sure, sure. Who else would be in the position to do this other than someone who has the ability to start a company and run one?
Say it to damn him. I just said it to say, you know, when we look at it and say yes, but he did the right thing with Glasswing like I think he did like a thing with Glasswing, right? He did like the 5050 of it. It's like, no, I'm not going to give him a huge amount of credit for Glasswing.
Right. I'm not going to sit there and go like, round of applause for giving it to CrowdStrike, but not to, you. Know, I'm just saying it's. Extreme.
Give it to you. Know, I'm just saying it's extremely consistent with the view that I've presented here. Of course. And I, I agree, and this is one of the things is that you can view almost everything anthropic does is consistent with safety ism, right?
Uh, and I think that, that, that's one reason why I have come to believe between that and Dario's, the fact that Dario does much like the Chinese do, writes everything he thinks and plans down in long winded essays that he thinks no one will read. And then you read them. It's like, you know, there are. When I prepared for the video, I mean, I probably read like 20,000 words written by the guy, right?
Uh, he's read all that stuff. He's prolific at writing these things. Okay. Um, yeah.
You know, and this is why I believe that Dario believes all this, but what Dario, there is a part of AI safety ism, which is that like, you know, if you go back to the origins of like, if you think like Nick Bostrom, like, um, superintelligence, which would be like 2015 book, right. Uh, which would have been basically around the time when yod would have been writing all this stuff. You know, basically one of the positions that's argued is like that AI is aligned, fully aligned with our goals.
But, uh, given that it's aligned, whoever controls it has like immense power, right? And this is kind of your concern of like, what if the CCP had it right? That'd be like really bad, right? If like, if like the Chinese Communist Party has the ability, has this recursive self-improving thing, and then you get there first.
And Dario believes in this world where like whoever, if, if we can control it, which I think he thinks is a little bit unclear, but as he assumes is like, let's just assume it's true because there's some probability where it's true. Um, you know, there was this whole wing of safety ism, which is like, but then what if the company who's in charge of it decides to do something evil with it? Right? Okay.
And, and like, that's a part of this, which is like, seriously, like, if you believe in this recursive self improvement plan where like suddenly whoever gets the best models will be permanently ahead. You have to be concerned about what the company who controls it would do with it, right? So here's my argument to you. So if you've read all his essays and stuff like that, what's clear to me from his essays and you can't always know the mind of someone, right?
My, in my view, what Dario would want the absolute most, okay, is we get to the other side of this risk situation. Either it burns down, we build it back up, or we mitigate it. Somehow. We get to the other side.
It. We have a we have ACI. We're now building up humanity like all the good things are happening or whatever. There's still companies.
He's still one of the companies. He's making tons of money. There's tons of open source models and they're doing great to everyone has tons of AI and he's still making money and the company is great and thriving. That is from his essays, the thing he's trying to build.
I have never once heard him say that he wants there to be open source models. If there's a quote on that, like send it to me because I haven't heard him say it. Yeah. I mean, there's no indication to me that he doesn't want open source models other than he doesn't want the most dangerous stuff to go into open source models immediately, because that would be bad for the reasons we talked about.
No, because he's talked to pretty extensively about how he believes one of the problems with open weight models is that you can remove any guardrails that might exist. So I guess like so from that perspective, it'd be hard for him to argue for open weight models because then you could just remove the guardrails. Right? Well, I mean, that's saying the same thing I'm saying, assuming that that we're something we could potentially solve.
He's not. In other words, he's not trying to limit the intelligence that people have access to other than for the reason of this risk thing. So he's not trying to get he's not trying to go to a place. He's not trying to go to a place where him and his rich buddies control intelligence and people have to go through him.
That's not what he's trying to do. No, I think he is. I think he's doing it for safety reasons, though, like he very much would love if I think if you told Dario tomorrow that like all AI goes through anthropic, right? Um, he would say that's great because it allows him to ensure that everyone is safe.
No, but what I'm saying. Is surely you agree he would say that that that would be good because it would allow him to keep everyone safe. Yeah, I don't know. I think he would probably like other people trying the same thing at the same time.
Yeah, I agree. They would probably. Yes. Yeah.
Yeah. Agreed. Agreed. So what what I would challenge you on is I think um, and maybe we could find quotes.
Maybe we could use AI to do this. Um, I think you're wrong about the, the final thing he's trying to get to, I think he wants there to be open models, lots of competition. He's not the only company. Everyone's safe, everyone's fine.
And everyone has access to tons of intelligence, one of which is him. Yeah, but I'm not gonna. I'm not going to let us play out this world where, like, we have to first hand over all this power to these guys, and then one day they will cede it back and make sure that everyone lives in harmony. Like, I think it's just the best course of action.
And I think, sorry, I probably have to wrap up, but like, and I'll let you say something after this. But the best course of action is fundamentally, everyone's competing for building the best AI models so that that way, like we don't end up in a world where it's controlled by one or 2 or 3 people that maybe we shouldn't have in control, right? Let them all compete, let everyone have access. And then like, if it turns out that there's some sort of like, if when we get closer to some of these really tangible risks, like, okay, like let's have a discussion maybe, but like right now, I'm not, I'm not willing to cave on any of this until I, until there's like some sort of reason to, why would we hand over to the, you know, Napoleon, just so that about some fear, we don't know if it's true.
We are extremely close. It is happening. Look, Dario just got surprised by this. Anthropic just got surprised by this.
This is happening a lot faster than I think. You think it's happening. Which would explain his behavior. So, um.
Yeah, I would just say, uh, great conversation. Uh, I'm glad we did it. And, uh, I think this really does illuminate the differences, which, like I said before, we're not disagreeing on what we're trying to achieve. I think it's just this, uh, this middle path that we're disagreeing on.
Yeah. No, I think this was good. Um, I think obviously, I think anyone listening probably now can form. I think there's very little left to ambiguity.
Like people can now decide which, which way they think. Well, that's a success. Yeah, exactly. So thanks for doing this.
Um, yeah, totally. I will then just stop the stream. I guess I will leave, you'll be on for one second. So okay, for everyone listening, see you guys.
Uh, I'll probably be on Twitter, so I'll probably post some stuff. So, uh. See y'all.
Other episodes covering the same guests and topics, from across The B2B Podcast Index.
Eric Ries on Why Good Companies Go BadPodcast Archives · on Anthropic92 / 100
The 18x Midas Lister Betting $3B on AI (and calling most of it fake) | Navin Chaddha, MayfieldThe Peel with Turner Novak · on Anthropic91 / 100
183: Why Trusted Data is the New AI Moat (w/ Rick Kranz @ AI Marketing Automation Lab)Move The Needle · on Anthropic91 / 100
Ep 92: xAI Co-Founder Unpacks the Future of Model DevelopmentUnsupervised Learning with Jacob Effron · on AI Safety87 / 100
Fighting Fire with Fire: How CyberProof Is Automating Cyber Defense with Edy AlmerCyber Sentries: AI Insight to Cloud Security · on Anthropic80 / 100
32 - When AI Attacks - Part 2Cyber Compliance & Beyond · on Anthropic78 / 100AI Predicts the Text of Answers52 / 100Most Companies Aren't Anywhere Near Ready for AI61 / 100We're All Building a Single Digital Assistant67 / 100Why AI Will Replace Knowledge Workers66 / 100Why I Believe in SOTA Models Over Custom Ones49 / 100