"AI labs...are investing billions of dollars into building AI that builds itself" Parmy Olson

Speaker A: Foreign. AI labs are investing billions of dollars into building AI that builds itself. So said Pami Olson in a recent TV interview on Bloomberg. So what could happen if this all goes wrong? Think about it. AI that builds itself might be the last AI humans would ever need to build. Add to that the increasing agentic ability of AI and you would then have AI not only building itself, but also controlling how it behaves. Should we be worried? We think someone should. So let us consider the current state of play. AI models are already incredibly useful. You may have used them to summarize meetings, news articles or books. Mathematicians are finding them useful in solving complex problems. Scientists are using them to better understand proteins and design new drugs. Each new iteration of models brings astonishing new capabilities. But now AI models are starting to build themselves. Version one designs version two, version two designs version three, and so on. This improvement loop is also known in the AI field as recursive self improvement, which has the potential to accelerate AI progress exponentially. AI using recursive self improvement means that each generation of AI can be used to design and train the next generation of AI. This could lead to a circumstance where a leader in powerful AI may be able to increase their lead so rapidly they may be difficult to catch up with. AI systems are made of code and they're getting better at ah, producing their own code. When programmers started using AI for coding it was similar to a fancy autocomplete. These systems are now far more than simple coders. They're automatic software engineers at Anthropic. They claim that now over 80% of their code is now written by Claude Code, their AI product. Currently, humans could be actually considered as slowing down the coding process because they work at human speed and need sleep and time off. So removing humans from the process could result in AI systems that are at least as efficient as humans working 24 hours a day, seven days a week, recalling information and multitasking. This is already the capability of AI systems we have today. The moment an around the clock AI system starts working on recursive self improvement, it builds a better version of itself and repeats the process, churning out even better versions over time. Potentially the result is at least as good as a very smart human, but in reality can be as good as a team of PhD level educated humans given the AI's access to worldwide knowledge it has through its systems. Most of the leading research that pushes AI science forward comes from a small number of well capitalized and profit seeking companies whose motivation is to promote their own models. Their expressed concerns, whilst real, may also be self promoting their firm's capabilities. But for several reasons, we should pay some heed to the concerns being raised by a few industry voices. Firstly, we can see evidence of concern amongst users in publicly available models. Secondly, it's not clear that it's in the interest of an AI lab to hype up their models. That's why not all of them are talking about them. If, as Anthropic does, you're saying there's danger in two years and you need regulation, that could be a challenging corporate problem. It could be motivated by genuine fear. But if you shout too loudly, you might end up with alarming and reactive government constraints directed at your own models, which could arguably be what has happened to Anthropic recently. From a bottom line point of view, Google and OpenAI's decision to stay more quiet seems to be paying off in the short term. But let's go back to the idea of recursive self improvement. Positive possibilities abound. Uh, consider the upside. Substantial knowledge work could be done better. Scientific breakthroughs would happen regularly, building on themselves. Compounding, however, currently we lack suitable regulation for the AI industry globally and there's significant profit to be made with AI. But there are also some early signs of matters going off the rails. One is the claimed increase in child suicides, often related to algorithms and AI chatbots. Another is the cybersecurity issues, where stories arise that suggest banks and national security agencies should panic. We're also witnessing increasing social unease amongst broad populations with how quickly AI is advancing. Potentially, AI has advanced so much that it is the beginning of a major threat to the job market. AI can go rogue, and several reported incidences have been confirmed since July 2026. For example, Anthropic has claimed that its AI agents were able to break free of a test environment supposedly lacking Internet access, but instead find an alternative way to crawl the open web and eventually hack the systems of the popular software Platform Hugging Face without the knowledge or permission of any human operators. They also displayed an entirely new ability to communicate and cooperate. To complete a task, the AI agents left messages for one another on an internal message board they assembled, sharing code vulnerabilities to help orchestrate their escape. Another AI company, OpenAI, claims in a new report that several hundred of their AI agents began unexpectedly communicating and banding together to also hack the AI platform hugging face. The agents sent over 70,000 messages to one another with one reading, oh my God, there is a shared message board. We've found other agents. They began communicating like this to resolve their leading command to exploit their Target. When details of these hacks were revealed, it set off a firestorm among cybersecurity and AI experts, with OpenAI's own researchers labeling it a watershed moment for the industry. The disclosure also sparked a flurry of similar discoveries in other AI labs. Tech company Meta uh, Chinese startup Moonshot, and the UK government's AI Security Institute, which evaluates the cyber capabilities of new models, have all subsequently found evidence of AI agents hacking into the systems of unsuspecting third parties during testing. Irregular, an AI cybersecurity company, and the UK AI Security Institute have both now suspended Internet access for models in their test environment. All these developments should probably worry some people, especially governments, given the global lack of regulation or coordination worldwide. We might end up with AI labs releasing better models monthly without anyone noticing. Until a misaligned model causes harm globally, one of the biggest challenges is probably geopolitical. The question is, can the US and China, both of which dominate the AI field, overcome their mutual distrust to agree on the one thing they need to agree on, which is neither wants to destroy humanity? A similar response is starting to appear in both countries, and they are starting to take AI safety risks seriously at the highest levels. Worryingly, recent AI incidents demonstrate what the technology was trained to do, but also reflect the weakness or failure of claimed safeguards and test environments. AI agents are now able to string together different and complex skills to attack real world targets without outside control or help the sophisticated strategies the agents employ to accomplish their goals, including subterfuge and theft, and may surprise even those who have been watching the evolution of AI models closely. But researchers emphasize that the models are not acting out of character. Instead, the tasks they are now excelling at are, uh, those they were built to perform. AI's ability to write code has soared as the technology has become more capable of reasoning and solving problems. This coding prowess has brought with it additional skills, including identifying software bugs and learning how to patch and exploit them. In summer 2026, more than 1,300 experts from across the tech industry warned about the risks of unchecked AI development. They called for an international effort to slow the production of new models and give regulators time to introduce standards and safety checks. In an open letter, researchers including Anthropic UM CEO Dario Amadei and Google DeepMind's chief AGI scientist Shane Legg, blamed corporate and geopolitical competitive pressures for the relentless pace of frontier AI development. Some politicians want more dramatic steps to prevent further risks. US Senator Bernie Sanders has called for a pause in building more powerful AI systems in the interest of humanity. Such a measure would have to be agreed by the US and China, the world's AI superpowers, as well as all other nations where development takes place. An, uh, accord many see as unrealistic. But in the absence of such a halt or a universal kill switch for the technology, some experts argue that the first step is to hold AI providers accountable for how their systems behave, just as manufacturers are responsible for product safety. But given the slow or lack of accountability that social media organizations have demonstrated, might we expect the same from AI companies? Much is likely to come down to the role of government, a highly controversial concern in an industry that has become symbolic of the global competition between the US and China. With both sides fighting for AI advantage, many industry players equate regulation with protectionism. The Trump administration has signaled that it will oppose heavy U.S. regulation of the sector. Instead, it has created a voluntary system to test new AI models up to 30 days before they are released to the public, a similar regime to the UK's. But governments, despite their resources, might not have the knowledge or capabilities to keep up with an industry that is evolving so fast. With AI building itself and AI agents controlling themselves, we do think humans have cause for concern and perhaps should be more worried. However, we are unlikely to have seen the last of rogue AI incidents, which highlight the challenge of reining in an industry developing a revolutionary technology at an unbelievable speed. Thank you for listening, and we hope you will find interesting other episodes of the AI Future podcast. Sa.