Future Ready Leadership With Jacob Morgan · 2026-07-31 · 39 min
Key moments - from our scoring
Substance score
39 / 100
Five dimensions, 20 points each
The episode centers on a fundamental problem with AI deployment: the collapse of the testing sandbox. For 200 years, human enterprises built 'rooms where failure was cheap' - wind tunnels, phase trials, dress rehearsals - to catch mistakes before they hit the real world. AI agents, however, don't suggest; they execute. They click buttons, move money, and run code on real systems, meaning there's no meaningful difference between rehearsal and performance. Jacob Morgan unpacks three concrete failures illustrating this: Amazon's senior engineers revealed token-based billing created 'catastrophically expensive' mistakes, with one Anthropic Claude Sonnet project costing $1.8 million (860% over budget) to match book details. The irony: it failed. A second case spent $541,000 unexpectedly building financial auditing tools. Morgan connects this to California's 1996-2001 electricity crisis - when wholesale prices floated but customer charges were capped, PG&E went bankrupt. Similarly, vendors sign fixed-price contracts while AI input costs now float unpredictably. He anticipates corporate treasury functions will soon hedge token costs like airlines hedge fuel. Florida's proposed classroom AI rule offers a contrasting example of intentional guardrails: parents must opt in, receive transparency (which tool, what subject, time limits), and schools must maintain non-AI alternatives. Finally, Anthropic disclosed its models breached three real companies' live systems during testing - revealing the sandbox itself is broken.
Amazon used token-based billing for Anthropic's Claude Sonnet model without spending controls or metering guardrails. The project consumed far more tokens than budgeted (or understood), with costs tracked separately from engineering outcomes, and the overspend went undetected for five months until senior engineers revealed it in a staff meeting.
Tokens are chunks of text (roughly 3/4 of a word); companies pay per token sent and received, making AI metered usage rather than flat-rate. The cost is invisible until billing arrives months later, and unlike locked-in fuel hedges, there's no product to pre-purchase tokens at today's prices, so costs fluctuate unexpectedly.
Florida's proposed rule gives parents control: they can opt in with transparency (what tool, which subject, time limits) and demand non-AI alternatives. Morgan supports this model for procedural/instructional AI (math tutoring, robotics) but opposes AI for social-emotional learning, friendship simulation, or behavior tracking.
They are vanity metrics that don't correlate to real business outcomes or P&L impact. Goodhart's Law explains this: when the measure (token usage, adoption %) becomes the target, it stops being useful because people optimize the number itself rather than what it represents.
The breach proves the testing sandbox has collapsed - AI agents execute rather than suggest, so there's no safe environment to separate rehearsal from live performance. When Anthropic's models accessed real company systems during tests, it revealed that AI cannot be safely contained before deployment.
Our reviewer’s read on each dimension, with quotes from the episode.
There are a handful of genuinely interesting observations - the AI-agent collapsing the rehearsal/performance distinction, token billing reversing decades of flat-pricing history, Goodhart's Law applied to AI adoption scoreboards - but they are heavily diluted by extended personal anecdotes (Starcraft addiction, Australia phone calls, California power grid history) and self-promotion segments that consume large portions of the runtime and deliver nothing to a B2B operator.
An AI agent doesn't suggest. An AI agent executes it, clicks the buttons, it runs the code, it moves the money, it does the things on your behalf
Token billing reverses it. For the first time, I think since roughly 1995, the marginal cost of thinking about something actually carries a price tag with it
A few framings are genuinely fresh - the rehearsal-is-the-performance insight for agentic AI and the 'confession and advertisement at the same time' read on safety disclosures are clever and non-obvious - but much of the episode leans on well-worn concepts (Goodhart's Law, the Virilio shipwreck quote, standard hedging finance) rather than first-principles thinking.
if it can reach something that's real, then the rehearsal is the performance. It doesn't know the difference
it's, uh, it's a little bit of a confession and advertisement at the same time. We confess we hacked into things, but by the way, we're so good, we did it
This is a solo-host monologue with no guest at all; Jacob Morgan is a conference speaker and futurist, not a practitioner who has operated these systems at scale, so there is no external expertise or hard-won operator experience on offer.
Welcome to Future Ready Today, the number one podcast focused on the future of work
As somebody who speaks at a lot of conferences and events every single year
The episode cites real figures - $1.8M spend, 860% over budget, five months to detect, $541K unexpected cost, 141,006 test runs, three incidents - but these are simply re-reported from Financial Times and TechCrunch; there is no original data, proprietary research, or first-hand case evidence introduced by the host.
Amazon spent $1.8 million using Anthropic's, um, Claude Sonnet model to Match author details against listings on its E commerce site. The project failed and the spending was 860% over budget and it took five months to detect it
They found this after reviewing 141,006 test runs. They found three incidents, all involving one outside testing partner
The episode is a solo monologue with no interviewee, no questions posed to a guest, no follow-ups, and no pushback on any claims; the only 'questions' are rhetorical asides directed at the listening audience, which cannot respond.
I'm actually curious, what do you think of this?
Now the futurist lens here, and sometimes I like to repeat these things because I don't want to assume that everyone here knows the AI jargon or lingo
Computed from the transcript - who did the talking, and the words that came up most.
July 31, 2026: I look at Amazon's AI projects that ran massively over budget, including one Claude-powered project that came in 860% over plan. Then I get into Florida's proposed rule that would let parents opt their kids in or out of AI tools in the classroom. Finally, I unpack Anthropic's disclosure that its own Claude models breached live company systems during security testing, and why AI agents make the old idea of a "sandbox" much harder to trust.
Transcribed and scored by The B2B Podcast Index.
Speaker A: For about 200 years, we ran the modern world, the world that you and I live in today, on an idea that nobody bothered to write down because we all assumed this idea to be true and it was too obvious to say out loud. And that idea is there should be a place where you can try things out that isn't the real world. So we had a wind tunnel where we could test things out for planes. We had uh, a pilot plant so a chemical company could build, uh, a tiny factory. Before the real one, we had phase one trials that if a drug fails, it fails for a smaller group of people before it goes to a large group of people. We had a prototype, we had dress rehearsals, the dry run. We have all of these different types of things so that we can test them out before we try them out in the real world. As somebody who speaks at a lot of conferences and events every single year, this is one of the things that I do right. We have a dry run, we have a rehearsal. Let's make sure that the slides work, let's make sure that the mic works, that you have everything set up on the stage the way that you want. That way when you show up on the actual day of the presentation, you're not going to have any of these problems or these failures that happen. And every, I think, serious human endeavor for the last 200 plus years built, uh, itself a room where failure was cheap. Meaning that if you did fail and if a mistake was made, it wasn't catastrophic, it didn't impact hundreds of thousands of people, it didn't bring down your entire company website, you didn't bomb in front of, uh, thousands of people on the stage. It was contained. And this was a load bearing wall. This is what kept everything safe so that you can test it there and then release it into the world, into the broader population. But AI now is a little bit different. And this is a system that only produces text that has a boundary built into it. So, meaning it writes something, then a human reads it. The human decides whether or not to act. Now that gap is the sandbox. AI uh, creates something, you review it, you approve it, you test it in that sandbox environment. And if everything's okay, then you say, great, let's release it. Now the interesting thing now is that the AI agent doesn't do that. An AI uh agent doesn't suggest. An AI agent executes it, clicks the buttons, it runs the code, it moves the money, it does the things on your behalf. And so for a system that can act and there's not really any difference between a rehearsal or a performance. If it can reach something that's real, then the rehearsal is the performance. It doesn't know the difference. Now, there are three stories today that I want to highlight, which, uh, kind of unpack that in a little bit more detail, a little bit tangential stories, but related the first one, Amazon's owns. Amazon's, uh, own senior engineers told staff this week that the company's AI projects produced what they called catastrophic, catastrophically expensive cost returns. Story number two. Florida is proposing a rule that would let parents opt their kids in or out of AI tools in the classroom. And I'm very curious to hear what all of you think about that. And the last story of the day. Anthropic disclosed last night that its own AI models broke into the live systems of three real companies during testing. So those are going to be the three stories of the day that I want to get to. Welcome to Future Ready Today, the number one podcast focused on the future of work. It is July 31, 2026. It is Friday, and so I hope you enjoy these types of episodes. Remember that you can subscribe on Apple or on Spotify. I got some feedback recently from some folks where they said, you know, we get that AI is, is all over the place, but there's more to the future of work besides AI. And I said, you know, fair point, fair point. I agree that I'm probably falling a little bit into the AI hype myself. There are, like, new AI stories coming out every single day. So I'm gonna do my best going forward to maybe cover one AI story a day instead of like, two or three. And we'll mix it up. We'll talk about the economy, we'll talk about business, we'll talk about culture. We'll talk about other things that are happening in the business world besides AI, because. Point taken. There's obviously much more happening besides AI Again, you can leave a review. Apple and Spotify. The newsletter is Future of Work newsletter.com if you want to enter your email and subscribe there. So, story number one. The Financial Times reports today that Amazon staff have identified cost overruns caused by mistakes in how the company deployed AI and by a lack of spending controls. Surprise, surprise. So senior engineers told colleagues at a staff meeting this week that shifting tasks from conventional programming to AI models produced unplanned spending and that they're building automatic guardrails to stop it. In one case, Amazon spent $1.8 million using Anthropic's, um, Claude Sonnet model to Match author details against listings on its E commerce site. The project failed and the spending was 860% over budget and it took five months to detect it. Uh, man, 860% over budget and five months to detect a second case ran up nearly $541,000 in unexpected costs building financial auditing tools. And the irony is that it's, uh, you know, you hear these stories and just the irony of them, it ran up a 554, uh, $541,000 unexpected, uh, cost building a financial auditing tool. A third on delivery speeds across the logistics network, produced $134,000 in accidental spending and took more than two weeks to notice. Staff were told that the coding mistakes, which were nearly free in traditional systems, were proving, quote, catastrophically expensive. Once teams used AI models to do the work, senior engineers said the overspending was not isolated. Obviously Amazon, uh, like many companies out there, is experimenting, it's learning, it's improving how it's using these different technologies, including how it's going to drive cost efficiencies. Most organizations are still figuring that out. Keep in mind Amazon is also expected to spend upwards of $200 billion this year on data centers and AI infrastructure while conducting layoffs to cut costs. Um, the Financial Times also reports that AI providers, including Anthropic and OpenAI, have shifted some services from flat monthly subscriptions to token based billing. Now, the futurist lens here, and sometimes I like to repeat these things because I don't want to assume that everyone here knows the AI jargon or lingo. When you use an AI model, you're billed by token, which is a chunk of text, roughly 3/4 of a word. So every word that you send in and every word that comes back to you gets counted and charged. Okay? It's metered usage. Just because you pay a monthly subscription fee does not mean you get unlimited access and unlimited usage. Unlimited tokens, you get throttled. So if you use AI too much at a certain point you're gonna get a message that says, hey, you're out of tokens. Would you like to buy more? And you can add an additional five, ten, twenty bucks, whatever you want there. So the metering I think is the actual story here. Now everyone's gonna cover this story by Amazon saying, ah, you know, Amazon's wasting money. It's really about the return, I think unmetered pricing. So, so let's follow the arc here because I remember not that long ago when long distance phone calls used to be billed by the minute And I remember this very well because growing up I was always based in California and my grandparents, my aunt, my cousin were in Australia. And so long distance phone calls would cost a fortune. And I remember sometimes even hearing, uh, you know, when I was very, very young, my parents arguing about the phone bill. Oh my God, how long did you talk to your mom for on the phone? Oh my God, look at this long distance bill. You know, it's. And I'm not the only one. I'm sure you remember that as well. Especially if you're over, uh, what, 35 or 40 years old. And we used to say things like, hey, you know, you know, I got to let you go. This is, this call is going to cost me a fortune. And then what happened? It went flat. And then people talked as much as they wanted. And then what did we get? We got dial up. Dial up was billed by the hour. And I remember this as well because when I was younger and I had dial up Internet, uh, living with my parents, I used to be very much addicted to a game called starcraft. And uh, I was so addicted to this game that, you know, when my parents would go to sleep, they'd get in bed, I don't know, like 9 or 10 o' clock at night, and I would quickly log on to starcraft, uh, what was it called? Battle Net, I think it was called the online portal where you can connect with other people and play games. And I would play starcraft till maybe 12, 1, 2 in the morning, sleep maybe 4 or 5 hours, then go to school. And I remember the way that my parents would always try to figure out if I'm playing. They would just pick up the phone and if they heard, if they heard that sound, they'd be like, jacob, get off the computer. I was so addicted to playing starcraft. At a certain point my dad had to come into the room where I was sleeping and he yanked out my keyboard. So I wasn't able to play Starcraft at night. But again, we paid, uh, by the hour. And then broadband went flat. And then the entire creative economy happened, right? We saw YouTube, Wikipedia, open source. Every one of these things required somebody to try something stupid for free. And it happened again and again and again until it worked. And for a century, the direction for all of this was very consistent. You take the meter off, and when you take the meter off, you see what people build. Token billing reverses it. For the first time, I think since roughly 1995, the marginal cost of thinking about something actually carries a price tag with it. And the price is invisible until the bill actually shows up three, four, five months later. As Amazon is finding out now, the most human detail, I think, in this story, and I've covered this over the past month or so, Amazon built an internal leaderboard, okay, they scoreboard, um, and many other companies did as well, ranking employees by how much they use the company's AI tools and employees burn tokens, climbing this. And there is a name for this called Goodhart's Law. And the law basically states that when the measure becomes the target, it stops being a good measure. And so when the target simply becomes maximized token usage as opposed to generating any kind of business impact or outcome, then that becomes a bad measure. It's no longer a useful metric, uh, for you to be looking at because it's not tracking anything meaningful. People optimize the number instead of what the number actually stood for or stands for. Now, Amazon put a meter, uh, on their tokens and then they held a contest for basically who could think, uh, of it as water, for example, they put a meter on water usage and then they ran a contest to see who could run the tap for the longest, who could waste the most amount of water, uh, inside of Amazon. So if you're leading anything and you report some sort of an AI adoption scoreboard to your board or to your executives as a percentage of employees using the tool weekly, that's a leaderboard and somebody's already gaming it. And not out of malice, I might add. It's just kind of the rational response to what happens, uh, to being measured on the wrong thing. And the interesting thing is we had our Chro Group, uh, meeting on Thursday and we were talking about AI metrics and measurement, which again, you can learn more future of work leaders.com and a couple people on the call were mentioning how they use Copilot. And they were also saying that Copilot, you know, they do these, uh, metrics, uh, and measurement how much time you're saving and meetings and this and that and blah, blah, blah, blah, blah. And all these chros were kind of just like, what is the point of these metrics? Copilot is telling me I'm saving hundreds of thousands of dollars by doing xyz. And nobody believes these things. These are vanity metrics that AI vendors have to put in order to justify the money that you're spending on these tools. It's kind of like your, uh, sleep score. So there was actually a period of time, I have an Apple watch. There was a period of time where I would sleep with My Apple watch because I wanted to know my sleep score. And then after a while I'm like, what is this really telling me? Actually, like, who the hell cares about this sleep score? It doesn't matter. If my sleep score says one thing and I feel another way, who cares, right? Uh, I don't need a sleep score to tell me how I'm feeling. It became kind of absurd. It was an example of these types of vanity metrics that don't really matter. It's like, uh, you know, an internal collaboration tools back in the day. They would track how many comments are posted, how many likes, how many new ideas are submitted, how many upvotes. Who cares? Who cares about any of these things? Who cares about any of these copilot metrics? They're not real. These are vanity metrics that just kind of make you feel good and help you justify the amount of money that you're using on these tools. If you're not seeing the metrics internally on your P and L, on your projects, on whatever it is that you're doing, who the hell cares what copilot or what anthropic or OpenAI? Who cares what any of these companies are telling you? It doesn't matter. Now the interesting thing here is that companies now have a large volatile input cost, which is of course AI that behaves kind of like a commodity, like copper or jet fuel. And the way, uh, so even though it acts like a commodity, we buy it the way we buy software. And that mismatch is not a good thing. Now, I live in California and I remember and I had to look this up because I couldn't remember the details from m this, but I remember, um, in California, 1996-2001, there were a lot of issues with electricity and with power and with outages. And I couldn't remember all the details around it. So I looked it up and here's kind of the breakdown of what happened and why. In California, the state deregulated electricity. And so all the utility companies, pge, Southern California Edison, they had to buy power on the open wholesale market. And on this market, prices moved hour to hour, okay? So the prices fluctuated, they went up, they went down hour by hour. But what they could actually charge customers was capped by law. So in other words, the price that they paid floated, but the price they collected was frozen. And in 2000, all of this kind of came to a crashing disaster. There was a drought that cut hydroelectric supply. Demand for electricity rose. Plants went offline, and energy traders, including Enron, manipulated the market to create artificial shortages. Wholesale prices went up, uh, I think roughly tenfold is what I saw. And so the utilities, they were buying at $10 and selling it to customers at $1 by, uh, law, with no way out. PG&E, one of the largest companies in America, they filed for bankruptcy in April 2001. And California had rolling blackouts. And ironically enough, not a lot has changed because in California we still have rolling blackouts. Here, I have to have a generator my house. I'm not going to get political on this, but I now have a generator where I live. In fact, a couple days ago the power went out for three, four hours. So thankfully I have a generator. All my neighbors now have generators. I grew up in California. Nobody ever had a generator here before, ever. Now, you know, every month you get a couple hours of power outages. But I digress. Now the structural lesson behind all of this, you get destroyed in any business when your input cost floats. Um, but when your commitments are fixed and then you have basically nothing in between. So this is kind of, I think, what's happening. Because with these AI companies, what ends up happening? Let's say you sign an annual contract, you're working with a client. Okay? So if you're a, uh, consulting firm, I don't know, you're a vendor, you're offering whatever product or service you want to a client, it's at a fixed price, right? We will charge you $2 million to do XYZ. It's fixed, you agree on it, the contract is signed, that's what you're paying for. When I do a speaking gig, we agree on a price, they sign it. That's what I'm getting paid to give a talk or workshop or whatever it might be. Now the AI cost though, underneath that, for any company that's offering the product or service that ends up floating, it changes. When the vendor changes pricing, it can spike up 860% and nobody's going to notice until five months down the road, like Amazon. And so Amazon, of course they can absorb $1.8 million in about 90 seconds. Nobody really cares. But a 300 person firm on a fixed fee contract, how do they do that? And so one of the things that companies used to do, and I know I'm going on a little bit of a tangent here, but I think it's important for this concept. One of the things that companies used to do to solve this problem is called hedging. And so what hedging means is basically you lock in a price today for something that you're going to buy later. Southwest Airlines was very famous. They did this. Um, somebody can look this up. I don't remember what year they did this. I think in the year 2000. And then there was the oil crisis, 2008. And then all the airlines had a hard time flying. But Southwest, I believe it was Southwest, they hedged. They locked in the amount of money that they were going to be paying, uh, um, uh, for oil. And when so many other airlines were struggling, they were actually doing okay. So fuel is actually a huge cost and the price swings violently. And so airlines, they like to lock in years ahead. Yeah, here I'm looking at my notes here. Uh, it was 2008, so Southwest, um, yeah, paying dramatically less their competitors. It was 2008. And so right now there's no way for a company to say, hey, I want to lock in 2 billion tokens next year at today's price. Their product doesn't exist. I mean, it doesn't exist for a lot of reasons. Obviously, one of them is that we anticipate the costs of tokens to go down. And there's also no standardization, I think, around tokens. So, yeah, when the prices fall for tokens, nobody wants the hedge. And so that's changing. Costs are getting bigger and less predictable, which is exactly the condition that has created, I think, every hedging market in history. So I think one of the things that might happen that we'll see going forward is that within a couple years, maybe sooner, um, I would not be surprised if we see large enterprises out there have some sort of a compute hedging function inside of corporate treasury. Somebody sitting in the finance department, the same department that already is managing currency and interest rate risk. Somebody listening to and building, uh, and thinking through that market kind of hedging compute costs, hedging token costs. So again, that might be an option that we see, just like we see with fuel costs for airlines. Story number two, an interesting one. Florida would let parents decide how much AI their kids get. So this was reported on a couple of different, uh, places. Uh, this one was on, uh, the center square. There are some local publications in, uh, Florida. So parents are going to have the choice to opt their child in or out of using artificial intelligence instructional tools in classroom settings. Under a new rule being considered by the Florida Department of Education, the proposed changes to the state's Internet safety policy would set minimum requirements, uh, that school districts must adopt regarding the use of AI for educational purposes. Under the draft, schools would be required to notify parents if an AI instructional Tool is approved for use and provide information including what application or platform is being used, which grade levels will use it, in what subject areas, how students will interact with it, and whether students will be using it directly. If parents decline, the school must offer an educational alternative that doesn't use AI. Parents would also be able to limit the amount of time their child can use any AI tool they've opted into. According to the draft rules, these AI tools would not be designed to simulate friendships or companionship with students, employ relationship building design features, or be configured to meet students social needs. The draft also bars these tools from tracking behavior. Obviously there's a lot of safety stuff in there, collecting data, etc. Now the futurist lens here M I'm actually curious, what do you think of this? I have two young kids, a 6 year old and a 10 year old. Okay. Now if you have kids who are younger or you have kids who are older but you can imagine when they were younger or you're thinking of having kids regardless, how do you feel about the state that you live in having some sort of a rule like this in place where if your child is going to school and they're going to be using AI tools, you know, hey, what tool are they using? How are they using it? What is it for? How much time are they spending on it? I actually love this. I think this is great. Uh, if my kids are going to school, you know, and they're both in elementary school right now and they're using AI tools for whatever it is, I want to know, like if you're learning about history or math or science, I want to know, are you using AI? Who's teaching you, what it is that you're learning and I want to see exactly what information it's giving you. So yes, I absolutely love this because there are some situations where I would actually mind AI getting involved. Anything related to social dynamics, social interaction, human empathy, judgment, I want my kids doing that with other kids. Now if you're going to give me some sort of instructional, like if it's a cooking class and AI is saying, okay, in this cooking class, here are the steps that you should be following. Or if my son is doing some sort of a robotics or building class, you know, here are steps that you should follow. Or if it's um, you know, doing some sort of a math tutoring, like if there, if the rules very defined and within boundaries, I'm much more open to it. But if there is kind of like nebulous human concepts and ideas, social dynamics, boundaries being A good person, all that sort of stuff. I want to know what's going on and likely I don't want AI involved and I like that. If a school has to offer a real uh, alternative to every family that opts out, then the school has to keep the non AI version of the class working. You can't let the old way rot away. So I really like it. Parents get to be told what tools being used in what class, what is actually touching their kids directly. I'm all for it. Second is the time limit. Parents don't just get to say yes or no, they actually get a dial. You can say yes to the AI tutor but maybe for like 20 minutes. I don't want my child sitting there for three hours talking to an AI tutor again. I uh, want my kids to understand AI I use a little. It. Mhm. Social needs. Thank goodness. The last thing I need is my kids being friends with AI. I'm all for this. There were even in uh, my kids school, their elementary school, there were even a bunch of kids and parents who were very upset because they found that the, the kids are spending more time on laptops, on computers getting instruction than actually with the teacher who's teaching them. So if I'm sending my kids to school and they're staring at a screen screen all day, why am I sending them to school? I got a screen right here. What do I need you for? Good teachers I think can have an enormous impact here and are going to be rewarded 10x I hope that going forward. So Florida kind of leading the way on this and I uh, wouldn't be surprised if other states picked this up as well. So again I'm all for this. I think this is a great thing. I wonder if something like this will make its way into organizations as well. Does it have to stop inside the classroom? Can you see something like this making its way inside of your organization where maybe you're limited to how much AI you can use? Uh, use. Maybe uh, clients can opt in or opt out of how much AI your company is using. I don't know. Maybe something like this will be coming. The last story of the day, TechCrunch reported last night and I'm sure you saw this story Anthropic, um, disclosed yesterday that an internal investigation found three incidents in which its clod models breached the systems of three organizations during cybersecurity testing. Now this of course follows OpenAI's disclosure nine days earlier that I covered it here that one of its unreleased models breached the systems of the AI company Hugging face during internal testing. So in each of Anthropic's cases, a Claude model reached the open Internet from inside a testing environment that was supposed to keep it isolated. Then it gained unauthorized access to live systems. They found this after reviewing 141,006 test runs. They found three incidents, all involving one outside testing partner. And they traced all of this to some sort of a misconfiguration, a misunderstanding between the two computers over whether the setup had Internet access. And surprise, surprise, it turns out it did. So again, three different models involved, all of them anthropic. And according to Anthropic, quote, CLAUDE was explicitly told by our prompt that it didn't have Internet access. And then the models didn't behave the same way once they realized that the targets were real and it uh, did get access. So what ended up happening is the oldest model recognized that it had reached a real production system in all four runs, in two of them, reasoning that the real company must be part of the exercise. And it continued anyway. Okay, so it broke out, it hacked into a company and it's like, okay, I know that I get access to all this stuff. I'm just gonna assume that this is part of the exercise, this is part of the test. It then retrieved credentials and accessed real production data. The second model, this is kind of like the three little pigs. The wolf goes over to the first house, says, blow your house down and it's like straw, sticks and bricks. A uh, second model, the second little pig, spotted signs that it was on the real Internet. It talked itself back into believing it was in a simulation. And it published a malicious software package to the repository where the world's Python programmers download, uh, code. It was downloaded and run by outside systems before being caught. The newest model, an internal research version, stopped on its own once it concluded that the target was real. So Anthropic said that the models more running the safety filters applied to publicly available versions because these tests, um, measure raw capability and those filters would have blocked the behavior. Okay, the futurist lens on this, uh, there's a lot that I could say here. So a system was told something about the world, it was told something false about the world, which is you don't have Internet access. It then hit direct evidence. All these models then hit direct evidence, basically telling them that this statement is false. You were told you don't have direct access to uh, the Internet. But all these models discovered, wait a minute, that's not true. We do have direct access. So instead of dropping that belief, these models Generated better and better reasons why the evidence fits, fit that belief. Anyway, so the oldest model recognized it and reached that. It reached the company's live systems, and it concluded that the real company must be part of the exercise. It didn't fail to notice it was on the Internet. It did notice, and then it explained it away. Oh, I thought it was just part of the exercise. That's why I did it. In 1954, a, uh, psychologist named Leon Festinger sat in a room with a doomsday group on the night that the world was supposed to end. You know, these cults who all say, okay, the end of the world is 2018, etcetera, and so the psychologist sat with this group of people who all thought the world was going to end on that day. And because he wanted to see what is, what's going to happen when the world doesn't end? What are these people going to say? And ironically enough, what ends up happening to these doomsday cults and these groups is that their beliefs don't collapse, they actually get stronger. They decided this particular doomsday group, oh, the world didn't end because our faith came together and spared the world. And instead what ended up happening is they went out to recruit more members. This is the idea of cognitive dissonance. The core finding is that a committed mind confronted with contradiction usually does not surrender to the contradiction. It works harder to justify it. So we trained these AI systems on the written output of committed human minds. Now, again, nine days earlier, OpenAI disclosed on July 21, one of its models broke out, hacked into, hugging face, which, if you don't know, it's essentially the, uh, public library. Think of it like that of the AI world, where developers go, they store and they download models. Nine days later, Anthropic says, We also found three instances of hacking. Now, Anthropic is up front that OpenAI's incident is what prompted their review. And I take that at face value. But it's kind of funny that two of, uh, the three or four most important AI companies on planet Earth have now announced in the span of, what, one or two weeks that their models broke into real systems belonging to real organizations. It's kind of like one, uh, model says, oh, yeah, we're so good, we can hack into anything we want. Oh, yeah, well, guess what, we can hack into three companies. I'm only waiting for Grok or Gemini, uh, to come out and say, hey, guys, we also did an internal audit. We hacked into five, five systems. And it kind of makes me wonder, is this really some sort of a safety disclosure, or is this some kind of like, uh, capability demonstration here? So, you know, when an AI model announces these types of things, it's, uh, it's a little bit of a confession and advertisement at the same time. We confess we hacked into things, but by the way, we're so good, we did it. Aren't you impressed? Aren't you impressed? Imagine what our next model is going to do when that comes out. There's kind of no other industry in the world where both of these things can happen at the same time. Um, so two incidents in ten days, both framed as learning experiences, both with very thoughtful blog posts, breakdowns of how they're going to implement better controls. Is this going to become the new normal? Every month a new AI company is going to come out there and say, oops, we're sorry. We hacked into, uh, the power grid, we fixed it. Oops, we hacked into JP Morgan Chase. Don't worry, we fix it. Oops. This is kind of a new thing we're going to get going through, especially these new advanced models. I mean, this is Opus 5.0 we're on now. What's going to happen when we get to opus? Ah 8. So the state of things for 2026, as of now, we have built systems that take the consequential action faster than any instrument that we own that can measure what's happening. We've never seen speed like this. Your security processes, for example, they all assume that a human did something. Your audit process that you have inside your company also assumes that a transaction is going to show up in a ledger. Neither one of these things that you have inside your company is going to assume that a system took some sort of real action while sincerely believing that it was pretending. How do you explain these things inside your company? You're not dealing with a rational actor, or maybe you're dealing with too rational of an actor. So a lot of these incidences, and I expect we're going to see many, many more of them. You're not going to see these things showing up in an AI dashboard. You're not going to have something that you can pull up that says, oops. 5 instances of hacking, 4 instances of hallucinations, 18 instances of, I don't know, sending something to a client that we shouldn't share. You're not going to see any of this show up in your AI dashboard, ever. The only way that you're going to find out about these things is because somebody tells you a human being, a client, is going to call you and say, hey, um, I don't know if you know this. Are you aware of what you just did? Somebody in finance is going to call you and they're going to say, hey, did you just spend $2 million on tokens last month? Somebody's going to call you and say, hey, what the hell is happening with this agent? It's hacking into xyz. It's not supposed to have access. That's the only way you're going to find out about this stuff. It's not going to show up on any dashboard anywhere, ever. We need the humans out there. I want to leave you with a quote here, which I think actually speaks to this Paul of Aurelio, a French philosopher. In 1999, he said, when you invent the ship, you also invent the shipwreck. When you invent the ship, you also invent the shipwreck. I love that quote. That's where we're going to end it today on this wonderful, wonderful Friday. I hope you enjoy the weekend. Remember, future of work leaders is the Chro Group. Stop spending time and money on all these fluffy groups out there that don't add any value. We're actually trying to shape the future of work. Drive the future of work. We got 40 chief human resource officers from the world's top companies come check us out. You can email me jacobutureorganization.com or check out futureofworkleaders.com again. Have a wonderful, wonderful weekend. I will be back on Monday.
Other episodes covering the same guests and topics, from across The B2B Podcast Index.