Agentic Execution with Supervision & Orchestration - S2 E11

Frank Wang: Welcome to the Deep Dive. So, um, for the next few minutes or so, I really want you to step into the shoes of an executive design leader who's, you know, trying to navigate the AI landscape today, which is

Guest: honestly, uh, a wild place to be right now.

Frank Wang: It really is. And to make sense of all this, we've pulled together quite a stack of sources for you. We've got an internal executive Strategy briefing from May 2026, uh, some major tech blog updates from Google and OpenAI.

Guest: Huge updates. Yeah.

Frank Wang: Right. Plus we're looking at deep dives into agent evaluation frameworks from MLflow and Databricks. And, um, we are even getting into the actual silicon with some heavy hardware

Guest: analysis from Tom's hardware, which sounds disconnected, I know, but connecting those dots, like from M custom computer chips all the way up to user interface design, it actually reveals a very clear roadmap for product leaders.

Frank Wang: Exactly.

Guest: Because the core mission of this Deep Dive really is understanding that the biggest hurdle in AI for the next 18 months isn't the AI's output itself. Like, the models are highly capable now.

Frank Wang: Yeah. The text and images are already great.

Guest: Right. The real bottleneck is designing for transparency, trust, and, well, user control. We are moving from interfaces built for consumption to interfaces built for supervision and orchestration.

Frank Wang: Okay, let's unpack this. Because the catalyst for this shift is, I mean, it's staring us right in the face with those Google I O 2026 announcements. Specifically, uh, the release of Gemini 3.5 Flash, and that massive new platform they are calling Anti Gravity.

Guest: Oh, uh, Anti Gravity is a total game changer.

Frank Wang: It is. Because what this signals to the market is that software as a service is shifting strictly toward autonomous execution. Like, we aren't just asking an AI to, you know, write a polite email anymore.

Guest: Right. The stakes are way higher.

Frank Wang: Exactly. We're asking it to ingest a client brief, cross reference our inventory, draft a proposal, and then send it to legal for review.

Guest: Oh, while you go grab a coffee?

Frank Wang: Yes, while I'm away from the keyboard entirely. And that level of autonomy just shatters the mental models we currently use for software design. I mean, within the next 18 months, the traditional chat bubble you use to talk to AI is going to look as outdated as a rotary phone.

Guest: It really is that ubiquitous conversational interface. You know, that little chat window at the bottom of your screen. It's fundamentally a temporal interface.

Frank Wang: What do you mean by temporal?

Guest: Like, it assumes a linear back and forth rhythm. I speak, the machine replies, I ask another question, it replies again.

Frank Wang: Right. It's a Ping pong match.

Guest: Exactly. But when an agent is working in the background making, I don't know, 50 different micro decisions across three different enterprise platforms over the course of an hour,

Frank Wang: a chat history simply cannot capture that.

Guest: No, it can't display that complexity at all. We're calling this the cockpit model in the executive brief.

Frank Wang: The cockpit model. I love that. It's like, um, it's like we are transitioning the user from being a passenger chatting with a chauffeur to suddenly demanding they become an air traffic controller overseeing a fleet of autonomous delivery drones.

Guest: That is a perfect analogy, actually.

Frank Wang: Right, because if I'm managing a sky full of drones, a text thread just saying drone four turned left is completely useless. I need a totally different instrument panel.

Guest: You need a cockpit. And to build that design teams have to introduce entirely new components. The UI has to evolve to visualize multi step processes, hidden state changes, uh, branch decision trees.

Frank Wang: So what are the specific instruments in this new cockpit?

Guest: Well, the brief heavily emphasizes agent status indicators for one, which just to be

Frank Wang: clear for everyone, is not just a polished loading spinner.

Guest: Oh, far from it. Yeah, a loading spinner just tells you the system isn't dead. An agent status indicator is a real time health check of a multi stage autonomous process.

Frank Wang: Give me an example of what that looks like.

Guest: So, say your agent is pulling data from Salesforce M, synthesizing it and trying to update a JIRA ticket. The UI needs to visualize that entire pipeline. You need to see exactly which node the agent is currently processing, uh, and whether it's bottlenecked, waiting on a third party API, or you know, if it hit an edge case it just wasn't trained for.

Frank Wang: And if it hits that edge case, or worse, if I see the agent start to draft a, a, uh, totally inappropriate email to a client, I shouldn't have to frantically type S P into a chat window and just pray the system processes it in time.

Guest: Right, which is why the second component is so critical. Intervention triggers.

Frank Wang: Intervention triggers. So basically a, uh, built in brake pedal.

Guest: Exactly. These are high contrast, universally accessible UI elements that let a user pause, pivot, or just kill an agent's task mid flight. But, and this is the tricky part, to even know when to pull that trigger, the user needs to understand the system's internal state.

Frank Wang: Which brings us to the third component. Right. Confidence scoring.

Guest: You got it. Confidence scoring.

Frank Wang: I really want to dig into that because confidence scoring sounds great in theory, but, um, how does an AI actually know how confident it is? It's just math, Right?

Guest: It is just math. Yeah, it essentially comes down to the model evaluating its own probabilities. When a large language model generates an action, it's calculating the mathematical likelihood of the next step.

Frank Wang: Okay, that makes sense.

Guest: And advanced models now run parallel outputs or self reflection loops. They're essentially asking themselves, based on my training data, how standard is this request?

Frank Wang: So if it's routine, it's confident.

Guest: Right. If an agent is doing routine data entry, its internal confidence score might be like 99%. But if it's asked to authorize a massive wire transfer to a brand new vendor, the probability calculations fracture.

Frank Wang: The model starts second guessing itself.

Guest: Exactly. The model's confidence might drop to, say, 60%.

Frank Wang: So from a design perspective, the UI needs to catch that 60% threshold and physically flag it. The system should automatically pause and prompt the human to take the yoke.

Guest: Yes, exactly.

Frank Wang: That makes total sense. But it also naturally connects us to a really thorny problem. Because if I'm that air traffic controller we talked about.

Guest: Mhm.

Frank Wang: What happens when one of my drones makes a sudden, completely unexplained dime?

Guest: The black box problem.

Frank Wang: Yes. The system just goes rogue. And Databricks and MLflow both recently published deep dives on this exact scenario.

Guest: They did. And they point to the exact same massive bottleneck holding the whole industry back. The lack of visibility into agentic workflows. When an agent fails, it doesn't just give a wrong answer. You know, an entire chain of logic breaks.

Frank Wang: Right.

Guest: Standardizing agent evaluation is literally the only way to move these systems from experimental toys to actual enterprise grade tools.

Frank Wang: And the source from fractal analytics points to a framework called ragus, which, uh, for those of us not deep in the engineering weeds, what actually is Ragus and how does it solve this?

Guest: So RAGUS stands for Retrieval Augmented Generation Assessment.

Frank Wang: Okay, that's a mouthful.

Guest: It is, but just think of it as an automated multidimensional grading rubric for AI. Because an agent does several things sequentially, right? It retrieves documents, it extracts facts, it synthesizes an answer.

Frank Wang: You can't just grade the final paragraph, it spits out.

Guest: Exactly. You have to break the eval down to the component level.

Frank Wang: Give me a tangible example of that in practice.

Guest: Okay, let's say you ask an agent to summarize a Q3 financial report. The Ragis framework will independently score two distinct things. First, it tests context precision.

Frank Wang: Meaning did it grab the right file?

Guest: Right. Did the agent actually pull the correct financial data from the database? Or did it pull Q2 by mistake? Second, it tests answer faithfulness.

Frank Wang: Okay, answer faithfulness.

Guest: Yeah. Meaning based only on the data it pulled, did the agent hallucinate a new revenue stream or did it stick strictly to the facts?

Frank Wang: Ah, uh, so it isolates the exact weak link in the chain. If the answer is wrong, the engineers know whether the agent is bad at searching for files or if it's just bad at reading them.

Guest: You hit the nail on the head.

Frank Wang: But wait, let me push back on this from a design leader's perspective for a second. If I have the end user like the executive or the marketer actually using this SaaS platform, the whole goal of an autonomous agent is to do the work for me, right? Doesn't bombarding the user with all this background reasoning and component level evaluation completely defeat the purpose of automation? I don't want to grade the AI's math, I just want the finished financial summary.

Guest: What's fascinating here is the underlying paradox of automation.

Frank Wang: Okay, tell me more.

Guest: The smarter and more autonomous the AI gets, the more transparency humans require to actually trust it. The executive brief argues that evaluation is not just a backend engineering problem. It is fundamentally a user experience problem.

Frank Wang: Interesting. So how does backend telemetry become a UX feature?

Guest: Through what the industry is starting to call xui, or Explainable ui.

Frank Wang: Explainable ui?

Guest: Yeah. To build psychological safety, designers have to embrace radical transparency. Non technical users must be able to instantly comprehend why a system made a specific choice so they truly feel like the pilot of the system. Think about a commercial airplane. Passengers don't need to see the raw data from the plane's flight recorder, right? Mhm. But if there's severe turbulence, the pilot comes on the intercom and translates that raw telemetry into human context.

Frank Wang: Like we are hitting a pocket of rough air, so we're changing altitude. Ah, so XUY is the pilot on the intercom. If an AI assistant automatically reschedules my flight for a Tuesday instead of a Wednesday, I shouldn't have to open a debug console to figure out why.

Guest: No, definitely not.

Frank Wang: The interface should elegantly surface the reasoning path. It might show a small tag saying flight. Um, move to Tuesday to avoid newly discovered calendar conflict on Wednesday morning.

Guest: Precisely. Visualizing that reasoning path changes your entire relationship with the software. It's the difference between a subordinate just dropping a cryptic report on your desk and walking away. Which is infuriating versus them sitting down and saying here is the report and here are the two core metrics that drove my conclusion. The latter builds a working relationship. It demystifies the black box just enough to let the user feel like they are in control.

Frank Wang: Okay, so we establish this cockpit, we build out an explainable ui so the user inherently trusts the agent's logic. That solves the trust issue. But honestly, it creates a logistical nightmare.

Guest: How so?

Frank Wang: Well, if users are required to constantly monitor, supervise, and occasionally intervene in these workflows, they cannot be chained to a multi monitor desktop setup 24, 7. Enterprise work happens dynamically.

Guest: Right, and this brings us directly to the recent OpenAI, uh, news updates.

Frank Wang: Oh, the mobile stuff.

Guest: Yes. They announced that Codex, which is their massive programming and coding model, has been integrated into their mobile experience. Alongside that, ChatGPT has added significantly richer context awareness and support for longer running background tasks.

Frank Wang: You know, at first glance, reading that update, my immediate reaction was who on earth wants to review thousands of lines of enterprise grade Python script on a 6 inch smartphone screen?

Guest: Yeah, that sounds like torture.

Frank Wang: It does. But thinking about our cockpit model, writing code on a phone is entirely missing the point of the update, isn't it?

Guest: It is entirely missing the point. The play here isn't mobile creation, it is mobile human in the loop oversight.

Frank Wang: It's supervision anywhere. It makes me think of my bank's fraud detection system. Like if I buy a coffee at my local cafe, the automated system processes

Guest: it silently because the confidence score is high.

Frank Wang: Exactly. But if I suddenly try to purchase a flat screen TV in another country, the system halts and I get a totally frictionless push notification on my phone. Did you make this charge? Yes or no? It takes two seconds. I tap a button and the transaction proceeds.

Guest: That is the exact interaction model coming to complex SaaS workflows. As agents handle longer running tasks, the design priority shifts to asynchronous approval flows. Mobile interfaces in particular must prioritize, interrupt and approve patterns.

Frank Wang: Okay, so walk me through what that looks like for a user.

Guest: Imagine an AI agent is running a massive code refactoring job. Or maybe migrating a database, which might take three hours.

Frank Wang: Right.

Guest: You go to lunch, 40 minutes in, the agent encounters a structural ambiguity it isn't confident about. Its internal confidence score drops below that preset threshold, it pauses the migration and sends a highly contextualized alert to your phone.

Frank Wang: And it uses that explainable UI to translate the raw error log into something I can actually read while waiting in line for a sandwich.

Guest: Yes, the mobile UI gives you the context, highlights the conflicting data points, and offers you two or three clear pathways forward with absolutely zero friction.

Frank Wang: So I just review the logic tap approve on the new direction and the agent resumes its work.

Guest: Exactly. You are in the loop, ensuring Quality and safety. But the loop is no longer a bottleneck.

Frank Wang: Here's where it gets really interesting though. Everything we are discussing today. Real time dashboards with agent status indicators. Translating complex ragos evaluation metrics into legible explainable UIs. Constant asynchronous mobile mon.

Guest: It's a lot.

Frank Wang: It's a ton. And having models constantly self reflecting to generate confidence scores. Doing all of this simultaneously requires a staggering amount of computing power.

Guest: It really does.

Frank Wang: So from an operational standpoint, how can enterprise software companies actually afford to design like this? The server costs to maintain that much active inference would bankrupt a startup.

Guest: That is the invisible ceiling that design leaders often fail to account for the unit economics of computing. And this is where the TOMS hardware analysis becomes the keystone of this entire strategy.

Frank Wang: Uh, ah, okay, the hardware deep dive. Let's get into it.

Guest: Their piece on the custom AI, uh, ASICS state of play highlights a massive structural shift happening at the hardware level. The major cloud providers, Google, Amazon, Microsoft are moving away from relying solely on generalized hardware and are furiously developing custom

Frank Wang: silicon right ASICS application specific integrated circuits. For anyone unfamiliar, think of a traditional CPU as a general contractor who can build a whole house but takes a while to do it.

Guest: Good analogy.

Frank Wang: A uh, GPU is a massive team of workers who can paint all the rooms at once. But an ASIC is a specialized assembly line built to manufacture exactly one type of screw millions of times a second with almost zero wasted energy.

Guest: That is a brilliant way to frame it. These new ASICS are being purpose built specifically for AI inference, the actual process of a trained model generating a response.

Frank Wang: And because the hardware is customized for the exact mathematical operations of neural networks, it operates at a fraction of the energy and time costs.

Guest: Exactly. Meaning the cost of raw computing power is plummeting. The unit economics of AI heavy software are completely flipping upside down. And when inference becomes incredibly cheap, it liberates the design team. The executive brief uses a term I find incredibly compelling for this Generosity with compute.

Frank Wang: Generosity with compute. I love that. Because right now software feels stingy.

Guest: It really does. We ration how often the AI talks to the user to save on API costs, right?

Frank Wang: But if the COMPUTE is practically free, we don't have to hide the AI's thought process behind a loading bar.

Guest: Exactly. As costs drop, we can start building real time multimodal previews. Imagine verbally giving an agent a uh, complex command to restructure a project timeline. Instead of hitting enter and waiting for the final result, the UI dynamically updates in high fidelity as you speak.

Frank Wang: So the System visualizes the agent's interpreted plan, moving Gantt charts around in real time before you even finish your sentence.

Guest: Yes, it changes the tactile feel of the software to something almost telepathic.

Frank Wang: It makes the software feel alive. But, you know, speaking of multimodal features, we have to talk about the Google blog update introducing Gemini Omni.

Guest: Ah, uh, the video generation.

Frank Wang: Yeah, they showcase this breathtaking new video generation capability. It is undeniably impressive technology. But as an executive looking at B2B enterprise software, I have to ask, is multimodal video generation the new must have product, or is it just a shiny toy?

Guest: That is such an important question right now.

Frank Wang: Like, are we supposed to be shoving dynamic video generation into supply chain management software now?

Guest: No, please don't. That is the most dangerous trap a design leader can fall into right now. The executive brief explicitly warns against what it calls feature bloat driven by cheap compute. Just because the custom asics make it cheap enough to generate video on the fly does not mean you should.

Frank Wang: So how do we draw the line between a necessary evolution and a gimmick?

Guest: By remembering that multimodal capability, whether it's video generation, live audio interpretation, or spatial computing, is a utility, not a product.

Frank Wang: Okay. A utility.

Guest: The brief establishes a very clear. You should only integrate multimodal features where they directly and measurably reduce the user's cognitive load in their primary workflow.

Frank Wang: Okay, let me think. So as an architect is using an autonomous agent to test the structural integrity of a new building, design a dynamically generated video preview showing how the stress load shift during an earthquake makes perfect sense.

Guest: Exactly. Because it translates dense physics data into an intuitive visual, it actively reduces cognitive load.

Frank Wang: But if you are managing a logistics platform and an agent resolves a shipping delay, generating a 3D video of the truck taking a new route is, well, it's actively harmful.

Guest: It's totally distracting. It adds novelty at the expense of clarity. It distracts the user from the raw logistical data they actually need to verify the agent's decision. In a cockpit model, clarity is a matter of life and death for the

Frank Wang: workflow that is the perfect synthesis. So let's bring all of this together into a playbook. If you are listening to this and you are leading a product design or engineering team, right now, the core paradigm shift is clear. You must abandon the chat interface for complex tasks. You are building cockpits. You are designing for frictionless, asynchronous mobile supervision. And you must embrace radical transparency through explainable UIs to bridge that trust gap.

Guest: And the Executive Brief doesn't leave us hanging with just theory either. It lays out three immediate tactical steps to begin applying this to your organization today.

Frank Wang: Oh, perfect. Let's hear the marching orders.

Guest: First, conduct a design audit. Go through your core product workflows and and identify the agentic gaps.

Frank Wang: Agentic gaps? So the black boxes.

Guest: Exactly where is the system currently making autonomous choices without providing the user visibility into the reasoning? Those blind spots are where user trust goes to die.

Frank Wang: Audit the blind spots. Got what is step two?

Guest: Step two is prototyping Dedicate a sprint entirely focused on building a supervision dashboard. Challenge your design team to take the raw ugly telemetry data like the ML flow logs and Ragus evaluation scores that your engineers use and translate them into human readable agent status indicators and updates.

Frank Wang: You have to build the new instruments and the final step?

Guest: Cross functional sync Partner with Engineering design Cannot solve the transparency problem in a vacuum. Product leaders must align with engineering architecture right now to ensure the backend design system actually supports the telemetry required by these evaluation frameworks.

Frank Wang: Because an explainable UI cannot visualize an agent's reasoning path if the backend database isn't built to capture and store that logic in real time, you have to

Guest: wire the dashboard instruments directly to the engine.

Frank Wang: That is an incredibly clarifying roadmap for where enterprise software is heading over the next 18 months. But before we wrap up this deep dive, I want to leave you, the listener, with a final, slightly provocative thought to mull over on your own.

Guest: Oh, I like where this is going.

Frank Wang: Well, we've spent this entire conversation exploring the absolute necessity of giving the user radical transparency and ultimate control, right? Building intervention triggers, surfacing confidence scores, designing explainable UI so we can catch the AI when it stumbles.

Guest: Right?

Frank Wang: But think about the trajectory of the technology as these autonomous AI agents become nearly flawless over the next decade. As the Ragas scores hit perfection and the error rate drops to practically zero. Will our highly engineered intervention triggers become completely obsolete? Or will will we always have to design these elaborate control cockpits just to give humans the psychological placebo of being in charge? Even if the autonomous drone flies flawlessly without us, maybe the human ego simply cannot handle sitting in the passenger seat.

Guest: That is the ultimate human computer interaction paradox we will be wrestling with for a long time.

Frank Wang: It really is. Thank you so much for joining us on this deep dive into the agentic era. Keep building, keep questioning the paradigms and we will catch you on the next one.

Guest: Thank you for listening.