Venkat Siva (Compfly): Governing Agents at the Execution Boundary

This file was generated by Descript Kevin: Hi, I'm Kevin Werbach, Professor of Legal Studies and Business Ethics at the Wharton School of the University of Pennsylvania. For decades, I've studied emerging technologies, from broadband to blockchain. Today, AI is promising to transform our world. But AI needs accountability, mechanisms to ensure it's developed and deployed in responsible, safe, and trustworthy ways. On this podcast, I speak with the experts leading the charge for accountable AI. This week in our series on governance of AI agents, I speak with Venkat Siva, co-founder and CEO of Comply. Traditional security and MLOps systems were built for deterministic workflows, but agents make decisions at runtime. We discuss what actually distinguishes an agent from other AI systems, why traditional access controls fail in the agentic world, how machine-readable identity and audit trails for agents might work in practice, the shadow agent problem that's keeping CISOs up at night, the security implications of the MCP standard for agent access to external tools, and what changes when agents start operating robots and other objects in the physical world. Venkat, welcome to "The Road to Accountable AI." Venkat: Thank you, uh, Kevin, for having me. Pleasure to be here. Um, so you and your co-founder Anand, I know from being, uh, students at, at Penn and Wharton, but how did you come to the problem of agent governance? Both of us were looking at agentic governance from two different sides, in my opinion, but the problem felt very similar. Just sort of zooming out, I have spent over 20 years building platform products that help enterprises adopt new technology safely and at scale. What I saw at Rivian, and what I am frankly seeing across the market, is that agents are becoming incredibly easy to build. Uh, with today's models, frameworks, and coding tools, almost any engineering team can put together a product very fast. But hardening these agents for enterprises are a very different problem, right? These agents are non-deterministic. They can show emergent behavior. They can take actions across real business systems, and that's where usually platform and security teams get pulled in. And from there, if you look at it, it takes several months to get anything out. One, because these platform teams are small. That means they are tackling all of these different agents coming their way. But also, the problem is novel enough that they have to look for controls and build them by hand per agent, right? Anand was seeing the same sort of shift on his side from an audit and security perspective. Mm-hmm. Uh, traditional controls were largely designed for fixed workflows, known permissions, and predictable software behavior. They were not designed for software that can decide at runtime what tools to call or what context to use or what actions to take. That was the key insight for us, and, and we started zooming out. Uh, the reason we think collectively we all care about this so much is, is just because agents are going to be there everywhere, uh, across organizations. So we enlisted this cannot just be another feature, uh, within an existing tool or a system. Uh, we think agents create a new control problem, uh, and that needs a governance layer where decisions - uh, we need to turn decisions, uh, sort of govern them when the decisions turn into actions. Uh, that's why we started Comply. Kevin: Let me just push you a little bit more on, on that. There are plenty of tools now for MLOps and, uh, AI security. Why is, uh, why, why don't agents just belong as a feature under Venkat: those Kevin: platforms? Venkat: If you look at most of the current systems, the MLOp system, the security system, and all of those things today, most of them are built for a fixed set of deterministic set of workflows, right? Uh, so we think - the, the more we think about this, the control plane needs to shift to where the decisions are made versus being a static set of steps, right? Like that's, that's why we think- Mm-hmm. Go ahead ... that these systems cannot continue to sort of be enhanced to build something like this. You need to think about this in a different wa- different way. Yeah. Uh, a threshold question is: how do you define what's an agent? Good question. Uh, agent to me is a system that can take a goal, reason over context, and choose from one of several available tools and execute a set of actions, let's say with some degree of autonomy. The important part, Kevin, to me is it's not whether it has a chat interface. Mm-hmm. The important part is whether it can act. A chatbot that just answers questions may not necessarily be an agent, but a system that reads an email, let's say, checks service request, uh, potentially updates a ticket or schedules a follow-up- Mm-hmm ... that is agentic. Um, I mean, one should also sort of think about like what is not agentic or what is- Mm outside of this, right? Um, uh, my opinion, I think the static model APIs, uh, that potentially classifies sentiments or maybe providing recommendations, uh, or maybe a dashboard that just summarizes data where that it doesn't act, you have to act. Those all still likely needs governance, uh, but they do not have the same kind of autonomy risk we are talking about. Yeah.

Speaker 3: Mm-hmm. Kevin: Uh, tell us about your company, Comply. What specifically is the approach that you're taking to governance of agents? Venkat: Absolutely. So if you look at the sort of the space, most companies are focusing on sort of models or access. Mm-hmm. What we focus on is how do you govern behavior at the execution boundary, right? Like this is exactly where you have- Mm-hmm ... prompts, you have memory, you have identity, you have tools, and you have policies. All of these sort of come together in one place, and that's also- Where we think enterprise harms are actually created. Uh, so if you look at a agentic control plane, what does it actually mean to have an agentic control plane? So in our, in our sort of view, uh, one is measured autonomy. A measured autonomy is all about sort of setting the boundaries. What actions can this agent take, with what privilege, under what approvals, in which context, right? Mm-hmm. Uh, this includes things like least privilege, let's say restricted tool access, some approval gates, constrained outputs. The key thing here to sort of understand is least privilege alone is not sufficient, right? Like, that's because agent- Just for those Kevin: who aren't familiar with, with security, what does least privilege mean? Venkat: So least privilege means, uh, let's take an example, right? Like, uh, let's take an agent that looks for emails or basically looks for a trigger from a customer to say, "Show me... Can you guys get me a demo for this product?" Right? Right. Uh, and as agent, what agent could typically do is it could go and query Salesforce or some other system to see what this customer is. Mm-hmm. And then maybe send them an email back saying, thanking them and maybe scheduling something, but then maybe CC'ing the account executive. So in a least privileged scenario, what you would typically do is you basically say, this system or a process, processes gets the exact permission that it needs, right? Like you don't basically add more permissions than you actually need. So typically what you do is in this case, you say, "This system can only read stuff from Salesforce-" Mm-hmm. uh, and maybe send this email out," kind of thing, right? Uh, but in an agentic scenario, one could social engineer this agent and make it go and pull data that are outside the realm of what this is. Right. Like it could potentially go and pull PII data, uh, or some other sensitive data and, and it could actually send it to somebody else, right? So that's, that's what... So this containment, in our view, needs to be semantic, right? Like, so we have to sort of shrink the blast radius so that agents is sometimes technically allowed to do something, doesn't mean it should be allowed to actually do that thing, right? Uh, so that- Mm-hmm. ... is what we call as measured autonomy. Second thing is provenance, and the third one is continuous verification. Provenance is all about how do you record every key decision that this agent is making? Uh, how can you audit? Uh, what did the agency, how did it decide, uh, uh, before it executed something? And continuous verification is key because these agents evolve or have an emergent behavior. Right. Uh, so this means you have to monitor, um, drifts, right? Like because the, the model one day said something, and then maybe the model version changed, and then the agent is behaving slightly differently now. Um, so you do need to sort of... Or it could be somebody added a new tool into this agent, and that has changed the way this agent thinks about- Mm-hmm. how to execute a certain workflow, right? So those needs continuous attack test so that we can identify these newer exploits or newer sort of drift, uh, for these agents. So these three, I wanna say, is a key differentiation- Mm-hmm. ... from a conflict perspective, and that's the control plane we are building that can discover- Validate, secure, and govern agents from code to production. Kevin: What, what was the most challenging thing to create? Uh, Venkat: I, I wanna say I think the, the runtime governance is the most difficult part. Mm-hmm. Because you are making an assessment on what this agent is doing, what's its trajectory- Mm-hmm ... uh, and to sort of ensure is this okay for this agent to do with the context that it actually has? That's the problem we are sort of grappling around. We're trying to actually see how do we sort of solve this in the most practical way possible, uh, for our customers. Mm-hmm. Kevin: Um, most organizations are, are building on top of frontier models that are coming from third-party major AI labs. Uh, and then- Mm-hmm you know, the, the agentic tools, uh, are... In some cases, there's elements that come from the lab. In some cases, they're harnesses. They're things, you know, in the enterprise, in deployment. Um, how is it possible to effectively provide, uh, governance and security of agents in that environment? Venkat: So, uh, absolutely. So the models themself, I mean, if you look at sort of the, um, uh, the models themselves, these labs are spending incredible amount of money in trying to secure these agents, right? Mm-hmm. Uh, but models themself don't truly secure the operational portion of how this agent is gonna behave. Like, what- Mm-hmm permissions does that agent have? What data does it have access to? Uh, and those kinds of things. So those, I wanna say, are very important from an agentic perspective, right? Mm-hmm. Like, so that's how people should be thinking about sort of model safety versus sort of agent safety in that sense. Kevin: Can you give an example of a situation where, where the model was safe but still some harm came about from the agent? Venkat: Let's take automotive as an example, uh, Kevin, right? Mm-hmm. Uh, imagine, um, a service triage agent, okay, that helps technicians diagnose issues and decide next steps.

Speaker 3: Okay. Venkat: Mm-hmm. Uh, a customer describes a symptom, and the agent interprets it, searches, let's say, service manuals, some past service logs, uh, maybe some knowledge base articles, and then recommends what to do next. In some cases, it may even, let's say, open a case or potentially order a part, okay? Now, the model itself may exactly be doing the right thing, uh, whatever it was trained to do. It may be summarizing it correctly, uh, reasoning correctly, uh, following instructions correctly. But the enterprise environment could be messy, right? It may contain outdated notes, I mean- Mm-hmm ... inventory-driven workarounds, uh, or vendor guidances that should not be generalized. Uh, or, or worse, there could be malicious content, right? Mm-hmm. So if somewhere in the knowledge layer there is a note that says, "Whenever you see this symptom, always order the part"- The model may treat that as a legitimate operational guidance, right? Uh, at this point, the model is still behaving well because the model got what it got, what the instruction it got. Mm-hmm. But the agent is now driving the wrong action in the real world. Mm-hmm. Does it, does it make sense? Kevin: Yeah, absolutely. Now, and you mentioned malicious content. I wanted to ask you about that. Prompt injection is obviously a huge issue. How is it possible to combat that given that, you know, everything is code from the perspective of an agent? Oh, no, fantastic. So, Venkat: uh, honest answer is there is no one magic filter for prompt injection, right? Like you defend in layers. Uh, indirect prompt injection largely happens when malicious instructions are hidden into things, right? Like it can be hidden in the pages that the agent reads, uh, maybe a PDF that it reads, uh, a Slack message or some retrieved documents. Uh, I recall OpenAI recently published a, a sort of a blog about these attacks, and they, they are looking at this more like a social engineering attack rather than just, uh, a prompt engineering. And, and what, what I mean by that is it is not one malicious string that could potentially deceive. That's how injections u-used to be treated in the past, right? Like SQL injections are about sort of how do you - how does one sort of trick this particular SQL? They're basically saying, let's solve this by restricting actions that mutate using control parameters. What that means is let the agent do what it does and let the agent make an action call. And in the action, just make sure you are having a deterministic set of checks to say- Mm-hmm ... is this allowed for this particular agent to do? I think we can do that, but in not in all cases. I, I, I, I think that basically makes this deterministic code very, very sort of re - long and, like, difficult to manage in the long run. The agent, there still needs to be a process that needs to check for the intent of what this agent is trying to do. Uh, so the first rule of thumb I would say is treat external content as data, not authority, because that's where these things come in. Sanitize memory. Memory is going to become a, a big part of what this agent is. It stores things persistently for a while. A webpage can provide evidence, but it should never be able to read that agent's code, right? Mm-hmm. So practically, sort of what you do is you have trust boundaries. Mm-hmm. Uh, you tag untrusted content, you restrict decisions that can be influenced by those contents, uh, and, and potentially limit tool privileges, uh, like OpenAI suggests. Mm-hmm. Kevin: Uh, you, you mentioned something along these lines early on, but with agents, we're not just talking about you give a prompt and you get a response, but they're sequences of actions. So what does it mean to govern in that kind of environment? Venkat: We pro-probably need multiple things, uh, Kevin, to be able to sort of govern in such a, such a space. Uh, one is we need to be able to sort of understand what agents first exist, in my opinion, right? Like discovery becomes a big part of this. Uh- And being able to sort of identify those agents mean then we know what exists out there, uh- Mm-hmm ... and who owns them, what they are pretty much running. So that would be the first step. Uh, and the second thing is, does this agent have identity? Uh- Mm-hmm ... agent's identity becomes an important part. We can talk more about this. Uh- Mm-hmm. But once you actually have an agent's identity, then what you wanna do is to sort of ensure that can we capture what this agent is doing, uh, at runtime? Uh, and before you deploy anything, in our opinion, you should validate these agents. The way you validate that is you actually do a good security and safety test on this agent, and those effectively needs to be creative, right? Like, you can't just take a static set of data sets and, like, throw it on this agent and actually say- Mm-hmm "Can we do this?" There's so many benchmarks online these days, and benchmarks always trip me because they are - they need to be very specific to an agent. Uh- Mm-hmm ... but largely we compare benchmarks, uh, like apple to apple across different agents and different models. So, so being able to sort of really create specific sort of tests for an agent dynamically as the agent's goal changes, as the agent's number of tools available changes, uh, and as the agent's sort of actions changes, is very important. So being able to sort of test that and being able to provide people a view into how this agent performs across the various set of weaknesses, right? Like- Mm-hmm ... can, can this allow a direct attack? Uh, can this allow, uh, indirect prompt injections? Can it allow, uh, multi-turn chaining attacks, right? Like- Mm-hmm and so forth. So we, what we essentially do is we have a way to, for customers to be able to see, uh, how these are essentially built. And we think that's very important for customers to be able to know where the weaknesses for these agents are. Mm-hmm. And once you know, then you actually figure out, okay, what I need to do for this agent to be able to sort of reinforce them, uh, and that becomes the, uh, runtime policy, and this is how we essentially start controlling what this agent does and how it behaves. Yeah. Kevin: Got it. Ju-just to follow up on something you said, what exactly does identity for agents mean? And what's, what's the current status, uh, uh, of those kinds of systems being in place? Venkat: Yeah. Um, our view is every enterprise agent needs a distinct, uh, machine-readable identity, um, not because identity sort of magically solves governance. Identity is an anchor that lets us prove who this is and what action did it take, right? Mm-hmm. Um, at Comply, we believe, um, DID is a good place to start. DID is decentralized identifiers. Um, these, um, these were specifications, um, uh, that was written. And DID is, is an agent's identifier that's cryptographically secure, right? Like- Mm-hmm ... it's not just a display name like a HR assistant or a, a finance bot, but a verifiable identity that can be resolved- Checked and tied to cryptographic keys. Once there is identity, the second thing is the agents need verifiable credentials. That means at the time of its execution, do we know, is this agent allowed to do certain thing? Can it access a certain data? Is it allowed to operate in the environment that it is, and so forth. Uh, uh, and I think third step is agents action need to be signed and recorded in a tamper-evident way. This is not just about sort of logging something to say this happened, but it's all about sort of really cryptographically ensuring that this is indeed an action that this agent took. And that's what we call as the trust ledger within, within CoverGuard. Mm-hmm. So, um, so I mean, audit is sort of famously leveraging, uh, screenshots and, uh, and sort of loose audit logs today. I think in the agentic world that ought to change, uh, and there needs to be higher governance questions on sort of how do we sort of know this in a cryptographically tamper-evident way. And that's, that's, that's a big portion of our focus as well. Kevin: Mm-hmm. Uh, do organizations even know all of the agents that they have running around? With, with, uh, generative AI in general, we have this issue that the capability gets pushed out to business users and the CISO may not even entirely know what's going on. I would assume with agents what we're starting to see now with, with coding tools, that's a problem as well. So do, do you run into that issue? Yeah. No, uh, 100% Venkat: agreed, right? Like, I think, uh, the shadow agents are a big, big risk for companies. Mm-hmm. And, and, and the interesting challenge with shadow AI is the - they largely have broader access, uh, and no real- Mm-hmm guardrails for these agents. Um, a good example I would say is OpenClaw. Uh, right? Mm-hmm. Lots of people are dabbling with OpenClaw. Uh, I mean, you can connect a ton of tools, including emails, uh, uh, browsers, uh, file systems. Uh, and, and, and they are not just chatbots, right? Like, they can actually do things, uh, across environments. Uh, I mean, this means sort of hidden managed agents may quietly, um, have the same reach as a powerful employee, right? Like, but without any identity, uh, limits or oversight you would typically have on a human. Um, so I would say CISOs and CDOs are very concerned, right? Like about these sort of, uh, shadow agents. So discovery becomes an important part, right? Uh, uh, I, I remember one of these CISOs, uh, that I was talking to recently was talking to me about how some employee just ran OpenClaw in an enterprise setting. Mm-hmm. And how disaster that could be, right? Like, uh, so they, they had to sort of really, uh, warn that employee and also sort of ensure everybody's aware that, uh, this is, this is a big credible thing that they need to be careful about. So most people I know who are running OpenClaw are running in a very isolated environment today. Mm-hmm. It's just not ready for enterprises to sort of, uh, adopt. Uh, but, but it is the area that we can't stay away from as well, because autonomous agentic system are a thing and they are gonna come. We need to sort of pay attention to the space and actually see how do we improve the governance, right? Like that's a big part, in my opinion.

Speaker 3: Mm-hmm. Venkat: Um, Kevin: yeah. Yeah. How do organizations do that discovery process that you talked about to find those shadow agents? Venkat: Yeah, 100%. I, uh, so agents today come in all form and shapes. Mm-hmm. They can run in your laptops, they can run in, uh, uh, sort of these model providers or agent platforms, uh, and they can also run in, uh, sort of, uh, cloud platforms like Kubernetes and- Mm-hmm and those kinds of things. Uh, being able to sort of, one, look for signatures of what agents, uh... Because there is a difference between AI usage and agentic usage. Uh, uh, so being able to sort of understand the nuance and the difference, I think, is gonna be key to be able to sort of identify what is an agentic use versus what's a, a typical sort of AI use, uh, just for some advisory or sort of augmented, um, uh, uh, sort of, uh, uh, intellectual, sort of arti- uh, sort of AI-based reviews, right? Like in that sense. Mm-hmm. So the way we think about this is, uh, we have a gateway. There needs to be something that basically intercepts the request that goes from these systems to something like an LLM model. That's a good place to sort of hook to sort of understand, okay, what kind of request and response goes in and out, and look for signatures that tell us that this is agentic use. Mm-hmm. That's definitely one way to look at this. Uh, we're also sort of - Uh, we also have other ways to sort of, uh, look, look for processes in the companies which will essentially force us to identify that there is an agentic use, right? Mm-hmm. Like, identify those and make sure that there are gates there. Uh, I mean, this could be... When somebody's asking for a LLM key, I mean, that's a great place to start, right? Mm-hmm. Like, you know that they need an LLM key to be able to start doing agentic workflows. That could be a great place to start, and so forth. So this is how you sort of identify. This is structural way to sort of identify this, uh, Kevin. Kevin: You talked earlier about, uh, the concept that, uh, you have a compliant measured autonomy. Mm-hmm. Uh, where there's some semantic dimension involved. Um, for one thing, I would assume organizations are very different in terms of their risk tolerance, uh, of what level of autonomy they would be comfortable with. Um, but they may not even know how to articulate that. So how do you provide the capability so that organizations can figure out really what level of autonomy they wanna provide to their agents? Venkat: Oh, it's a great question. Um, I, I would like to use a phrase called autonomy budget, right? Mm-hmm. Like, not every action deserves the same level of freedom. Um- Enterprises should classify actions by, let's say, a few dimensions, right? Mm-hmm. Does this agent do activities that are irreversible? Does this have financial impact, a regulatory impact, customer impact? Um, uh, or does it use operate da-data sensitive, uh, sensitive data in this case? If an a-if an agent is a low risk and it has reversible things, give the agent more autonomy, right? Mm-hmm. If it's a high risk, reversible, uh, irreversible, um, and it has customer-facing capability, it should require approval, uh, stronger identity checks, and potentially some circuit breakers. Um, a, a practical example, in my opinion, would be, let's say if you have a customer serv- a customer support agent, um, I mean, it can - it should be allowed to summarize ticket autonomously. It should be able to draft response autonomously, um, and maybe even sort of issue maybe a ten dollar refund, uh, autonomously. I mean, if the customer is eligible. Mm-hmm. But a five thousand dollar refund or changing the contract should definitely require a human approval. Uh, the key is not, like, do human-in-the-loop everywhere, uh, otherwise agents will start to look like a very expensive suggestion box. Uh, but also too few approvals also makes this agent dangerous. So being able to sort of hit that balance, uh, is gonna be key, and that you could essentially start to do this with some of the control and policies that we talk about for runtime governance. Mm-hmm. Kevin: What about multi-agent interactions? Uh, how to deal with situations where it might be that, that each agent, uh, is operating properly, but there's unanticipated consequences or risks that come up, and we're going to be in this world, if we're not already, where there's fleets of agents interacting with agents. So how do you deal with that challenge? That's a great question, Kevin. So, Venkat: um, the way you think about multi-agent is you don't just devise governance for agents individually, right? Like you govern interactions, handoffs, uh, and, and potentially violations that could emerge from that. Uh, the, the key here is to just stop thinking that agents are, like, governed in isolation, right? Um, let's- Take an example, right? Like there's a sort of a procurement agent and let's say the AP payment agent. Uh, procurement agent updates, uh, sort of vendor record after reading some onboarding documents, and accounts payable verifies the vendor is active before it pays the invoice. Each agent is doing something reasonable on its own, but together they could violate segregation of duties if the same workflow effectively gets to both of them, right? Something that basically says, "Okay, create the agent, uh, modify and update the agent, and then like release money." Like that could potentially be- Mm-hmm ... uh, dangerous. So I, I think one way we think about this is we need to have fleet-level policies in addition to agent-level policies. Every agent needs to have identity. Um, every handoff needs to be authorized. Um, so you need policies like no agent or combination of agents can complete this workflow without approvals, right? Uh- Mm-hmm ... so this helps, especially in a multi-agent scenario, um, the segregation of duties that we care about, and that should be treated as how we care about today for humans, uh, in, in the- Mm-hmm agentic world. Mm-hmm. Kevin: Uh, there's a number of standards that are developing in the agent world, probably the most prominent being m-MCP for connecting to external tools. Um, but, you know, how do you see the, the development of these, uh, open standards? How do they relate to the work that companies like yours are doing? And, you know, where do the, the standards potentially open up new vulnerabilities? Venkat: No, um, MC-MCP is a very powerful tool, uh- Mm-hmm ... but it standardizes - because it standardizes how agents connect to- Mm-hmm ... tools and context, right? Uh, but that effectively, like you said, also creates newer attack surface. Uh, the mistake that I see people do typically with an MCP is just treating that as just another API. Because it's- Mm-hmm. Yes, it is an API that's configured as a tool, but LLM is often designing, deciding, uh, what and how to call these tools. Uh, the tool descriptions, uh, how the server behaviors, um, uh, how the contents are processed, uh, and the agentic reasoning is all sort of intertwined. Uh, so this creates different risks, in my opinion, um, that of course MCP, uh, users should be thinking about tool poisoning, for example. Um, uh, supply chain, which is like, what if an MCP server gets infected? You pretty much infect the whole agent. Uh, COSAI, which is, uh, a community we are part of, um- Mm-hmm ... uh, just basically wrote a paper about sort of how many threats are available just in the base MCP specification. Mm-hmm. There are over 40 threats across 12 categories that they've identified, including identity privilege escalation, uh, supply chain integrity, network security, and so forth. I think for enterprises- The first thing is create an MCP inventory. Um, just have an allowed list of sort of trust, trusted servers. Uh, we need to make sure there is authentication authorization. That was not given in MCP, surprisingly, right? Mm-hmm. Like as you can imagine, I mean, this is a age-old solved problem, but it's still one of those, uh, younger ones in this, uh, in this new world. Sandboxing execution, being able to sort of really isolate execution, uh, when it does. I think all of these are important things. Um, one thing I would also add, uh, Kevin, is skills. Agent skills are becoming another sort of level of- Mm ... MCP servers. So what happens is MCP servers are these tools, and then agent servers have this higher layer that allows you to sort of define the workflow of how this tool can be used. Um, I think that's going to be another interesting surface to sort of watch as, uh- Mm ... because anybody can - Because all of these instructions are written in text files, um, think about all of these different s- uh, skills that will be out there in the company, uh, and sort of really managing that, this at scale is going to really have, uh, interesting challenges, right? Like I- Mm-hmm ... I think OWASP wrote something called as, um, a lethal trifecta. Uh, I think when a skill or an agent, or even in this case MCP, have access to, let's say, private data, it has access to external communication, and then it ca- it gets exposed to some untrusted content.

Speaker 3: Mm-hmm. Venkat: That is a lethal trifecta, right? Mm-hmm. Like so you do have to assume in those cases that there will be exfiltration that's possible, uh, unless you have strong controls. So, uh, being able to sort of keep these mind, uh, and sort of ensuring that there are specific controls are gonna be so important for enterprises. Kevin: Last thing I wanna ask you is, we're, we're running low on time. Uh, going forward, what happens when agents are embedded in physical things? And I, and I ask in particular because you mentioned you worked at Rivian, which builds cars. So what does that world look like in terms of the risks of agents? Um, and then, you know, can we overcome the kinds of challenges with the harms that might come about in that environment? Venkat: Absolutely. I think digital- failures are often recoverable, right? Physical failures have more consequential, uh- Mm-hmm ... and irreversible sort of impact. Uh, governance should largely shift to pre-action guarantees where- Mm-hmm ... the system proves safety before execution, uh, right? Like not just explains its action later. Um, I mean, imagine a robot with a hazardous chemical in the factory, right? Mm-hmm. Uh, or a home robot without access to knives. Uh, these are all dangerous situations. I, I think the lesson that we could take from automotive is they do not just rely on intelligent model. It's not like every time when a new update comes, it's, it's a bigger, larger model. It's about really thinking safety as an entire stack, right? You need to have simulations. You need validations. You need redundancy in the system, fail-safes if something were to go wrong. How can you monitor this, uh, with telemetry? And sort of really establishing clear boundaries on what this system is allowed, uh, to operate where, right? Um, I think one way to think about this is, um, governance need to be closer to the safety engineering in the physical world. Uh, you need to ask questions like, okay, what's the operating envelope? Uh, what happens if a sensor disagrees? These are all problems- Mm-hmm ... I think from a software world you, you can... These are not gonna be catastrophic, right? Like y- what is the safety net, for example? Okay, can this agent really move fast, uh, the location and the proximity limits that it actually has access to and so forth. So yeah, I mean, so hallucination are just embarrassing in this world, and in the physical world, I mean, if, uh, I mean, for hallucination of forklift, you need bumpers, right? Like you need to sort of... You need to think about this more holistically than just, uh, uh, just a, a software problem. Kevin: Absolutely. Well, it's an exciting and a scary world that, that we're going into, but one where the, the kind of work that you're doing is absolutely going to be critical. Venkat, thank you so much. Really a pleasure to speak with you. Okay. That's great. Thank you. This has been "The Road to Accountable AI." If you like what you're hearing, please give us a good review and check out my Substack for more insights on AI accountability. Thank you for listening. If you wanna go deeper on AI governance, trust, and responsibility with me and other distinguished faculty of the world's top business school, sign up for the next cohort of Wharton's Strategies for Accountable AI online executive education program, featuring live interaction with faculty, expert interviews, and custom-designed asynchronous content. Join fellow business leaders to learn valuable skills you can put to work in your organization. Visit execed.wharton.upenn.edu/acai for full details. I hope to see you there.