The AI Control Loop: When AI Goes Rogue - with Craig Thomas of Wallarm

[SPEAKER_00]: Hello listeners, today we are dropping another episode in our series The AI Control Loop. [SPEAKER_00]: Out Enterprise is governed the AI they've already deployed, sponsored by our friends at Walmart. [SPEAKER_00]: Walmart is the AI Control Platform for Enterprise AI, protecting every AI workload, API, and application in production, giving CSOs the governance they need and CIOs the speed they demand. [SPEAKER_00]: Organizations choose Wallarm for a complete inventory of APIs, AI agents, and AI apps using patented AI ML-based threat detection and blocking the operates at production speeds. [SPEAKER_00]: In this episode, Craig Thomas, senior solutions engineer Wallarm, examines what rogue AI actually means in practice, where the risk materializes, and what it takes to move from detection to control. [SPEAKER_00]: Craig, thank you for being on the show today. [SPEAKER_00]: Thanks for being on code story. [SPEAKER_00]: Yeah, great to be here. [SPEAKER_00]: Appreciate it. [SPEAKER_00]: I know. [SPEAKER_00]: Absolutely, really excited to dive into our topic today around when AI goes rogue. [SPEAKER_00]: Excited to dive into that. [SPEAKER_00]: Excited to hear your perspective and bringing in all of your experience and including your experience with wall arm. [SPEAKER_00]: But before we jump into that topic, tell me a little bit about yourself. [SPEAKER_00]: Tell me in my audience a little bit about you. [SPEAKER_01]: So actually, my background, I spent over 10 years in Department of Energy in multiple roles from a network engineer all the way up to a CSO of the nuclear weapons plants. [SPEAKER_01]: So bringing the experience from the customer side and a highly regulated environment, then spent several years running a consulting company where working with those same highly regulated industries. [SPEAKER_01]: And so, [SPEAKER_01]: When I joined Wallarm, they had worked with APIs and worked with automation for quite a while, and really interested to how we bring that security to the customers, right? [SPEAKER_01]: Both from an API and an AI perspective is that that continues to evolve. [SPEAKER_01]: And then outside of work, pretty hands on, like a tar or spend time in the yard, like the mess with my long quite a bit. [SPEAKER_01]: a lot of my weekends revolve around kids activities or shows and baseball and football and so keeps me pretty busy and always somewhere to be. [SPEAKER_00]: Appreciate that overview. [SPEAKER_00]: It kind of gives me an idea of who I'm talking to. [SPEAKER_00]: Let's dive into the meat of it. [SPEAKER_00]: So again, our topic is when AI goes rogue and when we say rogue AI, what do we actually mean there? [SPEAKER_00]: What do you guys actually mean there? [SPEAKER_00]: Is it only malicious AI [SPEAKER_01]: kind of level setting where we're coming from too, right? [SPEAKER_01]: AI is really interesting. [SPEAKER_01]: The fact that it really does permeate all parts of the business from IT, cyber, compliance, obviously the business itself. [SPEAKER_01]: It's adding value and accelerating work, but how do we better discover, observe, enforce and govern that and do it at the speed of AI, right? [SPEAKER_01]: And so, some of those are starting with rogue. [SPEAKER_01]: And to your question, rogue doesn't necessarily mean militias. [SPEAKER_01]: And in a lot of times, it isn't militias. [SPEAKER_01]: most common rogue AI scenarios we're seeing are similar what we saw in the API world right they're not attacks their legitimate systems but they're behaving outside their intended boundaries so their intent was fine but the outcomes weren't from that perspective but we can't look at that and I'll look at that some three various categories unsanctioned AI that AI that nobody approved it it's running in your environment and it gets into shadow a bit unmonitored AI so we have AI that no one's [SPEAKER_01]: unpredictable AI so that gets in a bit of the point to the road but it's approved and watched but it's doing things no one anticipated whether those be from the business side or the security side so that you've got agents emerging kind of as that biggest emerging category because they're doing things on behalf of users and also rather than just a specific question to a chatbot and a specific answer now it's going and taking actions on that behalf and that's introducing even more rogue actions at scale [SPEAKER_00]: Absolutely. [SPEAKER_00]: Now, that makes it clear when we think about Rogue AI doesn't necessarily have to be something malicious, but it's going off on its own, and what are the most common ways that AI systems drift outside of intended boundaries? [SPEAKER_00]: And once an organization understands what Rogue AI looks like, where does that loss of control typically begin and who is responsible for preventing it? [SPEAKER_01]: the operating definition I like to use is it's working as a design, but not as intended. [SPEAKER_01]: And so this goes back to computer science 101, right? [SPEAKER_01]: It is, we're capturing a developer or designer designs at one way, but then it's being used for other ways, whether maliciously by bad actors or just from the software responding, different than you expect. [SPEAKER_01]: But one of the most common ways is prompt injection. [SPEAKER_01]: It's an underrated drift mechanism, [SPEAKER_01]: Support ticket, scrape web page, user message, it can redirect an agent's behavior. [SPEAKER_01]: No one touches the model, right? [SPEAKER_01]: With drift events on an AI particularly, you've got various components and several of those are out of your control. [SPEAKER_01]: Model updates may change the behavior without you changing a single thing or single code deploy. [SPEAKER_01]: If you're calling a hosted LL in the provider updates the model, your agent's behavior can shift. [SPEAKER_01]: No pool request, no change management, [SPEAKER_01]: Back to the conversation of what broke and why you've got to go a layer deeper, and that's a big way that AI systems can drift because of From the user interaction or the agent interaction to an API to a tool to an LLM. [SPEAKER_01]: There's various components in there that can all be changing and outside of your general control. [SPEAKER_00]: Let's move into the shadow world, right? [SPEAKER_00]: And shadow AI is the unsanctioned or unmonitored use of AI tools, apps, or embedded features by employees within an organization. [SPEAKER_00]: Basically, usage of unapproved AI, right? [SPEAKER_00]: It's in looking at shadow LOMs. [SPEAKER_00]: How do these things create risk even when no attacker is involved, right? [SPEAKER_00]: Let's define that. [SPEAKER_00]: And if AI drift often starts with normal business activity, [SPEAKER_01]: I'd define shadow LMs or that they are the new shadow IT, right? [SPEAKER_01]: A developer spins up an LLM endpoint in AWS or Azure or somewhere else, because the internal approval process takes weeks, if not months for various things. [SPEAKER_01]: And so typical behavior with shadow I and shadow LMs, right? [SPEAKER_01]: It's because it takes too long. [SPEAKER_01]: They have a legitimate use case on business case. [SPEAKER_01]: what they want done, and so there's no security review, no data handling policy, no logging. [SPEAKER_01]: Or, but you bring in a new third party tool with its own AI built-in, calling LLMs that you never approve. [SPEAKER_01]: Need to be able to see that, surface that, and stop it, or at least provide feedback to your vendors, right? [SPEAKER_01]: So that you understand what's going on there, and you understand the full risk picture. [SPEAKER_01]: But maybe a little bit outside of the scope of this conversation, but S-bombs, software bill materials, and AI bill materials, they're crucial to understand truly what makes up your software and understand all your risks. [SPEAKER_01]: Those are point-and-time, maybe from your vendor, but the fact that you need to be able to understand and build those, you can understand the risk and stop that as well. [SPEAKER_01]: Now, I'd ask, kind of, I kind of add one other thing. [SPEAKER_01]: Unsanctioned agents, [SPEAKER_01]: But they inherit the developer's credentials, right? [SPEAKER_01]: So developer goes in, just puts his credentials in, and now they get a broader role than they probably should out. [SPEAKER_01]: So the agent, the same person as me, or you, right? [SPEAKER_01]: Maybe in an administrative role, which is much more than it needed to actually do the task at hand. [SPEAKER_01]: And so those are a couple of the things from Shadow LM, you're really looking at. [SPEAKER_01]: One big stat that kind of comes out 72% of corporate AI tools and active use [SPEAKER_01]: not future state problem that's current state today, so we have to get that under control for sure. [SPEAKER_00]: Yeah, no doubt. [SPEAKER_00]: And it's clear already from our conversation that this is important and it's a hard problem to solve. [SPEAKER_00]: Why can an AI action look legitimate in isolation? [SPEAKER_00]: Still create serious business security or compliance risk when viewed as part of a larger sequence of actions or a workflow. [SPEAKER_00]: As these shadow systems become more embedded in their [SPEAKER_01]: One, because our tools aren't keeping up right, tools that are using signatures or specific, this is allowed, this isn't allowed, just can't keep track when you're doing chaining actions together. [SPEAKER_01]: In that case, individual actions pass every check, but the sequence could fail the audit. [SPEAKER_01]: Maybe reading a customer record is fine, summarizing it's fine, passing the summary to a third-party API, all good, right? [SPEAKER_01]: But doing all three automatically at scale without a human review step, it could be a conversation around GDPR or other things that you just don't want to have. [SPEAKER_01]: It goes back to even different parts of data and isolation or fine, but once you train those together, you can have the classified or other security events in the national security space and similar things are happening here. [SPEAKER_01]: And then by next step, Noah's on Chained Agents, right? [SPEAKER_01]: They multiply the blast radius, because rather than just a single agent doing a single thing, Agent A calls Agent B, which calls Agent C each step, looks good, authorize API calls with the end result. [SPEAKER_01]: There's a capability, no one really explicitly designed or approved. [SPEAKER_01]: They just emerge from the chain and these agents learning what they can do and how to get what they want done. [SPEAKER_01]: And so then you're a lot of times you're losing context. [SPEAKER_01]: Indian session tracing really is the type of ability that makes this sequence level analysis possible. [SPEAKER_01]: We can't believe you have to be able to trace every AI session from the initial prompt through every model call tool, [SPEAKER_01]: get a single record as otherwise you're losing that context and you're not able to reconstruct that and understand the true risk of what's going on to be able to eliminate or minimize risk and take action in a larger environment. [SPEAKER_00]: Let me make sense and I think your answer probably opens up. [SPEAKER_00]: or leads into an answer to my next question, how do APIs and integrations and connected systems amplify the impact of the seemingly legitimate actions? [SPEAKER_00]: Now, I think I could probably as an engineer, cherry pick some of the things you're probably gonna say from the last answer, but what changes once those actions begin flowing across these workflows, across APIs, applications, and interconnected systems. [SPEAKER_01]: Yeah, it goes back to, I know conversations you've had in previous weeks, right? [SPEAKER_01]: APIs really are the central nervous system of AI, right? [SPEAKER_01]: Every meaningful action or AI agent takes eventually. [SPEAKER_01]: It becomes an API call, well, that's to a database, to a SaaS tool, to a payment system, identity provider, MCB servers. [SPEAKER_01]: If you're not watching that API layer, or you're not watching the agent, and then that integration surface enormous, it's unemotored. [SPEAKER_01]: We'd powder you protect, which you don't know, modern enterprise environments have thousands if not tens of thousands of API endpoints. [SPEAKER_01]: Many of those are undocumented, they might be third party, yeah, as it might be your own APIs. [SPEAKER_01]: And so it might be endpoints that should have been deprecated a long time ago. [SPEAKER_01]: The issue now becomes even more because an AI with broad permissions, they'll find those, they'll call them, they'll be able to do what they need to do to get things done. [SPEAKER_01]: But you no longer dealing with just users or even just scripts that are dealing with specific APIs that you're dealing with agents that multiply this and it's being able to integrate AI security, API security because of this amplification effect becomes crucial. [SPEAKER_01]: A lot of folks treat these as separate problems, API gateway for API traffic, AI gateway for AI traffic. [SPEAKER_01]: separate tool for AI behavior, but AI didn't interact with all these external systems to APIs, and that's where data-exfiltration risk, typical materialized, separating the two creates a blind spot, exactly where the risk is highest, and so you've got to be able to see that full picture. [SPEAKER_00]: Right on. [SPEAKER_00]: Okay, let's switch over to leadership of CIOs and CIOs. [SPEAKER_00]: What kinds of unexpected outcomes worry these folks the most today? [SPEAKER_00]: When AI systems are operating across these [SPEAKER_00]: And as that connectivity expands, what are security and business leaders most concerned about? [SPEAKER_01]: Great question. [SPEAKER_01]: A couple of things, one that jumps out to me right away is PI or sensitive data showing up where it shouldn't, right? [SPEAKER_01]: And these have real world consequences and real world finds. [SPEAKER_01]: So in AI workflow, process a support request, routes it through a summarization step, stores the output in the log and that log contains the customer social security number. [SPEAKER_01]: No one really designed for that data path, but it happened because the model was helpful. [SPEAKER_01]: They got what it needed, but it now logged that information where it shouldn't have. [SPEAKER_01]: Now, I think another big thing is, yeah, I can't explain what it did problem. [SPEAKER_01]: And so that can be from the CISO's hat or from the CIO or business's hat. [SPEAKER_01]: A CISO can generally reconstruct the human attacker's path. [SPEAKER_01]: Reconstructing the AI agents decision tree across a multi-step workflow with external data inputs. [SPEAKER_01]: It's generally, it's just a hard thing to do, right? [SPEAKER_01]: regulators, board members, incident responders, they all want the story, but today a lot of organizations can't tell it. [SPEAKER_01]: And any other kind of interesting story is unexpected answers or behaviors. [SPEAKER_01]: We actually have an ongoing conversation with the potential customer. [SPEAKER_01]: They built their own chatbot and going to change some details here to not reveal anything. [SPEAKER_01]: But you go ask this chatbot a question for instance. [SPEAKER_01]: So what do I feed to my new puppy? [SPEAKER_01]: Right? [SPEAKER_01]: 99 times out of 100 or 99 times out of 1,000 responds with the skew for puppy food, a link you can buy it straight from their website. [SPEAKER_01]: However, one time out of that, it actually responds completely randomly. [SPEAKER_01]: And not only randomly, but in a really bad way to potentially feed your puppy something that's poisonous. [SPEAKER_01]: So that can't happen. [SPEAKER_01]: We have to understand what those responses are and make sure they're accurate 100% of the time because otherwise [SPEAKER_01]: So, those are a couple of the two big things know about unexpected outcomes, but from a cyber side as well as the business and liability side that jump out to me. [SPEAKER_01]: And then kind of a third is just that reputational risk from out, but it's not just access. [SPEAKER_01]: A.S. system that publishes incorrect to a biased content at scale or gives inappropriate recommendations to customers like I talked about. [SPEAKER_01]: It's a brand and legal risk. [SPEAKER_01]: It doesn't fit neatly into traditional security frameworks. [SPEAKER_00]: I appreciate you walking through that. [SPEAKER_00]: What does meaningful oversight actually look like when AI systems can act at machine speed? [SPEAKER_00]: And that can we talk about interconnected systems and impact there. [SPEAKER_00]: That can really amplify the effects of some of these actions. [SPEAKER_00]: So what does that meaningful oversight look like? [SPEAKER_01]: From a security practitioner, your policy alone is not oversight. [SPEAKER_01]: We've always said that, right? [SPEAKER_01]: Paperwork doesn't make you more secure. [SPEAKER_01]: Policy document does not stop an agent mid-call, right? [SPEAKER_01]: So real oversight fires actual runtime enforcement, the ability to inspect, block, or write limit that AI generated API traffic in real time. [SPEAKER_01]: That closed loop model, really to no see, stop, and prove. [SPEAKER_01]: You need to know what the AS systems are running and what they've connected to. [SPEAKER_01]: You need to see what they're doing in real time. [SPEAKER_01]: You need to be able to stop the behavior before the consequences compound. [SPEAKER_01]: But increasingly, you need to be able to prove to auditors that you did this, right? [SPEAKER_01]: Most organizations have no, at least partially, and they've got some logs, but see stop and prove definitely are the gaps. [SPEAKER_01]: And then humans in loop do matter, right? [SPEAKER_01]: But they need to be at the right point. [SPEAKER_01]: You can't put a human in the loop for every API call. [SPEAKER_01]: You put humans at decision gates for high risk actions, boundary crossing transactions. [SPEAKER_01]: But then the system has to surface those moments automatically, right? [SPEAKER_01]: So, [SPEAKER_01]: as you're looking at tooling and other things, those key aspects need to bubble up so that humans can take action quickly, understand the context around that, and then the tool has to be able to enforce these policies, like you said, at machine speed. [SPEAKER_00]: That makes sense, and enforcing the policies as quick as the actions are being done. [SPEAKER_00]: That seems clear, that would be a meaningful, that would be meaningful oversight. [SPEAKER_00]: Okay, how should organizations distinguish between the experimentation they want to encourage, [SPEAKER_00]: In the unmanaged AI behavior that they need to control, one challenge is balancing governance with innovation. [SPEAKER_00]: And as an engineer as a startup founder myself, that I, that holds true day-to-day, how do organizations avoid slowing down AI adoption while still maintaining control? [SPEAKER_01]: The security organization can't put the no in an innovation, right? [SPEAKER_01]: We've heard that many times, right? [SPEAKER_01]: So the answer is not slow everything down. [SPEAKER_01]: Organizations that over restrict AI experimentation, one you'll lose your best engineers, you'll lose your best people to competitors that don't. [SPEAKER_01]: As well as the business is asking all of these quarterly earnings calls and other things are asking specifically what the AI strategy is in plan. [SPEAKER_01]: So we have to be able to support that, but do it in a secure way. [SPEAKER_01]: So I think the first thing is, hey, define the sandbox clearly. [SPEAKER_01]: Make it easy to use, no. [SPEAKER_01]: If you can go out, you can innovate, you can test, but do it truly with the Dev Environment, isolated credentials, scope, API access, no production data. [SPEAKER_01]: logging enabled. [SPEAKER_01]: So you can understand that you can experiment quickly. [SPEAKER_01]: And that sandbox needs to be a first class experience, not just an afterthought. [SPEAKER_01]: And it needs to be usable, right? [SPEAKER_01]: We've all dealt with Dev environments that didn't mirror production at all and caused all sorts of issues once it went to prod. [SPEAKER_01]: Being able to do that to be able to support continuous AIS bomb going forward here as well so that in those environments you can understand what [SPEAKER_01]: as well and then create a fast path to production right shadow AI grows when that official path is slow So your point is you can't slow it down people are going to get around it one way or the other right either They're going to do their own thing outside your boundaries or they're going to leave so a lightweight AI system intake process This security room is data classification runtime monitoring the completes in days not months or years in some cases removes the incentive to go around it [SPEAKER_01]: And then I think the last thing is you're tooling being able to easily enforce new new sandboxes or new environments that are stood up is important too, right? [SPEAKER_01]: So rather than having to go from your security team or your developers to the infrastructure people and all it, it'd be able to just click a button from some of these AI governance and observability tools. [SPEAKER_01]: do the enforcement monitoring and enforcement in these new environments with the click of a button in seconds rather than like as weeks or months also is important so you can get that visibility of the enforcement as you see new things spin up. [SPEAKER_00]: Okay, I'm tracking with that. [SPEAKER_00]: Okay, Craig, there's one more question I've got, and we know that many organizations can detect risky AI behavior. [SPEAKER_00]: That's a fact, but if they can't stop it in real time, what critical gap still remains? [SPEAKER_00]: Even with these governance programs in place, many orgs are still operating reactively. [SPEAKER_00]: Kind of close out our conversation, what's the key difference between detecting AI risk and actually controlling it? [SPEAKER_01]: as a security professional or as the CSA. [SPEAKER_01]: You don't want just another dashboard to tell you how bad you're doing, right? [SPEAKER_01]: And detection without prevention is just documentation of damage, right? [SPEAKER_01]: Knowing an agent exfiltrated data after the fact doesn't elinex filtrate it. [SPEAKER_01]: It doesn't keep you from finds and other regulations. [SPEAKER_01]: Bray our systems operating at machine speed, but the gap between that action taken and the action detected can mean thousands of API calls and gigabytes of data before you're ever alerted to that. [SPEAKER_01]: And also, at machine speed, looking at logs and some of your traditional sims just aren't able to keep up with AI speed actions, these metrics, the traditional metrics, they just look even catastrophic. [SPEAKER_01]: So one minute response time is just way too slow. [SPEAKER_01]: of an agent made 50,000 API calls in that minute. [SPEAKER_01]: And regulators even are moving toward real-time accountability, right? [SPEAKER_01]: So you've got the EUAI Act, which is coming fast upon us. [SPEAKER_01]: And the SAI are a math, emerging SEC regulation. [SPEAKER_01]: It all signal that we detected eventually just won't be sufficient. [SPEAKER_01]: So the expectation is shifting towards, like, demonstrated ability to prevent, not just to discover. [SPEAKER_01]: And so that's what definitely we are looking at. [SPEAKER_01]: How do we close this gap? [SPEAKER_01]: Most organizations have that SAM, the AAPM, the AAAPI gateways. [SPEAKER_01]: But none of these tools see inside a session and stitch all those pieces together. [SPEAKER_01]: They just see headers, they see latency, they see logs. [SPEAKER_01]: But they don't see the prompt, the payload, the data return. [SPEAKER_01]: We're looking to be able to see all of that at the kernel level and act on it in line at machine speed. [SPEAKER_00]: Oh, that just makes so much sense and thinking about reactive versus control and processing that in real time just really brings it to light. [SPEAKER_00]: How important how fast this moves. [SPEAKER_00]: So Craig, I really appreciate you being on the show today. [SPEAKER_00]: It's clear that Rogei doesn't necessarily mean malicious AI. [SPEAKER_00]: Actions can appear as legitimate business actions and are often difficult to determine. [SPEAKER_00]: and the chaining of these events through interconnected systems, obviously, and clearly, through your answers amplifies the impact quickly. [SPEAKER_00]: You have to bring the speed of governments to be as fast as the speed of action. [SPEAKER_00]: A business must keep that balance between control and innovation, but to control AI risk, businesses have to move to real-time action to these issues in order to prevent the damage and to get out of that reactive mode. [SPEAKER_00]: So I really appreciate you explaining all this sharing your perspective and being on a showcrack. [SPEAKER_00]: As you can see from Craig's answers, Rogue AI is something to take seriously. [SPEAKER_00]: Even if it's not malicious, AI is powerful. [SPEAKER_00]: And businesses need to take stock of how people are using this technology. [SPEAKER_00]: With a balance of ensuring open posture towards innovation. [SPEAKER_00]: If you'd like to learn more about Walarm, you can visit Walarm.com. [SPEAKER_00]: That's W-A-L-L-A-R-M.com. [SPEAKER_00]: And thanks again for listening.