AI Red Teaming Isn't Enough: Shoshana Cox, AI Security Researcher

Speaker A: These boxes, these spaces are absolutely massive. We've determined that it's not feasible computationally to run a search through them. So what that means is we can't search and find all the attacks. So with a traditional pen test, you can pretty much find all the attacks that, uh, you know, may be relevant to a system. You cannot do that with AI systems. It's not really possible to do. So, um, what you can do, because there are effectively an infinite number of attacks, is, you know, spray a bunch of prompts at something and say, oh, my God, See, look, all these attacks worked and then, look, I blocked them all. Now, like, you can do that, but did you actually really provide meaningful security? Uh, I would argue no, you absolutely did not. I think that is the crux of what's wrong with AI red teaming right now.

Speaker B: Um, hello, everyone. You're listening to Cloud. Next your go to source for Cloud Innovation and Leaders insight brought to you by Global Dots. Our guest today is Shoshana Cox, an AI security researcher focused on how AI systems actually fail in practice. Back in 2022, she approached this space from a defense perspective, assuming many of the core problems were already understood. But over time, she realized the industry's understanding of AI security was far more broken than it seemed, something that led her to challenge some of the dominant approaches we see today. I'm Ganesh, the awesome solutions architect at Global Dots, where we research innovations every day, so you don't have to. But we're not here to talk about me, we're here to talk with Shoshana. Shoshana, thanks for joining us. So great to have you.

Speaker A: I'm so happy to be here. Thank you so much for having me.

Speaker B: Um, before we get into the juicy stuff, I really want you to outline your history in AI and how you came to this point where you are today. Because I know you're a rockstar and I followed you on LinkedIn and read a lot of your posts, but a lot of people would claim to be AI experts, and they're definitely not. And I want, uh, the listener to understand that you definitely are and for you to lay down the credentials so that people know they should be listening, basically.

Speaker A: Um, yeah. So I've been doing this for a really long time. I started working with AI and security back before it was cool, uh, before the genai revolution in 2022. Um, so I was, um, back in about 2010, working on, um, looking at how these systems could break. And what I saw was very disturbing. And, uh, there was a great amount of literature, uh, out there for how to mathematically attack these systems back then. And I just sort of, uh, dove into it. So we found that there were, um, serious, uh, uh, vectors for attack for all AI systems from predictive AI to now, including generative AI. And obviously, um, agentic inherits those vulnerabilities as well. But these go back, uh, to the beginning of AI itself. So they're not new. Uh, and anyone who's been working on this for a very long time certainly recognizes the new quote, unquote, new vulnerabilities and prompt injections. Um, but yeah, that's how I got started. Uh, I was a red teamer initially and then started poking around in AI, uh, and since then, uh, I wrote a paper in 2022 that was the first, uh, AI security operations paper in machine learning, security operations, whatever you want to call it, um, with the first architecture for exactly how to implement this defensive, um, security. And then I got a patent, um, in 2024 for defensive, uh, federated architecture. And I've been working on the OWASP AI exchange, serving on the core author team there. Uh, I do work on federated AI. I, uh, am also the AI policy lead and I served on the team that helped, uh, prepare requirements for the EU AIX technical sections. So, uh, that's kind of what I've been working on up until now. I, uh, started my own startup a few months ago. So we're building in stealth. And um, yeah, uh, I kind of enjoy working at the bleeding edge of AI research and security.

Speaker B: Very cool. Very, very long list of accolades that nobody could possibly argue with. Uh, and specifically around this point, I'd love for you to flesh out AI red teaming. And if you can start with so people understand what AI red teaming is before we jump into it, just so they know we're talking about the right thing. Uh, and then if you can break down for us the myth of that, uh, and how that has pervaded into software today, because I think it's very interesting.

Speaker A: Yeah, so I mean, that's kind of the problem though, isn't it? Like, what is AI red teaming? So for people, I don't know how much people know about security testing and so forth, there's offensive and defensive security, and uh, the defenders are the ones that are hardening systems and trying to make sure hackers don't get in. Uh, and then you also have good guys who attack systems professionally. And those, uh, are the red teamers, those are the offensive security professionals. Uh, so the idea, uh, is you can do anything from you Know a penetration test and test someone's application or network security all the way up to a full red team engagement, which typically in security meant that you would be um, testing all types of systems, potentially like not just the computer systems, but also potentially physical security, um, social engineering might be on the table in the rules of engagement. So this idea of red teaming is um, the idea that we are the red team, that's the offensive team and we are going to just attack this from all possible angles. So, um, AI didn't used to have that back in the day. AI security was its own thing, but it wasn't called red teaming. You could attack these models, you could test for them. When it started becoming red, uh, teaming to my understanding is after the Genai, um, big launch and the adoption and what happened was people found that these models had vulnerability, uh, vulnerabilities. Excuse me, how did they find that? Uh, they found it by typing in natural language prompts to see if they could trick the model. So I believe, my understanding is that this was analogized to the social engineering aspects of red teaming because they kind of said, okay, prompt engineering or uh, prompt engineering, prompt injection and so forth. This is social engineering. Right. So when we do it for security, we are red teaming the AI. There is your long winded back history of why it came to be called that. If you haven't noticed by now, I disagree with that nomenclature. I do not think that is the right name. I think it should be called something else, but not that.

Speaker B: Let's pick a name just for fun. Um, what would it be if we called it something sensible?

Speaker A: So what is being done right now in the industry? Nothing. You should not do that at all. Uh, if you were going to actually test AI systems, I would call it AI Security testing. Um, if you really wanted to do a red team operation, to my thinking, you need to like show up on site like on prem, and see if you can, I don't know, attack the model some way that way. But um, yeah, that's how it got its name. I prefer security testing. It's more straightforward and hopefully conveys what you're actually trying to do.

Speaker B: So like by definition the real AI red teaming would be turning up with a parcel at the people's office and seeing if you can get in there and then attack the AI machine with a baseball bat or something like that.

Speaker A: Right. I would just see if I can get someone to badge me in. That's usually the easiest way to get in is follow somebody and then like, I don't know, I'LL find the, The Gibson server room and proper, proper, proper

Speaker B: Hollywood style, just with, like, a boy, uh, the all gray boiler suit. You come in as the cleaner or something like that.

Speaker A: Um, I like it.

Speaker B: So let's go back to why it doesn't work, though, because let's just say that I work in an industry where I've had people demonstrate things to me that are definitely products claiming to stop this. And if you talk about prompt injection, that's, I wouldn't say totally understood, but maybe understood to some level. And you know that you can stop some prompts coming in, uh, ironically, by using AI to listen to the language that's coming in or stopping certain keywords. But essentially that's. That. That is unstoppable in, like, mathematically provable. That it is not. It's not something you can defend against. That's what you're saying.

Speaker A: That's correct, yeah. And. And, you know, I would argue that using AI to, To monitor these things is just idea all around. Um, shouldn't be done. Not a good idea from an architectural or security standpoint. But that's another conversation for another day, maybe. Uh, yeah. So, uh, mathematically we understand that. Uh, every model. Let me back up a little bit, actually. Every model comes with, um, a set of attacks that will work against it. Um, and this is like, it's called a subspace. An adversarial subspace is. You can think of it like a box that's attached to the model. Um, and these are all the attacks that'll work against that model. Right. Uh, and these boxes, these spaces, are absolutely massive. We've determined that it's not feasible computationally to run a search through them. Um, they're pretty huge. They're sparsely clustered. It's a mess down there. So what that means is we can't search and find all the attacks. So with a traditional pen test, you can pretty much find all the attacks that. That, uh, may be relevant to a system. You cannot do that with AI systems. It's not really possible to do. So, um, what you can do, because there are effectively an infinite number of attacks, is spray a bunch of prompts at something and say, oh, my God, see, look, all these attacks worked. And then, look, I blocked them all. Now you can do that. But did you actually really provide meaningful security? Uh, I would argue no, you absolutely did not. I think that is the crux of what's wrong with AI red teaming right now.

Speaker B: So these are like, basically the Donald Rumsfelds of AIs. These are like the unknown unknowns.

Speaker A: Basically the Donald Rumsfeld of AI.

Speaker B: No. You never heard that? There's a famous quote about Donald Rumsfeld when he was talking about the known knowns. It was talking about the war in Iraq or something and he was like, we got known knowns. We've got unknown unknowns. And we've got unknown unknowns. It was like the most ridiculous statement ever a human's ever made in politics. But anyway, y.

Speaker A: No, I'm sorry, I was only aware that Al Gore invented the Internet. Since we are talking about ridiculous political statements, I've learned something today. Thank you.

Speaker B: Uh, Tim Berners Lee might have had something to say about that, but that's okay. Uh, he's a penniless guy somewhere in the uk, living in a, uh, very conservative house. So these unknown unknowns, though, because you don't. These. I'm sorry, I forget what you called the space. What did you call this?

Speaker A: Like subspace adversarial. Like it's trying to fight adversarial.

Speaker B: But how, if you don't know they're there or we don't know the size of them, like, uh, uh, just for my brain and probably everybody else's brain who's new to this as a topic, like how. How do they, how do they get there? Or how do you ever measure this space? Or, or a better question, what the hell do you do about it if it's this like, unknown mystery box that lives with inside these models?

Speaker A: So that's three questions. I, um, will start with the first one. Uh, they get there because whenever you compress a high dimensional space into a lower dimensional one, um, you break certain assumptions, right? You create angular representations among, uh, the concepts that the model so quote, unquote learns, right? But those are not as rich in information as the actual relationships, right? So if we model something from our daily life, we are, um, inherently presenting less detail than the detail we ourselves understand, because we have to. Because, you know, you're simplifying something for modeling purposes. It's exactly the same in AI. Uh, so when you make these models, uh, boom. All of that compression has to go somewhere, right? So you create this adversarial space. Like now you have these angles where attacks are possible. Um, and that's just an artifact of changing the dimensionality of a representation. Um, it's. I can say that. And people are like, what? So I'm happy to go into more detail, but on the question too, I like it.

Speaker B: It sounds very. I was going to say it sounds very Stephen Hawkins I don't want to talk about that guy because he'd now appeared in the Epstein. It sounds, it sounds very, uh, quantum mechanicsy, which, you know, there's lots of very cool words there that all came together in a beautiful sentence, but it totally, it totally makes sense. Um, sorry, I interrupted. Onto the next part.

Speaker A: Yeah. So, um, my paper that came out a few months ago with my co author, Nicholas Wenzel, if you're not following him on LinkedIn, you should be. Um, he's done brilliant foundational work in this space, like actual mathematics and engineering. Um, um, so that paper laid out a system by which you can start to map the boundaries of these subspaces, um, using some pretty cool new techniques. One's called centered kernel alignment to measure similarities of models and some kind, uh, of hacker techniques. I shoved in there, um, kind of doing things the way hackers would a little bit. So the marriage of these, of these two methodologies, the mathematics and just a little teeny bit of black hat, if you will, um, wink, uh, I think is a good starting place for, uh, anyone who wants to do serious research on that. So that's how do we measure them? But more to the point, what do we do about that? Um, this is where threat modeling comes in. You absolutely cannot defend against every possible attack to your system. If you are running AI, we, um, can back up a little bit and say maybe you don't need AI for that. Right? Like, if you don't need AI, you shouldn't put it in because these are intractable vulnerabilities that you will be responsible for. For. All right? If you're dead set and you need AI, and that's the solution for you, you need to start with threat modeling, because the threat model is going to tell you what to test for. And that way you're not trying to boil the ocean. You're not trying to just spray random prompts that somebody told you were good. You know, you're not wasting money on the tokens. The time, the red team engagement. You have a threat model that says, this is what I want to test for. And this is how, let's go.

Speaker B: The horse has bolted on that one. Um, because it's too late. Everybody decided they wanted AI everywhere, all the time. Everybody's obviously backpedaling, trying to solve it and scrambling to do anything to help. And lord knows I'm 43 years old at this point. I'm like, playing catch up. I see all these kids doing things and I'm like, my God, you know, I. Why is an Open claw running my whole life. Maybe I'm, uh, maybe I'm doomed. But the people who are in the high positions are even older than me and they definitely don't have a clue what's going on. And you know, even, even something like, hey, you just need to provide a threat model for this. I'm pretty sure they'll, the thing they're going to do is go straight to like chat GPT or Gemini and say, how do I build a threat model? Because nobody really knows. So. So people are just desperate.

Speaker A: Ah.

Speaker B: And this is why I totally hook, line and sinker fell for all of these various prompt, injection, AI, Gateway, AI, Red team, whatever. There's a whole suite of people making a whole lot of money out of venture capitalists building these things. And to some extent they appear to work and they've got a flashy UI and blah, blah. But um, like it's just, it's better than doing nothing. But I feel like, you know, it's a bit like a. Well, the uk, I'd say it's like a chocolate fire guard. Basically. That's what it appears to be.

Speaker A: So what American.

Speaker B: A chocolate fire guard. So a fire guard is like a piece of metal that you put in front of your fireplace so that the sparks don't come into your house.

Speaker A: Oh, okay, okay, I get it.

Speaker B: I could have said a chocolate teapot. That probably is, Makes more sense. It's basically like a chocolate teapot. So what, what, apart from like offering people that they do some threat modeling, you know, for the people whose the horse has bolted. Like what, what's the, what's it? What's a. What's. Basic advice, you know, what's what. What's like real stuff that they can do where they're not going to waste money.

Speaker A: Oh, um, you got a threat model, man, that's it. That's it. That's all you can do. You, you guys, you guys, the horse is out. If you want to put it like that. I would put it a little less delicately, personally, but you guys, you're going to have to do the threat model or else you're going to be liable when this thing breaks. Now consider your use case. If it's an internal chat bot or something, maybe you have, uh, a limited blast radius for what can go wrong when you start deploying agents, when you start deploying something publicly, uh, you're effectively giving it hands and mouth to speak to the world. So you're gonna be responsible for that. And ah, if I were you, I would take that Responsibility. Seriously, because, um, governments are already showing that they are.

Speaker B: Well, uh, I seem to see something in the news literally on a daily basis about something atrocious that a chatbot has said or some ridiculous 99% discount on giving a hotel room or whatever. Basically like a non stop reel of um, of stupid things that people have, uh, AI has basically done because people have tricked it.

Speaker A: That's all it ever does is it tells people to kill themselves and then like gives them incredible discounts. Like those are the two use cases. I don't know.

Speaker B: Yeah, uh, well, I mean, uh, they're like machines that are trained on 10 years of Reddit posts. You know, they're, I could only imagine they're highly sarcastic and all kinds of things, so.

Speaker A: Oh God, it's way worse than that. It's way worse. I won't. This is my, my soapbox. That's very depressing. But the things are trained on way worse than Reddit. They're trained on the open Internet. Like Reddit is nice compared to some of the things these things have seen. That's why they have the reinforcement learning with human feedback. And that's why the people that do that get ptsd. These are not nice machines and they must literally be trained to not be totally evil.

Speaker B: Yeah, I have seen like a mini documentary about that because there's basically paid by the penny kind of our workers out in wherever, uh, name a sort of third world country where it's cheap for the labor and they just have to sit there and go through these obscene responses and filter the results. And yeah, this is like, uh, well, yeah, people are getting sick, but yeah, humans are getting literally sick training these things because they're so vile. What comes out of. Yep, um, definitely I would like to go, I would love to go down that rabbit hole, but maybe it's a bit too depressing.

Speaker A: It's a little sad. I know I told you nobody wants to hear this because I'd say it and people are like, wow, that sucks. Anyways, what can we do to see.

Speaker B: Yeah, well, you know, we live in a very, very strange industry because all of the, you know, I'm chatting on a MacBook and a whatever Philips screen full of microchips that probably were artisanally mined by some poor bastard out in the Democratic Republic of Congo. And that's a, that's a whole massive shit show also. So we, we, we're just like stacking bad karma on top of bad karma basically to build a super AI bad karma machine. But that's like a uh, that's a spiritual vent. That is definitely probably not going to resonate too much with the tech audience, but who knows, maybe they will.

Speaker A: Let's go down that rabbit hole. Let's construct the Tower of Babel. Why not?

Speaker B: Great. Let there. Well, basically guys, it looks like this karma is a real thing. You have a soul and what you do to the planet is going to come back to get you. So if you build everything out of bad things that's crushing loads of people and then you build other stuff on top of that, well, surprise, surprise, the end result is a bad thing. So that's where we're, that's where we're heading. Super AI Evil. End of game machine.

Speaker A: We sound old again. Also, you kids need to get off of my lawn too. Another thing.

Speaker B: Yeah, perfect. Yeah, my daffer deals are supposed to be coming up there. Stop playing board games, etc. Um, so I'll try and like pull us slightly back in. So horses bolted. These poor people, they need to do threat modeling. What about the people who actively knew that this was a, uh, snake oil operation? And we can, and we can't leave this in. It's totally up to you because I know you. Basically some interesting stuff came out because you were talking about it and saying, hey, this is, this whole thing is snake oil. And people came for you and said, you better stop saying that. Snake oil. And then it turned out that it was snake oil and you were right and you did a very long post on LinkedIn. It was basically like, I was right and you were wrong. Uh, give us a breakdown of that and what sort of. Well, if you're allowed to. What sort of nonsense came your way because of that and from what kinds of people?

Speaker A: Yeah, so I have to be very, very careful what I say here. Um, a lot of money was being made in this and I will go ahead and confess that I was dumb and did not realize what they were doing because I thought, okay, everyone knows this, right? I'm gonna publish this first paper in Defense 2022. Let's get on it, guys. And um, I just assumed, okay, AI Red teamers. These guys are on it. Right. I didn't think about it further because I was, you know, working on the EUAI act and a bunch of other stuff. So in the course of doing that, it came out and came to my attention because I'm on the team that sets red teaming requirements. Um, what the state of the art was and what these people were doing. And my head absolutely caught fire. So the first thing I did Obviously was start going to people privately because, like, again, I'm so dumb. I was like, you guys did, you know, like, I thought maybe they just didn't realize or something. Like, you know, this is not real. Right. And, um, time after time, I was told that we don't care as long as people pay for it. Um, we're trying to build a business here. It doesn't matter to us. A few people trying to, like, I guess, neg me into join. Yeah, I know I'm looking at your face. They did that verbatim. Verbatim. I was told that by a CEO. I won't name the name. But anyway, anyway, like you said, people,

Speaker B: I. I can't believe it. I can't believe that people would just be in an industry to make money without caring. That, uh, is such a shock to the capitalist mentality. Outrageous.

Speaker A: I know. I too, was murdered in my feelings by that.

Speaker B: It was the shattering moment.

Speaker A: Yeah, I grew up a lot that day.

Speaker B: The Alice in Wonderland moment. So basically, they told you to ignore it and shut up because we're making cash money. Then what?

Speaker A: Yeah, or worse. Some of them tried to, like, neg me into, like, telling them how it worked. And some of them tried. I. People were mean, like, really mean. And I eventually was like, okay, well, I have to do something about this. Um, so I ended up writing a couple of posts about it and in my newsletter. And, um, uh, that's when I started getting threats. Like, I actually said something to a couple of people publicly because at this point I was getting pretty frustrated with this, because I'm watching, I don't know even how much money has been made off of people charging enterprise for red, uh, teaming services. I know for a fact that these guys selling courses out here are like 2000, 1500 to $3000 ahead, basically. And you pack a course with like 20 people. You do this multiple times a year. You are raking in cash off of these poor people who think they're learning a skill and none of it's real. And you're sending them out into industry to do what? To do worse than nothing. And then I had to say, not only are you not just doing nothing, you're not just charging for nothing, you're publishing these repos of prompts which make it easier for real hackers to do real attacks. So anyone, any system that you've read, teams that any logo you stuck on your stupid site is now vulnerable. So, as you can see, I feel pretty passionately about this. And, um, yeah, that was the point at Which I got threatened. That was the point at which, uh, an organization reached out to me and was like, you got to shut up about this or else. And I won't say what the or else was. Um, but it was bad and it was in writing. And, uh, I couldn't believe it. I was absolutely shocked. And then that's when I published. I wrote 15,000 words about it and dropped six or five different, um, volumes on my newsletter because I was like, don't threaten me. Don't threaten me. That's how you get 15,000 words written about you.

Speaker B: Sounds like one of those situations where you also need to state openly that you're not feeling suicidal and you've got no, no plans to end your life and you're quite happy. I don't know. It sounds very dark, basically.

Speaker A: Yeah, yeah. I've been a little bit on the edge here, and I will, I will. To your point, I will, uh, affirm that statement. I am not suicidal. I have no plans to disappear. I'm in Washington, D.C. i plan to stay here loud and proud. So let's just make that known.

Speaker B: Well, hats off to you. It's very difficult to be, uh. It's very difficult to have, like, a, uh, north compass, a moral north, when money's involved and especially where a job is involved. You know, uh, I won't go into, like, ins and outs, but I know I've definitely done things that weren't totally aligned with, with my north compass due to the fact that I was employed with a certain company or my role was to, you know, you're in a certain role, you don't rock the boat if you don't want to get sacked, basically. So it takes, I was going to say it takes some balls, but you're a woman, so it doesn't take balls. It takes a lot of courage to do that. So for that, chpa, indeed takes a lot of chutchpah for that. Uh, and we definitely salute you for that. Or I definitely salute you for that. And, and, and it's pretty awful that you, that you're basically. You know, it's. It's classic whistleblower mentality. The whistleblower comes out and, and they go after the whistleblower, basically. Um, and I can't imagine how much money is it. I mean, you talk about the training courses, which is, yeah, okay, that's like, definitely snake oil. But these vc, the guys making startup applications and software, the amount of money they're taking from VCs, and they regularly announce, Yay, we did 25 million funding round B, you know, 50 million funding round C. And those. Those tools are doing exactly what you said.

Speaker A: Yeah.

Speaker B: So very, very murky waters.

Speaker A: Um, I'm gonna be real with you. This is not financial advice. Asterisk. Asterisk. But like, I. My number one, um, one of my largest demographics of readers is investors, um, on my newsletter. Uh, and because I've had so many people that have started coming to me more and more and being like, hey, is this real, Shosh? And I'm like, yeah, that is vaporware, my friends. If you want to invest in it, hoping it gets acquired by someone stupider, that's your business, not mine. But that is vaporware. So there's a lot of it going around and you hate to see it.

Speaker B: Um, definitely interested. Should I buy silver? Gold.

Speaker A: Everyone is big on gold right now. YouTube told me so.

Speaker B: You can't go wrong with that gold.

Speaker A: And the green drink. You gotta get some of the green drink.

Speaker B: Well, AG1.

Speaker A: Is that what it is?

Speaker B: I don't know. What's the green drink?

Speaker A: I don't know. All the podcast.

Speaker B: Uh, yeah, it's Ag. Athletic Greens. Ag1.

Speaker A: Oh, M. My bad.

Speaker B: Uh, pulling us back. And we go way back to actually talking about building the boundaries of these things. So I'm sorry, I forget the name of your colleague you mentioned who's building the ways to find the boundaries in a mathematical way.

Speaker A: That was me with my colleague, Nicholas Spunzel, and he provided the, um. So basically, it's this technique called centered kernel alignment, and it's a means of measuring the similarity of two neural networks. Um, we found that there's potentially other applications. Ah. Nicholas's work demonstrated that you can use it to measure these things, and it does actually, in fact, provide a good measure of similarity, um, when you're evaluating attack transferability. So this is kind of a major breakthrough. And then, um, in our paper together, we were able to apply that to say, okay, here's a method you can use to find the actual boundaries of these subspaces, like the boundaries of this box. So, um, anyway, sorry, that's a little bit more of the background on how that works.

Speaker B: No, that's very cool. So now that we've got the boundaries, what does that mean in real world terms? Um, does it mean that you can see something even if you don't know what it is? You can analyze it and say it falls within the boundary. If you know the boundary space, can you know something's inside the boundary? Does that actually help to solve the problem in the future?

Speaker A: So this is not something I've really talked about much publicly before, but one of the big methodologies, uh, for attacking AI in the wild is to construct a surrogate model first. Um, if I were to attack AI, that would be my first stop. I would make a surrogate model of anything that I wanted to attack. I would craft attacks against that model. Those attacks will most likely transfer. So what we're talking about here is attack transferability. Um, and attackers in the wild don't need your model, they don't need your data, they don't need any of that. They need to understand the domain that your model is trained on in order to construct these attacks. So you can see why every model is very vulnerable. If we can hypothetically, and this is where I'd love to see research go in the future, uh, begin to map out the boundaries of these subspaces. Right now they're very, very large. Um, but I think we could begin to understand how they overlap with, we could begin to understand and have some idea of attack transferability. Right? So if I know that, um, the more similar a model is to my model, the more likely attacks are to transfer, then I can start testing different models and their attacks, right? So I can test one that is very, very dissimilar to my model and see if the attacks will transfer. If they do, I know. Oh my God. Okay, that's a boundary. A model that is very dissimilar to mine will still have a tax transfer. Move the boundary out when we're, start when we're crafting these attacks. Um, and in that way we can start to maybe map this a little bit. Now keep in mind it's high dimensional space. Um, so it's a challenging problem. But, uh, the intention here is to begin to understand what attacks will transfer. Uh, and what are the implications of that for my system.

Speaker B: I can't even imagine how that will apply in the real world. I'm too thick for that information to go in, but beautifully presented. Uh, we definitely hit the bar of where I was able to follow you in the conversation. Uh, uh, what's the. If you had to throw a magical number at the problem or. Well, first of all, is it solvable ever? Basically, because initially the problem, the speaking to you and getting the feedback, it was like, this is a, this is a mathematical flaw in the systems. It's just there, it's not solvable. And then you talk about surrogate models and high dimensional space and looking at attack models and different things. And then to the layman, which I Definitely am. You start thinking this sounds like she's onto something. And actually maybe it is solvable after all. But the what is there a time frame on it if something magical like this can actually solve the problem?

Speaker A: Oh, I love to hear you say that. Yeah, it is solvable. That's why I'm not, I mean I'm not actually an AI whatever pessimist or anything. You just have to do it, right? Like I just want this industry to adopt real engineering standards instead of treating everything like it's silly pretend sci fi, like we're making real systems, let's act like adults, let's make them real. So yeah, I agree with you. I don't think you're wrong at all. I think the problem is solvable, but it's not using the old toolkit, right? And it's not solvable if we pretend like it doesn't exist. So when we combine um, these ideas of threat modeling for our system designed for test, right? So we design systems for testability. We threat model so we know what to test and then our testing is focused and efficient. We can also test to understand what attacks will transfer. To see how easy this is going to be for hackers. We put all of these things into uh, let's call it a security portfolio. And this goes into our model monitoring. So whether we're monitoring a model itself, uh, or an agentix system that's being deployed, we should have in place a certain set of thresholds, right? Like these are the ranges in which everything is normal. So I will make up a magic number. This is a meaningless number. But for demonstration purposes, bear with me. Let's say I have a ratio of red blots, red dots to blue dots going through my system, right? And if my ratio of red dots to blue dots becomes um, 27% higher for the red ones, then I know because of my predetermined thresholds, oh, we're under attack. Oh, something's going wrong in my system. So because of that I'm able to trigger, say, human in the loop, human review, um, some type of model review, some other type of data engineering system to remedy this. That's how we deploy AI securely.

Speaker B: My favorite part of that was human in the loop because it's just that solves all the problems of people losing their jobs to AI. They can just be humans in the loop.

Speaker A: Yeah, we would need a lot of them.

Speaker B: Well, that's all right. There's going to be a lot of lawyers and consultants at work soon. I mean, you think so?

Speaker A: I don't think so.

Speaker B: Look at all those. What about McKinsey? McKinsey is just, like, dropped off a cliff like that because all they did was stuff that AI does automatically for them now. So we can put all the McKinsey people and, um, we can put them in AI in the loops, and then they can just be AI monitors. I don't know. Maybe that's just my personal feeling on people who work for McKinsey.

Speaker A: I'm not an economist, so I'll yield to your expertise here.

Speaker B: Well, super cool. But it's. It's a nice way to sort of come to the close of the podcast because at least it has a, uh, light at the end of the tunnel, positive feel about it, where it's not because I don't believe in being an AI doom and gloomer either. I've never. There's so many. So many reasons that people think it's the apocalypse and this and that and the other. I just don't believe it whatsoever. And I think the statement, you know, we just need to be adults in the room, I think that's it. Basically, instead of just like, releasing these power toys like children, it's like having a moment of being adults and then using them correctly. So very cool. Um, anything you want to get off your chest that we didn't get off your chest is part of this.

Speaker A: Oh, man. Gosh. All types of stuff. Uh, no. What I really want, uh, people to do is take the message seriously that AI is serious engineering. It requires serious engineering. Um, it's time to take it seriously. It's going to become very expensive to not take it seriously very soon, and I would really rather people listen to me about that now than me have to say, I told you so later, because I don't actually enjoy that. I would rather see everything work.

Speaker B: Um, maybe you enjoy that I told you so a little bit. I don't know.

Speaker A: But by the time you make me have to say it, yeah, I enjoy it. I'm not gonna lie.

Speaker B: So we always like to ask every person the DeLorean question. I'll say that again. The DeLorean question. Um, which is, if you could go back in time to when you first started in this space or just in your professional career and give yourself one piece of advice, what would it be?

Speaker A: M. Don't give up. Be meaner faster.

Speaker B: Be meaner faster. That's a new one. I like that.

Speaker A: Thank you.

Speaker B: Um, and then we have five quick fire questions, which are pretty fun to ask.

Speaker A: Okay. Are there rules that I need to know to Answer this. Or can I just go?

Speaker B: Uh, I mean, they're supposed to be quick fire, so it's the thing that comes to your brain. So try not to think too much about it.

Speaker A: No thinking. Got it. A Candia.

Speaker B: No, no thinking. One tool or app you can't live without.

Speaker A: I don't. I don't have any of those. I don't care about tools or apps.

Speaker B: One tool or app that you hate?

Speaker A: Uh, all of them. I hate all of them. I think that they were all a mistake. Except grep. Grep is good.

Speaker B: Grep is good. You know, I once, uh, met a guy who was heavily involved in helping build Linux kernels. And he was such a super nerd. And I was like, I was quite young at the time. I was like, what's your favorite Linux command? Or what's the most. What's the best thing about computers? And his answer was grep, actually.

Speaker A: Yes. That's a sensible person. Thank you.

Speaker B: Yeah.

Speaker A: Yeah.

Speaker B: Well, someone who programs Linux kernels tends to be pretty sensible. They're not, you know, not too rock and roll. Uh, what keeps you up at night? Professionally or unprofessionally?

Speaker A: Uh, definitely thinking about influence operations and how AI can be used adversarially. Not just adversarial attacks against AI. Um, from a national security perspective, that is an evergreen problem, and I need everyone to be more serious.

Speaker B: Got it.

Speaker A: Sorry, that's long. But that's.

Speaker B: You could be as long. You could be as long as you like. Can you share the worst ever experience of your career?

Speaker A: Um, probably getting threatened over talking about this. I don't know, man. It's been rough out there. That's why I would. I say to girls that are coming into this space, be meaner, faster, I got your back. Because, like, it's rough out here. I could. There's. I got stories that range from everything from people stealing my work to sexual harassment to on and on and on and on and on. And you just, you take it, you update your priors, you move along, you keep it rolling. Um, but yeah, it's hard to say at this point.

Speaker B: Well, definitely this is a side pitch, but we've also done a couple of podcasts with some, uh, like, influential CISOs and stuff in the UK. One of the topics we came back to quite a few times was women in tech. The lack of women in tech, the problems that arise with women trying to come into tech. I didn't want to touch on it too much to assume that there were problems. I don't know. And, you know, the Problem is middle aged white men and I'm a middle aged white man. So it's some somehow whatever I'm it basically. But I'm uh, not it also.

Speaker A: But I don't know, I don't want to blame men for this problem. I don't actually think that's the case. Um, I, I do think, I think that the problem is systemic, which means that it doesn't necessarily need conscious maintainers. Like I don't think there's like a cabal of dudes sitting around like planning, well, we've, we're going to make sure that Shoshana has a hard time. I don't think that happens. But there are micro decisions and there are systems that are set in place. And yeah, you come to like, I'm not, I, I, I not a, um, cry about things that happen to me kind of person, but any woman that's been in this space for more than five minutes is like, yeah, the sexism turned out to be real.

Speaker B: I mean it's just, it's just the only, it's the only feedback that every, every woman agrees with basically. And loads of, loads of good women leave the space because of it. Like smart, uh, I've worked with very smart women. They said they couldn't work in an XYZ department anymore and then they left. And it's kind of the churn is unbelievable. I would like uh, as and particularly if you're high profile, I think I would consider you, I don't know, quite high profile. But you know, with that also comes more pushback. Did you, did you, Was it, was it was it was.

Speaker A: Have you seen my comments?

Speaker B: Yeah, I have seen your comments, but in particularly. So we're going right back to when you were talking about raising these problems initially and you were like a bit naive about the fact that they knew. And uh, what difference do you think it would have made if you were a dude bringing up those problems? How different do you think your response would have been or, or the feedback or anything? Does that play into it? I feel like it must have done.

Speaker A: In my opinion at uh, the first time I brought it up. I would have been immediately hired and handed a fat salary, in my opinion. Yeah, that's how I feel. And I'm going to tell you this. I've been uh, I've had every role in the data stack up to Chief Data Officer as a red team lead. I've interviewed men, I've been on teams of men interviewing men and I've seen the different ways that men get Interviewed versus how, uh, I get interviewed. And I'm telling you what, man, it's different. And again, I'm not going to sit here and complain because this made me better. I'm better at, uh, everything because it was all harder for me. But like, it's different. So, yeah, I have to think that if I had been a dude and I had come and you know, talked to these people about it, uh, it would have been received very differently. But that's just my opinion.

Speaker B: No, I mean, it totally resonates. We had, uh, we had a, we had, we talked about it a good number of times on the podcast, to be honest. Uh, including one girl who's the head of DevOps, who. Someone started like cracking onto her in an interview, like chatting her up in the interview. And I was just like, it's so fucking ridiculous. It's unbelievable.

Speaker A: But I had a. There's a very well known gentleman in the space. I won't say more because people know who he is, that, uh, asked me out to a, ah, resume review, um, bought me dinner and this, I was struggling at this time. Like, this was, this was a while back. Instead of reviewing my resume, he propositioned me. And in the course of doing so, I guess he didn't realize that I was Jewish. He went off on like a giant anti Semitic grant. So like, what a night for me. I tell you what, no resume review happened. Uh, and I got to experience all that.

Speaker B: Great times, great time. Out of pure interest, was the guy so autistic that he couldn't pick up on body language, that you were being repulsed? I mean, having, having, being anti, like the whole thing, you know, I'm gonna

Speaker A: be real with you. I don't think they care. I don't think they care because I'm not buying. I'm autistic. And I can tell when someone doesn't want to talk about something like, okay, you. I don't think they care. But again, we're getting into this is my opinion. And again, I don't blame men. Like, I'm not like, oh, men are the, the devil here or anything, but I do think there's a system that rewards certain kinds of behaviors and punishes others. And that's is based on my observation. Um, and I think that it's hurting the tech industry because it cuts out a lot of really talented and um, diverse viewpoints. Not diverse for diversity sake, but diverse technologically.

Speaker B: We do. And, and it's totally true. And uh, weirdly, not weirdly factually, there, there are places that don't have this problem. One of them is Israel, where I work with a lot of Israeli tech companies and it's about 50 women in maybe, uh, not 50, but a very much higher percentage than is in Europe generally. Uh, and I, I'm sure it's because of the military, because the military just, uh, trains people equally and then churns them out and then everybody arrives with a, with a similar sort of mindset and that's how it goes. But once you get into the professional workplace, it's different there. It is different. I, it's just different, basically. So it's not a matter that women can't live in the tech world, because Israel is a perfect example of where there are loads of women in very high tech positions who are super smart and it's not a problem or they don't need to leave. There isn't this high churn. But, well, uh, elsewhere you have to be a badass like you that takes the challenge and then, you know, sticks the finger up and has to, like, trawl through the mud to become a lettuce flower and pop out the other side.

Speaker A: That's nice. I have to say that I'm sorry to, but it's very important to me to say this. I would not be where I am if men had not stood up for me and helped me and whatever else. So there have been a ton of good dudes that I'm not. Don't make the perfect, the enemy, the good, you know what I mean, who have been like, hey, let her talk, or like, boosted my thing when I needed it and so forth. So the reason I bring that up is just be one of those dudes. Be one of those dudes for a lady, and that's all you gotta do.

Speaker B: Yeah. Very sound advice. Uh, I don't know how, uh, how many deaf ears that falls on. Uh, we have got a couple of minutes left. I'll go back to the final speed question.

Speaker A: Oh, sorry.

Speaker B: If you weren't working in tech, what career would you choose?

Speaker A: Whoa. Probably policy. I'd be in policy. But if I wasn't to do any of that, I would be in music. Um, so I guess I am kind of in policy. So, yeah, I would love to be performing and doing music stuff on stage. Super fun.

Speaker B: Very cool. Um, that's it. We come to the close. Totally awesome chatting to you. It was one of my favorite episodes. I really thank you for giving us the time. Totally loved it. Any parting words?

Speaker A: I had a wonderful time. Thank you so much for having me. This was a minute in the making. And I'm glad you pinged me again and. Uh-huh. Had the patience and forbearance to keep trying. Thank you.

Speaker B: That's it for today. Thank you all for listening. If you're ready to rethink your cloud practices and cyber security strategy, then the team and I at Global Dots are at your disposal. We've been doing it for over 20 years. It's what we do. And if I don't say so myself, we do it pretty well. So have a word with the experts, don't be shy, and remember that conversations are always for free. I'm Ganesh the Awesome, and this episode was produced and edited by Toma Mulvidson, sound editing and mix by Bren Russell. Stay tuned for more episodes.