From Blue Team Challenges to AI Innovations: A Conversation with Jason Haddix

I really feel that input validation is a losing game with AI because natural language has so many, like a natural language system has so many bypasses. And like, even if you block all special characters, like some of the techniques, you can literally just describe them with words. And so like there's always gonna be a creative bypass in this type of system. Simply Defensive brings you the industry's top practitioners, innovators, and leaders to inform, educate, and join us defensive.

Hello, and welcome to the latest episode of Simply Defensive. I'm Josh Mason, the only person here who hasn't keynoted a Wild West Hacking Fest, and with me as always. Wait, what What's up? Was last year.

All right. And it was the, closing keynote. I don't know if that counts or so it does. does it.

Honestly, it was the better of the two. Jerry's pretty upset about it. It was pretty damn good. Uh, and with us today, uh, our good friend Jason Haddix.

Jason, thanks for joining us, man. Hey, thanks for having me. And I feel like you need an introduction if you all don't know who Jason is and you're listening to us, like choices, life choices so Jason you you're running arc anum. For those who don't know you, you've been the siz o at UB soft.

You've been a bug crowd. You're doing stuff with flare. And you, in my opinion, are one of the thought leaders in the AI cybersecurity space. Just all Around.

you seem to have, a good grasp of how to attack it, how to use it. Wade, you have taken Jason's class on using ai. yeah. I took the Red Blue Purple AI course, I wanna say maybe two years.

There's a second cohort that was going through. I think. been a year and a half ago. Yeah.

Yeah. I saw you on Jerry's. It's on what is the fireside chat? And I was like, okay, I need, 'cause I had been playing around with AI for a while, but I hadn't really dived into it yet.

And honestly the course just skyrocketed my experience like to hear. ridiculous. And it I came back and had instant like use, I'm still using the stuff today at my current org and I am signed up for the course again. And so gonna be another one in December.

We don't know when this is gonna air, but that's, to say. I do wanna shout out one thing though, in that class way back then, you were talking shit about CTI and Little bit. A little, and now you're a field CISO for flare, so I just thought that was hilarious. But.

So I think the con, I think in that class, the conversation was that I've had a lot of different hats and I've had to manage a lot of different groups inside of security it was always hard as a leader. To prove value of CTI. Not that I didn't think that CTI was valuable. It's just hard for them to prove what we're doing every day is cost effective as opposed to other groups in security, which give you an artifact pretty much after they do anything right, like pen Or And I think CTI and threat hunting suffer from this both a little bit.

And sometimes your CTI people are your threat hunters. It depends how big your organization is Right? some teams wear many hats. But I think that was one of the things it's it was just like I remember.

I remember looking at some investigations and then some threat hunts and then looking at artifacts and it just wasn't where I wanted it to be. And that could have just been my personal experience. It's every, place is different. But like I do think that there are some very pointed, kind of components of CTI that are really valuable.

So the cyber I think that. The credential kind of leakage problem. yeah. Is one of the biggest ones.

If you look at any fucking breach, oh, sorry. No, we can cuss. Go for it. You're fine.

You're good. Yeah, Yeah. If you look at, if We're grownups. Across the industry it's like something that's 85% of 'em are started by some sort of either leak credential or someone's been part of a different breach, password reuse or something like that.

And so like that whole part of. is really important is to first of all, identify which accounts have been leaked, how they have been leaked, what the mitigation is going to be for that. Whether you're going to roll a credential, whether you're going to reset a password, whether you have to reformat a whole goddamn machine. 'cause there's malware on that machine, right?

Yeah. And I actually see that last one I don't see a lot of people talk about. Like when as part of Flare, I've learned a lot about the CTI underground, and when you get Steeler logs and you look at those and you're like, okay, an employee's laptop has been infected with actual Malware. that, that exists usually at like the system level, right?

Like on the kernel level, depending on they had an exploit or something like that. And so a lot of customers like, oh, we'll just roll the account. It's. No, you have to like like actually nuke that system and start from Okay.

And that, that's a harder discussion too, if it's a personal system, Oh yeah. someone's Yeah. that they logged into corporate resources with, how do you go tell a personal. Per oh, your, the laptop that you and your kid and your wife uses at home got malware on it because someone downloaded a crack for Fortnite or free B bucks or whatever, and I need you to reformat your reformat that box because it's it's got malware on it that disclosed corporate credentials or cookies or whatever.

Yeah. To your point to to give you a little more credit. So I did raise my hand up and I was like, oh, I'm the CTI instructor at Antis Siphon, and you did let me come on and defend CTI and then you, were, all for it. So Not more of just poking fun, but with, definitely with the.

Imaging stuff. Now, especially with more remote work, like I, when I, one of the big corporate overlords I used to work at, if there was any signs of malware, it wasn't even a question like, oh, we're gonna get, no, you just go get a new PC where we're wiping you complete. Nowadays, I don't think I've seen that since remote work at all, at any of the orgs I'm at. And it'd be much harder, like I've been nuked twice by antivirus 'cause of doing funny stuff and they have to send out a new Laptop.

I think Josh, was it you that I was talking to at Wild West and we, were talking about the struggle to even get machines at the beginning of COVID. Was that you that I No, but I, recognize that I saw that myself. There was a little while where we like we, I was a cso, UB Soft, and we, we need. needed machines for new employees.

And during COVID, during the first year and a half, Dell wouldn't even sell us anymore. And so what it meant was everybody went home to work and we had new employees still coming on board during COVID. And now for a little while it meant that we asked them to use their personal devices, Wow. were pre-owned.

Yeah. Yeah. Yeah, so it's, not an easy question with, that blip of three years, three and a half years too, that that happened to where everybody like moved to remote work. It's, an infinitely hard balance to, do.

Yeah. Oh, so much I was a government contractor at the time and there was so much of, you definitely can't log in from your personal and to do stuff. That was a real difficult thing to do. Yeah.

Yeah. I wonder, I never even looked at, so I wondered if the VDI market like spiked then, right? Oh yeah. Yes, thing we looked at.

Yeah. a hundred percent because that was our we solution. video. Oh, okay.

Yeah. We, I mean that we had so many employees that we had 22,000 employees, so the V, so any VDI solution that we would've pivoted to first of all would've taken forever to implement. And then second of all, would've been so expensive and crushed our security budget and our IT budget. And And then you gotta hope those, it out.

you gotta hope those VDIs are then set up properly, right? There's management and like yeah. and all that Use a different golden image and you're like, wait a second, Yeah. you're not supposed to have access to that.

Yeah. Exactly. Yeah. Yeah, let's pray that someone doesn't put a hard coded admin account on there and it's got a hash and then some low privileged user just pulls that sucker out there and Yeah.

the uh, instructors were happy because finally we had computers that we could mess with just for fun, that had Cali because we were using vSphere, and they just spun, or they just turned around and used the golden images that we had for classes. Just took those pools and just aot of them to us for usage to then log into the courses To allot them to students. So it was it was, I don't know. I wasn't in charge of it.

I wasn't the vSphere admin, thank Goodness. yeah. But that was a mess. All right.

I have a good AI question for you. And when we discussed here before. So me as like a blue teamer, right? I've seen like the, Soar come up, right?

And everyone's we're gonna automate everything. Everyone's gonna leave. No one's gonna have a job 'cause we're gonna be able to do it. I felt like in my, career, people did use Soar, but everyone was still scared to do like manual pushes and, blocks and stuff like that.

Like I'd never seen someone go full bore with it. Okay. I feel like people are thinking they're gonna do the same thing with AI now. But if like people didn't even use SOAR for that, why are they gonna trust AI more?

Does that make sense? Alright. I think that, in the first two cohorts of attack of of Red Blue Purple ai which people don't know what that is. It's a course that we run we spend one day getting you from zero to hero to understanding the current kind of models and ways that you can consume those models.

And then we spend a day talking about how to apply it to security roles. And and we look at some of the more cutting edge. Projects out there that are already using AI for blue teams, red teams, and purple teams. And so I feel like in that the the first few cohorts, I was very bullish on that it would happen very fast that we would.

We would automate some of the detection work, some of the glue that puts those systems into other change management systems and asset systems and all this kinds of stuff. And it would really happen very quickly. Now since then, we have been like doing some consulting, going to organizations. As part of this new random service that we built called the an AI scaling Assessment where we go into a business and we're like, let's look at everything you're doing in security from a security point of view.

And then I will, basically consult with them and say, here's where you can use AI for these teams. what I have started to learn over the course of doing these assessments is that, organizations move gally slow for change like this. And so I knew that from being a CISO in certain orgs, but it's even more apparent when you work with companies in the finance sector or in the healthcare sector or the automotive sector, which some of our, which are some of our biggest customers. And so Like it, it just, I have had to slow my role a little bit and be like, cool.

Although the capability exists for you to do some of this cutting edge stuff, like using MCP to use natural language to execute investigations or to build custom dashboards or all this amazing stuff, it all exists. With a couple of with a couple of vendors and some open source stuff, it's in order to get that into any of these organizations, it's gonna take a year and a half, pretty much. Yeah. at minimum.

And so that was something I've been learning over my time with AI is that even if the tech is there, I. Sometimes the businesses are not ready to move, the security teams are not ready to move. And that, that's just the technology portion of it, like the implementation. There is also people inside of these organizations who are anti ai.

And I've had a lot of this in my interviews with these security teams and they're like, I. They don't, first of all, they have some latent fear of being replaced. Which is normal. It's human.

And that's totally a thing. And then also they've had bad interactions with their usage of AI where they, no one had really trained them how to use any of these models really well. So they tried to use it for something and it didn't work well, like the first two times. And then they were like, F ai, is so junk.

Like it's never gonna take the place of anybody. And then you have other people who, Just like they inside of political, inside of political organizations, they try to protect their fiefdom from change because they have a well-oiled machine in operations or in the blue team or in the sock or whatever. And they finally have just gotten their product that they wanted in there. They finished all that onboarding and they don't want to change any part of that 'cause it basically gives them mental anguish to think about doing any change.

Yeah. I recently, okay, so I'm at a newer org. I meant one password. I've been there for almost coming up a little over six months.

Yeah. Yeah. Thank you. Yeah.

It's pretty fun. We're right off the bat, so we're very project oriented project and like making issues and stuff like that. And for security, like I haven't seen. close of like more of a developer life cycle than anywhere else.

Yeah. right off the bat, my boss tells me is Hey, your issues are lacking. Like you need to be more verbose. You need to be better, like what you're gonna do with them.

And I'm like, oh, okay. didn't make new issues. I just made a, I made a, good bot that to I give it the idea, here's the template and then the, best part is he came back to me, he's dude, you're rocking it on issues. And I'm like, oh no, I made a bot for that.

Here, it is. And he is just give this to the team. Everyone's just gonna use this for issues now because everyone else. So stuff like that Yeah.

just I'm not seeing enough people that call out the little things that you can use it, that are Oh my gosh. We're huge on that Here. our big thing, man, over the summer was like, if you we have, you know, a, an enterprise AI for like everyone to use, like use that one for work. And, use it for as much as you can.

It's got all the features, so like that puts all the fences on it that I don't know, someone signed off On. and, uh, the CT like CTO and CEO signed off on it. And so it's got all the features, we've checked it out, like it's got the fences, so thus, use it. And for the small things, for the big things.

I ran deep research like 20 times this week On like crazy ideas and I'm, yeah. Because I got, I'm taking over like the company podcast as well, because cool. you can never have too many podcasts, in my opinion. Everywhere.

Just Josh, you're crazy. I'm bored and You talk to my wife more on signal than you talk to me. whoa, whoa, that's gonna, that's gonna be weird. That is not true.

Hi, Julia. You're great. and Julia hi. Yeah.

That does sound really weird. You gotta admit. Yeah. The autistic in me is just like that.

I don't know how to respond to this. Okay. for everyone who's listening, Jason and Julia are really awesome and, uh. Yeah, we're all good friends.

But Jason's really freaking busy and so Julia handles scheduling anyways. Um, okay. Whew, way to ask something. Oh man.

Man, ask something. What do you think about I've been looking at all the good, like the new detection stuff that's been coming out more for ai, like detections.ai we had like their field ciso, Aaron on and stuff like that. What do you think about those for right now?

Do you think that's just natural progression? I think Yeah. I think that the I wrote an article on this a while back and it was like the, it was also from this experience of going into orgs and talking to 'em about ai and I feel like there's there's an adoption. Ladder or something like that, whatever you want to call it.

Stages of adoption and they will be blocked by different things. But the stages are, really clear, at least in my mind. And it's okay, so I mean you remember when you took Red, blue, purple AI and the first thing we talk about is just building A GPT Yeah. how important.

System prompts are right. And we teach a custom prompt engineering methodology in that class. And I've talked about it in talks before, but I really haven't released it or anything other than in the class. But but prompting is really important.

So like most orgs they have access to, like what Josh was saying was like, okay, we buy access to a co to, or we have Microsoft and we've turned on copilot everybody so everybody can make Copilots. So the first kind of adoption is building what we call custom bots, right? Which is just adding a system prompt to do a task or to build a bot to do a single task in copilot. And so that's like, level one adoption.

Like at the end tail level, one odd option you might add rag to it, right? Where you add custom documents or data sources to that bot. So it can basically have answers to questions that are not in the training data. And so that's at the end of your kind of like stage one journey.

And then you move on to stage two where you do things like agents where you break out a whole system which has access to individual little bots that do individual things and bring them back into a plan. and I think right now the general industry at large is still at the tail end of probably that first stage where you know, people are starting to learn, oh damn, like AI is now good enough to write detection, engineering rules in my favorite tool, or help me build signatures or even help me like make my EDR better by like doing custom stuff that was never available to me.

I see this in the red team side all the time. It's like I see red teams who didn't have the money to. As part of their team, have a tool developer who did custom tooling or build custom phishing frameworks or do all this stuff that they just, they couldn't do it before. And now with ai, they're able to do a lot of this stuff which is, really cool.

Which often requires a lot of knowledge of some C two or whatever. And yeah, so I see a lot of people at that end stage right now in, in the industry. I haven't seen as much talk about or at least in the blue team phase, like I don't feel enough people talk about like how essential using rag. Is in order to like really build yourself up.

Yeah. I've, I have pretty much profiles of my entire company. Our entire tool sack, our logging, how, what's the fields in those logs provided that to a bot. And now when you wanna build detection, it tells you exactly what logs, the query, the, and it's amazing that I I don't know why more people aren't doing it, or people at least explaining the customization at some of these like blue team talks, but.

I think that the domain has gone past prompt engineering to now what we call context engineering, right? Which includes rag, it's The whole context engineering thing is, it's a really simple idea. It's like how do you add information on top of the model? To make the purpose of your bots or your your API call or whatever.

Way better. One of the classes that we're building right now is not actually a security class. It is a class just on that. And it's Wow.

do context engineering. And it's like rag is one you said, right? And How to do really good prompt engineering. And what we did is we went out Stop.

at all of these AI first companies and eventually their system prompts get leaked on the internet and to GitHub. And so we started reverse engineering them and being like, okay, this is how they're calling tools reliably. This is how they're structuring order of operations for. The ai, this is how their agent architecture looks like.

This is how their rag looks like where they're pulling it, how they're pulling it. And then we also did that from the model vendors as too. So like the model vendors have their system prompt, right? There's two layers of system prompt that you know are in.

Anytime you use open ai, there's the. the one set by OpenAI, and then there's also the one set by the user, and then you know, the use and then the, or the developer, and then the user chats with it. All of the model vendors system prompts have also been leaked. So we started looking in there too and being like, okay, what are the best practices that they use to make these bots better?

And so it is I don't have any benchmark data or ever whatever, but after doing all that research and and implementing some of those things like rag I think in, even in the second cohort, we might have said examples of what you want output to look like is really important, like Yeah. examples. I think we, we said that in the course and and just like structured system prompting and like tool calling URLs, all this stuff and like metadata fields that you can add. It's like it's night and day difference.

It is I Yeah. just in my head, like a 20 percentile effectiveness, like increase from just. Asking a straight question to the model Oh yeah. Yeah.

Have you seen any other ones telling their bots to use methamphetamines? I have not for a little while. Yeah, so what he is referring to is one of the is one of the tricks in, I guess it was prompt engineering at the time, but there was some white paper studies about. Urgency prompting of which one was telling your AI that it was on methamphetamines.

And so that was one of the, what we called in the course, we called it like silly machine tricks, Ah. that we have in our methodology. I still use that today and I don't know if my bots that I published publicly would be as good as they are if they didn't have all those little tricks in them. Honestly.

And Yeah. couple new ones I think in the class that, that we have added but we have also removed a couple too. Yeah it's a evolving field. Sometimes the models get good enough, you don't need things like that anymore.

And it's hard to know when too, to really, to remove the silly machine tricks because the the you have to do a bunch of benchmarking in order to figure out, like if it's having an effect. And sometimes we don't have time to do that benchmarking, like right away, yeah, I have like my stupid trick with GPT five is think hard about that. So that it'll use the pro and then, tell it to go out and search the internet to find the, find examples so that it won't just like hallucinate on whatever it's trained and it'll go find proof.

Uh, when you were talking about your new course, I was like, oh man, I really want to do that. And then it hit me that, uh, I'm going to learn how to do that and I'm just going to use it to use a rag to work on my DD campaign. And. Oh, yeah.

Yeah. You don't even need a DM anymore. You just feed it all to the, bot, and then you're So and look. What's next?

Dude? No, Nah. It, oh. good enough.

It's good enough. You just drop, you go get a PDF of the, The Dungeons Master Handbook and drop it in. Yeah. Yeah.

one, one aspect that I have I, know is giddy is starting to be a little bit more a little bit more mainstream is like defending your prompts, right? And making sure people can't reverse engineer them. Yeah. to give an example we all know FedEx.

I have a well-known bot that does the Palantir's a DS framework and you to give it a detection, it completely fills out the whole framework for you. And then one day FedEx came to me, he is Hey, here's your exact prompt for this. And I'm like, not upset about it but should I be protecting this? And he's yeah, man.

This is your thoughts. This is, I'm like, ah, all right. What do That's a good point. Yeah.

At what point does a prompt become ip? yeah. So this has been a discussion since the beginning of. Of like basically the GPT store and and I think a lot of people originally thought, oh, the GPT store on OpenAI.

It it's basically a user sets up a system prompt and then publishes a bot. And so a lot of people have done it and originally they thought they were gonna monetize those bots. But think that prompt injection being so to use and there's so many bypasses to any security control, you even put in the prompt that, you know in when that simplistic model of like just a user interacting with one bot and there's no classifier or guardrail in the middle or anything, system, prompt level, detect protections are like the least effective out of everything basically.

And so there's always gonna be a way to. To dump the system prompt. And a lot of the times when you're thinking about defense of system prompts or just defending an AI system in general I think I tweeted about it this morning. It's like you have to shift away from your normal security mindset of like input, validation, because that's where we all go immediately, right?

Yeah, I was app a AI waf, right? And You're like, oh I'm gonna use I saw some guy no, no fault of his, he's a great researcher, but I saw him post on LinkedIn and he is he's this is just the same problem we had with web apps. And it's we're just gonna make sure that special characters can't go through a system or whatever. And so I was trying to think about oh, he comes from.

The web app world and like input input malicious input, detection and then output and coding are the the two big things that you learn when you're in web apps and, even other protocols when you're hacking other types of stuff. But I really feel that input validation. Is a losing game with AI because natural language has so many, like a natural language system has so many bypasses. And like even if you block all special characters, like some of the techniques, you can literally just describe them with words.

And so there's always gonna be a creative bypass in this type of system. And so really I feel like a lot of people need to move. Move towards like classifiers and guardrails in line with your AI system. And those are basically sentiment analysis to see on the output.

It's okay, does this look like a normal response? We usually give the user yes, no. Okay. Return it to them or Yes.

Return it to them. No, it has like. Random agent data, it has all this stuff, no drop, connection or whatever. And I think that is a mind shift that a lot of the security people are gonna have to make in, this era.

But of guardrails, classifiers, and prompt based protections, prompt based protections are the weakest. Usually you're able to bypass them. I still think they play a role in defense in depth. If you have all three of those and you are also doing several stages of routing for a user question does it satisfy these requirements?

Does it contain these keywords? Okay, then I'm gonna route it to the LLM instead of just directly to the LLM. If you have all four of those things. really hard to crack.

I've been up against a couple systems really recently that had all four of those things. So LM based routing that was contextual. A classifier, guardrail and system prompt defenses. And it was an extremely hard test.

Extremely hard yeah. Wow. I love the defense in depth reference. That's it.

Like I, I ha The only thing I have thought about that too is exactly what you said, like the web app, the waf just like monitoring that. But all those other parts that's, pretty crazy. Yeah I've got one of our researchers who I get to interview about a zero day that. He gets to publish on Tuesday and I'm, excited.

It's fixed. He was demoing it to me and all of a sudden like it stopped working. I was like, oh, sick. He's got videos of it working.

Good. Yeah. But it's pretty wild. Yeah, and it's good that it's fixed.

I was talking to someone the other day. I was telling him that one of our researchers on the team, he found like on one of the, I'm not gonna say which one, but one of the biggest model vendors out there. He found a mass assignment vulnerability, which is I think the first time in, an API chat completion or an AI chat completion. API I think has had a mass assignment vulnerability in that kind of structure.

And so I like to tell people we're still in the infancy kind of stage where we're finding bugs with these systems at every layer. And it'll be like that for the next probably year and a half, two years. And it's a great place to do research if you, like doing research and you wanna dive into something, there's a whole stack of stuff you could look at. You could look at the models, you could look at all of the.

Apps around the models you can look at prompt based detection, prompt based hacking prompt injection methods, like there's so much research going on right now. It is so cool to go to the conferences and see so many talks yeah. Where's. we're at a time.

Are do, You asked. First, you ask yours and then I'll wrap up with mine because it'll sound. we end the podcast on one question, whereas what's one piece of advice you'd give a blue teamer right now? Could be someone just starting out, someone with years of experience.

Doesn't matter. I guess it's to embrace this new tech, right? I think that I meet a lot of people who, you know on, whatever side blue or whatever, but who just are not ready to accept that this is a mainstream piece of technology that we're all gonna have to use. I, fear for people who are very.

to adopt it just as a tool. You don't need to make it your whole life, but understand that it is a powerful tool that can do small functions of your job really well and make you faster. I don't see many complete automations of people out of their jobs right now. That is the narrative, but I don't see many of 'em right now.

There's still human in the loop. over the place, but it does make good people fantastic. And so embrace the technology. Take there's hundreds of getting started with AI classes out there that are completely free YouTube channels.

Just learn how to use some of the front some of the frontier models. And then as like we do in the class, right? Take your job and think about what are the things I do day to day? And then rank them about like, how annoying they are to you.

It's like what, in this. like super annoying or requires me to like transpose data or like glue together two systems and then just start chipping away at those micro problems using AI and you'll become like a rockstar. You just. You just do, you build tools.

You have a bot that helps you do things and then you share it with your team. And that, and then when like later on we do get to some automation stuff or people are getting displaced maybe, which I don't think we're, it's gonna be like crazy, but if it does happen, you're the person they call. They're like, oh yeah, you made that bot, you're a, you're our AI guy on the blue team, right? So, embrace the technology, it's a tool.

Don't buy into the hype that it's gonna replace everybody and use it as, you Can. Love it. My question is last, wait, one last piece. okay.

Okay. watch yeah. Yeah. on YouTube.

Like Yeah. Yeah. Fire is a great content creator around tech and ai. Yeah.

When, you, I like from that class. I learned about him from that class and I've been watching it from then on, but. He's great. Yeah.

I love that. I love that. Where can people, uh, keep up with you? What's the best place to, uh, know where you're at, what you're doing?

So I am at J Haddix on Twitter. So like my personal ramblings and sometimes our company stuff goes there. But our website is ARKanum, A-R-C-A-N-U m-sec.com, and we have a blog there.

Where we talk about research, we also have some resources there. We have a GitHub where we have recently been pumping out tools and resource sheets for free for everyone. So if you find the AR, canem security ar canam information security GitHub, there's a bunch of resources on that now, especially on GitHub pages. So you can check us out there.

And then we have a newsletter too, which is it's a newsletter called Executive Offense. And usually if we have a big tool release or a resource cheat sheet or something we're releasing, we'll release it on the newsletter. So if you find the Executive Offense newsletter written by Jay Haddocks, then you, won't ever you'll always see what we're putting out there. Awesome.

Those will all be in the show notes. So if you can hear it or see this you can find them There. Jason, thank you so much for joining us. Uh, hope to see you soon.

Play some d and d, play some, uh, magic with you, uh, or at least have a drink and hang out. Thanks, Ben. Bye y'all. See you, Wade.