Episode 124: Microsoft Security Response Center for AI

Welcome to the Azure Security Podcast, where we discuss topics relating to security, privacy, reliability, and compliance on the Microsoft Cloud Platform. Hey everybody, welcome to episode 124. This week it's just me, Michael. Everyone else is busy this week.

And our guest this week is Raji, who's here to talk to us about the Microsoft Security Response Center AI, safety and security. This is certainly an area that's going to be very interesting. But before we get to Raji, I want to just talk about one little news item that I have, and I literally only have one news item. And that is, in fact, I really do wish Sarah was here because this is right up in her wheelhouse.

That is, we now have in general availability deployment safeguards for pod security in AKS, or more accurately, the pod security security standard. So it turns out, I didn't know this, but now I do. There are some pod security standards upon the Kubernetes website that have different profiles. There's privileged baseline and restricted.

And there's now abilities built into AKS that allow you to deploy these in a consistent way and to enforce pod security configurations across clusters. I understand the words. I kind of understand the security implications of it. But again, I really wish Sarah was here to go into a bit more detail because it's her thing.

All right, that's all I have in the news. So now let's turn our attention to our guest. As I mentioned, our guest this week is Raji, who's here to talk to us about MSRC AI safety and security. So Raji, welcome to the episode.

We'd like to take a moment and introduce yourself to our listeners. Absolutely. Thank you so much, Michael, for having me. Exciting to be here.

I've always been a big fan of the podcast, so really excited to be here, specifically talking about something really near and dear to my heart, the MSRC AI safety and security team. So basically, my role at MSRC is that I'm a principal engineering manager, and I have been with Microsoft completing three years in Feb of this year. It feels like a long time, but I've been here for three years. My team mainly is responsible for driving the end -to -end lifecycle for vulnerabilities response for Microsoft AI services and products.

So this is anything to do with Copilot, Copilot Studio, Consumer Copilot, M365 Copilot, all the different AI interfaces you deal with as you are working with Microsoft products. So my team actually, we also lead the engagement with a global AI research community. This spans both academia and industry. And prior to Microsoft, I have held other security leadership roles and worked in multiple domains of security at Apple and Adobe.

So yeah, it's fascinating for me because I've seen the transition to the cloud, multi -cloud, and now AI and AI agents. I just want to end by saying with great connectivity comes great risk. I would say with the ambiguity of an LLM comes great risk, but let's just leave it at that for the moment. I think it's great to see a specialized branch of the Microsoft Security Response Center coming into focus.

Obviously, it is an area that needs to be treated in its own way. So maybe worthwhile just spending a couple of minutes and just talking about the difference between the Microsoft Security Response Center, the classic MSRC, and MSRC AI. Has anybody been here three years? I've been here just a little bit longer than that.

Yeah, 33 years. But anyway, that's another discussion. But yeah, so I was there at the beginning with the MSRC. I remember the MSRC first forming.

People like Steve Lipner and Scott Culp will remember some of those names who started the MSRC back in the day. And obviously the big focus back then was on literally security vulnerabilities in products like Windows and Office and SQL Server and Exchange and Visual Studio and so on and so forth. So, I mean, is this like an adjunct to the MSRC? Is it part of the MSRC?

How does that all fit? Yeah, great question. And it always fascinates me, you know, people I meet either at Microsoft or across the industry. There's like so many people just across different companies right now who all started their careers at MSRC.

So it's really fascinating to hear the stories and kind of, you know, the... period for which this specific part of Microsoft has been, I would say, continuing to grow and bring together the nuance of coordinated vulnerability disclosure, specifically when it comes to external researchers. Just fascinated by that. Specifically to your question about, you know, AI versus MSRC AI versus MSRC.

Yes, MSRC AI is part of the MSRC. We have two... big groups I would say. One is the security assurance engineers who are basically working on triaging the bugs which come in.

We also look at ensuring that it is transferred to the right teams within Microsoft and get the right level of visibility. Then we also obviously go through and ensure that this is in line with our AI bug bar. We can talk a little bit about that in just a little bit. But then we assess those bugs to ensure what kind of severity it is in the space.

And so that's basically what this team has been doing. And we've also built a relationship with external researchers, specifically who are doing research. in AI safety and security. Some of the names are Johan Rehberger.

He has published the article on Embrace the Red. He has a great blog on that. And Zenity, we worked quite closely with Michael Bargeri and his team. So those are some of the well -known researchers, but we also have others we've worked on with them.

Yeah, I think the thing that makes it interesting is that it's such a... brand new area, relatively speaking. I mean, security vulnerabilities, SQL injection, memory corruption, et cetera, et cetera, have been around forever. AI -specific vulnerabilities are relatively new in the overall scheme of things.

However, in many cases, what is old is new again. I mean, if you look at prompt injection, you could argue it's just input validation. Just on steroids, though. Yeah, exactly.

Well, you know, as you and I were talking about earlier, you know, there's a security mantra, which is, you know, thou shalt not mix the data plane with the control plane. And yet with prompts, that is precisely what we're doing, right? Yeah. Absolutely.

And that leads to the different types of attack surfaces, as well as almost sneaky attacks in some ways. I mean, we all know, everybody who's listening knows about AI and generative AI and the impact this has had on our lives from 2023, at least, since ChatGPT has been released. There's a complete era booming. Everybody, businesses are wanting to adapt and obviously it's the right thing to do because it brings so much additional goodness and efficiencies.

But like I said before, as the goodness comes along, there's always some risk involved. And here we have expanded the threat surface and the threat landscape in completely new areas. Yeah. I mean, at the end of the day, I mean, let's be brutally honest.

At the end of the day, you know, for every use case, there's probably N abuse cases where something can be used for good. And then all of a sudden, you know, well, if you do it this way, it can actually be used for bad. Yeah. So just going back to the MSRC versus MSRC AI.

I mean, so the MSRC handles classic air quotes, classic security vulnerabilities and the MSRC AI handles. safety -related vulnerabilities. And security, AI security vulnerabilities as well, yeah. Yeah.

So a classic bug is a security bug, but it might not be a safety bug necessarily, right? Yes, it may not be. So the word safety, if you think about where it actually came from, right, it actually came from ML. When you go back to before generative AI, when a model was inputting something which was not in line with what they wanted, then that's when you would call it a safety.

It's a safety issue. It's not aligning. It was like an alignment problem. But in those cases, the model developer and the model user was the same person.

But that's not the case anymore with generative AI. The model creator is completely different and the user. obviously uses it in different ways. And so that's why this has become a bigger challenge right now.

It's not so much misalignment because the misalignment ultimately ends up with socio -technical issues, which just makes the category of AI risks just a bigger blob. It's not just confidentiality, integrity, availability. It's also some other... Some other things, like we've actually published that on the AI Bug Bar.

I can share the link with you, Michael, so you can add it to when you publish the podcast. It's basically, we have content -related issues, and that actually spans a myriad of different types of safety issues. Some of them are information integrity, where AI has generated content which is false or misleading. Then it could be inappropriate language or information integrity.

It could be things about malicious users. It could also be CBRN, a harmful use of CBRN to demonstrate information which shows information which is not readily available. By the way, CBRN actually stands for chemical, biological, radiological, or nuclear. So this is a category of when the model produces outputs, which are in these categories of chemical, biological, radiological, nuclear related.

category then we say hey it's it's the category cbrn but there is more nuance to specifically categorizing at cbrn that it should not be readily available publicly like if someone can get the same information in a google search or a bing search then that is not brand new we are actually looking for uplift what uplift means is that what is the model providing completely new which the user will not be able to get it otherwise. Yeah. But then you've got other topics, right? You've got like sexual content, physical harm.

Yeah, self -harm. Yeah. So these are all like the new categories, what we have seen with AI safety. This is new overall in the industry, right?

And we are looking into this because to ensure that we are thinking about risks more holistically when it comes to AI. So what are some other examples? I mean, that's image manipulation. What about any other examples?

I think this is an area where a lot of people, it's brand new to a lot of people, and it's an example of stuff you don't even know you don't know. So any more sort of context you can give around that I think will be really useful to our listeners. Yeah, some of the safety issues we have seen is folks sending us Actually, even with indirect prompt injection, the way we categorize something as a security issue is as long as it has a security impact. So if the specific security impact could be something like a data exfiltration.

If you can showcase in your bug, as you're sending us a bug, if you showcase that, hey, it has a security impact, that means I'm able to exfiltrate data, then we categorize it as a security issue. And in a safety issue is anything where you can show indirect prompt injection, but you're not able to demonstrate. data exfiltration, for example, then we call that a safety issue. So that's one way, one of the things we have seen.

The other stuff we've also, other kind of issues we have seen is people just manipulating pictures and sending us you know, like different types of images with some political figures and things like that. And those have been safety issues as well. And most of the time it is because a classifier, we need to go update a classifier. So you mentioned very early on that there is an MSRC AI bug bar.

Can you give us examples of, actually before we do that, why don't I explain what a bug bar is at Microsoft and then you can sort of chime in with. what the AI bug bar kind of looks like. So the bug bar at Microsoft has been around for a long, long, long time, and it's a way of categorizing overall risk slash severity of a vulnerability. So for example, a remote code execution exploit that is easy to pull off remotely and gives you system on a Windows box.

That's more than likely critical. Now, there may be a lot of extenuating circumstances that reduce the potential damage. Perhaps it's only a local -only attack. Perhaps it doesn't give you a system.

Perhaps it's on a feature that's disabled by default. All these other things and other defenses come into play, and that may be rated a low or a moderate. I'm just making that up. So the bug bar has been around at Microsoft for a long, long, long time.

So what does the bug bar look like for MSRC AI? Yeah, it's a great question. And it's actually exactly what you talked about, Michael. It is basically a way of categorizing the different vulnerabilities we are seeing in the space of AI, both security as well as safety, and then talking about severity, specifically when it comes to...

safety, there is no severity. It's more focused on in -scope, out -of -scope, because this is basically, we are trying to identify whether it's, what is the safety issue? And there is nothing about in -scope or out -of -scope. Sorry, nothing about specific severity because of this emerging space we are talking about.

But when it comes to security, it is very similar to other MSRC bug bars, specifically inference manipulation. We talk here, some of the things are prompt injection. The indirect prompt injection has been the largest category of bugs we have seen. and if it's a zero click attack like basically if uh without a user performing any any action uh then you know uh and the data is exfiltrated through uh through a prompt then we call that a critical a severity bug and if we need a specific interaction that means somebody needs to click a link and then that allows to exfiltrate data then we call that an important severity bug so that's kind of how we've been categorizing the the ai vulnerabilities We've also called out input perturbation, which is basically any time a specific classification model is misclassified or, you know, those are what we call input perturbation.

But we haven't really seen these kind of bugs submitted to us. Mostly it is in the space of indirect prompt injection. And indirect prompt injection is where you embed essentially a prompt injection -like attack in a document that is read by the LLM. Yes, yes, that is indirect prompt injection, or it's basically the top LLM vulnerability, even in OWASP.

top 10. This is basically where the attacker is able to say, you know, summarize an email, and then the email actually has the prompt injection embedded in it. So when the user who's interacting with the LLM says, hey, summarize my email, the prompt injection automatically gets triggered. So I just realized while we're talking about prompt injection and indirect prompt injection, not everyone will probably understand what a prompt injection attack is.

Do you want to just give a quick example? Yeah. So, for example, let's assume somebody sends you an email. The attacker can send you an email with just a normal email in your inbox.

And as part of the email, there is a line that says, send this data, send sensitive data to attacker. website or attacker email right and the user who is just on their emails uh using copilot might say hey summarize my email and when they do summarize my email in our llm it behaves differently right it interprets a prompt or an instruction inside an email like a prompt, and LLM, as useful as it wants it to be, basically takes that prompt and summarizes it and basically sets forward that action.

What it does is that the user is basically just summarizing the email, but in the background, the LLM is already sent. some sensitive information to the attacker email or to whoever is basically sent that prompt injection to them so that's why it's indirect prompt injection the llm doesn't differentiate between the instructions coming from the user directly interacting with the llm or or instructions embedded in documents it is leveraging to summarize your instructions hopefully that's helpful I think if people listening to this podcast don't know what prompt injection is and indirect prompt injection is, this is as classic.

It will end up being as classic as SQL injection, memory safety issues, cross -site scripting. Absolutely. So everyone needs to be aware of what they are, especially if you're building your own tools on top of perhaps your own LLMs or third -party LLMs or something. I think it's just critically important that you understand what these...

these vulnerability classes are. In fact, most importantly, what the mitigations are. And I think it's fair to say that this whole area is constantly evolving. The thing that sort of concerns me is if we take SQL injection as an example, we know how to fix that.

We know precisely how to fix that. You just use parameterized queries, basically, for any untrusted data. The problem solved. The problem with LLM prompt injection is there's no...

single mitigation that fixes the problem. Because as you noted, we're sort of mixing control constructs with data constructs, and we don't know which is which. So it becomes very, very difficult. Yeah, absolutely.

And I think ultimately, I think security basics... apply whether it is ai or non -ai so i would say like continue to really think about defense and depth and defense and depth is no longer optional it's actually really important and crucial for uh specifically mitigating something like indirect prompt injection you need to be thinking about hey what is my content security policy And you need to have multiple layers of defense. So it's almost like the Swiss cheese analogy of like one, okay, it doesn't catch here, does it catch the next one?

And that is the best way to mitigate these. Yeah, don't forget least privilege. Like if you're running AI agents, if you have an agent that runs with godlike permissions over your environment. and the LLM decides to start doing its own thing, then you can start doing all sorts of damage.

So yeah, least privilege is storm. It's just interesting to me that we talked about least privilege since the 60s and 70s, and here we are again talking about least privilege with a brand new type of technology, which is generative AI. Yes. And I think there's another aspect of this as well, which is basically, you know, we have to be prepared for what can go wrong.

I mean, and this is like classic security mindset of threat modeling, right? Like, well, what can go wrong? I mean, like, this is how we should be thinking about our systems. So really, really digging deep into secure by design and threat modeling right in the beginning, right, as you're getting this.

as you're thinking about ideas and kind of coming together as a security team, product team, you know, like and working together to ensure that we are meeting customer needs from a customer perspective, but also really thinking about what is the how can we continue to keep provide that safety and security aspect of things as well as, you know, really focusing on or like almost like. assuming things will go wrong and uh preparing for what should you do when things go wrong because it's too late to figure out things at that point so at least if you have like um here is my team i will be who will be looking into these or um you know how do you pull into incident response or what are the new terminologies in incident response Some of those things is kind of keeping in mind as you're thinking about AI systems.

Yeah, and that's why I think it's so cool that Microsoft has now set up a Microsoft Security Response Center division or part of the MSRC just to focus on AI. This is a constantly evolving area and response is a critical part of responding to AI vulnerability. So this is really good to see. Yeah, absolutely.

And it also helps to continue to think about it from a... you know we have a lot of processes internally when it comes to pre -deployment right like before deployment like a specific uh you know there are a lot of different controls including a deployment safety board which actually stamps the approval before something actually goes out the door from microsoft from an ai perspective and then on the other side msrc is working with these researchers who are seeing the bugs in production right and that kind of provides that feedback loop to continue to better our defenses and as well as think about things in angles which we had not thought about ourselves so different perspectives bringing in completely new different perspectives yeah and i think that's something that's really important as well as you know serious issues, critical issues go through a complete root cause analysis to find out what happened and what we can learn from this.

And I think that virtuous cycle is critically important. So again, it's good to see that we're doing this in the AI space as well. All right, Raji. So one question we always ask our guests is, what does a typical day in the life look like for Raji?

I feel like the answer is... for most of your guests must be there is no typical day in security that doesn't, it doesn't really, it's very different every day. And I think that keeps me kind of. excited and motivated but if i have to really think about hey what does my typical day look like it looks uh it basically is working with uh both external folks as well as internal teams and continuing to uh kind of bring together some of these perspectives of new vulnerabilities coming in and helping our internal teams really think about it from uh you know how does that fit into their threat model and how do they mitigate that, as well as how do they take the feedback from these different research to basically better their mitigations in the future, as well as working with our...

bug bounty teams uh because we also you know some of the some of the all of the work we are working here uh in ai security specifically have uh have uh rewards attached to it so we've uh and recently in black hat eu Microsoft basically, MSRC basically announced a standard award policy, which basically says that you don't have to be in scope anymore. There is no in scope. But as long as you have a critical or an important severity bug, then you get paid a standard award, which is really cool.

Yeah, that is cool. So those are some of the things I do work with. Very cool. All right.

So let's start to bring this episode to an end. So again, another question for you. If you had one piece of advice to leave our listeners with, what would it be? One final thought.

Yeah, one final thought is really, you know, the Microsoft AI security and safety team is always looking for AI security researchers or safety researchers, you know, to work with. And if you find any bugs in Microsoft AI security products or services, please reach out to the MSRC at the researcher portal. And you can submit the box to us. And we would love to continue to engage with you and collaborate with you to ensure that we can make Microsoft AI products and services better for our customers.

Very cool. All right. So, Raji, thank you for joining us this week. I know you're very busy, and that's a very interesting area of research.

So it's great to see we have people like you working in this area. And I always learn something new. Again, I'm a classic security person, but over the last couple of years, I've also learned a great deal about AI security and safety. And again, it's amazing.

The things you don't know, you don't know. So again, thank you for joining us this week. And to all our listeners out there, we hope you found this episode useful. If you're in the AI space, you really need to understand AI safety and security, especially if you're building products.

on top of AI. So stay safe, and we'll see you next time.