Key Governance Risks in AI Deployments

Welcome back to the Risk and Cybersecurity Podcast. I'm Lauren Ross, a senior manager in our cybersecurity practice, and today I am joined again by Steve Ursillo, a partner in our cybersecurity practice, and Morgan Haag, senior manager at Meditology Services. Today we are discussing managing compliance for internal versus vendor AI, and this is the third episode of our AI Compliance Series. Steve and Morgan, welcome back. Thanks, Lauren. Great to be here. Thanks. All right. So I wanted to start off by talking about internal AI when you have your own large language model. So Steve, first question for you. What governance challenges arise when managing the full lifecycle of an internal LLM? Well, I think, you know, anytime an organization is evaluating whether or not they're going to use something internal versus using a provider, there's a number of different considerations there based on what their requirements are. But some of those requirements should obviously be the, the risk management elements of that, right? So ultimately, you know, the risks are the risks. So everything that we've talked about in the last couple of episodes, from AI governance down to some of the more operational aspects of risk management, need to be considered whether or not you're whether you're, you know, building your own LLM or you're obviously using a third-party LLM. It's just a matter of, you know, the level of transparency, oversight, and depth that you can get, you know, in using your own LLM versus using a provider. And those are just things that inherently come with the nature of, you know, typical third-party risk management challenges. That said, when you're talking about your own internal LLM, then, you know, that's, that's something that obviously can take a situation where you can, you have an advantage where you can build in some of these risk management tactics into the design and execution of the LLM. So like any other, I'll say SDLC project or any other project that you're building out on for a particular system, you embed risk management throughout the entire lifecycle from defining the requirements all the way down to the execution, depending on whether you're a waterfall or you got some type of agile, you know, development cycle. But ultimately, you know, doing the proper risk management elements from the governance aspect of it all the way down, like I said, to these individual risk management areas for specific AI risk becomes really important. And if you navigate that early enough, then you can design and build in certain safeguards and capabilities and interoperabilities in order for you to, you know, prevent detect, respond, manage, and monitor, you know, these different types of risks or different types of operational challenges that might come forth. So that said, you know, data governance becomes once again, you know, a big area to consider. So if the LLM is being deployed in-house and understanding the boundaries of the LLM, what type of data it's going to access, what systems are brought into play, if it's doing it to any type of document repository, if it's, if it's accessing the data that's expected and making sure that all of your data management practices are aligned, making sure that the data quality and the sensitivity of any type of information is protected for leakage or cross-contamination, anything like that, sovereignty issues, things along those lines. You're also going to be getting into the governance around the training and the, and the overall management of the LLM. So making sure that the training data is not only intact, but it's secured, it's reliable, it's not stale, it's free of bias. And of course, it's maintained in such a way where you can, you know, understand what training data has been involved in developing different types of models for purposes of versioning and/or, you know, validating certain types of output or the overall evolution of the LLM as it goes forward. Model governance becomes another aspect of things to consider, right? So you got the data governance and you got model itself, you know, validating the accuracy, the robustness, reproducibility of the results that the model may be generating, making sure that you're defining certain tolerable levels of performance and you're trying to prevent any type of hallucination or drift within the model, making sure that you're monitoring that and you have the right mechanisms in place in order to cast that level of transparency. Of course, you're going to be designing front ends and other types of safeguards within the actual processing of the model to make sure that you're mitigating any type of adversarial attacks on input, like any type of direct or indirect poisoning attacks, prompt injections, anything like that. So just the model governance aspect of it becomes a big element. And then of course, the operational governance where you're setting in these continuous monitoring elements for all the things that we just talked about to, to really manage that drift, bias, or misuse. And then you, you know, you can't forget about your overall general control structures and ITGCs, or internal general controls that you'd expect to have as it pertains to basic security hygiene and incident response. So if there was a potential incident or an attack on the LLM or any type of, you know, inappropriate disclosure of information or information that was inaccurate, that there's a response policy intact in order to mitigate communicate that in order to make sure that those particular elements are handled appropriately. Then the very real thing around all of this is really just making sure you are in a position to understand why governance matters. Of course, with your own model, you know, you're going to have users relying upon this, whether it's internal users, external users, depending on the nature of the business or the services you're providing. And that trust is going to hinge based on consistency and accountability, right, and traceability. So making sure that the more mature you you design these processes and you aid in the level of transparency, the better you're going to have in obviously being able to reproduce and provide evidence that you're governing it properly. If a service organization is using AI and they're, you know, casting output that's going to have an impact on either financial reporting or in a SOC 2, their service level commitments, then it's important that they're able to demonstrate with maturity around these basic safeguards and some of the governance and risk elements of AI in order to ascertain confidence around their service level commitments. So those are some of the big, big differentiators. Obviously, if this is something that's done by a service provider, it then becomes the same risk management elements, but really taking into account more of a third-party risk approach to making sure that the provider is doing the right thing and understanding the shared responsibility model there. Thanks, Steven. You know, over the last couple of episodes, we've talked about various compliance frameworks. So Morgan, from your perspective, how does ISO 42001 support responsible development and deployment of proprietary AI models? Yeah, I think, and to your point, we've talked about it a little bit in some of the other conversations, but really it just puts into practice, or at least gives you a general roadmap to kind of get through some of the basics from an AI ethics, from an AI secure use perspective. You know, when you think about all of the different considerations, like Steve mentioned, you know, data governance, model governance as an example, there are so many different flavors of how you can accomplish that, or so much nuance in terms of, well, maybe, you know, authority body A says that you need to be at this level, or, you know, nation-state B says you need to be here. 42,001 provides kind of a pretty general standard for organizations to meet and kind of aspire to, where now they have a little bit more of a playbook in terms of what they can meet. What are the kind of minimum necessary benchmarks they need to be able to hit. And so that crosses off a few things, right? I think the first piece is kind of taking out the gray, the maybe fog of it, so to speak, where, you know, people talk about having to have AI ethics. What does that actually mean? AI secure use, what does that actually mean? It translates that into something that's a little bit more actionable and tangible. But then throughout the nature of the requirements in 42.001 as well, it talks about and provides for organizations a means to provide structure for the risk management system that's built around their model. So thinking about how to tailor that system specifically to AI, how does that differ from something like 27001 that's been in the industry for a while now, and then also guides them through a lot of the kind of basic principles that if they're not familiar with AI specifically, they might not be very familiar with, right? If you're an organization thinking about jumping into the AI space. And so, you know, that might be identifying which of your systems might be designated as high-risk AI systems or evaluating kind of the risks of all of the relevant systems that feed your data model or your pipeline itself, things like that. Right. And then the details around impact assessments or stakeholder engagement even, and then thinking about kind of the practical considerations of developing model cards and what needs to go in there. You know, 42,001 gives you the kind of roadmap to be able to accomplish those things. And so it's a really, really actionable method or means for organizations to take that step. And even outside of just kind of providing you that kind of development and deployment kind of guideline or baseline, it provides assurance for your buyers, right? So for your clients, for you as a third party, like Steve mentioned, there's a good degree of assurance there if you're going with 40001. That, hey, look, we're not just kind of flying by the seat of our pants. We've taken, you know, kind of a measured approach to how we're going to implement this system. And as a buyer, it's always great to have a little bit of that rubber stamp there. And so those are the main kind of considerations and kind of, I think, areas where it can provide some support for organizations. Yeah, thanks, Morgan. And Steve, you know, another framework we've talked a lot about is SOC 2. So when offering AI as a service, How should SOC 2 be scoped to reflect AI-specific risks? Yeah, thanks, Lauren. So, you know, similar to what we kind of discussed in the past there, you know, SOC 2 is a reporting framework to help an organization really display its maturity around service level commitments. So whether or not that's security, availability, confidentiality, privacy, processing integrity, so on and so forth. So like anything else, typically an organization is going to look at the boundaries of their report. They're going to look at the systems that are included in there. And if they've identified their AI registers and their inventories of all of their AI-based systems that are part of those boundaries, then it's important to disclose that in the system description. SOC 2 has particular elements that are required for the description criteria, DC 200, and those elements can fit perfectly within the proper categories there. Even though it's AI-specific, you would just drive back what's relevant to the governance around AI versus some of the tactical risk mitigation around AI built into the system and the system model. Model. So while we know that you'd be going through your, your typical points of focus and the criteria to cover the adequate parts of the system depending on your scope, you're also going to think about those AI-specific elements that we've, we've talked to in the last couple of sessions here. And as we drive kind of back and we think about, well, what's relevant there from a completeness perspective, I'd strongly encourage organizations to take a look at 42,001 from the governance aspect of the model, and then also the risk management framework around AI risk management framework, NIST, NIST AI management framework in order to think about, you know, the actual control implementation to properly manage some of the risks that are inherently and directly related to their models. Those can— because like I said, the reporting framework is broad enough, it's based on COSO, you can drive the elements of that in the particular categories or criteria of the Trust Service categories in the Trust Service principles. Now, there are enhancements that are underway the AICPA SOC 2 Working Group in order to add specific elements of points of focus around AI, which I think is going to continue to provide more transparency and value. But as it stands right now, provided that you have the framework and you have the right risk mitigation elements according to best practices, you could weave those into the proper elements of the Trust Service Criteria now, right? Again, because it's a reporting framework based on COSO, you just have to have those standards aligned. So when you're thinking about that, you want to scope in those right components. You want to think about the AI model. You want to think about the training infrastructure, the training pipelines. You want to think about the inference infrastructure, you know, where these training models are deployed to make predictions or classifications or decisions, anything like that. Those are all going to be part of the system. You think about the governance structure around the AI, the roles, responsibilities, the oversight committee that's responsible in the risk management aspects of it, having that in scope, thinking about that risk management structure. What's the strategy? How is it executed? What's the frequency? You know, if there's different types of automated and continuous monitoring, what are the elements that you could bring forth there to give some transparency on how those risks are being mitigated? And of course, it all drives down to service level commitments. So you're obviously going to be thinking about, you know, defining those commitments, the accuracy, the fairness, transparency, reliability of AI inputs, processing, and outputs to make sure that you're aligned with what you need to enforce the service level commitments that the auditors are opining on. These, these particular AI outputs in many cases are defining a benchmark. So if you have the anticipated output as a criteria for the benchmark, the auditor is going to be kind of assessing against that criteria for purposes of proper processing elements, depending on if processing integrity is part of what you're bringing into the mix. Then you'll think about additional control elements, human in the loop and other types of validation checks that might take place in order to, once again, especially in high-risk areas or certain industries where there's high-risk elements like finance, healthcare, you know, HR, so on and so forth, you know, there's gonna be different types of these validation checks that you'd see on a regular basis to make sure whether they're human or automated to make sure that the system's behaving the way it needs to. It also gives a good idea or a good way for customers and regulators to know that, you know, a lot of the decision-making and the output from these systems is not left unchecked. It's following a particular model. Model. So overall, you know, making sure that the SOC 2 is once again covering the governance and the risks elements of AI, just expanding into, you know, the new and emerging technologies that are surrounding the platform that's being covered or the system being covered, making sure that those typical risks are covered in the way that they're expected, and then making sure that, you know, you're building a real trust around the AI as a service system Or even if AI is part of an existing system and it's not necessarily as a service, you know, it's just integrated as part of the system being covered, making sure that there's enough transparency and best practice risk management controls in there so that user entities, customers, business partners, and/or regulators have the ability to gain the transparency they need to provide that level of trust. Thanks, Steve. And as we think about vendor AI and third-party large language models, Morgan, from your perspective, what are the key elements of effective vendor your due diligence for AI systems? Yeah, it's a good question. I, you know, I don't know honestly that there's a significant difference in the foundations between what you'd want to consider from an AI vendor perspective and what you'd want to consider for just a general vendor. But I think there's a couple of key areas where the level of focus or maybe the level of scrutiny, if I'm going to ramp that up a little bit, I think one of the main things is transparency. So this is kind of a tenet you think about when people say responsible AI. There's a lot of kind of independent groups that are, are, are pushing that, I think for good reason. And transparency is one of the main components. It's shared a lot in, or in a lot of ways with the privacy domain as well. But it's basically, hey, look, if I'm using your service, if I'm using this model, uh, with my data, right? Let's say you're a highly regulated entity like a, a healthcare organization. If I'm sending patient data to you, um, I wanna make sure that I know exactly what's happening to it, right? And so when you're engaging in these relationships with the vendor and potentially going to use a service or a product that leverages some sort of a model, it's important during that process that not only are you just like sending off a security questionnaire, but you've seen, you know, ideally data flow diagrams, architecture diagrams. You understand exactly where the data is being sent. For a lot of organizations now, they kind of have fairly complicated architecture, or maybe only one specific environment or instance is certified or has been attested to from a security control perspective. And so it's important that you have a very, very clear picture on what their environment looks like with regard to where your data lives, where it's being processed, and where it's being transmitted kind of back and forth. And so I think the transparency component is really, really important and a point of focus. And then I think outside of that, obviously, is just making sure that you're validating what they're saying. So it's not just a matter of, you know, hey, look, I'm going to send you a couple of questions. Let me know what your take is on this as a security function. For that vendor as an example, but getting some sort of assurance. So I think a SOC 2 at a minimum, you know, broader levels of assurance, especially in the AI side, whether that's, you know, ISO 42001 or HITRUST AI certification or anything other above and beyond that where they can provide some sort of assurance that yes, we are doing these things and we've been, you know, verified by some independent party is really a must. And it might be that you're that organization, right? If you have a really, really close relationship and you're putting a lot of chips on the table, I'd probably recommend some sort of first-party validation of those things. You know, have a walkthrough with that organization of, of how they're leveraging your data. What are the kind of core outputs and inputs of the model? Just so you're aware how it works. And that kind of speaks to another piece, which is just making sure you're comfortable with their operational maturity. If this is an organization that spun up 2 months ago, they have very few clients, and you're not totally sure if they even developed the service or product, or if it's just something they spun up using, you know, cloud code, it's important that I think you have a very clear perception, or maybe not perception, but a clear understanding of their maturity and their ability to kind of manage your data accordingly. I think Sam Altman at one point said that we're in the fast fashion era of applications and services, and I think to some degree that we are, you know, I think the burden of entry into the marketplace is been significantly lowered, which I think is a good thing in some cases, but it can also be a real concern. And so it's important now more than ever for security owners or even people that are just in the contracting department to apply scrutiny to these organizations and say, hey, look, you might have a rubber stamp here in one form or another, but can I actually trust you with, with this data? And so I think above and beyond that, you know, I'm sure we'll talk about this as well, but, you know, having some sort of internal function to review third parties is great. Having a really toothy agreement or contract that has some sort of repercussions and punishments outlined is also a must-have. And so that's just another piece above and beyond just some sort of third-party risk or third-party vetting process, you know, putting pen to paper on what's going to happen if. Yeah, so maybe you could talk about that a little bit more. How can organizations, you know, ensure contractual safeguards are adequately addressing these evolving AI risks? Yeah, absolutely. I think honestly what we're going to see is a little bit more across the board. Most organizations are going to have some kind of default agreement, right? If you're in the healthcare space, you're typically going to have a BAA, you know, associate agreement for everybody that might be consuming or ingesting patient information and in any other industry there's going to be something consistent with that. I think the bar for those standard kind of template agreements is just going to be raised a little bit. I think when you think about the contracts and the content that need to exist there, having really robust data use clauses is a must. So I think in the privacy space those have existed. I think they're going to continue to be expanded out quite a bit, right? So there might be explicit explicit prohibitions on how to use certain data in certain which ways, kind of more than we're currently seeing kind of in, in the environment. There's also going to be requirements, I think, for, or maybe something you should consider, change management. So if a model is changing, we need to have a very, very explicit notification of what's the change inclusive of. How does it change if our data is being adjusted? Any change to anonymization, things like that. In the same way, there's a lot of typically expectations around right to audit, things like that, evaluation rights as a service kind of consumer, or as anybody that's providing data within the confines of an agreement. Those are going to become more and more popular, I think, to some extent. I'm just making sure that, hey, look, if I'm giving you these droves of information, I need to be able to validate that it's being treated properly in some form or fashion. And so that's another piece. I think outside of that, just there's always going to be kind of continuous enforcement, or maybe not enforcement, but continuous validation from a regulatory standpoint. Like, hey, is this agreement reflective of any of the reg— regulatory requirements, or even legal requirements at this point that we're beholden to? I think as the AI space continues to evolve, there's going to be legislation at some point that does pop in. And so it's going to be a really, really important consideration to ensure that in your agreements, things like notification timelines are consistently updated. So it's not just you as a contracting party for your service organizations, but for anybody that you provide services to as well. It's really, really important that you kind of maintain a very, very firm grip on that regulatory environment because I think it's going to, in the next couple of years, speed up quite a bit and it's going to be a little bit of a headache for legal departments. Which is probably okay for the general public, but I think just something to consider. So yeah, from a contracting perspective, I think it just comes down to, hey, look, take a really hard look at who's using your data and make sure that you have a really clean paper trail in your agreements to understand what they should and shouldn't be doing. And then also include some of that penalty structure generally. So like, hey, look, if you don't do this, that's a termination of the contract, just to avoid the headache going back and forth if something does happen. So, yeah, thanks, Morgan. And Steve, I know we've spoken about this a few times throughout the series, but, you know, as it relates to these third-party vendor AI tools, what are some of the mechanisms that exist to monitor some of the concerns like model drift and other ethical concerns in these third-party tools? So, yeah, I think that the inherent challenges that when you're using— there's a lot of benefit, but there's inherent challenges when you're using a third-party AI AI tool, only because obviously, you know, there's a, there's a quest to get a better level of transparency on everything and make sure that you're comfortable with how all the risk mitigation strategies are put into play. Ultimately, you know, understanding the nature of what the AI tool is doing, understanding how the AI tool interoperates with your systems and other systems that play into the boundaries of your datasets. Become really important in talking or working with, you know, the particular vendors, just making sure, especially if you have, as, you know, Morgan had talked about before, the right clauses in order to hold them accountable, making sure that they're sharing certain artifacts that would give you some assurance on the fact that they are in fact monitoring for anything that's happening out of the ordinary, such as drift or any type of ethical issues that might might transpire from the use of, of using their LLMs or anything else that they're putting forth. So getting certain artifacts can help for maybe some drift indicators. What's their, what's their process and what do their retraining cycles look like? What's their governance around, you know, such processes? And then what types of monitoring logs are they using? Are they establishing benchmarks and kind of guardrails on the processing of information so that if anything starts to operate outside those bounds, There's a clear indication of, you know, something that needs to happen in order to right-size the ship. Those are the typical types of things that, you know, you would expect to kind of ask and get some input on from a monitoring perspective. That's going to give you a little bit more transparency to know how often the model's updated and what's happening in the way of just making sure that the model stays current and it doesn't doesn't become obsolete. Making sure that you get independent verification. So once again, if there are fixed benchmark datasets to catch any of this, if there's some way to test against that, whether it's something you're doing or whether it's something that you're working in conjunction with the provider, or maybe they're having an independent assessment on that, you know, think of it as almost like a health check, just making sure that there's an independent validation that's taking place. Could be their internal audit group, it could be an external provider, whatever the case may be, just helping you do what you need to do there. Also making sure that they're taking into account any type of ethical or adverse impact checks. So collecting— you can certainly collect maybe user feedback or just error cases and things that they are monitoring in order to make sure that, you know, these particular elements are not coming forth, whether it's done through your own systems as a compensating measure or part of a shared responsibility, or whether or not you're doing it, you know, in conjunction with the provider that allows for that level of transparency. Anything that's outside the bounds or anything that's a flagged outcome so that you can, number one, know what are the parameters to get me there, and then number two, how many times have we fallen off that? And is there any, any type of measurement that is cast up to show compliance over time? Certainly looking for signs of any unfair treatment or output that would be, you know, to the detriment of what your objectives would be, or any type of information that might traverse outside of a boundary that you would expect. Those are all things that you'd want to have some level of transparency that they're managing, making sure that you're reviewing their overall governance structure. So asking the questions and getting some information on what's their risk committee strategy, what's the program look like, what, how often do they meet, what's the cadence, what are some of the actionable items that come out of that? You know, how are they actually operating with their models? Are they using any third-party products and providing services for their customers? Is that taken into account not only in their organic or their own AI, but also any third parties that integrate with their AI? And as they start to incorporate, you know, certain aspects of agentic AI, understanding how that risk management cycle applies to the nature of, you know, these workflows and workstreams. If the organization is doing any type of red teaming or vulnerability management around AI, that's, that's another great way to get some transparency there. There are products in the marketplace now that allow you to interconnect your, your LLM stack against, you know, these particular platforms that continuously test for different types of AI risk. All the stuff that we've talked about here from injection attacks to bias to, to drift, any, anything like that. They will also allow for, if you upload your, your AI policies and procedures, they'll actually test against those policies and procedures. So as part of overall threat and vulnerability management, you can incorporate some of these red teaming or ask the vendors if they're incorporating these red teaming strategies into their mitigation strategies. Some of these products are actually expanding upon not only doing the red teaming piece, but also taking modules and wrapping them around the LLM. So the stuff that they're testing from a red teaming perspective, which by the way is an offensive security measure, as acting as an adversarial attacker against the LLM, the ways in which they're using different logic to attack, they're also using modules to wrap around the LLM to make sure that it's prevented or detected. So that's another very interesting way that organizations can start to get ahead of putting something either incorporated in or around their LLMs to try to help manage that risk. Think of it almost like, you know, a web application firewall in front of a website that's protecting against different types of web application attacks, right? So just another way to put in more safeguards and manage against policy. So, you know, I think ultimately the vendors are going to move fast. The third-party risk management area moves pretty quick. Making sure that you have these particular elements designed into your overall strategy, asking these questions and gaining that transparency, I think would be a good way for you to make sure you don't have a blind spot as you're going through these evaluations. Yeah, awesome. Thank you, Steve. All right, well, thank you so much, Steve and Morgan, for all your insights here. And I know this is a continuously evolving space, but there's a lot folks can do now to get ahead and safeguard their data. Thank you all for tuning in to the Risk and Cybersecurity Podcast. We hope you've enjoyed this AI compliance series, and don't forget to like and subscribe so you can always be up to date with risk and cybersecurity. Thank you.