Var Shankar: AI Governance for Smaller Organizations

This file was generated by Descript Werbach: Hi, I'm Kevin Werbach, Professor of Legal Studies and Business Ethics at the Wharton School of the University of Pennsylvania. For decades, I studied emerging technologies from broadband to blockchain. Today, AI is promising to transform our world, but AI needs accountability mechanisms to ensure it's developed and deployed in responsible, safe, and trustworthy ways. On this podcast, I speak with the experts leading the charge for accountable ai.

Most AI governance guidance is designed for large global enterprises, but that leaves a significant gap. Var Shankar, executive director of the Council on AI Governance is working to fill that gap. He previously led the Responsible AI Institute and worked at the Reg Tech startup enzyme. In addition to practicing law at Grava, Swain and Moore and working at the Clinton Global Initiative, we talk about why smaller organizations.

Actually need AI governance and what they need, why workforce AI literacy is so important, and AI governance lessons from the healthcare and financial services sectors, as well as other important points in this conversation. Our pleasure to talk with you. Welcome to the Road to Accountable ai. Shankar (2): Hey, Kevin, pleasure to be with you.

Thanks for having me. Werbach: Tell us a little bit about your path to working in AI governance. Shankar (2): Yeah, so I'm a lawyer by background, and my interest has really been in using AI to improve service delivery. So that is both in terms of the substance of the service being delivered, for example, better healthcare, but also in terms of form.

So providing services in the right language or in a more accessible way. And so that's kind of how I got started in Responsible Tech. I worked on these issues for, uh, the government of British Columbia. Both digital government and COVID response and you know, AI started coming up more and more.

I started looking for clear guidance on, on responsible AI use, and found that there was very little joined and kind of went, eventually went on to lead the responsible AI Institute with the support of other large member organizations in healthcare, pharmaceuticals, financial services, energy, and others. Was putting out really clear guidance on what good looks like for AI governance. And working on certification programs. Then went on to work on it from the startup side, joined Enzy, which is a RegTech AI compliance company, and we were kind of tackling exactly how organizations can work across functions to both drive responsible AI outcomes, but also to put together the documentation that demonstrates responsible ai.

And then more recently joined the council on AI governance. Where we've been more focused on providing open resources for AI practitioners within organizations. Yeah. And so I think that, so the council is the brainchild of my colleague and, and kind of close collaborator, Alexis Cook, who was previously an AI engineer and developer advocate at Google.

And her view, which I share, is that there is. A gap in clear guidance for most organizations because the kinds of AI standards we're seeing are best suited for large, sophisticated, kind of multifunctional, you know, global organizations. And about half of American workers are employed by organizations that have fewer than 500 employees. And so our effort is really to provide those organizations and practitioners with.

Open current and, and pragmatic guidance that's aligned, you know, doesn't take any single perspective, just kind of brings together the best practices. Mm-hmm. For example, you know, this week we're releasing an intellectual property primer for people using LLMs. Just kind of, you know, a few pages on, on what good do's and don'ts would be at a smaller organization.

And then we also have our AI governance playbook, which provides kind of a mental model, uh, so that that's the gap that we fill. Yeah. And other than being shorter and simpler, uh, is there anything different about how small to medium enterprises should be looking at AI governance compared to the major big firms? Yeah, a couple of things.

They need to be much better about asking vendors the right questions because typically, you know, they're not building. Mm-hmm. Uh, they're buying almost all of their ai. And then also, you know, at smaller organizations, a few people do a lot.

So the roles and responsibilities, sometimes you just don't have, uh, either you don't have the right expertise or you're just not clear on where those lines are between, you know, expectations of who's gonna be doing what. And so those are two of the main areas that come up frequently at smaller organizations. What are you hearing from the, the organization that you're talking to, either in terms of just generally how interested they are in AI governance or what are the, the big problems that they're highlighting for you?

Yeah, so I think there is a lot of interest in this now. I, I think, you know, people at all kinds of organizations are asking about how to run a program and, you know, it's, it's, there's a lot of kind of typical questions around how do we inventory what we already have? So for example. If we are using some sort of platform that uses agents within the platform, is that part of a master inventory or is it do, do we just log that we're using this system?

You know, those sorts of really concrete questions around a question around issues like inventories. Then already mentioned, you know, how, who owns governance internally? Mm-hmm. How do we ensure that nothing is slipping through the cracks?

It's really important, especially with regulatory reporting as well as kind of. Customer expectations coming into place. And then procurement of course is super hot mapping, existing privacy risk model, risk management requirements. So I, I'd say generally, you know, there, there's.

Compared to a few years ago when I used to speak with organizations, we used to have to make the case for why you should care about AI governance. I think that case has been made, you know, for a variety of reasons, and now it's more, yeah. What is it that, that you find is most pushing those organizations to feel the need for AI governance? Is it regulation or perception about coming, regulation, customer trust, what is it that that's primarily on their mind now?

I think it's a few things. So I think that there's str, there's pressure, board pressure and CEO pressure frequently. On making sure that you're using emerging technology to drive value or to cut costs. So I think that that's a big one.

Definitely risk and compliance, you know, the regulatory stick. A fear of missing out. I think you see a lot of other companies adopting AI are talking about how they're adopting ai, and so I think in every industry and in companies of all sizes. There's a little bit of, you know, we, we should be using this regard.

We should at least be experimenting with this to see where it goes. And I find that we frequently have to kind of say, okay, just because it's out there doesn't mean you need to be using it. You need to kind of go, go slow, go methodically. It has, of course, significant potential, but I do think that there is a little bit of anxiety around, uh, competitive pressure driving it as well.

Those seem to be though reasons why companies are investing in ai, but that would seem to suggest that, that they might be disregarding the need for governance and, and responsibility and and risk management. So where does that pressure come from? So I think it's. A need for quality and predictability because especially with generative ai, it's, I think people correctly don't trust, you know, you don't want to give an AI agent, for example, browser access or tool access or, or a lot of memory.

And without kind of understanding how you're kind of, without having visibility into it, without having, being able to control it. And so I think organizations see that. Another thing that's driving it is that just shadow AI use. So the concern that, okay, we could just say, you know, all this things I just mentioned, you could just say we don't allow AI use in our company.

Your employees are probably going to be using it anyway, and they're probably gonna be using it in ways that you don't want them to be using it. And so I think that that's also, in addition to the regulatory push, really driving a need for responsible ai. Let me dive in a little bit more concretely to some of the, the work that you've done. So the, the Council's AI governance playbook talks about four focus areas, and correct me if I, if I get it wrong, but strategy, risk, and compliance.

Workforce and literacy and then procurement and management across those areas. Where do you find there's the biggest gap? That organizations are not investing enough? Yeah, so, so my view is that that would be.

Literacy and workforce. We already talked about how strategy is, is getting attention from CEOs. Risk is getting attention because of the regulatory stick. And then, you know, procurement gets attention usually when something's not working the way it should.

Literacy is often kind of a longer term play. It's more like more vitamins than painkillers, so it tends to fall through the cracks a little bit. It's, you know, organizations say, okay, we'll add this to our training module and we'll go from there. And.

You can't really scale an AI governance program without literacy. It's, it's really kind of a, mm-hmm. A, a key piece. You have pretty good external resources now, whether, you know, Udemy, Coursera, universities, and then AI labs themselves kind of put out pretty good content.

But what you really need is to get everyone in your organization thinking critically around what the opportunities are, and then also what the risks are. Mm-hmm. Uh, and you can do that. If you can learn by doing so, you need to involve them in pilots.

You need to involve them in documenting what they've learned. And people are already very busy, very stretched. Mm-hmm. Uh, it's kind of an uncertain economy, so it's really difficult to resource this kind of big AI literacy, change management push at most organizations.

So we often see that as, as kind of the, the, the missing piece. So let's talk a little more about the educational side of it. You ha have actually been teaching AI governance at Purdue, and so I'm curious what your experience has been with the students in terms of, you know, what they struggle with, what they get wrong or what they potentially get right. Yeah, and I'd be curious to how this maps to yours.

So I've, as you mentioned, you know, I've been helping with the Grail lab at Purdue, building out their AI auditing coursework at the master's level. And then I've also worked with students in, in various capacities at, at other masters in law programs. Uh, and generally I feel that they have. Positive visions and are optimistic about what AI can achieve, which isn't always the case, you know, at at large organizations with people that are much further along in their careers.

Students also, I feel, tend to understand the companies and models really well. Like they, they tinker far more than I expected that they would, and they also have a good grasp of the pitfalls at for a given use case. You know, if you're brainstorming, for example, in a given use case fit, is this fit for purpose? How do we understand what it's doing?

Could it be biased? Those sorts of questions seem to come naturally. On the flip side, I think. The areas where they tend to learn a lot and grow a lot is organizational governance, uh, and then also networks and networks of people.

I think governance is organizational. You know, we're asking questions of who approves what, what gets inventoried, how do you assess a vendor? So even if you have good instincts at the use case, you still need to learn. All of the kind of governance tools that you would use at a big organization.

And then also there is a lot of that organizational governance is interpersonal skills. You know, how are you talking to people in different functions? How are you kind of bringing them on board with your ideas? And then when I see networks, I mean that students particularly.

Early in their careers tend to underestimate, you know, the extent to which developing long-term trusted networks and relationships is important. Mm-hmm. Not just generally, but also for your practice of AI governance. So, you know, who are you gonna call to bounce an idea or an issue off of?

Or are you a regular at a chapter meeting of, of some industry organization where you can quickly address, you know, a new issue or even just keep. Keep current with AI governance approaches. Mm-hmm. Uh, so those are the areas they, they tend to learn a lot in.

Yeah. Your distinction between the, the organizational structures and, and the use case and the tools is, is interesting because some of the structures, like for example, the ISO 40 2001 standard are about management systems. So what, what exactly is the gap? If you understand those frameworks and understand the organizational piece, what, what exactly is it?

This is not just for students, but for. People in companies as well. What are the more operational tools that they need to understand better? Yeah, so I think that ev, every organization has a different way of doing things.

So you can't just overlay something like ISO 40 2001 onto an organization. You're going to have existing functions, existing personalities, as well as existing governance from cloud and, and kind of. Pre other IT frameworks, data governance frameworks, cybersecurity frameworks. And so there's this constant dance of making the case for why this is needed and what's new.

And then also kind of making sure that the processes that you put in place are sustainable. And a lot of that, you know, as I mentioned earlier, comes down to having good relationships within the organization. Being able to communicate things to different audiences and being able to understand, you know, not everybody needs to be a computer science PhD, but being able mm-hmm. Able to communicate why this is important requires just a little bit of technical depth, so.

Mm-hmm. So that, that's kind of where the organizational governance theory, where the rubber hits the road. Mm-hmm. You've done some work on looking at application of AI governance in different sectors, and in particular, I wanna ask you about healthcare.

I know you, you've done some analysis and some writing. W what's different about healthcare in particular, and, and is it a, an effective or, or an appropriate or good area to do this work? Yeah, so I think healthcare has already done a great job of, because of necessity. You know, it's, it's, we trust the healthcare system, or we tend tend to at least trust components of the healthcare system.

And that trust is a really valuable kind of infrastructure that you might not have in other industries, and it would make sense to kind of build on. So I think there's, there's a really specific, at least in the US, a really specific regulatory architecture. There's hipaa, there's kind of FDA, the FDA's really taken the lead on some of these AI issues as they pertain to medical devices. And then there's also state level requirements.

Mm-hmm. So you already know what you're working with in terms of regulations, which isn't always the case in, in other industries. You're also, you know, these are, these are healthcare organizations are very complex. They're used to handling a wide variety of use cases with a lot of kind of variability.

So, you know, you, you might have to. For example, assess third party risk in a diagnostic model, a scheduling tool, uh, or some sort of ambient scribe, you, you are, are already accustomed to looking at a number of different use cases. Mm-hmm. It's not one size fits all.

Then I think that they are, they tend to be already aware of. It's not just the system, but it's where the system is used by a physician or other medical professional that really helps determine outcomes. And so again, that's already embedded in their knowledge. And then finally, they, they already have a lot of emphasis on, on post-deployment monitoring of the systems.

And so there already exist a, a, a number of. Good pieces upon which you could develop. Mm-hmm. You know, and so some sort of assurance model, that's not to say of course, you know, there, there's, there are a lot of different kinds of, or healthcare organizations, they kind of govern things in different ways, but generally there are these kind of existing pieces that, that lend themselves to.

Mm-hmm. And ai, AI assurance model. And also there's a desire, you know, you, you have this trust that's built up over kind of centuries to be worthy of that trust, right? Mm-hmm.

So there's also a real desire to do it within these organizations. Yeah. You, you mentioned AI assurance model. What, what exactly is an AI assurance model?

And, and maybe talk a little bit about what you're seeing in terms of the development of that assurance ecosystem more broadly. Yeah, so I think assurance is the connective tissue between an organization and other organizations. So it's how you would demonstrate to regulators, to customers, to the other organizations you do business with, and then to the public more generally, that your internal governance is real and that's, that it's effective. So in some cases that can be self-reported.

You know, you, it might be kind of a low risk use case or you might be the type of organization that doesn't require a lot of additional governance. You're kind of relying on the vendor to do a lot of that. And then in other cases it's, you know, in higher risk use cases, it can be through an audit to have someone external come and look at your processes. So.

I think to your point about the assurance ecosystem for ai, it's important to be realistic on, on what you're able to achieve. So I think at this stage it would be good to just understand. We want people to share information with each other, uh, so that they can gauge and price risk. So we're not going to be able to eliminate risk the, the way that the speed at which technology is moving, we're probably gonna have an increasing gap in risk, but.

Can we at least be able to have clear information that everybody kind of agrees with and then be able to, to either accept it or at least price it and potentially transfer it. Mm-hmm. So that's kind of where AI assurance comes in. Now I wanna be really clear about, you know, it's not an alternative to other AI governance methods, so it's not, you know, regulation is really important and has kind of democratic legitimacy and regulators of course can incorporate assurance tools.

But really it's kind of its own beast. And then internal governance is really where all the work is happening in your organization. So assurance is separate and you can have an assurance ecosystem regardless of what kind of legal regime or regulatory regime you have. But the reason I really like it is the tech is moving quickly.

You need really specialized expertise to, to govern it. And so it seems like a lot for internal compliance teams and regulators to tackle alone. If you kind of bring researchers, Professors, standard setters, auditors, everybody on board, then you have a better chance of kind of closing that information symmetry and governing AI better. There are now a number of firms out there that will do AI audits, and you've mentioned you're teaching and master's programs that are training AI auditors and all the major audit and assurance firms, you know, have practices in this area for a company, especially the the small to medium companies that, that you're focused on.

Can they easily now find someone that they can trust to do those kinds of assessments? For them, I'd say it, it really varies by industry and by what kind of assessment they're looking for. I think in regulations really do drive markets in many of these cases, so the New York City local law around automated employment decision making has a really standardized report format, and so. That's really kinda stimulated the development of a market around it.

In other cases, smaller organizations don't necessarily know what to ask for. Mm-hmm. And they also to, to at risk of kind of getting in the weeds aren't clear about what exactly is being assessed or certified. Because once you take, especially if you take a foundation model or, or a kind of system from a big lab.

If you make enough changes to it, it's really quite different from the model that you purchased. And so is that a separate kind of object of analysis or is it, are there elements of it that you can kind of break off? These are questions that are really difficult for smaller organizations in particular. And so I think that that it kind of re, it's more helpful for really common industry use cases where.

You have the same patterns over and over again. And so a smaller organization can kind of piggyback on that and say, okay, I see, you know, I'm a small bank or small financial services organization. I've seen the kinds of audits that a big bank does, and so I'm gonna piggyback off of that. Mm-hmm.

So, yeah, to answer your question, it kind of really varies. Um, but, but, but you're right. Increasingly, the ecosystem has, has, is, is developing well and has come a long way over the past few years. You made an interesting point about foundation models because a lot of our initial ideas about assurance are, you know, you build some machine learning solution for an application and we're gonna test it for bias or for accuracy or whatever else.

But now increasingly, we're in a world where companies are building on top of these foundation models that they have no control over. But as you point out, they may be modifying them. So how can the assurance ecosystem develop? To help companies that are increasingly using these generative AI structures that, that are gonna look fairly different than the kind of machine learning approaches that original kinds of, uh, audit world came about around.

So I think that, you know, at, at its surface, theoretically, it seems like a really good model for assurance design because, you know, the AI labs have understand a model's capabilities better than anybody. The organizations that are deploying them. Are best positioned to understand fitness for purpose, and then end users, whether they're individuals or companies, they can gauge whether they're actually getting value. And so there is an incentive and kind of need for all of these different players to share information.

And you know, as you're kind of alluding to, there's also a couple of complications. So. There's only a handful of cutting edge AI labs. The deploying organizations tend to be a bit limited in what they can demand from those cutting edge labs.

And then of course, the rate of change is quite significant. And so you can say things like, okay, everybody needs to share information in these specified formats. Labs need to do model testing, deployers need to be responsible for the context of deployment. And you have regimes like the EU AI Act, trying to make that very explicit, but.

You know these questions of, of. What is the thing that we are trying to provide assurance for in, in, in the case of a deploying organization or even what level of third party access is, should be allowed for a cutting edge model, you know, at what point does that become a security risk? And we've tried a number of different approaches to this. We tried with AI Safety Institutes kind of peering in and doing analyses of cutting edge models.

We've tried providing really structured access to a handful of researchers. Then a lot of the time we also just rely on the mm-hmm. Labs themselves to, to, to evaluate their models. And so I'm not sure that those open questions are going, we're going to see a resolution for those anytime soon, but I am optimistic that.

If we can start with the most common use cases in the largest industries that give us some common patterns where you have, you know, the sector specific nuance, then you can have the economies of scale to make mm-hmm. Assurance exercise worthwhile, and then smaller players can kind of piggyback on that. Yeah. Along those lines, I asked you before about healthcare, but you also were a co-author of a paper in 2025 that proposed a certification framework in financial services and, and actually did a pilot.

I'm curious what you found in that work. Yeah, so this was kind of pilot that we did with the Standards Council of Canada. To test some AI system level certification criteria with a bank that had a customer facing use case? I'd say the, probably the main takeaways were, you know, we had, we had.

Really reduced the certification criteria and simplified them as much as we could. Mm-hmm. But still, you know, just, just by, by the nature of AI and the kind of differences between certifying something like a dynamic system, we found that our certification criteria were interpreted in different ways by different actors, and that we needed to kind of simplify and standardize the scheme. Even further and leave some discretion to the auditor.

And this again, starts, starts to get back at why it's an ecosystem and why auditor education is so important. The other kind of surprising finding for me was that our public sector partner, the Standards Council of Canada is. One of the most forward thinking organizations on this topic, kind of anywhere in the world, but even they, at the time of the pilot didn't have the expertise available to play the standard setter role for something that dynamic. Usually, you know, the things that are being certified are, are quite static, so it really highlighted the need for more capacity in the public sector.

More training, more technical expertise to be able to oversee, even oversee a, a certification program. First is, I think it's really important to, you know, in, in this AI and responsible AI and risk world, we tend to focus a lot on risk, and that's understandable, but I'd really start with. That positive vision? Mm-hmm.

I think it's, it's, it helps you change your mindset to problem solving and kind of creativity. And so really trying to understand, okay, what are, what are we trying to do with ai? Is there a way that we can do this in a way that, and that's significantly less risky? So getting a sense of the strategy, getting the sense of how it's gonna help you provide value, I think is really important for any risk, risk or compliance professional.

Beyond that, I'd say getting. Observing how your IT governance is working right now. So you'll already have data governance, information security, analytics, you know, risk. All of these functions you'll already have grappled with some of the major issues.

We discussed it with the council's playbook, whether it's third party risk or visibility. For who's using these systems across the organization. And so just by observing how is this actually working and how can we kind of overlay some of the AI concerns on top of it. You'll also get a sense of which ecosystems you're already plugged into.

You know, do you, some organizations might really closely follow FDA guidance, others, you know, if you're a government contractor, maybe you're following the Pentagon's guidance. Are you kind of. Really close to NIST or really close to iso. I think those sorts of questions can help determine like the path of re least resistance for your organization in getting really good AI governance guidance.

So that's what I'd recommend. Great Var, thank you for all of your insights. Really appreciate it. Thanks for having me, Kevin.

This has been the Road to Accountable ai. If you like what you're hearing, please give us a good review and check out my substack for more insights on AI accountability. Thank you for listening. If you want to go deeper on AI governance, trust and responsibility with me and other distinguished faculty of the world's top business school, sign up for the next cohort of Wharton's Strategies for Accountable AI online executive education program, featuring live interaction with faculty expert interviews and custom designed asynchronous content.

Join fellow business leaders to learn valuable skills you can put to work in your organization. Visit exec ed.warden.upenn.

edu/acai for full details. I hope to see you there.