Are leaders deploying AI faster than they can effectively govern it with Zahra Shah

Speaker A: M

Speaker B: Not every company needs a large language model. So when you are recommending a solution then you have to look at, okay, if they're solving an hr, uh, problem, perhaps they need only a small targeted model and then they can fine tune it for their requirements because all the different models are suitable for different tasks. So then you'd make sure that you know the person who's recommending the model to you. The recommendation is based on what is good for the company.

Speaker A: Hello and welcome to Tech Tomorrow. I'm David Ellerman, chief of Software Engineering at Zulker. Each episode we tackle a big question to help you make sense of the fast changing world of emerging tech. And today I'm joined by Zahra Shah, an expert in responsible AI and frontier technology. Zara serves on the board of Nexaquanta, an AI transformation consultancy firm, and is also chair and a founding member of UK AI's Women in AI working group. She has extensive experience advising companies on AI governance and ethics and is passionate about creating responsible AI policy. So who better to help me answer today's question, are, uh, leaders deploying AI faster than they can effectively govern it?

Speaker B: What we have really noticed is that if you're using AI tools to deploy code, literally they're changing on a daily basis. So whereas governance takes time to really keep up with the rapid changes. But one of our key approaches to governance is to make sure that we focus on responsible AI framework. So we include looking at, ah, ethics by design, safety by design. Actually a lot of companies we notice, do not even realize that they're already using AI. A lot of their employees are already using what we call shadow AI without their knowledge.

Speaker A: Shadow AI. Yeah.

Speaker B: Yes.

Speaker A: Emerging is a term, isn't it?

Speaker B: Yes. Uh, and then the other thing, uh, we also experience is that companies do not really know what data is being used, which is I would say the most crucial aspect of an effective AI system. So it's almost helping and guiding companies to find and select the right use case because it's almost like if you select the wrong problem to solve, you're never going to get a return on investment. So that is something, I suppose for some organizations it is painful, but it is better to do that at the beginning, at the outset, where you do the due diligence, you do what we call requirements analysis, you come up with the proper use case, you look at your strategy, you ensure that you have responsible AI principles that you are adopting, things like explainability, transparency. Are, uh, you complying with data privacy regulations? You know, you're making sure that Whether you're building it in house or you're going with an external vendor, uh, you have to look at things like bias. Do you have a strategy for addressing that? How do you manage if that particular system is hallucinating? What will you do? How do you mitigate that? So you're thinking about all of that in the beginning.

Speaker A: I mean, that makes perfect sense. To be able to understand and prioritize the risk means that you have to understand what those risks actually are. So there's a degree of understanding. And one of the problems with AI machine learning over the last, since it's become more corporately adopted, maybe over the last 20, 25 years, but particularly just in the last few years since the sort of the transformer models and LLMs and that stuff, is that there's a certain degree of black boxness around it. And that's not just exclusively to those, but it's been before, explainable AI transparency, repeatability, all those things are kind of hard to do. And if a board wants to adopt a, uh, particular AI approach because they've been sold on an idea, it kind of suggests that they've got to actually understand, there's got to be some initial movement to actually say, well, okay, here's the promise, and then let's now build in all of the understanding into that in order to do what you just said.

Speaker B: Yes, and also, I know for some people might be overwhelming. So another approach that we suggest is to do a pilot first, perhaps in the case of a knowledge assistant or an AI agent that helps your employees with, let's say, HR policies. So something that is low risk and you can try out the pilot, so it builds confidence so you can look at all of the risk mitigating factors. When you're developing the pilot, you ensure that you're following your AI strategy. You have a, uh, responsible AI policy in place, you have somebody who is overseeing it. If you're not big enough to have a proper board to an AI ethics committee, so at least there should be somebody on your existing board who has that responsibility. And if, let's say you don't have somebody with the expertise, either you can get external expertise or you can train somebody. And then as you're deploying the pilot, you also will need to train your staff members. And what we find is actually often there's a big gap between what the system can do and how employees use it. And if they're not trained on how to use a system correctly, I still see a lot of people, they're using AI system like Google 2.0. So they need education and training. How should I ask the right kind of question, which we call or refer to as prompt engineering? You need to provide the right context. You need to provide the right level of guardrails when you are questioning the system. When you're doing a pilot, it gives you the opportunity to kind of test that and then you learn from it. And you can give yourself as an organization some leeway. Instead of the KPIs for the pilot being, oh, we must get return on investment straight away. Perhaps the KPI should be. Did our, uh, XYZ guardrails work well or did we mitigate specific biases so you can have realistic KPIs at the pilot stage to give everybody the breathing room to come up to speed?

Speaker A: And I was just thinking for your example there, I mean, you gave the example of an HR system and obviously the. That's a great example because there's a lot of personal information in there. For anybody listening to this, that is gonna be thinking, okay, so I hear a lot about responsible AI or ethics. What does it mean on a daily basis? And you mentioned guardrails and bias in terms of there's a project starting next Monday which is, as you say, is gonna be a pilot to do something, maybe some sort of chat assistant within the HR system. What would a sort of responsible tech on that team try to inject and want to expect from it?

Speaker B: So you first would look at the data. What data is being used in that organization? Because a lot of organizations have multiple databases. And if you are using data which is what is classified as personal data, then do you have permission or consent of your employees to. Because there might be, let's say, medical information, which is very sensitive. There might be information that some employees do not wish to share with other employees in the organization for whatever reason. When you are designing your data sets, then you have to make sure that you take all of that into account. You make sure that the data is accurate. There's no missing information. When you're labeling data, you also make sure that it reflects diversity. So you'd have to take that into account when you're designing your data set. And then of course, now we have the EU AI Act. So if the organization has a footprint across Europe, then they will need to also comply with the EU AI Act. So then when you're designing it, you have to make sure that you, you are complying with all the necessary regulations. And then when you're selecting the AI model or whether you're developing it in house, or for example, if you're using something like Claude. So then you have to also figure out, where will I store, for example, the information. You obviously have cost constraints. Then you would also look at energy efficiency, because some models might be, let's say, more cost effective than others, but they're very expensive to run. So you'd have to look at all of those considerations. You don't want to be stuck in a situation where you ended up going with a certain vendor because you had a good price, but you didn't realize you were tied to that vendor and now you cannot go with any other vendor. And then you realize that it's taking up so much of your memory space, it's very expensive to run for the company. And then you also realize that you're not even using all of the capabilities of the model.

Speaker A: Vendor locking is one of the topics we come back to on this show. And yes, it's far more common than most leaders realize. It usually happens not through a single dramatic decision, but through a series of small, sensible sounding ones. A team picks a model that solves an immediate problem. Data starts flowing into a proprietary fine tuning pipeline. APIs get baked into the product. Internal tooling, prompts, evaluation harnesses. All of it gradually shapes itself around one provider's quirks. By the time anybody steps back and asks the question, switching costs can quietly have become substantial. When a leader finds themselves locked into a specific vendor, uh, the route back to a more neutral position starts with abstraction. Anytime you call a model in production, that call should go through a thin internal layer of your own, one that hides the specifics of which provider you're using underneath. If you. It also means owning your prompts, your evaluations, and especially your data so you can repoint the same workflow at a different model and measure honestly what you lose or gain. And it means routinely running parallel benchmarks against at least one alternative. So you always have a credible plan B. It's important to remember that every company has unique needs and there are many vendors and solutions available. The key is to find the best fit for each specific context.

Speaker B: Not every company needs a large language model. So when you are recommending a solution, then you have to look at, okay, if they're solving an hr, uh, problem, perhaps they only need like a small model. They don't need something that does X, Y and Z, because all the different models are suitable for different tasks. So then you'd make sure that you know the person who's recommending the model to you and an application to you. Is not just doing it for monetary concerns. You know, that the recommendation is based on what is good for the company. And it might be that they don't need a very fancy model, perhaps they need only a small targeted model and then they can fine tune it for their requirements. And then the other thing you'll also have to look at, there's something, a technique that we use, referred to as retrieval augmentation rag. In very simple terms, what that means is the rag can actually help to reduce or mitigate hallucinations. So for example, in some models where there's more risk of hallucinating, you can use this rag methodology where you use real time data to mitigate that risk. So there are various things you have to take into account to make sure that you are recommending, uh, the right solution to that company and you're not just recommending the solution that's gonna make you the most money. So you have to understand your risks and then prioritize the key risks and, and then have a mitigation process and then monitor it and track it and monitor it.

Speaker A: Many things about the regulatory compliance or putting the right guardrails in place, checking for biases, it's such a fundamental job that it's hard to see that you would do this without being intrinsic to the project. And yet we see some sort of ethics theater in some companies where there'll be something, where there's a sort of a checkbox of things that are applied after the fact. Most, maybe a system might be checked way down the line. Everything that you've just said is like, okay, sleeves up, day one. This is how we're going to work together on doing this. And there are a number of challenges. Like a system in terms of its fairness, accessibility and maybe even its sustainability footprint are all things that seem to me in this day and age to be essential to put in right at the beginning. And where people think, well, we have an ethics officer. It's like, what is that? You know, this is intrinsic to the actual work.

Speaker B: Definitely. And also I think because of the nature and the speed of advancement in AI, people are afraid as well of the accountability. What we have seen is that nobody is willing to take accountability. The legal person thinks it's the technology person's responsibility. The technology person thinks it's, you know, the product person's responsibility. To be very honest, AI is, uh, a tool now. It's being used in every part of the company. So if it's used in finance, the responsibility still remains with the director of finance. If it's used within the legal department, then the responsibility sits with whoever is in charge of the legal department. Similarly with marketing or Sales or hr, you can't say that things went bad because of the AI system, because the AI system is there to help you. So another way of dealing with that accountability issue is to have a committee of people within the different departments working together. They can share the responsibility as well and then do as I suggested, almost like a pre mortem look at what can go wrong. And often the person who understands that business function is best placed to figure out the various risks of what could go wrong from a business point of view. And perhaps a person who is, uh, has expertise in AI can then figure out what can go wrong in terms of the system point of view. And the legal expert can figure out, oh my God, we can be fined or we can go to jail. So you would need people from the different, I would say, areas of expertise,

Speaker A: uh, within a committee, multidisciplinary team.

Speaker B: Exactly. And in this way, you know, it can also help with the fear because then you're working together by addressing risks and at the outset you prevent the risks from becoming issues. And you also take this view that it is a challenging thing. It's not easy because it's changing so quickly. So you give people a bit of leeway.

Speaker A: You touched on something earlier that I think is worth diving into a little bit. You talked about the EU AI act and their relationship to, uh, having to satisfy that there's a host of different sort of ethics, ethical frameworks that span from homegrown through to maybe the oecd have one, NIST have one. But just coming back to it, there is, I think, quite a confusing sort of plethora of either regulatory standards that one might have to build against or guidance in terms of how you're supposed to behave and operate. And it's better than none. But I think that some people might be inundated with potential opportunity and I think it may change the way people react in the light of how they respond to governments. We in this kind of like, uh, unique situation in the UK where we're seeing the regulatory drift that's happening in the U.S. and the tightening up of more regulation in the EU. Uh, and I just wondered what your feelings were. You know, you're a C level person within a company in the uk, say as an example, because we're in that sort of middle ground.

Speaker B: It is a very interesting kind of global situation, actually. We have the US which is sort of almost with very little or minimal regulation. When it comes to AI. And then you have the eu, which some people would argue as overregulation. Because the EU AI act also follows a risk based approach. It's not a principles based approach. And that approach is very hard to enforce when it comes to AI systems because AI systems are very dynamic, so the risk changes depending on how the system is being used. So a low risk system, if it's not used correctly, can become a high risk system very quickly. And I feel that the UK is in a very interesting situation. UK can utilize this opportunity to come up with its own responsible AI regulation which is balanced, which is I would say taking the middle way between the US and Europe and come up with our own regulation which is principles based, which is innovation friendly, which protects minorities and vulnerable people in this country. Because you've seen what happened with grok, AI and Ofcom had to take decision to investigate X. And then there were changes to the Online Safety act. And now uh, currently there's a consultation going on whether we should ban social media for young people. So a lot of these issues that uh, will keep coming up because of the advancement in AI. And then obviously now because we look at what's happening in terms of the geopolitical situation in the Middle east, it makes you realize that we also have to look at sovereign AI. That UK must ensure that our sensitive aspects of our country, literally our sovereignty, relies on making sure that our sensitive information data AI infrastructure should really be supported by UK companies. So that's something that they have to think about. And then you have to also think about making sure that the benefits of AI are equally divided across Britain, that we don't just focus on London and also that we really focus on supporting the public to upskill people. That's a huge challenge. We'll need to be able to upskill people across the UK and particularly in underserved parts of the UK regions in the north, and make sure that when we are coming up with data centers or other AI related projects under the UK government's AI Opportunities Action Plan, and I know that the action plan does take into account regional hubs, which is good, but this is something that they also have to make sure that we have to be careful about causing an AI divide which within the country that might cause division.

Speaker A: There are a lot of polarizing predictions about how AI will change the job market. And the reality is we're not seeing wholesale displacement, but we are seeing a quieter, more uneven shift. Tasks within roles are being absorbed by AI rather than entire jobs being eliminated and the people who learn to work alongside these tools are pulling steadily ahead of that those who don't. The risk is a slow widening of the gap between the AI fluent and everybody else when it comes to what leaders could be doing to upskill staff, both technical and non technical. The mistake I see most often is treating upskilling as a training course rather than a capability program. A two hour prompt engineering session does very little on its own. What works is giving people meaningful problems to solve with these tools in their working context and with proper coaching and time to experiment safely and critically. That has to extend well beyond the engineering organization, finance, legal, hr, uh, marketing and operations. Even anyone whose job touches information is touching AI now, whether they know it or not. Training and upskilling are of course important. But another key factor in successful AI adoption is trust.

Speaker B: The other trend that's happening globally is that people also see seeing this kind of erosion of trust in AI because there are a lot of companies who are not using ethical AI principles. And as people discover that and they've had unfortunately bad experiences, there will come a time where people will want to look for companies that they trust. And uh, in that scenario, UK can take the leading position. This can be a country where people can companies where they feel, let's say in the US you feel that there are certain companies, you can no longer trust those companies because they don't follow any regulation. But then if UK has this regulation, perhaps even what UK AI, ah, is working on, like a trust mark or almost like a sort of a, uh, trademark that if companies are developed or trained or built in the uk, that means you can trust them because they comply with XY regulation. So we are uniquely positioned actually to lead in responsible AI globally and be the light actually for other countries. UK can be the country that takes that, I would say, brave step to make sure that we do things the right way.

Speaker A: Which leads us perfectly to the central question of the episode. So in your opinion, Zahra, uh, do you think that leaders are deploying AI faster, uh, than they can effectively govern it?

Speaker B: I would say in some situations, yes. And my aim is not to blame anybody or criticize anybody, but it's just the nature of the situation. It's changing so rapidly that a lot of leaders, they feel that they have no option. And there are companies or nations that are basing decisions on fomo. Literally, they're not making decisions based on rational objectivity. They're kind of driven by this fear, oh God, if I don't do it, then I'll be left behind. So it's a very difficult situation. But actually, I was very impressed by what I recently saw with the government of Singapore. They recently launched in January under the World Economic Forum, the world's first governance framework for agentic AI. I kind of analyzed that framework, and it's a very good framework because it's based on an agile model. And it also looks into kind of an iterative approach, some of the principles that I mentioned, it does take that into account. And I was thinking that perhaps as we work, because UK is part of the Commonwealth, perhaps the UK could work with other Commonwealth countries like Singapore, and, uh, we can share best practice. And UK has always taken sort of a leading position in terms of legislation and regulation, because that's a strong point. So we can do that in terms of responsible AI and work with other countries, and some other countries have expertise in perhaps helping us in the successful implementation of UK's AI opportunities action Plan. Because we need skills, we need people who can sort of help us in terms of using sustainable sources of energy, which has become, as you know, with the geopolitical situation, most countries realize that they have to look at, you know, alternative sources of energy. So a lot of these things, these challenges, we can kind of address them by working together with other countries, foreign.

Speaker A: Thank you for listening to season three of Tech Tomorrow, brought to you by Zulker. If you'd like to learn more about what we do, you can find links to our, uh, website and more resources in this episode, Show Notes. You can also listen back to all previous episodes right now in your podcast app of choice. Until next time,