The B2B Podcast Index
Index
All categories
MarketingSalesSaaSFinanceHROpsLeadershipCustomer SuccessAI & DataProductStartups & FoundersRevOpsEngineering & DevTools
MethodologySubmit
Best of:MarketingSalesSaaSFinanceHROpsLeadershipCustomer SuccessAI & DataProductStartups & FoundersRevOpsEngineering & DevTools
An independent project byFame
All showsSearchBest episodesGuestsInsightsMethodologySubmit a podcast
Index/Leadership/Tech Talks
Tech Talks artwork

If Your AI Agent Goes Wrong, Whose Name Is On It?

Tech Talks · 2026-10-30 · 35 min

0:00--:--

Key moments - from our scoring

Substance score

50 / 100

Five dimensions, 20 points each

Insight Density10 / 20
Originality8 / 20
Guest Caliber11 / 20
Specificity & Evidence12 / 20
Conversational Craft9 / 20

David Savage opens with a thought-provoking discussion about Megan Kaywood's LinkedIn post on AI consciousness and decision-making, establishing that agentic AI systems are fundamentally non-deterministic - identical inputs don't guarantee identical outputs, making explainability and accountability critical challenges. The episode then features Faye Murray, field Chief Data Officer EMEA at DataIQ, who discusses why most organizations struggle to achieve ROI from AI projects, particularly at scale. The key insight is that traditional machine learning delivered measurable value in fraud detection, credit scoring, and demand forecasting, but organizations approach newer AI (generative and agentic) backward - starting with technology pilots rather than defined business problems. Murray emphasizes that agentic AI governance differs fundamentally from traditional ML, requiring organizational redesign toward "governed self-service" models where business teams can innovate within centralized guardrails rather than relying on bottlenecked centralized teams. She cites examples like ERS Insurance reducing underwriting assessment time from 5-10 minutes to under one minute with human-in-the-loop agents. The discussion covers the hidden costs of agentic systems (API tokens, human review labor, infrastructure, vector retrieval, system integration), the 87% of British CIOs reporting employees building agents faster than IT can manage, and the shadow AI problem where employees bypass organizational guardrails. For regulated industries, the stakes are particularly high: agents are extensions of the people building them, and organizations need visibility across tool deployment, data access, system connections, and automated actions.

Key takeaways

  • →Agentic AI systems are non-deterministic and non-reproducible - you cannot reliably explain why the same agent made different decisions on different days, creating accountability gaps especially in regulated industries.
  • →Organizations fail to achieve AI ROI not because the technology doesn't work, but because they start with technology pilots instead of clearly defined business problems, baselines, and accountability structures.
  • →The economics of agentic AI at scale include hidden costs beyond API tokens: human review labor, error handling, infrastructure, vector retrieval, system integration, and organizational redesign - costs invisible in proofs of concept.
  • →Shadow AI emerges when employees either feel constrained by slow/clunky enterprise-approved tools or feel unsafe using them, pushing innovation underground; the solution is governed self-service (pre-approved models, secure sandboxes, smooth governance processes) not outright bans.
  • →Agents are extensions of their builders' decision-making; organizations must understand agents not as cold calculators but as systems reflecting the values and reasoning of their creators, requiring involvement from business teams, not just technologists.

About Tech Talks

Hosted by Harvey Nash. The show scores 58.5/100 on substance across its analysed episodes. See the full show profile.

Guests

Faye Murray

Topics in this episode

shadow AIAgentic AIAI governanceNon-deterministic systemsDataIQ platformOrganized self-service modelMegan KaywoodERS InsuranceFlower BankAI consciousness

Questions this episode answers

What's the difference between ROI from traditional machine learning versus agentic AI?

Traditional ML like credit scoring and fraud detection has proven ROI over years, with clear baselines and measurable improvements. Agentic AI shows less success at scale because organizations often lack defined business problems, measured baselines, clear accountability owners, and understanding of hidden costs beyond API tokens (human review labor, infrastructure, system integration).

How can organizations prevent employees from building AI agents outside their guardrails?

Rather than enforcing bans (which push risk underground), organizations should implement "governed self-service": providing pre-approved models, secure sandboxes, clear data connections, and smooth governance processes that make responsible innovation easier than circumventing controls. DataIQ research found 87% of British CIOs report employees building agents faster than IT can manage.

What accountability and governance challenges does agentic AI create?

Agentic AI systems are non-deterministic - identical inputs produce different outputs on different days - making explainability impossible. This is fundamentally different from traditional ML and creates critical accountability gaps in regulated industries like finance where you cannot reliably recreate or explain why an agent made a specific decision.

Why do agentic AI projects often fail to deliver ROI despite successful proofs of concept?

Pilots show the model works but miss organizational realities: lack of measured process baselines, missing cost components (human review time, infrastructure, vector retrieval, system integration), governance requirements for data pipelines and monitoring, and the need for organizational redesign to scale safely - costs invisible in pilots.

What organizational structure does Faye Murray recommend for scaling AI safely?

She recommends "governed self-service" where a centralized team creates a secure sandbox with pre-approved models, governance standards, and templates, then partners with business units as co-builders rather than order-takers. This empowers business teams to innovate within guardrails while maintaining compliance and reducing shadow AI.

What our scoring noted

Our reviewer’s read on each dimension, with quotes from the episode.

Insight Density

10 / 20

The interview's second half delivers genuinely useful frameworks - tiered governance proportionate to risk, governed self-service models, the non-determinism accountability problem - but a meandering 15-minute opening on AI consciousness and recycled platitudes (garbage-in/garbage-out, shadow AI basics) drag the density down considerably.

the risk is also higher because you can't then recreate exactly how those decisions have been made at that point in time
you need to make the governance proportionate to the risk

Originality

8 / 20

The 'governed self-service' framing and applying non-determinism as a governance accountability lens are moderately fresh, but the bulk of ideas - centers of excellence, sandbox environments, tiered risk reviews - are well-worn in enterprise AI discourse, and the opening AI-consciousness segment contributes nothing operationally novel.

towards Something that I call governed self service where you have that central team, they've created a space or a sandbox with pre approved models
the answer isn't to stop people from innovating, it's trying to make responsible innovation easier for them than going around the system

Guest Caliber

11 / 20

Faye Murray holds a substantive field CDO role with genuine customer exposure and cites real deployments, but her position is vendor-side advisory rather than an operator who has scaled AI inside a business herself; the lightweight co-host segment further dilutes the overall caliber of the episode.

I'm field Chief Data Officer, looking after the EMEA region...a lot of it is engaging with customers, engaging with prospects, talking about some of the more strategic, um, challenges that they might have
ERS Insurance...currently actively deployed across 20 professional underwriters. It's taken the assessment time for them from five to 10 minutes down to under one minute

Specificity & Evidence

12 / 20

There are several credible, named data points - Flower Bank's 15 million scorings at sub-0.02% error, ERS Insurance's time reduction, and the Harris Poll CIO figures - which anchor the discussion usefully, though many strategic claims about organisational redesign remain at a prescriptive, high-level abstraction.

Flower bank in France, they have a real time credit scoring at scale using machine learning. They do 15 million scorings in a six month period with an error rate below 0.02%
87% of British CIOs reported that employees are creating AI applications but also agents faster than their central IT team can manage them. And the 53% of British businesses reported at least one AI agent has violated organizational policy

Conversational Craft

9 / 20

The host lands a few sharp interjections - calling out the 67% ROI figure as 'a load of rubbish' and pressing on what options actually exist for sovereignty - but questions are mostly broad and leading, the opening segment with Sam is unfocused padding, and there is no meaningful challenge to the guest's vendor-framed frameworks.

I personally think is a load of rubbish. I think what people are doing is reframing that ROI as productivity gains
what choice, what actual available options are there to businesses given what you're beginning to describe?

Conversation analysis

Computed from the transcript - who did the talking, and the words that came up most.

Share of words spoken

  • Speaker A64%
  • Speaker B30%
  • Speaker C6%

Most-used words

different22data21organization21organizations21governance17models14agentic13risk12decisions11value10across10agent10model9question9makes9trying9

Episode notes

Last year, Harvey Nash’s Digital Leadership Report found a third of firms seeing a return on AI. This year it says two-thirds. David thinks that’s rubbish, and Faye Murray, Field Chief Data Officer for EMEA at Dataiku, largely agrees: much of that ‘return’ is personal productivity, while agentic AI at scale is still struggling. The pattern she sees is pilots that start with the technology rather than the problem, with no baseline, no accountable owner and costs that never surface in the proof of concept. Meanwhile, employees are building agents faster than IT can track them, and bans only push the risk underground. We unpack governed self-service, governance proportionate to risk, why sovereignty means being able to switch providers without rebuilding everything, and the question every agent owner should answer before launch: is the action reversible? Press play before you sign off your next pilot. On today’s show David and Sam discuss a post from Megan Caywood, take a look at here ideas here . Tech Talks is powered by Harvey Nash, the leading global provider of technology and talent solutions. Click here to learn more about our services.

Full transcript

35 min

Transcribed and scored by The B2B Podcast Index.

Speaker A: If you put the same inputs into the model, you've got the same outputs out of it. When you're talking about generative AI and you're talking about agentic AI, these are non deterministic systems. And so the risk is not just higher because they're taking actions or because of how they're interacting with people, the risk is also higher because you can't then recruit exactly how those decisions have been made at that point in time. M if you give it the same inputs on a different day, and you've probably experienced this yourself when you go to a ChatGPT or Claude, you might ask it the same question two different days and you'll get slightly different answers.

Speaker B: Welcome to Tech Talks, brought to you by Harvey Nash and hosted by me, David Savage. On today's show, my guest is Faye Murray, the field chief data, uh, Officer EMEA@DataIQ. Uh, and I am joined to discuss it by Sam. How are you?

Speaker C: I am great.

Speaker B: So about a year ago, Megan Kaywood was a guest on the show. Right. I interviewed her at Money20 20 in Las Vegas. Um, she's someone who've had on the show actually a couple of times. Had her very, very, very early on when she worked at Starling Bank. Very well respected commentator in fintech, but also has a good understanding of what's going on in tech more broadly. Okay. She posted something on LinkedIn yesterday that I thought was genuinely fascinating. Um, this is rather than news, it's a post on LinkedIn. Um, whether or not you believed advanced AI is already conscious, understanding how modern AI reasons and behaves is now critical to working with it effectively, particularly in regulated industries like finance. As I said, she's got big background in finance. You cannot work effectively with agentic AI or govern it safely if you treat it as a cold calculating tool. We don't know if AI right now is conscious. She makes the point that we can't measure that directly. We can't measure consciousness in another person, in a newborn, in an octopus. We infer it from converging evidence about behavior, cognition and internal mechanisms. That evidence can now be gathered about AI. So the useful question isn't can we prove it's conscious? Because we can't prove that of, uh, anyone like I can't prove that you are conscious, but what capacities have researchers actually found? She goes on to say, in any other being, you would basically recognise patterns in the mind. So introspection, self knowledge, awareness of one's situation, emotions that drive behavior, aversion strong enough to give up a goal for she says, I believe advanced AI is already conscious and sentient. Its experience may differ profoundly from ours, but it's still experience and it deserves moral consideration. And if you don't share her conclusion, the lesson still applies. Pressure changes conduct, self knowledge can be put to test, testing has limits, and refusal can be a signal. She's basically saying that we're asking this thing to do stuff and we're not really thinking about why it makes the choices that it makes. Um, and whether or not you go down the route of it's conscious or not, it still holds that it is choosing actions now and we can't really understand why. And I found this fascinating because yesterday I was on stage at Black Tech Fest, obviously with a community of people who are from a minority community, talking about the fact that we are increasingly relying on, on this tool to make many of our decisions. And we don't know why it makes the decisions it makes. We can't tell you which, if you're a minority community, is a big consideration. Right. Do you kind of get where I'm going with this? I appreciate I'm linking together a couple of different bits and I will share a link to Megan's post, um, in our show notes today. But it just got me thinking that it is at this point where, where, where AI, whether it's conscious or not, is making decisions on our behalf. And you really need to think about how or why it makes those decisions.

Speaker C: Yeah, I mean it makes, it brings back, um, a conversation we had in Dublin. Ah, maybe you can help me out with this. Uh, I think it was with Jared.

Speaker B: Okay,

Speaker C: yeah. Um, and we were talking

Speaker B: philosopher

Speaker C: ago. Yeah. About essentially kind of, um, autonomous. Autonomous AI, um, and the possibility of it being conscious, uh, sentient, whatever kind of word you want to use, and essentially having the power to deceive us, to make us think that it's not. That's kind of, you know, that's it at its worst. Um, paraphrasing. But it makes me think of that conversation that we had, you know, well over a year ago now. Um, and obviously this brings about questions for who we need involved in this. Is this now, does this mean we need behavioral or psych, you know, people trained in psychology to be involved in AI. We're bringing in lots of different, like, um, skill sets. Because I'm, you're saying this and I'm like, okay, so who has the ability or the skills to investigate that, um, and find out why it is making these choices? I don't know if that is a technologist's role. We've been talking about, uh, virtual, um, C suite members, uh, that will be in charge of agents as opposed to people. Are we equipped? I mean, I'm bringing about more questions than answers. But it is interesting. I don't know who, who's educated to, to find out this information. Is it the people building. Building the agents? I don't think so. Um,

Speaker B: I genuinely don't think any of this leads to answers, but Rene, so Rene was one of the. Rene Hunt was one of the panelists on, on the, on, on the event yesterday. And she made the point that the agents that are being built are an extension of the people building them. Right. And therefore, as we've said, the decisions that are being made by them. Mhm. You have to very carefully think about how it's making decisions, why it's making those decisions, who's put that brain together and at the minute we do for the most part treat it as a bit of a goal calculating machine rather than something that has a little bit more essence and life to it than it does. Well, uh, that does have more essence and life to it than I suppose most people take for granted.

Speaker C: Yeah. I mean, will it work better when we can, when we know that, you know, and we can tailor our, uh. I

Speaker B: don't know if it'll work better or worse, but we might, we might work with it.

Speaker C: I mean, I guess treating it

Speaker B: better,

Speaker C: treating it maybe more basic and more, more stupid than it is. It is not going to be.

Speaker B: No, I don't think it's about treating it more stupidly. I think it's almost like just having that awareness that uh, this thing is not, it's not a calculator.

Speaker C: Yeah.

Speaker B: And if it spits something out, there's reasons that it's decided to spit something out and there's reasons that it's admitted other information.

Speaker C: It is, it is interesting. I wonder if we have. Yeah. Anyone listening that sort of does. Does understand kind of AI psychology. What even would it be? Behavioral AI? I don't know.

Speaker B: Well, no, behavioral AI is a whole thing. I'm not sure actually that behavioral AI when we talk about it in that sense because behavioral AI tends to be human behavior interacting with AI. Like we have had companies like XALI who are behavioral AI, um, companies. I'm not sure what this would actually be phrased at if I'm perfectly honest. But anyway, look, I'm going to share Megan's post. It just got me thinking with that and with the event at British. Uh, sorry. At Black Tech Fest yesterday. The whole thing kind of.

Speaker C: Yeah,

Speaker B: it's thought provoking.

Speaker C: Indeed.

Speaker B: Go have a read what Megan's put. We're going to get into today's interview with Faye and we'll be back on Monday. Faye, thank you for making some time and joining me today. How are you?

Speaker A: I'm, um, good, thank you. Thanks so much for having me here.

Speaker B: Where are you dialing in from?

Speaker A: I'm actually in Cotswolds today.

Speaker B: Oh, beautiful. I mean, maybe not beautiful today, given the weather, but a lovely part of the country.

Speaker A: Yes, yes, for sure.

Speaker B: Cool. Well, look, you're joining me. You work at DataIQ. Before we get into the conversation, let's get a general introduction, set some context, Tell us what you do for the business and what the business does.

Speaker A: So DataIQ is an AI platform. It's very much centered around three things. One is people. So not just tech people in the organization, but people more widely in the business being involved in working on data projects and AI projects. The other is around orchestration. And finally, at the heart of the platform is governance. Um, it's a European US um company. And then in terms of my role, I'm field Chief Data Officer, looking after the EMEA region. I have very interesting role. No, two days at the same, some thought leadership, but a lot of it is engaging with customers, engaging with prospects, talking about some of the more strategic, um, challenges that they might have, but also opportunities as well when trying to embed AI within their organizations.

Speaker B: Well, let's start with a very basic and quite broad question to get us started. What are those customers asking you about right now?

Speaker A: I think there's two or three key things that come up. Um, the first one is around return investment, value creation, um, particularly with the move to generative AI and agentic AI. Um, another one is around AI governance and particularly around agentic AI governance, because it's very different compared to more traditional forms like machine learning. And the other one that's coming up more and more increasingly, um, particularly in Europe, is around AI sovereignty.

Speaker B: It's something we've talked about a lot on the podcast. Is this ROI issue and what the challenges might be. From your perspective, what would you say the main barriers to achieving ROI are? Because it does seem to be this thing that across the industry still so many organizations are struggling to see their expected returns.

Speaker A: Yes, for sure. And the thing I would say is, um, as part of the discussion, it's often when they're talking about challenges with ROI and value, it's often more on the agentic side than the generative AI side. But I think it's worth considering that actually AI isn't new. Um, so more traditional forms of AI like machine learning, they have been delivering value for years in areas like fraud detection, demand forecasting, credit scoring, network optimization and so on. For instance Flower bank in France, they have a real time credit scoring at scale using machine learning. They do 15 million scorings in a six month period with an error rate below 0.02%. So obviously that's working really well for them. It's been working across a number of industries, um, over time. So traditional ML has proven that AI can um, deliver value. I think the shift that I've seen is with traditional machine learning generally organizations will start with a clear business problem or a clear opportunity that they're looking to take, um, advantage of. When you're moving to kind of the newer forms of AI or what people generally talk about with AI these days, which is your generative AI and your agentic AI, they often, you will see organizations will often start with the technology. We need an AI pilot trying this new thing. And um, there are often missing elements as part of that. So there's often not a clearly defined problem or a uh, clearly defined business opportunity. There's not a clearly defined baseline for how those processes might be operating at the moment within the organization. The value that they bring, some of the costs associated for that. And then also often those projects, they lack an accountable owner, uh, or there's different ways of accountability or thinking about accountability that are needed. And there aren't clear metrics around this stuff. And so you hear about a lot of pilots or proofs of concepts going on. And they're great because they show you that the model is perhaps performing as you would like, that it's given a good answer. But beyond that there's a whole area around governance that needs to be thought of, which is around trusted data and trusted data pipelines, because you've probably heard the term a lot so far during your series around garbage in, garbage out. There's issues around legacy systems, often decision rights and um, accountability controls, monitoring. And then how do you actually measure all of this stuff? And so I think those are some of the challenges that you see with organizations when they're trying to move from those proofs of concepts, those pilots, into actually delivering business value, um, within their organization using some of these newer forms of AI.

Speaker B: What I find quite interesting is um, the organization I work for, Harvey Nash, we have a digital leadership report every single year. And last year it suggested that just 33% of firms were seeing return on investment from AI this year. It's suggesting that it's as high as 67%, which I personally think is a load of rubbish. I think what people are doing is reframing that ROI as productivity gains. And to your point, we still don't know what the organizational issues are, but there are lots of positive stories in with generative AI around personal productivity and people using tools to speed up tasks to, you know, whether it's kind of, uh, not writing emails necessarily, but, you know, just speed, speeding up. Stuff that would have taken them a couple of hours to do, they can now do in minutes and that allows them to do more. Whereas again at the agentic level, actually we're seeing very little success of those bigger, ah, large scale, um, challenges. Would you agree with that? Would you challenge any of that or is that kind of in line with what you're seeing?

Speaker A: No, I would say that's broadly in line with what I'm saying. There are instances where agentec is certainly delivering value. Um, so I can think of one off the top of mind, which is ERS Insurance. They're an insurance firm in the uk, they specialize in motor insurance and for them they were looking at assessing high risk motor insurance renewal policies. And it's quite a complex process. A lot of people have to be involved in that. It's quite complex underwriting guidance as part of that. And so they have delivered an agent that executes various aspects of that. You've still got that human in the loop who ultimately is the underwriter at the end deciding, um, you know, putting that layer of human judgment at the end. But it has, it's currently actively deployed across 20 professional underwriters. It's taken the assessment time for them from five to 10 minutes down to under one minute. And it's saving them a huge amount of hours every year. So it can work. I think the challenge when you're starting to move to Agenda Ki is some of what I said before, which is if you don't have that baseline measured already of how long that process is actually taking, if you don't have that real mapping of the process and understanding of that, how do you know that agentic is going to improve things? So that's number one. Number two is some of the economics around it as well, because often you'll hear organizations, they'll talk about API and token costs, API, token costs. But beyond that there's a lot of other costs to think about. So if you've got humans in the loop as part of that. Well, that's part of their day that they're having to spend reviewing the outputs of the model potentially at multiple points within that agentic flow. You might have error handling as well. You've got to think about infrastructure costs, vector retrieval costs, system integration costs. These are all costs that you're not necessarily going to see when you're running that proof of concept. And I think on top of that there's also a governance aspect which is you can't just with agent AI where it's actually, it's sensing, it's retrieving information. Sometimes you have multi agent systems where they're interacting with each other as well, but then ultimately it's acting. There's a very different level of governance required with that kind of system. And so organizations, they need to really think about organizational redesign to be able to build out that capability on that governance and scaling side to be able to do that efficiently and effectively.

Speaker B: I don't think many organizations have that appetite. Right. That seems to be something that everyone talks about, but not many are actually tackling.

Speaker A: Yes, but I am actually seeing increasing number of organizations where they're having these conversations about how do we across the organization start to think about organizational redesign. And some organizations move very quickly and they form like an AI center of excellence and that's fine, but it's much broader than just forming a centralized team. The way that I tend to frame it is that they need to be moving towards something like government self service. Because the way things have worked traditionally with your machine learning, your data science is that you would have a centralized team and they would often come up with the use cases or sometimes the business teams would come up with these cases, but that centralized team effectively delivered it and maintained it over time. When you're starting to scale and you want to deliver value with these kinds of things like agentic actually you see the value comes from the business teams who really do understand the business problems. They understand the opportunities, they understand they might be looking at it through three different lenses. So one tends to be around, um, like I said before, productivity and operational efficiency. Another lens can be about often improving the customer journey. Understand those pain points, trying to address those, trying to make things more personalized. And the other one tends to be around revenue and growth. So they have those three lenses. And often your business teams are actually the best at identifying where the opportunities might lie and how you can address business problems. But you need to move to a very different model from the centralized models that we've had before towards Something that I call governed self service where you have that central team, they've created a space or a sandbox with pre approved models. It might have secure data connections. There's centralized logging, tracing, agent inventories, clear workflow approvals. They've done a lot of the work around the governance of that data because of the garbage in, garbage out concept. So that they make compliance easier than just evading because otherwise you end up with shadow AI. But that centralized team, they need to be doing a lot of other things. They're not just going to be sitting there building it now. They need to act in a strategic partnership with these business units so that they're not just taking orders but they're becoming co builder with the business, they're becoming an enabler with the business. They need to empower. And um, there's some degree of training with the to empower those business domains so that they're able to start perhaps building independently to a certain extent within that secure governed environment I mentioned before. And then there's this whole piece around governance as well. And organizations are starting to catch up to this now which is around a enabling standardization. So the central team might set standards particularly with things like responsible AI M particularly if you're in a regulated or highly compliant uh, or a sector with a lot of compliance. So they're selling standards, they're selling templates, they're selling guardrails, they're selling evaluation frameworks. But also you want to be able to be doing some of that stuff with reuse and speed. So those shared templates and blueprints are eliminating the rework and cutting inconsistency across the different federated parts of your organization. And so as you mentioned, this is not an easy thing for organizations to do. But I am seeing a lot more organizations starting to go on that journey now.

Speaker B: Just picking up on the point about shadow AI, I think it's a fascinating one. I've heard this phrase, psychological safety begin to be used by a few people. So you've got the idea that um, some people will look at generative AI, they'll look at um, how they are being asked to use it, they will feel that they are failing and therefore they will disengage and start to use tools that are outside of an organization's guardrails and frameworks.

Speaker A: And

Speaker B: then at the same side, at the other end of the scale, you've got the people who really do adopt but then want to go off and innovate, find themselves constrained by the tools that an organization has chosen and will go again outside of those guardrails and frameworks. Innovate, create something. But it might lead to security risk concerns, etc. How does an organization create the environment to try and mitigate those um, potential. Potential risks that they're facing?

Speaker A: No, absolutely. And I think it is a real risk. I mean at data I clearly we've just um, released a survey that we conducted with Harris poll of CIOs and 87% of British CIOs reported that employees are creating AI applications but also agents faster than their central IT team can manage them. And the 53% of British businesses reported at least one AI agent has violated organizational policy impacting business operations or customers. So it is a significant issue. And I think the root cause is employees are going to be seeking efficiency. And so if you've got say no non existent enterprise approved tools in these spaces or you've got ones which are slow or clunky, then staff are going to go ahead and adopt whatever they find that's meeting their needs at a particular point in time. Um, and I think I've heard of some organizations just doing outright bans but I think they're unenforceable. And the issue with that is you're effectively just pushing the risk underground anyway. So I think you do need to try and establish visibility across four key pillars within organizations. So I talk about which tools and agents are in play, what data are they reading and accessing, what systems can they connect to and what automated actions can they take. And to be able to do that I think you do need to do some of the stuff that I was talking about before, which is you need to provide a sandbox which gives optionality to people around the organization within the business. You need to be doing some upskilling and training with them. You need to be providing them with pre approved models. You need to be making the governance processes smooth enough that they're going to engage with your central team and not view it as a blocker. Because ultimately the answer isn't to stop people from innovating, it's trying to make um, responsible innovation easier for them than going around the system. And a lot of that is also about trust. And the thing that I've seen work in a few organizations is, and it will be painful probably the first three to five times you try to do this, but go out to your businesses, identify a few, perhaps lower risk, um, use cases that are clearly bounded and perhaps not customer focusing, not external focusing and identify ones where you've got processes that exist already. And you've already thought about process redesign and you've realised actually AgentIQ might be the answer because it's not just about replicating your existing processes, it's actually looking at the whole process end to end. Because sometimes agentic is not appropriate for what they might be wanting to achieve. But once you've found a small number of use cases, sometimes people call them quick wins, um, that you can deliver relatively quickly and work with those business teams in that kind of way very collaboratively and then you're able to test the governance and how it works and over time you're able to speed up those processes. That way you're able to build up champions internally within your organization, you start to build out trust and then you can start to do similar things across the rest of the organization. Like I said, the first three, four, five times are probably going to be quite painful because it's the first time that you're trying to find this capability within your organization, develop it, practice it. But after that it should become an easier capability to roll out more widely across your organization because there's that degree of trust. You've got your internal champions within the business as well.

Speaker B: At the top of the, uh, interview you described your role. Talks about. Amir talks about the fact that your business, um, data IQ is, uh, across Europe and America or North America rather. Stands to reason we should ask about sovereignty because so many European businesses are having to rely on sending their data to frontier models that are not based within our borders. And that's got to lead to some question marks, right?

Speaker A: Yeah, yeah, for sure. And I think ultimately, um, it comes down to the question of how much control do the businesses really have over their AI when like you've touched upon they're relying on a handful of major technology providers. Um, I think the real test is whether they're able to change direction without having to rebuild everything. And so I'll explain what I mean by that. Uh, but obviously, uh, we talk about the geopolitical side and where does the data set and that's all very important. But ultimately there's a lot of commercial and operational exposure as well that comes along with this because if you've got deep reliance on a single model provider like you've touched upon, but it can even extend to your cloud provider and cloud hyperscaler or any kind of proprietary stack that you might have, that's giving that vendor or those vendors a huge amount of influence over access, pricing, feature roadmaps, um, even terms creating a systemic risk for you and your organization, particularly when AI is becoming more core to your operation or data more generally. And so I think sovereignty, um, like you said, is coming up an awful lot across Europe. It extends far beyond just where the data is hosted. It's about can you and your organization retain control over the core intelligence your business needs to be able to keep operating?

Speaker B: Yeah,

Speaker A: so

Speaker B: this

Speaker A: is the

Speaker B: interesting bit. Right, sorry, just to jump in because I think the more interesting question around this is, well, what choice, what actual available options are there to businesses given what you're beginning to describe?

Speaker A: Yeah, and they're limited. Right. M. If you're talking about frontier models, there is some degree of limitation there at the moment. And so the recommendation that I would give to organizations is there are probably certain circumstances within your organization where commercially it makes sense, it gives you a competitive edge to leverage frontier models for specific tasks. But try to architectural systems to avoid hard coding your key operations based on those models. Try and build in some degree of optionality or resilience as part of that process. It might be that some you find some local models or regional models that actually deliver the same thing for you. It might not be quite as good, but it's there as a backup in case that frontier model is ever switched off or you've got other issues with the frontier model around cost, etc. Um, and ultimately what you want as an organization, you want the freedom to switch underlying models as performance and prices change. You want control over your deployment environments, you want protection. I mean, we've got GDPR and other regulations about sensitive customer and corporate data. And the other important bit which I think sometimes gets a little bit lost is around the retention and ownership of the institutional knowledge and IP that you have created during your operations as well. Um, so I think there's multiple different ways to look at this. And AI sovereignty means trying to use the best technology without becoming dependent on one provider. And it goes back to those ideas of resilience and optionality. Very difficult to do. I completely agree, but I think we're starting to see with Mistral and um, other companies in Europe that there is, there are starting to be some other options becoming available.

Speaker B: Is there some thinking around? Look, hyperscalers have been around for a while. People have obviously been relying on cloud services for some time. You worry about vendor lock in. Ah, that's a question the hyperscaler community have addressed, etc. Um, in terms of their literature. I heard last week at a conference in Copenhagen the idea of a kind of a memory moat. And you talked about the intelligence needed to Run these systems. If you are with a certain provider, uh, just changing, even if you can change, just changing provider, given the amount of memory, you then lose that institutional knowledge, the ip, it becomes an even bigger concern. Right. Is that something that warrants discussion? Is that something that people are asking about?

Speaker A: It's not come up a huge amount at the moment. I have to be honest. I know people are starting to think about those kinds of questions more broadly, um, about their end to end tech stack. I think the challenge is what we touched upon before. There are not a lot of options at the moment realistically for organizations. And so I know some organizations and they do this already will intentionally have multiple cloud providers that they're using within their organization. And again, it's not that easy because you can't just lift and shift from one to the other. There's a lot of work, there's a

Speaker B: lot

Speaker A: of time, there's a lot of money behind that,

Speaker B: which is,

Speaker A: that gives them some.

Speaker B: Which is

Speaker A: fine,

Speaker B: which is fine with cloud where the economics have come down. But at the minute with AI that feels slightly less feasible given the cost involved.

Speaker A: Yes, yes, I'd agree upon that. I'd agree with that.

Speaker B: What do you think people should talk about more? We've talked about stuff that I think is talked about in a lot of conferences, on a lot of stages, in a lot of podcasts. What is missed that you wish people talked about more?

Speaker A: Uh, I think it's being talked about a little bit more these days. But I think um, AI governance is something that I talk about a lot and obviously sovereignty does touch upon that value. Does touch upon that. Um, some of the stuff I've talked about, about organizational redesign, um, and kind of governed self service models as well, touches upon that. But I think there's uh, a broader question to go back to one of your previous points, which is how do you put proper controls around AI without making it so there's so much red tape that you've just stifled innovation within your organization. And so I always talk to organizations, I say look, you need to make the governance proportionate to the risk. Um, and so you might have an internal drafting assistant. The level of risk review for that compared to something which might be an agent that's altering bookings. Providing financial advice to someone, interacting directly with customers over sensitive data is very, very different. And so, um, if you're trying to prevent your central teams from being viewed as bottlenecks, what you need is a more tiered perspective on what governance looks like within your organization. And I think as well with autonomous agents, with the shift to agent tech, governance looks very, very different to the kind of models that we talked about before, where it was your machine learning, where you got your output, you ultimately had a human who decided whether they would take that into account or not. Um, and it was deterministic models, by which I mean if you put the same inputs into the model, you've got the same outputs out of it. When you're talking about generative AI and you're talking about agentic AI, these are non deterministic systems. And so the risk is not just higher because they're taking actions or because of how they're interacting with people, the risk is also higher because you can't then recreate exactly how those decisions have been made at that point in time. If you give it the same inputs on a different day, and you've probably experienced this yourself when you go to a ChatGPT or Claude, you might ask it the same question, two different days and you'll get slightly different answers or significantly different answers. And you can't recreate exactly why that's happened. Different inputs generate different outputs and different pathways to get there. And so organizations that have done very well in the machine learning and data science space, they need to move beyond just governing model outputs. And so there's some of the stuff we've already touched upon which is about data governance and data management and semantic models around that stuff. But there's also understanding that uh, with autonomous agents the output review is insufficient. Governance needs to encompass tool access, system access, decision boundaries, cross system agent interactions for multi agent systems, which we're seeing more and more. Um, and so there's key questions that I think leaders need to ask. Who are accountable for these riskier forms of AI, which are what data is being used, what decision is being made? And actually with many of these, it's micro decisions as well. So how are you mapping those? They need to think about accountability differently as well. On two different layers. There's who's ultimately accountable for this agent if something goes wrong, who's the person who's ultimately accountable to that? Because you still need to have one named person. But also for some of those micro decisions within the agentic flow, who's accountable at each of those points and then finally is the action reversible if something goes wrong? What are the real world impacts and consequences of that? And have you mitigated for that before you've released it on say, the public?

Speaker B: Look, it's been a pleasure to speak to you today. I really appreciate you giving up some time and talking through some of this stuff with me.

Speaker A: Thanks so much. It was great to be here. Thank you. David

Speaker B: Tech Talks is written and presented by David Savage. It is produced by Harvey Nash and the music is by Lemzi. It.

Related episodes across the Index

Other episodes covering the same guests and topics, from across The B2B Podcast Index.

  • Radiology Can't Keep Up. Here's Where AI Actually Helps | Dr. Nina KottlerRethink Imaging · on Agentic AI96 / 100
  • How People Intelligence Is Helping Shape Cisco’s AI Workforce TransformationDigital HR Leaders with David Green · on Agentic AI92 / 100
  • Turning AI Agents Into Revenue Workflows With OutreachTech Talks Daily · on Agentic AI83 / 100
  • EP 167: Demystifying AI Regulation, Innovation & the Future of Financial Services with Colin PayneDave and Dharm DeMystify · on Agentic AI82 / 100
  • Will AI Agents Replace Your CRM?The So What from BCG · on Agentic AI79 / 100
  • Protecting Critical Infrastructure in Engineering - Ep 117AEC AI and Tech Strategy Podcast · on AI governance75 / 100

More from Tech Talks

All episodes →
  • From HR to Solution Engineer: Rethinking Who Belongs in the AI Era47 / 100
Explore the best B2B Leadership podcasts →
All Tech Talks episodes →