Why AI doesn’t need to be perfect to be trusted

speaker-0: Hello and welcome to the Digital Lighthouse where we get inspiration from tech leaders to help us navigate the exciting and ever-evolving world of digital transformation. I'm Zoe Cunningham. We believe that meaningful conversations can illuminate the path forward, helping us to harness the power of technology for innovation, scalability and sustainability. In this episode, I'm delighted to introduce Alex Wolff, who's the Director for Public Sector Health and Transport at Software.

Over the last 20 years, Alex has worked with government departments, the NHS and private healthcare providers to design and implement intelligent digital solutions that make a positive difference for the people using them. In this episode, we're going to talk about safety and security in critical public sector infrastructure projects. We'll cover how new and emerging tech is affecting the landscape and discuss the surprising conclusion that being safe doesn't always mean going slow.

Alex, welcome to the Digital Lighthouse. speaker-1: Thanks, Zoe. It's really nice to be here. speaker-0: Over the last 20 years, you've worked across large scale projects in the public and in the private sectors.

So what's special about the public sector? speaker-1: It's really hard to answer that without making it sound like the private sector isn't special in its own way as well. Because I've really enjoyed both. Actually, doing the two together has been really interesting because I think there's a lot they l can learn from each other.

But I think for me that the public sector, what I really like about it is that challenge that you don't get to pick your customers. You know, and a lot of the services we're doing there are ones that people they don't have a choice to use. speaker-0: Right, you're not choosing who to appeal to. You're saying these people need this service and we need to provide it.

speaker-1: Exactly. They don't have a choice to say, Well, I don't like paying my taxes here, so I'm gonna go pay them in Finland instead. Or they do, but it involves more aeroplanes, you know? So I it there feels like more of an obligation to to build really great services, but to build them that everyone can use.

You can't kind of segment and just say we're gonna focus on our ninety percent customer. speaker-0: Yeah, and the stakes are there, right? They're important services that need to be delivered and need to work. speaker-1: Yeah, very much so.

⁓ one of the things I've been really fortunate in my career is being able to choose where I want to make an impact. And I think for me, public services, ⁓ a ⁓ lot of what you do really does make that impact, you know. You're you're building things for people that provide them housing or provide them kind of the money they need to live their day to day lives, or or just make their life slightly less frustrating on a on a typical day, hopefully. speaker-0: Right, yes, okay, so on that note, if we look at the current landscape of public services, how error-free is this whole system of technology and people?

speaker-1: I'd love to say error free. Like, ⁓ one of the interesting things about working in in public sector is you meet a lot of people who are passionate about their work and a lot of people who they could move to the private sector and they could balance kind of the impact and salary differently. But speaker-0: they really care about what they're doing is what you're saying. speaker-1: Much so.

That said, you know, what we see a lot is systems working just under incredible pressure, like the overall system. You know. I was looking at something recently from the MOJ. Okay, yeah, the court backlog since COVID has more than doubled.

You know, I know some of the people working in there and ⁓ their teams haven't doubled, you know. ⁓ so there is this kind of large pressure, large workload, and yeah, in an environment like that, with the best will in the world, mistakes are gonna happen. speaker-0: Yes, right. Okay.

So, and the reason that I asked that question is because looking forward and looking at new technologies that maybe introduce errors, it's important to look at what's happening currently rather than comparing against an idealized state. speaker-1: Yeah, it's it's one of those and I'm sure somebody will have put a name to this, but it at the minute errors happen in lots of kind of small doses, you know. And nobody's name gets attached to that. Right.

In aggregate. Whereas if you're the person who signs off and says, We're gonna put a new system and it might have a ninety eight percent success rate and it might be higher than the current one, those two percent errors are still gonna have your name attached, you know, and it's still gonna be on the pages of the newspapers with your name. So so yeah, I I try and bring it back to the not is this perfect, but you know, if we're looking to make a change, is this better than what's already there?

Does it help rather than harm? speaker-0: Yeah, and so and hence can we use that to incrementally work our way forwards towards perfect, right, eventually, I guess. speaker-1: In really good to incrementally, ⁓ because yeah, it's the only way it works, right? ⁓ if you sit there planning for perfect, you're gonna be sat there planning for perfect for a very long time.

Whereas incrementally you can still go to bed going, Is today better than yesterday was? Yes. speaker-0: Yes. Right.

And when we're delivering things at such high stakes, that's actually really important. We don't have the luxury of planning everything out and saying, well, we'll have to just live with what we have already. We have to keep getting better. speaker-1: You mention high stakes.

⁓ it's interesting, isn't it? Because I think it's interesting. Because there are certainly some cases where we just can't tolerate error, you know. Places where we have good appeal systems and where kind of issues come up but they're kind of well corrected and ultimately resolved and and maybe kind of in those cases error in parts of the system is okay.

There are other things like prison sentences or, you know, parole decisions or speaker-0: Right. Yes. speaker-1: whether we give people a really important benefit where kind of errors can have real and immediate harm. And it's probably we've got that greater obligation to make sure that by the time the decision turns into action, it is the right one.

It's that balance, like where where can you move fast and correct later versus where do you have a system that needs to be right first time? speaker-0: Yes, amazing. So looking at taking out just the technology aspect of what we've got already, what we kind of assumed in the past about how computers work and how and how a computer system can deliver, I guess, accuracy and, you know, getting these decisions right. Yeah.

speaker-1: ⁓ I definitely assumed that computer systems would be able to take on at least ninety percent of my admin by now. I think that's true. But I think kind of when you think of computer systems, I certainly think of them as kind of these predictable things, you know, that are the same today as they they were yesterday, you know? speaker-0: Right, you give it a sum, you say what's one plus one?

The answer's two yesterday, it's two today, it's two. speaker-1: It's two tomorrow. And we're we're kind of hitting this point now with systems where you give it a sum and you know, most of the time it's two, but occasionally it might be three or one. And that can be really disconcerting.

⁓ because for the last kind of fifty, sixty years, we've been investing authority in computer systems. ⁓ we know you asked about error earlier. We know that we make errors because speaker-0: Yes. speaker-1: We're human, we make errors ourselves.

You know, we come home and go, ⁓ it was at seven, not seven thirty. certainly I do. But ⁓ I think with computers we've got this kind of we put them on another level, and that's why when they do get it wrong, ⁓ and I kind of were thinking the horizon scandal as as one kind of really terrible example of this, it can be so harmful because it takes us so much longer to realise that as a possibility. If we now move into a point where actually that's more of kind of a core aspect of what they are.

They're probabilistic instead of deterministic and I'm thinking about systems with AI. Then if we invest that with the same authority, that it's it's not gonna go very well, you know. speaker-0: Yes. All right.

Okay. So a building on that. If we think about, you know, these critical systems that we're creating, there's so many things now that we can't take for granted. Is there a way to enumerate those things and maybe think through what the consequences are for building things in the future?

speaker-1: I think that enumerating it, I mean, as you say, there's definitely things like probabilistic, not deterministic. There's some examples like, you know, heavily influenced by con context rather than just the clear instructions that are given. But I think that actually one of the interesting things is as you start to list those out, you can see paraboles to the way we used to paraboles parallels to the way we used to operate, which is how people work. ⁓ So people aren't deterministic, they're probabilistic, you know.

They don't always give the same answer. They're heavily influenced by context, you know. And they can be confused if you give them too much information, similar to the AI systems. They can be misled, you know, by a bad actor, you think the con man going up to a cashier in a casino, you know.

But we built successful systems that have humans operating within them. I think what it comes to is when you're you look at it as a critical system. speaker-0: Right. speaker-1: What do you mean by system?

What's the boundary of that? Right. You know, I I used to think of it as kind of a computer system, you know, the box that or laptop, or depending on how old you are, you know, small USB stick that you'd run your operating system on, you know, and that was a system. But there's another sense of it, which is that kind of wider thing, the kind of like a solar system.

Yes. With all the interrelated parts that kind of combine. And if we think of AI as being one of those interrelated parts, but other parts being humans or audit steps or correction steps, then you can end up with something where even though each part isn't foolproof and can make errors, the overall system gives you the right result in these kind of critical environments. speaker-0: And in fact, that's a fairer comparison, isn't it?

Because we have no systems that work with one human where it depends on, you know, the state of that human in that day and whether they get the answer right in that moment or not. We run literally none of our systems like that. speaker-1: Yeah, and whenever you see it creeping in, you don't normally see calm, ordered results with it. Right.

Because it's the workload and it's it's the stress and the like. So so ha yeah, having the layers we talk about human in the loop, mm but you know, it's it's that combination. How are the two systems working together, the kind of the person and the machine? speaker-0: Yeah, and I suppose, are there kind of principles that we can pull out of how to responsibly build a system?

speaker-1: There definitely are. I mean, so there's the field of AI safety is v ⁓ kind of now moving from that, looking specifically at individual AI bits. Okay. To kind of taking that step back and look in an end to end system that involves AI.

How do we make sure that that overall works? And there's probably three categories of things we can look at there. Okay. Yeah.

So as much as possible if we can kind of build things right, so architect it right. speaker-0: Yes. speaker-1: put make the right choices early on, then we eliminate the risk from ever arising. speaker-0: Yeah, so actually planning, pre-planning.

speaker-1: Yeah, exactly. Like these systems can or these parts can be hard to correct, you know, ⁓ in every circumstance later. So why don't you kind of plan it right to start with? So we do things like use case, you know, making sure that we're working to the right use cases and that's really deliberate.

You you mentioned principles, kinda making it a deliberate choice of are we answering, you know, questions, are we augmenting a human decision or are we automating one? Mm-hmm. You know, any one of those for a particular circumstance can be the right choice. Yes.

But making sure that you're there deliberately and not kind of wandering into one will make sure you put the other surrounding bits around it. speaker-0: Yes, exactly. Exactly, like being precise about what you're doing so you can put the right guardrails in. speaker-1: Very much so.

And yeah you mentioned guardrails. I mean, that's another kind of of the principles. You can put in place technical guardrails that seek to kind of minimize harm, you know, so make sure that you're not just trusting an AI component with all your data and all your actions. Right.

Yes. You know, because there will come a day when it will just nibble away at the rules you've set, you know. So how can you kind of work out in the same way you would with a new employee, what does this need access to? What does this need to be able to do?

You know, and be conservative with speaker-0: Exactly, yeah. speaker-1: And then probably the most important for me is make sure that it's never the be all and end all that you've got supervision and appeals. You know, if you've got a system being run with an AI component, make sure that there is a human whose whose name is attached to it. Yes.

You know, that means that you've got somebody who who is really thinking through the repercussions of this, making sure that full circle it works for a hundred percent of citizens, say in public sector and not just the ninety percent to the cost of the ten. But also that if things aren't working right, that there is a person who can say, actually no, this this decision isn't the correct one and can use that thing that we have ultimately, which is discretion, you know, to to go the other way and to ma to correct the outcome.

speaker-0: Yes. And I wonder if it also perhaps relies on a general level of education at the senior leadership level so that when there are errors, people are able to look at it and analyse it in the right way. So not just obviously having this one person, you know, one named accountable person, I can see how important that is, but actually making sure that the people around that the whole governance system has the understanding to be able to hold them to account. speaker-1: It's really hard because we're in this transition period where there's so much change and people are getting used to so many new concepts.

It's kind of the paradigm shift moment. But I think you're you're a hundred percent right. It's that how do you start to treat these ⁓ these things as correctable? Yes.

You know, rather than as kind of the bulletproof machine, you know, two plus two equals four, forever will. Because that brings in not just different controls, but also a different mindset. speaker-0: Yes, right, exactly. A new way of thinking about it.

and on that note, I'm very aware that, as you said at the start, it's really easy, particularly when looking at these critical government services, to look at how they can go wrong. And I feel like we've covered that quite extensively. So on the flip side, like, what are the positives, if we get it right? Like, why is it so important that we do invest the time to properly understand this and look at you know where it can take us.

speaker-1: I love that because you're you're right. Why do this at all if there isn't that major upside? I think some of it comes from looking at those backlogs and thinking, what would it look like to turn this round? Because we can look at the stats, which is a large number of people waiting for their court case, but actually behind each one of those is someone who's just not able to move on with their life.

They're waiting for a family situation to be resolved, or they're feeling in the right on something, there's that sense of aggrievement. How do they move on? It's the benefits, the peel that someone is waiting for. Yeah.

So that there's real cost to the delay. ⁓ I think one real positive is if we can use this to kind of scale our effort and to give people things like justice or answers much quicker. I think the other thing though is looking at people who have maybe struggled with the transition to digital. You know, quite often digital is about how do you fit within a process, how do you fit within kind of a box, you know, how do you follow my way of thinking about the world?

speaker-0: Here's a form. Yeah, yeah, it's your job to fill in this form. Yeah. speaker-1: Yeah.

And if we could turn that round a little, you know, so that you're using AI almost like that personal concierge for each person, you know, we know that humans already do a really great job, citizens advice bureaus, charities, you know, volunteers in helping people navigate government systems. ⁓ and it feels like there's a huge opportunity to use AI to provide that kind of personalized service to each person. Yes. That kind of meets them where they are.

Yeah. ⁓ so yeah, I'm really positive about that. I think that will be be a great outcome. speaker-0: Because what you're saying is that the possibilities are doing more and delivering more of these critical services that as you say we're already behind on and doing it better.

You know, that's where we're heading for. speaker-1: Yeah, very much so. The reason to move forward is not just to take what we're doing right now and deliver it cheaper. Yes.

It's to be able to ultimately deliver something that feels really different. You know, that uses information it already knows about you to take work off your plate. You know? ⁓ no one gets up in the morning thinking, you know, today I want to fill in a parking permit application.

They think I want to have mobility. You know, and to do that I need a car somewhere near my house. And to do that I need to fill in this form and it's a step, step, step. But what if we can use it to eliminate some of those steps and just go back to giving them the mobility or giving them the access to healthcare or education that they're looking for, you know, without some of that life admin.

Right. And speaker-0: or actual barriers, even worse actual barriers. speaker-1: Yeah. ⁓ it doesn't feel that far away that we'll be getting that.

We know that kind of within the center of government, they're already working on what their kind of agentic experience has looked like. It's a technical term we haven't used yet today, but you know, fundamentally, what does it mean to have systems that don't just respond to your immediate requests but try and predict them and facilitate them and proactively offer you solutions? speaker-0: was going to say to use a human term, like think around it essentially what you're asking.

speaker-1: Think around it. The kind of yeah, the thoughtful person who says, Hey, you know, I don't know if it's useful, but so it feels like that starts to open up possibility. I think the other thing is if we don't move ahead, say, in public sector, to do that, it feels likely that third parties will tr come in to look to fill that void. ⁓ and then kind of people's relationships with their government will be intermediated, which speaker-0: Right, right, exactly.

speaker-1: From a point of view of trust and from a point of view of a really positive governmental system that with engaged citizens feels, you know, worrying. Right. speaker-0: It's complex. it could, it could, there are good possibilities, but also less good possibilities.

speaker-1: ⁓ definitely. I think it's complex is the perfect description that applies to the speaker-0: That's answer to all technology questions, yeah. speaker-1: Yeah. But no, I'm I'm really excited.

⁓ I think that there are risks and I hear people talk about the risks, but the way we reap the benefit is by having that discussion not from a there are risks so we can't, but more there are risks, therefore we need to. Yeah. speaker-0: Exactly, exactly. And surfacing the risks is the important part of how you deal with them, right?

If you don't know about it, you're not going to be able to address it. Okay, brilliant. So just finally, quickly, I've been very heartened by the progress we've seen over the last 20 years in terms of government services being delivered digitally and how that's definitely affected my interactions with the government in a super positive way. What things do you think are on the horizon that maybe the public can expect to see over the next five years, the next 10 years.

I'm not going to hold you to it, but just, you know. speaker-1: ⁓ I'm yeah, I will come back in five years time and listen to this podcast and go, ⁓ wow spot on or not. ⁓ I think it's I think it's gonna be a really interesting time. Britain was ⁓ for a long time viewed as very much the lead in the world on how we offer digital services.

We were first or second, you know, in the world rankings on that, and that's thanks to a lot of hard work from the people in GDS that then spread out to the departments. Over the last few years, kind of as we've dealt with some of the thornier problems, we've slipped down in the rankings a little, still, you know, in the top ten but slightly lower. It feels like over the next few years there's a real hunger and appetite, ⁓ and the right people in the right roles to kind of make that happen again, you know, and to say, with this new technology coming through, how do we leapfrog back to providing, you know, one of the world's best services?

That's likely to come in a few areas. ⁓ In the UK, because we're a large country, seventy million, ⁓ and because we have really strong rules on privacy and you know, as a vibrant democracy, some things are harder. ⁓ so sharing data across government has always been one of those things that w can unlock so much, but it's just just always been that kind of five years time thing, a bit like fusion, you know? It feels like in the next five years that really will become a reality.

⁓ one of the things that AI can do is help kind of map the data across departments in a way that would have taken a lot of manual effort in the past. And because there's such compelling use cases for when you get the data right for AI, it also feels like the funding and political will is there. Right. What that means for you or me speaker-0: Yes.

speaker-1: is that when we go to apply for a new driver's license, you know, our photo comes across from our passport, you know, our address data can be suggested from something we did, you know, of ⁓ health recently or for the council. Yeah. ⁓ but that should all be done in a way that's really kind of respectful of your privacy and your preferences. Yeah, yeah.

And secure definitely. ⁓ so I think there's a lot of positive opportunity come in for how we join up government services. ⁓ and then in terms of your experience of that, there's a GovUK app and that's about kind of bringing everything together into just that one place that then switches from being reactive to proactive. So it's a notification of, you know, hey, you need to update your passport with the AI options, you need to update your passport, check this is correct and hit go and we'll do it for you.

Yeah. You know? so how can we broaden people's in you know awareness of what they're entitled to across government, get the data on them more accurate so that there's less work for them, and ultimately kind of reverse the trend in kind of slightly declining trust for government services and kind of it ⁓ enjoyment of, you know, to get back to kind of top of the rankings for delivering a really, really positive experience for people. That for me, I think next five years, next ten years Flying cars and zero rate tax.

Not sure. speaker-0: ⁓ Thanks so much Alex, thanks for sharing your experiences and helping us to shine a light for others. speaker-1: ⁓ it's been really nice. ⁓ thanks Thorough for having me.

speaker-0: This digital lighthouse episode was edited by Steve Folland and produced by Patrick Anderson. The theme music was written and recorded by Ben Bailo. A huge thanks to our sponsor, Softwire, for their continuing support right from the inception of the show in 2019 to the present day. If you love the podcast, please let us know with a rating and review on your platform of choice.

We're always looking for feedback to ensure we're making the best show possible. And if you'd like to take part, please drop us a line at thedigitallighthouseatsoftware.com. You've been listening to the Digital Lighthouse with me, Zoe Cunningham.

Thank you for sharing your time with us and stay safe on this wild technological ride that we're all on.