AI Can Fix Your Broken Research Repository

Paul Boag: Welcome to the Boag World show, the longest running web design podcast where we look at user experience, design, conversion, rate optimization and working in the web. On this month's show, we're going to be discussing AI powered user research repositories and then in the very next breath, a warning about the dangers of AI. So make of that what you will. My name is Paul Boag and joining me as always, is Marcus Lillington. Hello, Marcus.

Marcus Lillington: Hello, Paul. Uh, you're right, mate. How's it going?

Paul Boag: Yeah, not too bad, not too bad. Uh, we've been. It was really interesting. We, uh, we were having a moan in the Agency Academy, Slack channel. I don't know whether you noticed that. Um, and, uh, I didn't know.

Marcus Lillington: I've been too busy, Paul. Uh.

Paul Boag: Oh, well, you've actually been doing work. We're all feeling a bit burnt out and, um, and we're blaming AI for it. So. Yes, that's where we're currently at.

Marcus Lillington: It's because that's the thing that is already a recognized thing.

Paul Boag: Yeah. And it really is as well. It used to be you sat down and you did one thing right, but now I've got like three or four agents on the go, all doing different things and you're having to hold it all in your head of, you know, and you have to check and review and all of the rest of it. And so, yeah, I'm a shit ton more productive. But I'm knackered. And I'm not the only one. Several people were moaning about it. So it's a thing, is it? If you read about it, Some of them.

Marcus Lillington: Yeah. Even though I have for the first time in a very long time, come up with some reads for this week which are a little bit. Well, they are quite AI related. Yeah. I'll see if I can dig one out for the next show. Because it's basically a. I think it's called AI Burnout, which is to do with basic. Basically. I'm gonna repeat pretty much what you just said, but having to manage too many things. Yeah, you become. You best basically become a manager, um, rather than a doer. Ah, that's a bit too strong, but you know what I'm saying.

Paul Boag: Yeah, yeah.

Marcus Lillington: I'd probably be quite good at this because I've always just had a million things on and, um, I'll just skip between them. Um, but I'm not a big user of AI. Use it a bit. But when there's a specific task where it's gonna help, I don't help it run. Uh, Whereas I probably could cope with it. But I can completely understand that we're not. I think I saw a quote that, you know, we're only meant to sort of deal with, you know, whether we're hungry or. Yes. Or, you know, or whether it's wet and we need to get under some, you know, some shelter. And having to deal with, you know, very complex issues, multiple ones, uh, and make sure that you're on top of it is going to make you go, Blair. Yeah.

Paul Boag: Ah, I reckon I can only be productive at that level for about four hours a day. And after that, I begin to kind of start, uh, dropping things and flipping up. Yeah. We glitch. What about you?

Marcus Lillington: How.

Paul Boag: How are you? Things going all right with you?

Marcus Lillington: They are. Okay. Um, uh, Headscape. We won a client, a new client the other day.

Paul Boag: Yay.

Marcus Lillington: And, I mean, bearing in mind how long. Long I've been saying we haven't done that. I mean, that's. I think we should have a little party. Not a particularly big, Big, um, thing. Um, but for another American university, so that's nice. Oh, good. Um, and we're also courting another, um. Courting in the. In the old sense of the word. Yeah. Another American university, um, which we would love to work with. So, yeah, I guess our. Our marketing work that we've been doing, uh, maybe starting to pay off, or it could be complete chance. Who knows?

Paul Boag: Uh, you don't know. That's the annoying thing about it. You know, you ask, how did you find us? And inevitably the answer is, I don't quite know. I can't remember. You know, every time.

Marcus Lillington: Oh, I think I just looked it up on Google, really. No, you didn't. At least I know where the. Where the. One of these. I know where both of these came from. One of them was, um. Ah, we do a lot of work for the University of Michigan. Ah. And this guy runs a journal for, um, another university, which I'm not going to mention yet because I don't want to jinx it. Um, and he. He was looking to sort of, how can we improve our website. Found the one we did. Thought that's what I want.

Paul Boag: Right.

Marcus Lillington: Um, got in contact with the people at Michigan that run it, and they said, oh, Headscape did it. Go talk to them. So that's, you know, perfect. Yeah. So that's got nothing to do with our marketing? No, I realize.

Paul Boag: No.

Marcus Lillington: Um, the other one has because. Because she came. Um, I basically decided that because we've done work for law schools that I should be talking to other Law schools and like cold email basically. But we. But with a good story. Yeah, yeah. And a couple of people came back and she was one of them. Um, but she knows that's good, you know, so. Yeah, anyway, also Paul, I was down your neck of the woods the other day.

Paul Boag: Yes. You were showing off your breakfasts to me.

Marcus Lillington: Well, yes, exactly Paul. You. All I'm going to say is you must go there for, for breakfast. It was, you know, you and I are in our sort of aging state, mustn't eat that sort of thing very often. But every now and then.

Paul Boag: Hang on a minute. I'm a lot younger than you. I want to remind you. Don't. Don't tar me with your, your.

Marcus Lillington: With my great age. Yeah, exactly. Yeah, yeah, I know you've had, you've had to um, as I think probably discussed, maybe not on this podcast but on other podcasts you've had your issues with what with, with your eating. So I was just looking after you, Paul.

Paul Boag: That's very kind of you. Thank you very much.

Marcus Lillington: Anytime.

Paul Boag: So anyway, should we talk about topic for today? So I was quite surprised that ah, I haven't already talked about this topic and I had to triple check to make sure I haven't because it's something I'm really enthusiastic about. I think I vaguely mentioned the IDE of AI generated Personas before. We have talked about that, I think

Marcus Lillington: more than once, Paul. Yes.

Paul Boag: Yeah, but it's kind of a little bit broader than that that I want to talk about today. I want to talk about user research, um, and user research repositories. Right. Um, because there are an, yeah. An increasing number of the clients that I work with um, do user research. They've got their own in house UX teams. Um, and they, they might even have specialist UX researchers, although in a lot of cases have just got kind of generalist UX people if that makes sense. Um, and so the typical pattern from what I see in most organizations is that these people go away and do research normally for a particular project. Um, they then give a PowerPoint presentation to stakeholders, um, who either nod along and then completely ignore everything that was just said or nod along and agree. But in either case that's then that research done and dusted. Right. Um, and that's great maybe in the short term if they, they nod along and agree and you go and do something about it. Um, but it does mean that that knowledge just evaporates. You don't get long, any long term um, value out of that. So the solution to that has traditionally been to create some kind of research repository. Um, but the problem is, well, who actually is using these things? Right. We create these repositories of information about all the different testing that we've done. Um, but oftentimes that feels quite cumbersome to do because if you're not on top of it, say for example, you decided to create a user research repository today you might have done years worth of user research that's in PowerPoint presentations and you know, kind of user interview transcripts and surveys and all of this stuff is all over the place. And bringing it together and putting it into a repository feels like quite a big job. And even if you do that, you're then left with what, a load of documents that have become a dumping ground that are quite hard to kind of navigate and find anything of value if you're working on a specific project. And yeah, there are tools where you can search on them and that kind of stuff, but that's pretty, pretty hard as well. Um, and so most of the time if anybody is using these repositories at all, it's just other UX professionals. Right. The very people who kind of understand the research anyway. And so what's the benefit kind of thing? Um, and everybody else is just ignoring them. And so this can all feel like a massive wasted opportunity. So what I'm kind of interested in is how AI has changed the game around all of that. I know it's, I feel like it's all we, I uh, said this last time, all we talk about is AI

Marcus Lillington: but I'm going to talk about something different on this subject, by the way.

Paul Boag: Oh, are you?

Marcus Lillington: Yes. Now fashioned, Old fashioned.

Paul Boag: Okay. What were you going to talk about, I want to know.

Marcus Lillington: Well, there is another way of doing this, um, of ensuring that things stay front of mind. Yeah. Um, outside of people in UX and for writers, designers, whatever is print them out big and stick them on the walls. Which is yes, a really, really good thing to do.

Paul Boag: It is, it is a really good thing to do. Absolutely. I agree with that.

Marcus Lillington: And because I've been, I, I've. I wrote an article basically saying that a couple of months back, the conclusion of which is, you know, because if you, I can remember we looked at some good examples of Personas that, you know, high fidelity, well designed Personas that for. I think it was M. Mailchimp maybe. Anyway, they're the ones and they were, they were really good. But what we've, and um, we've done that from many clients over the years, but what we've ended up getting. The ones that you stick on the wall are less and less and less and less detailed to the point of. Actually all you need is the face. Maybe that's a little bit too. It's a reminder that they exist. And then. Because unless it's. Unless you don't interact with these things very often, you know what each of these characters are talking about Personas, and you probably know about the, the user journey maps and what kind of thing as well. So all you need is the face. I then went on to this last month to say, but what about having. Should we be having faces on Personas? But I'll save that thought for another day. Yeah, but, but, but I see. What I like about what I think you're going to say, Paul, is it'll happen automatically with what you're going to say.

Paul Boag: Well, not quite. I think there's a. I think there's a slight difference between what you're talking about and what I'm talking about. Because you're talking about Personas, right? And, and I agree that there is a big advantage of having Personas on the wall to remind people. Totally agree with that. However, a user research repository is a repository of all the user research you've ever done in its raw form. Right?

Marcus Lillington: Not just Personas.

Paul Boag: Not just Personas. Yeah. So, so, and, and that is, you might argue, well, not everybody needs to know or have access to that information anyway. But the problem comes in, um, that you often then end up repeating research in slightly different forms or research doesn't get done at all when, when actually if you did created a repository of research, then people could just go to that, to, to find out, you know, what has been done before. But uh, it's kind of overwhelming. And that's where AI begins to change the game. Right? First of all, AI is really good for creating a user repository. User research repository, because you can just throw everything at it, you know, every PDF, every old Persona, every survey, every bit of data that you've got, anything you can lay your hands on, you can just pump it into AI and it will structure and organize that, ah, information into a repository for you, right? Which is incredibly powerful. So where it used to be this huge painful task to create a user research repository, now it's pretty easy to do in comparison. Um, but the best bit is you can then make that content accessible to absolutely everyone, not just a UX specialist, because traditional, um, search on these kinds of repositories require you to know specifically what you're looking for. While AI, ah, lets you have a conversation with that repository. So once you've got it in place, you can then put a chat interface on it and say, I'm working on this feature in this product, you know, and I'm doing this kind of thing, or I'm trying to appeal to this audience or whatever it be. What research have we already done on that that might be relevant or might be useful? So you can be much vaguer than you can be with a traditional search. So a product manager could ask something like, you know, what do users think about our checkout process? And get this kind of synthesized answer that might come across, you know, come from like five different research studies, um, that they never even knew existed, um, and that it means a lot of new research requests can um, also get answered. Let me explain what I mean by that. So let's imagine somebody goes along and types in, you know, well, what I just said, what, what do users think about our checkout process? If you set up the AI in the repository correctly, it will be intelligent to know, uh, and go, do you know what? We've got no research on checkouts at the moment, but here's some best practice for the product manager. But at the same time I'm going to make a note of the fact that we've got no research on that and send that to the user research and UX team so that they can now go away and do some user research in that area to fill in the gaps in the repository. So it's a great way A of creating the initial, um, uh, repository B, enabling people to access it better, but C, identify gaps that need to then be filled, you know, um, with more, um, you know, more research is a,

Marcus Lillington: is can AI, uh, can you rely on AI, uh to reliably do that though, really?

Paul Boag: Yes, you can, yeah.

Marcus Lillington: Uh, at least a question along that, using your example. I want some. My experience of AI is if you ask it something, doesn't know it makes it up.

Paul Boag: Yeah, that's because you, that's because it hasn't been trained properly. Right. Okay.

Marcus Lillington: You just did say if it's been set up properly.

Paul Boag: Yeah, so, so what you would do, you do, you do two levels to it, right. You could even argue three. First of all, you've got your basic instructions that say very explicitly, do not make shit up. Right. Uh, you know, or whatever other requirements that you need from it of how you want it to respond. Then as part of those instructions you say before you return the answer to the user, go through this quality gate, check Right.

Marcus Lillington: Okay.

Paul Boag: So you know, have um, you know, does it meet these requirements? And then finally you could get it to even go. Now I want you to check back against the original sources that you mentioned to make sure they're all existing and um, accurate. So if you add in these kind of layers of checks and balances in to make sure that what you get back is reliable. Okay, but even if, let's be super pessimistic and say 1 in 10 things that it answers is incorrect. Right. That's still better than having nothing at all. And even that incorrect, if it gives you an incorrect answer, it's still going to give you generic best practice. It's not going to suddenly say, oh, everybody thinks your checkout process is wonderful. Right. It'll go. I, worst case scenarios. It will go. I can't find any specific research on this. So what I'm going to do is make up some general best practice when it comes to checkout. Right.

Marcus Lillington: So in this scenario, I, I agree entirely. Um, yeah, yeah, it depends on what

Paul Boag: you're using it for. For example, medical advice. I would have a much stricter, you know, criteria on its answers. You know, so it depends.

Marcus Lillington: I give my money away.

Paul Boag: Yeah, that kind of thing.

Marcus Lillington: Yeah.

Paul Boag: Use some bloody common sense. Um, and you can also caveat these things. And you know, if you use Claude, for example, every time I start Claude, it's got a little line under it that says something like Claude makes up sometimes double check everything. Um, and you know, people, yes, are inherently lazy and don't do that. But um, it's still in this case that it's the consequences of it. Making stuff up is not the end of the world.

Marcus Lillington: It's a walled garden, which makes it fine.

Paul Boag: Um, so, uh, the reason I like this is that A, it gets, you know, it creates an easy interface for people to start asking questions about the user, a bit like those virtual Personas. B, um, it has huge cost saving benefits because before you're commissioning any new research, you could check what you already know. So those two things are really good. Now I don't think that A, this replaces your big posters on the wall, um, but I would maybe my eye. So then what you could do is you compare this repository with those digital Persona virtual um, Personas I talked about before. So in case you didn't listen to that, that basically what you do is you take the content of your um, uh, repository, you get AI to analyze it and then create a set of Personas based on the data that's in your user research repository. So now you've got a set of Personas that, uh, and you can create a chat interface on that set of Personas so that people can then talk to those Personas and get answers for them. Now, does that replace your pictures on the wall? Absolutely not. So what you do. Because that still means people have to think, oh, you know, oh, yes, I need to use that amazing AI tool. Right? Yeah, yeah. So what you do is on those posters, you put a QR code, so you snap the QR code and now you're talking to that user that's on the poster or a virtual facsimile of them. Right. So I just think, I think that's bloody amazing as a, as a way of, of getting the user's voice into an organization, organizing the research that's been done, identifying gaps in it. And I just don't understand why more organized. Well, I do understand why more organizations aren't doing it is because their UX teams are too small and too busy to do this. So hire me to do it. There you go. See. Oh, that was good, wasn't it? That was subtle. Yeah. So, um, yeah, that's, that was basically all I wanted to say on that. That, uh, particular subject. App of the Week or App of the month or whatever it is.

Marcus Lillington: So I thought this is App of

Paul Boag: your life, Paul, up of my life, yes, it is. Um, but I did pick this one kind of on purpose, um, because it relates to the conversation that we've just had. So the, the App of the, the month that I must have done this as a recommendation before because I'm obsessed with it is a tool called Notion. Um, you've probably heard a Notion, um, because they're, they're quite a big tool. And, and yeah, Marcus is right. I'm utterly obsessed with Notion. I run my entire business on it. Um, you know, if, if it, if Notion goes down, I go down, I cease to exist. Right. So, but I wanted to talk about it specifically today because I am pretty convinced it is the best platform out there for doing these user research repositories. Um, there are supposed specialist platforms that do it. Um, but actually I don't think they're as good as using something like Notion. What Notion gives you, it gives you two things. One is it gives you utter flexibility in the sense that you can structure your user research repository however you need to specifically for your organization, and it can, um, also take in pretty much any form of user research that you might have done. So whether it be a survey, whether it be data and analytics, Whether it be a diary study or a user interview or whatever it is, or user testing, whatever it is, you can bring it into Notion pretty simply and pretty easily, um, in order to create this repository. And then second, Notion has got a really powerful AI agent built into it that can run through this repository and, um, uh, you know, access everything in IT and search on it and that kind of stuff. So I think it's a really good tool. But that said, I'm talking to, um, the rni. Rnli. Yeah, that's right. Is it B. Lifeboats people? What is it?

Marcus Lillington: Rnli?

Paul Boag: Yeah, for some reason I wanted to put B.

Marcus Lillington: National Life Boat Institute, I believe.

Paul Boag: They're not a client, but I've just got chatting with them and we've been, We've been talking about, um, user, uh, research repositories and in, in their organization they use, um, Co. Co, uh, Pilot, um, and they've got. So. So they can do the same thing in something like SharePoint, for example, where they have all of their repository information in SharePoint and then click, co pilot just looks at SharePoint and pulls out what it needs to. So you can do it in a lot of different systems. But Notion, I think, is the one. Perhaps, perhaps I'm by. Well, no, I am biased because it's the one I really know, but it's a good one. It's definitely worth checking out, um, for organizing any kind of information like this. So I thought I'd give it a mention.

Marcus Lillington: I, uh, can't help but, uh, want to ask you the question. So I'm going to ask it. Paul, what if Notion did go down? Have you got backups in different places?

Paul Boag: What, what if you lose Internet access?

Marcus Lillington: Well, yeah, I suppose. Yeah, yeah. I'll just write letters to people.

Paul Boag: Yeah, exactly. I mean, there are, um.

Marcus Lillington: Yeah, fair point. Yeah.

Paul Boag: You know, there are things in our businesses that are single points of failure. It's a conversation that we've had. And I would feel differently if notion was. It's like every, every time Amazon Web Services go down, half the Internet stops working suddenly, doesn't it?

Marcus Lillington: Um, Cloudflare has been one of recent Cloudflares, another twice.

Paul Boag: Yeah, yeah, yeah. So we do have those kinds of dependencies.

Marcus Lillington: Um, yeah, fair enough. Yeah.

Paul Boag: So that's my kind of attitude. Notion is big enough. Right. That I'm confident in them, they're profitable. Right. Which I consider another big, um, factor in their favor. Um, but I mean, we could end up in a world where we don't want to be using American apps, for example.

Marcus Lillington: Um, yeah, yeah, some people I know, some people, you know, uh, have dumped all of their American affiliations or subscriptions, whatever you want to call them.

Paul Boag: Well, we're also, we're also finding um, an increasing problem with um, organizations. I've got several larger organizations I work with that can't use American software to, because of GDPR and, and, and that kind of side of things because they're just not compliant. Yeah. You know, if you're going to hold any kind of private data in them, which, which I have to say on a user research repository, you absolutely shouldn't be doing. Right. It should all be anonymized and you don't want to hold people's personal information on that. So it's a, you know, it's not relevant in that particular case but you know, in notion I've got contact information and you know, confidential information held in that and it's like, should I be.

Marcus Lillington: Possibly not. Although we, we're having an ongoing nightmare due to consent management at the moment. Um, uh, a lot of our clients were using Cookie Pro which uh, run by Bought by One Trust I think, um, and that the Cookie Pro product is being discontinued or it's being changed to be a ten times more expensive. As you can tell I'm not really part of this, this particular conversation. So we've been looking at alternatives and you think you find yeah, that's the one. Then you go, no, it's not, it's not accessible. Um, and all of this kind of keep going down, down this route and then you kind of come back to the conclusion that actually maybe the expensive one is the one you want.

Paul Boag: Yeah.

Marcus Lillington: And it's just like everything else these days that everything's just getting more and more expensive the other.

Paul Boag: It is interesting because the cookie notification thing drives me nuts. I mean I've talked about this but before but it's a waste of, absolute waste of time. Worst piece of EU legislation ever. Really good intentions, you know. I understand but all of the ad people that they were trying to target have moved to browser sniffing anyway. So it's not like helping at all with the, with the problem that they were trying to solve and just making an enormous pain in the ass for everybody else's sites. And then the other one of that, which I feel a little bit differently about, um, and it's more nuanced is the, the new legislation around um, unsuitable images, you know, so, so if you're going to use pornhub, you now have to prove that you're an adult, etcetera Obviously I, you know, I, I kind of fully accept that. I think that's very sensible. But the big problem that it's created is that a lot of companies that host imagery. Right. Yeah. In America have gone, screw it, we're not going to support that anymore, you know, but we're not, we're not going to comply with that. And so things like huge numbers of images now just, you cannot see this image in your region because it exists on websites absolutely everywhere across the Internet, but are being pulled from America, who don't want to be liable for this. And so he's just broken big chunks of the web because they'd not thought through the consequences. It's really hard to get this kind of legislation right. I was speaking to, to somebody from Ofcom about AI, um, boyfriend, ah, girlfriend things, you know, these fake boyfriend, girlfriend, um, and they're, they're thinking about how to legislate around that and they wanted to pick my brain over it. And it's like, it's really hard to get right without having unintended consequences.

Marcus Lillington: The one with the, the imagery, it's kind of like, well, the idea behind the Internet was that it shouldn't be, it shouldn't be kind of, I can't find the right word, but you know, closed down in any way. It's meant to be just this free platform. But equally, of course, as soon as you do that, then you end up, well, you end up with what we've got in the world now. Yeah. So. Which ain't great. So doing something, I think in this particular example is better than nothing. And if it's broken.

Paul Boag: Yeah.

Marcus Lillington: A lot of other things I do,

Paul Boag: I feel in that case I agree with you. Yeah, I do.

Marcus Lillington: Yeah.

Paul Boag: Because at the end of the day all it need means is that, you know, websites that want to operate in the EU are going to have to find an EU based image provider. Right.

Marcus Lillington: Yeah.

Paul Boag: And, and yes, that is a big inconvenience. But when you're talking about, you know, the safety of children, I think that's a, a very, you know, a price worth paying. I don't feel like that with the cookie legislation because instantly it became meaningless because, you know, um, the companies that were the offenders, the ones that the EU wanted to target, just switched technology. That's all they did, you know. So anyway, let's talk about your, your um, your interesting reads because you had some good ones. I, I, the first one in particular I had no clue about.

Marcus Lillington: Me neither. Um, this Dan at Headscape found this one, um, which Which I sort of was like, really? Anyway, so it's. I thought to start off, I've got a couple of interesting reads, a couple of articles this week. Well, I thought I would, um, take a different look at AI because obviously Paul's such a massive fanboy. Um, um, maybe we should be a bit careful. Or maybe that's not, that's not really what it is, but it's just looking at it from a slightly different angle.

Paul Boag: Yeah.

Marcus Lillington: And I'm picking on Google as well, particularly. I don't know why, but it just happens to be. Be the case. So the first one is a Tech Juice article, uh, entitled Google is Quietly Rewriting Headlines with AI in search Results. Which is like. Is it? Um, and then I started reading the article. Apparently it's been doing this for years anyway. Um.

Paul Boag: Oh, really?

Marcus Lillington: Yes. Uh, I'm not quite sure how it's been doing it, maybe even manually, but I don't know, it has been, it has been rewriting the titles, uh, of certain content in its, in its articles. But with the advent of AI with Gemini, I assume in this case it's gone into overdrive. And it said, I think it said that, let's say four months ago, we're just, we're trialing this and then a month after that it's in use and it's happening every day. Anyway, to use an example of the kind of thing that it's changing, the one that's come from this article, uh, it's a. It had an article that was originally titled I used the Cheat on Everything AI tool. And it didn't help me Cheat on anything. That was shortened to Cheat on Everything AI tool, which obviously completely changes its meaning. You go from this is rubbish to here's get your Cheat on it on Everything AI tool.

Paul Boag: Um, that's the fundamental problem with AI. Well, it's not the problem with AI, it's the problem with how people are using AI is you have to have a human in the loop.

Marcus Lillington: Loop.

Paul Boag: If you take a human out of the loop, if you have no human judgment in it, you're gonna get like that. Do you know what I mean?

Marcus Lillington: Yeah, well, absolutely. But I mean, I still, I'm still left. I'm still left a little bit not knowing where this is all coming from. Because unless, unless you've got your conspiracy theory hat on. And I'm not a conspiracist at all. But I'm thinking, well, with that particular example, you could say, well, AI is being pro AI. But then I keep coming back to why what's the point? Even like so before AI, why was Google changing the term to encourage more?

Paul Boag: Well, no, because they don't really care about traffic that much. They don't care whether you click through. In fact, they prefer you not to.

Marcus Lillington: All I can assume is that it's meant to sort of help. Better summarize. So we are helping you understand what the content is behind this title because it's all you're getting is the, is the title of the link, isn't it?

Paul Boag: Yeah, yeah.

Marcus Lillington: There are other articles, there are other examples in this article where um, um, the title's been changed to use words and subject matter that don't exist in the article at all. So again, it is really weird. But you know what I kick over to is what's wrong with just keeping the original headline?

Paul Boag: It will be something, it will be something about driving traffic when it. There's got to be of, of driving click throughs or try uh, improving engagement or some. There's some logic there. But I mean, yeah, I, I would agree. It, it's a, ah, it's a, a solution looking for a problem. Um, there's a lot of that with AI at the moment of people using it, um, to solve something that's not really fundamentally a problem.

Marcus Lillington: No. So that was a real eye opener for me. Um, you can't trust Google to recreate, uh, your headlines that you spent minutes over.

Paul Boag: I don't care anyway. I don't use Google, so don't give a shit. Well, I think people do. Yeah. I don't use Google search. I should add, it's not that I'm against using anything Google, but yes, you're right, the vast majority of people still do.

Marcus Lillington: So Google bashing part 2. Yes.

Paul Boag: Now this one I've got more opinions on. This one I do care a little bit more. More about. You are right. Like such a clickbaity title. You, you noted that. I noticed.

Marcus Lillington: Yes, this is the title. Testing suggests Google's AI overviews tell millions of lies per hour. Yeah. Which I did like that made me smile a bit. Um, but this is an Ars Technica article. Um, and there appears to be something in it. Um, and I wrote the words in my notes. We all use the AI summary overviews that Google presents us with. Then I thought, well, I do. So I guess everyone does. A lot of people use, uh, the AI summaries. And I guess what I mean by that is, uh, and this is purely anecdotal on my behalf, quite a lot of the time I'll be looking up something, I don't know, like the golf club that I went to over the weekend down near you. I'd look that up and if it told me what the address was, I'd believe it. I wouldn't go through to the website and check it. Um, but anyway, so this article talks about some analysis that was done, um, by the New York Times that suggests that AI overviews are incorrect 10% of the time, which is interesting. You used that figure earlier. Earlier, um, in this podcast, Paul. Um, they used them an example, uh, in the article where when asked for the date on which Bob Marley's former home became a museum, AI overview cited three pages, two of which didn't discuss the date at all. The final one, Wikipedia listed two contradictory years and AI overview reviews confidently, and that's a really important word, confidently chose the wrong one. Um, and there are many, many other examples of this. Google of course said well, you know, your research is flawed, blah, blah, blah, which it is. Um, but it then went on to sort of summarize saying, well yes, of course we can't get it right all the time, or AI can't get it right all the time. So that goes back to what I was just saying. My beef with all of this is if I'm like, if I'm the average user, people aren't checking, they aren't going through and checking responses, they're not checking that the address for the golf club is. They don't go through the website anymore. So being one being wrong 1% of the time, if we're talking about the entirety of the Internet is potentially a problem.

Paul Boag: Yes and no, is my response to that. First of all, I think this isn't a Google specific problem. I think this is a problem with large language models. Um, that they are predictive and they don't always predict accurately. Um, and so yeah, they will, will get things wrong. They're also reliant on the data that was, you know, that is provided to them. Um, of which, you know, you talked about. The Wikipedia one had two contradictory dates. Um, and so I don't think it's fair to pick on Google other than the fact that I guess they've got a bigger responsibility to be accurate just simply because of the scale of usage. Um, but here's where I'm less sympathetic is that we do have a responsibility. Um, and um, we fully accept that, that um, we tend to trust humans, we tend to trust what's written on the Internet. We shouldn't do that either. Right? Why is AI any different? You're oh, shocking. Something on the Internet's not true. You know, it's like. And so the truth is, uh, you know, why should I trust this article? I bet there's things in this article that's incorrect. I bet there's things that you tell me that are incorrect. That we live in a world where things are inaccurate. So it comes down to an individual responsibility. It's a, um, risk benefit analysis basically that we have to make every single day with every piece of information that we're fed, which is comparing the effort with validating whether that thing is true or not against the risk of getting it wrong. Right. So, for example, um, let's take your, your, your address. You know, that you wouldn't go through and check that that address is real. I would personally check. Because me getting lost is a bigger pain in the ass than clicking through and check. I'm not saying you were wrong doing it. I think we're all different. Right. You know, it's a bigger pain in the ass than, than just clicking through on that website and double checking that piece of data. So I don't think it's. Everybody makes out it's Google's problem or AI's problem. I just think it's. We, uh, accept it in every other aspect of life that things will be inaccurate. Why do we suddenly hold AI to a higher standard other than. And you nailed it earlier, the word confidently. M. Right. That I think AI says.

Marcus Lillington: Yeah, yeah.

Paul Boag: All it needs to add is something like. It looks like the address of your place is, uh, rather than the, the pl. Your. The address is. And this is where you get into a user experience thing and conversational interfaces. Yeah.

Marcus Lillington: Have a prominent link rather than, you know. Yeah. Right, right. But I, I think we're gonna have to maybe agree to disagree a little bit on this one because, uh, even though I, I think you're right in the only way that this is going to get fixed is if people change their behavior. But what is annoying is that you've got an extra step here added because of this, uh, potential extra step where in the past you used to type in East Dorset Golf Club or whatever it was. It's changed its name. It's Posha now it's the country club. Um, and the top link would be East Dorset, which I'd click on. There would be no reading the AI summary. I'd just click on it and go there, uh, and find what I was looking for. But now I read the IO summary that confidently tells me something that I go, yeah, that's right. And walk off. But what you're. But you're right. I need to. Now go. Okay. I need to go and check that on the website. So I've added. An unnecessary step has been added which is just annoying.

Paul Boag: I. I didn't know. I get that. No, I fully accept that. I think we. We do agree over that. I've literally just gone. I mean, perhaps I've turned the AI off on something. I never use Google, but I don't get an AI summary. I. I didn't ask a specific question.

Marcus Lillington: It is very. Sometimes it appears, sometimes it doesn't. Maybe it doesn't. Yeah, maybe address. All right.

Paul Boag: Yeah, address. When I added address to the end. It did. And then I. But immediately below that is a link to click through. Yeah.

Marcus Lillington: Do people click on those links? No, they don't. Is my. That's.

Paul Boag: And, uh, I would agree with that. So what you're saying is there's an added level of cognitive effort.

Marcus Lillington: Yes. Of.

Paul Boag: Of having to ask yourself, do I trust this summary?

Marcus Lillington: Based on what you're saying about most of what's on the Internet is a load of old tripe anyway, then nearly every time it should be. Well, I better had. So what was the point?

Paul Boag: No, I almost. I almost take the opposite approach. See, I see it the other way is. I see it as well, it's just as likely the bloody website hasn't been updated and is wrong. Right? Yeah, maybe.

Marcus Lillington: I mean, you can't complete. Obviously completely trust that. But the organization's website is got to be the place to find their address.

Paul Boag: It's got to be the definitive secondary place. Yes, absolutely. No, I accept that. So what's the answer then? Is it to get rid of those entirely?

Marcus Lillington: Uh, yes, but they're not going to people like them because they confidently tell them stuff quickly. But if 10% of it is wrong, then that's a problem. I don't know what the answer is, Paul. I'm not here to give you answers.

Paul Boag: That's part of it. I'm just. I'm trying on DuckDuckGo now. See, now DuckDuckGo is giving me a search answer as well. So it's giving me an AI answer. See, I like it. I just.

Marcus Lillington: That's. Yeah, I just said that I think they're great and I believe them. And it's like, this is handy. Don't have to faff about going to websites anymore. It just tells me the stuff. And then this article tells me that only one in ten. All right. Oh, uh, no, Sorry. You know what? I meant one out of ten's wrong anyway.

Paul Boag: But you see, even that I'm not convinced of that number. It depends on what you're asking.

Marcus Lillington: Wishful thinking, Paul.

Paul Boag: No, that's based on, on the benchmarks associated with the large language model. You see, in that particular example of Bob Marley, it wasn't the AI's fault. It correctly quoted Wikipedia. Wikipedia was wrong.

Marcus Lillington: Oh, it had. Or it had three. It had two or three dates and it picked. It confidently picked the wrong one.

Paul Boag: Yeah, but what does that mean? I mean to say it confidently picked it. I mean, what is the wording? Let's have a look at what the wording actually is. That's the key is the wording. Yes, I mean it is confident. The Dorset Goldfoot, uh, and country, uh, Club is located in Hide, near, uh, postcode is.

Marcus Lillington: And it's probably right.

Paul Boag: Well, it, Yeah, I suspect it is. Yeah. It's in. Yeah, just down the road.

Marcus Lillington: Lovely part of the world. Gorgeous. Just.

Paul Boag: Oh, everybody just won'ts and whinges about any new technology that comes along. And so, you know, you. Yeah, I, I haven't got a problem with it. It's an education thing. People just need to uh, to realize that AI gets it wrong sometimes. It's like going, why doesn't notion make my breakfast for me?

Marcus Lillington: Well, don't, don't think that's quite what that is.

Paul Boag: That isn't fair. I know, I know.

Marcus Lillington: Although wouldn't it be good if AI did do that? Why doesn't A, AI, uh, paint my living room for me? Wouldn't that be bloody fantastic?

Paul Boag: To be serious. My. The serious point there is that AI is claiming to make my breakfast and paint your living room, but is not doing either. That's the fundamental issue, isn't it? It's the clay, it's the claiming with confidence. Have they got any kind of disclaimer? All AI responses may include mistakes written in tiny text, little TED text.

Marcus Lillington: Ah.

Paul Boag: So I guess it, you know, they're covering. Although that doesn't appear. Interestingly that disclaimer doesn't appear until you hit the show More button.

Marcus Lillington: Yeah. And quite a lot of the time you get what you want or you get what you think you want right at the top.

Paul Boag: Yeah. Which is where the, the message needs to be. So you know, it's not. It does have a link.

Marcus Lillington: Uh, the links are in the top. Okay, fair enough.

Paul Boag: Yeah. But it could at least say. See, this is basically what we're saying is this is a user, User, uh, uh, interface problem.

Marcus Lillington: Well, how kind of I think it should just be dumped and just go back to clicking through to the. What's more likely to be the definitive source? What's wrong with that? Nothing.

Paul Boag: Yeah, but they're not doing that.

Marcus Lillington: No, they're not going to. But.

Paul Boag: No, but. All right. Within the constraints of the business model that they're operating under, which is that they want you to stay on the website so they can show ads. Right. We can wish all they want to. Oh, let's strip out. I wish that they'd get rid of all of the ads. I wish they'd go back to the clean interface they used to have when they were making no money. You know, that's a, it's like a ridiculous argument.

Marcus Lillington: But so many things have been ruined by this. I use the Monzo banking app, which is still pretty.

Paul Boag: Okay, why don't you pay for search then?

Marcus Lillington: I'm paying for search, Paul. Don't be ridiculous.

Paul Boag: There you go. See, that's the problem. So therefore you are the product. It should be subsidized. It should be like the nhs, right? It's tapped, which is taxed on it.

Marcus Lillington: Well, to be honest, I mean, that's not a bad idea. Everyone uses it all the time, so why shouldn't we just be taxed? And then.

Paul Boag: It's not the American way.

Marcus Lillington: Clean.

Paul Boag: Yeah.

Marcus Lillington: No, certainly not the American way. No. God, no.

Paul Boag: It's quite an interesting question.

Marcus Lillington: When Obama comes in, do you want, do you want, um, free health care? Oh my God, no. And we're all like, what?

Paul Boag: Why wouldn't you want that?

Marcus Lillington: Anyway, still, don't we?

Paul Boag: We need BBC search. Uh, that's what we need.

Marcus Lillington: Yeah.

Paul Boag: Funded by a license payer fee. BBC make it happen. There we go, Done. Um, Marcus, that. We went off a complete tangent then, didn't we? Which is a good job because the show was running short time wise. So now.

Marcus Lillington: Now.

Paul Boag: Yes. What's your joke?

Marcus Lillington: I have a joke. Paul Edwards shared this one and it made me giggle. So I'm entering. Sorry, start again. I'm entering the annual give helium to a sheep contest again. And I'm a bit nervous. Last year the bar was very high.

Paul Boag: I, uh, knew it'd be something to do with bar and I was trying to think, what would a higher pitched bar be? But yeah, no, that was better. That was a good one. All right, I approve. They're getting better recently. Marcus, I think you're, you're dialing in on the things that amuse me. That's what it is. All right, that's great. Thank you very much everybody for joining us. Um, I'll be interested if you, any of you do listen to this, you hit us up on, on x or, or LinkedIn or whatever to see what you think about some of these subjects. Because I, I really, I'd be interested in what other people think.

Marcus Lillington: It's an interesting one.

Paul Boag: All right, thank you very much, Marcus. Thank you for listening, everybody, and we will talk to you again next month. Goodbye.

Marcus Lillington: Bye. M.

Paul Boag: Sam.