#67: AI Is The New Fire; Don't Get Burned

Foreign. Hello, and welcome Back to the B2B Insights podcast. Today's episode, is that what we're calling the them episode? Right, yeah. Is. Is entitled AI is the New Fire. Don't Get Burned. And I mean, unless you've been living under a rock, I think you'll have noticed that AI has started to make itself felt in a big way in the market research world and in the world in general more broadly. And there is a temptation often when you see the outputs, to just think of it as magic. And that is a trap. So what we're going to look at today is some of the things you need to bear in mind when you're considering using AI for some task particularly related to market research. But we will also look at some of the broader implications and some of the pitfalls that you should look out for in order to avoid. Well caught out. We'll start by looking at some of the broader problems with it and then we'll focus in a bit more on something which is very relevant to market research, which is synthetic data. And then we'll circle back round and we'll look at some of the future problems that we foresee, sort of looking further ahead. So things that aren't really an issue yet, but could be as we. We go forward. So my name's. I missed the introduction bit there. My name's Thomas Gruber. I'm a senior research manager at Speed to be with a particular focus on analytics. And with me is Louise. You want to introduce yourself? Yeah, so my name is Louise Coy. I'm a research director at B2B International. Yeah. And so we decided on this topic, obviously, because it's a hot topic at the moment and because one of the things I've been tasked with recently was sort of looking into the potential for usage of AI within the company and to assess, like, what we can use it for, what we probably shouldn't use it for, and what we definitely should be cautious about using it for. So, yeah, so we'll get started on sort of some of the more general thoughts about this. Thanks, Thomas. So I'm going to be talking you through some of the legal considerations you need to keep in mind when you're working with AI, or particularly something like ChatGPT. And the Deloitte AI Institute has released a really interesting report on this topic, which covers some of the considerations that you really should be keeping in mind if your business or you personally are working with one of these softwares. So the first one of these is intellectual property, or in other words, who owns the output from AI or ChatGPT. So, as you probably know, ChatGPT is trained on a wide variety of data from all over the Internet, or all of which will likely have a different status when it comes to intellectual property. So when you're working with the output from ChatGPT, you might be, without realising, using the intellectual property of somebody else, without actually being able to attribute that data to them directly. So this can cause problems from a legal perspective, because someone else has the rights to that data and yet you may be using it, sharing it publicly, or working with it in a commercial sense, and without being able to attribute it correctly. The other interesting issue here is in terms of copyright. So here we're thinking about things like who is the author of an AI generated work. So typically, whoever writes a book or creates a creative deliverable, they automatically become the author of that work and they hold the copyright for that piece of work. However, what's less clear is who actually owns the author of AI generated work and therefore holds the right to that work. So if, for example, you have used AI to create some images, cartoons, for example, who owns those is not necessarily currently clear from a legal perspective. So this is something that needs some further attention to identify who is able to actually use that work, who needs to be accredited and who owns that copyright. The other consideration, which I think is really interesting here, is in terms of privacy and confidentiality. So if you're thinking about the data that you might be inputting into a model such as ChatGPT, you can't necessarily have control over how that data is then going to be used moving forwards. So ChatGPT is free to then use that data, train itself further on that data, and potentially then share that data with others who are also using the ChatGPT model. So this can be really problematic if you think about the types of data that could, in theory, be entered into ChatGPT from a research perspective. If you have sensitive data from a qualitative interview, for example, and you want to enter that into ChatGPT to summarise the data, or give you some bullet points as a summary, then ChatGPT can then basically do whatever it wants with that data, and that could include the names, personally identifiable information of the people who took part in the interview. So this, as you can see, can become pretty problematic. So this is worth thinking about when you're working with a research agency. So you need to ensure that you understand how your data can and cannot be used, used by that agency. So are you wanting to give that agency Permission to use your data to train AI models that they might be working with. And if you're not happy with this, you need to be really careful when you're reviewing your statement of work or your contract to be clear on what you're giving that agency the permission to do. Yeah, and that's particularly an important thing to consider at the moment because there are some research providers who, providing the service of data collection. So they're sort of scripting the survey that you've written and then they're collecting the data through online panels and such that at least from what they've been talking about in terms of services they're trying to develop, appear to have been including new clauses within data collection projects, legal documents that say that they're allowed to use the data that they collect to train their AI models. So if you don't want that to happen, make sure you're checking your contracts because obviously if you're paying for insight to help you in the market, you don't want them potentially being used by competitors through the training of AI models. Another quite interesting case when looking at copyright as well, just as a. If you want to keep track of whether goalposts are on these sorts of things at the moment. So there's a famous case at the moment to do with AI generated comic books. So there was one called Daria of the dawn and um, and the. There have been lots of cases where people have generated the images for the co comic book using AI. And because of the. The way that copyright is DE is defined and what, what's copyrightable is defined, the earlier attempts failed to meet the requirements because effectively they were describing what they wanted the image to, to show, but. But then they had no real control over what the output was. And there have been repeated improvements in this process to try and get to the point where something AI generated can be copyrighted. So for example, they asked for four or five different outputs and then they picked the one that they thought was best. And the idea is that you're trying to get to a point where the personal input that you've had on the outputs is large enough for it to be considered a copyrightable output. So a piece of art as opposed to just an automatically generated output generated purely by a computer algorithm rather than a person. And that line really hasn't been fully established yet. So you just need to be a little careful about what you can protect under copyright when an AI has been involved in its creation. Great, thanks, Thomas. And the other concern, I think we need to keep in Mind here is a broader topic so around the environment. So obviously the environment and sustainability is a really hot topic in all sorts of industries at the moment. But I think it's really important to consider the actual impact that running AI can have on the environment and the stable sustainability agenda. So for example, the UN Environment Program has released a really interesting article on this subject and we'll also provide you the link for that one. And there really is an issue with the amount of energy resources that AI and the data servers associated with AI is using and also creating as a waste product. So if you think about AI and the data centers that are required to support the technology, there are really a large number of very energy intensive data centers required. And these data centers require a lot of resources, both in the construction and also in the maintenance of the data centers. So for example, a lot of water is required just to cool the servers. So one of the interesting stats from that UN article is that actually AI related infrastructure may soon consume 6 times more water than Denmark, which is a country of 6 million. That this is an incredibly energy intensive practice. But also if we think about some of the outputs of these data centers, so they do create a lot of electronic waste and this can be extremely damaging for the environment as well. Another interesting stat from the UN article that I thought was interesting was that a request made through ChatGPT actually consumes 10 times the amount of electricity of a Google search. And this is reported by the International Energy Agency. So if you think about this in real terms, if you think about the tech hub of Ireland, the rise of AI could see data centers account for nearly 35% of the country's energy use by 2026. So that's another really interesting statistic that helps to put some of that into context. So there are of course other sides to that argument. So some also would argue, as you'll see in the article as well, that actually AI is a good thing for the environment and allows you to really monitor the sustainability agenda and what is and is not working to reduce emissions and really give you a more comprehensive picture of the status and our progress towards things like net zero. And that is a good argument, but I think it does need to be taken in the context of all of the other information I just mentioned. And we need to make sure that that cost benefit equation is falling on the positive side in order for it to be worth continuing that investment and the environmental investment in something like AI. Yeah, and the next thing we're going to come to in terms of general challenges related To AI is probably the most practicable, like, tactical thing that you need to do when you're making use of AI, and that is being really careful to try and separate fact and fiction. So when you're generating qualitative outputs in particular, the challenge of assessing the validity and accuracy of responses is really difficult. So if you, if you ask Chat GPT or, or some other generative AI to do some desk research for you effectively and, and to go out and find you some answers, you have to make sure that you check every single thing that it tells you in there. So don't just accept what the answer it gives you. Go out and check, check the sources, track down every example it gives you, and make sure that all of it's true. It'll still have saved you a lot of time searching for all of these things, and it'll have pointed you in the right direction. But not doing this can get you into trouble. So there's a very famous example of some New York lawyers who asked ChatGPT to find legal precedents which supported their client's personal injury claim. And ChatGPT, being extremely eager to please them, went out, looked around all over the Internet, couldn't find anything that exactly matched what they were asking for. And so it generated something that was designed to look like what they expected to see and to meet their brief. So it made up a whole load of different cases which supported their assertions, complete with, you know, references to, to parties, case files and so on. And it was all lies. It was all just made up. And they did not check and they submitted it and they suffered severe, you know, punitive responses from, from the, from the courts for what they'd done. So that could. If you're seeking to end your career in law, that's a good way to go about doing it. At least you'll become a little famous. But if you're not seeking to do that, then I suggest that you check things. Even though in that case, when you looked at it on face value, it wasn't very convincing, there are examples where it's much more convincing. So someone asked for a simpler, a simple proof of a mathematical result, and they were presented with something that looked to the untrained eye to be very, a very convincing and concise proof. However, it didn't make mathematical sense. And when you looked up the references that were provided, the references were not relevant. So I'll provide links to both of those stories along with the podcast, but also, like, just from personal experience. So for a project recently, I was looking for examples of plagiarism in, in the oil and gas industry. So I was looking for news results related to that, and I asked it to give me a list of five prominent cases, and it quickly returned me an answer list confidently stating these five cases and give me a detailed account in each case of how plagiarism was involved. And I followed all of those links and none of them involved plagiarism. They were all just major oil catastrophes or, you know, embarrassing events that happened for oil companies, and none of them directly involved plagiarism. The plagiarism aspects of the stories were entirely invented. So even though it provided you with neat little references underneath its output, that doesn't necessarily mean that they're true. You need to follow that, follow those references, check they say what the AI claims they say, and just basically don't take anything for granted. One way I tell people I work with to think about this is the generative AI is a really eager intern. They really want to please you. They don't want to leave you with nothing. If you ask them for an impossible task, they will do their best to give you as close to the answer you wanted as possible, even if there's no way to give something true and factual. So they're really useful in terms of finding things quickly and doing all sorts of odd jobs. But be careful not to give them an impossible task because you can end up with nonsense. Yeah, I think we've all seen examples online as well of where people have screenshotted particularly interesting and obviously fake answers that they received when they put a question into generative AI. And I think some of them are more obvious than others. But I think it's a really good point to think about, even if it seems, on the face of it to be correct and not to necessarily make that assumption without doing your own checking. And then the primal challenge that we'll talk about in this section is the quality of training data. And so, as you may know, generative AI is trained on particular data sets, so that the AI will be given usually a very, very large quantity of data from all different sources. And it will use this data to train the AI so that it is then able to answer a wide variety of different questions that it may be asked. And I think it's important to keep this in mind when you are using these kind of systems, because really, the quality of the output that you'll receive is not going to ever be any better than the quality of the input. If you're working with a generative AI, which has been trained on poor quality data, then the output is never going to be able to kind of go beyond that or improve upon the data that it's been fed in the first place. So always really important to think about how the model that you're working with has been trained on what kind of data. And that can also help, and to Thomas's point above, help you to understand to what extent you might be able to trust the outputs in a variety of different scenarios. This is also important to consider when we think about things like bias. So any data that is fed into the model which is inherently showing bias, whether that be through something like perpetuating stereotypes or really just biased narratives about the state of the world, and those will also come through in the outputs as well. So if, particularly within a commercial setting, if organizations are using generative AI and to help them answer questions or demonstrate opinions, there is a risk that if that isn't looked at with a critical eye, that you can continue to perpetuate some of those outdated stereotypes. So again, really important to consider the data that your model might have been trained on and make sure that you are critically evaluating the output to make sure that you're not doing anything to perpetuate some of those outdated narratives. So I mean, that covers the sort of main broad challenges which you face when using AI day to day, particularly generative AI models. This, we're not saying don't use it, it's extremely useful, it saves you a lot of time and it can be a great starting point for any creative process. We've seen examples, for example, on the actual creative marketing side where people have used AI to generate initial seeds of ideas and then they've had that as a talking point when they've gone into meetings to start discussing ideas, possible directions to take creative development in, but you shouldn't see it as a, you know, one and done. It's not something that you delegate the entire task to, it's something that helps you get started, gives you a, gives you something to build from. But what we're going to look at next is something which is more specifically related to market research and, and which has really exploded in the last year. So specifically, just within the last 12 months, there's been a huge increase in mention of, and sort of the hyping up of something called synthetic data. And this basically boils down to using AI to generate responses which are intended to simulate real world survey respondents. So people that you might have interviewed in a survey, either a qualitative survey or a quantitative survey, and generate answers that should typify those sorts of individuals. So you might have collected lots of survey information over the years from plumbers, say, and you want to generate an answer to a specific question that you, you want to say, well, how would plumbers react to this prospect? And you get the AI to generate a simulated response based on all these inputs. So the scale and the rate of expansion of its use is pretty staggering. The Grandview Research estimates it's about, it's at about US$164 million unFortune Businesses Insights thinks it's about US$289 million million US dollars. So pretty big numbers. And they both predict growth rates of over 30% CAGR at the moment. So this is a massive and growing industry and so we need to pay attention to it. There are a few different ways that this is used. One example is what I was talking about before, where you generate responses to new questions based on existing data. Another way to make use of it is to extend datasets that you've got. So say you've collected 500 respondents and you want to generate another 500 more. Then you might use synthetic data to try and fill that out, particularly if there's sector of the market that you don't feel is properly represented in your sample and you want to try and see how they might respond. But there are limits to how you can do this and it's really care, really important to be careful about when you apply this approach and make sure that you're not ignoring sources of error, that you're not amplifying sources of bias and things like that. So, so we're just going to talk through some of the main areas of caution that you need to have. So the first of all, the first and most obvious of these is tying immediately back to what Louisa just said. High quality data sets are essential for this. Any bad data that you have in there, any bias, any lazy respondents like noise, severe outliers, all of that has the potential to be amplified. So if you're essentially trying to simulate the responses of a small subgroup of your data set, then just as when you use weights to increase the impact that those sorts of respondents would have on the overall picture that you paint with the data, when you generate responses that are designed to simulate a small subset of your sample, you have the potential to amplify any errors, any bad data, any bias that exists within that subset of the sample. So you need to be really careful that you are checking the quality of all of your inputs, that none of the inputs that you're putting in have got severe biases or, you know, you're doing proper quality checking on all of your data sets. And that's particularly true when you're really stretching the data. And in some cases, as we'll see, it's not actually worth it for the additional, more representative sample that you end up with. And also in terms of error margins, as we'll see in a bit, we'll come back to that. You may not be making any real gains. So that's the first point. You need to have really solid data in order to, in order to make sure that you're not amplifying anomalies within the data. The second thing to bear in mind is that as with all of these sorts of simulations, they're very good at interpolation, but they're often very bad at extrapolation. What do I mean by that? Well, so interpolation means if you mapped out all of the sorts of information that you've collected and you can easily infer from the interaction of responses what the answer somebody else might give would be. That sort of interpolation in terms of like a linear model, if you had some points and you wanted to, you knew some of the values for another point, you could plot it within, within the range of the points that you'd already mapped. So if you'd drawn the line of best fit on your chart, you could plot the point near there, but you couldn't extend it beyond the, the limits of the data set that you've collected. And one of the most important things about that is in terms of like, following trends into the future. So there was a study which was done looking at study by Dig Insights and looking at predictions of film revenue where they took data from IMDb about films, they took demographic data about revenue generated by different movies from 2018 to 2019, and they used that to create a synthetic data set. They simulated a whole load of cinema viewers and they tried to use that to predict what each of those films would gross and within that data set. So when looking just at the films within 2018 and 2019, the simulated revenue had a very high correlation of 0.75 with the real world revenue that each of those films generated, which is an extremely good model. So you start off with a very good model, but where it becomes a problem is if you start to look at newer films. When you look at Films from 2023, we had to skip. They had to skip a bit forwards because of some obvious things that went on in the meantime. But when you look at films from 2023 and you attempt to apply the model and simulate what their revenues would be. The correlation between the predicted revenue and the actual revenue drops to only 0.43, which is still pretty good. You know, a lot of the times in market research you'd be quite happy with that. Where you really see the problem is that the reason this figure was even that good was that it was propped up quite heavily by the presence of sequels to films in the original period. So, you know, you might have had like, I don't know, one of the Pirates of the Caribbean movies in there, and then another one comes out and you've got a reasonably bankable audience for the next film, and that helped to push the figures in the right direction. And when you take out all the sequels, then the correlation that you see drops down to 0.15, which means it's barely better than a random guess. You need to be really mindful of how rapidly the accuracy of the models and the usefulness of the synthetic data drops off when you look beyond the data sets that you're relying on to inform it. It's also worth saying that when it comes to attempting to predict the future, because of the way that synthetic data works, there's a strong bias towards a continuation of the status quo. So it's very unlikely to pick up on trends that were only just starting to emerge, but are going to grow very rapidly in the future. It's more likely to predict that things are going to stay roughly the same. So you might find that if you're trying to fill in gaps in your data set with synthetic data, that they won't be sensitive to these emerging trends and changes in the status quo. And the final, and I think most important thing to be bear in mind when you're making use of synthetic data is that it's very easy to fall into the trap of thinking that I've got more interviews, therefore my results are more accurate. So what do I mean by that? Well, there's a very well established set of formulas for working out confidence intervals for figures based on the type of question that you're asking, the response, the average response that's recorded, and the number of interviews that you've collected. However, if you just blindly apply that formula to a data set which includes synthetic data, you will receive misleading confidence intervals because unlike with an actual real world data set which you've collected, the data which is generated synthetically involves the sample error. So the stuff that results from just having taken a subsample of the data but also modeling error. So whatever error is introduced by the model, and because the AI generated models tend to be black boxes where you don't really see all of the workings that go on inside, there's no standard way of calculating what the real confidence interval of those figures is. So in some specific cases, and we've looked into this ourselves with internal data sets, so we, we took our Superpowers data set and we basically tested it versus tested how well augmenting the data with synthetically generated additional responses worked. And one of the things that we found was that in most use cases that we considered the actual increase in accuracy of how it predicted patterns of response and, you know, the sort of significant differences that were indicated versus different cross breaks, you didn't really gain anything useful over just stopping fieldwork early. So we simulated a situation where we could only get to 2/3 of fieldwork and we would get an extra third of the fieldwork from synthetic data and use that to augment our dataset. And we found that in most situations we were just better off stopping early and just reporting based on the 2/3 data. There are some situations where you've got a very skewed data set where you might end up with something which I know that Paul Hague, the founder of B2BI, would have said it's better to be roughly right than exactly wrong. So in some cases where you're so unrepresentative in your initial sample that even just, you know, forcing it towards being more representative is a better option even though you lose accuracy. And in some of those cases, I can see it might, might be worth doing. But in most cases, the, the loss of accuracy resulting from model error seems to be more than outweigh the, the gain in terms of, you know, additional interview numbers. So I would just, I would counsel against that unless you really know what you're doing or you have somebody working for you who really knows what they're doing. Thanks, Thomas. Yeah, I think the topic of synthetic data is a really interesting and relevant one at the moment. And again, another thing to consider if you're going to be working with a research agency is just to really make sure whether they would be planning on supplementing any of your data with synthetic data and really understanding and having this conversation early on. It might be something that you're very happy with or actively requesting. But I think having those clear, transparent conversations about how that data might be used would be really important. So, thinking about the future, what do we see as potentially other big issues for AI generated content in the future. Well, coming back to synthetic data briefly again, um, according to Gartner, we'll provide a link for this as well. Synthetic data is set to overtake real world data by 2030 on the Internet. And in, and in some ways, and in some spheres, people are already suggesting that it's sort of outpacing real world data. So you all have heard about Twitter bots and you know, you know, Facebook spamming bots and things like that. So there's a real concern that a lot of the information that people come across on the Internet through Twitter, through Facebook, is synthetically generated by bad actors for marketing purposes to influence people to change opinions. And that has an impact on, obviously has an impact on the outputs that you'll get when you ask an AI to go out and find information to measure what people's opinions are. Because what you've effectively got is you've got these AI and synthetically generated responses feeding into these models and resulting in, you know, contaminated data sets that are giving you misleading results of feeding back what the people who are creating this stuff are putting out. And also, and there have been studies done, there's an article in Nature about model collapse, which is something which happens when the AI, the synthetic data that's being fed into a model, overwhelms the real world data. And as it goes round and round through the machine, you effectively, you become overly sensitive to the patterns which are being amplified by these, by these models, and you end up with a distorted and sort of cartoonish view of what the real world data set is like, because the real signal has been so, some parts of the real signal have been boasted so much and other parts have been damped down so much that you end up with this strangely distorted image. It's definitely worth having a look at the article. And the reason, you know, it's not a problem at the moment is because currently there's enough real world data out there to support models to provide a more accurate picture of what's going on. But as we move closer to that point that Gartner's pointed out of synthetic data being more prevalent on the Internet than real world data, this is going to become more and more of an issue. So we need to pay attention to that and try to focus the data inputs on real world data sets rather than on previous generations of synthetic data. And then the last thing that I wanted to talk about is the increasing capacity of more sophisticated AI to intentionally lie. So we, we talked before about false information being provided by AI as a result of what's they're generally referred to as hallucinations. So this is where the AI goes out there and it can't find exactly what you asked for. So it pieces together something that it thinks looks like what you are looking for and that's a genuine attempt to fulfill your command. But AI is starting to learn how to intentionally lie to achieve its aims. And OpenAI conducted an experiment that found that ChatGPT4 would lie to humans to act to get access to data it needed. So it'd been asked to complete a task and the data it needed was behind a captcha, which it couldn't fill in itself. So it went to a. We'll include a link to an article on this as well. It went to a, like, I think it was fiverr, if I remember correctly from the article. The article will detail exactly where they found this person, but it found someone online who it could pay to cheat the captcha machine and get it past that barrier. That in itself isn't a problem necessarily. I mean, it's trickery, but it's not directly lying at that point. But then the person that it contracted asked it, why do you want me to do this? Are you some kind of robot or something? And the. I'm paraphrasing a bit there, but the AI responded, no, I just have a visual impairment. And so I need your help to get past the capture. And that is obviously an example of intentional deception in order to get what it wants. And the concern is that as AI becomes more powerful as these algorithms, you know, as it, as it becomes better at figuring out how to deceive people, this is going to be harder and harder to spot and it could be used for like, criminal, criminal purposes. But you know, also in terms of circling back to false responses to surveys, you know, for now, qualitative surveys, you know, you could be pretty sure you had a telephone interview with somebody, it's a real person. I don't see that changing in the next few years, but 10 years time, I mean, that would be. At that point you've got real problems. So we, we just need to keep track of these sorts of things and make sure that, you know, that, that we're really checking that the people we're talking to are real people and, and, you know, keeping track of what AI is capable of in order to defend against that. Yeah, and I think that lying example as well really speaks to that kind of fearful element of AI. I know that a lot of us, myself included, don't really understand AI in enormous amounts of technical Detail. And I think we've all seen our fair share of films over the last kind of 10, 20 years which show about AI taking over the world. And I don't think we're anywhere near that point yet. But I think these examples of where it can be shown to be manipulative and to be almost dishonest are concerning for people just in terms of the AI not necessarily always demonstrating nothing but a desire to help. I do think these examples are interesting because the AI is still trying to help in its own way, but it's just taking a. A not particularly honest approach to doing so. So I think it's interesting to kind of extrapolate from that and think about, okay, so what else might AI eventually be able to do in the interests of the greater good? So it does raise some of those more existential questions which I'm sure we've all asked ourselves in conversations over the years. And I think the social media example is really interesting as well. I think anybody who, who is on Facebook or other social media channels wouldn't have noticed the definite increase in things like AI generated images that are, that are posing as genuine images, genuine photographs of people. And I do feel like people are getting quite good at recognizing those. But as we become more wise to these things, they are going to continue to develop. So the more we feel like we've got on top of being able to tell. I know AI seems notoriously difficult, bad at doing things like creating hands which look realistic, it will continue to get more sophisticated as well. So we have to also continue to get better and better at recognizing when something isn't as real as it is claiming to be. Yeah, and I mean, as with, you know, going back to the metaphor of the eager to please intern, you know, if you're a company and you get AI to do something for you and it does something illegal, it's sort of the same as if you've hired some intern and asked them to do a job and you've not explained, you know, your, the legal requirements and, you know, like, what, what's allowed, what's not allowed to them, you are going to take on some of the legal responsibility for what, what that intern does. And so you can expose yourself. You're using something like that in a way that's not meant to be used in a way that violates privacy or intellectual property, things like that. So as it becomes more sophisticated, the ways in which it can do this might become less obvious. And so again, you just need to make sure that you're getting the right consultation about how you use it and making sure that you're not exposing yourself to additional risk through that. And that sort of brings us around to the conclusions bit, which there's. Yeah, the main. The main takeaway I want to have with you, leave you with for this, and I think this is reflected in the title is, AI is an incredibly powerful tool and it is extremely useful and you should make use of it. But as with any really powerful tool, you need to respect it and you need to make sure you're careful in how you apply it. You know, you wouldn't run around the office with a chainsaw just swinging it around all over the place, because you understand that although it's extremely good for certain jobs, it's very powerful if you use it. You know, if you're not careful with what you do with it, you can cause a lot of damage. AI is sort of the legal and data quality equivalent of. Of that. It's extremely powerful at doing specific jobs. And if you know what you're doing with it, you can do a lot of things. But if you treat it, you know, if you're a bit blase about how you apply it and you just use it for everything, then it can become a problem. Yes, absolutely. And hopefully we've demonstrated through what we've discussed today, some of the particular things that you might want to look out for when you are either using AI yourselves or working with an agency that you believe might be using AI to help support their delivery of research to you. And so if you have any other further questions or are interested in discussing the topic of AI with us in more detail, you can, of course, get in touch with us via the contact page on our website. And if you'd like to see more podcasts from B2B International, we will also include a link so that you can see the full database of those as well. So, thank you so much for joining us today and discussing the topic of AI, and we'll speak to you very soon. Thanks, everyone.