Data Quality, AI, and the Future of Survey Research with Mario Callegaro

Speaker A: I hate when people say human in the loop. No, no, no. Researcher in the loop. The same prompt will give different answers. I just finished teaching an eight weeks class on AI assisted survey to the students at University of Maryland in the survey program. The other thing we need to deal is anxiety because you need to realize that you will never keep up.

Speaker B: Today on Survey and Beyond, we are joined by Mario Caligaro, founder of Caligaro Research. From pioneering online planning methodology to leading research at Google and, and now advising companies on AI assisted research. Mario is here to show us why data quality matters more than ever and how technology can either strengthen or weaken it depending on how you use it. Mario, welcome to Survey and Beyond.

Speaker A: Thank you. Yeah, I'm very excited.

Speaker B: So to kick off our discussion today, could you maybe walk us through your career journey? I know you started with survey research, you transitioned to user experience and market research. So just tell us a bit more about how this played out in your life.

Speaker A: Definitely. I started my career I was a sociology student in Italy at University of Trento. And after I took my first research methods class I got very excited about surveys and especially question on design. And so to the point that I actually even did the same exam twice, which is something that actually you can, I checked, you can still do. In Italy you can do an exam twice where the second time you just work on a specific subtopic, you agree with the professor on the topic and you generate like a small paper or something by option. It's optional. Yes. Not mandatory. And so the second time I did it actually on questionnaire design because I was really excited about writing questionnaires, you know, how do you write the questions? You know. So that's where I got very excited. And then I was lucky enough that the University of Toronto had uh, a collaboration with um, University of Berkeley in California and we were uh, using their own service software to do telephone interviews. And so I was recruited to run basically a small Cathy lab and with the help with a uh, professor who came to teach us, Tom Piazza, who came to teach us how to use the software. And everything we did, we did few studies like telephone service study. So I was training the interviewers which were mostly students and managing this 20 workstation kind of telephone interview center. That's where I got into really listening to lots of surveys and asking questions and understanding how sometimes we think our questionnaire is great and then you start talking to people and they get confused on a specific question, for example. So that was really an eye opener. Then I started working at the university. I Was in a uh department called Sociology and Social Research for a few years until eventually I applied for a scholarship at ah, the University of Nebraska because they had a master program in survey social methodology. Still now there are many and the scholarship was from the Gallup organization which I guess everybody knows. And luckily I got the scholarship, I was able to move to Nebraska and I did a master and then my professor eventually convinced me to also do the PhD. So I spent seven years there, give and take. And then after that my first job was actually as a survey scientist for Knowledge Panel which is a probability based panel of the US population. The company doesn't exist anymore but Knowledge Panel is still around and the company's been acquired by Ipsos. So you can still see, you know, Ipsos Knowledge Panel. And that was great. I was there for a couple of years and I was really working on you know at the time I'm talking about 2007 8, 9 web service were really taking off and so we had all the issues of how to design a question or how do you show questions on a screen. We didn't have mobile phone issues then. It was mostly desktop basically and understanding and obviously Internet speed which was very slow. So there was customers who wanted to see show like a video which was now, you know, we don't even think about it. But there was some technical challenges so I learned a lot. I had an excellent team. Then two years later I got a ping from LinkedIn, a recruiter from LinkedIn saying hey, we are looking for a survey scientist at Google. I was like what? I mean Google surveys. I mean funny enough I was already living in Mountain View which is a headquarter, but I only drove around Google and I saw people going around by bicycle, you know, the funny colorful bicycle. But then as much I didn't know anything, I was about Google search, you know, and then I did a bit of research or maybe they do some surveys. They may, they talk to their customers. Okay. And that's exactly what the job for was for a team which is called quantitative marketing. And my first project was running a survey of our advertisers. And so that's how I basically started my career at Google.

Speaker B: And now you are running your own company, right?

Speaker A: Yeah, exactly. Yeah, I left almost two years ago running my own company consulting on surveys, search of obviously market research because I was in marketing first and then user experience. So I also did a uh, transition from let's say market research to user experience search later on in Google. So I spent about eight years in market research and seven in user experience, which I thought was great. So I had a lot of fun in doing that and seeing how research methods change, you know, and also getting an idea of the difference between B2C and B2B. It's very different. So consumer research is one story. When you go to B2B it's a very different ball game and you need to understand how to do it properly because there are, I think it's just much more difficult to do in general. And if we have time, we can even discuss that now.

Speaker B: I would just love to hear your perspective about data collection because that's our main topic. And your career covered many areas as you mentioned. It's incredible. And I wonder, you have seen data collection happen in all these different areas. How different was data collection from one area to the other? How did this shape the way you think about it?

Speaker A: So if we talk about the difference between market research and user experience research in terms of data collection, you have something in common which are uh, you are talking pretty much to the same users but from different angles and at different stages. So when you are in market research, generally you talk to your users after the product is already being launched. So the product should be out there and you talk to the users or customers, you know, depending on your definition. Generally the difference between user and customers in some companies, like if you call somebody is a user, if the product is free and if they pay their customers. But from a research point of view they are still the people use your product with different level of expectation obviously. So in market research, uh, there is this stage difference. So you start talking to customer after everything's been launched and then you ask them different questions. It's mostly about their experience. Obviously there are lots of questions related to the marketing that you need to do about the product and uh, everything else. While in user experience you generally start talking to users initially as soon as you have some kind of demo or prototype. So you're really much closer to the product, you are closer to the engineers, you're closer to the designers because you're also in the UX thing. While generally from market research are a bit more distant from the designers. Especially in a huge company if the company is more like, it's much easier to make these connections, you know. So in big companies, you know, there's the marketing team and the UX team and everything is different. Also the chain of command is different. There's no overlap, few exceptions. There are some companies that have more like an insight team or something with different languages where the UX and marketing, uh, they are in the same team, which makes a lot of sense because you're talking to the same users. And many times one of the risk of having these two teams separate is that you might duplicate your research without even knowing it because there are some overlaps. So I give an example. The market research team might study Churn and trying to understand why some people stop using our product. Even the user experience team can do the same study because it's still relevant to them. And so sometime I was actually in a situation where I was in UX and someone did marketing was doing a Churn study. I was like, maybe I should know that. Also it's important even if you don't know it to have a uh, consistent and way to report and maybe a uh, common repository. There's also lots of software out there that you can use where every report from marketing and UX should go there so you avoid duplication. Everybody's learning and you don't need to start from scratch. Especially if the company is huge, the company is more or less much easier to know who's doing what. But when the company's big, like Google, there were cases in which uh, there was some duplication unfortunately or in any case overlap I would say. But yeah, the main difference is the stage in which you talk to your customers and the kind of questions. So in user experience you do a lot of also usability studies, uh, you do lots of qualitative interviews. You try to understand how people use your software. If you are in B2B. I had some colleagues who actually went into a company that was in cloud and they just observed how they were like setting up some cloud products just in a very, let's say anthropology kind of style where just to understand, did they look at the official help center or did they just search online how to do something, how did they start using? Did they just did it? Did they talk to somebody? It's fascinating to see this report. So that's another reason why I moved to ux. It was like I really want to be closer to the product. While in marketing you're generally a bit more distant. And so it was great working with designers and making some changes based on your research. Like we found this and so you see the change actually in the product and that's this kind of satisfaction. They say okay, you know, maybe hopefully the product is going to get better because of my research.

Speaker B: This is great because this is also something that I wanted to explore with you. And I think you already mentioned a lot which is when a Lot of people, myself included, think about market research and I think maybe more market research than user experience research is. We think about surveys a lot. Think about web surveys. That's how people are collecting data. But as you mentioned, there is so much more than that. And the data that you need is so much more than the data that you collect from m the users. You need to understand how people are interacting with a specific product, their behavior. So can you just dive into that a little bit more and just describe how do you collect data in this space?

Speaker A: You're right. And there are also industry reports showing that in market research, service is still the used data collection method for the good and the bad. So we know it. And in fact, when I was at Google, I was doing survey, uh, office hours, so people could just book like half an hour with me. And I spent a lot of time basically telling people not to do a survey, which was funny, you know, I was a survey person telling people not to do it. And the reason is because it was not the, the most appropriate or efficient data collection method. So many times it was not what the tool is able to do. Especially if you want really in depth knowledge, granular knowledge in a survey, you're not going to get it. So you should do qualitative studies or if you really want even more detailed, you look at something that, depending on the company, you call them logs or behavioral data or telemetry, depending on the company you work with, which is basically the interactions that are captured by the software that people are using and we can talk about that later. But you can also look at those that will give you the what people do. You don't know the why, but you do the what. So let's say if you have this brand new UI that your designer is very excited about and you see that nobody's clicking on that specific button, you see that from the logs, you don't know why is it out of sight. They don't see there's something that is happening. And so you then when you need to do qualitative and you do maybe a usability study where you see how people move their mouse, how they're interacting with the tool. So I would say in user experience surveys, they're very, very, very popular, but less popular than in market research. But even market research is just too many at best. In market research you might do a bit of focus groups and maybe some interviews, but the interviews, in depth interviews and usability studies are obviously very, very commonly used in user experience because it makes sense and that's Especially when something's very new. You have a prototype, you can show the users that they can click around, they can see how they do it, they can talk out loud and telling them what they are trying to do and they fail. You can probe. So you need to do it live, obviously, and trying to understand what was the friction. And then you can show this video to your designers. And guess what? You know, we can change this, we can improve that. So those are very commonly used method and there are gazillions of methods in user experience, et cetera. As soon as I moved, I bought a couple of books and I remember this one book, they have a hundred methods that you can use. Maybe they're very tiny, like calf salting. Okay, it's one method, but it's still so many. So in a way, when you are in user experience, so you should be like a Swiss knife, you need to know lots of methods. Because going back to your initial question and what my approach is, I need to know your research question. And so it's my job as a researcher to ask you questions about your questions. Because sometimes you just come to me like, I want to know this, but then it's not really what you want. But then when I ask you a few questions, like actually you wanted that and sometimes the answer is already there. So you don't need to do any research. So one research method is, look what we did in the past internally, oh, we already know it, great, so there's no need to do research. Which is the best answer. Immediately you can say, well, okay, now that I know that I want to know more. But then you don't start from zero. You already have this data point or maybe data points, and you say, okay, great, now I know more about the topic. But then you may come back to me in a week and say, okay, now that we have all this knowledge, we want to know on this specific thing because we are planning this new feature or something around that nature, I wanted to know more. So it's part of the difference. So know exactly what you want to measure. And then the second question asking, um, who are we going to talk to? So users is a bit vague. Do you want to talk to a random group of people who are using or do you want to talk to the people start using the product? So they are novices, which obviously they have less experience and they have a different experience and different also set of expectation. Or you're going to talk to the people who are already using it for, let's say a year, for example, because Obviously there's also self selection and if they're using it for a year, the people for some reason didn't see that that was useful for they had already gone. So they churn, you don't see them. Or you can also follow them. So you can see, follow this cohort of people. So let's say you get and you pick cell phone and so you can talk to them during the first two weeks they somebody got the phone, how is this experience? Blah, blah. And then you can talk to them after three months or after a year. And then you can also talk to the people who stop using it and asking them, um, why did you stop using this specific phone? What happened? You know, did you switch to another company or brand? And why? So that's how you follow up depending on differing such questions. And that question can be market research. Definitely. It can also be user experience research, maybe something that didn't work well in the phone and they couldn't do something and so they switched to um, something else.

Speaker B: Yeah. And how do you handle data quality in this context? Because you are gathering data from so many sources in so many ways, quantitative, qualitative, so it sounds like it can be quite complex. The way you control quality in your

Speaker A: data, that's a big topic. The main distinction is if you have your own list of people to talk to. So you have a list of customers or users that you can reach out in different ways. And that's one story. And that's why generally the quality is very high because they're using your product for sure, because you know it based on your internal data. In many companies you can actually reach out to them. There is some kind of opt in mechanism. So you are opted in when you use the product into something called, let's say Malcolm. So it's not everybody, so we need to remember that the sample is a bit skewed in a way, but you have permission to contact them. And so you can reach out, say, hey, we are doing a study on this product, do you want to do an interview with me? Or you can even send them a survey. If you need to do a survey or other methodologies, that's one story. And then the quality for these people is very high because you know they're real, there's no fraudster there, you can talk to them and everything. When you go off your list because you don't have a list or you want to reach potential customers, then that's where everything is gets very complicated very quickly because you say, well, we want to talk to potential Customers, for example. So what do you do? You choose some kind of online generally sample or online panel or like companies that provide users, uh, or respondents or whatever language. That's where everything gets very tricky because unfortunately, as we know, especially with the recent debates, there's a lot of fraud in the survey world. But even in other places where, where people are trying to pretend to be what not they are. And so if it's qualitative, it's a bit easier because you actually, you can see that is the real person and you can do some extra stuff. But when he's completely unmoderated, then it's more difficult to know if that's exactly the person you need to talk to. And so you need to pay a lot of attention. And in the survey world there are companies specialized now in let's say cleaning out Frost. So basically blocking them from entering your study before they actually even start with lots of different, very sophisticated technologies in a way. So the industry is developing, but it's kind of sad that we need to have this industry developing and to clean up your sample instead of just having a good sample to start with, which is obviously a huge topic that the market research industry is debating at the moment. But yeah, that's what I always think of a quality. And the other key concept I was also discussing with my colleagues was when you choose a company to work with, many times you need to rely on a third party company. I saw folks getting very excited about this company because it was cheaper and faster than other company. But my question is, well, maybe there's a reason it gets cheaper and faster. So that's where it gets very complicated very quickly. But if you're a good researcher and you know your topic and you did all the internal desk research, you talked to the expert, you are able to understand it. The study results are completely off or they don't make any sense. So they're completely different from let's say internal data. Because many times you have internal data or you should. And so that's another quality criteria. If a study is like completely comes out with very different outcomes than all the data collected so far, is it a one off? Is it just a, uh, fluke or I don't know, you got there unlucky or is it real? And many times it's not real.

Speaker B: So you can do quality checks at least to know correct.

Speaker A: Exactly. And so the other skill that a uh, researcher should have is not only in the methods, but also you need to know your product very well. So let's call it Substantive knowledge. So you need to read, if you find them, third party research, you know, industry data. So you have the big picture other because your competitors are doing research as you and many times they publish that. So you should read them so you have an idea of your space. And when you have an idea of your space, it's much easier to see if your results are completely not credible or if they are actually, you can rely on them. And that goes back to triangulation. So you have some data points, you did a study with your own customers, you get some number, you did a study with potential customers that cannot be completely different. Or you do a study, you do these competitive studies where you compare scores by your product and the competitors and you have some kind of baseline. So for example, you can say, you can ask questions about quality of the support that you get from these products so you can see how the perception of support is for your product versus your two, three competitors. And trying to work on that.

Speaker B: Something that I would also love to dig in is I know that now you are teaching and maybe you are also working on some projects related to AI assisted surveys. So from your perspective, and this is a hot topic today, everyone is talking about it, everyone is using it. So from your perspective, how do you think AI can improve? We are going to talk about how people are maybe overestimating a few things, but I would like to take a look at the positives first maybe.

Speaker A: So my mental model on AI helping is first we need to think about who's helping. So generally we talk about helping the researcher, but that's just one actor. Let's say in this equation AI can help the respondent if he's in a survey, for example, and I have some examples there, AI can even help the people collecting the data, let's say the interviewer or the supervisor, if that is the case. So this is just to start on a mental model, then looking at the positive. And as you know, everything is changing very quickly. I just finished teaching an eight weeks class on AI assisted survey to the students at University of Magna in the survey program and we went through 150 papers on anything you can imagine about the different stages of a survey project. From even starting doing a literature review, which you should always do, to actually writing the final report. So the whole cycle. And it's very similar in research in general. If it's not a survey, it can be qualitative research, it can be something else. And you have the general or generic tools like the one we all know. And then you have all the custom tools which, it's another huge theme because there are so many that you would never keep up. So there are some places where I think AI and LLM is working is pretty good. You know we always, if you with there is. I call it, I hate when people say human in the loop. No, no, no. Researcher in the loop. The human doesn't know. If you don't know a topic, AI writes in a way that is so confident you can believe anything. But if you are an expert or expert in the loop then you understand the limitation and sometimes you might disregard or you might re prompt and see what's going on here. So with that in mind think about doing a literature review or a uh, desk research how difficult it is and especially when you are new to a project, you'd start by reading lots of reports which you should read anyway. But having like a nice summary, it's something that it's not perfect but it's getting better and better. And there are generic tools like the deep research function in most common LLM tools that you can imagine. They're all called the deeper research. Anyway so From Gemini to ChatGPT to Copilot is all called the deep research. And so you have basically a summary across I don't know, 120 different websites which probably you will not be able to do manually. So that's something. But you can think about the same thing about your internal database. So if you have an internal database of research it gives you like a nice summary. So you start getting the know like let's say you move teams, you go to a new product. Well how do you get up to speed? Are you going to read all the research reports? No, you don't even know what they are. But you know if the company did a good job in archiving, ah them and now with an LLM on top to summarize them, then link them to the original reports. Okay. So you can check, you can learn more. Then you have like an initial starting point which will save you a lot of time to get up to speed. And also you become a substantive knowledge expert which you should be anyway. So that's where I think is a great way to help. Then we have all the tools that are doing I would say a good job in summarizing text, you know, like interviews and everything with different results and they're all very different. So you use three different tools, you get two different results. So that's something we need to deal with that let's just say coding open ended answer is another topic where lots of people are excited and it's getting better and better. You know, there are some good studies out there and then all the code writing code. So you know, if you need to write some statistical analysis code with your preferred coding tool can be Python R or anything. In this case there's a lot of help, you know, and uh, it's like having an assistant there. So that's definitely, I think a lot of people are, they would agree with me on this. Let's say assistant and help. As long as you have in mind that there are always hallucinations. Even now they're not going away anytime soon. An example I did with my students was to. I found a request for proposal online. Very common. You know, you get a request for a proposal which is, was for a survey. Obviously it was a, a survey in different counties was a um, let's say 40 pages RFP. And I asked them try two tools to see how they summarize it. If you are working a survey shop or in a company, in a market research company, you're going to get RFPs. So how do you process them? Do you read it all or you have the LLM doing a quick read and give you a basic info like number one, when is the deadline to answer? Because if you miss the deadline then you're going to get that job. So maybe that's number one, not very important and, and other things. And in many cases, even when I did my own test there were some hallucination. You know, I remember I had an uh, RFP with a scoring system which is very common. You know, we score each company with based on this scoring system. And the sum was different points. The sum was, I don't know, let's say up to 60. So 10 points for this tempo. And then this LLM tool completely changed the scale to a thousand. And when I repromp, obviously the LLM is going to ask for forgiveness, but that was it. So if I wouldn't have read the original report, that probably was going to be a mistake because that's key. And the students were shocked. You know, like what? You know this is 2026 and we were using the latest model. We're not using something so that's something just to keep in mind. I um, mean the mental model is you have a, it's like a uh, research assistant who's very eager to please you. But because he's an assistant and is, let's say young to speak, this entity doesn't know much and so doesn't know how to check for something that in that case, like if the scale is 60, you cannot just convert everything to a thousand. So that's something that just to keep in mind.

Speaker B: So do you think people are overestimating AI a little bit?

Speaker A: Yeah, for sure. Also because, you know, the marketing material is very hyped up and so you just need. But it makes sense, you know, there's a lot of pressure from the investors to get the money back and many companies are not making any money, by the way. So I'm not surprised that there is a bit of hype and I understand it's very exciting. So it's not that it's all negative and you want to be excited, you want to be an early adopter, but you just need to keep in mind that, you know, everything that is seen is always not perfect. Also, we need to deal with something that we are not used to do, which is uncertainty. So the same prompt will give you different answers. And for a researcher's mind is very hard to, you know, we don't do well with uncertainty. You know what. And so because the model is probabilistic and then the tools give you different answers every time. So what I did with my students is to use there is a tool where you can see the same prompt coming out from two different LLMs, left and right. So it's like on a table. Very nice. And you start getting the message that even with the same prompt you had very different answers. And so that's why it's very disconcerting. Like which one should I believe or which one is closer to what I really meant or not which one is. And some outputs are just shorter, some outputs are longer, you know, depending on still with the same problem that that's what we need. We need to be able to deal with uncertainty. And the other thing we need to deal is anxiety because you need to realize that you will never keep up and if you don't accept that, you're going to be stressed forever.

Speaker B: More and more. Yeah, no, but clearly AI is posing new data quality risks. So you now need to pay attention to different things. And certainly it's a tool, it can help in a lot of things, but it can also be dangerous if you don't pay attention and you're not careful to it.

Speaker A: Uh, not careful. And you're not an expert, especially so because it sounds really good, but it's not Only hallucination is also something might be missing, like something very important might be missing. And because you don't know, you don't know what you don't know, it seems okay. So doing that and a bit of more like traditional, um, and also reading, learning more from the originals. So it's good to find the original sources but then read the original sources because that's how you learn and you build your internal knowledge.

Speaker B: So looking ahead and taking into account all this new technology that is rising, how do you think the research teams will operate in the future?

Speaker A: Always hard to talk about the future. Uh, my best guess is well, ideally as I said before, the market research team should be closer to the user experience team from more like an organizational point of view. And if AI can help for example to transfer knowledge, having this repository, uh, for example of research done by different teams and make you aware that somebody else in the other team is doing similar research than you or they did it in the past where that's definitely very useful. I probably guess that companies will have their own internal uh, LLM tool trained on the internal data which makes a lot of sense. So you get more than a general tool which is trained on let's say the Internet, which still we don't know what it is, we just don't know where the data are coming from. That's another problem of LLMs that are difficult to reproduce almost never and also they are non transparent so you don't know which data sources were used to train and which were not, for example. And it's very hard to know. And so if you have an internal tool you kind of know where the data is coming from and also you can always double check because that's so I will envision then there are some folks in some conference saying that probably the difference between qual and quant is going to become more and more blurred with AI because now if you know less on the quant side, A.I. uh, can help for example on that. So I don't have a strong opinion yet, but I can see that coming. I don't know if it's positive or negative, but definitely this very strong distinction I guess is becoming more and more blurred. Also when you see job postings, they're looking for like mixed method. More and more folks who can handle a bit of data and also a bit of qualitative which makes a lot of sense. So probably we will need more, the new research would need to be more equipped in mixed method research than just saying oh, I only do stats or I only do qual. And because actually either you work together with qual, uh, and quant or you just do everything yourself because the quant will never give you all the answers and the qual, same story. So generally is when you add it together that you get the highest amount of insights and knowledge. So that's probably what's going to happen

Speaker B: hopefully and taking that into account and we might have a lot of young researchers listening to our podcast. Do you have any specific advice to them? What should they focus on? What kind of skills would be important for them to have?

Speaker A: Definitely. So we just discussed how everything is very daunting and stressful because there's something new coming out every day and new research and you study, you go on LinkedIn and you can never keep up. So I would focus on one or two topics that you really are passionate so you can follow those other. If you want to follow everything, you might do that as full time job and that was it. Basically. Skills definitely need to learn about how LLM work and it's technical, but you need to read a bit. Prompt engineering is still important. When I read a paper that is coming out, I don't even read the paper. I go to the prompt first and say what was done there? You can say which tool? But then the tool by the time the paper is out is already updated. But at least you have an idea about the prompt and you understand what was done, what kind of input was given to the machine in a way. So a different way of reading research, then you read everything and obviously if the prompt is not there, it's like, well, why I want to know. You know, it needs to be more transparent. So that's something. And then now more than ever there are lots of webinars, so it's much easier to keep up. Webinars, many are free. My general suggestion is join one or two or even more professional associations. They organize lots of webinars and training. So you keep up because the moment you leave your education and you start working, you don't devote your entire time to learn. So you need to keep up. And so that's generally a way to do it and so it's a bit easier. And then that community creating, uh, discussing with your colleagues and exchanging. Because if, let's say you have a group of people and um, everybody's passionate about different topics, then let's say you meet once a week, you do like lunch and learn. And anybody can say, well, you know, I learned about these new things or this new study came out, now I'm going to try to redo it with my own data. So that's another way to share knowledge and avoid the anxiety of keeping up with everything, which is Basically human impossible.

Speaker B: And with that in mind, do you have any resources you want to share with our listeners?

Speaker A: Just one. By the time the podcast is out, it should be out, but in case it's not out, I'll announce it. So I'm going to um, co edit a special issue of journal which is called Survey Practice. So Survey Practice is an online open access, so anybody can read journal from the American association of Public Opinion Research, which is really targeting practitioners. So the articles are not like journal articles in terms of length, they're a bit shorter but very practical. And with my colleague Sarah Ball from LMU Munich, we are editing the special edition AI Assisted Surveys. And we are looking for practical experiments or practical example of how AI was helping the researcher or maybe the expandant or maybe the interviewer or the supervisor in the survey world. So it's open to anybody and private companies and academic is the same. And we have taken advantage of the tool, which is all online, so you can show images, you know, you can show a gift or even a small video. How you, I don't know, let's say you did some uh, coding of open ended answers or you developed a tool to convert a questionnaire from Google Doc to this specific survey tool. That's just a couple of examples. So anybody can do them and then once it's out there they can talk to you. So create a bit of a discussion. So that call for papers is going to come out soon and hopefully by the end of the year. We are not waiting until the issue is full. As soon as a, uh, paper is ready, we are going to put it out there so you can start reading it right away. So that's, we wanted to make it very, very practical and also for students, like if you're teaching any kind of AI Assist or something, which exercises do you give to your students and how did they do it? What did they learn? That's another topic that we would love to hear from because a lot of people are now tasked by teaching students how to use AI. Well, if you need to teach, obviously you have the theory, you read research, but then you'll only learn, especially with AI, if you start prompting and see what happens. So which kind of exercises you give to your students and how do they do them? Are they excited, you know, what works, what doesn't work? That's something that I would love to get for the special issue.

Speaker B: Wonderful. Thank you so much, Mario, it was a pleasure having you here.

Speaker A: Thank you. Yes, it was a great discussion.

Speaker B: That wraps up another episode today. Mario and I discussed market research versus user research, the structural differences between the two fields, and how to avoid duplicate research silos inside large organizations. We also discussed ensuring data quality, why researchers must triangulate results with internal baseline data and maintain deep substantive knowledge of their products, and finally, the reality of AI in research, treating LLMs as eager assistance while staying vigilant against hallucinations, missing data, and model uncertainty. Check out the show notes for a, uh, link to Mario's upcoming special issue on AI Assisted Surveys. And don't forget to subscribe. Thanks for listening to Survey and Beyond, the Data Collection podcast by Survey cto. If you want to learn more about how Survey CTO helps organizations collect reliable, secure and scalable data anywhere in the world, visit www.surveyct. and if you are a fan of Survey and Beyond, consider leaving us a rating or review on your favorite podcast app. Your feedback helps more listeners discover these conversations and stay connected to the latest thinking in data collection. Don't forget to follow us on Apple Podcasts, Spotify or wherever you get your podcasts so you never miss an episode. On behalf of the entire Survey CTO team, thanks again for joining us and we will see you next time.