#188 - Why AI Can't Replace Qualitative Research with Sam Ladner of Workday

Speaker A: Back when I started doing qualitative research, I had to transcribe everything by hand, which is insane. Now people take it for granted. I had professors who were telling me, that's a really good thing. You'll learn the data really clearly and deeply, which was true. You know, it actually does learn the data really deeply and quickly. Uh, however, it's not really a good use of my time to transcribe by hand. And so it was very welcome when we started getting really good transcriptional functionality in off the shelf tools. And so the ability to have very boring coding done by the machines. I'm all for that. I think it's fantastic. It's a labor intensive, not very good use of your time research activity that you really do can offload to the machine. The problem is, of course, is once you start doing that, it's very tempting to offload sense making to the machine, which is the problem. You can't offload cognitive sense making to machines.

Speaker B: Hey, this is Erin May, and this

Speaker C: is Carol Guest, and this is Awkward Silences.

Speaker B: Awkward Silences is brought to you by User Interviews, the fastest way to recruit targeted high quality participants for any kind of research. We're very excited to have Sam Ladner here today as our guest. Sam has written two books and is working on a third, so has written Practical A Guide to Doing Ethnography in the Private Sector and Mixed Methods, A short guide to Applied mixed methods research. Forthcoming is the tentatively titled Practical Foresight, Strategic Foresight and Applied Settings. Uh-huh. Big juicy topics. And I am excited to dig into another big, juicy topic that you joked. You're like, I guess we got to talk about AI, which we indeed do need to do. But qualitative research in AI, which is, uh, a big topic and we're going to not cover all of it, but we're going to cover some of it. And I'm sure it'll be a great conversation. So thanks for joining.

Speaker A: It's my pleasure to be here.

Speaker B: How is AI changing how qualitative research can be done?

Speaker A: Well, it, it is giving us some labor saving, very welcome labor saving moments. Um, you know, back when I started doing qualitative research, I had to transcribe everything by hand, which is insane. Now people take it for granted. And I even had, I had professors who were telling me that's a really good thing. You'll learn the data really clearly and deeply, which was true. You know, it actually does learn the data really deeply and quickly. Uh, however, it's not really a good use of my time to Transcribe by hand. And so it was very welcome when we started getting really good transcriptional functionality in off the shelf tools. And so the ability to have very boring coding done by the machines, I'm all for that. I think it's fantastic. It's a labor intensive, not very good use of your time research activity that you really do can offload to the machine. The problem is, of course, is once you start doing that, it's very tempting to offload sense making to the machine, which is the problem. You can't offload cognitive sense making to machines. Machines don't think and they certainly don't make intuitive leaps and they don't, um, have abductive logic. They're basically big correlation machines, which is really what the core of the issue is. Qualitative research is not correlation hunting. That's not what it is and it never has been. It's more, it's closer to unriddling than it is to predicting what is going on here. What's the layers, what are the human nuances, what are the complexity, what's the entire picture? All of those things are what qualitative research is supposed to be for. And the fact that, I mean, nobody I think really argues that machines are better at that than humans are. I don't think anybody argues that machines are a lot better at math. Sure, math is very predictable, it's very understandable. But if you think about qualitative research as math or even just transcription, I mean, you know, obviously you're not really important as the researcher, so of course the machine should take over.

Speaker B: Yeah. What'd you call it?

Speaker C: Riddling.

Speaker B: Unriddling, I love that.

Speaker A: Unriddling. Unriddling.

Speaker B: Unriddling. Yeah, that's a good one. I like that a lot. Right. Because it's not going to find that correlation, that causation, like you mentioned. But maybe, uh, as we often see when we mix qual and quant together.

Speaker A: Right.

Speaker B: This could be an area to dig deeper into. Right. Or there's some signal here of where we want to continue to explore.

Speaker A: It could definitely do that, I think. I haven't actually experimented with that kind of mixed method, hinting that, you know, the machine might be able to do for me, know, with a big corpus of transcripts, it might be possible. It's kind of interesting as a, as a question, like is there a correlational opportunity here, uh, that I can't see as the human? I haven't tried that yet. I haven't even seen people try that yet. But what I have seen a lot of people do is I wasn't a very good unriddler, so I was a transcriptionist or a summary writer and I wasn't very good at that. And it's very labor intensive. So now I'm going to get the machine to do that for me.

Speaker B: Yeah.

Speaker A: Um, and you probably weren't very good at the task in the first place.

Speaker B: Yeah. It wasn't to oversimplify, to put, you know, because we need to put obviously everything into a two by two. Right. AI is going to be able to and should do what it can do and what maybe you don't like to do or don't find meaning in or don't want to do over and over again. Right. And, and so I think that I enjoy it or I don't enjoy it part is going to be a little bit different for everyone and it's going to change over their careers. Right? Mhm. How much is what AI can do going to change and how much has it changed? Right. Because as you mentioned, AI is good at math. That's sort of an unchanging idea. And we have all seen AI use cases increase over time. But fundamentally there are things that we believe AI Will never be great at. Right. And so do you have uh, a shorthand for thinking about that when it

Speaker A: comes to qualitative research, what it will never be good at?

Speaker B: Yeah. Yeah.

Speaker A: Well, AI can't think or feel.

Speaker B: Yeah.

Speaker A: So if you extend that a little bit and say, okay, what kinds of thinking and feeling coming out of qualitative research really gives, you know, the power of qual research? I mean really feeling is a big part of it. Right. AI M cannot feel partly because it can't think, but also because it doesn't have a body. If I'm going to be metaphysical, it doesn't even have a soul. Right. So some of the most affective, qualitative, uh, research that I've done in the past, and anybody listening to this will know this too. There's a moment where somebody says something to you that is just like, uh, my gosh, that's really. It hits me in my soul, in my heart. Oh boy. That is a really emotional moment. Those are the kinds of instances that the AI Will never be able to discern. Even if you're literally having a participant cry, AI Will not be able to interpret what that really, really means. So what you should probably pay attention to are the things that are the most human moments and those are the moments that the, the human researcher should be taking the wheel and be very, very present. And extremely advocating on behalf of the participants in those moments. So whether it's going to be in the actual interview itself or maybe you're on site and you're doing ah, an ethnographic observation, maybe it's going to be an online interview, a focus group, there's uh, a, maybe there's a conflict in the focus group and people are, you know, disagreeing. These are moments that the AI can never understand. And when you're reporting those moments, it's incredibly important to do it with, you know, your own human, um, insight. You need, you need your per, your participants emotional experiences to be adequately and responsibly represented. And when you outsource that to AI, first of all it doesn't understand. And secondly it's, it doesn't care. It AI is psychopathic. It really is like literally psychopathic. Has absolutely no interest whatsoever in serving human beings. None. Y Y Sycophantic.

Speaker B: Ch.

Speaker A: Sure. So it's going to tell you stuff you want to hear. These are something that humans won't do.

Speaker B: Yeah. Well, I love what you're saying because it's something that I've thought about a lot doing this over the years. Right. Is the human part of qualitative research is very important and that's only coming, I think, more to the forefront as AI is more of a part of it. And I love how you're describing the role of the human in that is like the insight comes from thinking and feeling.

Speaker A: Yes.

Speaker B: Like. Yes. From the rigor around methods and knowing how to actually practically do good research, but from that human feeling that some of which comes from experience and from building. Right. That uh, that instinct over time. And also you're, I think, making a distinction too between AI and an analysis context, you know, can do some sentiment analysis or things like that. So there's, there's something there, but that is different than thinking and feeling.

Speaker A: And I'm not really a fan of that. Yeah. Sentiment analysis.

Speaker B: Okay. Yeah, go on.

Speaker A: Yeah, I'm, I'm not a fan of sentiment analysis. Yeah. You know, the first time I think I kind of came into contact with sentiment analysis was probably about 10 years ago.

Speaker B: Uh-huh.

Speaker A: Give or take. I'm trying to remember exactly the same moment, but I remember thinking what this is. You just went through this text and you looked at keywords and told me that these are the keywords that represent the emotional experience. And I thought that's like, you know, poetry being reduced to a bunch of words. Like that's not what it is. Right.

Speaker B: Mhm.

Speaker A: So Sentiment analysis is. It gets to the core of really what the contradiction here is. Sentiment analysis is correlation. Uh, it's mathematical, it is mechanical, it is not actual human emotion. So when you are like so excited, like now, I can do this sentiment analysis at huge scale. Well, guess what? You've already defeated the purpose. Yeah. Emotional experiences cannot be scaled. That doesn't even make any sense. Less like saying the Pope is weak on crime. What? That doesn't even make any sense. Right? Like, what are you even talking about? Sentiment analysis is not emotional experience. It's not conveying participant experiences. It is not a human translation. It doesn't make any sense. So when people say, oh, I want to do sentiment analysis, really what they're saying is, I want to know when people are upset. But I don't really care enough about

Speaker B: it to find out how I hear what you mean. So let me ask a related question, right? So totally understood on the mathematical nature of sentiment analysis and of asking AI to do it at scale. Compare that to a person, a human being, tagging what they're hearing in interviews. Uh, not necessarily at scale. Maybe there's 10 interviews or whatever it is, or. But trying to find patterns and make sense, Right? And make it mathematical, if you will. In a way, is. Is that okay? In a way that.

Speaker A: But they're not doing it mathematically. If they're doing it right, they're not doing it mathematically. You know, qualitative coding is not about frequency per se, although frequency is one of the elements of it. It's about understanding the holistic nature of the problem at hand. So let's take for example, let's say we've got 10 participants and, you know, we're asking them about their emotional experiences. Their experiences, period. And there's emotions about using a product, right? And there's one person who has a very, very negative experience. In quantitative research, you're like, that's an outlier. Take them out.

Speaker B: Mhm.

Speaker A: Qual is not like that. You have to explain all the outliers. So it is not a frequency issue. You don't get the out of using frequency to say, hey, they don't matter. Uh, it's not important. It's small, right? It's not small. You need to explain the entire picture of why is there one person who really, really gets annoyed using this product? What is it about this pattern that you know, doesn't exist in other people? Well, you need to explain everything. Your explanation as to how this entire corpus, um, of data came to be must explain the outlier. And if you can't do that, you're not doing quality research.

Speaker B: Yeah, yeah. All right, well, let's talk about where AI can fit in in a positive way in qualitative research. Because I know there are a lot of ways that it's, you know, you don't recommend and we'll talk about those as well. But where have you seen and where are you seeing that change, you know, over the years as AI enters workflows more?

Speaker A: Well, yeah, like I said, I mean, the first big one, I think for me personally was the transcription. That was huge. And then being able to reliably do closed coding. So and when I say closed coding, what I mean is, um, you know, a predetermined set of codes. Key tags. Right, Keyword tags. I tell the AI this is my coding system. And you know, I've had, I've done this before many times. Like I've had a coding system that's a pre existing coding system. And I as a human have gone in and coded based on that. And now I can get the AI to do that pretty predictably because a predetermined coding system is usually pretty straightforward. Right, right. I want people to tell me, like, I want you to code when people are finding difficulties, or I want you to code when I start asking them about workarounds. I want you to, you know, summarize for me those workarounds. That is a real innovation. That's a, that's, that's really freed a lot of people up to do this. What's the difficulty and has always been the difficulty is that a lot of qualitative researchers don't graduate past that themselves as humans. Right. Like, they haven't graduated past what I call it the laundry list problem. Like, this happened and then this happened and then this happened and that's the end. And you're like, yeah, well, so what?

Speaker B: So what exactly?

Speaker A: You know, like, what does that mean? You know, so it's done a really good job of laundry list type coding. And it's freed me up to go. You know, I've got these 10 participants and there was just that one weirdo that just really, really was frustrated. And I didn't expect them to be frustrated in this way. And it's kind of weird and I need to understand a little bit more about it. I can go and I can spend more time unriddling that riddle.

Speaker B: Yeah.

Speaker A: And I can do it in lots of. Maybe I can go back to the literature and I can say, you know, you know, what are kind of the psychological Profiles that I'm looking at here? Or what are the uh, sociological pressures that this person has, Maybe somebody else doesn't have. Maybe I'll go back to the literature and see if there's a techno socio, technical explanation for what this is going, what's going on here. It frees me up to do that work, which normally I would have to rely frankly on my crystallized intelligence to do that. And I often didn't get to do that kind of stuff because I was busy coding.

Speaker B: Right, right, mhm.

Speaker A: So now that I'm not as busy coding, I can do that work. I've seen a lot of people start doing that now because they've had, you know, feedback from, you know, their stakeholders saying, yeah, well you know, what's the difference between you and AI? And they're like, oh God, I guess I better come up with some kind of difference. Right? Well guess what, you know, be careful what you wish for because now it's here.

Speaker B: Yeah, yeah. I'd ah, love to hear more about the coding and transcription in terms of um, just for folks listening who are maybe at different levels of using this or not using this. I know you had a coding system for years that you've kind of honed in and works well for, for you. Let's suppose someone didn't have that. Should they just, you know, practice for years without it or any tips to kind of get.

Speaker A: Well, it depends. Right? It depends. Like I've had many different closed coding systems that I have developed like that. I know I'm going to be doing the same kind of study over and over again. So a classic one you're going to see for most UX researchers is going to be the user journey. Right. There's an easy one. This is where they describe how they started using this tool or this product or whatever. So the user journey is a pretty good one. Uh, the UX honeycomb is also a really normal one. You know, you're going to see, oh, these are all the things where they've, you know, shown that we're not providing a good user experience because we haven't got accessibility. For example, it's one of the elements of the UX honeycomb. Um, these are very standard kind of things. Right. If people haven't been using those familiar, they're not familiar with those things, they should probably take a look at them because those are easily repeatable. You know, the kinds of coding systems that are going to be more advanced are going to be um, more emergent. And I don't want to say Grounded theory, because really that's overstating the rigor. But you know, many people may be familiar with grounded theory, meaning like the data. Tell you what the theoretical explanation for what's going on is. Right. Well, you're going to be looking inductively into the data a little bit more. So you're going to generate maybe a very specific study, specific, you know, new emergent coding system. Uh, you're like, oh, now I can make sense of that one rando that just really didn't like this. Oh, you know what? Now I understand. They had a very specific way that they like to integrate new technology into their workflow. I'm just making this up. Nobody else had that. And that specific way came from, I don't know, you know, they got once bitten, twice shy kind of experience where they lost a bunch of data once upon a time. So they're very, very, you know, gun shy of adopting a new product into their workflow. That makes sense. So, you know, what if this person had it? I'm going to guess there's other people that have the same thing and I might even call it something. I might even make it up. I mean, I'm unriddling. Why does this person, everybody else loves it. They don't. Why? Ah, uh, it's because they have kind of a trauma induced data fear. I'm going to make that up.

Speaker B: Sure.

Speaker A: These are the kinds of unriddling that you should be doing on a regular basis. You know, if you're doing even the same study over and over again, there should probably be variations in what you're hearing and you probably want to work on. Like, why is this happening on the edges? What are those margins?

Speaker B: There's other.

Speaker A: And it's also really good for you to think about. Is this person kind of like a, a customer from the future or a customer from the past? Right, right. Like, oh, they're, they're evoking stuff that we heard years ago or they're evoking stuff that I've never heard before. And there's something else going on with this person. Maybe this study isn't enough for me to unriddle it. Maybe I'm going to have to keep a, put a pin in that and come back to it later in another study.

Speaker B: Yeah, just the theme I'm hearing from you that I'm loving is this emphasis on the rando, the outlier. Um, and you know. Yeah. And um, I'm just wondering. It's a little bit of a side tangent, but, um, I get a sense that Maybe, um, you have a passion for the, these allyers. Is that where you find the good stuff? Or is it. Or are you merely saying pay attention to that and maybe it doesn't. Isn't the key to unlocking the whole thing after all, but that there's something there worth paying attention to?

Speaker A: I think the reason I bring it up is because it was really something that taught me a lot. Um, when I was in graduate school, I remember having a qualitative methods professor who was telling me, and I didn't understand this at the time because I didn't, I wasn't advanced enough. I didn't quite get it. I was like, oh, that person's just weird. Like, I don't, I'll just take them out. And uh, I didn't even have a, like I had took undergraduate statistics, you know, So I thought, I, I don't even know if I called them an outlier. But I think I just like implicitly thought that they don't matter. And she's like, no, no, no, no, that's not what call research is. You don't get that. You don't get to do that. You have to explain them. And I was like, what? Why? Oh. And I didn't know how. And it was very confounding to me and it was very concerning. And I thought my whole like, logic of how I did this work just completely got upended. Like, why am I going to explain one person? What is the point of that? And that threshold that I walked through was, it's not about frequencies, uh, it's not about math. It's about holistic understanding. And if I, if my interpretation of what all of these data mean, all of these people, what all they said, if my interpretation does not explain the outlier, then I haven't done my job.

Speaker B: Yeah, yeah, love that. Uh, okay, so back to our closed coding. So whether folks are borrowing, you know, from a standard system that exists or developing their own over time as you have, you have it. How did you get it into the AI brain? What are their tools? You're using dedicated?

Speaker A: Oh, yeah, no, it's pretty easy. Well, I mean, I've got my preferences for the tool that I use. I, I use, uh, Max QDA is my, my favorite tool. A lot of people be familiar with that kind of tool if they, uh, like Atlas Ti or Nvivo. I haven't used Atlas Ti or Nvivo recently, so I don't know kind of what their AI tooling is. But I know in, uh, MaxQDA, the AI tooling that they have allows you to do coding based on a very specific description. Like, so you'll make a code and you'll say, or you'll, you'll tell the, the system auto code, these things. And this is what to look for.

Speaker B: Mhm.

Speaker A: You know, there'll be keywords. It's actually a really good exercise too because you, you realize, I don't think I know what I mean when I say that. Like maybe I should explain it a little bit. Maybe I get a clarity. And so if you give it a little bit of clarity, it's actually pretty good for very basic stuff like, I mean, UX honeycomb stuff like usability problems. You can describe that very easily. You can describe accessibility problems very, very easily. I mean it understands quite quickly. So giving it that. I was used to doing this anyway for my own coding because I would always write a code memo like, what is this code supposed to mean? Because sometimes you forget, right? You don't even know. And if you collaborate with other people, you need to write a code memo. You need to tell them like, this is what I coded. So you just treat the, the AI as if it were a collaborator. That's kind of not very smart. Right. And needs a little bit of extra, you know, clarity. And if you find that it misses the mark on a few of them, like it'll give you an output and say, okay, I'm going to code these ones and you can actually tell it. Oh no, no, no, no, that's a, oh, that's a synonym. No, no, no, that's not what I meant. You can, you know, not code. You know, I've got 43 instances of whatever I told it to do. Two of them are totally wrong because I can see a preview. So I'll tell it, don't do that and don't do that. And then it'll code it for me and it'll be so much easier. So then I can do something which I really like to do is not the coding part, it's the, the chat with the coded segments.

Speaker B: Mhm.

Speaker A: Like, I think this explains what's going on. Is that a fair assessment? And it will, you know, take with a grain of salt, uh, what it says, uh, because it's not as smart as I am, but it will, it will help me understand if I'm kind of reaching the right level of investigatory, you know, power. And I don't have to like quality check it because it's a closed system. It doesn't have hallucinations. So it co, like this is the thing that makes me crazy about all the so many of these tools is that like, why would I buy a car that doesn't have brakes sometimes? And I. Oh, make sure you make sure that it's breaking. Like what, what? No, you have to like this is a, uh, must have feature. No hallucination. So what MaxQDA does, which I like, is it only, it only works on the data that you feed it.

Speaker B: Yeah.

Speaker A: So it's not going to make up stuff. And uh, that is great. So the coded segments it comes back with that literally happened in the transcript. It's right there like, and it cites it, it, you can go to it. It's a hot link. You can go right to it. Some of these tools don't do that. I've actually, yeah, I see so many of these people posting stuff on these very complex workflows that they've engineered themselves through cloud code or, or ChatGPT and Python and they put them together and it's like this agent based thing and then they, they find out that there's like tons of hallucinations and I think, uh, why did you, why don't you just buy a completed off the shelf tool?

Speaker B: Yeah, it is very funny. I've had that same question with a lot of cloud code workflows and marketing or other spaces where it's just the wild west, right?

Speaker A: It is.

Speaker B: And uh, people are trying to be relevant and use these tools and there are some, I think, good uses for them. But yeah, it's like, guys, there's a SaaS tool that literally already does this.

Speaker A: It already does this.

Speaker B: Yeah, yeah. So it's funny. Awkward interruption. This episode of awkward silences, like every episode of awkward silences is bright to you by user interviews.

Speaker C: We know that finding participants for research is hard. User interviews is the fastest way to recruit targeted high quality participants for any kind of research.

Speaker A: We're not a testing platform.

Speaker C: Um, instead we're fully focused on making sure you can get just in time insights for your product development, business strategy, marketing and more.

Speaker B: Go to userinterviews.com awkward to get your first three participants free. Okay, so you're using the transcription, uh, closed coding AI. This is helping you out, freeing time. That's the AI dream, right? Any other use cases where AI has truly been helpful for you in your work?

Speaker A: Uh, well, this is going to sound hilarious, but one way I use AI a lot is to analyze my dreams.

Speaker B: Okay, great.

Speaker A: That is not a workflow. It is not something that's about productivity. But I have access To. To kind of a base level Jungian analysis. I've done Jungian analysis and usually the. The skill of the. They call it the analysis, the analyzer and the alisand. Whatever, it doesn't matter. The Jungian anal analyst is really well familiar with worldwide folklore and symbolism, whereas ChatGPT only knows what's happening in. On the web. Like, it's just a fraction of what's actually known about folklore and symbolism. So it gives me something to talk about, you know, something to think about. I like to record my dreams, and I like to think about them. And it actually is reaffirming to me when I start looking at it and I realize that it doesn't know symbolism, which means it still can't replace me. It never will.

Speaker B: Uh, you record your dreams, meaning you wake up in the morning and, um, a voice memo, like your memory.

Speaker A: I do a voice recognition into notion.

Speaker B: Uh-huh.

Speaker A: And I don't even know half the time what I'm saying. And then I go back and I read it sometimes, and I was like, what? What's that?

Speaker B: Wait, okay. And so Chat GPT doesn't know what's up with all the folklore. So how are you AI ing it?

Speaker A: Well, I'll take the text that I generated from, like, my voice recognition and I'll put it into. Sometimes I'll try cloud, Sometimes I'll try ChatGPT and I'll say, analyze this using Jungian analysis techniques.

Speaker B: Huh?

Speaker A: Right. And normally it will give me something like, oh, driving symbolizes power and autonomy. And m. I was like, yeah, that's true. But it won't. It won't say anything about. It's curious that you had a red Volkswagen, you know, because red Volkswagens in the 1930s were like, blah, blah, blah, and they were used in this advertising campaign. Like, they don't do anything like that. Nothing like that, you know?

Speaker B: Yeah. Wait, so why is it useful to you if it doesn't do anything?

Speaker A: It just gives me a first pass.

Speaker B: It gives you a first pass.

Speaker A: If you think about what your. Your dreams really are, they're like your subconscious already knows what it's trying to. It's trying to talk to you. It's trying to work some stuff out. It needs a little push, and it can't talk to you in English. It can't tell you in literal terms what is up with you.

Speaker B: Right.

Speaker A: But it can communicate in ways that are symbolic. And so you have to, like, weed through the symbols. And sometimes you're like, oh, my God, that's totally true. Like I had this, the symbol of honey in a dream. Turns out that's a very known symbol. ChatGPT did actually call that out for me and I was like, that's interesting. I hadn't thought about that.

Speaker B: What is honey?

Speaker A: Honey is like a mana, uh, from heaven.

Speaker B: Oh yeah. I was gonna say it seems like it'd be a good thing. Not like honey falling out or anything like that.

Speaker A: Yeah, that's a bad one. That's a bad one. Yeah. Honey is like, uh, the sweetness of life.

Speaker B: Okay.

Speaker A: The joy of life.

Speaker B: I'm going to try that out with my whisper flow. Um, chatgpt.

Speaker A: That'll be fun.

Speaker B: All right, so that's AI good use. We've talked a bit about how not to use AI. Uh, but anything else you want to add there? I'm doing qualitative research. Uh, my boss is pushing me to use A.I. you know, CEO mandate, got to use A.I. we've got fewer resources, big tech budget, whatever it might be. Mhm. How do I not go too far astray?

Speaker A: Well, there are times when you're going to have studies that are low stakes. Maybe M. They're also not enough time to be designed properly. I wouldn't lose sleep over using AI for those at all. Uh, you probably already know as a researcher, some of the studies that you have done that were like, not really important, you know, and you even wondered if you should even spend the time doing them because you kind of already knew what you were going to find. Those are perfect use cases.

Speaker B: Is that typically evaluate evaluative sometimes?

Speaker A: Yeah, sometimes. But they're going to be even things like, uh, uh, you know, even, even generative type research where you're like, I don't think this is going to be a really compelling idea. I don't think there's anything here. Yeah, I've been embedded with this user base for a really long time. Okay, people want to know. Let's go find out. Yeah, I wouldn't lose sleep over using AI for that.

Speaker B: So the research question isn't make or break for the business play with AI. See what you get from it.

Speaker A: See what you can get. See what you can get. But you know, the only way to really have, like to know that you haven't gone too far is to also know what you consider to be your little, you know, most important thing. Um, I read a book a while ago by a CIA analyst who uh, talked about being a rebel at work. That's the name of the book, Rebel Rebels at Work. And she talked about going into these meetings and she had like a. Like a hobby horse, like a pet project that she always wanted to bring up. And one of her colleagues called it her little red wagon. And she dragged her little red wagon into the meetings and they're like, oh, what's in the wagon's here? What are we going to talk about this time? It's her. It's her little pet project, her little red wagon. So you should probably have your own little red wagon. And. And it's going to be something that, you know is important. Uh, you think is got legs somehow. Like it's. It's like a. It's a brewing problem or it's a brewing opportunity or something that you know is important. And nobody has quite figured it out like you have yet. Those are the ones that I would not let AI take over for me. I would do AI to enable me maybe to do more research on that topic, for example, like, maybe I'll be able to get it to code a bunch of past research about that topic now that I understand more about it.

Speaker B: And I'm gonna know what's BS and what's not because I.

Speaker A: This is my passion topic.

Speaker B: Yeah.

Speaker A: Yeah. This is my little red wagon. What's your little red wagon? If you don't have one, maybe you should find one.

Speaker B: Yeah, I agree. Makes work more fun. Aaron's talking about whatever again.

Speaker A: Exactly. I just read a really great memoir by, uh, Catalyn Carico. She is one of the co. Inventors of the MRNA vaccine.

Speaker B: Hmm.

Speaker A: And her little red wagon was MRNA for many years. And they literally called her the MRNA M. Lady. Oh, here she is again. The MRNA Lady.

Speaker B: Oh, my God. Many worse monikers a person could have.

Speaker A: Well, now they call her, you know, Nobel Prize laureate.

Speaker B: Yes, exactly. Exactly. Uh, I love that. That's a great story. Okay, so we're going to use AI for not important studies. Great. When are we not? So I imagine you have some strong opinions on synthetic users and. Or AI moderation. Yes.

Speaker A: AI moderation. Okay. Yeah. I don't know why you would do that. Why don't you just do a survey? I guess maybe it's faster than, uh. So here's. Here's something.

Speaker B: Yep.

Speaker A: There's a reason why you might use AI moderation. Maybe you would normally use it as a cert. Do. As a survey. Right, right. Maybe you. But you would require users to. You spend a lot of time and effort and burden the users, the. The respondents with giving the information. Maybe an AI interviewer might make that easier for them. And it's a very simple topic, maybe. I guess the thing about AI moderation, though, is that you have to accept that what you're really doing is telling human beings you don't give a shit about this.

Speaker B: Right. Or maybe like this much of a maybe m. No, like marginally more than a old school survey or. No. You think?

Speaker A: No, no, I think, uh, you like old school surveys also telling people you don't really give a. About them. Like when, when you've taken a survey and you're like, what the hell? It's like as if you know nothing about the topic at all. And you're, you're. The potential answers that you're having me choose have nothing to do with my experience. And then you're like, if it's other, give me this 20 page. You know, like what? Like, you obviously don't care about me. This is extractive. Yeah, I don't want that.

Speaker B: Well, one of our earliest episodes was with Erica hall, and it was why surveys almost always suck. And I, you know, almost always, you know, I think 75% of it is that surveys suck and 25% of it is survey design, which almost always sucks. Right?

Speaker A: Yes, I agree with Erica on that. Honestly, they're totally abused. Have been from the beginning. Right. Um, there's good reasons to have surveys. Absolutely. But it actually, a lot of people don't realize that the work that it takes to design a good instrument is far more than most people think. So AI moderation is going to be the same thing. You're going to have an AI moderated what could have been a survey, and you're like, oh, okay, well, I'll make it easier for the participant. They'll just talk out loud and that'll be that. And we'll transcribe it automatically. You still haven't done the work. And when I say the work, I mean caring. You still don't care enough.

Speaker B: Right. And if it, uh. And if you care and if the thing matters and if you have the skill, you could argue in many cases, you might as well put that time and skill and effort into a different method. Yeah, most of the time.

Speaker A: Absolutely. And you know, like, let's let us look ourselves in the face if we don't have the courage to admit that we don't care enough to put a real human in front of another human.

Speaker B: Yeah.

Speaker A: Like, what are we even doing? Right.

Speaker B: Good question. You've written about how companies often privilege scientific or quantitative truth. How does AI amplify or challenge that bias? We've talked a bit about scale and you know, Sort of, you know, maybe to your point about, well, this research question doesn't really matter. I'm going to appease my stakeholders with some um, AI M scale and that'll be that.

Speaker A: Yeah, that makes it sound very cynical. Yeah, it's not totally an incorrect inference, but you know, you played the long game as a researcher so sometimes, you know, you can get your stakeholders, they're going to show up with complaints, Right. And the complaints will often come as quantitatively minded, quantitatively framed complaints. But in reality they're not quantitative at all. They just don't have the sophistication to tell you that they don't understand the nature of the thing they're talking about. They, they use off the shelf language, they talk about sample size, they talk about numbers, they talked about prediction. But really what they're saying is I don't know what's going on here and it makes me really anxious and nervous. Right. So using quantitative scientific techniques for instances that are not quantitative or scientifically problematic is really appeasing people. And sometimes it's worth appeasing them to some degree for a long game. You know, if you're using AI to do truly qualitative work, you're not going to solve the problem like ever. Really. No amount of scale will ever solve that.

Speaker B: Yeah. So what I'm hearing so far is AI has surgical targeted uses in qualitative research, but is in no way a substitute holistically for qualitative research.

Speaker A: How could it be logically? Because qualitative research is not about prediction, it's not about numbers at all. Probability theory is underpinning all of AI. How the hell would that have anything to do with qualitative research?

Speaker B: I want to talk about the future. You've written about, thought about, think about lot about the future and how AI can or cannot help organizations to think about their future as a business, uh, about better futures for the world, for their business.

Speaker A: You know, I'm writing every morning for my book and every morning I find myself getting stuck in different places. You know, some days are better than others, but the places that I often get stuck is when people are uh, like I'm trying to explain to the reader how to deal with like a future focus problem that doesn't get the future focus it needs. Right. So people are constrained temporally, they're very short term and they, they don't think long term. These are very normal things for human beings, um, especially in this time and place, this temporal backdrop, this time scape that we have. When I think about What AI can do to help people think about the future. I think it can't do anything because that's not the problem. Right, the problem. In fact, it might even make it worse because people become accustomed to immediacy, and immediacy doesn't actually improve your ability to think about the future. In fact, it makes it worse. People who are much better thinking about the future or people who are thinking about the afterlife, you know, people who have spent their whole life thinking about heaven are way better thinking about the future than people of today. The idea that we somehow don't have enough knowledge, uh, the problem is we don't have enough knowledge. So that's why we don't get ready for the future. That's why we get caught, like, you know, unawares. The evidence does not support that conclusion. We have more knowledge now than we've ever had in the history of humankind. And yet people, companies are blundering over and over and over again into the same problems. What is happening with Allbirds Shoes?

Speaker B: I see.

Speaker A: This was a very predictable thing. They had a winning product, they tried to expand, they made a bunch of products that nobody cared about, and now they're becoming an AI infrastructure company. What?

Speaker B: I know, it's like an Onion article, but, you know, maybe they'll be the ones laughing.

Speaker A: Yeah, not the Onion.

Speaker B: Ah, not the Onion. Yeah.

Speaker A: Uh, crazy, right? So now you're saying to yourself, okay, why? What would AI help us do? What AI should be helping us do is delineate the boundaries of known knowledge and unknown. Uh, knowledge should be making us understand regularly that we are discounting the future through, uh, you know, showing us regularly how little we're paying attention to the future, how often we are thinking about the immediate present or maybe the near future. AI should be reminding us on a regular basis of these kinds of things because it's vigilant. It's way more vigilant than humans are. And it's not constrained by social norms. So it can, like, break in and say, hey, you guys, you're not doing the future focused work. It can do that. It's probably not going to because it's going to be designed to do things that are to suit the powers that be, which is make it cheaper.

Speaker B: Use more AI.

Speaker A: Use more AI. Yeah, yeah, use more AI.

Speaker B: So AI is no good at, um, helping us dream up the future. Who is? Or what is is it? Back to the emotion and the feeling and the imagination, these human traits?

Speaker A: Yeah, well, I'm glad you used the word imagination, because imagination is the core of what future foresight is about.

Speaker B: Yeah.

Speaker A: The ability to envision is fundamentally what it takes to be human. So you cannot say that a non human actor can imagine because it cannot. AI cannot imagine. It can only use what's known. It's a prediction machine. It uses past information. It can't come up with something entirely new. Clearly, AI is a tool that can help us, but if you want to think about, um, who or what can make us think about the future better, it's going to be a deep sense of, uh, empathy. Um, it's going to be fiction and creativity. Um, these kinds of envisioning tools, they're literally, there's a place in the brain the default mode network, which is like your shower thoughts. You know, you're doing the dishes and something comes to you. That's your default mode network. You're mind wandering. What actually happens with human brains is when they start getting demented, they no longer imagine they can't mind water anymore. This tells you exactly like, AI isn't mind wandering. It's not doing any of that stuff. Creativity is a human trait. So enhancing your own creativity, there's, um, a science of creativity, that's probably the best thing that you can do. But the next best thing you can do is to recognize that the, the timescape that you're in is artificially constrained and it is having a huge pressure on time. Discounting, discounting the future is always something that humans have done, but it's extremely more common today. So. But you can, we know there's techniques, there's psychological techniques for yourself and for your stakeholders, little tricks, priming techniques that you can use. We all have studied social psychology. We know what priming is, we know what anchoring is. You can use all of those tools to get your stakeholders to stop asking for quick wins. Or, you know, you can use those techniques.

Speaker B: Great. Last question before we hit up our rapid fire section. Just parting advice for what research leaders in particular can be doing right now to make sure AI strengthens rather than weakens their qualitative research practice. I'm a research leader, yes. And I am invested, um, in quality.

Speaker A: Okay.

Speaker B: I believe in it. I care about it today. The company believes in it. Best thing I can be thinking about to make sure that AI as it changes in the months and years ahead, has the right place in my qualitative practice.

Speaker A: While you reserve storytelling and creative storytelling, empathetic, emotional storytelling, you reserve that 100% for humans to do. Uh, you never ask AI to envision the future because it'll do A really boring, flat portrait. Uh, you never ask AI to tell the story of the humans that you investigated. You. You get it to do the stuff that it's boring and good at. It's vigilant, it's consistent, but it's really bad at telling good emotional stories. It sounds very flat. Just get it to get it to do the things that, that are. It takes vigilance to do. Not the stuff that takes empathy to do.

Speaker B: All right, Rapid fire section. Favorite interview question for research. Uh, for a research interview.

Speaker A: Oh, well, I have many favorites, but, um, one that I like to pull out when things are going really sideways. So imagine that I'm a younger relative and I really want to be successful at this thing that you do. What advice would you give me? Oh, I love that. And they immediately start giving you stuff they never said. Like you're talking to them for 20 minutes about their jobs and they've never said any of this. And all of a sudden they're like, you know what they always tell you? You got to pay attention to like the, the code book. Don't ever do that. Don't waste your time.

Speaker B: That's great. And is it, uh, it's important that it's a relative.

Speaker A: I say a younger relative because they always have a younger relative in their head. They come to like they have a niece or a cousin or somebody, you know that they even their kid, right?

Speaker B: Mhm. Yeah. That's a great one. What non research activities have you given you the most unexpected benefits for your research practice? So non research activities benefiting your research practice?

Speaker A: I, uh, would say exercise. Exercise is huge. And not just because it's physically very good for you, but also because, you know, when you go to exercise classes, you realize how bad many people are giving direction.

Speaker B: Especially for me, like any kind of dance class, it's like I'm lost.

Speaker A: Exactly, exactly. I have been in many of the. I had one dance class that I took where she's like suspend. And I was like, what?

Speaker B: What my disbelief that I can do.

Speaker A: Yeah, what are you, what are you getting at? And everybody was lost. So, um, exercise is fantastic for you health wise, mentally, but it's also really good to see how leadership works and doesn't work.

Speaker B: Mhm. Yep. Love that. Two or three resources that you recommend to others.

Speaker A: Oh, so we're talking about books here or what are we talking about?

Speaker B: It could be a book, a blog, you like, a, uh, TikTok, what? You know, whatever.

Speaker A: Uh, I read pretty much everything that Paul Krugman writes. I find his Work. Very interesting. And he's earned the right to talk. Give a hot take that I'm willing to take. There's so many books that I have read over the years that really help, but books about writing are the books that are really starting to come back to help me these days, because writing is thinking. There's a really good one called, um, the Elements of the Elements of Style. I think everybody knows that one, but it's called the Elements of Academic Style, and it talks about writing academic work. And it applies for applied researchers, too, because you're always trying to explicate big ideas and little ideas. And he talks about how you can connect those. It's a good one.

Speaker B: Wonderful. Paul Krugman. Love that He's, I think, made a nice, uh, living on substack.

Speaker C: Yeah.

Speaker A: Oh, I'm sure he has.

Speaker B: Yeah. Yeah. Good for him. Love to see people go on their

Speaker A: own and do well.

Speaker B: Uh, what is the last book that you read that you recommend to researchers?

Speaker A: The last book that I read that I recommend, Teresa. Well, I can tell you the last book that I read was, uh, Breaking through by Catalyn Carico. That was that memoir of the Nobel Prize winner. But I'm reading something right now called Once Upon a Time in the west, which is not the one you think it is. It's a series of essays.

Speaker B: Okay.

Speaker A: And it's talking about how the Enlightenment basically, um, you know, alienated us. So it's more about science. Right. It's understanding the beauty of science and the failure of science.

Speaker B: Nice. Where can folks follow you?

Speaker A: Uh, you can find me on LinkedIn, of course, but I'm also on Blue, uh, sky. I am not on Twitter, which breaks my heart because I was on Twitter for many years.

Speaker B: It was fun. It had a good run.

Speaker A: It had a good run, but it was time to go. So I don't live there anymore.

Speaker B: Yeah. And you have a website, Sam Ladner. And we'll. We'll link all of that and your books and everything you mentioned in the notes.

Speaker A: Yes. And I have a newsletter. I have my newsletter, which I host on my own website. Uh, so it's SamLadder.com newsletter. You can sign up directly there. There's no substack involved. There's no foregrounding of other people that you may not agree with. Um, it's just my newsletter, so that comes out every couple of weeks.

Speaker B: Wonderful. Well, Sam, thank you so much for joining us. I learned a lot and had a great time talking with you.

Speaker A: It's my pleasure. Nice to talk with you.

Speaker B: Thanks for listening to Awkward Silences brought to you by User Interviews Theme music by Fragile Gang. Hi there Awkward Silences listener. Thanks for listening. If you like what you heard, we always appreciate a rating or review on your podcast app of choice.

Speaker C: We'd also love to hear from you with feedback, guest topics or ideas so that we can improve your podcast listening experience. We're running a quick survey so you can share your thoughts on what you like about the show, which episodes you like best, which subjects you'd like to hear more about, which stuff you're sick of, and more just about you, the fans that have kept us on the air for the past five years.

Speaker B: We know surveys usually suck. See episode 21 with Erica hall for more on that. But this one's quick and useful, we promise. Thanks for helping us make this the best podcast it can be. You can find the survey link in the episode description of any episode or head on over to userinterviews.com awkwardsurvey.