Rethinking where UX research belongs | Joel Barr | #76

Speaker A: Right now, UX is fighting for its very life. And until we get past the whole Kumbaya thing and having real conversations about the value of, of measuring and understanding the user experience, we're going to die to the robots because we are unwilling to have conversations.

Speaker B: Welcome to UXR Geeks, where we geek out with researchers from all around the world on topics they are passionate, passionate about. I'm your host, Petra, uh, a senior UX researcher and a research advocate. And this podcast is brought to you by UXWeek, the UX research platform for recruiting, conducting, analyzing and sharing insights all in one place. Our guest today at uh, uxrgeeks is Joel Barr. Joel has spent years working closely with product and design teams. He is someone who's been thinking deeply about how UX works in real organizations beyond just methods and deliverables. What I find especially interesting about Joel's perspective is that he's not just focused on how we do research, but on where research sits and how that shapes the impact it can actually have. In a recent piece, he explores the idea that UX might be much more effective and if it was embedded in operations rather than product, which is what we'll be unpacking today. Hi Joel. Welcome to uxrgeeks podcast. I am really happy to have you here today and hello to our listeners. So this is actually my first episode as a host and you are my first guest, so it's a new experience for me. But I'm excited to be talking to you today because we will be talking about a, uh, bit of a provocative topic about your hot take, let's call it that way. And you have argued that UX research shouldn't sit under product, but it should move under operations. Right?

Speaker A: Not just research design too.

Speaker B: Okay, so let's unpack it. Let's go back a little bit. So can you tell me a little bit more? How did you come to this idea? Was there anything wrong that you see happening with UX research? The way it is positioned standardly in the organizations?

Speaker A: Actually, I wouldn't say it's wrong, but it certainly could use improvement. Right. There's very few in life where you have an absolute good and an absolute bad. There's always gray areas. Any UX researcher will tell you that. Right? Uh, it's all gray, it's all beige. I've enjoyed a career that when I started out, it was a field first business. In other words, if you were going to be in ux, it was pretty well that you would be out in the field with your Users. And then we got away from that. We started doing more remote research. And then Covid happened and we all went remote and then we all went back to the office. But user research and UX design never really got back out into the field. And so my thesis, if you will, is that we got folded up underneath product during COVID more than anything, and tied ourselves to product metrics to show our value to an organization. My thesis is that is misplaced. Where we should be is under operations with sales and implementation and retention, such that we tie the value of user experience into the selling of that user experience for a customer. And in my career, the most valuable and the most impactful pieces of research that I've ever done with my teams has always been in the field. Always. If you do it remotely, you only get a little vignette. Yeah, of a, uh, user's life and where they are, uh, using your products. If you're in operations, if you're out with sales and implementation and retention, you get to see the entire user ecosystem of how they're accessing your product and why they choose to access your product in a given context. You don't really get that looking through a window, looking through a window on a computer, if you can even do that anymore. Of course, nowadays there's tools everywhere that purport to do all that work for you. And then we have the rise of the machines who say that you don't even need to look at, you don't even need to talk to humans anymore. We can just generate synthetic humans and you can just do research on that.

Speaker B: That's pretty scary. I want to focus a little bit on that, move from product to operations, because in some organizations it's the way it's done, it's the way people are used to working. And in some organizations it works right, because the researchers work closely with the designers, with the product managers, with the development teams, and they are enjoying this way of working. But what we see happening is the democratization of research, so they're asked to do less and less. So what would you recommend to these researchers who are in the middle of this change? Because just moving to the operations is not possible immediately. Are there any small steps that we can take in between?

Speaker A: So you're right, like it doesn't work for everyone if you have a completely virtual product. Let's suppose you're zapier or you're in the WordPress space, which is a gigantic virtual market where there isn't much physical component to product offerings. My thesis doesn't Work for them. And I'll just be honest there, it's got limitations like any good science does. If you're sitting on a product and you have a physical product or a software as a service that goes out to customers, then your first call really is to sales, right? And understanding their KPIs and understanding what they need to do to meet their metrics. And through that effort, talk to people in implementation, ask them when they go out into the field, see if you can get a seat on that plane or in that car and just go out with them. And while they're doing their work in product or doing retention or customer success, you're right there also in the field with them, but finding out the real stressors of users in the field when the rubber meets the road, as it were. So whereas people understand product on the ground, you're at the ground level, you are in situ. If you're an anthropologist, you're in the middle of the house, as it were, rather than looking at the house from the outside and just guessing what goes on inside. So the real steps are just making inroads inside your company and having conversations and finding out how can UX affect the outcomes of sales professionals and implementation and the rest of the people in operations. If you can work on increasing their KPIs or exceeding them M, it becomes a very easy conversation to have because you're helping them succeed. You are the conduit between them and the user and through you the enterprise will end up building better products because you're on the ground right where the products are, instead of just in a theoretical capacity.

Speaker B: And you have argued that when UX sits under the product, it becomes a, uh, support function, that it's not the ideal way for UX research to work. Right. But I like this term servant leadership and that's how I see UX research. In organizations. We are not the leaders telling people what to do. We should connect with other teams in the organization. We, we should connect to sales, we should connect to marketing, we should be connected with developers and so on. And through this servant leadership, through asking them questions and collecting the information, we should create this insight and give that to them so they can make smarter decisions and work faster and so on. And having said that, when I read your thesis, I was taken aback a little bit because when I saw the notion like X should move under operations, it was hard for me to grasp. But when I look at it this way, it makes way more sense to me. So is there anything that resonates with you when it comes to servant leadership.

Speaker A: They say that if you get above a certain level, your role becomes strategy and then you just accept that you're not going to be in the field anymore. I don't find that to be true. I think the best leaders lead from the front and not from the rear. And I think that I could be a director of ux, I could be a vice president, grand poobah, uh, of ux, right at some gigantic enterprise. And I'm still going to be in the field with my direct reports who are also in the field, not because I'm there to watch over them, but because I'm just as interested in solving the problems of humanity as it relates to enterprise product as the rest of my researchers are. I think, you know, the best leaders I've ever had in my career have been field first UXers. In other words, we can design anything we want on paper and we can design it in some kind of software figma or whatever, but until we're out in the field and we actually watch humans use our work, we have no idea really what we're designing for. It used to be, gosh, before COVID large enterprises regularly had laboratories. We could simulate real life in the use of our products. We've gotten far away from that now and, and you really only see it in the largest of enterprises. Right? Some of them have mock stores that they bring customers into just to watch them. I think the best work is done out like in situ, even if it's just watching people in an ethnographic kind of way, just watching how people behave. There's tons of ways we can affect product, tons of ways. And so I think there's validity in the servant leadership standpoint. I think there's validity in uh, disagreeing with the entire thesis. I didn't want everybody to just be like, oh, I totally agree and just pass it off. I wanted to have debates, discussions. Right now X is fighting for its very life. And until we get past the whole Kumbaya thing and having real conversations about the value of, of measuring and understanding the user experience, we're going to die to the robots because we are unwilling to have conversations.

Speaker B: But there is still the demand for research, but it's changing the way it works, what they want and how fast they want it and et cetera. And some researchers are like, that's the only thing they can do is the uh, behind the screens test.

Speaker A: User results may vary. Right. My thesis is essentially that if we're, if you're unwilling to, or unable to fight for the ability to get in front of real physical users, then maybe you're not at the right organization. If that's not valued by your organization, if you espouse the same values as being in the field as I do and others do, we're measuring things like value and trust and stress. One of the greatest studies I ever did was around stress levels of individuals running from a terminal to a rent a car counter. And we measured how stressed out they were while having to use the app on the run. And we got our little stopwatches out, we measured how far the distance was, and then we measured just how well the product worked in the hands of a normal, regular, everyday user, but also how stressed out they were and how much or how little the app helped alleviate their stress by its ease of use. You'd never get that remotely.

Speaker B: So what I'm, um, gathering here is it's not only about the move from product into operations, but it's also about the infield research. That's something what needs to be brought back. So we are in direct touch with the users and the way they are interacting with the products that we are developing. If you can walk me through an example of an organization before making this shift and after.

Speaker A: I won't tell you the name of the company, but it's very large and, and you probably order from it. And one of the things I was tasked to do is, is to go into the warehouses of this particular organization to work on the interface of garage doors and lift extenders. Not the sexiest work in the entire world. Bit like watching paint dry at times. The problem was, is that this particular organization came out with this new way of opening and closing garage doors and extending and descending ramps. And I'll just give you a little spoiler alert. Garage doors go up and down, ramps go in and out. It's all you really need to know about what they can do. But the way in, the way in which you get those doors to open and close and ramps to extend and descend was this fancy one button and it'll take care of everything. As it turned out, they didn't actually study a single user before they created this newfangled way of doing things. It was basically was uh, by product thrown into engineering, developed and thrown out into the field in the old fashioned product first way, right? And we'll fix whatever goes wrong in the field. We'll watch the JIRA tickets, as it were. It turns out that warehouse workers are not particularly tech savvy. And so uh, the call went out. Nobody is using this new interface that's worth millions and millions of dollars that it took to develop it, nobody uses it. People go and do the manual way instead of which wastes on average about a minute and 30 seconds of time. And so they deployed myself and others out to these warehouses to find out why people aren't using this particular interface to make garage doors go up and down and ramps to go in and out. It became quite clear no user was consulted in its design. And you know this because one of the things that this particular company's warehouse's safety mandates is that all users, all employees, wear gloves to protect their hands. Unfortunately, the screens that they have to interact with aren't club friendly. You have to type in little tiny letters that are not great for gloves. There's no language support. So if English is not your first language, there's not a lot of options for you to work on. And as you might imagine, warehouse workers come from different places, as do we all. And a lot of them just didn't understand the verbiage of what was being asked of them to do. And then there were the other problems with notifications and that sort of thing on the screens themselves. It's a very complicated system. It's intended to be simplistic in its design, in its function, but when it goes wrong, or when the user does not do things in the right sequence, the product gets out of hand rather quickly to the point where a door goes down and is no longer functioning in these warehouses. No air conditioning, no heating. Which really got down to the grits of why these folks weren't using this product and forwarding up our recommendations not to product, but to operations and engineering. Changes happened overnight. They were rolling out updates week after week. And now they have 95, 98% usage rate for this one system. So that was a real eye opener. Uh, this is a very large enterprise that does a lot of business. And here we are, little user researchers out in this little podunk ass town in the middle of Kansas, reporting back to this place on the west coast that their system that they invested millions and millions of dollars in, one, doesn't work for the user, two, doesn't meet the user's needs, and three, needs to change or nobody's really ever going to use it. And they were rolling out changes in weeks, not months, not years. And it was all evidence based and videotaped and reported on. And here's what needs to change, and I'll give you another great example of this on the doors, on the interfaces, when you're Doing opening and closing operations. There was an ear splitting beep that would go constantly while it was doing these operations. Just made your ears ring, sound loud and just at the right tone to where it makes your ears ring afterwards for about two minutes. And the manual mode did not have that beep, only the new mode bit. And to be honest with you, these guys were taking styrofoam uh, cups and taping them to the speakers. They were uh, doing all sorts of ways to get it to stop beeping. And it was a terrible quality of life thing. Engineers don't know about that product, people don't know about that. To them it just, it's that's way it's supposed to work because of safety. And you're trying to alert everybody around you that you're operating a really big door that can crush them. But if you're a user you're right up against that speaker and it is loud. And so one of the most immediate fixes that we sent back from the field, you gotta either figure out a way to lessen that beep in volume or just take it out altogether or make it more pleasant because my God. And it took em a week and they eventually just took the beep out altogether.

Speaker B: And you have that in SaaS as well. When users find a workaround to something what was not intended that way and suddenly they're using your product completely differently to what you have intended, you couldn't anticipate that. And you get that through research so

Speaker A: far that's the case. Yes. I'll give you another one. Same idea. And we were at a company that sold, oh shoot, Hondas. I think it was Honda's foreign cars. Uh, they were big users of our products. They were probably one of our biggest customers. And we get there to Vancouver and we find out that they're not using our product necessarily. They were so tired of the product working poorly that they went and built their own that sits on top of it. So you want to talk about workarounds, they made an end to end workaround that sat on top of CDK so that it would work in the way that worked for them. And be like taking a uh, car that you wanted to go across a river and putting floats on it. You're still driving the car but the floats are what keep you above the water. And, and that was a very eye opening experience. And so we reported back, we're like, hey, you're going to want to study these guys more because they're making your software work by building something on Top of it, like a reskin or refunctionality of your software and it's working great for them. And I tell you, that raised some alarm bells.

Speaker B: Well, as it should. Yeah, definitely. Okay. But if I circle back to the operations, right? So I still see in product a lot of strategic decisions happening. So. So if the research is moved from product, do you think that there is the risk of losing this power of the, of the impact of practice?

Speaker A: I don't think so. Because strategy is only so good as the evidence allows, right. If you have incomplete evidence to back a product decision, if you're lacking research, what are you really strategizing around? I think that should be an open question. If you don't have like on the ground research, what are we really strategizing around?

Speaker B: Yeah, but that's what I'm thinking. You also mentioned at the beginning that if the lead or the CEO is not listening to the research and all the main decisions are not based on research, then you are developing for no one, really. So that made me think whether this is a structural issue or a leadership issue, because I have experiences with a company where the CEO was coming to the research calls, the GM was listening to them on a regular basis. So they stayed in touch with the user and they understood the user problems and pain points and et cetera. And that to me was leadership where research was right in the center of all the things.

Speaker A: My thesis does not apply to everyone. I am fully aware of that and I make no assertions that my thesis or my ideas are ubiquitous. In other words, your mileage may vary. And if my point of view is something that you enjoy and you want to put in your own organization, do. If you don't enjoy my point of view, it's okay. I, that's why I made it. I want you to push back on it. I want you to say, you know, that this doesn't work in my organization. Okay, that's okay. But yeah, like leadership I think is a big part of it. I've interviewed at a number of companies lately where the head of UX was never in UX to begin with. They sat in marketing or they sat in product and they got Knight it into X is a kind of a fascination or because they needed the chair filled by a body that they didn't want to go out and hire. And now they're trying to hire field and very experienced researchers and designers to sit on their products more. And I'm just like, man, we could do so much better. We could do so much better. And I understand why decisions like that are made. But I think where we're getting to now is where humans are beginning to be able to delineate or be able to tell the difference between companies who are phoning it in. Right. Or using slop as it were to comprise their experiences and those that actually put the work in and study. I think there's. Humans are starting to see a real difference there.

Speaker B: Yeah, and we touched on that. The usage of AI tools when the companies are trying to substitute the researchers with the AI tools and they think like the good enough result is all they need all or the good enough research is all they need.

Speaker A: Which is funny because AI user research is so easily manipulated. Every so often I'll run an experiment where I'll take insights from a study I've done or something like that from an old study and throw it in AI transcripts. Then it, ask it to analyze it, you know, come back with results and then I'll say to it, no, that doesn't work for me, I really need more negative response. And it just says okay and it just rewrites it negative. And I'd say oh yeah, that's too negative. Can we make it a little more positive? Oh sure, that just does it. That tells me AI insights, especially if you're using synthetic users, there aren't very few guardrails that account for integrity of the research evidence. In other words, if you want to use synthetic users or artificial intelligence to back your product point of view, it's very easy to just ask the AI uh to come up with whatever you need to back your point of view. And I think that's a real danger now, right? I think that's a danger because everything's are all about saving money on labor. And why wouldn't they be? Because what's coming out is not increasing their sales or their bottom line. So they don't have the money to afford to hire anybody or even the travel for field research. Yeah, I don't blame them at all. But I think it's coming to a point where there's a fair amount of, it's not slopped directly, but there's a fair amount of finagling, tweaking going on and product insights that either UX doesn't feel empowered to speak up or they do and they're shown the door. And I think we're going to come to a point where humans, everyday humans, general population humans will be able to start telling the difference between product backed research or research backed products. And those that are just slop yeah,

Speaker B: well, in next week, we actually conducted a, uh, meta analysis on, um, the synthetic parts. So we have collected everything that is out there, all the studies that have been conducted across different industries. There was healthcare, there was technology, and et cetera. And that's the, the findings that we gathered is that the synthetic participants should not be used as methodology. They are a tool to help you get started, to come up with your research plan and so on. But it's not something that should be your final outcome because it's just too vague, it's biased. You cannot go into so much depth. You don't get those random wow moments with synthetic participants. And with AI, you cannot have AI as a human element. Right. It's still not there. So I'm completely there with you.

Speaker A: Not yet. Uh, every day they try and replace X research every single day. And I knew that. I knew when I signed in I was going to be the underdog for life. It was just what. It was part of what you accepted. We didn't have seats at the table for the longest time. We had to fight for those. We have to fight for our very existence to be not replaced by some automated moderator. And those robotic moderators are terrible. I was watching one. I forget what company came out with it, but the user said the third question that they didn't like talking to automated systems. And the thing goes, oh, that's nice. And just picks right up with the script right where they left off. And I thought, oh, my God, what? Any human researcher would see that and be like, okay, maybe I should. Do you want to end the call if you're not comfortable talking with me? I'm at. Because people have different traumas, different triggers. We have to be mindful of that as researchers. But the, uh, robot moderator just went straight on and it affected the whole rest of the conversation. And they put it out as a demo. It was a couple of years ago and I just, I remember going on the comment section and just going off and being like, why on earth would you do that to a human?

Speaker B: Yeah. There is so many things happening that we as researchers don't like. And I think it has its due date. Like, people eventually will realize that they cannot get the real insights, the, the good quality of research from these AI tools. But it's going to probably take some time.

Speaker A: I don't even talk to recruiters anymore that use automated systems to talk to me. I just. Hi, this is such and such from some kind of recruiting company. Is this such a stand. No, I just hang up. I don't care if it's the greatest job in the world. If they're not willing to commune with me on a human level, I just have no time for them.

Speaker B: Yeah. I think also a context is really important when it comes to the AI tools. I was gathering that from what you were saying as well, that AI tools do not have that context and research should be connected. You have mentioned that research should be connected to sales and to the KPIs and to the revenue and, and understand the retention and so on. So what can researchers do today to better connect their work to revenue and retention and trust?

Speaker A: So the way I would advise people to go about having those conversations is to try and figure out the schedules of the folks that are going out into the field on site and get. And I will tell you that it will not probably fit in very neatly with a product agile sprint. Right. Because sales and implementation retention, customer success, they all go out at different times. But if you can get on the schedule to, or on their travel schedule or at least understand where they're going, you can start having conversations ahead of time and it may take a matter of months, it may even take years to pull off. But eventually, if you keep making the case for the need to be in the field with your users understanding how and why and where they use products, you're going to change the entire way the entire enterprise looks at their human users in ways that robots can't do. And I think that's one of the. Going back to what you said about the synthetic user researchers, the one place robots can't go or understand yet, because who knows, 40 years in the future that might be different, but they can't understand the context of where humans exist. To an AI, the world looks in the same way than a mirror at a funny park or an amusement park looks. Right. It's all topsy turvy and all that. That's an AI's understanding of humanity. And when we look away from the mirror, we see the world around us. And I think that a lot of enterprises is enamored with the way the world looks in the funny mirror.

Speaker B: I want to turn it to something positive because we were talking about a lot of dangers, a lot of risks and et cetera. Uh, but I do think that there is a future for research and I was gathering that from you as well. Right. That eventually the management will learn that developing the products without talking to the actual users and seeing them use their products and so on will just not make sense. Right. So to sum it up, you have advocated for the infield research, for moving research from under product to the operations so they can support all the different teams and gather all the information that is in the company and so they can be in touch with the users. Right. Is there anything else that I'm missing here?

Speaker A: So you're tracking correctly, but the metrics of the value of ux, both design and research, being tied to sales and not CSAT or NPS or something like that, something arbitrary is where really we show our value in a way that companies cannot ignore. For so long we've been asked to show our value to what are your, what is your value to the organization? We've struggled really hard as a field to try and make that salient. And I feel like if you can tie it to things like customer retention and sales and implementation numbers, suddenly the value of the user experience becomes very real for a lot of companies. And I saw it at the companies that I talked about.

Speaker B: Well, perfect. Thank you so much, Joel. I think we have covered a lot and we have also some takeaways and some practical applications of what the researchers can start doing today to be ready for this change and for everything what's coming to us. So thank you so much for coming, Joel and I will see you around.

Speaker A: My pleasure. M

Speaker B: thank you for listening to UXR Geeks. If you enjoyed this episode, please follow our podcast and share it with your friends and colleagues. Your support is really what keeps us going. If you have any tips on fantastic speakers from across the globe, feedback or any questions, we would love to hear from you. So reach out to geekspodcast uxtweak.com special thanks goes to my colleagues to our podcast producer, Katarina Novikova, our social media specialist Daria Krasovskaya, and our audio specialist, Melissa Danishkova.