Quality Of Evidence

Hi folks, this is all things product with Petra Bille and Teresa Corks. And we're so happy you here! Theresa the other day I heard his hallway conversation at one of my clients... and i was thinking of dead something for our podcast.

Theresa and I need to discuss that topic.... Do we actually really need to go see users? do We, Actually Really Need To do user interviews because. Have all these feedback coming in All the time and isn't that enough user research done?

What's your take on That? yeah so product teams i think are drowning In data And we have lots of sources. right. we have Behavioral analytics if we if we've instrumented our Product we have.

for If we have a sales team they're talking to customers all day. Almost everybody has a support team or you're getting support tickets. Some companies even have like feedback forms where people can just submit all types of stuff. I do think product teams should be using all of this.

But i think there's, this concept Of the quality of evidence That teams need to understand and i'm actually working on a blog post On this. so hopefully by that time. This episode comes out weekly. how timely.

yeah amazing? Yeah but Um, I've written about this before like i have an ask Teresa blog post called something Like what do I do with all the insights that come from sales calls and support tickets? And things like That. What I've Written in The past is that I Think About those as Signals that tell me what to explore In my interviews and the reason for that Is that Those signals rarely come With enough context To know it to Do Right.

so if someone Sends in a Support Ticket and Says Hey, I'm trying to do this thing and I'm stuck. Like rarely does the customer actually write all of detail on what they're doing or why it went wrong. They just write a symptom that says like This feature looks broken And you'll be like It's not broken! What is going wrong for me?

You can't act on that. It's a signal. somethings wrong here But don't know enough how fix it. Dangerous here, sometimes you look at it and think what that means.

You know how to fix it because projecting your own experience as a product expert onto one sentence of feedback then if you go talk with the customer they are trying something like total oddball outlier and it's actually really important that we collect that context. And would you say, because in my coaching what I usually tend to reflect back on my coachy when they ask a question like that is Henrik Niebergs Triangle which is if you have one tentative data there could be an App Store review the same idea and verdict.

And then you can find a piece of qualitative evidence that it is a good idea, Then your good to go basically but You always have to make sure that you test for all three. so Is there an expert in the organization supporting? The evidence is their signal That you are talking about In random quantitative feedback data stuff qualitative insight on top of it. I would get more specific than that, so i'll give an example...

I get exposed to a lot of customer data now because I have my partnership with Vistali where we're doing AI-generated interview snapshots and AI generated opportunity solution trees And I get to work with A LOT OF customer interview transcripts! What I'm learning is like We use this term Interview for a wide variety of research activities And, I mean no surprise. So like i'm gonna give a real example A team did an interview where they literally were talking to a customer and the customers shared.

uh...I have it usability issue when I rotate my laptop My iPad from portrait-to-landscape. Here's the issues that im having. We probably see evidence of this issue in our behavioral analytics.

Maybe we see people switching between portrait and landscape, try... And then like when they're in Landscape that can't find a button They never push it right? Like We could probably see evidence of a problem In our analytics. We probably even conduct A survey and learn People prefer Landscape to Portrait.

We just did an interview where someone showed us the Problem. Can I now fix It? maybe Here's what I wanna know before I fix that problem. What were they trying to do when they encountered that problem?

Why is landscape better than portrait, like? what problem are they trying solve by switching the orientation of their iPad and what specifically is going wrong? And i'll tell you in this specific interview where got to see a transcript The interviewer didn't get into any data right. so now we go build feature just guessing.

Okay, well we see people want to go from portrait-to-landscape. Let's make sure landscape works better. Works better. how?

Yeah I don't know right. so this is why We teach story based interviewing. i wanna collect the Story. what were you doing?

when did This come up? What like what? where are You trying? what need arose that caused you To flip your iPad?

Did it actually solve Your problem? Is there still an unmet need here? I want that whole context. And the challenge we have is like sales calls, if i'm talking to a prospect and they say hey We need a solution That will flip into landscape mode Okay why Most of the time The sales person would get part- The good old product management question.

Yeah most Of the time the sales rep is Like yeah we Have that feature Checked the box move on But thats not really Appropriate feedback Until I Get the Whole Story. Oh, the whole vibe coded software tools. This work will be more important than ever because it is easier then ever to release all these mediocre software tools where nobody thinks about this exact. okay?

Is this button really here today? do we need it? can't It be elsewhere? Can I look different?

why Do people needed in The first place All These kind of questions? so yeah People get Better an interview. Yeah, I mean like i'm gonna say Somewhere between ten and fifteen years ago. I wrote a little video about the ladder of evidence.

It's it was just this framework? I came up with where I tried to communicate. To get quality evidence We have to move higher up The ladder. so as you go up the ladder that effort goes Up but the value also goes up And its Just This idea Of Like.

We're inundated with these low value signals all the time, but they rarely carry enough signal strength to actually tell us what we should be building. But they feel like they do and the reason why they feel that way is because we are experts on our product. so when you look at those signals But we don't always know what they mean and We often project our own experience onto a low-quality signal And use our experience to turn it into a higher quality signal, but the problem is that translation.

We often get wrong a lot. We don't realize like they were using our product for something. We never thought They would have used it. four yeah, and we could still.

so I could picture eight teams challenging each other on Jumping two assumptions too quickly and nowadays Your favorite LLM could do the exact same thing. So you can have something like the devil's advocate skill if You planning to look at data, then they could always say what have you really understood? The data is it just a signal? Can you kind of act on the data?

that could be something people could be doing and practicing a bit more right? Yeah so I Okay, so part of my work with Vista Lee II. Because i'm generating AI generated opportunity solution trees I have to think a lot about what's a strong enough signal. To say this should be an opportunity on your tree, right?

And so when we started... Right now we only support two interview types. it has to be story-based interview. that is the strongest signal for an opportunity.

or because most teams aren't very good at story based interviewing We also support general interviews. So in general interview as i just ask you direct questions You tell me about your experience. Don't love those types of interviews. I actually think that's still a pretty weak signal.

But if we didn't support that format like almost there will be no product right? Yeah, and long term our goal is to help use the tool to teach teams Like show them a signal strength And then teach them how to collect it better signal over time like it. but what's interesting is We're seeing a lot of teams submit transcripts and they don't classify in either of those categories. They are actually product demo, so their demoing the products to customers.

How do you like this customer? Yeah! Their team meeting... So my guess is it's stakeholder interview.

And then thinking about stakeholder interview as a customer interview This one is a tricky. One we see ones where it's not a usability test. the interview participant isn't giving The Participant. I'm sorry, the interviewer isn't given that participant a task and asking them to think out loud.

It's not use ability tests but they're basically asking for their usability preferences. so though the interview will be like what? Tell me about your experience with our product, which sounds like a really good open-ended question. But it's not because It is not grounded in specific instances and stories.

And so then the participant will be like well I don't Like landscape mode? Then they'll tell you what You Don't Like About it. They'll enumerate What they don't like about it. The interviewer will write all that down.

They just collected a bunch of, like probably idiosyncratic preferences with no surrounding context. And so this has really opened my eyes to how much misunderstanding there is about what a good interview looks like and also just... What data is good enough for us make decision on? There's this pragmatic piece.

I can't wait you be perfect interviewer before making product decisions Right. Like lots of teams are making decisions with no interviews and I can't like Go for perfection to be like. we can't use this data because it's not story-based. We can't used as data cuz you asked unreliable feedback?

We still have to figure out, like what is the signal here even if its low quality evidence? Yeah So this has been really fascinating for me to just think through. like Can i come up with a rubric for quality of evidence which i did ten, fifteen years ago with my ladder of evidence. But now how do I pair that?

With like... How much confidence can you have in this signal and what types of decisions should you make on it?" Yeah! It's really fascinating.

That is really fascinating. And Now You Have To Bake It Into A Product. By the way just to mention i think This Is a Very Good Example For A Strategic Product Decision. Do you love the decision?

no, you don't like to decision but it is a very strategic decision to say like even if we don't think that's the perfect format for an interview. We still take it with still work within but then we educate user towards what actually thing is good. Interview and better strengths of evidence or better evidence quality however you call it and I really liked at its just like magnifying glass here. So this is what we usually talk about when we say like, that's a strategic product decision?

Well I think also it's very practical product decision. right because if i look at the total addressable market If we limited to like We're only gonna pull opportunities from story based interviews which in my world That Is What I Would Do! We might have twelve customers. Yeah And do all your former trainees Even people that have been through training, it doesn't mean they've kept the habit or they've practiced a habit.

Or they continue to develop the habit. like story based interviewing is skill but takes work to learn and works to maintain. It's disciplined. do over-and-over again The reality.

most organizations aren't asking of their teams so let them fall by the wayside. And here are other kicker. If you did conduct an interview, it's not a great interview. It still better than having never talked to your customer.

so I don't want give the team feedback like we can use this because now just discourage them from talking customers and that was worse right? Yeah So its really is spectrum where like The worst thing you could do is Never talk with customer. i think best thing You Can Do Is collect a rich story about their experience. And then there's whole spectrum in between.

And the way that I think about this is, if i'm on a story-based end of the spectrum there's less risk in those opportunities. I can be more confident than their real needs. but it doesn't mean If im towards like kinda crummy and interviewing That theres no signal. There IS A SIGNAL THERE.

Its just not as strong As a signal. Yeah So This Is Something I've been thinking alot About. It's like a fun UI challenge of how do you communicate signal strength? And in way that doesn't discourage someone from continuing to interview, but motivates them to move down the spectrum or little bit.

To get better at interviewing. and so this is really fun. it's fun to give little nudges. we have ability now say instead asking next time try asking.

This is so cool, right? Yeah. And I love that as always you're publishing it on your blog for people to read as well if they want to draw their own conclusions about how day could improve. You know i kind of have too because like If you're going to build really opinionated software which Is what our goal is and her opinion in his story based is better.

um we have to. We have to communicate the why. yeah Like we have To be really transparent About Like. if we give you an indicator of signal strength, We have to be really transparent about how we're communicating that signal.

Yeah So yeah. And then as a coach. the thing that really motivates me is I think this provides now we're looking at. We're working in the context of your work were working with your real interview transcripts.

Were working at the moment. where? You trying to synthesize them? And to me, this is a great coaching moment.

To just give these little nudges of like next time. try This. yeah see the difference between this and this. See how?

This gives you more context for thinking about a solution. Yeah How this is very vague. I guess it's a very fun Coaching moment in the context of a problem. think About ya.

i see why You love it Teresa. i see where you Love It. Yeah Super problem. yeah so cool.

yeah i Think we wrap it up and i say Thank you, Teresa. Thanks, Petra!