Season 9, Episode 8: Designed for Insight: Blueprints for Mystery Shopping

Speaker A: Hello and welcome to the Experience Perspective. This is an Ipsos podcast which is for anyone interested in improving experiences for your customers and colleagues across all sectors. Across this series, we will bring you insights and inspiration from around the world whilst we have a chat with experts within the field. Now, let's get started.

Speaker B: Hi, everyone. Thank you for listening to the podcast. Another interesting episode ahead. Before we get there, I just wanted to say a quick reminder that please do like and follow wherever you listen to your podcast. That way you'll get to know first every time that we release new episodes. As I said today, another interesting conversation on the way and today's guest is Andy Firth. Andy, you're a colleague of mine at Ipsos with a very particular area of expertise and specialism. So to start, it'd be great if you could introduce yourself and tell us a bit about what it is you do.

Speaker C: Thanks, Rob. Hi, everyone. Yeah, my name's Andy Firth. I am, um, head of advisory, the advisory and analytics team at Ipsos Channel Performance uk. I see the role of my team comes in really at the very end when all the kind of hard work's been done, the operations and we've got this great treasure trove of data and the role of my team is to go in and analyze that along with client data and basically tell our client what it means.

Speaker B: What's the.

Speaker C: So what, what's the commercial impact? Well, as the now what to help them, um, kind of action our insights and drive their business forward.

Speaker B: That is amazing. And actually it's, it's kind of mirror image of that topic that we're going to be focusing on today because like you say, you come to this data so often at the end and you're looking at this data on mass and finding these themes. But the purposes of today's podcast, we're going to rely on your expertise to talk about if that's what we can get out at the end of a project. What are the considerations that you should have at the start of a project to ultimately design and build a mystery shot program that can give you the greatest amount of usable and actionable data at the end? So, yeah, really looking forward to getting your advice and steer and insight on that. But before we start, maybe we'll start a little bit broader because I think the first question that would be great to share with us, our listeners, is when it comes to mystery shopping as a methodology, where do you think it particularly excels or where do you think the best uses of mystery shopping tend to be in your experience?

Speaker C: Well, the division Mystery shopping sits in within ipsos is called channel performance. And there's a reason for that, because I think it's the best methodology for objectively assessing the performance of a channel. Uh, so that might be telephone, online, email, but predominantly it is face to face interactions. And that, you know, is often or mostly in stores. But we do plenty of work, uh, on transport, for example, and also in delivery. So the real strength of it for me, for clients is knowing what is happening on the ground, let's say in a store environment. That gives you a real detailed, objective assessment of what is happening, not based on recall and um, predominantly focusing on what the employees are doing and how the operations are working. You know, we can ask mystery shoppers to provide feedback on their own feelings. That's more the remit of customer experience research. But for me this is how are your employees performing? How are your operations working? And we know that's important because when we put it together with commercial KPIs, for example, we tend to see really strong relationships. And I don't think there's another methodology that can get to those insights, get

Speaker B: to those kind of moments of truth, those absolute moments of interaction between the business, the entity and the customer. Yeah, absolutely. So when you think about, again, obviously we are thinking about what can come out at the end of this and the many insights that can be drawn from this type of data. But if you go right back to the start and if we're advising maybe an imagined client or even a group to think about what needs to go in initially, what are the ingredients that make the most impactful Mystery shopping programs? What do you think are the key things that clients, organizations, individuals should consider at that stage?

Speaker C: Yeah, well, I mean, in many ways Mystery shopping is a research method. So you need the same pillars as any strong research program. So at the core of that, uh, of course is the questionnaire. Uh, I mean the questionnaire or survey, whatever you want to call it, has to be well designed and it has to be designed with the business objectives in mind. Uh, of course. And from an analytics point of view, there are elements of a questionnaire that really help us in analyzing the findings and understanding the why. So we have methods and structures around how we create really good questionnaires. And actually Mystery Shop and questionnaires can be quite long. No questionnaire should be too long, but they tend to be more detailed than say a research questionnaire. Uh, because the shoppers know in advance what they're observing and what they're, what they're answering. Of course, you have to have a really good sample. You know, for a lot of our clients who own their own network, you would tend to advise that we need to go and assess all of your stores. But a lot of the time we use smart sampling and say actually if we're going to focus a bit more resource, let's do it on the, maybe the really big or quote unquote important stores and possibly those stores that are underperforming. If you were a client with a big estate, you'd probably want to focus on the ones with the highest footfall and the underperformance because that's where the most potential is. For example, obviously we do do a lot of mystery shopping where we are going into third party retailers. You can imagine a manufacturer wanting to know how they're products being recommended in another, in a partner retailer. So again we have to design the sample around that because sample is really important. The third bit that really isn't used in other research methods is of course the scenario or the mission you're giving the shopper and their instructions around that. You know, most other researchers are surveyed with a customer. We are instructing shoppers to go in and follow a mission so we can assess that interaction. So again, that's really important to get right. And of course, as I was always say, the analysis of all that data is really important, but we can't do a good job with the analysis unless we've got the data itself and that relies on the survey and a good robust sample.

Speaker B: So when you say a good robust sample, what are ah, and I know these vary based on the client, based on their estate, sometimes based even on their industry and setup, but is the kind of any kind of broad rule of thumb about the size of a data set you would need to be able to do some of that analytics, some of that deep dive work on it. What kind of sample sizes broadly would

Speaker C: we be thinking of? Well I mean for analytics, the more the more the merrier. Uh, yeah, obviously you know, we're doing quite a lot of work with our US colleagues who have big programs. I mean obviously the US is this kind of the size of Europe. Thousands and thousands of surveys which allow us to just do so much good work. I think we would always say that maybe 100 is uh, a kind of minimum for some sort of top line analytics. But we really prefer to get into the realms of 3 and 500. If we're going to do some of the analytics modeling, including combining Ms. Data with commercial KPIs, that really gives us confidence um, that what we're saying, uh, is true, if you like, and we can put our kind of statistical confidence levels on that.

Speaker B: Well, that's an interesting. Because already we're starting to move from one theme here into something that feels like it's got a real impact, like you said, because on the one hand, a mystery shopping assessment at an individual level is at the heart of the interaction, often between client and customer, but also is a moment in time. Right? It's a single snapshot in a single location. But once you start building those numbers up, like you say, that robustness, that statistical confidence that builds with those additional data points, you can absolutely see how that can, you know, be incredibly useful when it gets into the hands of you and your, your very expert team. So sample sizing, I think that's, that's really useful and really handy. I want to bring us back really quickly to a point you made at the very start. So you talked about the importance of making sure that your questionnaire, uh, chimes with and reflects kind of key business objectives. Is there any particular themes that match very well or ways that maybe clients can step back and do that type of checking, that type of checking and balancing, or what types of things are you talking about there?

Speaker C: Well, the first question my team always asks, uh, and that may be at the start, it may be one of programs running for us to be able to, you know, provide the client with insights that they can use and are meaningful, is, why are you doing this mystery shopping? And I think everyone in our team, from, from operations to project management, validators, allocators, analysts, really, they try and understand that. I think it helps all of us. And we tend to bucket the reasons into groups if this helps. One of them is just what we call compliance. And these will tend to be run by compliance teams in a client. So if you take age verification, for example, age verification teams in a client that sells, uh, over 18 products, for example, they need to know when that product is being sold either in store or delivered, that the person buying it is being asked for their id. It's almost as simple as that. But that has such an impact if you get that, uh, wrong. And really the only way you can measure that objectively and not based on recall is mystery shopping. That's compliance. They don't really care a lot of these teams about whether the person's been friendly or did this or did that. They ask for their id. And there's lots of compliance with financial services. That's just one example. The second one, I'd say is sales conversion. And that tends to happen in retailers who own their own network or luxury or auto, where really, when you think about it, the primary objective is to sell that, uh, product at the point of purchase. If someone comes inquires about product X, the role of that employee or that store, when you think about a store, they're expensive or a dealership, they're expensive things to run. So really they need to be driving revenue whilst keeping their overheads low. So what we're measuring is the likelihood of that sale being converted. So we call it sales conversion. I mean, you know, sales is sometimes a dirty word. It shouldn't be. If you don't make the sale, you don't have a customer. The third one, very quickly that's related to that is what we call product recommendation. And that's where manufacturers are looking to assess employees in their third party retailers who sell their product. So think electronics, bed, mattresses, phones. How else do they know how their product is being spoken about in their third parties? Again, objectively, in detail, without recall. So that's mystery shopping. So the scenario is the customer goes into a third party electronic store and says, I'm looking to buy a tv. What happens, what's recommended first and the manufacturer of those TVs, it's really important information for them. Yeah. And the last one is what I would call kind of customer retention. And that's, I guess when we use shoppers who are already customers of somewhere like a bank or they are already buying something like in hospitality, like a meal or petrol, in oil and gas, they've already bought it, so we're not trying to convert them, but what that location is doing in the employees and maybe trying to get them to buy a bit more, which is an upsell, but also make sure that those customers are going to come back again. And, um, they're loyal and they've had a good kind of experience. So that makes sense. We kind of bucket it into four. Um, I'm always happy to hear if anyone thinks their mystery shopping program doesn't fit into those. But that's quite a neat way of, of covering things up because that allows you to focus on the business objectives and the insights that are important.

Speaker B: Nice. And I think you mentioned there, so, so the takeaway for me there is you've obviously got core measurements that are important to be included objectively. You know that your real core are there. But also there's an element there that you're describing that it also needs to reflect in a way the customer's journey or the real world experience. And yes, there will be a number of other questions that will add some color and add some detail to what those interactions look like. But those core measures are all important. So the counterbalance of that, then I suppose you've got these kind of two contrasting elements. You've got one thing that you want, a mystery shopping program that absolutely aligns to your goals and matches your real world kind of customer journey to capture all that important data. When that data starts to come back in. And I know we've talked about there's an analytics element that can certainly be added to this, but what are the best ways to use a mystery shopping program? So if I'm a big business and that data is coming in, I'm, um, imagining there's quite a lot of that data that can be actionable, that can be informative for clients. What do you feel some of the best uses of that can be?

Speaker C: If you take a classic mystery shopping program where mystery shoppers are assessing the stores of that client, say they got a thousand stores, I don't know, fashion retailer. Are we going to every store? There are two key ways that I would say they should use it. There is at the tactical level. So every store would receive a really clear report on what happened at the assessment in that store. And the store manager, as well as their regional manager should be taking those insights and kind of actioning things and saying, this is where we fell down. This is important. We need to do something about it. Yeah, that's tactical. The other end is strategic, which is really where my team come in. And this is where we are talking generally to head office people. You know, these might be head of ops, retail people, whoever's got a stake in the retail network. And we are telling them these are the big themes we are seeing. We can see there is a relationship between these types of behaviors or processes and your commercial KPIs. So this is where you need to be focusing at a strategic and a tactical level on improving those. And if we can identify the kind of levers, investment levers, you need to pull to improve those behaviors. Now that could be training, better resourcing, stronger employee wellbeing, whatever it is, then we can help you improve those. Which means you will see stronger commercial KPIs, which at the end of the day is what senior people in businesses ultimately care about. It is that commercial background really and improving those KPIs.

Speaker B: Well, that actually was going to be my next question. You started to answer it actually. Really, or in fact have answered it really. Well, because I was going to also ask about how these programs tend to evolve and what people can learn, you know, how it can help shape your business and kind of shape your actions. But I think the two examples you've given there are great, because I'm assuming that, you know, we, we identify this area of focus, we're talking to the client about, like you say, maybe training or a level of investment in resources. But then I suppose what you're looking in the next wave or round of mystery Shopping is the impact of that, right? Whether that's moved certain needles, whether certain metrics have changed, whether the feedback from mystery shoppers is kind of reflecting those changes. So it does end up being a very, uh, much an evolutionary process, but a continually relevant process, I suppose, because there is that connection between, you know, ourselves and the clients that you are working with them as you make these evolutions and forever slightly change these focuses to measure these key changes and key steps.

Speaker C: So that's the. That's the ideal. You have to work closely with your client. If after quarter one, you said these are what we might call the key behaviors, this is what we think you can do, improve them. It's kind of incumbent then on the clients to try and do something about it. We can help them activate it. But then the beauty of it, uh, if they have done something about it and we see those behaviors improving and we see the commercial impact, you can track that over time. And once you prove that, you know, they might have sorted this one bit out really well, okay, then there's something else now that you need to do, because you're never going to reach absolute perfection, but you should be improving, you know, wave on wave almost, if you can. Iterative approach as you kind of described.

Speaker B: And like I say, with some of the analytics elements and I mean, even with some of the key measures, you know, the core questions we were talking about, those are all things that hopefully you're seeing have an impact not only within your mystery shopping program. So an improved score or improved performance in a certain area, we would argue that a lot of the time we're seeing that in the broader business as well. You know, that those interactions and those performances at a store level are actually having a far bigger impact further afield and been seen across a whole host of metrics in the business. In terms of my next question, you've talked a little about the objectivity and the fact that mystery shopping as a methodology is perfectly suited to some of these things. How would you say we would ensure that the observations of Mystery shoppers are captured in such a way where it always is usable, it's always reliable, reliable, accurate, unbiased. What are the kind of steps that need to be considered when building a mystery shopping program in relation to those,

Speaker C: do you think It's a great question. And, um, one of my kind of favorite questions, if you like. But I mean, the first and underlying fundamental principle is that the shoppers all are, uh, anonymous. You know, that's the whole point of mystery shopping. The staff member can never know that they're in. It sounds obvious. Um, but the way you do that is the shopper has to be as, as, as real as possible. Yeah. The employee has to think that they are a real customer. Um, and we, I think we're the best in the business at doing that at the same time, while the customer's doing that and, you know, the closer they are to reality, a lot of them are customers. This is what they are doing the better. But at the same time, they are observing the things that we need them to observe. So that's the first thing. The second thing, and I think in all the analytics we do, we tend to find that it is the objective metrics that we are measuring tend to come out as the things where we can find a strongest relationship with business outcomes, such as commercial KPIs. And also they tend to be the things that are most easy to action because they are objective. So one example of that might be friendliness. You know, we would never say don't be friendly, but the difference between a question that says, how friendly was the member of staff? On a scale of 1 to 5, we'd probably include a question like that. Uh, yeah, but, but the elements that kind of characterize friendliness, like, did they say hello, was it a warm greeting, did they ask you your needs, did you get a friendly farewell? They're the objective metrics that we would score whilst maybe having a how friendly were there on a scale of 1 to 5 in there, but not scored because it's too subjective. Yeah. And what do you do with that? Uh, anyway. Yeah, okay, go away. You've got to make that store a bit friendlier. Or only 60% of your stores actually gave this greeting. You need to work out which stores they were and how we can help improve that metric. Yeah. And does that make sense? That's the kind of. And that's just based on a lot of analysis. It's not just me, um, you know, hypothesizing that.

Speaker B: No, absolutely. And you, you, you talked about the very start, just to bring us back again ever slightly. You talked about. Well, I suppose there's two things here. One of the things is we've already talked about, you know, creating mystery shopping scenario that follows the customer journey, that has these really key objective kind of touch points that we want to measure in the middle. But you also talked about the fact that one of the things we pride ourselves on within IPSOS is how well those mystery shoppers are prepared for those assessments and how well they know those scenarios and the things to look for. What is it about that that's so important?

Speaker C: Yeah, I mean as I said, my view is that the job of the shoppers is to assess the employee and um, the operational processes around it. There's no other method that does that. Unless you can tell me one, I don't think there is. And that, and that's, and that's what sets, that's the usp, that's what sets mystery shopping apart. So it's a really important job for these shoppers and um, for any shoppers that listening what you do is absolute foundation of everything we do. We couldn't do our analytics without good data. So they have to be that real person. As I said, a lot of the time they are. And I think that really helps if they are, as they just are, a real customer. But they have to be because it helps them just come. They're just doing it as naturally as possible, but they are also observing very specific things. And the sooner they can kind of relay those observations in the survey after they finish, the better I think, because one again, One of the USPs is that this isn't recall. They are looking at it in the moment, which is, you know, voice of customer, customer service are really important. Of course you need to know what your customer thinks in their experience. But asking them an obvious, very objective question like, I don't know, how long did you queue for afterwards? They won't, it won't be accurate. Whereas a shopper will have timed that queue to perfection and we can say to clients, look, this is the, this is the, you need to get queues under X, Y, Z, um, to improve your commercial KPIs or whatever it is. And I think that's really uh, important.

Speaker B: So highly trained shoppers, these really kind of well understood scenarios and some of these measures requiring absolute accuracy. You know, this isn't about.

Speaker C: Sorry Rob, I was just going to say I'm backed up by, you know, photographic evidence or in some cases video recording that's as objective as you can possibly get.

Speaker B: Compare that to other methodologies or other things people might be used to in these same retail settings like customer satisfaction surveys that the general public might fill in and other aspects. It's not that there isn't a place for those, but this is a specific purpose that Mystery Shopping is able to deliver that sits outside of the remit of those other things. And I think that's, that's such an important aspect, isn't it? It so much of the time people might look at this data in combination or be comparing this data, but certainly what Mystery Shopping is doing is following a very distinct methodology for a very distinct purpose and delivering on a very specific mission.

Speaker C: So yeah, thank you.

Speaker B: That was, that was really, really useful.

Speaker C: Yeah. If we think about age, uh, verification and ID checks couldn't contact a customer after they've been, they've gone and bought, I don't know, alcohol or something and asked did you, was your ID check. I um, mean a, you'd have to find customers of a certain age where they're likely to get it might remember you have to do that objectively and without recall to be as accurate as possible.

Speaker B: Yeah, no, that, that's really helpful and really clear. And I know we've kind of touched on a few of these already because you've mentioned certain industries where it's um, well used or certain industries where there's engagement with clients at the moment. But thinking about some of these, do you have any, I don't know, favorite examples of where you found projects to work really well or been really impactful or I suppose the reverse where, where we've had to make adjustments to the way people might be assessing their processes. What are the standout stories for you when you look across Mystery Shopping at the moment?

Speaker C: I remember the first time that uh, we were given sales data from a client. I think it was sales conversion data. It was a luxury chain. And the first time we, we put that together with our Mystery Shopping data and you just saw a beautiful pattern. I almost describe it as a work of art, which is quite sad. And you could see that they're low performing stores on Mystery Shopping and actually seen their sales decline. The medium performing stores have seen a small growth and that top performing stores had seen a really nice growth of X percent or whatever it was. And we were just like that. That's what more do you need to, I mean there's a lot more to do after that, don't get me wrong. But what more do you need? And you can put nice, you know, pound, pound signs on that by saying, look, if all your stores had done this, we think the gap is £10 million or whatever it is. That's really nice. And the role of the employee is so important in that sales conversion. And that's why we see strong relationships at the other end. We're doing loads of work, oil and gas in the US at the moment. And the metrics that are influencing sales are just completely different. Obviously, it goes without saying, they're very infrastructure, they've driven, very process driven. You know, did you consistently have fuel available? It's pretty important. So, uh, and so the difference in the diversity in what we're measuring here, not just the sector, but actually in the metrics that matter, I think is just really interesting and shows the massive depth and spread of what, what we actually do in mystery shopping.

Speaker B: It must be such a delightful thing because, like you say, to have the confidence, you know, when you're talking about, you know, for a number of our clients, I would imagine large levels of investment, you know, to move some of these needles, to change some of these areas of performance, to have that confidence, you know, that statistical robustness that says, you know, based on your data, here we, here is where the investment might be needed, but here is the likely outcome if this lands well and can be achieved. So, you know, that's. That's fantastic.

Speaker C: Yeah, back to the earlier point, you know, a lot of the time we need quite a lot of data to, to have the confidence. We will only show data if we are sufficiently confident in it. And I think that's really for ipsos and integrity that is so important. And, um, on kind of commoditized markets, uh, where things like price and geography are important reasons for a customer to buy something. These operational processes, you know, they do shift the dial on sales, but you kind of need a lot of data to prove it confidently.

Speaker B: So for people who have listened to this podcast and who are interested and keen to find out more, knowing you as I do, I know one of the ways they can find out more is they can come to you. They can absolutely find Lenny. But what would you advise as a first step? If someone's listening to this and says, you know what, this is a conversation I'd love to have. This is something I'd love to find out more about. Where could they go?

Speaker C: Well, yeah, come directly to me if you're setting up a new program. Again, we would like a chat, but,

Speaker B: Andy, thank you so much. Really fascinating walkthrough. The importance of design and the impact of mystery shopping. I'm sure that have been a really interesting listen for our audience. I know I've certainly loved it. Thank you ever so much for sharing those stories and for taking the time.

Speaker C: Thank you, thank you,

Speaker A: thank you for listening to the Experience, Perspective and Ipsos podcast. Make sure to subscribe to us on all your favorite podcast apps to have new episodes sent straight to you as soon as they're published. We're also available on Spotify. If you'd like to reach out to us, please send an email to experienceperspectiveipsos.com.