7 Marketing Measurement Myths Costing Companies Millions

Speaker A: If you look at, like, who we are as humans, we were born to experiment.

Speaker B: Every few months, they would just cut all their spend down to zero.

Speaker A: It is impossible for a channel to not work. It is very possible for it to not work for you.

Speaker B: This is a limiting belief that holdouts is equal to incrementality testing, where in reality, you can test anything. Where the data tells a story that's gotta be told. It's the art and the science.

Speaker A: Get the numbers right.

Speaker B: All right, welcome, everybody, to another episode of brandformance. Reminder. We start a new series within brandformance, where Sundar. Hi. Sundar and I, uh, go over your favorite marketing science topics. So we kicked it off a couple of weeks ago with our first episode where I grilled Sundar on his background and his experience. But, uh, today we're going to do something interesting and different. I was thinking about sort of what topics would be fun for our audience to cover. And I thought there's a lot of misconceptions, myths, confusion in the space of marketing science. And so we could just take, you know, five or 10 interesting topics and break them down and just have a discussion about why those are sort of misconceptions and how to think about those topics from first principles. So there we have it. That's going to be today's conversation. Sundar, thanks for joining me. How's your day going?

Speaker A: It's going well. Thanks for having me. And I was, uh, I was super excited about this because when we briefly chatted about this, I realized a lot of these myths end up being limiting beliefs, and there's a blockers for why teams don't do things. So, like, busting these is actually more important than just like, oh, that's not what actually happens. It's like, hey, this is. This is the reason you're not doing something. Um, so that was just gonna be really cool.

Speaker B: All right. And, you know, this is also an interesting time. Like, this is 6am in the Bay Area in SF. So you see me in my hat because my hair is m. A mess, and the lighting's like. It's still not, like, fully bright. Sundar's in Amsterdam, right? Or are you traveling?

Speaker A: No, I'm in Amsterdam this week.

Speaker B: Yeah, so it's like 3pm or 2pm what's the like right now? Yeah, 3pm all right. Look at that. We're going global. Um, so it's going to. It's going to be a fun one. All right, let's kick it off. Sundar with the first one. And actually, you suggested this and we can maybe combine, you know, both of these, um, myths into sort of one bucket. So you said one of the biggest limiting beliefs is testing is for big companies with big budgets and testing requires complicated tools. Talk to me about why you think this, this is not true. And you know, I, I have lots of, uh, feelings about this as well. So tell me what you think about this.

Speaker A: Yeah, I'll start with. I think we've mixed like testing for like statistical rigor. And so statistical rigor is super, super important, don't get me wrong. And there's a time and place for different levels of it. For example, in a medical test, of course, like any clinical trials need to have the highest level. But if you look at like who we are as humans, we were born to experiment. We are actually incredibly natural experimenters. And so example I always have is, you know, our ancestors probably looked at a plant and we're like, well, let me, let me take a bite of that. Okay, well that didn't work out really well, so I'll stop doing that. I mean that was a sample size of 1 and, and you just never eat that fruit again. Right? Are we now going to go and be like, ah, you didn't have enough sample size. But like testing in itself, experimentation is actually a very human experience. And for me, that limiting belief of hey, you need a lot of budget or size is really not true. And it sort of ties in with this idea of you need complicated tools. And the best example I have is if you're a company, which is, I would argue almost 75 to 80% of companies probably have one main channel. Most of that time is probably meta. Right? You've got meta as your biggest key channel especially. And I come from a world of consumer tech, so that's, that's, you know, whether meta, Instagram, if you've got one channel and you just, if you want to go really extreme and just turn it off, like you have a pretty good experiment, like you don't need complicated tools to just be able to look at a trend line and be like, yep, this is when we turned it off, this is when it dropped. And that right by definition condition is a pre post experiment. Very unsophisticated by, by anyone out there. But I believe it's very, very accurate to the sense of like, is this giving you enough signal to make a decision on meta? Sure, you might make the decision of, hey, I'm going to pull back to 50% spend and the true answer could be like 42. But the fact that you went from 100% to 50% spend is the win. Not that you didn't get the 50 to 42, right. Like these little, like we, you know, data, science actually for me gets you that marginal benefit, but human intuition and a lot of it, like both you as a marketer and me as a marketing scientist have used intuition a lot more often than people give credit for. We kind of look at things and we spot patterns, we have gut feel. And so you don't need big budgets. And actually bigger budgets actually complicate things, right? If you've got a simple one or two channel, like you can make pretty easy decisions. And when you have that, when you only have a few channels and you've got this gut instinct, you then don't need complicated tools. I mean I, for the longest time in my career, even still to this day, just use SQL and Google sheets. Like, I'm a firm believer that that'll get you quite far. And if you're not comfortable using just those two tools like a SQL and Google sheets, then you're hiding behind complicated tools. There are of course companies that need bigger, more sophisticated answers, but they're again looking in these sort of nuances of like, where do exactly I put my allocation. But if you're an early stage company, like, don't let that be the reason you don't experiment.

Speaker B: I love it. No, I completely agree with you. I've, I have had so many conversations with marketers at, uh, smaller businesses who come to us for like, hey, could this work for our brand? And I give them just a very simple tip. I'm like, I was actually at a conference like a couple of months ago and, and this is a, uh, startup that sells into governments, like local governments, city governments. I'm like, you are the perfect candidate for doing testing, right? Because you sell in all 50 states, you're selling to all these counties. Go run an experiment just in these seven counties. And what that means is you're worried about like, you can't reach them. Like do local radio in those seven counties and see if you get more meetings with those, you know exactly where they are and, and there's no ambiguity in terms of, you know, what's possible with that testing. You don't need complicated tools. I was like, yeah, set up a spreadsheet and see what your trend line is and you know, run it, see what happens. So I completely agree with you. Here's a fun example. Even a large company, I followed this guy Brian, I'm forgetting his last name, but he used to be the CMO at True Classic, which is one of the darlings of, you know, D2C. And I think he was at a smaller brand called Nude before that. And what he mentioned was at True Classic, every few months they would just cut all their spend down to zero. Just cut it completely. He said it publicly on X and then they would build it back up slowly. And so that created this like, idea of zero based budgeting within marketing, right. Where you just always are paranoid about, are you overspending than you need to? And it just creates this, like, you know, culture and environment. So even if a True Classic can do this, even you can do this.

Speaker A: And yeah, not only can you do it, but I actually think what you just pointed out is the biggest thing that is a limit is again, this blocker limiting belief, which I'm going to reference throughout this episode, is that leader was willing to take that risk.

Speaker B: Yes.

Speaker A: And like, if you're willing to do that, that's Honestly, to me, 80% of the reason why companies don't get into experimentation. Because you're just not willing to go far enough and feel the pain deep enough or experiment hard enough to actually see signals. I mean, again, if you've got this belief that, let's say meta is so incremental but you're too scared to turn it off to prove it, that's a huge red flag to me in both leadership and testing culture. And also just the environment that you're in. Right. You should be in an environment that goes, yeah, let's like f that, let's go, let's try it. Um, and you, as a leader are responsible for creating that culture. Otherwise you can only imagine how many decisions trickle down to like, ooh, I don't want to do it because I'm scared I might lose my job. And then you've just got, you've got, you're just waiting for a company that's willing to take that risk to just beat you.

Speaker B: Yeah. Okay. So a related topic to this is the flip of this. Like, I was talking to a head of growth a couple of weeks ago and they were like, hey, I really don't think that incrementality testing is, you know, kind of aligned with scaling really fast. And I was like, huh? Tell me what you mean. Like, why is that a belief in your head? It's like, I just don't want to do holdouts. Like, we need to be investing more. And I'm like, okay, hold on to that thought. Who said that incrementality testing is just Holdouts, you can do the opposite if you are in a hyper aggressive go, go, go mode. Why not do a pulse up in the specific geographies that the model recommends? And a light bulb bulb went off for them. I was like, oh yeah, we did this launch like three weeks ago. I should have done that as a pulse up in like these key metros.

Speaker A: I'm like, yes.

Speaker B: So again, like, this is a limiting belief that holdouts is equal to incrementality testing. Where in reality you can test anything. You can test a level of budget, you can test different creative, a new channel, a new sort of attribution and bidding logic. Right. Have you seen this come across, uh, like, have you come across this as well?

Speaker A: I have. And I had this question. I don't remember where it came up, where someone was like, do you like, what's the difference between a B testing and incrementality testing? And I was like, oh no. The fact that you've asked that means you sort of have not, um, realized that incrementality testing is just a subset of a B. And. And like, I think there's this, whatever it is, a blocker that A and B have to be different, but A, all it's saying is there's an A and a B, and A and B are not the same. That's literally the only difference. So I have seen this, I have heard this. I think it's maybe because we use terms like incrementality, but we. When you get into like the next level and you have, you know, more nuanced conversations, a lot of people are talking about marginal, uh, incrementality versus channel incrementality versus maybe creative incrementality or channel. Whatever, whatever. There's usually a prefix to the word incrementality. So I have heard this. This is another really funny limiting belief. And you're right. Like, I don't know where it comes from. I haven't figured out because it's not like the companies that are doing or not doing holdouts are actually doing pulse ups either. You're just blanketly not doing any experimentation. You're just like, well, if I can't do the extreme, I can't do any of it. It's like, well, okay, that's not how you treat email, it's not how you treat product. For. I don't know why you think you have to do this marketing. I think part of it is because it feels like there's. The signals are a bit harder to capture on marketing. Sometimes, of course, a product you have first party data, et cetera. So yeah, I have heard this thing. This is, I'm so glad you brought this one up because I was like, wait, I don't, I don't know where it comes from, but it is effectively not true. I think you probably have to go a little bit more, let' say extreme in marketing where you're not just doing like a 1% pulse up like you, you know, probably gotta do at least a 10, 15, 20, like minimum kind of thing, but.

Speaker B: Exactly.

Speaker A: But the line items are bigger so like you kind of need to, to do that as well.

Speaker B: Okay, here's, here's another one. I get this question from prospects and customers very often where people will like, hey, is TikTok incremental? And I'm like, the question behind the question is like incrementality testing proves whether a channel works or it doesn't. And that itself is a limiting, uh, belief. And so tell me what you think and then I have a couple of questions like, follow up to that.

Speaker A: Yeah, so I mean, specifically, I guess the question would be like, is a channel just purely blanket incremental or not incremental or not?

Speaker B: Or like, hey, if an incrementality testing fails, right? Fails in the sense like there is no lift, you ran a holdout or you ran a pulse up, you didn't see lift, that means we're going to give up on the channel and move on to the next one, right?

Speaker A: Yeah, yeah, yeah. So this is a, this is a really good one because I get the, I get the almost like other end of the spectrum where people have just jumped to the conclusion. They're like, this channel doesn't work for us. And I'm like, let's unpack that. You're telling me, let's look at LinkedIn that has a billion users. TikTok, that has over a billion. And Facebook or uh, you know, meta has over a billion. Your target audience does not live on those three platforms. So as a digital company, are you going after people, uh, not on the Internet, like, help me rationalize and make sense of this. Right, so exactly what you've said. So if you start with the idea of the. It's showing that it's not incremental. I think there's a huge asterisk that you need to add that goes, our efforts on this channel are not incremental. Not that the channel is not incremental. It's our efforts. So whether I've seen, I mean, if you think about the number of variables that you kind of have to get right, which is why marketing is so beautiful and so hard. Obviously, the creative. But then you have to get the right audience that you target with that creative. Then you have to hit them with the right frequency, and then you have to run it long enough to be in a window where they're willing to convert. Uh, or you catch them in the moment where they're ready to buy. And this is excluding everything from macro stuff to competition, you know, so, like, it's. It is literally, it is impossible for a channel to not work. It is very possible for it to not work for you, but, like, you just haven't cracked it. Even a consumer tech company could crack LinkedIn if they did it.

Speaker B: Right.

Speaker A: Right. Which is like, well, LinkedIn is always B2B SaaS. I'm like, yeah, I'm not saying, you know, I'm not going to say it's super efficient. I'm not going to say it's the best. But, like, you could use.

Speaker B: I'll give you a great example.

Speaker A: Yeah, please.

Speaker B: Yeah, I saw. Okay, this is wild. I used to see Rolex ads on my LinkedIn feed all the time. Right. And I'm like, smart. Okay, that is very smart. Right. And so, like, I, you, uh, know, I'm not a. I'm not a watch guy, so it didn't work on me, but I'm pretty sure it worked on a lot of people.

Speaker A: Yeah, yeah. And this is like, you know, it's just. That's actually where the alpha is sometimes. You know, it's. It's like this, like, uncommon thinking. So, like, if a channel doesn't work, it's because you haven't found the alpha in it. Right? You haven't found the. The. So, man, I know we're kind of going through these rapid fire, but I'm just thinking about how fun and like, so fascinating some of these misconceptions are, because I hear these exact phrases and I know you have, which is why we're bringing these up. But like, that term, like, I was just, you know, the last company I was at was like, oh, meta didn't work for us. It was. I'm like, what do you. Like how.

Speaker B: How do you. How do you make that statement? Right?

Speaker A: How do you make that statement?

Speaker B: Yeah, uh, yeah, yeah, exactly. Okay. And I do want to go through them in like, rapid fire. And we can, like, you know, wrap up at the end as well because I think this is, like, really fun. Okay, so. Mmm. Um, marketing, mixed modeling and MTA multi touch attribution. They disagree because they Often will. That's fine. But they disagree because one of them is wrong. What do you think is behind that statement? Why do you think this is right? Wrong? Like, how do you view this, like, debate between mmm, M M and M mtn?

Speaker A: Yeah. Ooh. I mean, it goes down to, again, this. If you start from this assumption that like, they're supposed to be. What is it? Like, uh, I'm, I'm trying. I'm blank on the word, but like, they're like perfect solutions. I think there's this Starting assumption that MMMs and MTAs are perfect solutions, which is already, like, going to lead to. Okay, well then naturally, if I assume these are right, if I get conflicting signals, then one of them must be wrong. But, like, that's. They're not. They're not purely right. They're not purely wrong either. And I think that's, that's the starting point. If you understand, like. Well, they're not meant to be purely right. Like, they are models that, uh, by the way, you're building, like you are feeding the inputs into and you are configuring it and it's telling you based on what you've configured its version of reality. Right. And, and those signals are meant to be conflicting in some ways. Or if you think about an mta, well, first off, MTA is only purely tracking digital if you have any offline channels. So already you, like, you're comparing apples to oranges because one, an MMM does include offline channels and brand and other halo effect and MTA doesn't. So are you comparing apples to oranges now then, like, if you say these statements, it's like you, well, did you not understood what you were doing when you set up mta? Like, you basically took all, uh, the trackable touch points that you had and gave weights to them to see which one you felt had the most. And you're like, okay, given what I know, that's the best. And then an MMM comes and goes. Okay, well, I see some other stuff. And, and that's okay, but not because they're wrong, because they just have different views, they have different lenses. And the. Really. And this is the part where I think you and I always re. Go back to like, it truly is triangulation. You know, it's, it's mmm plus some sort of attribution model, plus incrementality testing. And if you don't have those three working together, your system, your marketing measurement system is wrong. But none of those individual pieces are necessarily wrong. They're not. They're not right. But their job isn't to be right. Their, their job is to provide more information for you to make smarter decisions, but in tandem. I think this is like the equivalent of you just saying, okay, well, I have attribution and I purely judged everything based on attrib and it wasn't right. Well, yeah, but it was never supposed to be right.

Speaker B: And so I think there's a nice analogy here. Like, if you create a, you know, MTA and MMM are both models. Right. So think about it from the perspective of if you have a model airplane and you have a real airplane. Right. A model airplane is not going to fly. It's not going to follow the same, you know, principles of aerodynamics and pressure and lift like a real plane does. And you can create a model airplane that flies, but then it's like a real airplane, it's not a model. Right. Like, so the idea is you're trying to create a model that in some ways reflects reality, but it's wrong in some fundamental ways because it doesn't have all the dynamics, it doesn't have all the variables. It's not all knowing because it's not. Especially in marketing, it's not possible to be all knowing. Right. You don't know what's inside somebody's head. Yeah.

Speaker A: Which I think is a good maybe starting point to like, get everyone aligned on, including marketers. But of course, like C Suite, like all, all levels. Like, you cannot be completely right about marketing. Like, like, it is just impossible. And ignoring, ignoring even just the, like, oh, we don't have the right data. If I had perfect data, I would still be somehow wrong. Like, it is just too complex of a beast to model human behavior to, to be able to predict it. So I think if we could somehow be like, okay, let's just be less wrong. Like, if we started to be like, that's our new motto is let's just learn to be a little less wrong instead of trying. Right. I feel like we would have a better understanding and sort of appreciation for, for marketing science. Because even myself as a marketing scientist, like, I'm still, like, I'm just trying to get us, uh, closer to less wrong. I'm not. Like, I cannot nail how right we are. Like, I can be more confident that we're less wrong. Yeah.

Speaker B: Totally aligned with you. Hey folks, thanks for listening to this podcast today. If you're enjoying the show and if you're getting value out of it, we'd really appreciate if you drop us a five star rating on your favorite podcasting. App. Okay, here's another one. And this is often sort of another pretty like big limiting belief that I see from customers. You must have a certain amount of data and variance and granularity for your MMMs to work, right? Somebody says five years, somebody says, oh no, like 10 years. Somebody says, oh no, we can do it with one year. So the assumption here is more data is always equal to better models. And my thinking on this is like, well, maybe if your entire business pivoted two years ago and you know, are you really going to go back to year three and look at the historical data? Maybe not, right? Tell me what you think about, you know, is more always better or is there a different way to think about what makes a good model?

Speaker A: Yeah, that's, uh, a. That's a really great question. I think, like, you start with, you have to start with like, what is good data? And then the truism is, yeah, well, more good data is better. But if you don't have a really clear sense of what is good data for not only an mmm. And first off, there is no like, set threshold or like number of years, a number of touch points. Like, sure, you can approximate. It's almost like the Central Limit theorem. Like, yeah, fine, a hundred points is better. But like a thousand points, sure. But like, if those a hundred are not representative of what you need to do, then it doesn't really matter how many of those a hundred are. But for an mmm, you have to start with what is good data. And I think you kind of touched on it. Like if you have five years of data but three years of not reflective of who you are. If you're all birds, for example, are you using data from four years ago? I don't think so. It's got to be one. It's got to be relevant. I would say sooner and closer to now is obviously better, but the biggest point is variance, and you briefly touched it. Variance and volatility is so much more important to the mmm. And this is where I think people don't realize, like, mmms work by just being like, wait, this went up a little bit or more here or there. And I saw this result. And so I think this happened because this happened. But if you have just a flat line in spend, it doesn't have anything to work with. It doesn't have a way to say, well, I don't, I, I can't assume it's because of this, right? So variance is more important and of course, like, relevance and closer to proximity to time is more important, but there Is no set. There's no set to me. Like I, you know, uh, we used to at least at Uber, feel like a year or two is good, but for us we would have daily touch points so we could actually have hourly touch points. Right. So, so it's not even like you can't even like a week for me is very different than a week for you. And so this is one of those where, um, I don't, I don't have a mental model. Do you like, do you have like a number that you're like, hey, just pick this number of good touch points? No.

Speaker B: What I tell prospects if you, if you ask like our internal data science and technical team, ML team, they will say, let's aim for at least two years of data.

Speaker A: Yeah.

Speaker B: And the reason that they're saying is that, you know, from a pure data science perspective, there are some limits to what is possible if you only have six months of data. Right. And the limits are mostly around how confident can you be in the outcomes

Speaker A: from the models, especially around seasonality. Like baking in seasonality too.

Speaker B: Yeah, yeah, exactly. Right. So two years gets you at least two seasons. So there's something there. But then when I'm talking to a prospect and the prospect's like, hey, actually I'm fine with a, uh, lower confidence answer. I've gone from 0 to 2 million a month in spending over the last six months hyper growth. And I just want to know if I'm doing okay and I'm okay with a wider, you know, uncertainty. And at an aggregate level I know the answer that yes, we're doing okay because I can see the business results, but I'm trying to now get smarter and smarter and I recognize that six months is going to be limiting, but I rather start now than wait for another six months and do things by pure gut intuition and this data will be helpful to me. So that's a very nuanced answer.

Speaker A: Right?

Speaker B: Like yes, six months is not ideal. We don't recommend it. The customer understands it. But they also understand that what is their alternative? The alternative is to keep using their touch based attribution in platform reporting, which seems limiting and wrong to them because they're like, yeah, that's not working for us. So you have to like be practical in how you think about these things. And then obviously, you know, bias a lot on actually running experiments and what have you.

Speaker A: Yeah. But I also love this idea that they're now, even in this example, they're now setting a baseline. Right. So if you uh, just imagine this scenario you do in a year when you feel like you have more complete data, well, then you're like, well, this is the first time I've seen an mmm M. How do I process it? How do I share the results? How do I talk to my CFO and my CEO? Uh, but if you do this now, six months before and zero to 2 million is like. Is pretty great in terms of like, signal, uh, volatility. There's more to an MMM than just the model. It's everything after also the model comes out is where the real fun in games begin. So I even would advocate for like, err on the side of what you're saying. Like less confidence. But like, if, if yes, two years is great, but try it at a year because you also have to see how receptive your company is to an mmm. And I think a lot of people forget, like, yeah, there's like, you, you can be excited as you are, but there's going to be like a lot of detractors and there's a culture and a muscle that you have to build. So even in this ex data is better. But more reps of running an MMM is actually pretty useful too.

Speaker B: Indeed. Indeed. Okay, here's another one. MMM tells you where to spend your next dollar. Like you could you. You have a really accurate model and it will, you know, tell you exactly where to spend your money and you're going to make lots of money as a result. What do you think about that?

Speaker A: Yeah, well, when you, when you anchor to my. My previous statement, then my goal as a marketing science is just to be less wrong. I think my answer is probably pretty obvious, which is it's not like, first off, you can't achieve 100% confidence. I do think you can use MMM to forecast, which is what we had done. But it. A lot of it comes down to how much do you assume the forecasted period is reflective of the current period. And that's kind of the hardest part to forecast. Right. Is just the unknowns. Right. If you knew I'm gonna. If you're a pair of Nike and you're like, I know I'm gonna produce this much. Okay, great. Well, did you also factor in Iran blocking a lot of your shipping lanes? Probably not. So that's the kind of stuff that gets people in trouble and they never think about that. So to answer your question, I do think an MMM is capable. I think there's actually probably a validity to using it to forecast. But you have to be very like, you have to list all of the assumptions you Made really well. And documentation around forecasting is where a lot of people get in trouble. And so to your point, you know, maybe to your question, sorry, that's like you can't know how much your competition increased their bidding, uh, as much like maybe you've got a really good like tracker, but like, fine, but you can't really track like sentiment that well. And so if you're doing it within like the next, again, going back to consumer tech might be a little different for B2B. But if you're in consumer tech and you're doing for the next month, sure. If you're doing it for three months, fine. If you're doing it for six, I'm like, dude, just like it's, it's not even like we're going to reforecast in two months anyway. So you can use it. Be smart.

Speaker B: Yeah, no, I completely agree. Right. I think of it as a forecasting tool, not a prediction tool.

Speaker A: Yes.

Speaker B: And I think the difference there is really important. Right. Where people are like, oh, like if the MMM is telling me I should spend all my money on X, Y and Z, I'm just going to go do that. And, and then if, if, if the forecasting thing didn't happen, then you look back like, mm, must be wrong. Like yeah, uh, no, no, no, no, no, you're missing the point.

Speaker A: Right.

Speaker B: And so forecasting and predictions are completely different games. If it was a prediction machine and you could predict what's going to happen in the next two months, well, guess what? We'd all be, you know, at the beach sipping pina coladas because we would have, you know, invented a time machine and figured out how to make ourselves rich. That is not what MLM is meant for. It's supposed to give you like reasonable forecasts with the, with the right assumptions. And what's really important, like you said, is the forecast period behaves very similarly to the last two months, three months of current behavior. And what people also don't realize is when you are in a very stable, mature environment, that can actually happen. When you're actually in a high growth environment and the category is going through massive disruption, this is actually not going to work as well. So you have to understand what company you are and what category you are in and what are the dynamics before you lean too much into, you know, that forecasting mechanism, um, which breaks people's brains.

Speaker A: Right.

Speaker B: It's like, that's just how it works, man. Like go do some day trading and you'll figure this out.

Speaker A: Well, you actually also brought Up a really good point which is going back to like, your thoughts on like pulse up testing. This is a great example. If your forecast tells you something, well, this is a great opportunity to layer in this like, system and muscle of like, all right, well I'm going to go pulse up, see what happens, measure that, feed the results and instead of building a six month forecast or prediction, you're better off building three two month ones with like a pulse up. Right? So like, like this is where I think people don't realize, like it's truly is a system like testing and be becoming a profitable marketing engine is a system and it is a system that has to work in, in coordination over a very long time. And so like that's part of it. Like you know, you can get in if you get in trouble by just putting all of your eggs in one basket. Well like there's a reason, there's that phrase like just, just put in two and then see how it works, then put in a few more and then you get to get to where you want to go. I think actually quicker than like you said, I put all of it in one basket. It didn't work. Now I reset and then I'm like, I, I have to rebuild. M my entire company's faith in an mmm. Um, like that takes like change management is, is the hardest part of, of marketing signs and those changes and those transition periods take a lot longer than your uh, forecasting ability.

Speaker B: Yeah, yeah. I mean change management is the hardest, hardest thing and we see that with our customers all the time. Okay, another one here. Once you've established or set up your MMM and you've done a whole bunch of holdouts and incrementality tests, right, you've got your measurement, you're done. Now you can move on and just execute. And you've got your incrementality factors, you've got your mmm, um, you know, coefficients. Just, just go execute. Don't worry about measurement now you've got that in the bank. Go execute.

Speaker A: Well, clearly we've, we, we've labeled it as a misconception. So already right off the bat we should start with yeah, that's not true. And uh, I think the more important point is why it's a misconception. And if I could give advice to marketers on like just one like shape in the world that they should truly master. It's like a curve. Like the entire field of marketing, I think, runs on curves. And like that's, that's it. If you could understand how curves interact with each other throughout time. You, you've, you've become a master of marketing. And, and the biggest reason why these things don't hold is, is simply like when you go from one level of spend to another, just, just keeping everything else constant. Let's assume the world doesn't change, your customers don't change, nothing else changes. Just going from one level spend to another. You've now changed where in the efficiency curve you are and you're always going to go to less efficient part of the curve. It's just the natural law of marketing. So you can't just run an incrementality test at one period and then assume it's going to be fixed because incrementality at 5 million in spend is not the same at incrementality and 10 million in spend and that's just holding everything in constant. So kind of like you said, assume competition, product mix, all of that. And I would say the best, the best companies probably run incrementality tests, I would say once every six months and like recalibrate their mmms at least once a quarter, probably sometimes a little bit more frequently. But like that's the cadence. Like if you sign up for, for experimentation and in this sort of program you're signing up for life. And it's why like I think that the, the analogy of building a muscle is the most apartment because it's not just about consistency, but it's consistently also readapting and changing. Like uh, if you're, if you work out and you go to the gym, you've heard of plateauing. Like you can't just do the same thing for three to four months because eventually your body adjusts. And the same thing with a business, the same with the marketing. What was incremental six months ago, a tactic which was amazing is probably arbitraged away by now where everyone is doing it. And so you've now lost all of that incrementality. And so you've got to readjust and you got to refind a new, a new winning tactic and a new creative. It's why creative fatigue is a, is such an important thing, right? And especially as Meta and all the other channels invest more and you have to be so good at creatives, your creative literally could, could not be as good in like a week or two. So any incrementality you've measured now has to be kind of thrown out and re. Rethought of. And so that's a huge, a hugely dangerous misconception that you've thrown out and that one's not a limiting belief, luckily, but it is a huge assumption that companies make which is, you know, it's, it's a one and done test. It's a, it's a constant thing. And yes, I think like it's exhausting. Be like, well great, now I have to run another holdout. But like that's part of it. And I think, you know, once you get into that muscle, like you kind of enjoy doing it. Like you create this like program behind it. But like that's the only way to survive is enjoying it and creating a system around it. Or else to your point, like, or to your, to your misconception, it's not a one and done. Please don't do one and done. Like you're going to check back on it a year later and be like, oh man, I just lost a year's worth of progress baking in an assumption that I had from a year ago.

Speaker B: You gave me uh, an idea right now which is I think every scaled marketing team should have an incrementality manager. Yeah.

Speaker A: Um, a program manager test.

Speaker B: Yeah, that's it. They are literally the, you know, the. And I don't mean like it needs to be siloed, but just like you have a coach, right? For your team, there's an incrementality coach that's sitting there and just be like, hey, like when was the last time you exercised this muscle? Right? Like let's go, let's go put in the reps so we can play the game after that.

Speaker A: Right?

Speaker B: Like there's, there's gotta be some, something like that. That would be, that'd be cool. Okay, last one. And this is not marketing science specific, but I hear this all the time from brand marketers and sometimes like you know, CMOs as well. Brand compounds.

Speaker A: Mhm.

Speaker B: And my take on this is actually brand does not compound, it actually decays. So what I mean by this is you can't just do a whole bunch of brand marketing and expect that somehow it'll just compound in people's brains for the rest of their life. It'll actually decay just as fast as you built it up. And so you have to keep investing. It's not like typical compound savings.

Speaker A: Mhm.

Speaker B: Right. It's actually more like a garden where you're constantly watering and if you don't water, your plant's gonna die. Yeah. I'm curious what you think about that or you know, what's, what's your take on like brand compounds?

Speaker A: A hundred percent. And I think this is again like so One of the things I've been, I m might go on a little sidetracked just for this. I've been, I'm going to start pushing more for we need to stop treating brand as a vitamin and more as a painkiller. And I think if you did that, it like reframes your whole mind. I always hear, oh, we'll get more volumes, we'll get more volumes. And I've been saying, I'm going to start saying this more recently. Okay, if you go to your CFO and you're going to get so much more volumes. Well, what's their biggest first concern? Okay, where cacs are going to skyrocket. Brand actually prevents skyrocketing cacs with higher volumes. It's a painkiller. And the reason I bring that up is it compounds in the short term, but very quickly decays. I mean the half life of brand is absurd. Like, and I have a great example for this. We ran a campaign in the UK and we had this like really simple setup. And this is what I go back to. Like, even at Uber, like, we just ran simple setups and we ran, um, brand lift studies on Facebook. We ran one before the campaign, I think couple during and then one after. And we were about to run the campaign and then midway through the campaign we had to turn it off. Pre campaign. Let's assume our brand awareness is 50 during the campaign. It goes to 60, 65. We turn it off because we have to shut off the budget per reason. And then we measure it after within like two weeks of us turning off the campaign or maybe two to four weeks, that brand awareness is back to 50. It's like the campaign did not happen. Just did not happen. And that's when like we knew this. I think a lot of the brand marketers I work with at Uber are super smart and had come from places but they never had to like measure it. And like an Uber type setting, they just sort of were at, uh, places that just assumed a brand work. They were like, oh crap, we've got to be always on. Like, that is default. So this is where brand decays for sure. But brand also compounds where if you do show up consistently, you do have to keep adding more. You can't just stay at the same levels because that doesn't that, you know, again, you're offsetting decay. But it does compound in the fact that like, if you stay consistent and if you keep going, what happens is the speed at which I think people, uh, have make mental associations with your brand become faster and that compounding is real. Right? Like, again, like, uh, fine, I could. I know Nike's an extreme example and maybe not the best time in Nike's period to use it, but for 30 years, they were synonymous with, like, the sports brand. And, like, you could even be like. And everyone would be like, Nike. Right? So, like, that's built up over compounding. But you see it now. They've stopped. It's been a few years, and it's decayed quite quickly. And so it is compounding. It is decaying. It is both. But to your point, it's not just. It's not like a positively compounding thing forever. Like, you do have to water. I think the gardening analogy is really great. But, like, another example of gardening, though, is like, you get better at gardening the more you do it and you can.

Speaker B: I like it.

Speaker A: You know, and so, like, that's where of the compounding happens. Like, you layer on and you're getting better at branding. Blah, blah, blah. So. So it does compound and it definitely decays.

Speaker B: There we go. All right. It was. It was a, uh. I wouldn't say that it was a diplomatic answer. I think you answered that very well. So love it. Um, fantastic. That brings us to all the topics that I had in mind. Did we miss anything? Do you think there's another, like, big limiting belief or misconception or myth that we should bust?

Speaker A: Well, I know we focus a little bit more on just incrementality testing. M. Mmm. I do have a lot of in my head around you can't prove the ROI brand. There's some misconceptions around what you can and cannot prove that I would love to touch on at some point. Yeah, let me get back to you because I think there are probably some that we could dig up, but at a high level, I think this is. We got most of them.

Speaker B: Amazing. No, I like the idea of talking about brand and brand roi. Maybe we'll pick that up in the next one and demystify the whole idea of brand marketing. And do we think about that differently from performance marketing and all that fun stuff? Like, no better podcast than brandformance to talk about that. So tune in, um, next week. So we're going to start doing these more frequently, but in two weeks time, we'll do one on brand and performance and get into the nitty gritty of, um, ROI and how to think about those two domains. Sundar, always love chatting with you. Thanks for joining me. And, um, thanks everybody for listening in.

Speaker A: Yeah, thanks, everybody. See you next time.