The Future of Marketing Measurement Is Causal | Talgat Mussin

Talgat Mussin: I noticed that the main blocker of adapting this, it's not math, not tools, not uh, any other rational reasons, more like a perception beliefs and philosophical reasons. So decision makers, most of them don't understand how all this works. So that will be my primary goal in my new venture Communality Insider where I help decision makers to make sense of it, simplify and eventually when they tackle this they will win in business because there's so much money being wasted.

Paul Kovalski: Talget, welcome to the show. Hi.

Talgat Mussin: Thank uh you for having me.

Paul Kovalski: I'm excited to chat all things incrementality, measurement and budget optimization with you today. You're a uh leading expert in the space. To kick things off I wanted to contextualize your experience for folks. Could you just give a quick introduction into who you are and what your career journey has looked like from a uh incrementality and budget optimization perspective?

Talgat Mussin: Sure. I think my background is I'll say unconventional in terms that I didn't have a prior knowledge and expertise and sort of like build up I need to deal with. So I immigrated to US back in 2015 and by random chance of luck landed at Google as a contractor in 2017. And straight in the heart of the area of experimentation uh incrementality, causal measurement. I started in a GTM role but slowly start actually uncovering the passion to it and uh how fascinating this topic and how important it is in decision making not only related to marketing budgets overall I think essence of any effective decision making comes back to causality like cause and effect. That's what we actually trying to solve. Right. With different means, different words but essentially uh, you need to find out what works for your objectives and this actually uh can be solved with a causal measurement approach. So most of it was first introduction by Google of uh geo experimentation as a framework. The new approach to measurement first paper came out not first but influential one widely adopted called GBR Geobase regression ah very I would say complicated most of the day to day market here like it's a uh, uh multilinear regression and it's very complex to implement that has also has its own drawbacks. I'm not going to go in the details of it but the next paper came out time based regression which is much simpler using the Bayesian framework. And what helped me that within the Google I was on a part of the as a measurement lead marketing science helping advertisers to measure effectively and scientifically the investment in Google media channels and what several years I spent only on that and that's how it's become my thing and just summarizing the experience that I run probably more than hundreds of studies different shape and forms educated teams and I was in charge for the project Project Lift. It's a cross function initiative across internal Google teams helping analytical leads to learn and help advertisers to properly measure incrementality and just skipping a few steps. After Google I moved to Amazon DSP as a research scientist Even though I don't have any background, I don't even have a US education it's just again serious of luck connection and uh, uh actually being passionate about it helped me to get there so work there. It was uh also interesting experience exposed to programmatic world very confusing, very I would say blotted and very very complex world of open Internet on programmatic and after that moved to TikTok where I was more on the PMM side and helping the internally coordinate work between measurement teams and product teams and all in all came to the point to last year where actually I felt that I know enough to do on my own I don't need any corporate structures for it and also most of my let's say this is my thing all the way I did it for the nine years and I'm planning to do it for the rest of my life because so much can be done and there's so many gaps and so many uh areas which not being properly explained understood through different reasons so and I noticed that the main blocker of adapting this it's not math, not tools, not uh, uh any other rational reason it's more like a perception beliefs and philosophical reasons. So decision makers most of them don't understand how all this works. So that will be my primary goal in my new venture Incrementality Insider where I help decision makers to make sense of it, simplify and eventually when they tackle this they will win in business because there's so much money being wasted. So this is my intro.

Paul Kovalski: That's awesome. And I think I share a similar obsession with you behind understanding what's driving results, what's driving performance because when you run a multi channel media mix at scale you know that there is spend getting wasted basically at all times and identifying it and finding the areas to scale, finding the efficient spend is super rewarding and gratifying. Why do you think that so many advertisers and marketing leaders still struggle with this even today?

Talgat Mussin: I think there is one phenomena like a problem industry wide problem I was trying to like I tried to internalize and synthesize and put the structure and Zoom out and come up with some frameworks. Because what's happening nowadays there's a different platforms, the mini universe. They have their own incentive and agenda and the whole area of advertisers they are uh, confused because there's so many misleading. And what happened that back in the days when the first introduction of the personal computer and then smartphone it was a relatively simple world where attribution was sol attribution being used as the representation of the truth. So most of attribution dashboard perceived as a performance. And then over time things got really complex. There are so many devices, so many like most versus of the walled garden. And as a user where you come in and out and nobody, absolutely nobody, even Google have a full visibility of the users across whole Internet. And it creates this huge silent world of the mini universes like universe of universes. And then you come in and out. And back in the day when everything was simple, the promise of the one user, one device identification et cetera was easy. And then drastically the cost of the measurement M because measurement is expensive. But since there is a rise of performance marketing which kind of pushed back the brand marketing, that's a different story. And there's so many problems. But long story short, experts marketeers in the field over the years working on this, they didn't adapt to the new world at scale. There is some of course adapting, but majority still thinking the attribution it is representation of the reality. But what ends up happening? There's so many mini universes, so many like a uh, book of records databases. And the problem came out that the reconciliation they're trying to get a sense of it. And then that's why with iOS release of the privacy and overall trend of the privacy of the users across the board led to the situation that okay, this is not reliable, this is not working. But not all the company moved away from that conceptual understanding that this is not a representation of the reality. So that's why the holistic approaches as MMM and geo experimentation which also have more like holistic approach because you using the as a KPI top line not attributed through the filter of someone's logic of attribution. That's why like there is a huge huge misconception, misunderstanding. And the main problem there is there's no investment behind it to address this issue at scale. There uh, is a rise of the measurement size. And I see that renaissance of MMM faster, cheaper. But that's kind of tooling side that's great. But without addressing the overall conceptual understanding I see the like I know A lot of companies and I have uh, friends and colleagues like huge network in the area, they have a hard time to adapt it because fundamentally it needs to be new like a doctrine, new thinking should came in and it should come and then uh, explain it. Look, this is a different world and whatever you did for the last decade, you have some built up knowledge and filter, it's not relevant anymore. And it's really hard for people in Eastern uh to admit because that's become part of their belief and part of the identity and they cannot move away easily from that saying like oh like all those years I was doing it wrong. So that's the biggest problem. And for decision makers which will be my primary focus, they confused even further, like even more because there's so many conflicting truths, there's so many like sources uh, of truth. And they come so confused. They have a different day to day like a objective for the brain space. Like they dealing with the funding, the burn rate, hiring and then this coming this like they don't have a brain capacity to like properly uh, understand and they, they make a big mistake. They outsource a judgment. They I like to give this example like all this high stakes conversations happening, either measurement vendor or like test results. And my man comes into the table and they say okay, what's going on there some presenting and another problem that our colleagues from the field they over complicated most of like too much. You know, like they come into the state decision making and it starts saying like confident intervals, P values this and that. There's so many unnecessary complexity of the warning and decision makers they don't get it and they try to out of the judgment they call in the like a uh, monsterless data scientist. Can you explain this to me like what's going on? Is it like I trust you in our team, can you tell me yes or no, good or bad, how bad it is? And those folks usually they focused on the different things and then just thrown at them and they're like oh, it's a second uh priority and they don't have expertise and they doing something quick to appease the stakeholders and what quality is not there. So long story short, lots of problems, huge vacuum of the expertise and big misunderstanding that worlds change so much. Attribution is not equal to true performance.

Paul Kovalski: Right. And of course as a founder or CEO, um, you are solving multiple problems. This might be on the lower end of your list. And so the approach of course is to outsource it in a lot of ways. Similarly with AI, a lot of CEOs have extreme FOMO. And so they say I want to make sure I'm doing everything right. I don't want to be missing out on this, on this big trend. I want to get into your approach into measurement and incrementality testing. But I believe that there's still a delta between measuring media effectiveness and bidding and optimizing media. Because if there was no concept of bidding and optimizing towards a specific event, it would be a lot easier for us to just say let's have a media mix model, let's do incrementality testing. That's going to be our approach for optimizing our mix. The reason why we still have last click attribution and we still have deterministic and probabilistic attribution is because we need an event to send to the ad platforms to optimize towards. And so you have all these different uh, forms of marketing measurement. At a high level incrementality are kind of a way to experiment. But you still need data to flow into the ad networks for them for their platforms to know what to go after. So how do you make sense or how do you think about that kind of dichotomy?

Talgat Mussin: Really good question. And I think but it implies that platforms when you share the conversion signal optimizing for incremental users. Which is not always the truth because my understanding on high level. So each platform either over attributing or under attributing and depending on the scale is different and ultimate goal should be when you combine all those claimed reporting of attribution it should match to your actual sales of your CRM whatever. But actually cash flow. So, so but it's not, you know that for sure that it's not happening. Uh, there is a lot of like double counting inflated metrics et cetera. So some platforms you cannot treat equally in terms of the attribution most of attribution aggressiveness of the getting uh, claiming the credit. So some of them you feed them more data, they conveniently allocate it close to the final decision of the checkout and objective is not change the behavior of the user but just conveniently piggyback when the user already heading to convert. And that's the problem of the causality because again there's deviation big deviation happened wanting the media to change the user behavior to caused uh desired action or solving for attribution because attribution it's a different concept. It's like the purpose of attribution and where they overlap back in let's say 90s beginning 2000 where attribution was solving the causality and displacing it. And that's like reliable enough. But it's not the case anymore and I'm not sure if all the platforms perform better if you upload your conversion data. This should be I think carefully evaluated case by case because in some cases it hurts your business. Uh uh, and again this is very complicated topic and I cannot say universal answer, but I've seen cases where business decided not to share any conversion data at all. They just optimize for reaching frequency and just solving for each and then they see through incrementality studies that just doing the broader reach, giving the more incremental users again case by case.

Paul Kovalski: Of course. Yeah, it's something that I think about and play around with a lot because when you are bidding towards a specific conversion event, the ad platform will show impressions to those that are converting. But if you have a strong natural conversion behavior, those ads are being served to people that would convert anyway. And if you optimize away from I need to get as many purchases as possible to something else that's kind of a proxy for showing impressions to people that you might, that might not be in your I'm about to convert pool that is actually an incremental conversion and I'm exploring this all the time.

Talgat Mussin: I can share uh, quickly just to close this loop, I can share a really interesting case where I uncovered interesting phenomena. So I think what happened in meta in February, the huge algorithm change, I dig deeper for that uh, into this case and I found the two concept which is actually hurting some advertisers, lots of other getting hurt by this new update and I think Meta, uh, any decision maker who looking for the agency, they need to clearly divide their expertise and cases and then develop most ads before and after 2025 February this new update changed the game completely. And what's happened that maybe you notice as a user itself that when you as a user see the ad on Instagram and then you engage or watch or even click, most of it triggers the whole chains of event of like it's called ads clustering. So now you see a lot of ads in the same category. Like a lot of them. Like before you didn't show the interest intent signal. Now now you're getting bombarded with any type of product and service, you just show little interest. That's it, you being bombarded and you get exposed to a lot of their competitors with the same category which you didn't exist before or you didn't know existed. And the way how it's feeding the really advanced algorithm which is the social game changed completely. It was social graph. There was a knowledge graph, interest graph. So you in organic content. When you engage in certain topics then very fancy machine learning AI algorithm start to match this behavior to certain advertisers. And then it's all feeding this loop. And what happened? Eventually there is concept called scouting scouting and then piggybacking. So the advertisers in category who spend the most it end up educating the whole cost of user base. And then others who piggybacking on them they get a free free right and exposure. So what's happening? I I saw the huge case that advertisers spend most educating the whole category about the whole category of the whole huge pool of users. Because algorithm finding very aggressively the users who interested in this topic. And when you add a very generic and uncovering the whole category like you don't as a user you don't know this existed this offering product or services. Now you show the interest the scouts those who found you paid the most. But the whole category like a riding behind of it. And what's ended up happening the top spender waste most of the money but the whole category of smaller guys is rising and uh. Their awareness rising and their sales rising because the one guy being the fool overspending. So that's a different topic. But yeah, I'll share more in my resource.

Paul Kovalski: Talget, what is your framework or philosophy for uncovering and improving wasted and inefficient ad spend?

Talgat Mussin: Uh, I developed several like my own frameworks and I was really busy all these years implementing. But now I have a time to put it in the like a concept and structure and playbooks which will be in my resource. And by the way I decided not to do the newsletter. There's so many newsletters and world doesn't need another newsletter. So I I going to. I myself like like I signed up for so many of them and I'm not even offering some of them even you know world is very noisy. There's so much to like FOMO to read. But I decided to make a resource which is not a newsletter. It'll be, I call it incrementality Intelligent desk. So it will be the resource for decision makers marketeers to come in and there will be clear simple playbooks and guides to implement and follow step by step. So action. No, no water. Uh, because I understand that when you as a solopreneur for example you're trying to figure out what you value like of course uh, newsletter is good stuff. But for me it's all clear. I don't need to like figure uh out my value. I know what I'm good at and I need to just focus on the pure delivery. And then the most important part that is to change the mindset. That's why I'm addressing this and putting lots of effort to educate decision makers that it starts from the beliefs and it starts with the culture. Because no matter what you do with the tools, if incentive is not there that you probably aware that different stakeholders may have different true intentions and true objective. They don't share it publicly but internally uh, it comes to politics, it comes to agency who get a cut from the media budgets they in charge of and they're not interested for the efficiency. They're interested for continue increasing the budget because they get a cut. So that's like misalignment of incentives. That's the key. And I think the decision makers they're not aware how much money being wasted and how it can be prevented. And there's ways to do it like uh, I'm gonna share the seven day sprint, like how to do certain steps to stop the waste. Like go in the platform, turn off this stupid things which not supposed to be there. There are so many buttons and knobs and every platform is so confusing and then beliefs perception. And this is where I'm introducing the, which I was able to put in the thoughts like what's the path of the getting better at measurement? It's not like everybody says we need to get better at measurement. What does it mean? And I see from my experience from both sides there is a path and there is a levels to it. So I call it measurement elevation atlas. And the level one is the. I call them sleepwalkers. It's not their fault for the reason I described that before because of buildup and misconception and all this huge uh, gap in the expertise. I see that 70, 80% of the advertisers they're in that stage of sleepwalking. They're not aware they're wasting money because they accepting the different truths of the dashboards. The next Level I Level 2, it's Awakening. They start questioning like okay, how, how do we know is it true? Like we need to test this, we need to like uh, get to the actual hard evidence and we need to make have the courage to admit like what we don't understand and what we understand what works. And that is like a. But it's a bigger struggle. But in the same Time biggest efficiency gains happening there. The next level it's enlightened ah this already kind of figured this out. Now they addressing the waste and after when you clean up your system your system is like most likely bloated between 30 to 40% of your budget wasted. And once you clean up the system then you have a better like a signals. You uh reduce the noise and you have a better sensitivity for signal. And then you will find the nonlinear growth opportunity on channels. And as you know the growth is combination of factors. Different proportion of spectrum, different channels can give you the best ROI possible. But when you have a lot of bloat in it it's hard to detect. Even uh. As an MMM practitioner they know that all those evaluation metric MMM will be much much better when you clean up your system of the non incremental spread um MAPE and then R squared and all this like it will show. Okay now you you have a better prediction higher accuracy ah and most for enlightened and I I see that good trends companies. There is an organization level, there's a personal level. They're moving from the stages and next one they call it omniscient. This is like guys like a players killers. They like figure this out. You cannot bullshit them with it with the. With some narrative propaganda of the other like most from other stakeholders uh or other parties. So they know what works. They test it. They aggressively testing always test everything. And they see the test and learn. It's become their like a default setting. And on a reservation level on uh the personal level those experts exist. I see them that it's growing the population of them who they come into the company whatever they touch. They like super efficient. They cut the waste fast and then they help company to grow. Actually this is uh one of the most let's say growth unlocker as a competitive edge nowadays having someone like this in your site figuring this out and internally also educating and change the culture. But closing my monologue on the organizational side company who figured this out and you probably are uh aware that there's most players they figure out to the point where they this is such a lucrative business advertising this works so well. Why don't we do it ourselves. And they become advertising platforms themselves. And you see the rise of retail media network and thus Uber's Netflix bookings like Expedia. Like so many of them now doing their own ad network because it's such a such a lucrative business because the value creation is that so many the whole like a business community confused how to measure right that that's why the value is there. So I'll share more in my resource. But this is how I see and the frameworks depending on the case closing on the frameworks I'll say typical answer. It depends if uh. As a practitioner I try to do less of it. But sometimes I will do as a premium offering. Depending on the appetite, how fast you want to move, how aggressive you want to be. Because it require courage, it require moving fast and stopping everything what you do and addressing this issue. I noticed already that not all the organization are ready for it. It's too fast, too aggressive to update the beliefs. And that's why you need to start maybe slower. Depending on appetite you can start. Okay, let's do mmm. Okay. We see here and there uh MMM have a lot of multicolinearity and other issue. Let's do calibration with experiments. So I'll say that most popular approach nowadays start with MMM calibrated incrementality and get better at it. But there is a shortcut fastest way when most of just do this multi cell experiment aggressively and within a short period of time. You tackle this really fast within uh 2 months. But not all the companies are ready for it. That's why it's going to stretch for the like two years. So did the best.

Paul Kovalski: Yeah. I mean when you talk about organizational culture change, you're talking about behavioral change and that takes time and based on the level of the organization, size of the organization, the stage, it can take a very long time. We have about five minutes left. I want to move to rapid fire around a couple of kind of quick questions with what your hot takes are. Um, I'd love to hear about it. My first question. Everyone's talking about AI right now and we'll continue to talk about AI. Where does AI and incrementality collide and how should we be thinking about the relationship between those two things?

Talgat Mussin: Very interesting complex question. Because in my personal opinion before tackling AI this should come first understand efficiency and where you're actually wasting money before jumping in the form of AI. What AI practically can help is the educating and then automating and uh streamlining this manual tasks related to testing and comes out in mmm. There's a lot of operational efficiency can be accelerated with AI. But the critical part never ever in marketing outsource judgment to the AI. It should be like because AI is the trained on the past knowledge and past knowledge as we all know uh regarding the measurement it's not relevant anymore. So uh. The training set feed with the incorrect Non relevant information if you want to apply to the business AI cannot going to solve this. And there are so many issues where causality incrementality not properly addressed in any AI. And I personally tested all those LLMs with the hard question they come up with something like which will be which of course it was relevant like 10 years ago, 5 years ago, but not now. So long story short when they collide operational efficiency education like building some training programs for the team, changing the culture and like some interactive models testing I myself playing around for this with the educational part. But never ever I'll never recommend to trust in AI for the high stake decision making and the measurement because it's completely wrong.

Paul Kovalski: Regardless what's common experiment design mistake that you see even advanced teams making doing

Talgat Mussin: something for the sake of doing without proper hypothesis. Like everything should start with a simple hypothesis of the business objective. And the biggest unlock. Um when I was doing the workshop by the way I'm also offering those before tooling, before all those design. First you need to get clear on the business objective and then how translate in the measurable hypothesis. This is the biggest step of the moment like okay, is it measurable? Is it testable? Is it feasibility? Is there before jumping right away to testing you need to build this connection and actually those professionals like omniscient or enlightened they develop this intuition of that feasibility even before testing. Even before test they can play around with observational data and see is it possible to test or not Playing around with this noise and signals ratio. So that's sort of like an approach from Art of War Sun Tzu. I don't know if you heard about this like winning the war before winning the battle. Before the battle you need to be more certain than running blind and doing something. So the key is the biggest mistake I see how to translate business objectives to the testable measurable hypothesis which you have a data supporting that is measurable. Because I see some really stupid mistakes when they jumping ahead of the testing without this feeling and then they're trying to boil the ocean the small sample size they're trying to measure some different hyper which is not even possible to do. But yeah, there's so many of them.

Paul Kovalski: But this is a key one amazing last question. What's one mental model or book that has shaped how you make decisions Most

Talgat Mussin: of I really love work of the Doctor Benjamin hardly the series of books 10x easier than 2x and the most recent one impacted me so much. I'm going to implement in my business and life. It's called science of scaling. And the key concept is that treat your time as a tool. And most of them you can change all your decision making today if you filter out to the future you desire for a desired future. And then everything becomes so clear that you don't need to do 90% of stuff you're doing if you filter it right with your big goal you have. So that's why when I intuitively already kind of knew this when I left the big tech, I put this very delusional goal. People told me it's delusional, but I know it's possible. The bloat and waste is so big. Uh, I want to help businesses to optimize $1 billion. And it's possible. I know it is possible. And that's creating the value that kind of will be by product. Like of course I'll get some small cut out of it when I make it happen. But putting in the value first, helping the drive those results. Now I filter all my decisions through that one simple goal where I optimize $1 billion. And I already made some progress, like 20 million. But I need to change direction because it's too slow now. That's why I'm addressing this and finding the core issue of the decision maker, perception and the police. This is where I believe I'm going to change the slope of this traction, uh, and hockey stick, like move it faster. So scaling.com definitely recommend to anyone because it may change your life. Ah, in a way you cannot imagine.

Paul Kovalski: Love it. Talget, thank you so much for coming on the show.

Talgat Mussin: Thank you, thank you for having me.