Going Deep into Marketing Mix Modelling and Incrementality - Pranav Piyush

welcome to rev ops FM attribution remains one of the hardest challenges in marketing. It affects. Literally everything, how we're perceived as a discipline, the strategies, the activities we decide to do, how we justify our existence, all of it. And at the end of the day, there's still so little rigor in most companies around attribution.

Most teams I've worked with are doing something either very basic like first or last touch or using methodologies pushed by a vendor. Uh, that are kind of just invented out of thin air that aren't backed by a lot of evidence. I don't see that to be critical because I've used those methodologies too. I've been a consultant for those methodologies.

but I think there is a growing awareness as we scratch a little bit beneath the surface that a lot of the ways that we're doing attribution and historically have done attribution, especially in B2B, Don't have a very solid grounding. so today we're joined by Pranav Piyush is the CEO of Paramark, which is a company that provides media mix modeling software, incrementality testing software, and we actually got connected through a linkedin discussion. I was posting about attribution.

He was very, politely pushing back on some of the things that I was saying. So I thought it was actually a great opportunity to bring him onto the show and talk about what he's seeing so for now, thank you so much for joining us today. Thanks for having me. And I have to give you your flowers for being engaging and really sort of engaging in discussion and debate on LinkedIn.

I think that's the way it should roll. So excited to chat live about it. Agreed. So, I mean, Maybe let's just start.

I've set the stage a little bit with, some of the problems out there, but give us your view from 1000 feet. what is the state of B to B attribution today? How is it going from your perspective in the industry? things to be excited about is the higher level of conversation around the concepts of incrementality and even just the concepts of correlation and causation.

And that's really good. I'm glad that we're having that conversation. I'm glad that these types of podcasts are coming across. At the same time, there's almost a tale of two cities or two worlds or whatever, where there's a whole spectrum of conversation that is either still stuck in the multi touch attribution world or is inventing pseudoscience.

In the name of incrementality, and that gets me really upset because I have this point of view that the word attribution has been maligned. It's actually a really beautiful word, but it's literally just been maligned and something very similar is going to start happening with incrementality because Of a whole host of reasons that we can talk about that. So I'm both excited and optimistic, but at the same time, a little bit disappointed by some of the conversations going on. Yeah, I am seeing this play out the scene that you've described exactly.

And I go back to, 10 years ago or maybe even a little bit more when Bizible first came on the scene and it was like this amazing thing you could track all these different touch points and then you could choose all these different ways to divide them up take your opportunity credit and you just kind of slice it up like a pizza, like this touch point gets a slice and we do w shape or equal weight or whatever. and we never really stopped a question, I guess, as a marketer and like, I'll take ownership for this for myself.

We didn't stop to question like, is this justifiable? Does it make sense to do this? Does it actually produce like a, good outcome in terms of decision making if we do this and now, I guess there is a lot more pushback. against this.

Can you just walk us through like, is that sort of historical? multi touch attribution methodology. Is it a valid way of looking at the world? And if not, why not?

I give you three points of evidence that will hopefully help inform this conversation. So the first one, if you look at a whole variety of channels, and this is arguably sort of more biased towards larger brands, whether you're B2B or B2C. But if you look at channels like. Social video podcasts, Like the one that we're talking about right now.

These do not generate clicks or touches by definition. You are going to exclude a significant part of marketing and media from your models. If you rely on multi touch attribution models to assess the impact of these types of channels. And well, some people will tell me, well, like, what about view through attribution, right?

Isn't that the solution? And I'm like, yes and no, because that brings me to the second point, which is there is coincidence. There is correlation and there is causation. The three C's.

Everything that we are talking about in MTA world is coincidence. A Latin term that I just recently came across. Post hoc ergo proctor hoc. All this means is, after this, therefore because of this?

That's a logical fallacy. That is a very popular fallacy that just because something happened right prior to something else happening that we assume that there's a cause and effect relationship there where there isn't even a correlation. so that's the second point. And the third is when you add all the privacy changes that have happened in the last five years, people are realizing that even the touch based data that they do have is incomplete because guess what, 20 to 30 to 50 percent of people do not accept cookies.

And most of your first click, last click data is coming through either cookies or UTM codes. And now there's increasing evidence that UTM codes are probably going to get stripped out from pretty much every browser. Safari is already doing that in multiple cases, Firefox is already doing that in multiple cases. So, when you look at all three of those points, it's like, how can multi touch attribution work?

And that's how I generally think about this conversation. Now, it doesn't mean that you shouldn't track clicks and touches. That's not what I'm saying. There's perfect legitimate use cases for tracking a user journey.

And that's how I think about that data. It's behavioral analytics. It's not attribution. Attribution is very simply cause and effect.

If we're not talking about cause and effect, you can't call it attribution. So that is an important distinction to make tracking the touch points, understanding, these are observable facts that we can detect. We're not saying it's the entirety of everything that happened. It's just, we're saying that this happened and we could track it.

That can be useful taking it and then, you know, starting to dole out credit and saying, therefore this channel drove X million dollars in pipeline. That's a, that's a fallacy. And I think that makes sense. I I've yet to see a strong argument against, I think those three things that you mentioned are kind of devastating.

so let's then like, I've created a nice foil now for, MMM, or I've heard it say both media mix modeling and marketing mix modeling. I don't know which of those you prefer. Maybe just introduce us to that and why is it different and not suffer from those same limitations. I don't have a preference.

I think, you know, people can call it whatever it is. It's we invent so many new terms. what's interesting about MMM is that it's predated all of us. It's predated the Internet.

So this was actually invented, I believe, in the 70s or 80s by, you know, folks at PNG. This is folklore. I don't think there is good attribution for this, but I think it was literally built by a partnership of academia folks at M. I.

T. And Howard and practitioners that companies like P. N. G.

And they had a hard job, right? Because you didn't have any touch and click data. So how do you know which of your ads are working or not working? And they were working with, you know, T.

V. And radio and newspapers and these types of media channels. And so they had to invent something new. And so you had a bunch of these statisticians, who we now call data scientists, who in the 70s and 80s are like, Hey, there's actually a way that we can model the data about readership and listenership and find correlations between the increase or decrease in readership and Of PNG products.

So as you increase the number of ads in the wall street journal, maybe that's a bad example, how many more sales can be attributed to that region that that newspaper is distributed in. And that was the beginning of MMM. You look at time series data. So you're looking at day by day.

You have 10, 000 people reading a newspaper. You have 15, 000 people reading a newspaper. You have 20, 000 people reading a newspaper. And as that trend goes up and to the right, do you have a corresponding increase in your amount in your sales and vice versa?

When you decrease the distribution of ads through newspapers, do you see a decline in your sales? And now when you do that across multiple channels at the same time, you can build very sophisticated models. And these are all being done through spreadsheets and manual work back in the 80s. And now it's been digitized and we can talk about sort of what that's been like.

So when you think of it that way, right, let's talk about all three issues. You no longer have to sacrifice channels. This can work for newspapers. It can work for Google.

It can work for podcasts. It can work for pretty much any channel out there. Second, you are no longer making an assumption about one thing happening and therefore the second thing happening. You're actually looking at the statistical correlation between a quantity increasing and its impact on your sales increasing or not.

So you can imagine if those two numbers are going like this, there's no correlation. But if the both numbers are going like this, there is a correlation. So that's the second piece. And third, because you are not actually tracking individual users, you're making a, analysis on aggregated data.

You have no privacy concerns. We're not trying to spy on a certain user and say, did you click or touch on this particular piece of advertising? We're saying as overall numbers have increased in terms of impressions or reach or frequency. Okay.

Has your sales numbers increased? It's a very different approach and completely privacy friendly and future proof. So that makes total sense to me and let me ask you a question like I'm going to try to poke a hole not out of skepticism because I would just like to hear how you feel it. And this is probably coming from a place of ignorance because I am not a statistician or a mathematician.

so we've solved the post hoc fallacy because we've seen that two things trend in a similar direction, let's say, placing ads in the wall street journal, like you said, and sales, what it doesn't solve for, like, let's say that the reason why we decided to place more ads was because, you know, economic forecast, consumer confidence is high as we predict there'll be more demands. We're going to place some more ads so we have correlation, but we don't necessarily have causation because the increasing spending could have been.

Caused by that. How do you resolve that problem in MMM? There is a certain amount of correlation that you will not be able to convert into causation through MMMs. MMMs are not causal.

I'm probably the only MMM vendor that will that out loud and be willing to, you know, stand by it. So just because I'm talking about MMM being better than MTA doesn't mean that I'm saying that MMM is the end all be all and gives you perfect causality in your models. It doesn't. It is a correlation.

It's an estimate of causality, but it is not causality. To get to causality in marketing, you have just one option. That is experimentation. By the way, that is also the only known way of getting to causality in any other field.

I was just thinking that it sounds a lot like medical tests, like you can do long term studies and establish correlation. But if you want causation, you have to do a controlled trial. Precisely. So random control trials are CTS.

Everything in all other fields of science, rests on the shoulders of our CTS. And there is no difference in marketing. If you want to get to causality, you have to run experiments. So one of the things that we talk about at paramark all the time is the purpose of your marketing mix models.

Media mix models. Attribution models is not to establish causality. It is to understand the The hints of causality, it's estimates of causality. And if you really want to get precise about causality, that should inform a series of experiments that you are running.

And we can talk about how experiments can be run in media. It's harder than running an A B test on your website because you don't control the surface, right? You don't control TV, you don't control radio, you don't control meta. But there are, Reasonable ways of doing experimentation on that channel that gives you a sense of causality.

And I can talk about that if there's, you know, if you think that's a good time to jump into it. Yeah. I mean, let's, go there let's say a podcast. Here's a perfect example.

Obviously this is my own podcast. Let's say, uh, a company's looking to run a podcast. How should we understand causally if it's affecting, you know, Sales or not. That's an interesting one.

Podcast advertising far easier to test causality, right? Because if you're advertising on Spotify or on YouTube or any other channel where you have sort of podcast advertising, almost all those platforms are going to give you geotargeting capabilities. So if you're doing testing on podcast ads, the way to construct your test and control groups is by geography. And you'll tell Spotify that, Hey, target my ads just to, I'm just making this up San Francisco and Miami, and let's keep New York and Austin as the control.

And you're going to monitor your conversions from those locations over the next six weeks to see if there was a statistically valid increase in your conversions. As a result of you running those podcast ads, that is pretty much the blueprint for any channel. The hardest parts about this are identifying the right geographies, identifying the time frame, and identifying the amount of budget that you have to dedicate to be able to see an effect. That's pretty simple.

Podcasts by themselves? I think of them as an asset, not a channel. So let's talk about this, You and I are recording this podcast. Well, how does it make it out to people?

There's only one way you have to distribute it. So the real question is how are you distributing it? And does that distribution plan have a positive effect? On your pipeline.

So whether that's splicing it up on social, whether that's cutting it up into social ads, whether it's through email to my list of email subscribers, but that's where you get into testing, not the, whether me producing a podcast has an effect or not, it's like asking, you know, I created an ebook, but like that doesn't matter. What really matters is how did you deliver your ebook to your audience? And you can test that. If the tree falls in the forest and no one hears it, does it make a sound kind of thing?

But let's get into content because this is a huge question. I, work with a content team, and I've worked with, with many content teams over time, and everybody wants to know, like, is this working? Is it helping? And, I saw a post, I think it was Dale Harrison who also posts a lot on these topics, maybe it was on a post of yours.

It might have been around, around incrementality, like seeing a blog or not seeing a blog, and it's like, well, of course, the people that are reading your blog are people that are already more likely to buy. Okay. And I could see that argument, but then you're also saying, well, we're investing a ton, like hundreds of thousands, even millions in a bigger company in content, there has to be some way to be able to tell aside from the distribution channel. Whether that content is actually valuable, whether it influences people's decisions in any way, how would you go about trying to do that?

So first off, I would say, if you think about any creative production, at the end of the day, content is at the heart of what we do as marketers, right? If you don't have content, you have nothing, you're distributing nothing. So the idea that you have to justify your investments and content come from a place of severe. Anxiety and under confidence in you as a marketer.

Okay. So that's the first thing that I'm going to say, like, that's the wrong conversation to be having. Now you might still get forced into having that conversation because you have a finance team and you have a CEO who maybe don't understand that and you're having to talk about the justifications of why you're investing hundreds of thousands of dollars in content. I get it.

I understand where that's coming from. I would flip the script and flipping the script means if you can explain. That a hundred percent of your marketing budget, right? If you're spending 5 million a year are bringing an incremental 50 million in pipeline, then nobody cares how you spend the 5 million between content versus distribution.

The problem and the question arises because you don't have good math on your side about how much incremental pipe has been generated as a result of that 5 million in marketing. And I don't care if it's brand, performance, content, creative, paid, earned, owned, it doesn't matter. So, that's my answer to that question. Trying to pinpoint the efficacy of every single piece of content is a losing proposition.

If you had to do it, I would do it based on the engagement metrics of that content. What do I mean? You know, I had a conversation with Ashley Faust from Atlassian and I pitched this concept to her. I was like, look at the total volume of consumption of a content.

So if you're talking about a blog post, the total minutes that have ever been read about a certain blog post. If it's a video, the total view time. Guess what? If you go and talk to YouTube creators, that's what they're going to talk about.

The lifetime viewership of their content. They're going to look at the percentage of people who make it to the end of the video, right? That's the video retention rate. So you look at the consumption metrics of the pieces of content to understand if the content is good enough or not.

But again, it's You still have to figure out how you're going to distribute the content. And that's a different metric altogether. that makes sense to me. And I think I agree with you, but I want to, I want to drill down on it one more level, just because this really, and I'm sure you have a similar experience.

This really is a conversation we have all the time. And I think part of it, you're quite right. Is the. insecurity, I think we've developed as, as marketers in the face of like these purely short term activation, performance driven metrics.

and the way that executive teams ask questions to marketers. So there's that, but then I think there's also, you want to know sometimes like, is my content any good? Like I put out podcast episodes. I want to understand are these episodes good or the ones with the highest viewership, the best episodes, or is it, you know, there's lots of factors that come into it.

So to give a tangible example to just try to test. What you're saying. we have a, a blog post in my company. It's like a very, uh, top of funnel.

It's a blog post in NASA. It's something, it's a piece of content, very top of funnel, very general, not really product related. So of course it gets a ton of traffic. It's about like onboarding or something like that.

Something that's very relevant to everyone. So it's a lot of people, but there's no guarantee that the. People who are consuming that content, you know, have any sort of like product related intent are ever going to buy anything, connect that content back to our product. So I, totally buy into the engagement metrics and I, and that is the same feedback I give to my team.

And yet I struggle sometimes when you say it could be engaging, but is it engaging the right people that eventually will lead to the outcomes that you want? Like, how do you control for that? You know, It's the same topic as the question of a marketing qualified lead. This is the same exact conversation.

And if you think about the word qualified, who are we to qualify our prospects? They are qualifying us for whether we solve their need or not. So, that to me is not a measurement question, that's a strategy question of why are you putting that content out in the first place. And if you're putting the content out in the first place is to, you know, boost your engagement metrics, then obviously it's going to attract the wrong traffic.

You see what I'm saying? Because your strategy is focused on juicing a number rather than to serve your audience. If your strategy was to serve your audience, you would immediately think about a different way of measuring it, Or your measurement would be more real, so, I think we conflate different things when we talk about content strategy, like I don't care for the qualified metric. I don't care for qualified as a term.

We don't do any of that at paramark itself is because it's very clear who we serve. And if you don't find that out on your first visit to paramark, we did something wrong. So it doesn't matter what engagement metrics I had from that number. and again, it goes back to like having that conversation with your leadership.

It was like, no, no, no, I want to. Increase my organic visits to the website by 20%, And you're like making shit up to do that. But why are we doing that? Like, why does 20, why is 20 percent the right number?

basically The content engagement is a useful proxy of how valuable the content is. But if you're putting out content about like how to pick the winning lottery numbers, you shouldn't necessarily expect more people to buy your contract management software. Like it just doesn't work that way. And.

that's logical reframing it as a strategy rather than a measurement question, I think is interesting. So digging a little bit into this, and don't expect you to, like, unpack complex math here in a conversation, but how does it actually work under the hood? are the inputs? What are the outputs?

And then how are teams? How are your customers actually making decisions using this information? Yeah, there's a lot there. No, I think we should, we should talk about it.

So if you visualize the question of marketing measurement as a formula, that's the best sort of, that I have found. So on the right hand side, you've got your. Success metric might be pipeline, might be sales, might be orders, might be leads. whatever the metric is that you're optimizing towards, And on the left hand side, you've got every single marketing channel.

You've got a multiplier on that marketing channel that represents the strength of the correlation. Okay. And you have a few other variables that are representing seasonality. That are representing, the organic trend of your business that are representing other factors that may be outside of your marketing team's control.

When you sum up all of those things on the left hand side. You're trying to predict the right hand side. That's what's happening under the hood. And I'm not even kidding.

If you actually go look up the academic work, that's literally what's happening under the hood. So it's just a way to solve that formula. And you're applying a whole bunch of machine learning to predict or to simulate, based on all the data that you have, the answer to that equation. So how do customers make use of this?

When you get output out of a, an MMM, you essentially get a few different sort of interesting, tidbits of information you understand. The percentage of your success metric that came from a certain channel, or that can be attributed back to a certain channel. And again, this is an estimate. You understand the cost, obviously, of acquiring an incremental conversion from that channel.

so you get that across every single channel. And you also get what is known as a baseline. baseline. is essentially your organic demand.

Your word of mouth, your brand equity. These are all sort of things that were not driven by marketing or sales in the short term. These are sort of longer term things that are happening in your business. And then you also get a sense of what a future might look like if you were to invest more in every single channel.

Again, this is like, you can think of it as a forecast. If I put another 10, 000 into Metta, what will that do to my success metric? Okay. And you can imagine all of this being shown to you on a monthly or a weekly basis, depending on how often you are refreshing your analysis.

So that's what you get. So now you have a very rich understanding of, as you have increased or decreased your spend and your strategies in different channels, What has that done in terms of contribution to your metric? And if you invest more, what is the likelihood of that increasing even further or not? So our customers will use that to inform a series of experiments.

Where you take the most efficient channels and you figure out, Oh, if this channel is looking so good, can I just dump another million dollars here? What is the point of diminishing return? Right? So you run an experiment, run an actual incrementality test to test that hypothesis, and that becomes an ongoing set of experiments that you're constantly running every month, every quarter, every year.

And that's essentially your marketing roadmap. For other channels where it's highly inefficient, you've spent a lot of money, but it's not statistically correlated with your success metric, you might have a different hypothesis. Maybe we need to pull back on spending, or maybe we need to change the creative execution in that channel completely, Maybe static ads are not good, maybe we need video ads, maybe thought leadership ads. I'm just making this stuff up, So, the question is not necessarily to cut spend, the question is to find the winning channel.

Combination that'll help you extract even more growth. And the only way to do that is to constantly be experimenting. So summarizing, when you run an MMM, you get a whole bunch of output. Think of those outputs as informing your experimentation roadmap, and then go out and experiment on a monthly cadence.

I want to steel man your case here because on, on many levels, I would, I would love for this to be like the answer not, not that I have a horse in this race, but the, just the status quo was so bad for marketing attribution that it would be wonderful if this was the solution. So It feels like a big company thing. It feels like, yeah, all right. If I've got millions of dollars to play around, we're like an extra million dollars and spend for an experiment.

Like, sure. But if I'm a smaller company or even I'm at like a scale upstage company, 400 people, 50 million ARR, I'm picking a fake number. It's not the real number. Let's say a 5 million budget or even a 10 million marketing budget.

You don't necessarily have that sort of wiggle room. So what do first off, you're absolutely right that if you are just spending 100, 000 a year, all of the stuff that I said is way too complicated for you. And I do not recommend it. So this is not the go to methodology.

For seed stage startups or even series a startups or, you know, your mom and pop store around the corner, that's not it. This is meant for when you have lots of channels and a lot of spend to optimize. That's the reality. Now, what is the, threshold?

And, probably somewhere around a million or two, where you start to see that transition from mostly one channel to now many channels and this number will be different for different types of businesses, right? You could have an e commerce store where all you do is Facebook. That's it. That's your only distribution method and you have 10 million in spend on Facebook and you don't need to do attribution modeling because that's the only channel you have.

There is nothing else. So, there's a little bit of like just fake, you know, I'm giving you some like lines, but they're not precise lines. Now, having said that, if I were, and Paramark doesn't do advertising just yet, we are going to start in Q4, maybe in Q1, and the fundamental way that we're going to do this is through an incrementality test. So here's the fun fact.

You don't need to have MMM to be able to do incrementality testing. You can run an incrementality test, a geo test. So how does one do it? If I'm spending for the first time ever, I can look at all of my traffic today and where that traffic comes from by geography.

I don't need any privacy type of software to do that. That's just IP address matching with geolocation, Now I can see that, Oh, like 30 percent of my traffic is coming from California. The remaining is split across these five states, New York, Texas, Washington, whatever. I'm going to run this campaign just in Texas, and see if my traffic goes up, do my demos go up, does my pipeline go up, based on the location of the people who are entering the country.

The funnel. That's it. I don't need any complicated software to do this. So the interesting thing is, everything that I just talked about, you don't need software to do it on your own at a smaller scale.

You can do it yourself. You need basic math skills and basic, you know, understanding of how to do geotargeting in all the ad platforms. so that's my answer for smaller stage companies. That's really interesting.

is there anything to be said about confounding variables as we record this, you know, terrible events with Hurricane Helene on the east coast. of, the U. S. that's gonna affect demand to some degree for some period of time.

How do you adjust for that in these models? listen, I think the, for smaller businesses, when you're just starting, you literally will know what's happening in your test market. you're not going to set it and then not look at the news for the next six weeks, Right? so you can always kind of reset if you run into any issues that have a potentially negative effect on your test.

This happens all the time. Even in A B testing, you launch an A B test, like. Oh shit, there's a bug in our test version. Okay, we're gonna have to fix that bug and then relaunch the test.

So, that's a very acceptable answer to that question. It gets a little bit more interesting when you're at a large scale. When you're at a large scale, you may not have your eyes on every single DMA, and every single state and location. That's impossible.

And so the art is in constructing your test and control groups in a way that you minimize for the confounding variables. So you're not just looking at one whole, one DMA, one city. You're looking at a collection of test states. You're grouping them together that sometimes will avoid the noise that might come from like location specific variables.

Even having said that, if you were doing this test in March of 2020. I would probably not look at the results of that test. So the point that I'm making there is you always have to use judgment. That's why us humans will have jobs for a very, very long time is you're applying human judgment on top of the data and not letting the data do the judgment for you.

And that's how I think about, you know, confounding variables is you have to have a good hypothesis of what else could have happened that impacted this result that doesn't pass your intuition. That makes a lot of sense. I want to bring it back. to the original posts that got us chatting on this topic.

I was sharing. it was kind of like a history of one opportunity. One opportunity. Look at all these touch points.

This is interesting. And I can't even remember what exactly I was saying about it, but I think I was saying, you know, this isn't representative, but it's still kind of interesting. It's useful as a communication tool for sales. it's 1 opportunity.

It's not statistically valid now, let me ask you, am I deluding myself when I look at that and be like, Oh, this is interesting. Like, is it just irrelevant? Should we ignore it? What's your take on these like very local datasets and are they worth looking at?

It's a great question. humans are interesting creatures, right? we're visual in nature. We like to see things.

We like to be able to touch things. We like to be able to feel things. And there's a natural tendency for us to do the same thing when it comes to analytics. If it's not on a, chart, it's really hard for us to visualize, right?

Which is why MMM struggled so much because there's such like. Probabilistic statistical sort of type of things that it's people's intuitive mental models don't map to that way of thinking. so, that's what's going on in your brain when you see that laid out very neatly on a chart that this thing happened, this thing happened, this thing happened, this thing happened, and it gives you a sense of comfort, I know that this is what happened. So.

are you deluding yourself? Maybe that's a strong word. what I would say is it's I don't think that it adds anything to your reality is what I'm going to say. think about the additional information that you got out of that that you didn't already know.

So my way of thinking about this is if you are looking at that data, you're doing that to understand the behavioral journey of somebody. And that's literally like, this is what I did, and then this is what I did, and then this is what I did. If you view it from that lens, it's perfectly reasonable. Hey, we understand our customer's journey through the buying process.

Here's the typical things that are involved in the behavior journey. But does that mean that there is cause and effect? Probably not. And those two things are very distinct things.

And you have to just have an intellectually honest conversation about what are you actually looking at. I think that's reasonable. the way you put it about the way people process information. William Carlos Williams, a modernist poet from the United States.

He had this famous expression, no ideas, but in things. and I, I often think about that because I actually have a lot of trouble dealing with like mathematical abstractions. Uh, I really find that my insights and my understanding come from concrete particulars. So.

Would it be valid to say, I'm looking at this journey, obviously there's nothing even correlative about one opportunity's journey, let alone causative, but I say, oh look, they're like, before they buy, all these people started attending these workshops we were having, maybe there's something to that, and like what you said for MMM, it could inform a larger scale experiment, is that an okay? Way in your opinion to look at it. Absolutely. And I would also look at the individual things that are happening in that journey.

Right? So, the one that you described, workshops. Workshops, webinars, events. These are real things that are happening in the world.

As opposed to, I sent an email to my entire email database. And email showed up as a touch point in that journey, so you have to apply a human judgment when you're evaluating these journeys of like, what is really going? And I say this to many marketers, if you're not talking to your audience on a daily basis. What are you even doing?

Right? You can't call yourself a marketer. So, the other part of this is like, let's break out of our charts and visuals and let's go talk to actual human beings. That's going to tell you a lot more about whether it was the email or the workshop that got them excited to engage with your buying process.

So even those conversations, it's another example of something that it doesn't really scale mathematically, but it's very rich in terms of giving ideas, get like it feeds the sort of intuitive, emotional side of your brain, you could say for lack of a better word. Totally. And I go back to the PNG example, right? So tying it all the way back, MMM started with this partnership between, um, operators and academics at P.

N. G. Guess what? P.

N. G. Was also a pioneer in how you do customer research. They were spending time in people's homes, understanding how they used P.

N. G. Products, and they still, to this day, have a huge team of people who just does that. So you can have both methodologies to understand the impact of your products and your customers lives and how they make decisions at an individual level, but then really get to understand them and talk to them and observe them and their reality and also look at the aggregate impact of your marketing strategies on buying behaviors and your sales metrics.

It's not an either or it's a both. so great line of questioning So turning to incrementality again, you've, you've mentioned it a few times. I see some vendors using incrementality probably in ways that you would disapprove of, I'm sure that I've used incrementality. probably in ways that, that you would disapprove of in terms of, communicating about it internally and the.

The example I wrote down, which I think was the same, uh, maybe the same example from your blog post, all right, two charts, people that viewed a blog content and people that didn't, look, the people that viewed blog content have a greater chance, greater likelihood of converting, like, that's kind of seems reasonable, it's better than multi touch, we're not just saying, like, because they viewed a blog, we're giving it some credit, which is probably the most reductive, we're at least dividing them into groups, why is this, not useful, let's say, not valid?

Yeah, it's a great question in that analysis you are comparing two cohorts or two groups both groups had the option of Viewing the content or not viewing the content one group chose to view the content one group chose not to view the content So these are very different Different groups by themselves, and therein lies the challenge of claiming incrementality. If you take a step back and think about the concept of incrementality testing, right? We talked about test and control groups for test and control groups for RCTs.

Your groups have to be identical. So in an RCT world, in an incrementality testing world, you would have a test group where it's made up of both people who are viewing the blog and not viewing the blog and a control group where there is no blog because you're testing the effect of the existence of the blog. In getting to the conversion, Not just did they, did they choose to do it because already they've self selected into, it's like bias is exactly the term, so that's precisely it.

Anytime you have a question of incrementality, the immediate question you have to have is what is the control and the control has to be the non existence of the marketing or the media or the asset that is in the test bucket. Otherwise, it wasn't a causal analysis at all. and still on the subject of incrementality, I saw something, on LinkedIn the other day, which I actually found disturbing on some level. So I'd like you to comment on it.

It was the comment that, or let's say you spend, you bid on AdWords, you acquire a customer that way. We're not saying that AdWords was the only thing responsible for that, but at least like common sense that you would feel, I'm going to give AdWords a little bit of credit and then someone in the, in the comment, again, I can't remember who's like, actually there could have been no incrementality whatsoever from doing that. And in the sense that spending that money actually gave you some sort of incremental lift that you weren't otherwise going to get.

And that's disturbing, I think, because if you can't rely on the fact that this person came, we have a trackable touch point here that came in from this channel. If we can't rely on that to say that at least We're getting something from this. what can we rely on? What can we trust?

So, you know, it undermines a lot of the ways of thinking. In other words, that I think are very standard, very accepted. tell us about incrementality from that point of view. Yeah, so I'm going to play it back to you, right?

So the example is somebody bids on an AdWord, somebody clicks on that AdWord, comes and converts on your, you know, website. Is that incremental? Is it not incremental? Yeah.

I think it's the devil's in the details a little bit. I'll give you two examples, and I can prove incrementality or lack thereof in both examples, okay? If that AdWord purchase was for a branded keyword, And nobody else was bidding on that keyword. It's actually probably not incremental at all, right?

Because there's no competition for that keyword. And if you hadn't bid on that keyword, your organic search result would probably have taken the top result. Now, here's the caveat. Maybe your organic is awful because you just are new and you haven't done anything and you know that your organic result is on page three, then obviously that AdWord click is incremental.

There's no way somebody is going to page three. So the devils in the details, unfortunately, you can't have this conversation on LinkedIn post without all the additional context. Now let's take another example, right? If it's a non branded search keyword, It's a long tail keyword, non branded has nothing to do with your brand.

I would argue it's very hard. To not be incremental in that bucket. Again, the same conversation would apply, right? For it to have been non incremental means that that person would have done a non branded search, found your organic page, click through.

And then bought something. So your organic has to be a plus and generally speaking, that's like the chances of that happening are pretty low. So you have to view these things from the subjective lens. Now, here's my argument.

Run that as an incrementality test, and you will know the answer. It's not that hard. If you limit your AdWord by to a certain geography, and you can literally compare and contrast that with a test geography and a control geography, and you can see the incrementality of whatever strategy that you're employing. If you're in doubt.

If you're not in doubt, and you can have a reasonable conversation like this, you should be able to get to that pretty quickly. That answer seems commonsensical, at least, and I think, in part, I'm probably being unfair as I'm asking you to justify somebody else's comment, which doesn't make a lot of sense, but perceiving, I suppose, we're chatting a little bit before the show, just about how MMM and sort of related comments are becoming much more mainstream. I suppose they already do.

We're, mainstream for decades, as you were saying, but sort of unknown, like running along this parallel track to the world of B2B SaaS, all of a sudden, they start poking in. And even over the last six months, I see more and more other podcasts, more and more people popping up, more vendors. So it's really interesting time. And, in a lot of these discussions, MMM and brand, um, The notion of brand seemed to be fellow travelers.

I don't suggest the same thing, but there's a, on the one hand, we have sort of like multi touch attribution, trackable short term activation. On the other hand, we have like brand long term and MMM. And so that's why I sort of correlate those things together. And I think it's really interesting conversation is for the longest time brand spend, maybe it was either derided or people were afraid to do it because you couldn't justify it.

And now there's like a lot of people coming out, or at least a small group of vocal people coming out and saying unapologetically, brand is super important. We really need to do this. People are forming the consideration set before they even search. And maybe that's where that notion of is the search even incremental if they've already decided who they want to buy.

so maybe just talk about brand. How do you view brand and how does it relate to the MMM discussion? brand is another one of those words that has been maligned a lot. and the first conversation I have about brand is what is brand?

What is branding? And what is brand marketing? These are three completely different things. And we conflate them.

So when people, you know, ask me about brand is like, which version of brand are you talking about? Are you talking about brand as in the concept of brand, which is to me, the perception that an audience of people has about your product or service. It's merely a perception. or are you talking about brand marketing, which again, I hate as a term because it, way people think about brand marketing is, oh, it's just harder to measure marketing and I'll call it brand marketing, And so to me, brand marketing is no such thing.

Every piece of marketing performs. You just didn't have a way of measuring it in the past. And so what MMMs and incrementality testing have done have made it possible for you to measure any type of marketing, whether, you know, you call it brand marketing or performance marketing or what have you. It doesn't matter to me if it's a billboard, if it's TV, if it's radio, if it's, you know, a feel good, hilarious ad on YouTube, whatever it is can be measured through incrementality testing or MMM.

Now, if you flip the script and say, Oh, but I want to understand the impact of this type of marketing on my brand perception. Now, that's a very different thing because you're talking about peeking into somebody's brain and quantifying their perception. This is an incredibly hard thing to do, and it's almost not worth doing. So until Elon Musk commercializes Neuralink, and we all have direct access to each other's brains, I would recommend not trying to measure brand perception.

and there are better things to measure than that to understand the impact on the long term. So, again, not a satisfying answer, but if you're talking about brand marketing, It's a myth that you can't measure it. You can absolutely measure it. If you're talking about brand perception, brand awareness, it gets a little bit more finicky.

And I think it's very, very hard to do well, a On the subject of brand based on your experience in this field and working with your clients, as a marketer, you know, I'm in operations. I work adjacent to marketers. You feel trapped. You feel like, all right, AdWords is safe, directly attributable, direct response channels are safe.

I can justify them. As soon as I go into brand, it's very, very difficult. I don't have an MMM today. and yet if you don't invest in brand, what you find is everyone's going to fish at the same spots.

There's the, whatever, 5 percent of people that are in market. Everyone's competing, throwing money and you can't scale those channels all of a sudden they're like, all right, we're going to increase your targets by 20 percent and here's another 20 percent budget. You know, I can't turn this tap open any wider. There's no more water coming out.

So you need, you do. I believe strongly you do need to do something, whether you call it brand marketing or demand creation or whatever the terms are. I think they're all referring to a similar thing of trying to reach people that aren't in market today, get in their head so that they want to come to you when they are in market. How do you perceive the value of that?

Does that, do you have any data that suggests that like, yes, this really is a very important thing to do. So hundred percent. I may not like the word brand or brand marketing, but the concept is, a plus you have to be able to talk to people who are not actively searching for a thing in your category, because to your point, that's three or 5 percent of the market. For certain SAS categories, it's even lower.

So for the remaining 95, 97%, what are you going to do? Wait until they get into market. And have already sort of decided and you're going to be playing from, behind the starting line, or do you want to be 10 or 20 percent ahead of the starting line? And that's the perfect way to talk about brand, right?

Is you could be starting from ahead of the starting line if you spent enough time and energy sort of creating that perception. So I think it's incredibly important. I think more people need to figure out a way to measure those things. It's not that hard.

Frankly, you know, the one thing that I would say is if you can get a great analyst on your team that is not being pulled in 1000 different directions at the same time, it can be a game changer. Or, there's so many consultants and vendors out there in the ecosystem, wink, wink, that you should really have somebody who can be an extension to your, you know, marketing analyst, marketing operations team, that's giving you that, firepower to go test new channels. That you don't have the internal sort of infrastructure to be able to do.

And if you can make the case to go build that internal infrastructure, hell yes, go for it. But I find that that's a much harder. You know, hill to climb than getting an external solution just because hiring people is significantly harder in this new environment. there are a lot of different vendors popping up into the space right now.

you're one of them. Like, tell us about your vision for Paramark. why is it different? what are you doing that's special in this field?

I'd say three things. One is, I have a former VP of marketing myself. I've been in the hot seat. I've had to defend budgets.

I've had to present to the board, uh, reported into the CMO. I've managed large budgets and I know how anxiety inducing that week before QBR is. When you have to stand up in front of everybody and talk about how marketing is driving the business, but you yourself are a little bit unsure about how it's exactly going, and it's not for the lack of trying. It's not for the lack of creative.

It's not for the lack of strategy and vision. It's because your hands are tied behind your back because you don't have the internal infrastructure to do the measurement that you ought to. So that's our whole purpose. We are built for CMOs.

Literally the name Paramount comes from being on the side of the marketer. We're never going to go and sell to CROs and CEOs and CPOs. Our single mission is to make CMOs successful and. Really position them back into the leadership role that they should have always been in through better measurement, through calmer measurement.

not a death by a thousand paper cuts. So that's our vision. I don't get into features and capabilities. Like people can figure that out on their own.

I love features and capabilities. So I'm just going to ask, like, if I come to you, is it sort of like self service, like give me your data and, and all right, here's your bottle, go have fun with it, or. Or is it sort of blended with a, professional service, like to help interpret the data, to help run experiments. How are you partnering with clients in that way?

Complete white glove. We get you up and running in about four weeks. We hook into your warehouse. We hook into your ad platforms.

We hook into spreadsheets. You have slack access or team's access or whatever you use. And every two weeks you have a call with a dedicated customer success rep that is literally white glove hand holding you through the entire process, interpreting the data, designing experiments, recommendations, And obviously you have all the data that you can possibly need. So dashboards and reports that help you slice and dice every single campaign, every single channel, every single time frame that you can possibly need.

setting realistic expectations, let's say, are people signing up and it's like. Oh my gosh. I found the way the world makes sense now. Like life as a marketer is amazing now or are there still, and it's fine if there are, but are there still like challenges, ambiguities, uncertainties, does all that just go away or is it still a fact of life?

Think of it as the calmness because you know the way to get to an answer is running experiments. So that confidence of if there is uncertainty or ambiguity Let's go run an experiment. And that uncertainty will be removed in about six weeks. So it's a different way of operating, which is coming from a, position of strength and confidence and predictability and a way of knowing the answer.

Because you have experimentation as part of the platform. If you were just doing marketing, mixed modeling or attribution modeling, which is much more sort of looking back. Well, I don't believe it. How should I believe it and you get into these like philosophical debates rather than no, no, no, let's go run an experiment.

It will be very clear what's happening at the end of that. It's interesting. I've, uh, and I've, I've seen, you know, a good handful of vendors in this space. Now you seem to be emphasizing the incrementality testing in your go to market in a different way, or at least more, much more prominently than others are, and there is something.

To your point about like, all right, I've got this model, but do I believe it? But the, the testing, it feels active. It feels like you're, going on, on offense, so to speak. And like you said, you're resolving and puts agency in your control, which is very comforting.

Exactly. You can do something about it. You don't have to sit there and debate models. So let me just last question.

I'm just curious, you know, your own go to market, you have an unfair advantage. You see all the data, you see, you see what's working in theory. You know, you should be able to like, do anything. Does that, of course I'm being somewhat facetious, but, how are you thinking about, how you're getting the word out about what you're doing?

I say this all the time. Measurement is. Is Robin creative is Batman. So the secret to go to market, I could be sitting at a trove of data, but to differentiate and to make a place in the audience's mind, you have to have great creative.

So for us up until now, that has been our organic social activity and having good high quality conversations. On social some, you know, warm outbound and that's been sufficient to get us from zero to one to go from one to five. We're probably going to need to do something different and it's going to be figuring out new ways of reaching B2B audiences that are not the cookie cutter. And that's hard, right?

Everyone is kind of doing the same thing and we have some tricks up our sleeve, but you know, I won't talk about that just yet. You'll, you'll see it in market. So hopefully you'll see it in market and you'll tell me if you liked it or you didn't. that's great.

And I will say it's it's an exciting time just as an idea. it makes me happy to see, what feels like more substantial conversations. And I'm sure there's, there's a lot more debate to be had around measurement. It's not just going to resolve itself, but at least bringing marketing a little bit further, at least B2B marketing further along the maturity curve to a place where it isn't like, Starting from zero where every day is kind of day zero of how should we measure this thing, which really does feel like for the last 10 years.

it's been like that. So, but thank you so much for coming on the show. This was really, really interesting. wish you well, we'll watch yearly to see how Paramount does going forward.

Thank you for having me and great work on this podcast. It's awesome.