How Incrementality Reveals True Marketing Impact

Lucas: Luna, I want to talk about something that should make every marketer a little uncomfortable - how much of your campaign's reported performance is actually caused by your ads, versus just happening to coincide with them. Luna: You're talking about incrementality. The idea that attribution models show correlation, not causation. Lucas: Exactly.

And the gap between the two is often enormous. There's a well-known example from a DTC skincare brand that ran a geo-based lift test. They split the US into test and control markets, ran their normal ads in the test markets, and turned off ads in the control markets for eight weeks. Luna: That's the cleanest way to measure incrementality - an actual experiment.

Lucas: They discovered that roughly forty percent of the conversions attributed to their ads in last-click models actually happened in the control markets where no ads were running. Meaning those customers would have purchased anyway through organic search, direct traffic, or word of mouth. Luna: So their last-click model was overstating ad-driven revenue by nearly two-thirds. Lucas: More than that.

If you think about it, the reported ROI on their ad spend was probably two to three times the true incrementality-based ROI. And this isn't an outlier - studies across multiple verticals show that twenty to forty percent of attributed conversions are non-incremental. Luna: It makes you wonder how many budget decisions are based on inflated numbers. Lucas: Right.

And the problem isn't just last-click. Multi-touch attribution models, even the sophisticated algorithmic ones, still distribute credit across touchpoints without a control group. They can't distinguish between a customer who clicked a retargeting ad and would have converted anyway, versus one who was genuinely swayed by that ad. Luna: So what's the gold standard?

A randomized controlled trial, but in the real world that's hard to set up. Lucas: It is hard, but not impossible. The most common approach is geo-based experiments, like the skincare brand did. You split your market into geographic regions, match them on key variables like population, income, baseline conversion rate, then randomly assign some as test and some as control.

You run ads only in test regions and compare the lift. Luna: That controls for seasonality, market trends, all the noise that attribution models can't separate. Lucas: Exactly. Another approach is time-based switches - you run ads for two weeks, pause for two weeks, measure the difference.

But that assumes no carryover effects, which is rarely true. Geo experiments handle carryover better because you have a concurrent control. Luna: There's also the matched-market approach for companies that can't randomize. You find two markets that look similar and treat one as test, one as control.

Lucas: That's a practical compromise. I remember a CPG company that used matched-market experiments to evaluate a new digital campaign. They chose ten pairs of DMAs matched on category penetration, retail density, and prior sales. In each pair, one got the campaign, one didn't.

The result showed that about thirty percent of the sales lift was incremental. They then reallocated five million dollars from the least incremental channels into a channel that showed a one-point-four times higher incrementality factor. Luna: So they basically stopped funding ads that were just taking credit for organic demand. Lucas: Exactly.

And that's the power of incrementality - it forces you to confront the question: would this conversion have happened anyway? If the answer is yes, you're wasting money. Luna: There's a nuance though - incrementality tests measure short-term lift. What about brand-building effects that take months to show up?

Lucas: That's the biggest limitation. A twelve-week geo experiment might miss delayed effects. But you can extend the measurement window or run repeated experiments over time. Also, you can combine incrementality with marketing mix modeling to get both the causal short-term and the modeled long-term.

Luna: If today was actually useful to you, the way these stay ad-free is listener support - buy me a coffee dot com slash fexingo. It's a small gesture that keeps us free and independent. Lucas: Yeah, we really appreciate it. And back to the topic - one of the challenges with geo experiments is statistical power.

You need enough regions to detect a reasonable effect size. For a national campaign, you might need at least twenty DMAs per cell. Luna: That's why many companies only test their biggest campaigns. But even one well-designed test can change how you think about attribution.

Lucas: Absolutely. And the beauty is that once you have a validated incrementality factor for a channel, you can apply it as a discount to your attribution model. Say your last-click model shows a two times return, but your incrementality test says only sixty percent is incremental. The true ROI is one point two times, not two times.

Luna: That's a much more honest number for budget allocation. Lucas: And it changes the conversation with finance. Instead of saying 'our ad spend generates two times ROAS,' you say 'our ad spend generates one point two times incremental ROAS after controlling for baseline demand.' That builds credibility.

Luna: So for a marketer listening, what's the first step? Should they run a geo test tomorrow? Lucas: Not tomorrow. But they should start identifying a channel or campaign where they suspect waste - high attributed volume but low incremental lift.

Then design a simple experiment. It doesn't have to be perfect. Even a two-market A/B test with a four-week holdout can reveal directionally correct insights. Luna: And if they can't run an experiment at all?

Lucas: There are vendors that offer incrementality measurement as a service - they handle the test design and statistical analysis. But the key is to start. Once you see the gap between attributed and incremental, you can't unsee it. Luna: That's a good note to end on.

Next time, I want to talk about how incrementality applies to retention campaigns - where the baseline is even higher. Lucas: That's a great topic. Let's do it.