Marketing Analytics with Fexingo · 2026-06-30 · 9 min
Key moments - from our scoring
Substance score
70 / 100
Five dimensions, 20 points each
Attribution models, even sophisticated multi-touch and data-driven variants, suffer from a fundamental blind spot: they cannot see the counterfactual - what would have happened without the ad. Lucas and Luna explore why this matters through a real case study where a retailer's paid search attribution was overstated by more than 100% because they lacked a control group. The episode walks through three control group methodologies for digital campaigns: geographic holdouts (excluding matched metro areas or DMAs for 4-6 weeks), time-based holdouts (running campaigns on alternating schedules, best for always-on campaigns), and randomized customer list splits (for email and retargeting, requiring stratification to ensure balanced groups). The hosts address why marketers skip control groups - they feel wasteful, require upfront planning, and risk revealing uncomfortable truths - but emphasize that the cost of not knowing far exceeds the expense of proper testing. Even mid-campaign, synthetic control methods and media mix models offer limited retrospective solutions. The conversation culminates in practical guidance on calculating true incrementality and avoiding cross-channel contamination that can invalidate experiments.
Their multi-touch attribution model had no control group to measure the counterfactual. A geo-holdout experiment in two excluded metro areas showed organic sales grew faster there than in targeted cities, proving the attribution model was taking credit for sales that would have happened regardless of the paid search campaign.
Geographic holdouts (excluding matched metro areas or DMAs for 4-6 weeks), time-based holdouts (alternating campaign windows, best for always-on campaigns over 3+ months), and randomized customer list splits (for email and retargeting, requiring stratification and at least 10,000 per group).
Take the conversion rate of the exposed group, subtract the conversion rate of the control group, multiply by the number exposed, then multiply by average order value - this reveals true incremental revenue, not what the attribution dashboard claims.
It occurs when a control region still receives ads through other channels (TV, social, retargeting) even though the primary channel is held out, contaminating the control group and making it impossible to isolate the impact of a single channel.
Three reasons: control groups feel wasteful because you deliberately don't serve ads to a segment, they require upfront planning and cannot be added after launch, and they risk revealing that high-performing channels in attribution dashboards aren't actually driving incremental sales.
Our reviewer’s read on each dimension, with quotes from the episode.
The episode delivers concrete, actionable insights about control groups and their necessity in attribution - a genuine blind spot many marketers have. The specific retail example (2.1M attributed vs. <1M true incrementality) is concrete and illustrative. However, the explanations of the three control-group methods, while clear, cover relatively familiar experimental design principles; the density drops slightly in the latter half as the conversation becomes more explanatory than revelatory.
Their multi-touch attribution model said paid search drove about two point one million in incremental revenue... The geo experiment showed the true incrementality was probably under a million. The attribution model was off by more than a factor of two.
Even the best attribution models - multi-touch, data-driven, algorithmic - they all suffer from the same fundamental blind spot. They can't see what would have happened if the ad hadn't run.
The core argument - that attribution models confuse correlation with causation and require control groups - is solid and important, but it is not novel thinking. Controlled experiments and the counterfactual problem are established statistical concepts; the episode applies them competently to marketing but does not reframe the idea or challenge orthodoxy in a fresh way. The framing of 'uncomfortable truths' and platform incentives is a mild counterpoint, but underexplored.
Luna: Counterfactual being the key word. Without a control group, you're measuring correlation, not causation.
And that's the gap I want to talk about today. Even the best attribution models - multi-touch, data-driven, algorithmic - they all suffer from the same fundamental blind spot.
Lucas presents himself as an operator who works with retail brands and has real case-study data, which is credible for a podcast of this type. However, the transcript provides no details about his seniority, company, track record, or depth of direct experience running attribution experiments at scale. He appears knowledgeable but is presented without context that would establish him as a practitioner of genuine authority rather than a well-informed analyst.
So last week a retail brand I follow shared something that stopped me cold.
The retail brand I mentioned - they spent maybe 50 thousand dollars not serving ads in those two metro areas for three months.
The episode grounds itself in a real retail case with concrete numbers: 2.1M attributed vs. <1M true incrementality, 50K spent on holdouts, specific DMA matchings, and sample-size rules (10,000 per group). It names tactics (geo holdouts, time-based, customer list splits) and includes practical parameters (4-6 weeks minimum for geography, 3 months for time-based). However, broader claims lack supporting data: no numbers on how many brands skip control groups, no cited studies on contamination rates, no benchmarks for typical ROAS delta between attribution and holdout.
Their multi-touch attribution model said paid search drove about two point one million in incremental revenue... The geo experiment showed the true incrementality was probably under a million.
The retail brand I mentioned - they spent maybe 50 thousand dollars not serving ads in those two metro areas for three months.
Luna asks competent clarifying questions and identifies logical follow-ups (e.g., 'why do so many marketers skip control groups?', contamination risks, retroactive fixes), pushing the conversation forward. However, questions are often rhetorical setups for Lucas to deliver the next point rather than genuine probes that challenge or dig deeper. There is no productive disagreement, pushback on edge cases, or testing of Lucas's claims. The dialogue is collaborative and clear but lacks the edge that would come from skeptical interrogation.
But the cost of not knowing is way higher. You could keep pouring budget into channels that look great in your attribution dashboard but are actually just intercepting demand that already existed.
Cross-channel contamination is the number one killer of geo experiments. If your holdout city doesn't get paid search but still gets your TV ads or social retargeting, you're not measuring paid search in isolation.
Computed from the transcript - who did the talking, and the words that came up most.
Episode 84 of Marketing Analytics with Fexingo tackles the most overlooked gap in campaign measurement: control groups. Lucas and Luna walk through a real mid-2026 retail campaign where the brand's multi-touch attribution model credited $2.1 million in sales to paid search, but a simple two-city geo-holdout experiment showed actual incrementality was less than half that. They explain why most attribution models measure correlation, not causation, and why even sophisticated media mix models need a counterfactual to work properly. The episode drills into the practical logic of setting up clean control groups: geography splits, time-based holdouts, and randomized customer list partitions. No marketing jargon - just the numbers that matter, and the one question every marketer should ask before trusting their attribution dashboard. #MarketingAnalytics #Attribution #ControlGroups #Incrementality #CampaignMeasurement #MarketingROI #PaidSearch #GeoExperiment #MediaMixModel #Counterfactual #MarketingScience #MarketingEffectiveness #DataDrivenMarketing #AdWaste #MarketingPodcast #FexingoBusiness #BusinessPodcast #Analytics Keep every episode free: buymeacoffee.com/fexingo
Transcribed and scored by The B2B Podcast Index.
Lucas: So last week a retail brand I follow shared something that stopped me cold. They ran what they thought was a solid paid search campaign through Q2 - ended June thirtieth, so literally just closed the books on it. Their multi-touch attribution model said paid search drove about two point one million in incremental revenue. Luna: That sounds like a win.
What was the problem? Lucas: The problem is they also ran a simple geo-holdout. They excluded two similar metro areas from the campaign entirely. Those two cities - no paid search ads at all.
And in those holdout cities, organic sales actually grew faster than in the targeted ones. Luna: Ouch. So the attribution model was giving credit to paid search for sales that would have happened anyway. Lucas: Exactly.
The geo experiment showed the true incrementality was probably under a million. The attribution model was off by more than a factor of two. And the worst part? This wasn't a bad attribution model.
It was using last-touch and view-through with some algorithmic weighting. It was doing exactly what it was built to do - it just had no way to see the counterfactual. Luna: Counterfactual being the key word. Without a control group, you're measuring correlation, not causation.
Lucas: Right. And that's the gap I want to talk about today. Even the best attribution models - multi-touch, data-driven, algorithmic - they all suffer from the same fundamental blind spot. They can't see what would have happened if the ad hadn't run.
Only a control group can show you that. Luna: So why do so many marketers skip control groups? I've seen brands run million-dollar campaigns with zero holdout cells. Lucas: Three reasons.
First, control groups feel wasteful. You're deliberately not serving ads to a segment of your audience, and that feels like leaving money on the table. Second, setting them up properly takes planning - you can't just add a control group after the campaign launches. And third, if the control shows your ads aren't working, you have to face that uncomfortable truth.
Luna: But the cost of not knowing is way higher. You could keep pouring budget into channels that look great in your attribution dashboard but are actually just intercepting demand that already existed. Lucas: Exactly. And there are three main types of control groups that work well for digital campaigns.
Geographic holdouts, time-based holdouts, and randomized customer list splits. Let's walk through each one. Luna: Start with geography. The retail brand's approach - seems straightforward, but what are the pitfalls?
Lucas: Geographic holdouts are the gold standard for brand campaigns. You pick matched metro areas or DMAs - similar in demographics, baseline sales, seasonality - and you exclude one from the ad buy. The challenge is finding true matches. Two cities are never identical.
One might have a local competitor opening a store, or a weather event that shifts demand. So you need to run the experiment long enough - at least four to six weeks - to smooth out those local shocks. Luna: And you need multiple holdout regions if you can. One holdout is risky.
What if that city has a power outage or something? Lucas: Exactly. Multiple holdouts let you average out the noise. The brand I mentioned used two, which is decent.
But some sophisticated marketers use a cluster of ten or twelve, randomly assigning half as holdouts. Luna: Alright, time-based holdouts. How do those work? Lucas: Time-based is simpler but less clean.
You run the campaign Monday through Thursday, then stop Friday through Sunday, and compare. Or you skip ads every other week. The problem is that consumer behavior has natural weekly cycles. Friday spending patterns are different from Tuesday.
So you're comparing apples to oranges unless you control for day-of-week effects. Luna: Plus, if you're running a campaign with a short purchase cycle, a one-week gap might not be long enough to reset the baseline. Lucas: Right. Time-based works best for always-on campaigns where you can alternate weeks over a longer period - three months minimum - and then model out the day-of-week variation.
It's not as clean as geography, but it's easier to execute. Luna: And the third method - randomized customer list splits. This is the one I see most often in direct response. Lucas: Yeah, this is common for email and retargeting.
You take your customer list, randomly split it into a test group that gets the ads and a control group that doesn't. Because the split is random, the two groups should be statistically identical. Any difference in conversion rate can be attributed to the ads. Luna: But only if the randomization is done right.
I've seen brands do a simple 50-50 split without accounting for existing segment biases. Lucas: Big issue. If your list has heavy buyers and light buyers, a simple split might put more heavy buyers in one group by chance. So you should stratify - sort customers by past purchase value or frequency, then randomize within each tier.
That ensures both groups have a similar mix. Luna: And you need a large enough sample. A list of 5,000 won't give you statistical significance for a small conversion lift. Lucas: Right.
There are power calculators online, but a rule of thumb is at least 10,000 per group if you're expecting a lift of 10 percent or less. Smaller lifts need bigger samples. Luna: So what should a marketer do if they're mid-campaign and realize they have no control group? Is there any way to retroactively create one?
Lucas: It's tough. You can try synthetic control methods - basically building a statistical model of what would have happened using data from similar markets or time periods. That's what media mix models do, but they're only as good as the data you feed them. And they can't fully replicate a true randomized experiment.
Luna: So the takeaway is: plan your control group before you launch. Even if it's small. Lucas: Absolutely. And be honest about the cost.
The retail brand I mentioned - they spent maybe 50 thousand dollars not serving ads in those two metro areas for three months. That feels like a loss. But it saved them from wasting millions on a channel that wasn't actually driving new sales. Luna: And that's the core insight.
Attribution models are great for tracking which touchpoints get the last click. But they can't tell you if the ad caused the sale or just claimed credit for it. Lucas: Which brings us to the bigger question. If your attribution dashboard shows a channel with a 5X ROAS, but a control group experiment shows it's actually 2X - which number do you trust?
I'd argue the controlled experiment every time. Luna: And if today's episode saved you from over-investing in a channel that wasn't working, or helped you set up your first control group - honestly, that's worth a coffee to us. You can show that at buy me a coffee dot com slash fexingo. It's the smallest ask, but it keeps these conversations happening without any sponsor influence.
Lucas: Yeah, listener support is what keeps us independent. And that matters especially for a topic like this, where the truth can be uncomfortable for the platforms and agencies that sell attribution tools. Luna: Okay, back to the data. So once you have a control group, how do you actually calculate incrementality?
Lucas: It's simpler than most people think. You take the conversion rate in the exposed group, subtract the conversion rate in the control group, and multiply by the number of people exposed. That gives you incremental conversions. Then multiply by average order value.
That's your true incremental revenue. Luna: But you have to make sure the control group isn't being contaminated. If someone in the control group sees the ad via a different channel - say, organic social - then the control is no longer clean. Lucas: Huge point.
Cross-channel contamination is the number one killer of geo experiments. If your holdout city doesn't get paid search but still gets your TV ads or social retargeting, you're not measuring paid search in isolation. You're measuring something else. Luna: That's why some brands run full funnel holdouts - they turn off all paid media in the control region for a period.
Lucas: Which is the purest test. But it's also the scariest for a CMO. Turning off all media in a region for a month takes guts. The brands that do it, though, often come away with the clearest picture of what actually works.
Luna: Alright, so what's the one thing you want listeners to remember from this episode? Lucas: If your attribution model doesn't have a counterfactual - a control group, a holdout, a randomized test - then you don't know if your marketing is working. You only know what your dashboard says. And those are two very different things.
Luna: That's a good place to land. Thanks, Lucas. Lucas: Thanks, Luna. See you next time.
Other episodes covering the same guests and topics, from across The B2B Podcast Index.
How Kubernetes Topology Spread Constraints Create Scheduling HotspotsDevOps Daily with Fexingo · features Luna95 / 100
How B2B Brands Wreck Pipeline with Unsyncroned CRM DataThe Marketing Operator Podcast with Fexingo · features Luna92 / 100
Why API Webhook Payloads Should Be Signed Not VerifiedThe Developer Tools Podcast with Fexingo · features Luna90 / 100
How to Sell Against a Competitor Already in the BuildingSales Leadership with Fexingo · features Luna85 / 100
Why B2B Brands Are Using AI for Account PrioritizationThe Growth Operator with Fexingo · features Luna84 / 100
Enterprise Software Buyers Now Demand a Vendor Data Portability GuaranteeB2B SaaS Talks with Fexingo · features Luna82 / 100
Why Incremental Lift Testing Beats Attribution91 / 100How Brand Lift Studies Reveal True Marketing ImpactHow Offline Sales Drive Online Marketing ROIHow Contextual Targeting Replaces CookiesHow First-Party Data Reshapes Attribution