The Measurement Gap: iROAS, Geo Holdouts, and Progressive Truth

So the problem that exists as it relates to marketing measurement is that there's a measurement gap. Every brand really wants to understand the causal relationship between the advertising dollars they spend and the revenue they realize as a result. This is the central question of modern marketing. Does the spend actually work?

This episode of the e-commerce playbook is brought to you by Omnisend. If you're serious about growing your brand through email and SMS marketing, Omnisend has your back. It combines powerful automation, smart segmentation, and real-time performance insights, all in a platform that's intuitive and built specifically for e-commerce. With pre-built workflows, dynamic signup forms, and full multi-channel capabilities, Omnisend helps brands scale without unnecessary complexity.

Join over 125,000 e-commerce marketers who rely on Omnisend every day. Head to Omnisend.com slash CTC and start on the free plan today. That's Omnisend.

com slash CTC. Hey everyone, welcome to CTC's Canon series where we are walking through our methodology across the core disciplines, the CTC methodology, how we approach forecasting, modeling, measurement, med ads, Google ads, email strategy, creative strategy, and the data that has informed the methodology across these disciplines. We're deep diving into each one. This is part three on measurement.

We've walked through forecasting. We've walked through modeling. We are now on measurement, and then we will be sequencing into the other disciplines of creative strategy, meta, Google, email, and on down the line from there. So measurement is today's topic.

And this may be one of the topics that gets the most heated debate outside of meta ads. And then creative strategy will be up there as well. So we have some hot topic conversations coming up in these and these next three conversations that we'll have as we walk through our series. But marketing measurement, this is how CTC closes the gap between what your data says and what's actually happening.

Through incrementality testing, geo holdout studies and commitment to moving closer to reality over time. We could spend the next two hours just walking through all the different frameworks and approaches to measurement that exists in the direct-to-consumer e-commerce space and the brands that we work with. Across our data set of hundreds of brands, we are able to see and get insight into the conversations, the organizational structures, the platforms, the frameworks, the spreadsheets, the tools that are used to surround this conversation of marketing measurement from creating a weighted triangulation of marketing measurement based off of J last click, an Adobe signal and North beam to using also measured as the source of truth and then translating those in the platform ROAS targets to we're going to use platform attribution as the key source of truth on the line.

There's, there are an infinite combination of how brands are approaching marketing measurement. And our role and our responsibility is to identify what we see as being the most helpful framework for approaching this conversation that leads to the best decision making in pursuit of the business hitting their business objective, which is rooted in the first part of the series around forecasting and sublining on what that core objective is. So the problem that exists as it relates to marketing measurement is that there's a measurement gap.

Every brand really wants to understand the causal relationship between the advertising dollars they spend and the revenue they realize as a result. This is the central question of modern marketing. Does the spend actually work? A great measurement system, we believe, causes the gap between reality and fiction when it comes to interpreting that effect.

The wider the gap, the worse your capital allocation decisions, the wider the error bars are. The narrower the gap, the more confidently you can invest towards growth. So on picture two ends of a line, we have reality, the true incremental impact of your spend. We have fiction, what your platform dashboards report or whatever the tools are.

And there's the gap in between those things. The measurement system exists in that gap. The goal is not perfection. The goal is to move fiction closer to reality over time with increasing confidence.

So some first principles to ground us in this conversation. Before building any measurement system, there are foundational truths that must be accepted. There are opinions, these are constraints that govern any honest approach to marketing measurement. So the first of these three principles is that media efficacy is in constant flux.

The relationship between your ad spend and its revenue impact is not a constant. It changes day to day, week to week, driven by forces both within and outside your control. Any system that treats this relationship as fixed is lying to you. The second principle, you are always building an approximation.

At all times, your measurement system is an attempt to build the closest approximation to reality that you can. There's no perfect measurement. There is only less wrong. And the third principle, progressive truth is the mechanism.

Our approach is to build progressive truth. We want to move closer and closer to reality over time. Every test, every data point, every experiment reduces the error rate of the system. Truth is not discovered in a single moment.

It is accumulated over time and through reps and signal. So there are many forces at play when it comes to thinking about the efficacy of the media spin, which is really shaped by two categories of forces. And understanding the distinction between these two categories is critical to interpreting any measurement results There are forces within your control and there are forces outside of your control So what are the forces within our control The quality of your creative campaign optimization placement strategy, audience targeting, budget allocation, the landing page design and experience, and we can go on and on.

Forces outside of our control, general market demand, the macro economy, cost of ad inventory, platform algorithm changes, competitive intensity, seasonal demand shifts. So it's important for us to understand from a first principle standpoint that there are three core principles. Media efficacy is in constant flux. You're always building an approximation and progressive truth is the mechanism.

And then to understand that within that dynamic, there are two categories of forces at play, those within your control and those outside of our control. This common understanding of the grounding of these principles then allows us to start to discuss the method by which we are going to build the best approximation between these two points, given the principles we've agreed upon and the forces that are at play. The best mechanism for building progressive truth is through incrementality studies, specifically geo holdup tests.

These experiment designs are the gold standard for isolating the causal impact of advertising spend on revenue. From a very high level, geo holdout tests are where there are test regions in which the marketing is being delivered and the impact is being measured. And then there are the control regions which are not receiving marketing for the channel being measured. And what we are measuring is the revenue difference between the testing control regions relative to the synthetic control and the baseline that is established between those things.

that helps us to understand what the true incremental lift the measurement is. So geo holdout tests are identifying on a state DMA, GMA level, whatever the mechanism is to isolate specific geographies. Those are selected through a lot of smart data science to be able to understand the groups of those geographies that are going to be best representative and tied to the revenue impact for the business. But they're isolated, the impact of running marketing within a subset of regions versus not then gives us a high confidence level causal relationship between the marketing spend that we are deploying on a specific channel or tactic and what happens when we remove that from the media mix.

So this measurement of the media span and the incrementality study geo holdout lifts need to be measured across every point of distribution as well. So the incremental lift should ideally be measured through not just your own dot com because your advertising bleeds across channels. So you have your dot com, you have Amazon, and then you have retail dot com and then in store and on down the line for additional distribution channels. So an incrementality test gives you a snapshot of the causal effect of that revenue with some degree of confidence for that period of time.

It's critical to recognize that this is a single data point that may or may not be replicated again. right? Running a specific test at a specific point in time for a specific channel yields a specific result at a specific point in time. And if you were to try to replicate that test, even within the same channel that you are isolating similar budget levels, we're likely going to get a different result because it is at a different point in time relative to the prior test.

So this is where the concept of progressive truth becomes essential. A single test result is valuable, but inefficient, insufficient, the power comes from accumulation of these test results. So let's deep dive into a GLF study a bit more so that we share a common understanding of what that looks like. So we design and deploy these tests through statless data science platform powered by the data science team, which provides in-end management of GLF studies from recommendation to specific test outputs, validations, and results.

What we look is the power curve of these studies. We look at the results and we get them to a level to where the confidence level is at 90% confidence or better. So a P value of 0.1 or less so that we have a high confidence value against the revenue impact.

And then again, we measure this across every point of distribution that we have data for, .com, Amazon, retail.com, as well as every core customer cohort. So new customers and new revenue, returning customers and returning revenue, and on down the line so that we have a very clear understanding of the media's impact in each of these specific areas and the impact on the individual customer cohorts.

So when we think about the framework, we're grounded in the principles, we're grounded in the methodology, so incrementality geo holdout tests. So now we need to build the framework for how we deploy this into an actual operational execution that leads to action that we can take based on the input. So we want to build the largest database of test results, both in aggregate across our brands and for individual brands to continuously reduce the error rate of any presently applied measurement system.

So stage one is that we start with an aggregate benchmark across all tests ever run and apply that against the brand's platform reported revenue. So we have a database of hundreds of incrementality tests that we run across various platforms and tactics for a number of brands. And that gives us an aggregate benchmark that gets us very close in many cases to what the what the incrementality factor should be for for each of these platforms And we start with that approximation as the best source of truth right Prior to us even running the incrementality geoholdet test when we start working with a brand we use the aggregate benchmark result from our database of incrementality tests for that specific channel and tactic to get us closer to what truth is in terms of that channel's impact.

Still juggling multiple tools for email, SMS, and push marketing? Omnisend brings them all together in one powerful platform built for email. e-commerce. You can launch pre-built automation workflows, build audience segments based on shopping behavior, and even gamify your signup forms without spending weeks learning the tool.

It just works and it's built to scale with you. Every account starts on a free plan. And unlike other platforms, Omnisend gives you 24-7 human support no matter your tier. Check it out at omnisend.

com slash ctc or click the link in the description. So that's stage one is using the aggregate benchmark to inform what the incrementality factors should be for that specific channel and tactic. Stage two is that we are going to get the first test result back from an incrementality and geo holdout test specific to that brand and that channel and tactic. So we run a geo holdout test specific to that business, get the factor.

and then what we are going to do is that we are going to weight that individual result relative to its confidence level and relative to a relationship to the aggregate benchmark that we have. We're not going to move towards it all the way, but we are going to get closer to that approximation of what the impact is. Because again, that test result was at a specific point in time in a specific structure. And so there should be some impact of the aggregate benchmark as it results to weighting that.

And then stage three is that as more test results come in over time, the median that's of the set represents the measurement with the lowest error, right? So we continue to get more data signals and move towards to this progressive truth. So if you were to visualize what this looks like in practice, imagine a scatter plot of incrementality test results over time. So each dot on a scatter plot represents a single test.

Early on, the dots are sparse and widely scattered, right? So the confidence in any single test result is low, but as tests accumulate over a longer period of time, patterns emerge and the median converges. Error shrinks against that and we can get higher and higher confidence in terms of that channel tactics, true impact. So the starting point for any brand should be the aggregate benchmark.

The second an individual test result arrives, we wait it relative to his confidence and shift towards it. With each subsequent test, the system gets less wrong. The progressive truth and practice, CTC maintains one of the largest proprietary database of income mortality test results. It's not theoretical.

It's a real geo holdout test run across real brands with real dollars. We have hundreds of tests across a large number of channels as well, and various levels of confidence in those results as well. And what that leaves us with is channel level benchmarks based on this aggregate data set that we can use as a starting point to move a brand immediately towards higher confidence in terms of their media's impact on that specific channel or tactic. There are factors that we use that represent the relationship between the platform-related reported revenue and the actual incremental revenue from each of these channels.

And these vary, to name a few. Facebook acquisition campaigns, the median IROAS is a 1.14 across our data set. YouTube is a 1.

1. Google Ads non-brand, 0.67. Facebook non-acquisition, 0.

60 and on down the line. But we have the database across hundreds of stores, test results that where we're able to understand the IROAS, the incrementality factor, and then understand the distribution in each of these results as well related to this channel. The data based on our aggregate set of benchmarks though tells a really clear story, which is that Facebook acquisition, a median IROAS of 1.14 is the most reliably incremental channel.

A second trend is that Google branded search median IRO as a 0.27 confirms what the theory predicts platforms dramatically over report on last click channels or channels that are that are very close to the bottom of the funnel in terms of capturing demand. And the ranges show why single test results are insufficient. Facebook acquisition ranges from a 0.

5x to a 2.4x and on down the line. So these factors are not permanent truths. They are the current best approximation based on the aggregate of all available evidence.

As new test results come in, every number in this table will be refined. That is the system working as designed. As we've mentioned, a single test result is a snapshot, valuable but incomplete. And so the principle of always-on testing is important for brands to adopt in terms of thinking about measurement, that this is not a one and done, but that we are going to transform that snapshot into a distribution of potential incremental outcomes for each individual business.

So as each individual test result accumulates for a brand, the bounds of that distribution represent the total possible error in either direction. The median represents the point at which we are at any given time most likely to predict the future outcome. So always-on testing builds a distribution of outcomes. The median converges towards predictive accuracy.

This is a practice of measurement that, once aligned on the principles and the approach is going to be a continued practice and getting more and more data points to get higher and higher confidence in the true impact of our media measurement. As the database of test results grows, the system becomes increasingly sophisticated. Seasonal effects sale moments and other variables can be incorporated Rather than applying a simple median across all results the factor can be adjusted based on the conditions that most likely match the present moment What this enables operationally is apps to apples comparison across all channels.

So this is one of the most powerful benefits of this measurement systems is that it enables true like-for-like comparison between channels and ad products. So without normalization, comparing meta-acquisition spend Google-branded search is meaningless. The platform is reporting different attribution of windows with fundamentally different relationships to income mentality. Google Branded Search is a well-understood example of this.

Google Ads by default reports on a 31-1 or 37-1 attribution. Branded Search will dramatically over-report its return on ad spend. Branded Search is a final step action for many shoppers on their path to purchase. The customer is already going to buy.

They typed your brand name into Google, clicked your ad, and completed the purchase. The ad gets credit, but the purchase was still likely to have occurred had the ad not appeared. So the factors of channels living at different points in the customer purchase journey and different platforms using different attribution settings makes it very challenging to compare performance like for like and make clear investment decisions. Once we adopt an incrementality in iROAS practice, we're able to normalize across all channels so that the platform ROAS on meta may read a 3.

2X, but based on the incrementality factor, the true IROAS is a 3.7x. Then on Google, our platform ROAS is a 12.5x, but based on the incrementality factors and benchmarks, the IROAS is a 3.

1x. So now we can look at what was previously a 3.2 on meta and a 12.5 on Google branded.

We can look at as a 3.7 and 3.1 and look at those numbers like for like and make investment decisions accordingly. So by running incrementality studies on every channel and computing the incrementality factor, we create a normalized IROAS that allows us and the profit engineer or the brand to compare their performance to their media channels on a like-by-like basis.

And this is what enables real capital allocation decisions. These incrementality factors are not theoretical. We operationalize them into CTC's statless platform through the MMM roadmap, which uses tester-derived IROAS to prioritize channel allocation for each brand. So each of the most up-to-date test result factors live behind stat lists.

Every ad channel campaign and tactic are applied against their respective factors so that everywhere that we are looking at data from the daily forecasting and the dashboard to the tracker tabs for each of the media channels, everywhere we're looking, we're looking at the normalized incrementality adjusted revenue and ROAS for each of these channels that allows us to be able to make the most impactful decisions. To wrap this up, IROAS is subordinate to reality. Even a well-calibrated incremental return on ad spend should always be subordinate to the realities of revenue and contribution margin at any given time.

We should always assume some amount of error in the system. So an IROAS and actual business outcome are incongruent, meaning one is moving up, your IROAS looks really good, while the other contribution margin is flat or moving in the opposite direction. That is the signal to examine the underlying measurement system and recalibrate. Contribution margin and the business outcome lives at the top of the assessment pyramid.

IROAS is subordinate to that reality. If IROAS is improving, revenue is growing, and contribution margin is expanding, that's a signal that there's congruency. If any of those signals have adverse relationships to one another, then there's incongruency as it relates to that impact as well. One of the unique things that CTC brings to the table is the ability to design, deploy, and report on geo holdup tests as part of our core service offering.

But the real differentiator goes further than this. We have the obligation to operationalize those results in your ad account and to bring those effects to life in our decision-making and reporting. So rather than being a standalone measurement platform or tool where we design and run tests, deliver a report with results, maybe suggest actions, and then stop there and hand off to you or your other partners or your team. What we do as part of the profit engine and the profit engineer sitting inside of that is we design and run the test, we deliver the results with interpretation, but then we operationalize those in the ad account.

We calibrate the cost controls, the bid targets, reporting the realized business outcome, and we're accountable to the business objective at the end of the day. And the suggestions and test results that we're making need to impact those in a way that leads to the business outcome that we are after. So in conclusion, our measurement philosophy involves building an ongoing roadmap toward progressively better truth, one that allows to get us closer and closer to the approximation of reality and the causal effects of any given point in time.

The best measurement methodology will include a constant pursuit of new data points, a period application that builds, not undermines, subordination to business reality, end-to-end accountability, dynamic benchmarks from aggregate intelligence, and honest treatment of uncertainty. the measurement system is only as good as its ability to lead to a better allocation of crap capital across the available media channels. That is the standard. That is what we measure ourselves against, not the elegance of a model, but the quality of the decisions that it produces.

This was CTC Canon series on measurement. Looking forward to seeing you in our subsequent parts of going through the CTC methodology Canon.