- Attribution tells you which ads were near a sale. An incrementality test tells you how many sales would not have happened without the ads, by comparing a group that saw them with a group that didn't.
- Two designs cover most stores: platform lift studies, where Meta or Google randomly holds back people from seeing ads, and geo holdouts, where you switch ads off in some regions and compare sales with similar regions.
- The result is most useful as a correction factor. If a channel's incremental ROAS is well below its reported ROAS, raise its reported ROAS target until the incremental return clears break-even.
What does an incrementality test actually measure?
Every ad platform reports conversions it can link to an ad. Some of those buyers would have bought anyway: they knew the brand, had a full cart or came back from an email. Attribution only sees who touched an ad, not what would have happened otherwise.
An incrementality test builds that “otherwise.” It splits people or places into two groups:
- Test group: sees ads as usual.
- Control group: a holdout group that doesn’t see the ads being tested.
The difference in sales, scaled to the same size, is the incremental effect:
- Incremental conversions = test group conversions − expected conversions without ads
- Incremental ROAS (iROAS) = incremental revenue ÷ ad spend in the test
- Incrementality factor = incremental conversions ÷ conversions the platform reported for the same period
An incrementality factor of 0.6 means 60% of the platform’s reported conversions were caused by the ads.
Which type of incrementality test should you run?
| Design | How it works | Strengths | Limits |
|---|---|---|---|
| Platform lift study (Meta Conversion Lift, Google Ads Conversion Lift) | The platform randomly assigns users or regions to test and control and withholds ads from control | Randomized, run inside the platform, little setup | Measures one platform at a time; you rely on the platform’s own data and method; eligibility rules apply |
| Geo holdout | You pause or cut ads in selected regions and compare store sales there with matched regions | Uses your own sales data; works for any channel; captures effects outside the platform | Needs enough regions and volume; regions are never perfect matches; sales outside ads can add noise |
| On/off (time-based) test | You switch a channel off for a period and compare with before | Simple | Weak, because seasonality, promotions and trends change over time too |
Many stores start with a platform lift study on their biggest channel, then add geo holdouts for an independent read. Treat on/off tests as a rough signal, not proof.
Platform lift studies
Meta runs Conversion Lift studies through its experiments tools. Google Ads offers Conversion Lift based on users or on geography, and has been lowering the budget needed to run lift studies. Eligibility, minimum budgets and supported campaign types change, so check the current requirements inside each platform or with your account representative before you plan around one.
A lift study’s strength is randomization, so the groups start out alike. Its weakness is that it measures only conversions the platform can see; if the pixel undercounts, so does the lift result.
Geo holdouts
In a geo test you choose regions, such as states or Designated Market Areas (DMAs), and change ad spend in some of them. You then compare sales in those regions with sales in regions where nothing changed.
The outcome comes from your own records, usually Shopify orders by shipping address. That makes geo tests an independent check on platform reporting, and works for channels without lift studies. Meta’s open-source GeoLift package is one widely used tool for designing and reading these tests; your analyst can also use a synthetic control method of their choice.
How do you design a geo holdout test, step by step?
- Pick one question. For example: “How many sales does our Meta prospecting spend cause?” Test one channel or campaign type at a time.
- Pick the outcome metric. Use orders or revenue from your store, by shipping region, by day. Include new customer orders as a second metric if acquisition is the goal.
- Pull the history. At least several months of daily sales by region. More history makes matching more reliable.
- Choose regions. States work for smaller stores; DMAs give more units for larger ones. Drop regions with very low sales or unusual patterns.
- Match test and control. Select test regions whose past sales moved closely with a set of control regions. Tools like GeoLift do this by building a weighted “synthetic” control from several regions.
- Size the test. Decide the smallest lift worth detecting, your minimum detectable effect, and check whether your volume and test length can detect it. If the answer is no, test a bigger budget change, more regions or a longer period.
- Set the duration. Cover at least one full purchase cycle, often several weeks. Add a short period after the test to catch delayed purchases.
- Freeze everything else. No new promotions, price changes or big launches in only some regions. Email and SMS should go to everyone as usual.
- Run it and don’t peek to decide. Stopping early when the numbers look good inflates false positives.
- Analyze. Compare actual test region sales with what the control predicted. Report the lift, a confidence interval and whether it reached statistical significance.
Worked example: a Meta geo holdout
All numbers are illustrative.
A store spends $40,000 a month on Meta prospecting. Meta reports a 3.0 ROAS. The store’s contribution margin is 40%, so break-even ROAS is 2.5.
Design. The team selects test regions that account for about 20% of sales and pauses Meta prospecting there for four weeks. The control regions keep running as usual.
Result. Over four weeks:
- Test regions (ads off) sold $160,000.
- Based on the pre-period relationship, the control model expected them to sell $200,000 with ads on.
- Lost sales from pausing ads: $40,000.
- Meta spend withheld in those regions: $8,000 (rounded for simplicity).
| Metric | Value |
|---|---|
| Incremental revenue in test regions | $40,000 |
| Ad spend withheld | $8,000 |
| Incremental ROAS | 5.0 |
| Meta-reported ROAS (account, same period) | 3.0 |
In this illustration, Meta underreported its effect. That can happen when ads drive sales Meta can’t see, such as purchases after a view on another device or orders on another sales channel.
Now flip it. If the lost sales had been $14,000, iROAS would be 1.75. Meta’s reported 3.0 would look healthy, but the channel would be losing money on first orders, since 1.75 is below the 2.5 break-even.
Either direction is possible; that is why you test.
How do you use incrementality results to set targets?
The test result turns into a correction factor for day-to-day reporting:
- Incrementality factor = iROAS ÷ platform-reported ROAS
- Platform ROAS target = break-even ROAS ÷ incrementality factor
Using the second version of the example: factor = 1.75 ÷ 3.0 ≈ 0.58. To be truly at break-even (2.5), the campaign needs a reported ROAS of about 2.5 ÷ 0.58 ≈ 4.3.
That changes decisions. A team that was scaling at a reported 3.0 now knows it needs closer to 4.3 to break even on the first order. If repeat purchases make new customers worth more later, you can accept a lower first-order return on purpose; our guide to break-even ROAS covers how to set that floor.
Correction factors drift with creative, audiences and seasonality. Rerun tests on your biggest channels once or twice a year, and after major changes in spend or strategy.
What are the most common incrementality testing mistakes?
- Testing too little spend. If the paused budget is small next to total sales, the effect hides in normal day-to-day noise.
- Ending early. A test that looks positive after one week often fades by week four.
- Spillover between regions. People travel, ship gifts to other states and see national campaigns. Pick regions that limit this, and be cautious with nearby cities.
- Counting only the store. If you also sell on Amazon or in retail, ads can lift those sales too. A store-only metric undercounts.
- Changing other things mid-test. A regional promotion or a site outage in one group ruins the comparison.
- Treating one result as permanent. Incrementality changes with creative, spend level and season.
The method is similar to on-site experiments, with regions in place of visitors. Our guide to A/B testing for eCommerce covers sample size and stopping rules in more detail.
How does incrementality testing fit with attribution and MMM?
Incrementality tests answer a narrow question very well: what did this channel cause, during this period? Attribution answers a daily question roughly: which campaigns and ads are near sales? Marketing mix modeling answers a broad question with a model: how does total spend across channels relate to total sales over time?
Most mature measurement setups use all three. Tests calibrate the model and correct attribution; attribution guides daily optimization. Our explainer on marketing mix modeling vs attribution covers when each one is worth the effort.
Checklist: before you launch an incrementality test
- [ ] One clear question, one channel or campaign type.
- [ ] Outcome metric comes from your own sales data, by region and by day.
- [ ] Break-even ROAS is calculated, so you know what result matters.
- [ ] Minimum detectable effect is estimated and the test is sized to reach it.
- [ ] Test and control regions are matched on past sales.
- [ ] Duration covers at least one purchase cycle, plus a short post-test window.
- [ ] Promotions, pricing and launches are the same in all regions.
- [ ] Stop date and analysis method are written down before launch.
- [ ] A plan exists for turning the result into new targets.
FAQ
How long should an incrementality test run?
Long enough to cover at least one full purchase cycle and reach the effect size you planned for, which often means several weeks. Add a short window after the test to capture delayed purchases. Shorter tests are noisier and more likely to mislead.
Is a platform lift study enough on its own?
It is a good start, because the platform randomizes who sees ads. But it relies on the platform’s own conversion data. A geo holdout measured on your store’s sales gives an independent check, which matters most for your largest channels.
What is a good incremental ROAS?
Anything above your break-even ROAS means the ads caused more profit than they cost on the first order. How far above depends on your fixed costs and how much repeat revenue new customers bring, so set the bar from your own numbers.