A holdout group answers a simple question: what would have happened without this marketing? You split an audience at random, show the campaign or flow to one part and withhold it from the other. The difference in results between the two groups is the incremental effect. Everything else, such as seasonality, promotions and organic demand, hits both groups equally, so it cancels out.
Holdouts can be people (a share of your email list that never receives a flow) or places (a set of U.S. regions where an ad channel is paused, known as a geo holdout). Meta and Google also offer conversion lift studies that build holdouts inside their platforms.
Why it matters for eCommerce
Many orders credited to retargeting, branded search or post-purchase emails would have happened anyway. Without a holdout, those channels look far more profitable than they are. A holdout shows the real lift, which tells you whether a campaign earns back its cost or a discount flow is giving away margin to customers who were already going to buy.
How it works in practice
Decide the metric first (revenue per customer, orders, new customers), then size the groups so the test can detect a realistic effect. Keep assignment random and stable for the length of the test. For example (illustrative numbers): a win-back flow goes to 90% of lapsed customers and 10% are held out. After 60 days, the treated group produced $2.40 per customer and the holdout produced $1.90. The incremental value is $0.50 per customer, not the $2.40 the email platform reports.
Common mistakes
- Choosing the holdout by hand, such as “our least engaged customers,” which biases the result.
- Making the holdout too small to detect anything, then reading noise as a result.
- Ending the test early because the numbers look good after a few days.
- Letting the holdout receive the same message through another channel, which contaminates the comparison.