- Decide the sample size before the test starts. It depends on your baseline conversion rate and the smallest lift worth detecting, and halving that lift roughly quadruples the traffic you need.
- Run tests in full weeks, until the planned sample is reached, and avoid promotions that change who visits. Checking results daily and stopping at the first "significant" reading produces false winners.
- Pick one primary metric tied to profit, add guardrail metrics, and check that traffic split evenly. If you lack the traffic, test bigger changes or measure an earlier step in the funnel.
What does an A/B test on a store actually tell you?
An A/B test splits visitors at random between two versions of a page or experience and compares an outcome, such as conversion rate. Because the split is random, a difference you see at the end is either caused by the change or by chance. Statistics is how you tell those two apart.
Statistical significance answers a narrow question: if the change had no effect, how surprising would this result be? A result “significant at 95%” means a difference this large would show up less than 5% of the time by chance alone. It does not tell you the size of the true effect, and it does not guarantee the winner keeps winning.
Two errors matter in practice:
- False positive. You ship a change that did nothing, or that hurt. The significance level (usually 5%) controls this.
- False negative. You discard a change that worked because the test was too small to see it. Statistical power (usually 80%) controls this.
Most bad ecommerce tests fail on the second error. They were never large enough to detect the effect they were looking for.
How much traffic do you need for an A/B test?
Three inputs set the sample size:
- Baseline rate. Your current conversion rate for the page or step being tested.
- Minimum detectable effect (MDE). The smallest lift you care to detect, usually stated as a relative change (“10% more orders than today”).
- Significance and power. Commonly 95% and 80%.
A widely used approximation for a two-variant test at those settings:
- Visitors per variant ≈ 16 × p × (1 − p) ÷ d²
Here p is the baseline conversion rate and d is the absolute difference you want to detect. With a 3% baseline and a 10% relative MDE, d is 0.3 percentage points, or 0.003. A dedicated sample size calculator will give slightly different figures, but the approximation is close enough to plan with.
Sample size at a glance
The figures below come from that formula and are rounded. They show how fast requirements grow, not targets for your store.
| Baseline conversion rate | Visitors per variant, 20% relative MDE | Visitors per variant, 10% relative MDE |
|---|---|---|
| 2% | about 19,600 | about 78,400 |
| 3% | about 12,900 | about 51,700 |
| 4% | about 9,600 | about 38,400 |
Two patterns stand out. A lower baseline needs more traffic. And halving the MDE roughly quadruples the sample. This is why small stores struggle to detect small improvements: the math does not allow it on the traffic they have.
Choosing the MDE honestly
Pick the MDE from economics, not hope. Ask what lift would justify the cost of building and maintaining the change. If a new product page layout costs a lot to build, a 2% lift may not be worth it, so there is no point sizing a test to detect one. If you cannot reach the sample needed for a lift you would act on, rethink the test rather than run it underpowered.
How long should an eCommerce A/B test run?
Length follows from sample size: divide the visitors you need by the eligible traffic per day. Then apply three rules.
- Run in full weeks. Shoppers on a Tuesday morning behave differently from shoppers on a Sunday night. Ending mid-week overweights some days.
- Avoid shifts in traffic mix. A sitewide sale, a big email send or a new paid campaign changes who visits. If one lands mid-test, the result describes that mix, not your normal traffic.
- Watch for novelty. Returning visitors may click a new design because it is new. If the lift fades week by week, treat it with caution.
If the calculated length is many months, the test is not practical. A long test also suffers from cookie deletion and people switching devices, which mixes visitors between variants.
When should you stop an A/B test?
Stop when the planned sample is reached, in full weeks. Decide that before launch and write it down.
Why peeking misleads
Testing tools show live results, and it is tempting to stop the moment a variant crosses 95%. With a standard fixed-sample test, checking repeatedly and stopping at the first significant reading raises the false positive rate well above 5%. Early in a test, results swing widely. A variant can look like a clear winner on day three and be flat by day fourteen.
Some testing tools use sequential methods designed to be checked continuously. If yours does, follow its stopping rules. If you are unsure which method your tool uses, assume fixed sample and wait.
Valid reasons to stop early
- The variant is broken: errors, missing buttons, wrong prices.
- Guardrail metrics show clear harm, such as a sharp drop in checkout completion.
- The traffic split is off. If you planned 50/50 and see 52/48 on large numbers, check for a sample ratio mismatch. A chi-square test on the visitor counts tells you if the gap is beyond chance. A mismatch usually means a tracking or redirect problem, and the results cannot be trusted.
Which metric should decide an eCommerce test?
Pick one primary metric before launch. Choose the one closest to profit that you can measure with enough data.
| Metric | Good for | Watch out for |
|---|---|---|
| Add-to-cart rate | Product page changes, low-traffic stores | A higher rate may not become more orders |
| Conversion rate (orders ÷ visitors) | Most tests | Ignores order value and margin |
| Revenue per visitor | Pricing, bundles, upsells, free shipping thresholds | High variance; needs more traffic |
| Gross profit per visitor | Discount and offer tests | Needs product costs joined to test data |
Then add guardrail metrics that must not get worse: average order value, checkout completion, return rate for fit or description changes, and page speed. A discount test that lifts conversion rate while cutting profit per visitor is a loss.
Make sure the test tool and your analytics agree on orders. Our GA4 ecommerce tracking guide explains how to check purchase events against your store data.
What if your store does not have enough traffic?
Many $1M to $5M stores cannot detect a 5% lift on a single page. That does not mean no testing. It means testing differently.
- Test bigger changes. A new page structure or offer can produce effects large enough to measure. Button colors cannot.
- Test templates, not pages. Change the product template so every product page counts toward one test.
- Measure an earlier step. Add-to-cart rate has more events than orders, so it reaches a decision sooner. Confirm the effect on orders later.
- Fix clear problems without testing. A hidden button on mobile does not need a test. See our product page optimization playbook for what to fix first.
- Use holdouts for marketing. For email flows or ads, a holdout group that receives nothing measures lift without splitting a page. Our guide to incrementality testing covers the method.
Worked example: planning a product page test
All numbers below are illustrative.
A store wants to test a new product page layout. The product template gets 40,000 visitors a week, and the conversion rate from those visits is 3%.
- Set the MDE. The team decides a 10% relative lift (3.0% to 3.3%) would justify the build.
- Calculate sample. About 51,700 visitors per variant, or roughly 103,400 total.
- Calculate length. 103,400 ÷ 40,000 = about 2.6 weeks. Round up to three full weeks.
- Check the calendar. A sitewide sale is planned in week two. The team moves the test start so it runs after the sale.
- Define metrics. Primary: revenue per visitor. Guardrails: checkout completion and average order value.
- Pre-commit. No decision before three weeks unless the variant breaks or a guardrail drops sharply.
- Read the result. At three weeks, check the split is even, then read the primary metric and its confidence interval, not just the significance flag.
If the team had wanted to detect a 5% lift, the sample would roughly quadruple and the test would run around ten weeks, which is too long to be reliable. That is a signal to test a bolder change.
Checklist: before you launch an A/B test
- [ ] One hypothesis, written down, with the reason you expect a change.
- [ ] Baseline rate measured from at least the last few weeks of data.
- [ ] MDE chosen from the economics of the change.
- [ ] Sample size and test length calculated, rounded up to full weeks.
- [ ] No major promotions or launches during the test window.
- [ ] One primary metric and two or three guardrails defined.
- [ ] Test tool orders reconciled with store orders.
- [ ] Both variants QA’d on mobile and desktop.
- [ ] Stopping rule written down before launch.
- [ ] Traffic split checked for sample ratio mismatch at the end.
FAQ
What does 95% statistical significance mean in an A/B test?
It means a difference as large as the one observed would appear less than 5% of the time if the change had no real effect. It does not tell you how large the true effect is, which is why you should also look at the confidence interval.
Can I run several A/B tests at the same time?
Yes, if they are on different pages or audiences, or if your tool handles overlapping experiments. Tests that change the same element or the same step can interfere with each other, so run those one after the other.
Should I test with A/B/n variants instead of one challenger?
Each extra variant splits your traffic further and adds comparisons, which raises the chance of a false winner. On limited traffic, one strong challenger is usually the better test.