Conversion Lab · Experimentation
Experiments on product, cart and checkout with sample sizes set before launch, durations in full weeks and revenue per visitor as the judge. Fewer tests that mean something, instead of many that don't.
Sizing a test
An A/B test compares a control with one or more variants on randomly split traffic. Whether it can detect a real difference depends on four inputs, all decided before launch:
With a 2% baseline conversion rate, detecting a 10% relative lift (2.0% to 2.2%) at 95% confidence and 80% power takes roughly 80,000 visitors per variant. If the tested page gets 20,000 visitors a week, split evenly, that test needs about eight weeks. Aim for a 20% lift instead and the requirement drops to about 21,000 visitors per variant, around two weeks.
That is why lower-traffic stores should test bigger, bolder changes, and why a test that looks like a winner after four days usually isn't.
The primary metric
A variant can raise conversion rate by nudging shoppers toward cheaper products, or lower it while lifting average order value by more. Revenue per visitor (revenue ÷ visitors) captures both. It is our default primary metric for tests that can affect what people buy.
It has a cost. Revenue varies much more from visitor to visitor than a yes-or-no conversion, so tests on revenue per visitor need more traffic to reach the same confidence. We plan for that, and handle unusually large orders so one bulk buyer can't decide a test. A higher revenue per visitor also raises what your ads can afford per click; see the Break-even ROAS Calculator.
Rules we don't break
Checking results daily and stopping when they look good sharply inflates false winners. We fix the sample size and duration up front, or use a sequential method built for early looks.
Shoppers behave differently on weekdays, weekends and paydays. Tests run in whole weeks, usually at least two, so each day of the week is represented.
If a 50/50 test delivers a clearly uneven split, something is broken in assignment or tracking. We check for this sample ratio mismatch before reading results.
Each variant is checked on real phones and desktops, with and without consent, before traffic is sent to it. Broken variants produce meaningless losses.
Big sales change who shops and why. Unless the test is about the promotion, we pause launches around major sales.
A variant that only wins for one device or channel, found after the fact, is a new hypothesis to test, not a win to roll out.
What we test
Ideas come from the research in our UX & Checkout Review, not from a list of generic best practices.
| Area | Example tests | Usual primary metric |
|---|---|---|
| Product page | Information order, image sequence, subscription as default vs one-time, review placement | Revenue per visitor |
| Offer structure | Free-shipping threshold, bundles vs percentage discounts, gift with purchase | Contribution per visitor |
| Cart | Cross-sell placement, threshold progress bar, express checkout position | Revenue per visitor |
| Checkout | Field order, trust messaging, payment option order, where your platform allows | Checkout completion rate |
| Landing pages | Channel-specific pages for paid traffic vs the default collection | Revenue per visitor by channel |
Price and shipping tests have legal and customer-trust implications. We agree the rules with you before running them.
Testing and research tools we work in
FAQ
Until it reaches the sample size calculated before launch, and in full weeks, usually at least two, so every day of the week is covered. Stopping when results first look significant inflates false winners. Running far longer than planned adds exposure to seasonal changes and cookie churn. We set the duration in the test plan and stick to it.
It depends on your baseline conversion rate, the smallest lift you want to detect, and the confidence and power you choose. Smaller lifts need far more traffic. As an illustration, detecting a 10% relative lift on a 2% conversion rate at 95% confidence and 80% power needs roughly 80,000 visitors per variant. A 20% lift needs about a quarter of that.
Because conversion rate ignores order value. A change can raise conversion while shifting shoppers to cheaper products, or lower it while lifting average order value by more. Revenue per visitor combines both. It needs more traffic to read reliably, so we size tests for it and keep conversion rate and AOV as supporting metrics.
Peeking is checking results repeatedly and stopping the test as soon as it looks significant. Random swings early in a test often look like wins, so stopping on them produces winners that don't hold up. The fix is to commit to a sample size and duration before launch, or to use a sequential testing method designed for repeated looks.
Yes, with adjustments. Test bigger changes that could move results meaningfully, test on high-traffic templates like product pages, and accept longer durations. Where traffic is too thin to test, rely on research and ship clear fixes directly. Our Conversion Lab plan says which approach fits your store.
Start with the Free Profit Audit to see where conversion leaks, or book a call to plan an experimentation roadmap.