Conversion Lab · Experimentation

eCommerce A/B Testing That Measures Revenue

Experiments on product, cart and checkout with sample sizes set before launch, durations in full weeks and revenue per visitor as the judge. Fewer tests that mean something, instead of many that don't.

  • Sample size set up front
  • No peeking
  • Revenue per visitor first

Sizing a test

Most eCommerce tests are called too early

An A/B test compares a control with one or more variants on randomly split traffic. Whether it can detect a real difference depends on four inputs, all decided before launch:

  • Baseline rate: your current conversion rate on the page being tested.
  • Minimum detectable effect: the smallest lift worth knowing about.
  • Significance level: how much risk of a false winner you accept. 95% confidence is a common choice.
  • Statistical power: the chance of spotting a real effect when it exists. 80% is a common choice.

An illustrative example

With a 2% baseline conversion rate, detecting a 10% relative lift (2.0% to 2.2%) at 95% confidence and 80% power takes roughly 80,000 visitors per variant. If the tested page gets 20,000 visitors a week, split evenly, that test needs about eight weeks. Aim for a 20% lift instead and the requirement drops to about 21,000 visitors per variant, around two weeks.

That is why lower-traffic stores should test bigger, bolder changes, and why a test that looks like a winner after four days usually isn't.

The primary metric

Revenue per visitor, because conversion rate can lie

A variant can raise conversion rate by nudging shoppers toward cheaper products, or lower it while lifting average order value by more. Revenue per visitor (revenue ÷ visitors) captures both. It is our default primary metric for tests that can affect what people buy.

It has a cost. Revenue varies much more from visitor to visitor than a yes-or-no conversion, so tests on revenue per visitor need more traffic to reach the same confidence. We plan for that, and handle unusually large orders so one bulk buyer can't decide a test. A higher revenue per visitor also raises what your ads can afford per click; see the Break-even ROAS Calculator.

  • One primary metric chosen before launch, written into the test plan
  • Guardrails: contribution per visitor, discount usage, subscription opt-in
  • Conversion rate and AOV reported alongside, never instead

Rules we don't break

Six habits that keep test results honest

01

No peeking

Checking results daily and stopping when they look good sharply inflates false winners. We fix the sample size and duration up front, or use a sequential method built for early looks.

02

Run in full weeks

Shoppers behave differently on weekdays, weekends and paydays. Tests run in whole weeks, usually at least two, so each day of the week is represented.

03

Check the split

If a 50/50 test delivers a clearly uneven split, something is broken in assignment or tracking. We check for this sample ratio mismatch before reading results.

04

QA every variant

Each variant is checked on real phones and desktops, with and without consent, before traffic is sent to it. Broken variants produce meaningless losses.

05

Avoid promo noise

Big sales change who shops and why. Unless the test is about the promotion, we pause launches around major sales.

06

Treat segments as leads

A variant that only wins for one device or channel, found after the fact, is a new hypothesis to test, not a win to roll out.

What we test

Experiments big enough to change what people buy

Ideas come from the research in our UX & Checkout Review, not from a list of generic best practices.

AreaExample testsUsual primary metric
Product pageInformation order, image sequence, subscription as default vs one-time, review placementRevenue per visitor
Offer structureFree-shipping threshold, bundles vs percentage discounts, gift with purchaseContribution per visitor
CartCross-sell placement, threshold progress bar, express checkout positionRevenue per visitor
CheckoutField order, trust messaging, payment option order, where your platform allowsCheckout completion rate
Landing pagesChannel-specific pages for paid traffic vs the default collectionRevenue per visitor by channel

Price and shipping tests have legal and customer-trust implications. We agree the rules with you before running them.

Testing and research tools we work in

  • GA4
  • Convert
  • VWO
  • Optimizely
  • AB Tasty
  • Intelligems
  • Shoplift
  • Microsoft Clarity
  • Hotjar
  • Shopify Checkout Extensibility

FAQ

eCommerce A/B testing questions

How long should an eCommerce A/B test run?

Until it reaches the sample size calculated before launch, and in full weeks, usually at least two, so every day of the week is covered. Stopping when results first look significant inflates false winners. Running far longer than planned adds exposure to seasonal changes and cookie churn. We set the duration in the test plan and stick to it.

What sample size do I need for an A/B test?

It depends on your baseline conversion rate, the smallest lift you want to detect, and the confidence and power you choose. Smaller lifts need far more traffic. As an illustration, detecting a 10% relative lift on a 2% conversion rate at 95% confidence and 80% power needs roughly 80,000 visitors per variant. A 20% lift needs about a quarter of that.

Why use revenue per visitor instead of conversion rate?

Because conversion rate ignores order value. A change can raise conversion while shifting shoppers to cheaper products, or lower it while lifting average order value by more. Revenue per visitor combines both. It needs more traffic to read reliably, so we size tests for it and keep conversion rate and AOV as supporting metrics.

What does peeking mean in A/B testing?

Peeking is checking results repeatedly and stopping the test as soon as it looks significant. Random swings early in a test often look like wins, so stopping on them produces winners that don't hold up. The fix is to commit to a sample size and duration before launch, or to use a sequential testing method designed for repeated looks.

Can a low-traffic store run A/B tests?

Yes, with adjustments. Test bigger changes that could move results meaningfully, test on high-traffic templates like product pages, and accept longer durations. Where traffic is too thin to test, rely on research and ship clear fixes directly. Our Conversion Lab plan says which approach fits your store.

Run tests your team can trust

Start with the Free Profit Audit to see where conversion leaks, or book a call to plan an experimentation roadmap.

Free Profit Audit