Email Marketing

Email A/B Testing for eCommerce: What to Test and How to Read the Results

Most email A/B tests are too small to mean anything. Learn what to test, how big your test needs to be and how to judge winners by revenue, not opens.

TL;DR
  • Test things that can change revenue: offers, content, timing and flow structure. Subject line tweaks rarely move the business.
  • Judge tests by clicks, conversion and revenue per recipient. Opens are unreliable because of privacy features.
  • Small lists need bigger differences and longer tests. If a result is not significant, treat it as "no clear winner", not a win.

Why do most email A/B tests fail to teach anything?

Email platforms make testing easy: pick two subject lines, send to 20% of the list, let the winner go to the rest. The problem is that most of these tests:

  • Measure opens, which are inflated by Apple Mail Privacy Protection and other privacy features.
  • Use samples too small to separate a real difference from random noise.
  • Test details that would not change revenue even if they “won”.

The result is a long list of “winners” that never add up to better results. Good A/B testing for email starts with better questions.

What should you test in eCommerce email?

Prioritize tests by how much they could change revenue per recipient.

Test area Example Potential impact
Offer 15% off vs free gift vs no offer High
Flow structure 2-email vs 3-email cart flow, different delays High
Content type Product grid vs story-led email Medium to high
Product selection Bestsellers vs personalized recommendations Medium to high
Send time and frequency Morning vs evening, 2 vs 3 campaigns a week Medium
Call to action “Shop the collection” vs “Find your size” Low to medium
Subject line Question vs statement Low on revenue, useful for engagement

The highest-value tests are usually in automated flows, because a winning change keeps paying off every day. A Klaviyo flow audit often reveals which flows deserve testing first.

Which metric should decide the winner?

Pick the metric before the test starts, based on what you changed.

  • Subject line or sender name: clicks per recipient (better than opens), and check for spam complaints.
  • Content, layout, product selection: click-through rate and conversion rate.
  • Offer: revenue per recipient and, ideally, gross profit per recipient. A bigger discount often wins on orders and loses on profit.
  • Flow timing and structure: revenue per recipient over the whole flow, not per email.

Avoid declaring a winner on opens. With a large share of opens being machine-generated, open differences can appear without any real difference in human behavior.

How big does an email test need to be?

Size depends on your baseline rate and the difference you hope to detect. The smaller the difference, the more recipients you need.

Worked example (illustrative numbers): a store’s campaigns convert about 0.15% of recipients. To detect a 20% relative improvement (0.15% to 0.18%) with reasonable confidence, you need a very large sample, typically several hundred thousand recipients per variant. Most stores do not have that.

What this means in practice:

  • Smaller lists should test bigger changes (offer type, content approach) where differences are large enough to see.
  • Flows can run tests for weeks, accumulating enough recipients over time.
  • Clicks happen more often than purchases, so click tests reach a reliable answer faster. Use them as an early signal, then confirm with revenue over longer periods.

Learn the concepts of statistical significance and minimum detectable effect before running many tests. Many email tools show a “winner” without telling you whether the difference is meaningful.

How do you run a clean test?

  1. Write a hypothesis. “Showing the three bestsellers in the cart email will raise revenue per recipient because shoppers see social proof”, not “let’s try a new layout”.
  2. Change one main thing. If you change the offer, layout and subject line at once, you will not know which mattered.
  3. Split randomly. Use the platform’s random split. Do not compare this week’s email with last week’s.
  4. Set the duration and size in advance. Do not stop the test the moment one version pulls ahead.
  5. Use a holdout where possible. For flows, a small holdout group that receives no email shows whether the flow works at all, not just which version is better.
  6. Record results. Keep a simple log of hypothesis, variants, sample size, result and decision.

Worked example: testing a cart abandonment offer

(Illustrative numbers.) A skincare store’s cart flow sends a 10% discount in the second email. The team wants to know if the discount is needed.

Test: version A keeps the 10% code, version B replaces it with a free sample and stronger reviews. The test runs for six weeks in the flow, with about 9,000 recipients per version.

Metric A (10% off) B (free sample)
Click rate 8.1% 7.6%
Conversion rate 4.2% 3.9%
Revenue per recipient $2.73 $2.62
Gross profit per recipient $1.36 $1.57

Version A wins on revenue, but version B wins on gross profit because it avoids discounting every order. The team keeps B and tests a third version with free shipping. For more on recovering carts without discounts, see abandoned cart emails without discounts.

Checklist: email A/B testing

  • [ ] Each test has a written hypothesis
  • [ ] Only one main element changes per test
  • [ ] Winning metric chosen before the test starts
  • [ ] Opens are not used as the main success metric
  • [ ] Sample size and duration set in advance
  • [ ] Recipients split randomly by the platform
  • [ ] Offer tests judged by gross profit, not just revenue
  • [ ] Flow tests run long enough to collect meaningful volume
  • [ ] Holdout group used where possible for flows
  • [ ] Results logged with the decision taken

FAQ

How long should an email A/B test run?

For campaigns, wait long enough for most clicks and purchases to arrive, usually 24 to 72 hours. For flows, run the test until both versions have reached the sample size you planned, which can take several weeks.

Is subject line testing worth it?

It can improve engagement, but its effect on revenue is usually small. If your list is limited, spend your testing capacity on offers, content and flows, where the potential gains are larger.

What if neither version wins clearly?

Then the honest result is “no meaningful difference”. Keep the simpler or cheaper option and move on to a test with a bigger potential impact.

Turn this into a plan for your store

The Free Profit Audit scans your store, sizes your biggest profit leaks and builds a 90-day plan.

Free Profit Audit