Every A/B test has noise. Even if two versions of a product page were identical, their conversion rates would never match exactly, because visitors differ day to day. Statistical significance is a way of judging whether the gap you see is bigger than that normal noise.
Most tests use a p-value. It answers: if there were truly no difference, how likely is a result at least this extreme? A common threshold is 0.05, often described as 95% confidence. A p-value below 0.05 means the result would be unusual if the change had no effect, so you treat the effect as real. It does not mean there is a 95% chance the variant is better, and it says nothing about whether the difference is large enough to matter.
Why it matters for eCommerce
Without significance testing, teams ship changes that did nothing, or roll back changes that worked, based on a few days of random swings. That wastes design and development time and can quietly lower conversion rate. Significance gives a consistent rule for deciding which results to trust.
How it works in practice
Set the sample size before the test starts, based on your baseline conversion rate and your minimum detectable effect. Run the test for full weeks so weekday and weekend behavior are both included. Check the result once, at the planned end. For example (illustrative numbers): control converts at 2.0% and the variant at 2.3%. At a small sample the p-value might be 0.30, meaning the gap could easily be chance. At a much larger sample the same gap could fall below 0.05.
Common mistakes
- Peeking daily and stopping the moment the result turns significant, which greatly increases false positives.
- Testing many metrics or variants and celebrating whichever one crosses the line.
- Confusing significant with important. A tiny, real lift may not pay for the change.