- Apple Mail Privacy Protection records opens for emails nobody read, so open rate can no longer pick a winning subject line. Judge tests on click rate, placed order rate and revenue per recipient.
- The subject lines that pay are specific: they name the product, the reason to care and anything time-bound, and they match what the email actually contains.
- Stop fake "RE:" lines, constant urgency, discount codes in every subject and tests on audiences too small to give an answer. They cost trust, margin or both.
Why can’t you judge subject lines on open rate anymore?
For years, the subject line test was simple: send two versions, pick the one with more opens. That logic broke in late 2021, when Apple’s iOS 15 introduced Mail Privacy Protection (MPP). According to Klaviyo, Apple pre-downloads email content to its servers for users with MPP turned on, and that process counts as an open. A subscriber who never looked at your email can still show up as an opener.
The result is that open rate now mixes real readers with automated activity. Klaviyo’s own A/B testing guide warns that open-rate tests count recorded opens, including MPP opens, so a winner may reflect automated opens as well as real engagement. If a large share of your list reads email in Apple Mail, the noise can be bigger than the difference you are trying to measure.
Klaviyo lets you choose open rate, click rate or placed order rate as the winning metric for a campaign A/B test (placed order rate needs a Placed Order metric in the account). For subject lines, we recommend click rate as the automatic winner and revenue per recipient as the number you report. Opens can still help as a rough health check over time, but not as the judge of a test.
What makes an eCommerce subject line work?
A subject line has one job: get the right people to open an email that then earns a click. It is not there to maximize curiosity. A line that tricks people into opening and then disappoints them produces opens, no clicks and, over time, more unsubscribes and spam complaints.
The subject lines that tend to pay share a few traits:
- They are specific. “The linen shirt is back in 6 colors” beats “Something you’ll love.” Specific lines attract people who want that thing.
- They put the important words first. Mobile inboxes show fewer characters than desktop, so lead with the product or the reason, not with your brand name or a greeting.
- They work with the preheader. The preheader text is the second line of your pitch. If you leave it blank, many inboxes fill it with the first text in the email, which is often “View in browser.”
- They match the email. If the subject promises a restock, the first screen should show the restock.
- They sound like your brand. A premium skincare brand and a discount gadget store should not write the same subject lines.
The sender name matters too. Subscribers decide whether to open based on who sent it as much as what it says. Keep it consistent and recognizable, and test changes to it rarely and carefully.
Which subject line tests are worth running?
Good tests answer a question you can reuse. “Did version B win on Tuesday?” is not useful. “Do our subscribers respond better to product names than to benefits?” is, because it shapes the next 50 subject lines.
| Test | Example A | Example B | What you learn | Judge on |
|---|---|---|---|---|
| Product vs. benefit | “New: the Trail 2 running shoe” | “Lighter shoes for longer runs” | Whether your list buys on product or outcome | Click rate, revenue per recipient |
| Price shown vs. hidden | “Linen shirts from $48” | “Linen shirts for summer” | Whether price helps or filters out buyers | Placed order rate |
| Urgency vs. none | “Last 2 days: free gift with $75+” | “Free gift with orders of $75+” | Whether deadlines lift orders or just opens | Revenue per recipient |
| Personal vs. general | “Back in your size: the Oxford” | “The Oxford is back in stock” | Whether personal data earns its effort | Click rate |
| Short vs. descriptive | “Restocked.” | “Restocked: the 3 colors that sold out in May” | How much context your list needs | Click rate |
| Emoji vs. none | Line with one emoji | Same line without | Whether emoji help or look cheap for your brand | Click rate, unsubscribe rate |
Run each idea more than once before you treat it as a rule. One win can be luck, a holiday or an unusual product. Three wins in the same direction start to look like a pattern.
What not to test
Skip tests where the answer would not change anything. Testing “Shop now” against “Shop today” is unlikely to produce a measurable difference on a mid-size list. Spend your test slots on bigger ideas: the angle, the offer framing or the level of detail.
What should you stop doing?
Some habits make subject lines look better in the short term while costing trust, margin or deliverability.
Fake replies and forwards. Adding “RE:” or “FW:” to a marketing email to imply an ongoing conversation is deceptive. The FTC’s CAN-SPAM compliance guide says not to use deceptive subject lines and that the subject line must accurately reflect the content of the message.
Urgency in every send. If every email says “last chance,” none of them are. Subscribers learn to ignore it, and some learn to wait for the next “final” sale. Keep urgency for real deadlines.
A discount in every subject line. Putting “20% off” up front can lift clicks, but it also trains your list to wait for a code and lowers contribution margin on orders that would have happened at full price. Our guide to eCommerce discount strategy covers when an offer earns its cost.
All caps and stacks of punctuation. They look like spam to people and add nothing a clear line cannot do.
Declaring winners on small tests. A 5% test cell that produced 40 clicks against 34 clicks has not told you anything. More on that below.
Writing the same line for everyone. A subject line about a restock works better when it goes to people who viewed or bought that product. Often the biggest gain is not a better line but a better audience, which is where email segmentation pays off.
How do you run a subject line test that gives a real answer?
A subject line test is a small A/B test, and the same rules apply.
Test one thing at a time
Change only the subject line. Keep the preheader, sender name, send time and content the same. If you change two things, you will not know which one mattered.
Make the test big enough
Click and order rates are low, so you need large samples to see a real difference. Before you test, decide the smallest lift you care about. That is your minimum detectable effect. If your list is small, test only bold differences, or send each version to half the list and treat the result as a lesson rather than a race for the remainder.
Check statistical significance before you call a winner. Our guide to A/B testing for eCommerce explains sample sizes in plain terms.
Give it enough time
Clicks and orders arrive more slowly than opens. A test that picks a winner after one hour mostly rewards whoever reads email during that hour. For a test that chooses a winner automatically, allow several hours, and judge the final result on orders and revenue after a few days.
Keep a test log
Write down every test: date, audience, both lines, the metric used, the result and what you concluded. After a quarter, the log becomes your brand’s subject line playbook.
Worked example: when the “winner” loses (illustrative numbers)
All numbers below are illustrative.
A store sends a campaign to 60,000 subscribers, split evenly. Version A reads “The linen shirt is back in 6 colors.” Version B reads “You’re going to want this.”
| Metric | Version A | Version B |
|---|---|---|
| Recorded open rate | 44% | 48% |
| Click rate | 2.4% | 1.9% |
| Placed order rate | 0.62% | 0.47% |
| Revenue per recipient | $0.31 | $0.24 |
| Unsubscribe rate | 0.15% | 0.22% |
Judged on opens, B wins. Judged on what matters, A wins clearly. The vague line drew curious openers (and possibly more automated MPP opens), but the specific line drew people who wanted a linen shirt. At 30,000 recipients each, A produced about $2,100 more revenue, and fewer people left the list.
If this store had let open rate pick the winner automatically, the remaining send would have gone out with the weaker line. That is why the winning metric is the most important setting in any subject line test.
Subject line testing checklist
- [ ] Set click rate or placed order rate as the winning metric, never open rate.
- [ ] Decide what question the test answers and how you will reuse the result.
- [ ] Change only the subject line; keep preheader, sender, timing and content identical.
- [ ] Confirm the audience is big enough for the lift you care about.
- [ ] Allow enough time for clicks and orders before a winner is chosen.
- [ ] Write a preheader that adds information instead of repeating the subject.
- [ ] Check that the first screen of the email delivers what the subject promises.
- [ ] Track unsubscribe and spam complaint rates for each version.
- [ ] Log the test and repeat promising ideas before turning them into rules.
FAQ
Should we still look at open rate at all?
Yes, as a broad trend. A sudden drop in opens across all campaigns can point to a deliverability problem worth investigating. Just do not use it to compare two subject lines or to pick test winners, because automated opens can swing the result.
How long should an eCommerce subject line be?
There is no single right length. What matters is that the key words come first, since mobile inboxes cut lines short. Test a short version against a descriptive one with your own list, and judge on clicks and revenue.
Do emoji in subject lines help?
It depends on your brand and audience. For some lists one emoji adds visual contrast. For others it looks cheap and lifts unsubscribes. Test it a few times, judge on click rate and unsubscribe rate, and avoid using emoji as a substitute for a clear message. To see what better campaigns are worth to you, try our email revenue calculator.