Cold Email A/B Testing Best Practices
Want better email results? Explore these best practices for basic cold email A/B tests
Want better results from your cold email results?
A/B testing is one of the fastest tactics for figuring out what works and what doesn’t. And with the right approach, you can find a style that gets more clicks and replies.
But A/B testing only works if you set up and run your tests the right way. Many make simple mistakes that lead to unusable results and wasted time.
That’s why we’re breaking down how to A/B test cold emails and get results you can trust.
Basics of A/B testing for emails
Before we dive deeper into how to test cold emails, here’s a quick refresher on the basics of A/B testing.
What can you test?
Everything!
Every element of your cold emails can be tested, such as:
- Subject line – Long or short?
- Email body – Hard facts or funny memes?
- Call-to-action (CTA) – “Book a demo here” or “How about a call?”?
- Sender name – Use the English (Michael, Mike) or transliterated (Mikhail, Mykhaylo, Misha) version of your name?
- Sign-off – Formal (“Best regards”) or casual (“Best”)?
- Send time/window – Morning or afternoon? Weekday or weekend?
This list is far from exhaustive, but you get the point. Any aspect of your email can be tested, so long as you have two or more ideas for how to make it more effective.
Testing principles
For best A/B testing results, you should try to follow these principles:
- Test only one variable at a time – Your A and B email versions should be identical, except for the element you’re testing. For example, if you’re testing CTAs, then the email body and subject line should be the same.
- Set a clear goal and success criteria – Do you want more opens, clicks, replies, or conversions? What difference will you consider meaningful: 5%, 10%, or 20%?
- Analyze and iterate – After completing the test, check which variation performed better and use what you learned in your future emails (and next A/B test 😉).
Statistical significance
Say Version A of your email gets 15% more opens than Version B.
Is it due to the change you made or something else?
Mathematically speaking, the answer depends on the statistical significance of your test, which is calculated from the p-value. If your p-value is 0.05, your statistical significance is 95%.
In other words, you can be 95% confident that the difference in your email performance is not due to chance.
The reason statistical significance is important is because it tells you whether the better-performing email version is actually better.
While you can use statistical software and A/B testing calculators to compute p-values, for strong statistical significance, you basically need:
- A large enough sample size
- A clear goal
- Relevant metrics
- Appropriate testing duration
- Adjustment for external factors
And don’t worry. We’ll cover these later 🙂
Sample size requirements
Generally, you need at least 1,000 recipients per version, which means you need a sample size of 2,000 people if you’re testing two versions. One half should receive Version A while the rest should receive Version B.
But if you expect a large difference or you want to confirm a basic hypothesis, you can sometimes get away with a smaller sample size, such as 100–200 recipients per variation.
You can also use an A/B test calculator to find out how many recipients you’ll need for good results.
What factors can impact A/B tests?
Email test results can get skewed by external factors such as:
- Seasonality – People will respond to the same email differently during major holidays or high/low season for the product you’re selling.
- Email frequency – If you send a lot of emails, your recipients might develop “subscriber fatigue”, leading to lower engagement.
- Audience – Various demographics and interest groups can have inherent preferences that differ, so what works for one may automatically not work for the other.
To minimize the risk of skewing testing results, you can:
- Choose an appropriate testing time – Avoid running A/B tests during major holidays or a clear high/low season for your product, unless you want to test email performance specifically during these periods.
- Target only cold leads – To test cold email performance, only use recipients you haven’t engaged before. Granted, if you use B2B sales channels like social media, it’s impossible to be certain that all recipients are fully cold and have never heard of you.
- Segment your audience – Make sure all the recipients in your sample share key demographic and behavioral traits.
How to set up an email A/B test
Setting up an A/B test is fairly simple and straightforward. All you have to do is follow these steps.
Choose your variable
Choose the email element you’ll be testing, and make at least two versions of it. Keep the other email parts exactly the same.
If you’re new to A/B testing, you might want to start with testing email subject lines. They have the biggest impact on whether someone opens your email. After all, if people don’t open, they’ll never see the rest of your message (and you’ll struggle to test other cold email elements if people aren’t opening).
For instance, our CEO ran one A/B test that boosted open rates by 16%.
Determine your sample size
A bigger sample is better, but don’t forget you’ll need some leads for your future tests.
If you’re concerned about running out, you can limit your sample size to a few hundred recipients per variation.
But you might use a calculator to ensure your test will still be meaningful.
Decide your test duration
It will usually take several days for people to see your email and respond to it. A smaller sample may take longer.
You’ll likely want to run the test for at least 1–2 weeks, but no longer than 2 months. Any longer, and seasonality might start to affect your results.
Set up your segmentation
Decide which leads you’ll include in the test. You might want to include prospects from a particular industry, location, or lead data source.
Break your whole lead list up according to these criteria.
Make sure that each lead is included in one segment only. Overlaps can reduce the reliability of your results.
Configure your control group
The control group receives Version A of your email, while the experimental group receives Version B.
Make sure you distribute recipients randomly between groups to avoid selection bias. You can use a randomizer or random sort function to get this part done.
Figure out your timing and sending window
Usually, you’ll want to schedule both email versions to send at the same time, unless you’re A/B testing the sending window itself. In that case, remember to segment carefully so your target time corresponds to your lead’s time zone.
Ideas to A/B test in cold emails
Like we mentioned, you can A/B test any part of your cold email, whether it’s something small like spintax variations or word choice or something bigger like your email framework, style, or tone.
Here are a few concrete examples of potential A/B tests you could run on your cold outreach.
Subject line
A subject line has the biggest influence on your open rate, which is why it’s a good idea to test it first and build your future tests from there (because you can at least be confident people are opening and starting to read).
| Test variable | Variation A | Variation B |
| Length | 10-12 words | 5-6 words |
| Personalization | Mike, learn how ABC Corp can boost outreach! | Learn how your company can boost outreach |
| Question/statement | Want to learn how ABC Corp can boost outreach? | Learn how ABC Corp can boost outreach |
| Emojis | An emoji at the end | No emoji |
| Capitalization | Learn how ABC Corp can boost outreach | Learn How ABC Corp Can Boost Outreach |
When our CEO ran an A/B test for AiSDR cold outreach, his hypothesis was that shorter, personalized subject lines written in sentence case would deliver higher open rates. (Spoiler alert: His hypothesis was confirmed).
But what worked for us might not for you. That’s why you should run an A/B test.
Email body
Your email body is the biggest part of your email, so it has the most room for experimentation.
Some good starting points for A/B testing are email personalization techniques and possible value propositions.
| Test variable | Variation A | Variation B |
| Length | 250 words | 150 words |
| Personalization level | Segment-based | Hyper-personalized |
| Personalization type | Company | Person |
| Value proposition location | At the beginning | At the end / PS |
| Pain point | Revenue growth | Cost cutting |
When split-testing emails, body will probably take the longest to build, but the result will be worth the effort.
Another option for testing email bodies is to try various frameworks to see which ones your audience is most receptive to, such as:
- Justin Michael Method
- Yurii Veremchuk
- Founder outreach
- AIDA (Attention, Interest, Desire, Action)
- Jobs-to-be-Done
- 3-B (Brief, Blunt, Benefit)
CTA
Your calls-to-actions are arguably the second most impactful element after the subject line. They tell the recipient what to do next (if you’re using the imperative) or offer an opportunity (if you use a question-based CTA).
| Test variable | Variation A | Variation B |
| Type | Button | Text |
| Placement | In the middle | At the end |
| Link | Hyperlink | No link |
| Number | One CTA | Multiple CTAs |
| Wording | Imperative: Book a call | Question: How about a quick call? |
Even a minor and seemingly insignificant change to the CTA can dramatically improve your outreach effectiveness.
Sign-off and signature
It’s easy to think that no one pays attention to your sign-off and signature, but they still influence the impression you leave. You might want to find out whether a formal or informal “goodbye” will deliver better results, or how just a first name compares to using a full name and title.
If you’re especially curious, you might even try variations of your name’s spelling for different regions. For instance, will people respond more willingly to Michael or Mikhail?
Sending window
Testing the timing of your emails is straightforward. You choose two times (10 a.m. vs 4 p.m.) or two days (Tuesday vs Thursday) and send identical emails to see when you get better engagement. However, you might have to run a lot of these tests before you find the best time slot!
So what does “better engagement” really mean? The answer lies in your numbers.
Metrics to track
For best testing results, you should track the email outreach metrics that align with your goals.
Open rate
Open rate is the percentage of recipients who opened your email.
A higher open rate means more people are seeing your message. Without a decent open rate, all the effort your team puts into crafting a perfect cold email is wasted. That’s why we recommend always keeping an eye on your open rates and targeting them first for effective cold emailing.
Clickthrough rate
Clickthrough rate (CTR) is the percentage of recipients who clicked on the link in your email.
CTR measures how likely your recipients are to take the action you’re nudging them toward, whether it’s visiting your website, posting a review, or booking a meeting. This metric is indispensable for measuring CTA efficiency.
Response rate
Response rate is the percentage of recipients who replied to your email.
Measuring response rate is a must for emails that invite people to get in touch. It’s a key indicator of how successfully you pull them into a conversation.
Positive response rate
Positive response rate is the percentage of recipients who responded positively, for example, by showing interest in your product or asking for more information.
This metric goes beyond just getting a reply. It captures how many responses come from leads showing interest. You’ll want to use it instead of response rate if you prefer to be more selective in your outreach.
Conversion rate
Conversion rate is the percentage of recipients who took the desired action, such as signing up for a demo or filling out a form.
Conversion rate optimization is the ultimate goal of many cold email campaigns. Tracking your rate helps you see how effectively you can get leads to do what you want them to.
These five essential metrics can show what is and isn’t working for your emails. But if you want to find out why an A/B test went wrong, you’ll need to check your test for common mistakes.
Common A/B testing mistakes
Running A/B tests might seem straightforward, but there are many ways a test can go wrong.
Insufficient sample size
You won’t get reliable results with just a few dozen recipients in a sample. Usually, you’ll need at least several hundred recipients per variation: use an online calculator to get a precise recommendation based on your target.
Monitor early performance to see whether your sample works out as intended.
Testing multiple variables
If you’ve changed both your email body and CTA and observed a 20% difference in conversion rate, how do you know which change contributed more to this result? The short answer is: you don’t.
There are special techniques to test multiple variables, but if you’re just starting out with A/B testing, we recommend keeping things simple and testing one variable at a time.
Ending tests too early
You’re a few days into your test, and you see that people strongly prefer one variation of your email. Should you stop the test?
Well, no.
Early results can be biased, as many respondents won’t have seen and read your email. Let the test run its course to get the full picture of what your audience prefers.
Ignoring statistical significance
Not all A/B testing results should be acted upon. Remember that statistical significance is how you know when your A/B variation (as opposed to other factors) is influencing the results.
Make sure to track the statistical significance of all test results. Look for p-scores of 0.05 or lower: these indicate at least a 95% confidence that the difference isn’t due to chance.
Poor hypothesis formation
What exactly are you testing? What result do you expect and why? Make sure to formulate that as a clear, testable hypothesis that aligns with your goals.
Bad hypothesis:
“I believe using a personalized subject line will increase open rates.”
This hypothesis is too vague and nonspecific. If there’s a difference of 1%, should you take action based on it? This hypothesis doesn’t address that.
Good hypothesis:
“I believe using a personalized subject line will increase open rates by 10% because personalization is proven to engage users.”
This wording makes it clear what difference you expect to discover and why. If the actual difference is 10% or more, you can state that the hypothesis was confirmed and start using personalized subject lines in your emails to improve open rates.
And naturally, you can make your hypothesis even more specific by outlining the type of personalization that will trigger the results you expect.
Bad segmentation
When you bundle all your leads together, you can’t test how prospects with particular traits, such as location or industry, respond to a particular change.
To avoid this, segment your recipients by clear criteria. If you’re unsure about segmentation, start broad, using only one or two criteria. Over-segmenting will give you samples too small to be reliable.
How to set up a cold email test
Here’s a quick rundown of how to carry out a simple cold email A/B test.
Decide what to test
What’s the biggest problem with your cold emails right now? Are people not opening them? Are they not clicking the links or not converting at a good rate?
Choosing the right thing to test is A/B testing 101. If you want to improve open rates, focus on testing subject lines. For CTR and conversion, begin with email length and CTAs. After that, you can move to more nuanced A/B testing strategies targeting personalization, sending time, and value proposition.
Set your timeframe
Establish a clear timeline for each test. Allow enough time for respondents to react, but don’t drag your tests out unnecessarily. A few weeks for the test itself is a good rule of thumb.
Use a spreadsheet to map the start and end dates, target segments, and variables being tested. Avoid running too many tests at once so you don’t overwhelm yourself, and be prepared to adjust your timeline based on user feedback or unforeseen circumstances.
You should also build in some time for email performance analysis after the test.
And don’t just glance at the numbers. Dig deeper to understand the “why” behind the results.
Allocate team resources
Even if you’re using an advanced AI-based tool like AiSDR to run your A/B tests, it’s not (yet) possible to take a human completely out of the process.
Appoint team members who have the skills and knowledge to design tests and analyze their results.
Act on results
Take action based on statistically significant A/B test results. Update your subject lines, email templates, and sending strategies accordingly.
Keep a record of all test results, including both successful and unsuccessful tests. Use this knowledge base to plan your next email campaigns and tests, avoiding the same mistakes.
Give access to all sales outreach team members so everyone stays updated and on the same page.
Schedule follow-up A/B tests
When you get a clear variation winner, plan your next tests to optimize it. For example, if a shorter subject line performed better, test personalized and non-personalized variations of shorter subject lines.
Set a time to regularly review your testing plans. It’s essential to keep exploring new ideas and testing new hypotheses, as email marketing changes fast.
Add feedback loops
Include the metrics you’ll track in each test’s timeline and description, and follow them consistently.
You can also encourage recipients to provide feedback. For example, you can ask recipients to tap a thumb up/thumb down at the end of an email to show how it landed. This feedback can help you plan future campaigns and test more effectively.
