How to Do A/B Testing the Smart Way: Your Most Pressing Questions Answered

In this step-by-step guide to A/B testing, you’ll discover a wealth of useful information on how to test your store’s user experience and triple your company’s conversions. 

Diving into eCommerce is easy. But don’t get caught by the misconception that setting up an online business is a one-time investment. And I’m not talking about the money side of things. It’s more about your personal input measured in the time spent, and effort put in. 

You have to be experimenting all the time, testing everything that might positively affect your conversions — play around with the layout elements, check the spamminess of your emails, and so on. Because without it, you’ll be playing a guessing game trying to figure out what your audience is fond of and what encourages them to run away from your site. 

Below the top things to understand in A/B testing: 

• Why should you be planning to run A/B tests for your online store?
• What can we test? And what we should NOT test? 
• What results are we about to achieve implementing this strategy? 
• What are some of the things to consider before you start spying out the land? (Yeah, we should all be a little bit like spies to know your shoppers better and be able to predict their spending habits.)
• Are there any A/B testing tools that you can use to be more productive? 
• And finally, how do you analyze your A/B testing metrics and take action?

But, before we go any further, what is A/B testing? 

Now that you know what is in store for you, grab yourself a cup of coffee and get comfy, because this guide is your life for the next 20 minutes. 

Ready? Let’s take a closer look at what’s inside. 

I. Why Do You Really Need to Do A/B Testing?

The benefits of A/B testing are pretty obvious, aren’t they? These are increased traffic to your website, better conversions, lower bounce rates, improved user engagement, reduced cart abandonment, just to name a few. I’m sure you know them already.

But my gut says that there’s still something that holds you back from all of the good stuff that awaits you in the world of A/B testing. You feel a bit skeptical about this strategy, don’t you?

I know the feeling. And this is why I won’t elaborate on the things that may seem obvious, but rather focus your attention on some of the most crucial benefits that will push your business toward A/B testing.

Without further ado, here are my ideas: 

1. The Results Will Come in Quickly

As a business owner, you are likely to be too pressed for time to opt for yet another strategy, especially if you are not 100% sure about the outcome. 

This is not the case for A/B testing. 

Even a relatively small change can provide visible, actionable results. Bonus: you can do it with zero or little investment. 

This makes this strategy a good fit both for small businesses running on a tight budget and for enterprise ones having a lot of plates spinning in the air. 

2. Change Is the Only Constant; You Should Be Part of It

Neither the community you serve as a business owner nor your competitors will let you stay put. You’ve got to be hustling, serving your customers on the go, and fulfilling their most exquisite needs. 

Have you already noticed that vanishing content is becoming more and more trendy? 

• Stories on Instagram vanish after 24 hours.
• Snapchat messages disappear without any trace once they catch your eye.t

Immediacy brings about the highest engagement, as it feels more authentic and secure. And business owners have to adjust to the changing environment.

The same about landing page design, which is never static. We just optimized our websites to perform nicely on mobile devices, keeping things simple and consistent, when another trend came over — now you have to get your customers involved with interactive elements?! 

Could you have predicted this shift with A/B testing? You bet! 

You have to continuously adjust your website as well as the overall strategy to your changing environment. 

3. Avoid Ineffective Decisions Before It’s Too Late

Ideally, all the changes to your eCommerce website structure and layout should be well informed. But it’s in an ideal world. We all know how it happens in real life — some changes are implemented in a hit and miss fashion. 

Following your gut is good, but basing marketing decisions around data and A/B tests is a less risky investment.

Just think about it for a moment. You don’t want to lose your revenue because of a tiny discrepancy between your inner sense and your customers’ expectations.

Additional Resources: 
The Importance of A/B Testing: 10 Benefits of AB Testing 

II. Types of A/B Testing

There are a number of different elements that you can test using the A/B testing technique — from call-to-action buttons to the navigation paths. 

But all these types of A/B tests will ultimately come down to just a few. These are:

1. Element-Level Testing

This is the most primitive A/B test, one that doesn’t require too much of an effort. You test the variations of various elements of a single page, such as images, headlines, CTAs, and the like.

• Call-to-actions. You can test different variables, including color, size, shape, position, wording, and even special effects.
• A/B headline testing. This is what your potential customers see first thing when entering your site, seeing it in search results, or receiving your email. Hence, the quality of your headlines may directly impact your email open rates, Google Ads impressions, the effectiveness of your entire landing page, and so on.  
• Visual content, like images, videos. Can infographics, webinar recordings, or podcasts be compared against the plain text? You bet! Can we A/B test a longer infographic against a shorter one? Of course. 
• Product descriptions. By testing the length and design of product descriptions, you can significantly reduce your abandoned cart rate. This is especially true for Amazon product detail pages, where the options for customization are pretty limited.

The biggest misconception about element-level testing is that you should test one element at a time. Don’t get me wrong. It will definitely help you figure out which particular change makes a difference in your lead conversions.

But what’s in it for you if you don’t see any significant revenue growth? Don’t get me wrong, it’s a good idea, but it won’t get you anywhere. Minor changes generally lead to inconspicuous results. 

Even worse, if you only keep on trying small element testing, inconclusive results and puzzling outcomes will soon discourage you and push you to quit your entire testing program. 

This brings us to the next point.

2. Page-Level Testing

Page-level testing, also called multivariate testing, presumes that you are testing several elements at the same time. 

At this level, you compare different page layouts or several combinations of elements on the same page. This type of testing is proved to be more effective, though rather complicated.

• Testing the overall look and feel of your landing pages, you can potentially accomplish a 3-5x boost in conversion rates, which is A LOT.  
• A/B testing your content depth. Some users are too reluctant to dig deep into product specifications and are good with concise overviews. Others want a deep dive that describes the product in painstaking detail. 

Note: Multivariate testing is recommended only for the stores with a large number of daily visitors. Otherwise, it may take ages to obtain meaningful data from the test.

3. Visitor Flow Testing

Gee, that’s the tough one. I promise. 

It implies testing several navigation paths for your customer around the website. 

For example, you test two different types of navigation menus, or product filters, or categories. Alternatively, you can compare a one-step-checkout with a multi-step one.

This sort of testing brings about major changes to your store’s look and feel, to say nothing of the amount of effort it requires from the development team to make. 

However, done right, it can have the highest impact.

4. Overall Strategy Testing

You can infinitely change the color of your CTAs and follow your customers’ steps to find out what it is that vows them most of all. But what if the issue is way deeper than you think? What if you should change the approach to business development? 

• SEO A/B testing. Here at X-Cart, we are constantly trying new techniques in SEO. As of now, content syndication is in progress, which is showing some promising results already.
• Social proof testing. It’s not a secret that people trust online reviews as much as friends. (So naive of them.) But what type of social proof works best for your eCommerce business? Reviews on customer review websites like Trustpilot or Shopper Approved? Social shares? Media mentions? Or any other sort of user-generated content? That’s up to you to find out.

III. Essential Things To Consider Launching an A/B Testing Campaign

As a business owner, you may already have a bunch of rules that you follow running your A/B testing campaigns. You weren’t born yesterday, after all. And even if you are new to the party, your gut probably tells you that: 

• You should split your test groups equally to get the most accurate results.
• You should allow your test to run for at least a week (or better a month) to bring any notable results. If there is no winner after one month, kill it.
• If a test drops below a 10% lift in conversions, kill it, as well.
• Testing email variables makes sense only if you have at least 5K recipients to get the most useful data. (That’s what Mailchimp suggests for its users, too.)
• Content marketing takes between 100-150 days to work. But who would expect it to work faster?
• Using A/B testing tools, like Google Optimize, will simplify the process. Really? LOL. (Scroll down the page to see more tools>>)

I won’t kill your time describing the hackneyed rules for A/B testing that are copied and pasted over and over again on multiple online resources. You know them already. I’d rather give attention to the less evident things related, say, to buyer psychology or silly mistakes that always get overlooked.

Off we go! No fluff. I promise. 

1. Tie Your Website Together With a Strong Narrative

If you are looking to go from 0.001% to 11% conversion rate, you must put together a solid and coherent content strategy. 

This may sound like an obvious thing to go for, but it is often overlooked and buried deep under the massive A/B and split testing campaigns. 

Don’t rush to spend thousands of dollars on landing page redesign, content updates, and navigation re-make. Take a moment to ponder on the following questions instead:

• What story do you communicate to your visitors?
• Are you absolutely sure what your buyer personas are? Is there anything about your website that might bother them? (You can even ask them directly, but wait, one thing at a time.)
• What techniques do you use to retain existing customers? Are they effective?
• What impression do you want to make at the moment they land on your site?
• Are you vowing your website visitors? If yes, how?

2. Do Not Run A/B Tests for the Sake of Data

The A/B testing is not about the amount of data collected but about the quality and transparency of the results. 

If you launch a new A/B test every other day, it means you might not pay due consideration to the analyses of the results. Or vice versa: If you spend more than 20% of your time doing pure analytics reporting, then you are wasting your company’s time.   

What I know for sure is that there should be some kind of a balance between analyses and the number of tests.

Your goal is definitely NOT just collecting raw data. You want to nail it. But don’t push yourself too hard. It won’t bring you any good either.

Run another A/B test, maybe?

3. Pre-Test Data Should Be at the Heart of A/B Testing

Before you put any effort into A/B testing, you should make sure that there is actual scientific research behind your theory that backs up your test. Otherwise, you are just guessing.

What’s in it for you? 

1. User feedback is essential. Without it, you’re just chasing your own shadow.
2. Take a closer look at the stats collected by other businesses in your niche. Did you know that a staggering 38% of the so-called successful tests are proved to be false positives? Knowing that can hopefully help you interpret the results correctly.
3. Segmentation and personalization is a must as we are all different, and so are your customers. For the experiment to bring the most plausible results, segment your data before and after the test. 
4. Take a closer look at the data you already have. Analyze your heatmap. (We use CrazyEgg to accomplish this). Dig a bit deeper into Google Analytics’ data. 
5. Talk to your CRO manager, after all, if you are not the one yourself 🙂 They may share some valuable insights with you on the previous tests.

4. Be Ready to Fail (Cause Sometimes It’s Inevitable)

Fear of failure is just your ego talking. Don’t be afraid to fail. But be ready for it. 

Even if your A/B testing experiment is technically a winner, it doesn’t mean that your customers won’t curse.

Winning designs can lose when deployed to production or on follow up validation tests. This could be due to false positives or just due to some kind of a ‘novelty effect.’

If your experiment failed to deliver a eureka moment, get over it, and try again.

5. Get Down to Polls and Ask Your Customers What They Think

Not only can you test various elements of your webpage or email. You can also experiment with your A/B testing approaches. 

Sounds like a plan, isn’t it? 

Think bigger and add some novel techniques to your ideas backlog. Here are a few examples off the top of my head:

• 1 on 1 interviews with your loyal customers. Why not? This can be accomplished through screen sharing tools built-in Zoom, Google Meet, Slack, etc.
• Online polling — this is some of the easiest qualitative data to pull directly from the horse’s mouth. 
• User testing is when you literally spy on your user behavior, but in real time. Individual visitor session recordings, using the tools like Hotjar or Inspectlet, are an amazing way to gauge how your shoppers interact with your store’s interface and see where your shoppers may be getting frustrated. 

In fact, there are way more strategies to consider before you dive into the world of A/B testing. Below are some more homework for those who are on their way to becoming A/B testing wizards.

More Resources to Study: 

The Complete A/B Testing Kit for Marketers
My 7 Rules for A/B Testing That Triple Conversion Rates How to Do A/B Testing: A Checklist You’ll Want to Bookmark

IV. Creating a Strong A/B Testing Plan for Your eCommerce Business

But enough theory — let’s talk about how to actually do it. 

This A/B testing plan template will walk you through how to conduct pre-test heuristic analysis, create the conversion roadmap, put it in place, and, finally, measure results for better split tests.

So, where do you begin? 

Begin at the beginning and go on till you come to the end; then stop.

Jokes aside. I’ve split the whole procedure into three major steps: pre-test, mid-test, and post-test analyses. Let’s step in and tear it apart. 

I. Pre-Test Analysis: Getting Ready For A/B Testing 

Map estimated user journey 

The first thing you should pay attention to doing your AB testing research is your target audience and their preferences. 

1. Determine the ideal user journey.
2. Identify areas for improvement. 
3. Then, put on your customers’ shoes and walk through their estimated journey with the team.

Stumbled upon a nasty bug? Go fix it. Then, continue testing.

Carry out qualitative, quantitative, and competitive research

You should have as much data as you can possibly get to back up your test results (but without drowning in overwhelming data that you’ll never be able to analyze). 

Qualitative research: Here’s where all sorts of polls, interviews, surveys, phone and chat transcripts come into play. You must know exactly what your consumers really feel about your online store.

Quantitative research should help you identify which parts of your user interface look a bit botched and where the weak points are in your funnel. At this stage, you should look into your transactional data, analyze where your customers come from, what device they prefer to use, whether they return or not.

TL;DR: Collect all the data you already have about your target audience. 

Use Google Analytics to create funnels, build heatmaps with Hotjar, etc.

Competitive analysis implies a deep understanding of what your competitors are up to. You should monitor the things like pricing strategy, organic search optimization, customer acquisition, opt-in forms and landing pages design, and the rest.

Your competitors can be your best friends if you know how to use this ‘friendship’ to your benefit.

Set a clear goal for your A/B testing campaign

You’ve got to have a clear idea of the results you’re looking for. And since every idea starts with a problem, do your best to identify the issues your shoppers have interacting with your eCommerce website.

If you have more than one issue (and there’s likely to be tons of them!), your next step is to prioritize them. 

Place a bet

Placing a bet should help you carry out the previous task. You’ll want to generate a 95% confidence interval, or rather 98%, especially if the procedure takes a good deal of time.

Create a conversion path and hypotheses

Defining an independent variable, a dependent variable, and the desired outcome is vital at this stage. 

You should clearly understand WHO does WHAT and WHY.

The independent variable would be the unaltered version of the page (in case you are testing the page design). The dependent variable would be the idea you want to challenge.  

Define the size of your sample group

If you are testing some major improvements to your store’s functionality or sending an email newsletter to a massive customer base, you’ll want to roll out the changes to a small portion of your clients first, just to test the waters.

When the apparent winner is selected, you’ll be able to show the winning variation to the rest of your follower base. 

Thus, you will eliminate silly mistakes and have a better chance to grow your conversions. 

II. Mid-Test Analysis: Designing, Testing, And Having Fun

Here’s when the actual execution process rolls out. I’ll keep this short and sweet — just do it. But be sure to follow the rules below:

• Be all eyes and ears and carefully monitor the process of A/B testing. If anything goes wrong, you should be ready to jump in and fix it.
• Take advantage of the testing tools. They’ll save you a fair bit of time.
• Avoid running multiple tests simultaneously (unless you are ready to dive into parallel testing, which is another big thing to discuss, but here’s a guide if you’d like to give it a try). Done incorrectly, it may ruin the validity of the experiment.

III. Post-Test Analysis: Reaping the Harvest

Post testing analyses come down to a few major things. These are:

• Pulling the A/B testing results.
• Determining whether the results validate the assumptions.
• Rolling out the winning combination on staging.

Below are a few little things that you should take into account analyzing your A/B testing results.

Take the results with a pinch of salt

You should learn to be skeptical about the results, even if they look beautiful. If something sounds too good to be true, it probably is.

Second guess the data and triple guess yourself to be on the safe side.

Remember to take action based on the results

Plunging headlong into A/B testing is a good idea. But, whatever you do, always bear in mind your end objective. 

Never stop experimenting

There’s nothing in life (just like in eCommerce) that’s static. And nothing that remains static is truly ever alive. 

Keep pushing on the boundaries of what you do. Get your inquisitive hands on. Never stop testing. 

Are you still here? There’s more. 

Take a look at this awesome infographic crafted by Ayat Shukairy from Invesp that sums up all the ideas mentioned above and shows the whole process in a more digestible form. Alas, you’ll hardly be able to find it on SERP, and, to be frank, I have no idea why Google demotes this awesome piece of content.

You’ll also love this article by Alex Birkett featuring 29 guidelines for online experiments, split up by “the level of nerdiness or laziness” of the reader. He touches upon some bloody difficult things like non-inferiority tests, predictive targeting, sample ratio mismatch (Oh, my!), and flicker effect. 

Not this one, obviously.

V. The Top A/B Testing Tools Recommended by CROs

If you are not new to eCommerce, you might have already had some experience with A/B testing tools. 

eCommerce optimization services have been around for years, but the release of Google Optimize has changed the game completely. And there are at least three reasons for that:

1. It’s free, with an option to extend functionality with advanced features like multivariate and split URL testing. 
2. It integrates with your Google Analytics account. (Show me someone who doesn’t have Google Analytics.)  
3. Implementation is as easy as adding a tiny piece of code.

At X-Cart, our CRO experts use it on a going basis. Consider reading this definitive guide on how to do A/B testing with Google Optimize. It’s a 9-minute read, but the amount of info you get is priceless.

Also, we have a specific addon that will make the integration process easier for business owners:

Another awesome tool that I’ve already mentioned once in the previous chapters is CrazyEgg that allows you to see what people actually care about on your eCommerce website pages.

Just like Google Website Optimizer, CrazyEgg is among our favorite tools that we use to figure out why users are leaving our website and find the best ways to improve conversions — all through A/B testing and other techniques. 

We take advantage of Mailchimp’s A/B testing functionality, too, to define which version of our emails generate more value. 

There’s more. If you have your store running on X-Cart, you are fortunate to have a wealth of testing tools that will help you find out whether the idea is “hit” or “miss.” Some of them are listed below:

There are two more tools made to improve your mobile app’s performance through A/B and multivariate testing. These are Taplytics and Apptimize. They will let you experiment with the look and feel of your apps without much coding.

Over to You

If you asked me whether there’s one rule that you should adhere to running A/B tests for your eCommerce business, I’d say there is one: just keep pivoting. This has become the mantra for many eCommerce businesses in 2020 which, in its turn, taught us to live in the here and now and adjust to the new normal. 

Launching an eCommerce business is anything but a set-it-and-forget-it mission. Your potential buyers just won’t let you sit tight. Whether they be middle-aged businessmen, young mommies, or teenagers, a common theme comes up — they can’t have enough of everything.

Below is my rule of thumb: 

Got a crazy idea? Don’t waste time. Translate it into action. 

Don’t worry too much if the test turns out to be a flop. Embrace your failure, and keep looking for more efficient ways to serve your consumers.

Have you already tapped into A/B testing? If yes, what are some of the big lessons you’ve learned about it? 

Please share your thoughts in the comments section below.