Buckle in because we’re about to cover everything you need to know about A/B testing (sometimes called split testing, sometimes called A/B/n testing, the “n” refers to the unknown amount of test variations one may undertake).
Let’s quickly define things: A/B testing is essentially testing two different versions of a webpage to identify which version performs better to improve that page’s conversion rate.
1. A/B Testing: The Basics
ECommerce tools, platforms, plugins are constantly increasing their functionality and becoming more cost-effective! In recent years, large-scale A/B testing has become more accessible to startups, small and mid-sized businesses. Competition will only continue to increase and make web traffic harder to obtain, therefore maximizing a business’s likelihood to convert is vital.
The standard method for A/B testing is the following:
- Identify an independent variable you’d like to test (there are scenarios where testing multiple variables works better, but let’s keep things simple for now).
- Create a variation of that independent variable on an otherwise identical web page
- Split incoming traffic equally between the original and variation page
- Run the test for a predetermined amount of time, experts recommend no less than one week and no more than four!
- Whichever page tests better is the page you should keep
- Rinse and repeat with a new variable until the end of time
If you look for successful A/B testing examples online, you’ll find businesses that achieved a 433% increase in their click-through rate, a 70% increase in conversions and for major eCommerce sites, the change of a button resulting in million dollar increases in revenue. You’d be setting yourself up for disappointment if you expected the same results for your business, A/B testing is a constant data gathering grind. But, what’s exciting about those impressive breakthrough results that other businesses have had—you might just get one too.
When an A/B test involves multiple variables, it’s more accurate to call it A/B/n testing. If “n” represents the number of variables you can have, you can understand you could have an A/B/C test, an A/B/C/D test and so on. But, if you test too many variables at a time, it becomes very difficult to pinpoint what change in variable(s) is responsible for your results.
The primary purpose of A/B testing is to discover what is causing your users to bounce: discovering what features are causing customers to lose trust in you, become impatient, lose their way in navigation or what lack of features is causing them to seek competitors, e.g no returns policy.
How can you test what is causing customers to leave your site?The following will describe an example where it’s more efficient to test multiple variables at once.
This A/B/n test will have an original web page and 3 variation pages. The first variation will include a new returns policy, the second variation will remove the need to register an account, the third will include trust symbols (security symbols, certification badges, testimonials).
Including the original, we have four different web pages to test. Therefore, web traffic will be divided in four. After testing, each page will have a conversion percentage. Compare the variation page percentages to the orginal’s percentage. Variations that done better than the original should be implemented, those that done worse shouldn’t.
It’s important to remember that these changes aren’t final. The variations that didn’t do well may become useful in the future. Each variable may combine better with a new control variable, e.g no account creation might combine better with a returns policy.
2. Accounting For Variance
In a perfect world, the resulting conversion rate would align perfectly with your web traffic. It’s not a perfect world, so it’s important you understand external inconsistencies and any factors that threaten to make your test inaccurate.
Competitor Promotion
For example, imagine you implement a two week split test at the same time your main competitor starts a big promotion. Your results will be different to what you’d expect during a period without promotion. However, your competitor’s promotion will only affect your A/B test if your test aligns with the same time period as the promotion. If their promotion goes for two weeks and your split test also does, then there’s no problem. If their promotion goes for two weeks and your split test goes for four, then your test will be contaminated.
Traffic Segmentation
This refers to the way different customers have accessed your site, e.g some may be revisiting your site after reading your newsletter, others may have just found you organically using a search engine. It may be more likely the user who has read your newsletter is more likely to convert rather than organic traffic who has just begun researching your products and comparing them with your competitors. If an A/B test included both newsletter traffic and organic traffic, it’s the traffic type that may be causing different conversion results rather than the variation being measured in your test. Stopping this is simple, you segment the traffic and do an A/B test for each.
Sample Size
When running any A/B test there is always an element of chance. One half of user traffic may simply have been more likely to convert. Why? For any reason, because humans are… humans, and one might have converted simply because the sky was blue.
To lessen the odds of chance affecting your A/B test, you need to ensure you have an adequate sample size. An adequate sample size ensures your results are statistically significant (a result that cannot be attributed to chance).
It’s recommended you let each web page in an A/B test receive 5000 unique observations. To calculate statistical significance there are plenty of calculators online, just search “ab test statistical significance calculator.”
3. A/B Testing vs Multivariate Testing
These are two terms that are used interchangeably, however they are different—A/B testing refers to macro changes, multivariate testing refers to micro changes.
A/B tests are usually larger changes.
While, multivariate testing focuses on getting the best out of small elements, like the placement of an image or the colour of a button. Multivariate testing will often test multiple variations of a webpage and therefore a smaller percentage of web traffic will be split up between multiple page variations, e.g different page variations for different button colours. But it can become a lot more complex than that: one page with a returns policy, one page with a return policy and trust symbols, one page with a returns policy, trust symbols and a different button colour.
Multivariate testing is mostly done by larger organisations as they have a larger amount of traffic to split up between different page variations. For smaller companies, splitting up a small amount of traffic between multiple variations will usually result in unreliable data.
4. What You’ll Need For An A/B Testing Team
Designers
A designer creates all the new variations for your site, which could be something as small as a colour change to redesigning an entire landing page. The designer must be skilled enough to ensure that a design change will mean that only the new design element is tested.
Developers
A developer will be experienced with Javascript and other frontend languages, and be able to update your pages so that they stay functional on multiple devices. If a page isn’t working on one device or a page goes down—your test loses credibility.
Not all companies have designers and developers handy, or they don’t have the time to be constantly A/B testing. If you’re not A/B testing, you can stunt your business growth. It’s not bad to plateau for a while, but ideally you want to be moving forward again soon.
Many companies are able to quickly find talented designers and developers by working in coworking spaces. Especially smaller companies that want to work in a cost-effective environment, while avoiding the drawbacks of a traditional hiring process. To surround yourself with others who also have tried and tested knowledge on business optimization, consider joining a coworking community.
However, the most common solution is to hire designers or developers so that A/B testing can be taken care of efficiently, without business having owners having to shoulder any technical burden. If you’d like to hire expert designers or developers, I’d recommend the services of global development company Code Clouds.
