Pros and сons of split testing (A/B testing)

By Roman Berezhnoi November 11, 2021 189 views

Pros and сons of split testing (A/B testing)

Split testing or A/B testing is one of the most popular methods for website owners and digital marketers that allows the customers to unleash two different versions of a design on the world and see which performs the best. There are A/B testing and multivariate tests. For simplicity, it’d be better to use the term A/B testing or split testing to refer to any study where you measure design alternatives by feeding them live traffic, regardless of the number of variables being tested. But, even though this form of testing is gaining popularity with time, is A/B testing the best option to go forward in the digital marketing field?

The feedback from the digital agencies about split testing is positive, but the form of testing still leaves a lot of unknowns. Split testing can be a bit biased for the person who is setting up the test. As a result, the customers might not get the best service they are expecting. Apart from this, split testing leaves some unanswered questions.

Advantages of split testing

Some simple A/B tests you can run include testing:

  • Headlines and subheadings
  • Text
  • Call to action button and text
  • Social proof
  • Images
  • Testimonials
  • Links
  • Header and footer

Some more challenging or advanced elements that you can A/B tests include pricing structures, free trial lengths, navigation, free shipping, free or paid delivery etc.

Clear evidence

The reason why digital marketers around the world appreciate split testing is because their client can see how many users complete a transaction with site A over site B. In other words, you get clear evidence based on actual data and you can confidently conclude that if version A gets more conversions than version B, then version A is the design you should show all users in the future.

New ideas can be tested

If you have new ideas under your belt for an existing landing page, then A/B testing is one of the best ways to figure out whether it works or not. It can measure very small performance differences with high statistical significance because you can drive traffic at each option of design.

You can optimise a single step at a time

Whether you are working for a large site or many landing pages, A/B testing is one of the best ways to test. You can optimise one step at a time by starting your work in a small corner and then slowly moving to the website’s main pages.

It answers design questions

While testing site UI elements, you will come across different questions in your mind, and A/B testing answers specific design questions. For example you might wonder whether the green button is better than the blue ones for your website design, and this form of questions are quickly answered. Web designers can test different colours, placement of buttons, images, and the UI elements of a website. Split test results help a web designer to improve the website.

Affordable price

Split tests seem cheap. Once a web developer has created the two web page design alternatives (or the one innovation to test against your current design), then a web developer simply puts both of them on the server and employs a tiny bit of software to randomly serve each new user one version or the other. You don’t have to hire expensive usability specialists to monitor each user’s behavior or analyze complicated interaction design questions.

With these clear benefits, why don’t web specialists use A/B testing for all web projects? Because the disadvantages usually outweigh the advantages. In fact, A/B testing can bу difficult used even for projects that have one clear, all-important goal, a single KPI (key performance indicator) like sales for an e-commerce site, users subscribing to an email newsletter, users opening an online account, users downloading a white paper.

Disadvantages of split testing

It takes a lot of time and resources

One of the significant disadvantages is split testing can take a lot of time and resources. It takes a longer time to set up and takes a lot of help, even though you can use third-party services. There may be endless meetings about which variables to include in the tests. Once a set of variables have been agreed, designers and coders will need to effectively work on double the amount of information. But the main problem is getting conclusive results, tests can take weeks and months for low-traffic sites. 

Additionally, when you are starting out as a business website you will have hundreds of ideas that might work or won’t. Testing each idea isn’t feasible.

It is primitive

A/B testing perfectly works only for solving a simple dilemma. For example, what is the colour of the button that can give the best results for this landing page? But if the goals are not easy to measure or not be measurable by computer, then split testing will not provide you with a potential answer.

Unfortunately, it is rare that such actions are a site’s only goal. Sites that don’t close sales online can’t usually say that a single desired user action is the only thing that counts. Even for e-commerce, the amount of dollars collected through sales is not probably paramount. Yes, it’s good if users add products to the cart, fill in a form or call a salesperson. But it’s also good if your visitors leave the site feeling better about your product or company and place your business on their shortlist of companies to be contacted later in the buying process, particularly for B2B sites. Also for many sites, the ultimate goals are not measurable through user actions on the server.

Anyway if your only decision criterion is to determine which design generates the most conversions, you risk undermining other parts of your business site or landing page.

It can’t improve the existing flaws

If you have usability problems with your website, then do not expect that the A/B testing can help you to improve those duds. The fundamental flaws will still remain the same that your site has. A/B testing will not let the users know about the problems of your website, and your users will be frustrated.

End up with constant testing of the website

The data gathered is of no use, and an A/B test has to start from scratch. Say, for example, that you tested two colours of Buy buttons and discovered that the red button generated 1% more sales than the green button. Does that mean that you would sell even more with a blue button? Or maybe an orange button would increase sales by 2%. You don’t know, and to find out you have no choice but to try again with another collection of buttons.

Further A/B tests will have to start from a new baseline and other types of testing will only likely be applied to the more successful site, when they could have found equally useful information from the rejected version.

No real behavioral insights

The biggest problem with A/B testing is that you don’t know why people do some actions. You’re not observing the users or asking for their thoughts. All you know is that, statistically, more people performed a certain action with design A than with design B. It supports the launch of design A, but it doesn’t help you move ahead with other design decisions. You have no idea whether other changes might bring even bigger improvements, such as changing the button’s size, colour or the wording of Call-To-Action. Or maybe changing the button’s page position, rather than changing the button’s colour, would create the same or better results. Basically, you know nothing about why the button on B design was not optimal, which leaves you guessing about what else might help. After each guess, you have to implement more variations and wait until you collect enough statistics to accept or reject the guess.

A/B testing provides data only on the element that is tested

Split testing is not an open-ended method like UX and usability testing, where users often reveal stumbling blocks you never would have expected. It’s common, for example, to discover problems related to trust, where users simply don’t want to do business with you because your site undermines your credibility. It is about a lack of marketing strategy.

Many issues like trust, uninformative product description, grammatical errors, bad marketing strategy often have effect sizes of 100% or more, meaning that your sales would double if such problems were identified and fixed. If you spend all your time fiddling with 1-10% improvements, you can easily overlook the 100%-500% improvements that come from qualitative insights into users’ needs, desires, pains, and fears.

F5 Sudio’s recommendations

In fact, split testing or A/B testing has more problems than benefits. If you want to develop a successful business you should not make A/B testing the first method you choose for improving your site’s conversion rates. And it should certainly never be the only method used on your website. 

Qualitative research such as observation of user behavior is faster and generates deeper insights. Also, web specialists who provide professional UI/UX design services, use polls, surveys and other combinations of methods. Qualitative research is less subject to the many pitfalls that plague quantitative research. On the other hand split testing does have its own advantages and provides a great supplement to qualitative studies.


A/B testing can be fruitful when it is used with other testing methods. This way of testing provides a potential tool to shape your design and figure out the things that help your users to click CTA. However, A/B testing cannot identify ease of use, frustration or usability problems; hence you cannot rely on it as a total solution. Therefore, you should experience some sort of usability testing, marketing research, and UX research to figure out the frustration of the users and issues before you integrate A/B testing to test the different elements of website UI design.

Also, don’t hesitate to hire a web design company that has the ability to provide usability testing services at affordable price.

Comments ( 0 )

Leave a Reply

Your email address will not be published.

Contact us:

Thank you for your message. It has been sent.

We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.