A/B Testing – A website optimization technique
A/B testing is a method of figuring out which is the best version of something: A or B. In scientific terms A/B testing is a controlled experiment where A and B are the control and treatment. A and B are identical in all ways except one.
In web design we are using A/B testing, to identify which is the best way to optimize a website. Optimization can mean various things: conversion rate, new sign ups, more shares, etc. Let’s take a look at some fundamental rules that you should always keep in mind when A/B testing
Rule no 1: Test only one thing at a time
A/B testing should be used when you are testing two different versions of one variable.
A good example would be, if you wanted to find out which hyperlinks get clicked more. Those that are underlined or those that aren’t. In order to test this, we would create one version of the site with underlined links (version A), and one version where links aren’t underlined (version B). The variable being tested is the text decoration of the hyperlink.
Testing more than one variable at a time can be misleading and should be avoided
Rule no 2: Choose an appropriate sample
Choosing an appropriate sample in both quantity and quality is extremely important. A sample that is too small will lead to inaccurate results. Moreover the sample should be diverse and should not consist of only a specific group of people (i.e. let’s say that we test only men and no women) or else the result is going to be biased.
If your site receives more than 10,000 visitors per month, a sample size of 10 visitors (which is 0.10% of the monthly site visits) won’t get you reliable results.
Rule no 3: View all metrics
It is often the case that when A/B testing, you will see an interaction between the various metrics. For example, using a version that improves the click through rate of a button might consequentially decrease the time that a users spends on the site.
Whether a change in some metric is good or not depends on the scenario and the goals that we set. It could be that a change that decreases a metric is alright if it improves a more important metric. For example it is probably fine if you decrease the click through ratio of a button, if you’re getting more sign-ups overall.
Rule no 4: Implement your findings
Sometimes the conclusions of the experiment may not be the ones that you expect or like. In some cases the versions that work better are uglier than the original ones. However the purpose of a website is not to look pretty, but to achieve certain goals.
Regardless of how a web site looks, the design is successful if it actually achieves the goals that are set
Tools For A/B Testing
There are a lot of tools available for A/B testing. Here are a couple of them:
Google Analytics – Content Experiments (formerly known as Website Optimizer)
A free A/B testing tool from Google.
Visual Website Optimizer
An A/B testing tool, with features such as WYSIWYG editor, click maps, visitor segmentation and tag-less integration.
Landing-page creator with integrated A/B testing.
Hubspot (formerly known as Performable)
Inbound marketing software platform.