Jason Brewster - June 24, 2015
Website testing can be a daunting task. While it is important to find the platform that’s right for your business and to carve out some time for designers and engineers to create the tests – interpretation of the experimental data can cause its own sorts of pain. To help new testing specialists cope, we have developed short list of some of the most important bullet-points to keep top of mind as you begin and run an effective test.
1. Which Type of Test Should I Run?
There are two very popular testing methods currently A/B tests and multivariate tests. There is more information on this in our previous A/B test vs Multivariate test blog. In a nutshell, you use multivariate tests if you need to evaluate multiple variables.
2. KPIs – What should my key metric be?
- CR – Conversion Rate AKA the gold standard of success in marketing. This can be a purchase, a signup or pretty much anything that the experiment driver wants.
- AOV – Average Order Value is frequently undervalued; however, some companies with large price points and frequent upselling opportunities can see more overall revenue lift by focusing on it. While it is rarely used as the KPI, it does feed into the success of Revenue per Visit campaigns.
- RPV – Revenue per Visit indirectly combines the performance of CR and AOV. By focusing on RPV you can ensure that the variables that succeed in lifting just CR or AOV do not penalize the overall revenue improvement on site.
3. Sample Size – Am I Getting Enough Data?
To develop actionable results, the experiment needs visitors and conversion volume. Conversion volume is what allows an analyst to become confident in their final results. There are two levels of conversion volume.
First level of conversion volume: stable results
- By reaching a stable amount of conversion volume you can see consistent trends over time and at the completion of the test you can make a simple binary decision.
- Failure to reach this level of consistency would look like a website that gets a couple of transactions a day (or missing data from a couple days). In this instance it is important to select a micro-conversion that will yield something that will allow you to comfortably make a decision within 6-8 weeks.
Second level of conversion volume: accurate forecasting
- If your AB test has operated for a long enough period of time and the conversion volume is very robust, then you can estimate how much of an impact your variables will make. This can be calculated as conversion rate lift, or if you insist on a dollar amount, it can be projected with RPV. You will notice that during the early phases of discovery, sometimes a promising result is unrealistically positive. Typically this situation corrects itself over time as the law of large numbers takes effect.
4. Outliers – Should I Optimize Towards Outliers?
For users that have low conversion volume, outliers can heavily distort the results of a test. RPV and AOV are influenced by the amount of money users spend per transactions, if that amount is greater three standard deviations above the mean transaction total then it is considered an outlier. Sometimes outliers are business owners or call centers. Outliers should be observed and potentially removed before making any critical decisions regarding the test or final implementation.
5. Segments – How Do I Target & Personalize?
User segmentation is a fact of life for web analytics – for testing it is the bread and butter of incremental improvement. It is the primary hope for a brand to find a simple change that drives the bottom line for every user on site. What these brands don’t take into account is that their user base varies far and wide.
It is a much better idea to personalize the website towards a few targeted segments and get some wins rather than dismiss variables because they don’t perform well in aggregate.
Below are 2 likely scenarios:
- The brand tests an experience and it shows a 10% improvement across the website, they excitedly slate it for implementation. Everybody eats cake and goes home!
- The brand tests an experience and it shows a 0.5% improvement. There is much grumbling, however – the clever analyst reviews the segments. He realizes that the returning user hates the experience and the new user loves it (with an 8% lift in conversion rate!). He looks over the experiment details and realizes that using security badges on the checkout funnel aggravates long time users – but helps entice new ones into engaging with the brand the first time! He works with his IT department to target the new users with the new “security” variable. Later, during his quarterly analytics review he notices a marked improvement on new users’ conversion rate!
As you can see, by combining personalization and segmentation with testing it becomes easier to improve your website as a whole. So instead of running a 6 to 8 week campaign and getting nothing out of it, you retrieve value for the time, effort and work invested.
