A/B Testing Tips for Gauging User Experience

Jason Brewster - March 17, 2016

A/B testing tools are a powerful method of making informed decisions and making improvements to a specific set of key performance indicators (KPIs).

History of A/B Testing

A/B testing was commonly used for direct mail (the paper variety) prior to the internet. In 1923 Claude C. Hopkins wrote a book called Scientific Advertising which is often cited as the most original document that not only describes “split testing” (A/B testing) but also coupon tracking.

By tracking metrics based on the durability of their wares, manufacturers and quality engineers use forms of multivariate testing to improve the quality of their merchandise.

In the information age, every brand with a website can gain an intimate understanding of their audience and user behavior through direct usability testing and A/B testing methods. With these methods, they can begin experimenting with thousands of users and pivot their experiments to audience reactions instantaneously.

Usability and A/B Testing Together

As the old adage goes, “garbage in, garbage out.” No matter how complex, powerful and algorithmically-sound your testing tool is, if you do not have a strong reason to believe your actual ideas will work, then don’t use them.

In fact, it would behoove any brand to look at the data that is available to them and construct a long-term testing plan based on research before they even look at a testing tool interface.

The data available in web analytics is expansive, but it can be impersonal when you want to determine the deeper psychology of your users outside of finding a conversion chokepoint. This is when usability testing becomes valuable. By utilizing a lab, survey tool, or even your own employees, there are several methods that can be used to both generate good testing ideas and remove bad testing ideas (i.e. ideas unlikely to succeed).

Usability Testing Methods

Card Sorting – Utilizes a group of experts or users to brainstorm examples of ideal streams of user behavior. This can utilize software or simple index cards.

Moderated In-Person Usability Tests – Record the user behavior and reactions of a person as they attempt to move through a specific flow. There are also remote tests that leverage screen sharing software as well as unmoderated platforms that can help gather information quickly.

UX Surveys –Making a survey available to your website users can help build the narrative around why certain behaviors are being observed in analytics.

No matter which method you choose to learn more about your users, remember: if you can link together the data shown by web analytics with the results of the UX test, you’ll find the most solid results.

Testing Based on UX Data

Once you have determined what the most insightful tests will be, you must plan them out and build accordingly. Make sure that each test is measuring the correct KPI and to watch other metrics that may be affected. Keep in mind that some of your tests will require technical resources when prioritizing your testing schedule.

When building your tests, it is critical to integrate the test experience with your web analytics platform. This will enable you to match up the data found natively on your testing platform with your main web analytics data warehouse (it is unlikely that they will match up directly). This is very valuable to companies that have already invested in advanced custom web analytics tracking.

In summary:

UX & Analytics Philosophy in a Nutshell

1. Have a Plan – Don’t test without a plan.
2. Collect Analytics & UX Data – Leverage the data you have in analytics and supplement that with UX testing. Remember that there are cheap, low-tech methods of getting UX data as well as the more expensive, scientific ones.
3. Gather Robust Analysis During Testing – Make sure you are collecting an adequate amount of data from your tests to make a decision that supports your KPIs. By integrating with your web analytics, you can take advantage of any custom tracking already in place and manage any consequences that are occurring outside of your KPIs.