I sometimes see A/B testing and user testing being talked about interchangeably. These two types of testing are approached differently and at different times in a project lifecycle. Depending on what is being tested, their goals may be different.
Difference between A/B testing and user tests
User testing is really a specific method of usability testing. It implies a single user completing a series of steps provided by the tester. Although there are different techniques — moderated, un-moderated, speak-as-you-go, timed — the goal of the researcher is generally the same: we’re looking for things standing in the way of usability.
A/B and similar tests, such as multivariate testing is part of an optimization strategy. Researchers are looking to see if a specific variation can get more conversions — visitors moving toward a specific goal — than the original.
When to test
A/B testing requires traffic. All, or a portion, of the incoming traffic is randomly assigned to either the test (variation) or control (original) and we track the aggregate number of visitors in the test and control that perform the action the researchers are hoping to improve. Because it requires traffic, A/B testing can only happen on a site that is both live and is receiving sufficient traffic.
User testing, on the other hand, only requires some type of visual aid. Generally a functioning website is preferable, but user tests can be done on static comps or even a napkin drawing. Ever shown someone a quick sketch and ask “what do you think?” Try instead giving them a little background and ask them to describe how they’d accomplish a task. That is user testing — it can be done at any stage in a project.
Use Both
User testing is great for validating or testing a new feature. Ensuring that it’s discoverable and straightforward. A/B testing does no good if no one can even use it. Then, if it’s a feature that has a conversion aspect, A/B testing can be done to tweak the feature until it converts better.
User testing is better at giving a big picture: how all the elements work with one another. It provides a human insight that conversion numbers can’t. On the other hand, user testing is anecdotal and qualitative. A/B testing provides quantitative numbers.
On the other hand, keep in mind that A/B testing may favor pushy and aggressive UI. While you may optimize conversions, especially in the short-term, it’s just as easy to lose sight of a product that is a joy to use.
How to Use
I suggest user testing new features: anything significant enough to change how someone interacts with the product. Then A/B test any conversion funnel on the affected pages. If A/B testing seems to indicate better performance with more dramatic changes, I suggest making a very dramatic change in that direction, then user testing, then A/B testing the (significant) variation.
But that’s a perfect world with a very testing-centric organization. Personally, I find more value in user testing once there are in-browser comps to test against. Then, if the project includes an e-commerce component, A/B testing the purchase funnel. If the project is relying on pageviews, Google Analytic’s bounce rate and pages-per-session are effective metrics to look at.
