What is an attended, in-person usability study good for? What to expect?
When running an attended in-person usability study, there are 2 extremes:
- I want to prove something,
- I want to improve something,
… and it is all about money:
- I want to get money, to get something done,
- I want to spend the money which I already have, in the most efficient way.
Prove case
If you want to prove something you need data, usually loads of data. But this sounds already like analytics where data points are generated automatically while people are using your service. The first question is: wouldn’t analytics deliver as good (or better) results as a usability study?
The problem with many data points and a usability study is that you either have to sample your participant many times with different stimuli or many participants with the same stimulus to get statistically significant results.
There is another difficulty: you have to plan your test and make sure your tests will deliver these statistically significant results. This means that either your test cases are very different from each other or that you have many participants. In both case you must do really good estimations as you do not want to get more participants than necessary. They cost you time and money. In addition, you will have hard times to extend you test from 10 to e.g. 100 participants, just because of a tiny comma mistake. However, in most cases there appears magically the numbers 10 or 15 as the number of participants to be tested – but in most cases, there is no profound scientific explanation why you need these numbers. Test facilitators will just tweak the equation variables in a way that things fit their budget and their gut-feelings, which is s..t when you want to prove something.
In addition to a proper planning, there shall be a proper control of the environment and the selection of test participants, as otherwise your results might not be valid anymore.
As long as you could get away with tracking the input only (mouse movement, click, gesture, text, even speech), analytics are likely to do the job for you.
There are few exception, in which case you want to prove something with usability studies:
- Scientific studies: In some cases, it is simply not possible to utilize analytics.
- Your business model is built on gathering valid evidence based information ad to explain the phenomenon’s behind, like nngroup.
Improve case
The goal is to learn what is going wrong, why something is going wrong, and preferably with the least effort. It is usually enough to see few people failing to understand that the observed issue is
worth fixing. In some cases, you even anticipate the issue and you can see it coming the moment before people stumble across the error. This does not need any scientific proof. It is simply wrong. Usually 2-3 participants are enough for each test round. This means you can fix the errors you find right away and you do not bother more people than necessary with the error.
These tests are cheap. You just have to get 2-3 participants to observe, you just have to find and book 2-3 participants and this all can be done during a single morning/afternoon/evening. The test is fast. 2-3 participants allow you to get this done within hours and not within days. However, improvement is a continuous process and it this approach requires many rounds. This way you will also be able to re-test your updates when necessary. As your product is growing, also the user experience is improving little by little.
Another important aspect of “improve” is that usually you do not have to ask too many stakeholders for the improvements – as long as the improvements are evidence based and do not violate the business model.
However, there is nothing like validating the user experience of your product. This is a misconception explained very nicely by Kara Pernice.
To conclude
Money talks. In most cases you want to improve something while utilizing an attended in-person usability study. Proving anything with a usability study of this type is at least questionable in many cases. Check if there are other methods which deliver better results at lower costs.