The complete guide to running ineffective user tests

Everyone says you need to do user tests — but doing a user test badly is about as useful as not doing one at all. Trust me, I know from experience, having ran my fair share of useless user tests back in the day. Today, I’m here to share some of that hard-won wisdom on how not to do your user tests, what to do instead, and what user tests are actually good for.

So, without further ado, here’s the complete guide to running ineffective user tests:

Get angry at users for misunderstanding

You’ve designed the perfect app, you have some volunteers to run user tests with, and yet, despite the flawless design you’ve created, they don’t seem to understand the interface. Obviously, the correct response is to get angry at your users for not understanding the inherent perfection in your design.

Take over to show them how it’s done

Is the user doing it wrong? Are they missing onboarding cues and tool tips that are painfully obvious to you? Are they exploring the app in a different way than you expected? It’s cool, just take over the test and show them how it’s done. Take the mouse and/or phone away entirely, if you need to, so that you can show them how it’s supposed to be done, and then proceed with the test from there.

Don’t write anything down

As everyone knows, the human mind is like a steel trap with impeccable recall, and all of our memories are flawless, completely untainted by emotion. Because of this, you don’t need to take notes, screen-record, or use any kind of data collection method — you can simply rely on your memory after the fact, and make product design decisions based on that.

Find random test users

If you’re just doing testing right now, it doesn't matter who you’re testing with, right? You can find entirely random people off the street, and it’s all the same as far as the test is concerned; no reason to make sure that your test users are related in any way to your intended users.

Create leading prompts and questions

You need to make sure your users are doing the right thing, so it’s fine to give them incredibly specific instructions, like “Go to your patients page and then click ‘create patient.’” Similarly, after the test, it’s totally fine to ask very specific, bordering on leading questions — after all, how are you going to get the answers you want if you don’t make sure they understand what you mean?

Or, if you want to run a user test that actually gets useful results...

Be open to being proven wrong

Does it suck when you’ve put blood, sweat, and tears into what you think is the perfect UI, only to see actual users be confused once they interact with the prototype? Yes. But better now than after you’ve created the complete product. Make sure to pay attention to exactly how they’re misunderstanding the design — are they misinterpreting a specific icon to mean something else? Is that intuitive button actually not-so-intuitive? Are their user patterns different than you’d predicted? Is your super clever microcopy confusing more than it’s clarifying?

Stay out of the user’s way

Taking over and explaining everything to the tester doesn’t show you how users are...actually using your product. To get unbiased, useful data, you need to stay out of the user’s way while the test is going, even if all you want to do is shout about how they’re doing it wrong.

Take diligent notes

It’s incredibly simple to record user tests and take notes while they’re happening, with tools like Silverback or Screencast-O-Matic, Evernote, or even voice memos on your phone...so you might think about doing just that. Lookback is another great one we’ve used and they even have an Invision integration!

Getting a recording and taking notes can be super useful when you’re trying to make an actual product decision weeks later after you’ve finished all your tests. Bonus: having a recording to reference so that you know exactly how your users talk about their problem and your product is super helpful when it comes time to start marketing.

Make sure your test users are reflective of your real users

If you want to get accurate results from your user tests, make sure that you’re asking people to participate who are reflective of your real users. If your intended user is a highly trained neurosurgeon and you run user tests with random people you find in Starbucks… chances are you might get some misleading information.

(Note: this does not mean to get an entirely homogenous group of users to test with. That way also leads to bad product designs and/or product decisions that are likely to be perceived badly by some users, i.e. prompts to share a map of where you run on social media, which will read entirely differently to people who are likely to be stalked/harassed or have been stalked/harassed in the past.)

Keep your prompts and questions high level

Instead of the above example (“Go to your patients page and then click ‘create patient’”), you want to be using higher-level prompts, like “You have a new patient — try setting them up in the app.” Similarly, you want to ask open-ended, non-leading questions (“Tell me about your experience setting up a patient,” vs. “Did you have a good experience setting a up patient in the app?”).

What are user tests actually good for?

Now that you know how not to run effective user tests, let’s talk about what user tests are actually good for. User tests are especially useful for:

Validating a new design or feature in an existing product
Figuring out whether your users can actually use your app before you launch
A/B testing which design to go with when you have a few ideas/options

The one thing you need to remember?

User tests are not the same thing as validating the need for a product.

They don’t actually tell you whether people need or will pay for a product. You can have the prettiest, easiest to use product on the market, but if it doesn’t solve a painful problem for users, it doesn't matter — they won’t use it (or pay for it).

If you want further resources on user testing, we recommend:

P.S. This is what came up when I first searched “User Testing” on Unsplash: