Stop asking customers what they want

The focus group ranked the concept first. The survey said people would use it. The launch was quiet.

Watch the task

Jakob Nielsen, cofounder of Nielsen Norman Group, wrote the rule down in 2001, and the research habit inside most companies has barely moved. In First Rule of Usability? Don't Listen to Users he tells teams to watch what people actually do, to distrust what they say they do and to distrust what they predict they will do later.

The failure mode he describes is familiar in customer research. A team shows a few options and asks which one people prefer. If those people have not tried to finish a real task, they comment on the surface. A feature list and a concept board collect the same kind of vote. It records what sounds good.

His illustration is a survey in which half the respondents say they would buy more if a shop offered 3-D product views. The answer means the feature sounds appealing. People are guessing, and the guess is a weak reason to build it.

Liking is a weak proxy

Preference earns a place after someone has used a design and can feel whether it supports the task. Even then, liking and working come apart. With Jonathan Levy, Nielsen analysed 113 pairwise comparisons of interfaces built for the same task, published in Communications of the ACM in 1994. The correlation between measured performance and stated preference was 0.44. In later research on websites he found a correlation of 0.53. His reading of that second figure is blunt. You can predict about a quarter of how well a design works from how much people say they like it.

A score collected at a demo is a weak way to fund a roadmap.

Three steps from the truth

Asked about past behaviour, self-report sits three steps from what happened, in Nielsen's account. In a focus group, people bend the answer toward what they think you want to hear, or toward what sounds acceptable. Memory then drops the small details, including parts of an interface they never saw. What remains gets a tidy explanation. He has heard "I would have seen the button if it had been bigger" many times. The fact in the room is that they did not see the button. The rest is a story.

The question "what do you want?" asks for a prediction, a memory and a performance, all at once. Customers are concrete. They can tell you what they are trying to get done today. They are weak at designing the thing that does not exist yet.

What to run instead

Nielsen still wants a question. He wants a narrower one, asked at the right time. A survey completed before use produces answers that do not describe the work. A survey after use often gets nothing, because the person has already left. One question he says does work on a website is why the person came today. Motivation, on arrival, is something they can report. The pattern he trusts is a captive test. Watch the task, then ask what they preferred. A paper prototype is enough to learn that a flow fails.

If you own product, innovation or customer experience, change the research pack this month.

1. Write the task a customer is already trying to finish. Watch them attempt it with what you have, or with a rough version of what you might build.
2. Keep the feature vote, the concept ranking and the "would you use this?" survey if the room wants them. File them as a record of what sounds good. Leave them out of the funding decision.
3. If you still want a preference score, collect it after the task, and put the observed struggle on the same page.
4. Ask one question people can answer from the present. Why they came, and what they were trying to get done.

Our customer experience workshop maps the journey a customer actually travels, including the handoffs an internal org chart never shows. Bring a live journey and the evidence from watching it. A slide of what customers said they wanted is the pack this piece is trying to retire.