Posted by Read the full article on A List Apart.
Theoretically, usability testing is a great way of finding out what is wrong with the products and services we design. We sit the users down in the lab and ask them to perform certain tasks, to “tell us what you think—give voice to your stream of consciousness.” And on the whole, it works.
I’m no mind reader (apart from when I am playing poker) but sometimes I do get the impression that users tend to say one thing when they mean another. However, they are not to blame—context is. In the lab, you shine the lamp into their eyes and ask lots of questions until you reach the finale:
“Would you use this site/service/product/software?” you say.
“Oh yes,” they reply.
You smile at them, all pink and happy. Some of the nice users even add a flourish.
“I will look at it this evening when I get home.”
And the real charmers will throw in a knee-trembling claim:
“I will tell my girlfriend/husband/dog, she/he has been looking for exactly this sort of thing.”
They pocket their cash, say their goodbyes, and enter into the outside world. And the minute they step out the door, real life intrudes on their thoughts. Should they’ve peas or beans for tea? Catch the bus or the tube? They’ve stopped thinking about you and your site/service/product/software, even before they turn the corner. By the time they get home, they’ve forgotten all about you, your lab, and their promises—even when sitting next to their girlfriend/husband/dog. They don’t think about your site/service/product/software. They watch Ugly Betty instead. They might, in the worst case scenario, never think about it or you again.
This is, in part, due to the exam-style conditions users are under. The desire to perform well and the need to please can lead to answers that simply aren’t true. Especially, when there is a cash incentive. We can’t help being nice to people who give us money. Money aside, the artificiality of getting users to step through a series of tasks in a laboratory can lead to behavior that is different to how they would behave if they were in their natural “habitat” doing the same tasks of their own accord. Therefore, wouldn’t it be better if we could probe inside their minds and study user behavior and motivations in a more natural context?
Aha! Enter ethnography.
Ethnography: stalking your user (legally)
Traditionally, ethnography is performed by observing users often in the workplace, sometimes over a long period of time, in order to build up a pattern of user behavior—and it can be a tedious process.
An ethnographer and I once stood on the observation deck of a steel-rolling mill watching the mill operator put in his mid-morning bacon butty order over the intercom. “Do you want ketchup?” came the reply. The ethnographer turned to me and said, “My job is to ask, ‘What is ketchup?’” I laughed out loud in an unethnographic manner, but he had a point. It was a case of, as my old boss used to say, “I don’t know what I don’t know.”
At the end of a field study, ethnographers might tell you what users do (eat tomato sauce in the workplace) but observation alone will not tell you how they feel (they hate the level of automation, but love ketchup). This can be problematic because feelings and satisfaction are high on the usability consultant’s list of what is important and needs to be measured. Also, observation alone does not tell you what is really important to users and what is mundane. Probing inside users’ heads does.
Cultural probes are a “quick and dirty” way of looking into users’ thoughts. They allow you to capture what types of knowledge and aspects of users’ jobs are important as well as how they feel about them. Probes go beyond classical user study techniques which focus on either what people say (questionnaire and interviews) or what they do (observation studies).
What’s a cultural probe?
One of the simplest cultural probes is a diary. This can be as simple as a pocket-sized diary in which, for a week or so, users jot down specifics about when, how, and why they interact with a website or service. Online blogs and photo diaries on Flickr work just as well—with the added advantage that you, the usability consultant, can see what is happening to users in real time and you don’t have to spend a lot of time afterwards converting the information you get into digital form.
Alternatively, giving users a dictaphone to talk into instead of writing/typing notes can allow you to capture all types of potentially useful information and head off the excuse that the user didn’t have a pen handy or the dog was sick on the diary (oh yes, it can be like asking for homework).
Typical questions in a cultural probe are based on the basic interrogatives: what, when, where, why, who, and how. And just as important as “How are users interacting?” is “How are users feeling?” Ratings can be useful for feelings so that interactions can be measured by emotion. Knowing why or why not users interact with something may help designers tune the final version. Also, users often use products and software in ways designers didn’t foresee them being used, and this becomes more obvious outside the laboratory when users have space to record what they do.
A few tips to keep in mind when you launch a cultural probe:
- Open-ended questions are a great way of encouraging users to write extra information down. Questions such as “What would you do differently in this type of situation?” uncover all sorts of thoughts that may lead to new solutions. Identifying whether an event or situation came up unexpectedly or whether it was triggered by something else (“it was my cat’s birthday”) is useful too.
- Giving users the choice of how they record their thoughts and feelings—text, photos, and drawings—is a good strategy so that they can decide on the best way of communicating for them and don’t feel hampered or self-conscious.
- The diary in whatever form can be as small or large as desired. However, size does play a part in how it is perceived by users. In paper form, space for 40 entries can be overwhelming whereas 10 entries might encourage them to complete the book. The digital equivalent sends out less of a psychological message as users have as little or as much space as they need.
Most users are shy to begin with but once they are alone with their thoughts and a platform on which to express themselves many are very forthcoming. They record all sorts of issues they simply would not have been prompted to think about when completing a questionnaire or performing a task in the lab, conscious of the two-way mirror. Diaries and blogs introduce a sense of intimacy that encourages users to tell you what is truly on their minds.
When to probe
Using probes during the initial stages of a new project can help you generate design solutions that answer users’ needs. During one such study I handed out diaries to ten users and asked them to describe incidents, over the ten days that followed, when they felt that their mobile phones had let them down. I asked them to describe a solution—even a magical one—to their situation which would guarantee them a successful outcome to the problems they had. The users sketched out all sorts of solutions: a stylus to take notes on their mobile during a call, a mobile which could text a fax, a mobile which could open word documents or texts whilst in the middle of a phone call.
By understanding exactly how testers used their mobile phones on a daily basis—and specifically, how they used them when they needed to respond quickly to someone else—it was easier for me to see a general trend. Users wanted their mobiles to be more like miniature networked computers, not just telephones.
Analyzing the results: emergent behavior
We may be individuals, but we are all human, and it follows that we have similar sorts of concerns and irritations. Even if your user group seems to contain the most diverse bunch of people you have ever seen common themes will emerge. They always do.
By analyzing the results of your cultural probe, you can build up a pattern of how users behave: what they love and hate, what motivates them to do what they do and why. Solutions based on this knowledge can help you to give users what they need, rather than what they say they want. Results can also help identify unrecognized needs and invent new products—and create happy users who feel as if you can read their minds.
