Twitter for usability testing or doc testing? Sure, here’s how

My coworker went to SXSW Interactive this year, and I merely went to lunch with people near SXSWi and followed the #sxswi Twitter stream, and now I am browsing through the sketchnotes. I went to BarCamp Austin on Saturday and presented about FLOSS Manuals and the Book Sprint methodology we’ve been experimenting with. It went great, and afterward I even got a nice compliment “You are a force of nature, aren’t you?” That made me grin big!

But back to Twitter for usability testing. How can that be done? The CMS Showdown featured at SXSW Interactive actually came up with a way to do it, and then there’s a video on Vimeo showing one of the participants watching and commenting as the Twitter stream goes by on her screen.


JoomlaSxSW AmyStephen’s Review of the Review from Amy Stephen on Vimeo.

Basically, set up a time for a certain number of website users to try certain tasks on your website or application. While they use the application, have them log on to Twitter and make comments, including a pre-set hashtag in their Tweets. By the end of the testing period, you’ll have a record of micro-comments (140 characters or less) collected with the search.twitter.com tool.

Somehow this use of Twitter to “judge” your product or documentation makes a lot of sense to me. You pick a hashtag and a span of time, and ask people on Twitter to read the doc or try the product at the same time, putting their thoughts up as 140-character or less Twitter posts.

Now, be sure to save off the stream of comments because, as Jenny Levine noted, the stream of the moment is momentary.

A week or so ago, people in the technical education sector did something similar to what I’m suggesting – they all discussed a topic at the same by putting the hashtag #educhat into their Tweets. We’ve been talking about a similar organized chat time for FLOSS Manuals.

I’ve blogged about uses for Twitter before – usability testing is just one more use to add  to the list.

3 Comments

  • April 3, 2009 - 1:15 am | Permalink

    Thanks for the post, it was an interesting video to watch.

    I feel differently about using Twitter for usability testing, and hear is why…I have two thoughts:

    Having sat through many usability test, I have personally found that the most valuable information comes not from what the user says, but from the non-visual cues that sometimes prompt questions…e.g. “what are you thinking right now?”. It is an amazingly lucky happenstance (too overstated?) when you get a user that does “talk aloud” all of their thoughts and feelings. There is usually much prompting involved.

    My other thought is that many of my test subjects actually had poor typing skills. I would imagine that “documenting their experiences” would either a). influence their opinion or b). be such a burden that the couldn’t complete their tasks–not because of the tasks, but because of the recording.

    What do you think?

  • November 30, 2009 - 4:12 pm | Permalink

    Just had the same thoughts as Larry!

    When working on a problem, I speak out my thoughts, but only when completely involved with the problem and with myself.

    I personally think that many users talk to themselves in their heads even if not loudly. But only when they are left completely unperturbed. Asking them to perform a conscious task in parallel would break that thought train and its not good for catching usability data.

    But then documenting the usability test is also necessary! Video recording (esp. closeup of face) to study expressions and body language while doing a task would be better, in my opinion than candidates themselves recording it.

  • December 26, 2009 - 7:27 pm | Permalink

    Sorry it took me so long to respond, but I believe that using Twitter for usability testing is simply another way to use the Think Aloud method. From http://cutshall.myweb.uga.edu/edit8350/usability/methods.html#think – “The user is instructed to speak aloud their thoughts, decisions they make, any confusion, their reactions to what happens, etc. while interacting with the product. Encourage the user to tell the evaluator what they are really thinking as they use the product. As thinking aloud may not come naturally to many users the user may need to be prompted “and what are you thinking about now?” or may need to hear an example of thinking aloud by the evaluator.”

    The trouble with Twitter is that you, the tester, cannot as easily draw out responses. But, that’s also an advantage, because the tester cannot influence responses or lead the tester.

  • Leave a Reply