Extreme Evaluation

Michael B. Twidale, Graduate School of Library and Information Science, University of Illinois at Urbana-Champaign.
Paul F. Marty, School of Information Studies, Florida State University.

• Is there a quick cheap and easy way to demonstrate the power of usability testing to a skeptical audience?
• What do you do when you need to evaluate the usability of a system under extremely tight deadlines and budgets?
• How can we adapt rigorous experimental user testing to meet the needs of rapid iterative prototyping?
• If real life development constraints mean that it is impossible to do a proper usability evaluation, do we just give up?
• Can we integrate extreme usability into extreme programming?

This project looks at some of the problems with formal rigorous experimental user studies, chiefly their cost and time consuming nature. We don't want to eliminate such tests - they are the gold standard for obtaining definitive information. But there are times when they are just not feasible, and we think it is unfortunate if the only alternative to an infeasible 'proper' user test is no test at all. Lots of people have worked on how to minimize the cost of user testing with one highly controversial lower bound being the 5 subject user test. That debate revolves around the question: "Is five enough?". We want to ask "What can you get with just one?"

Sample References

• Marty, P.F. & Twidale, M.B. (2005). Extreme Discount Usability Engineering. Technical Report ISRN UIUCLIS--2005/1+CSCW. [PDF]
• Marty, P.F. & Twidale, M.B. (2005). Usability@90mph: Presenting and Evaluating a New, High-Speed Method for Demonstrating User Testing in Front of an Audience. First Monday 10(7) July .
• Twidale, M.B. & Marty, P.F. (2004). Come On Down! A Game Show Approach to Illustrating Usability Evaluation Methods. To appear in interactions.
• Marty, P.F. & Twidale, M.B. (2004). Lost in gallery space: A conceptual framework for analyzing the usability flaws of museum Web sites. First Monday 9(9). [HTML]