Usability evaluation considered harmful (some of the time)

ABSTRACT

Current practice in Human Computer Interaction as encouraged by educational institutes, academic review processes, and institutions with usability groups advocate usability evaluation as a critical part of every design process. This is for good reason: usability evaluation has a significant role to play when conditions warrant it. Yet evaluation can be ineffective and even harmful if naively done 'by rule' rather than 'by thought'. If done during early stage design, it can mute creative ideas that do not conform to current interface norms. If done to test radical innovations, the many interface issues that would likely arise from an immature technology can quash what could have been an inspired vision. If done to validate an academic prototype, it may incorrectly suggest a design's scientific worthiness rather than offer a meaningful critique of how it would be adopted and used in everyday practice. If done without regard to how cultures adopt technology over time, then today's reluctant reactions by users will forestall tomorrow's eager acceptance. The choice of evaluation methodology - if any - must arise from and be appropriate for the actual problem or research question under consideration.

References

1. Barkhuus, L., Rode, J. From Mice to Men - 24 Years of Evaluation in CHI. ACM CHI'07 - Alt.CHI. http://www.viktoria.se/altchi/ (2007).
2. Bush, V. As We May Think. Atlantic Monthly, (1945)
3. Buxton, B. Sketching User Experiences: Getting the Design Right and the Right Design. Morgan Kaufmann, (2007).
4. Buxton, W. & Sniderman, R. Iteration in the Design of the Human-Computer Interface. Proc 13th. Meeting, Human Factors Assoc. of Canada, (1980), 72--81.
5. Christensen, C. The Innovator's Dilemma. Harper Business School Press, (1997).
6. Cockton, G. Make Evaluation Poverty History. ACM CHI'07 - Alt.CHI. http://www.viktoria.se/altchi/. (2007)
7. Dix, A., Finlay, J., Abowd, G. and Beale, R. Human Computer Interaction, 2nd Edition, Prentice Hall, (1993)
8. Dijkstra, E. Go To Statement Considered Harmful. Comm. ACM 11(3): 147--148, (1968).
9. Edwards, W. and Grinter, R. At Home with Ubiquitous Computing: Seven Challenges. Proc UBICOMP. LNCS 2201, Springer-Verlag, (2001), 256--272.
10. Engelbart, D.C. and English, W.K. A Research Center for Augmenting Human Intellect. AFIPS Fall Joint Computer Conference, Vol. 33, (1968), 395--410.
11. Gaver, B., Dunne, T., and Pacenti, E. 1999. Design: Cultural probes. ACM Interactions 6:1, (1999), 21--29.
12. Gould, J.D. How to design usable systems. in R. Baecker, J. Grudin, W. Buxton and S. Greenberg (eds) Readings in Human Computer Interaction: Towards the Year 2000, Morgan-Kaufmann, (1996), 93--121.
13. Greenberg, S. Teaching Human Computer Interaction to Programmers. 3(4), ACM Interactions, (1996), 62--76.
14. Greenberg, S. 2008. Embedding a design studio course in a conventional Computer Science program. P. Kotzé. W. Wong, J. Jorge, A. Dix and P. Silva (eds): Creativity and HCI: From Experience to Design in Education -- Selected Contributions from HCIEd 2007.
15. Greenberg, S. and Thimbleby, H. The weak science of human-computer interaction. Proc CHI '92 Research Symposium on HCI (1992).
16. Gutwin, C. and Greenberg, S. The Mechanics of Collaboration: Developing Low Cost Usability Evaluation Methods for Shared Workspaces. Proc 9th IEEE Int'l Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises (WET-ICE'00). (2000).
17. Hewett, Baecker, Card, Carey, Gasen, Mantei, Perlman, Strong and Verplank (1996) ACM SIGCHI Curricula for Human--Computer Interaction. Last updated 2004-06-03. http://sigchi.org/cdg/index.html
18. Kaye, J. and Sengers, P. The Evolution of Evaluation. ACM CHI'07 - Alt.CHI. http://www.viktoria.se/altchi/
19. Landauer, T. The Trouble with Computers: Usefulness, Usability, and Productivity. Cambridge, MA: MIT Press. (1995)
20. Lieberman, H. The Tyranny of Evaluation. web.media.mit.edu/~lieber/Misc/TyrannyEvaluation.html, ACM CHI Fringe, (2003).
21. Newman, W. CHI Guide to a Successful Archive Submission. http://www.chi2008.org/archiveGuide.html, (2008).
22. Nielsen, J. Usability Engineering. Morgan Kaufmann. (1993).
23. Olsen Jr., D. (2007). Evaluating User Interface Systems Research. Proc ACM UIST'07. ACM Press. 251--258.
24. Pinelle, D. and Gutwin, C. A Review of Groupware Evaluations. Proc 9th IEEE Int'l Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises (WET-ICE'00). (2000). 86--91.
25. Snodgrass, A. & Coyne, R. Interpretation in architecture: Design as a way of thinking. London: Routledge. (2006).
26. Stanley Dicks, R. Mis-Usability: On the Uses and Misuses of Usability Testing. Proc ACM SIGDOC, (2002)
27. Sutherland, I. Sketchpad: A man-machine graphical communication system. PhD Thesis, MIT, (1963).
28. Suwa and Tverskey. External representations contribute to the dynamic construction of ideas. In M. Hegarty, B. Meyer, and N. H. Narayanan (Eds.), Diagrams NY: Springer-Verlag, (2002), 341--343.
29. Thimbleby, H. User Interface Design. ACM Press Frontier Series, Addison-Wesley, (1990).
30. Tohidi, M., Buxton, W., Baecker, R., and Sellen, A. User Sketches: A Quick, Inexpensive, and Effective way to Elicit More Reflective User Feedback. Proc. NordiCHI (2006), 105--114.
31. Tohidi, M., Buxton, W., Baecker, R. and Sellen, A. Getting the Right Design and the Design Right: Testing Many is Better than One. Proc ACM CHI, (2006), 1243--1252.
32. Weightman, G. (2003) Signor Marconi's Magic Box. Da Capo Press.
33. Zhai, S. Evaluation is the worst form of HCI research except all those other forms that have been tried, essay published at CHI Place, (2003). http://www.almaden.ibm.com/u/zhai/publications.html

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods