ABSTRACT
Usability testing is an everyday practice for usability professionals in corporations. But, as in all experimental situations, who you study can be as important as what you study. In this Note we explore a common practice in the corporation: experimenting on the company's employees. While fellow employees can be convenient and avoid issues such as confidentiality, we use two usability studies of mobile and web applications to show that employees spend less time-on-task on competitor websites than non-employees. Non-employees reliably rate competitor websites and apps higher than employees on both usability (on the 10-question SUS scale) and ease of use (on the 1-question SEQ scale). We conclude with recommendations for best practices for usability testing in the corporation.
References
- William Albert, Thomas Tullis, and Donna Tedesco. 2009. Beyond the Usability Lab: Conducting LargeScale Online User Experience Studies. Morgan Kaufmann. Google Scholar
Digital Library
- Sarah Anderson, Sarah Wandersee, Ariana Arcenas, and Lynn Baumgartner. 2013. "Craigslist samples of convenience: recruiting hard-to-reach populations." Retrieved August 13, 2015 from: http://fiesta.bren.ucsb.edu/~sanderson/CraigslistSurvey.Google Scholar
- Aaron Bangor, Philip T. Kortum, and James T. Miller. 2008. An empirical evaluation of the system usability scale. Intl. J. of Human-Computer Interaction 24, 6: 574--594.Google Scholar
Cross Ref
- Carol M. Barnum. 2010. Usability Testing Essentials: Ready, Set...Test! Elsevier. Google Scholar
Digital Library
- John Brooke. 1996. SUS - A quick and dirty usability scale. In Usability Evaluation in Industry, Patrick W. Jordan et al. (eds.). Taylor & Francis, 189--194.Google Scholar
- Mohamed A. Bujang, Puzziawati A. Ghani, Nur A. Zolkep, Shamini Selvarajah, and Jamaiyah Haniff. 2012. A comparison between convenience sampling versus systematic sampling in getting the true parameter in a population: Explore from a clinical database: the Audit Diabetes Control Management (ADCM) registry in 2009. In Statistics in Science, Business, and Engineering (ICSSBE), International Conference (IEEE, '12).Google Scholar
- Joseph S. Dumas and Janice Redish. 1999. A Practical Guide to Usability Testing. Intellect Books. Google Scholar
Digital Library
- Eugene H. Fram and Michael S. McCarthy. 2003. From employee to brand champion. Marketing Management 12, 1: 24--30.Google Scholar
- Wintre M. Gallander, C. North, and L. A. Sugar. 2001. Psychologists' response to criticisms about research based on undergraduate participants: A developmental perspective. Canadian Psychology 42, 3: 216.Google Scholar
Cross Ref
- David Gefen and Detmar W. Straub. 1997. Gender differences in the perception and use of e-mail: An extension to the technology acceptance model. MIS Quarterly, 21, 4: 389--400. Google Scholar
Digital Library
- Fredrick J. Gravetter and Lori-Ann B. Forzano. 2012. Research Methods for the Behavioral Sciences (4th ed.). Wadsworth Cengage Learning.Google Scholar
- Haredo Sahai and Mohammed I. Ageel. 2000. The Analysis of Variance: Fixed, Random and Mixed Models. Springer Science.Google Scholar
- Joseph Henrich, Steven J. Heine, and Ara Norenzayan, A. 2010. The weirdest people in the world? Behavioral & Brain Sciences 33, 2--3: 61--83.Google Scholar
Cross Ref
- Sari Kujala and Marjo Kauppinen. 2004. Identifying and selecting users for user-centered design. In Proceedings of the third Nordic Conference on Human-Computer Interaction (ACM, '04), 297--303. Google Scholar
Digital Library
- Mike Kuniavsky. 2003. Observing the User Experience: A Practitioner's Guide to User Research. Morgan Kaufmann. Google Scholar
Digital Library
- Jung-Joo Lee and Kun-Pyo Lee. 2007. Cultural differences and design methods for user experience research: Dutch and Korean participants compared. In Proceedings of the 2007 Conference on Designing Pleasurable Products and Interfaces (ACM, '07). Google Scholar
Digital Library
- James R. Lewis. 1986. Power switches: Some user expectations and preferences. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting 30, 9: 895--899.Google Scholar
Cross Ref
- Michael G. Morris and Viswanath Venkatesh. 2000. Age differences in technology adoption decisions: Implications for a changing workforce. Personnel Psychology 53, 2: 375--403.Google Scholar
Cross Ref
- Jakob Nielsen. 1994. Usability Engineering. Elsevier.Google Scholar
- Lene Nielsen and Sabine Madsen. 2012. The usability expert's fear of agility - An empirical study of global trends and emerging practices. In Proceedings of the 7th Nordic Conference on Human-Computer Interaction: Making Sense Through Design (ACM, '12). Google Scholar
Digital Library
- Robert A. Peterson. 2001. On the use of college students in social science research: Insights from a second-order meta-analysis. Journal of Consumer Research 28:450 - 461.Google Scholar
Cross Ref
- Khanyapuss Punjaisri and Alan Wilson. 2007. The role of internal branding in the delivery of employee brand promise. Journal of Brand Management 15, 1: 57--70.Google Scholar
Cross Ref
- Khanyapuss Punjaisri, Evanschitzky Heiner, and Alan Wilson.2009. Internal branding: An enabler of employee brand-supporting behaviors. Journal of Service Management 20, 2: 209 - 226.Google Scholar
Cross Ref
- Stephanie Rosenbaum, Janice A. Rohn, and Judee Humburg Rosenbaum. 2000. A toolkit for strategic usability: Results from workshops, panels, and surveys. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems (ACM, '00), 327--344. Google Scholar
Digital Library
- Allen Rubin and Earl R. Babbie. 2010. Essential Research Methods for Social Work (2nd ed.).Wadsworth Cengage Learning.Google Scholar
- Jeff Sauro. 2014. Measuring Usability with the System Usability Scale. Retrieved on August 6, 2014 from https://www.measuringusability.com/sus.phpGoogle Scholar
- Jeff Sauro. 2010. Do you need a random sample for your usability test? Retrieved August 15, 2015 from http://www.measuringu.com/blog/random-sample.phpGoogle Scholar
- Jeff Sauro and Joseph S. Dumas. 2009. Comparison of three one-question, post-task usability questionnaires. In Proceedings of the SIGCHI Conference on Human Factors in Computing (CHI'09), 1599--1608... Google Scholar
Digital Library
- Jeff Sauro and James R. Lewis. 2010. Average task times in usability tests: What to report. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI, '10). Google Scholar
Digital Library
- Andrew Sears and Julie A. Jacko (Eds.). 2009. Human-Computer Interaction: Development Process. CRC Press. Google Scholar
Digital Library
- Osama Sohaib and Khan Khalid. 2010. Integrating usability engineering and agile software development: A literature review. In Computer Design and Applications (ICCDA), (IEEE, '10), V2--33.Google Scholar
- Anne S. Tui, Jone L. Pearce, Lyman W. Porter and Angela M. Tripoli. 1997. Alternative approaches to the Employee-Organization Relationship: Does Investment in Employees Pay Off? Academic Management Journal, 40, 5, 1089--1121.Google Scholar
Cross Ref
- Tom Tullis and Bill Albert. 2013. Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics (2nd Ed.). Newnes. Google Scholar
Digital Library
- Brady T. West, Kathleen B. Welch, and Andrez T. Galecki. 2007. Linear Mixed Models: A Practical Guide Using Statistical Software. Taylor & Francis Group.Google Scholar
Index Terms
Utilizing Employees as Usability Participants


Jofish Kaye


Comments