skip to main content
research-article
Public Access

A Usability Study of Four Secure Email Tools Using Paired Participants

Published:09 April 2019Publication History
Skip Abstract Section

Abstract

Secure email is increasingly being touted as usable by novice users, with a push for adoption based on recent concerns about government surveillance. To determine whether secure email is ready for grassroots adoption, we employ a laboratory user study that recruits pairs of novice users to install and use several of the latest systems to exchange secure messages. We present both quantitative and qualitative results from 28 pairs of novices as they use Private WebMail (Pwm), Tutanota, and Virtru and 10 pairs of novices as they use Mailvelope. Participants report being more at ease with this type of study and better able to cope with mistakes since both participants are “on the same page.” We find that users prefer integrated solutions over depot-based solutions and that tutorials are important in helping first-time users. Finally, our results demonstrate that Pretty Good Privacy using manual key management is still unusable for novice users, with 9 of 10 participant pairs failing to complete the study.

References

  1. Ruba Abu-Salma, M. Angela Sasse, Joseph Bonneau, Anastasia Danilova, Alena Naiakshina, and Matthew Smith. 2017. Obstacles to the adoption of secure communication tools. In Proceedings of the IEEE Symposium on Security and Privacy (SP’17). IEEE, Los Alamitos, CA, 137--153.Google ScholarGoogle ScholarCross RefCross Ref
  2. Erinn Atwater, Cecylia Bocovich, Urs Hengartner, Ed Lank, and Ian Goldberg. 2015. Leading Johnny to water: Designing for usability and trust. In Proceedings of the 1th Symposium on Usable Privacy and Security (SOUPS’15). 69--88. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Wei Bai, Moses Namara, Yichen Qian, Patrick Gage Kelley, Michelle L. Mazurek, and Doowon Kim. 2016. An inconvenient trust: User attitudes toward security and usability tradeoffs for key-directory encryption systems. In Proceedings of the 12th Symposium on Usable Privacy and Security (SOUPS’16). 113--130. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Aaron Bangor, Philip Kortum, and James Miller. 2008. An empirical evaluation of the system usability scale. International Journal of Human--Computer Interaction 24, 6 (2008), 574--594.Google ScholarGoogle Scholar
  5. Aaron Bangor, Philip Kortum, and James Miller. 2009. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of Usability Studies 4, 3 (2009), 114--123. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. John Brooke. 1996. SUS—A quick and dirty usability scale. In Usability Evaluation in Industry. CRC Press, Boca Raton, FL.Google ScholarGoogle Scholar
  7. John Brooke. 2013. SUS: A retrospective. Journal of Usability Studies 8, 2 (2013), 29--40. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Zakir Durumeric, David Adrian, Ariana Mirian, James Kasten, Elie Bursztein, Nicolas Lidzborski, Kurt Thomas, et al. 2015. Neither snow nor rain nor MITM…: An empirical analysis of email delivery security. In Proceedings of the 15th ACM Internet Measurement Conference (IMC’15). ACM, New York, NY, 27--39. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Ian D. Foster, Jon Larson, Max Masich, Alex C. Snoeren, Stefan Savage, and Kirill Levchenko. 2015. Security by any other name: On the effectiveness of provider based email security. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (CCS’15). ACM, New York, NY, 450--464. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Simson Garfinkel. 1995. PGP: Pretty Good Privacy. O’Reilly Media, Inc., Sebastopol, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Simson L. Garfinkel. 2003. Email-based identification and authentication: An alternative to PKI? In Proceedings of the 24th IEEE Symposium on Security and Privacy (S8P’03). IEEE, Los Alamitos, CA, 20--26. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Simson L. Garfinkel and Robert C. Miller. 2005. Johnny 2: A user test of key continuity management with S/MIME and Outlook Express. In Proceedings of the 1st Symposium on Usable Privacy and Security (SOUPS’05). ACM, New York, NY, 13--24. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Shirley Gaw, Edward W. Felten, and Patricia Fernandez-Kelly. 2006. Secrecy, flagging, and paranoia: Adoption criteria in encrypted email. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’06). ACM, New York, NY, 591--600. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Ralph Holz, Johanna Amann, Olivier Mehani, Matthias Wachs, and Mohamed Ali Kaafar. 2016. TLS in the wild: An Internet-wide analysis of TLS-based protocols for electronic communication. In Proceedings of the 24th Network and Distributed System Security Symposium (NDSS’16).Google ScholarGoogle ScholarCross RefCross Ref
  15. Ada Lerner, Eric Zeng, and Franziska Roesner. 2017. Confidante: Usable encrypted email: A case study with lawyers and journalists. In Proceedings of the IEEE European Symposium on Security and Privacy (EuroS8P’17). IEEE, Los Alamitos, CA, 385--400.Google ScholarGoogle ScholarCross RefCross Ref
  16. Marcela S. Melara, Aaron Blankstein, Joseph Bonneau, Michael J. Freedman, and Edward W. Felten. 2015. CONIKS: A privacy-preserving consistent key service for secure end-to-end communication. In Proceedings of the 24th USENIX Security Symposium (USENIX Security’15). 383--398. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Stanley Milgram and Ernest Van den Haag. 1978. Obedience to Authority. Ziff--Davis Publishing Company, New York, NY.Google ScholarGoogle Scholar
  18. Damian Poddebniak, Christian Dresen, Jens Müller, Fabian Ising, Sebastian Schinzel, Simon Friedberger, Juraj Somorovsky, et al. 2018. Efail: Breaking S/MIME and OpenPGP email encryption using exfiltration channels. In Proceedings of the 27th USENIX Security Symposium (USENIX Security’18). 549--566. https://www.usenix.org/conference/usenixsecurity18/presentation/poddebniak. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. B. Ramsdell and S. Turner. 2010. Secure/Multipurpose Internet Mail Extensions (S/MIME) Version 3.2 Message Specification. RFC 5751 (Proposed Standard). Retrieved March 19, 2019 from http://www.ietf.org/rfc/rfc5751.txt.Google ScholarGoogle Scholar
  20. Karen Renaud, Melanie Volkamer, and Arne Renkema-Padmos. 2014. Why doesn’t Jane protect her privacy? In Proceedings of the 14th Privacy Enhancing Technologies Symposium (PETS’14). 244--262.Google ScholarGoogle ScholarCross RefCross Ref
  21. Scott Ruoti, Jeff Andersen, Scott Heidbrink, Mark O’Neill, Elham Vaziripour, Justin Wu, Daniel Zappala, et al. 2016a. “We’re on the same page”: A usability study of secure email using pairs of novice users. In Proceedings of the 34th ACM Conference on Human Factors and Computing Systems (CHI’16). ACM, New York, NY, 4298--4308. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Scott Ruoti, Jeff Andersen, Travis Hendershot, Daniel Zappala, and Kent Seamons. 2016b. Private webmail 2.0: Simple and easy-to-use secure email. In Proceedings of the 29th ACM User Interface Software and Technology Symposium (UIST’16). ACM, New York, NY. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Scott Ruoti, Jeff Andersen, Tyler Monson, Daniel Zappala, and Kent Seamons. 2018. A comparative usability study of key management in secure email. In Proceedings of the 14th Symposium on Usable Privacy and Security (SOUPS’18). 375--394. https://www.usenix.org/conference/soups2018/presentation/ruoti. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Scott Ruoti, Nathan Kim, Ben Burgon, Timothy Van Der Horst, and Kent Seamons. 2013. Confused Johnny: When automatic encryption leads to confusion and mistakes. In Proceedings of the 9th Symposium on Usable Privacy and Security (SOUPS’13). ACM, New York, NY. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Scott Ruoti, Brent Roberts, and Kent Seamons. 2015. Authentication melee: A usability analysis of seven web authentication systems. In Proceedings of the 24th International Conference on World Wide Web (WWW’15). ACM, New York, NY, 916--926. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Jeff Sauro. 2011. A Practical Guide to the System Usability Scale: Background, Benchmarks 8 Best Practices. Measuring Usability LLC, Denver, CO.Google ScholarGoogle Scholar
  27. Adi Shamir. 1984. Identity-based cryptosystems and signature schemes. In Proceedings of the 14th International Cryptology Conference (Crypto’84). 47--53. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. S. Sheng, L. Broderick, C. A. Koranda, and J. J. Hyland. 2006. Why Johnny still can’t encrypt: Evaluating the usability of email encryption software. In Proceedings of the Poster Session at the Symposium on Usable Privacy and Security.Google ScholarGoogle Scholar
  29. Yuanzheng Song. 2014. Browser-Based Manual Encryption. Master’s Thesis. Brigham Young University, Provo, UT.Google ScholarGoogle Scholar
  30. Andreas Sotirakopoulos, Kirstie Hawkey, and Konstantin Beznosov. 2010. “I did it because I trusted you”: Challenges with the study environment biasing participant behaviours. In Proceedings of the Usable Security Experiment Reports Workshop at the Symposium on Usable Privacy and Security.Google ScholarGoogle Scholar
  31. Thomas S. Tullis and Jacqueline N. Stetson. 2004. A comparison of questionnaires for assessing website usability. In Proceedings of the Usability Professional Association Conference. 1--12.Google ScholarGoogle Scholar
  32. A. Whitten and J. D. Tygar. 1999. Why Johnny can’t encrypt: A usability evaluation of PGP 5.0. In Proceedings of the 8th USENIX Security Symposium (USENIX Security’99). 14--28. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. A Usability Study of Four Secure Email Tools Using Paired Participants

            Recommendations

            Comments

            Login options

            Check if you have access through your login credentials or your institution to get full access on this article.

            Sign in

            Full Access

            • Published in

              cover image ACM Transactions on Privacy and Security
              ACM Transactions on Privacy and Security  Volume 22, Issue 2
              May 2019
              214 pages
              ISSN:2471-2566
              EISSN:2471-2574
              DOI:10.1145/3316298
              Issue’s Table of Contents

              Copyright © 2019 ACM

              Publisher

              Association for Computing Machinery

              New York, NY, United States

              Publication History

              • Published: 9 April 2019
              • Accepted: 1 February 2019
              • Revised: 1 January 2019
              • Received: 1 March 2018
              Published in tops Volume 22, Issue 2

              Permissions

              Request permissions about this article.

              Request Permissions

              Check for updates

              Qualifiers

              • research-article
              • Research
              • Refereed

            PDF Format

            View or Download as a PDF file.

            PDF

            eReader

            View online with eReader.

            eReader

            HTML Format

            View this article in HTML Format .

            View HTML Format
            About Cookies On This Site

            We use cookies to ensure that we give you the best experience on our website.

            Learn more

            Got it!