Abstract
Secure email is increasingly being touted as usable by novice users, with a push for adoption based on recent concerns about government surveillance. To determine whether secure email is ready for grassroots adoption, we employ a laboratory user study that recruits pairs of novice users to install and use several of the latest systems to exchange secure messages. We present both quantitative and qualitative results from 28 pairs of novices as they use Private WebMail (Pwm), Tutanota, and Virtru and 10 pairs of novices as they use Mailvelope. Participants report being more at ease with this type of study and better able to cope with mistakes since both participants are “on the same page.” We find that users prefer integrated solutions over depot-based solutions and that tutorials are important in helping first-time users. Finally, our results demonstrate that Pretty Good Privacy using manual key management is still unusable for novice users, with 9 of 10 participant pairs failing to complete the study.
- Ruba Abu-Salma, M. Angela Sasse, Joseph Bonneau, Anastasia Danilova, Alena Naiakshina, and Matthew Smith. 2017. Obstacles to the adoption of secure communication tools. In Proceedings of the IEEE Symposium on Security and Privacy (SP’17). IEEE, Los Alamitos, CA, 137--153.Google Scholar
Cross Ref
- Erinn Atwater, Cecylia Bocovich, Urs Hengartner, Ed Lank, and Ian Goldberg. 2015. Leading Johnny to water: Designing for usability and trust. In Proceedings of the 1th Symposium on Usable Privacy and Security (SOUPS’15). 69--88. Google Scholar
Digital Library
- Wei Bai, Moses Namara, Yichen Qian, Patrick Gage Kelley, Michelle L. Mazurek, and Doowon Kim. 2016. An inconvenient trust: User attitudes toward security and usability tradeoffs for key-directory encryption systems. In Proceedings of the 12th Symposium on Usable Privacy and Security (SOUPS’16). 113--130. Google Scholar
Digital Library
- Aaron Bangor, Philip Kortum, and James Miller. 2008. An empirical evaluation of the system usability scale. International Journal of Human--Computer Interaction 24, 6 (2008), 574--594.Google Scholar
- Aaron Bangor, Philip Kortum, and James Miller. 2009. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of Usability Studies 4, 3 (2009), 114--123. Google Scholar
Digital Library
- John Brooke. 1996. SUS—A quick and dirty usability scale. In Usability Evaluation in Industry. CRC Press, Boca Raton, FL.Google Scholar
- John Brooke. 2013. SUS: A retrospective. Journal of Usability Studies 8, 2 (2013), 29--40. Google Scholar
Digital Library
- Zakir Durumeric, David Adrian, Ariana Mirian, James Kasten, Elie Bursztein, Nicolas Lidzborski, Kurt Thomas, et al. 2015. Neither snow nor rain nor MITM…: An empirical analysis of email delivery security. In Proceedings of the 15th ACM Internet Measurement Conference (IMC’15). ACM, New York, NY, 27--39. Google Scholar
Digital Library
- Ian D. Foster, Jon Larson, Max Masich, Alex C. Snoeren, Stefan Savage, and Kirill Levchenko. 2015. Security by any other name: On the effectiveness of provider based email security. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (CCS’15). ACM, New York, NY, 450--464. Google Scholar
Digital Library
- Simson Garfinkel. 1995. PGP: Pretty Good Privacy. O’Reilly Media, Inc., Sebastopol, CA. Google Scholar
Digital Library
- Simson L. Garfinkel. 2003. Email-based identification and authentication: An alternative to PKI? In Proceedings of the 24th IEEE Symposium on Security and Privacy (S8P’03). IEEE, Los Alamitos, CA, 20--26. Google Scholar
Digital Library
- Simson L. Garfinkel and Robert C. Miller. 2005. Johnny 2: A user test of key continuity management with S/MIME and Outlook Express. In Proceedings of the 1st Symposium on Usable Privacy and Security (SOUPS’05). ACM, New York, NY, 13--24. Google Scholar
Digital Library
- Shirley Gaw, Edward W. Felten, and Patricia Fernandez-Kelly. 2006. Secrecy, flagging, and paranoia: Adoption criteria in encrypted email. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’06). ACM, New York, NY, 591--600. Google Scholar
Digital Library
- Ralph Holz, Johanna Amann, Olivier Mehani, Matthias Wachs, and Mohamed Ali Kaafar. 2016. TLS in the wild: An Internet-wide analysis of TLS-based protocols for electronic communication. In Proceedings of the 24th Network and Distributed System Security Symposium (NDSS’16).Google Scholar
Cross Ref
- Ada Lerner, Eric Zeng, and Franziska Roesner. 2017. Confidante: Usable encrypted email: A case study with lawyers and journalists. In Proceedings of the IEEE European Symposium on Security and Privacy (EuroS8P’17). IEEE, Los Alamitos, CA, 385--400.Google Scholar
Cross Ref
- Marcela S. Melara, Aaron Blankstein, Joseph Bonneau, Michael J. Freedman, and Edward W. Felten. 2015. CONIKS: A privacy-preserving consistent key service for secure end-to-end communication. In Proceedings of the 24th USENIX Security Symposium (USENIX Security’15). 383--398. Google Scholar
Digital Library
- Stanley Milgram and Ernest Van den Haag. 1978. Obedience to Authority. Ziff--Davis Publishing Company, New York, NY.Google Scholar
- Damian Poddebniak, Christian Dresen, Jens Müller, Fabian Ising, Sebastian Schinzel, Simon Friedberger, Juraj Somorovsky, et al. 2018. Efail: Breaking S/MIME and OpenPGP email encryption using exfiltration channels. In Proceedings of the 27th USENIX Security Symposium (USENIX Security’18). 549--566. https://www.usenix.org/conference/usenixsecurity18/presentation/poddebniak. Google Scholar
Digital Library
- B. Ramsdell and S. Turner. 2010. Secure/Multipurpose Internet Mail Extensions (S/MIME) Version 3.2 Message Specification. RFC 5751 (Proposed Standard). Retrieved March 19, 2019 from http://www.ietf.org/rfc/rfc5751.txt.Google Scholar
- Karen Renaud, Melanie Volkamer, and Arne Renkema-Padmos. 2014. Why doesn’t Jane protect her privacy? In Proceedings of the 14th Privacy Enhancing Technologies Symposium (PETS’14). 244--262.Google Scholar
Cross Ref
- Scott Ruoti, Jeff Andersen, Scott Heidbrink, Mark O’Neill, Elham Vaziripour, Justin Wu, Daniel Zappala, et al. 2016a. “We’re on the same page”: A usability study of secure email using pairs of novice users. In Proceedings of the 34th ACM Conference on Human Factors and Computing Systems (CHI’16). ACM, New York, NY, 4298--4308. Google Scholar
Digital Library
- Scott Ruoti, Jeff Andersen, Travis Hendershot, Daniel Zappala, and Kent Seamons. 2016b. Private webmail 2.0: Simple and easy-to-use secure email. In Proceedings of the 29th ACM User Interface Software and Technology Symposium (UIST’16). ACM, New York, NY. Google Scholar
Digital Library
- Scott Ruoti, Jeff Andersen, Tyler Monson, Daniel Zappala, and Kent Seamons. 2018. A comparative usability study of key management in secure email. In Proceedings of the 14th Symposium on Usable Privacy and Security (SOUPS’18). 375--394. https://www.usenix.org/conference/soups2018/presentation/ruoti. Google Scholar
Digital Library
- Scott Ruoti, Nathan Kim, Ben Burgon, Timothy Van Der Horst, and Kent Seamons. 2013. Confused Johnny: When automatic encryption leads to confusion and mistakes. In Proceedings of the 9th Symposium on Usable Privacy and Security (SOUPS’13). ACM, New York, NY. Google Scholar
Digital Library
- Scott Ruoti, Brent Roberts, and Kent Seamons. 2015. Authentication melee: A usability analysis of seven web authentication systems. In Proceedings of the 24th International Conference on World Wide Web (WWW’15). ACM, New York, NY, 916--926. Google Scholar
Digital Library
- Jeff Sauro. 2011. A Practical Guide to the System Usability Scale: Background, Benchmarks 8 Best Practices. Measuring Usability LLC, Denver, CO.Google Scholar
- Adi Shamir. 1984. Identity-based cryptosystems and signature schemes. In Proceedings of the 14th International Cryptology Conference (Crypto’84). 47--53. Google Scholar
Digital Library
- S. Sheng, L. Broderick, C. A. Koranda, and J. J. Hyland. 2006. Why Johnny still can’t encrypt: Evaluating the usability of email encryption software. In Proceedings of the Poster Session at the Symposium on Usable Privacy and Security.Google Scholar
- Yuanzheng Song. 2014. Browser-Based Manual Encryption. Master’s Thesis. Brigham Young University, Provo, UT.Google Scholar
- Andreas Sotirakopoulos, Kirstie Hawkey, and Konstantin Beznosov. 2010. “I did it because I trusted you”: Challenges with the study environment biasing participant behaviours. In Proceedings of the Usable Security Experiment Reports Workshop at the Symposium on Usable Privacy and Security.Google Scholar
- Thomas S. Tullis and Jacqueline N. Stetson. 2004. A comparison of questionnaires for assessing website usability. In Proceedings of the Usability Professional Association Conference. 1--12.Google Scholar
- A. Whitten and J. D. Tygar. 1999. Why Johnny can’t encrypt: A usability evaluation of PGP 5.0. In Proceedings of the 8th USENIX Security Symposium (USENIX Security’99). 14--28. Google Scholar
Digital Library
Index Terms
A Usability Study of Four Secure Email Tools Using Paired Participants
Recommendations
"We're on the Same Page": A Usability Study of Secure Email Using Pairs of Novice Users
CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing SystemsSecure email is increasingly being touted as usable by novice users, with a push for adoption based on recent concerns about government surveillance. To determine whether secure email is ready for grassroots adoption, we employ a laboratory user study ...
Private Webmail 2.0: Simple and Easy-to-Use Secure Email
UIST '16: Proceedings of the 29th Annual Symposium on User Interface Software and TechnologyPrivate Webmail 2.0 (Pwm 2.0) improves upon the current state of the art by increasing the usability and practical security of secure email for ordinary users. More users are able to send and receive encrypted emails without mistakenly revealing ...
Secure Email - A Usability Study
Financial Cryptography and Data SecurityAbstractSeveral end-to-end encryption technologies for emails such as PGP and S/MIME exist since decades. However, end-to-end encryption is barely applied. To understand why users hesitate to secure their email communication and which usability issues ...






Comments