Abstract
App stores provide access to millions of different programs that users can download on their computers. Developers can also make their programs available for download on their websites and host the program files either directly on their website or on third-party platforms, such as mirrors. In the latter case, as users download the software without any vetting from the developers, they should take the necessary precautions to ensure that it is authentic. One way to accomplish this is to check that the published file’s integrity verification code—the checksum—matches that (if provided) of the downloaded file. To date, however, there is little evidence to suggest that such a process is effective. Even worse, very few usability studies about it exist.
In this article, we provide the first comprehensive study that assesses the usability and effectiveness of the manual checksum verification process. First, by means of an in-situ experiment with 40 participants and eye-tracking technology, we show that the process is cumbersome and error-prone. Second, after a 4-month-long in-the-wild experiment with 134 participants, we demonstrate how our proposed solution—a Chrome extension that verifies checksums automatically—significantly reduces human errors, improves coverage, and has only limited impact on usability. It also confirms that, sadly, only a tiny minority of websites that link to executable files in our sample provide checksums (0.01%), which is a strong call to action for web standards bodies, service providers, and content creators to increase the use of file integrity verification on their properties.
- Mauro Cherubini, Alexandre Meylan, Bertil Chapuis, Mathias Humbert, Igor Bilogrevic, and Kévin Huguenin. 2018. Towards usable checksums: Automating the integrity verification of web downloads for the masses. In Proceedings of the Conference on Computer and Communications Security (CCS). ACM, 1256--1271. DOI:http://dx.doi.org/10.1145/3243734.3243746Google Scholar
Digital Library
- Karen Turner. 2016-07-15. Developers consider Apple’s app store restrictive and anticompetitive, report shows. Washington Post (2016-07-15).Google Scholar
- Swati Khandelwal. 2018. Flaw in Popular Transmission BitTorrent Client Lets Hackers Control Your PC Remotely. https://thehackernews.com/2018/01/bittorent-transmission-hacking.html. (2018).Google Scholar
- Linux Mint Website Hacked; ISO Downloads Replaced with a Backdoor. Security News - Trend Micro USA. https://www.trendmicro.com/vinfo/us/security/news/cybercrime-and-digital-threats/linux-mint-website-hacked-iso-downloads-replaced-with-a-backdoor. ([n.d.]).Google Scholar
- Hsu-Chun Hsiao, Yue-Hsun Lin, Ahren Studer, Cassandra Studer, King-Hang Wang, Hiroaki Kikuchi, Adrian Perrig, Hung-Min Sun, and Bo-Yin Yang. 2009. A study of user-friendly hash comparison schemes. In Proceedings of the Computer Security Applications Conference (ACSAC). IEEE, 105--114. DOI:http://dx.doi.org/10.1109/ACSAC.2009.20Google Scholar
Digital Library
- Sergej Dechand, Dominik Schürmann, Karoline Busse, Yasemin Acar, Sascha Fahl, and Matthew Smith. 2016. An empirical study of textual key-fingerprint representations. In Proceedings of the USENIX Security Symposium (USENIX Security). USENIX.Google Scholar
- W3C. 2016. Subresource Integrity. https://www.w3.org/TR/SRI/. (2016).Google Scholar
- S. M. Furnell, P. Bryant, and A. D. Phippen. 2007. Assessing the security perceptions of personal Internet users. Computers 8 Security 26, 5 (Aug. 2007), 410--417. DOI:http://dx.doi.org/10.1016/j.cose.2007.03.001Google Scholar
- Catherine L. Anderson and Ritu Agarwal. 2010. Practicing safe computing: A multimedia empirical examination of home computer user security behavioral intentions. MIS Q. 34, 3 (Sept. 2010), 613--643.Google Scholar
Digital Library
- Vaidya Rishi. 2018. Cyber Security Breaches Survey 2018. Survey. United Kingdom.Google Scholar
- Elissa M. Redmiles, Sean Kross, and Michelle L. Mazurek. 2017. Where is the digital divide?: A survey of security, privacy, and socioeconomics. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI). ACM, 931--936. DOI:http://dx.doi.org/10.1145/3025453.3025673Google Scholar
- Elissa M. Redmiles, Sean Kross, and Michelle L. Mazurek. 2016. How I learned to be secure: A census-representative survey of security advice sources and behavior. In Proceedings of the ACM Conference on Computer and Communications Security (CCS). ACM, 666--677. DOI:http://dx.doi.org/10.1145/2976749.2978307Google Scholar
- Elisa M. Redmiles, Amelia R. Malone, and Michelle L. Mazurek. 2016. I think they’re trying to tell me something: Advice sources and selection for digital security. In Proceedings of the IEEE Symposium on Security and Privacy (S8P). 272--288. DOI:http://dx.doi.org/10.1109/SP.2016.24Google Scholar
- Serge Egelman, Lorrie Faith Cranor, and Jason Hong. 2008. You’ve been warned: An empirical study of the effectiveness of web browser phishing warnings. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI). ACM, 1065--1074. DOI:http://dx.doi.org/10.1145/1357054.1357219Google Scholar
Digital Library
- Joshua Sunshine, Serge Egelman, Hazim Almuhimedi, Neha Atri, and Lorrie Faith Cranor. 2009. Crying wolf: An empirical study of SSL warning effectiveness. In Proceedings of the USENIX Security Symposium (USENIX Security). USENIX, 399--416.Google Scholar
- Devdatta Akhawe and Adrienne Porter Felt. 2013. Alice in warningland: A large-scale field study of browser security warning effectiveness. In Proceedings of the USENIX Security Symposium (USENIX Security). USENIX.Google Scholar
- Serge Egelman and Stuart Schechter. 2013. The importance of being earnest [in security warnings]. In Proceedings of the International Conference on Financial Cryptography and Data Security (FC). Springer, 52--59. DOI:http://dx.doi.org/10.1007/978-3-642-39884-1_5Google Scholar
Cross Ref
- David Modic and Ross Anderson. 2014. Reading this may harm your computer: The psychology of malware warnings. Computers in Human Behavior 41 (2014), 71--79.Google Scholar
Digital Library
- Antonio Bianchi, Jacopo Corbetta, Luca Invernizzi, Yanick Fratantonio, Christopher Kruegel, and Giovanni Vigna. 2015. What the app is that? Deception and countermeasures in the Android user interface. In Proceedings of the IEEE Symposium on Security and Privacy (S8P). IEEE, 931--948. DOI:http://dx.doi.org/10.1109/SP.2015.62Google Scholar
Digital Library
- Jeffrey L. Jenkins, Bonnie Brinton Anderson, Anthony Vance, C. Brock Kirwan, and David Eargle. 2016. More harm than good? How messages that interrupt can make us vulnerable. Information Systems Research 27, 4 (2016), 880--896.Google Scholar
Digital Library
- Mario Silic and Andrea Back. 2017. Deterrent effects of warnings on user’s behavior in preventing malicious software use. In Proceedings of the Hawaii International Conference on System Sciences (HICSS).Google Scholar
Cross Ref
- Robert W. Reeder, Adrienne Porter Felt, Sunny Consolvo, Nathan Malkin, Christopher Thompson, and Serge Egelman. 2018. An experience sampling study of user reactions to browser warnings in the field. In Proceedings of the Conference on Human Factors in Computing Systems (CHI). ACM, 512:1–512:13. DOI:http://dx.doi.org/10.1145/3173574.3174086Google Scholar
Digital Library
- Cristian Bravo-Lillo, Saranga Komanduri, Lorrie Faith Cranor, Robert W. Reeder, Manya Sleeper, Julie Downs, and Stuart Schechter. 2013. Your attention please: Designing security-decision UIs to make genuine risks harder to ignore. In Proceedings of the Symposium on Usable Privacy and Security (SOUPS). ACM, 6:1–6:12. DOI:http://dx.doi.org/10.1145/2501604.2501610Google Scholar
Digital Library
- Catherine S. Weir, Gary Douglas, Martin Carruthers, and Mervyn Jack. 2009. User perceptions of security, convenience and usability for ebanking authentication tokens. Computers 8 Security 28, 1-2 (2009), 47--62.Google Scholar
- Leona Tam, Myron Glassman, and Mark Vandenwauver. 2010. The psychology of password management: A tradeoff between security and convenience. Behaviour 8 Information Technology 29, 3 (2010), 233--244.Google Scholar
- Michael Fagan and Mohammad Maifi Hasan Khan. 2016. Why do they do what they do?: A study of what motivates users to (not) follow computer security advice. In Proceedings of the Symposium on Usable Privacy and Security (SOUPS). ACM, 59--75.Google Scholar
- K. Krombholz, K. Busse, K. Pfeffer, M. Smith, and E. von Zezschwitz. 2019. “If HTTPS were secure, I wouldn’t need 2FA” - End user and administrator mental models of HTTPS. In Proceedings of the IEEE Symposium on Security and Privacy (S8P). IEEE, 1138--1155. DOI:http://dx.doi.org/10.1109/SP.2019.00060Google Scholar
Cross Ref
- Joshua Tan, Lujo Bauer, Joseph Bonneau, Lorrie Faith Cranor, Jeremy Thomas, and Blase Ur. 2017. Can unicorns help users compare crypto key fingerprints? In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI). ACM, 3787--3798. DOI:http://dx.doi.org/10.1145/3025453.3025733Google Scholar
Digital Library
- N. Unger, S. Dechand, J. Bonneau, S. Fahl, H. Perl, I. Goldberg, and M. Smith. 2015. SoK: Secure messaging. In Proceedings of the IEEE Symposium on Security and Privacy (S8P). IEEE, 232--249. DOI:http://dx.doi.org/10.1109/SP.2015.22Google Scholar
- Ruba Abu-Salma, M. Angela Sasse, Joseph Bonneau, Anastasia Danilova, Alena Naiakshina, and Matthew Smith. 2017. Obstacles to the adoption of secure communication tools. In Proceedings of the IEEE Symposium on Security and Privacy (S8P). IEEE, 137--153. DOI:http://dx.doi.org/10.1109/SP.2017.65Google Scholar
Cross Ref
- Elham Vaziripour, Justin Wu, Mark O’Neill, Ray Clinton, Jordan Whitehead, Scott Heidbrink, Kent Seamons, and Daniel Zappala. 2017. Is that you, Alice? A usability study of the authentication ceremony of secure messaging applications. In Proceedings of the Symposium on Usable Privacy and Security (SOUPS). ACM.Google Scholar
- Checksum On the Go. Chrome Webstore. https://chrome.google.com/webstore/detail/checksum-on-the-go/fholnooplijidhdagedffljaphholpea. ([n.d.]).Google Scholar
- Files MD5 SHA1 Calculate 8 Compare. Add-Ons for Firefox. https://addons.mozilla.org/en-US/firefox/addon/calculate-md5-sha1-hash-che-1/?src=search. ([n.d.]).Google Scholar
- Certificates and Digitally Signed Applications: A Double Edged Sword. https://eventtracker.com/tech-articles/certificates-and-digitally-signed-applications-a-double-edged-sword/. (Feb. 2016).Google Scholar
- Nevena Vratonjic, Julien Freudiger, Vincent Bindschaedler, and Jean-Pierre Hubaux. 2013. The inconvenient truth about web certificates. In Proceedings of the Workshop on Economics of Information Security and Privacy (WEIS). Springer, 79--117. DOI:http://dx.doi.org/10.1007/978-1-4614-1981-5_5Google Scholar
Cross Ref
- Justin Cappos, Justin Samuel, Scott Baker, and John H. Hartman. 2008. A look in the mirror: Attacks on package managers. In Proceedings of the ACM Conference on Computer and Communications Security (CCS). ACM, 565--574. DOI:http://dx.doi.org/10.1145/1455770.1455841Google Scholar
- Bart Preneel. 1994. Cryptographic hash functions. Transactions on Emerging Telecommunications Technologies 5, 4 (1994), 431--448.Google Scholar
Cross Ref
- Computer Security Division, Information Technology Laboratory. NIST Policy on Hash Functions - Hash Functions | CSRC. https://csrc.nist.gov/projects/hash-functions/nist-policy-on-hash-functions. ([n.d.]).Google Scholar
- Bertil Chapuis, Olamide Omolola, Mauro Cherubini, Mathias Humbert, and Kévin Huguenin. 2020. An empirical study of the use of integrity verification mechanisms for web subresources. In Proceedings of the Web Conference. ACM, 34--45. DOI:http://dx.doi.org/10.1145/3366423.3380092Google Scholar
Digital Library
- Bonnie Brinton Anderson, C. Brock Kirwan, Jeffrey L. Jenkins, David Eargle, Seth Howard, and Anthony Vance. 2015. How polymorphic warnings reduce habituation in the brain: Insights from an fMRI study. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI). ACM, 2883--2892. DOI:http://dx.doi.org/10.1145/2702123.2702322Google Scholar
Digital Library
- Adrienne Porter Felt, Alex Ainslie, Robert W. Reeder, Sunny Consolvo, Somas Thyagaraja, Alan Bettes, Helen Harris, and Jeff Grimes. 2015. Improving SSL warnings: Comprehension and adherence. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI). ACM, 2893--2902. DOI:http://dx.doi.org/10.1145/2702123.2702442Google Scholar
Digital Library
- Mario Silic, Jordan Barlow, and Dustin Ormond. 2015. Warning! A comprehensive model of the effects of digital information security warning messages. In Proceedings of the IFIP Workshop on Information Systems Security Research. IFIP.Google Scholar
- Alex Poole and Linden J. Ball. 2006. Eye tracking in human-computer interaction and usability research: Current status and future prospects. In Encyclopedia of Human Computer Interaction. 13.Google Scholar
- Joseph H. Goldberg, Mark J. Stimson, Marion Lewenstein, Neil Scott, and Anna M. Wichansky. 2002. Eye tracking in web search tasks: Design implications. In Proceedings of the Symposium on Eye Tracking Research 8 Applications (ETRA). ACM, 51. DOI:http://dx.doi.org/10.1145/507072.507082Google Scholar
- Marcel Adam Just and Patricia A. Carpenter. 1976. Eye fixations and cognitive processes. Cognitive Psychology 8, 4 (Oct. 1976), 441--480. DOI:http://dx.doi.org/10.1016/0010-0285(76)90015-3Google Scholar
- Mauro Cherubini and Nuria Oliver. 2009. A refined experience sampling method to capture mobile user experience. arXiv:0906.4125 [cs] (June 2009). arxiv:cs/0906.4125Google Scholar
- S. Consolvo and M. Walker. 2003. Using the experience sampling method to evaluate ubicomp applications. IEEE Pervasive Computing 2, 2 (Jun 2003), 24--31. DOI:http://dx.doi.org/10.1109/MPRV.2003.1203750Google Scholar
Digital Library
- Giovanni Iachello, Khai N. Truong, Gregory D. Abowd, Gillian R. Hayes, and Molly Stevens. 2006. Prototyping and sampling experience to evaluate ubiquitous computing privacy in the real world. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI). ACM, 1009--1018. DOI:http://dx.doi.org/10.1145/1124772.1124923Google Scholar
Digital Library
- Clara Mancini, Keerthi Thomas, Yvonne Rogers, Blaine A. Price, Lukazs Jedrzejczyk, Arosha K. Bandara, Adam N. Joinson, and Bashar Nuseibeh. 2009. From spaces to places: Emerging contexts in mobile privacy. In Proceedings of the International Conference on Ubiquitous Computing (UbiComp). ACM, 1--10. DOI:http://dx.doi.org/10.1145/1620545.1620547Google Scholar
Digital Library
- Stephen S. Intille, John Rondoni, Charles Kukla, Isabel Ancona, and Ling Bao. 2003. A context-aware experience sampling tool. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI) - Extended Abstracts. ACM, 972--973. DOI:http://dx.doi.org/10.1145/765891.766101Google Scholar
Digital Library
- Florian Schmitt, Jan Gassen, and Elmar Gerhards-Padilla. 2012. PDF scrutinizer: Detecting JavaScript-based attacks in PDF documents. In Proceedings of the International Conference on Privacy, Security and Trust (PST). IEEE, 104--111. DOI:http://dx.doi.org/10.1109/PST.2012.6297926Google Scholar
Digital Library
- Michael Crabb and Vicki L. Hanson. 2014. Age, technology usage, and cognitive characteristics in relation to perceived disorientation and reported website ease of use. In Proceedings of the International ACM SIGACCESS Conference on Computers 8 Accessibility (ASSETS). ACM, 193--200. DOI:http://dx.doi.org/10.1145/2661334.2661356Google Scholar
Index Terms
A Study on the Use of Checksums for Integrity Verification of Web Downloads
Recommendations
Towards Usable Checksums: Automating the Integrity Verification of Web Downloads for the Masses
CCS '18: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications SecurityInternet users can download software for their computers from app stores (e.g., Mac App Store and Windows Store) or from other sources, such as the developers' websites. Most Internet users in the US rely on the latter, according to our representative ...
An Empirical Study of the Usage of Checksums for Web Downloads
WWW '23: Proceedings of the ACM Web Conference 2023Checksums, typically provided on webpages and generated from cryptographic hash functions (e.g., MD5, SHA256) or signature schemes (e.g., PGP), are commonly used on websites to enable users to verify that the files they download have not been tampered ...
Using Checksums to Detect Number Entry Error
CHI '13: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsNumber entry is a common task in many domains. In safety-critical environments such as air traffic control or on hospital wards, incorrect number entry can have serious harmful consequences. Research has investigated how interface designs can help ...






Comments