skip to main content
research-article

The Tip of the Iceberg: On the Merits of Finding Security Bugs

Published:28 September 2020Publication History
Skip Abstract Section

Abstract

In this article, we investigate a fundamental question regarding software security: Is the security of SW releases increasing over time? We approach this question with a detailed analysis of the large body of open-source software packaged in the popular Debian GNU/Linux distribution. Contrary to common intuition, we find no clear evidence that the vulnerability rate of widely used software decreases over time: Even in popular and “stable” releases, the fixing of bugs does not seem to reduce the rate of newly identified vulnerabilities. The intuitive conclusion is worrisome: Commonly employed development and validation procedures do not seem to scale with the increase of features and complexity—they are only chopping pieces off the top of an iceberg of vulnerabilities.

To the best of our knowledge, this is the first investigation into the problem that studies a complete distribution of software, spanning multiple versions. Although we can not give a definitive answer, we show that several popular beliefs also cannot be confirmed given our dataset. We publish our Debian Vulnerability Analysis Framework (DVAF), an automated dataset creation and analysis process, to enable reproduction and further analysis of our results. Overall, we hope our contributions provide important insights into the vulnerability discovery process and help in identifying effective techniques for vulnerability analysis and prevention.

References

  1. Debian Project. 2016. Debian security FAQ. Retrieved from https://www.debian.org/security/faq.Google ScholarGoogle Scholar
  2. O. H. Alhazmi and Y. K. Malaiya. 2005. Quantitative vulnerability assessment of systems software. In Proceedings of the Annual Reliability and Maintainability Symposium. IEEE, 615--620.Google ScholarGoogle Scholar
  3. Luca Allodi. 2017. Economic factors of vulnerability trade and exploitation. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security (CCS’17), Bhavani M. Thuraisingham, David Evans, Tal Malkin, and Dongyan Xu (Eds.). ACM, 1483--1499. DOI:https://doi.org/10.1145/3133956.3133960Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Jeff Alstott, Ed Bullmore, and Dietmar Plenz. 2014. Powerlaw: A Python package for analysis of heavy-tailed distributions. PloS One 9, 1 (2014), e85777.Google ScholarGoogle ScholarCross RefCross Ref
  5. Juan José Amor, Gregorio Robles, Jesus M. González-Barahona, and Francisco Rivas. 2009. Measuring Lenny: The size of Debian 5.0. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.464.51088rep=rep18type=pdf.Google ScholarGoogle Scholar
  6. Jason Bau, Elie Bursztein, Divij Gupta, and John C. Mitchell. 2010. State of the art: Automated black-box web application vulnerability testing. In Proceedings of the 31st IEEE Symposium on Security and Privacy (S8P’10). IEEE Computer Society, 332--345. DOI:https://doi.org/10.1109/SP.2010.27Google ScholarGoogle Scholar
  7. Leyla Bilge and Tudor Dumitras. 2012. Before we knew it: An empirical study of zero-day attacks in the real world. In Proceedings of the ACM Conference on Computer and Communications Security (CCS’12), Ting Yu, George Danezis, and Virgil D. Gligor (Eds.). ACM, 833--844. DOI:https://doi.org/10.1145/2382196.2382284Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Priyam Biswas, Alessandro Di Federico, Scott A. Carr, Prabhu Rajasekaran, Stijn Volckaert, Yeoul Na, Michael Franz, and Mathias Payer. 2017. Venerable variadic vulnerabilities vanquished. In Proceedings of the 26th USENIX Security Symposium (USENIX Security’17), Engin Kirda and Thomas Ristenpart (Eds.). USENIX Association, 186--198. Retrieved from https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/biswas.Google ScholarGoogle Scholar
  9. Matteo Bortolozzo, Matteo Centenaro, Riccardo Focardi, and Graham Steel. 2010. Attacking and fixing PKCS#11 security tokens. In Proceedings of the 17th ACM Conference on Computer and Communications Security (CCS’10), Ehab Al-Shaer, Angelos D. Keromytis, and Vitaly Shmatikov (Eds.). ACM, 260--269. DOI:https://doi.org/10.1145/1866307.1866337Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Fraser Brown, Shravan Narayan, Riad S. Wahby, Dawson R. Engler, Ranjit Jhala, and Deian Stefan. 2017. Finding and preventing bugs in JavaScript bindings. In Proceedings of the IEEE Symposium on Security and Privacy (SP’17). IEEE Computer Society, 559--578. DOI:https://doi.org/10.1109/SP.2017.68Google ScholarGoogle ScholarCross RefCross Ref
  11. Sven Bugiel, Lucas Vincenzo Davi, and Steffen Schulz. 2011. Scalable trust establishment with software reputation. In Proceedings of the 6th ACM Workshop on Scalable Trusted Computing ([email protected]’11), Yan Chen, Shouhuai Xu, Ahmad-Reza Sadeghi, and Xinwen Zhang (Eds.). ACM, 15--24. DOI:https://doi.org/10.1145/2046582.2046587Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Sandy Clark, Stefan Frei, Matt Blaze, and Jonathan M. Smith. 2010. Familiarity breeds contempt: The honeymoon effect and the role of legacy code in zero-day vulnerabilities. In Proceedings of the 26th Annual Computer Security Applications Conference (ACSAC’10), Carrie Gates, Michael Franz, and John P. McDermott (Eds.). ACM, 251--260. DOI:https://doi.org/10.1145/1920261.1920299Google ScholarGoogle Scholar
  13. Aaron Clauset, Cosma Rohilla Shalizi, and Mark E. J. Newman. 2009. Power-law distributions in empirical data. SIAM Rev. 51, 4 (2009), 661--703. DOI:https://doi.org/10.1137/070710111Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. CVE Details. 2020. Browse Vulnerabilities by Date. Retrieved from https://www.cvedetails.com/browse-by-date.php.Google ScholarGoogle Scholar
  15. Nigel Edwards and Liqun Chen. 2012. An historical examination of open source releases and their vulnerabilities. In Proceedings of the ACM Conference on Computer and Communications Security (CCS’12), Ting Yu, George Danezis, and Virgil D. Gligor (Eds.). ACM, 183--194. DOI:https://doi.org/10.1145/2382196.2382218Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Jim Finkle and Supriya Kurane. 2014. U.S. hospital breach biggest yet to exploit Heartbleed bug: Expert. Reuters (2014). Retrieved from https://www.reuters.com/article/us-community-health-cybersecurity/u-s-hospital-breach-biggest-yet-to-exploit-heartbleed-bug-expert-idUSKBN0GK0H420140820.Google ScholarGoogle Scholar
  17. Andy Greenberg. 2014. Hackers are already using the Shellshock bug to launch botnet attacks. Wired (2014). Retrieved from https://www.wired.com/2014/09/hackers-already-using-shellshock-bug-create-botnets-ddos-attacks/.Google ScholarGoogle Scholar
  18. HackerOne. 2017. The Hacker-powered security report 2017. Retrieved from https://www.hackerone.com/resources/hacker-powered-security-report.Google ScholarGoogle Scholar
  19. Munawar Hafiz and Ming Fang. 2016. Game of detections: How are security vulnerabilities discovered in the wild?Empir. Softw. Eng. 21, 5 (2016), 1920--1959. DOI:https://doi.org/10.1007/s10664-015-9403-7Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Jinyoo Kim, Yashwant K. Malaiya, and Indrakshi Ray. 2007. Vulnerability discovery in multi-version software systems. In Proceedings of the 10th IEEE International Symposium on High Assurance Systems Engineering (HASE’07). IEEE Computer Society, 141--148. DOI:https://doi.org/10.1109/HASE.2007.55Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Meir M. Lehman. 1980. Programs, life cycles, and laws of software evolution. Proc. IEEE 68, 9 (1980), 1060--1076.Google ScholarGoogle ScholarCross RefCross Ref
  22. Meir M. Lehman, Juan F. Ramil, Paul Wernick, Dewayne E. Perry, and Wladyslaw M. Turski. 1997. Metrics and laws of software evolution—The nineties view. In Proceedings of the 4th IEEE International Software Metrics Symposium (METRICS’97). IEEE Computer Society, 20. DOI:https://doi.org/10.1109/METRIC.1997.637156Google ScholarGoogle ScholarCross RefCross Ref
  23. Nancy G. Leveson. 2009. Software challenges in achieving space safety. J. Brit. Interplan. Soc. 62 (2009), 265–272.Google ScholarGoogle Scholar
  24. Frank Li and Vern Paxson. 2017. A large-scale empirical study of security patches. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security (CCS’17), Bhavani M. Thuraisingham, David Evans, Tal Malkin, and Dongyan Xu (Eds.). ACM, 2201--2215. DOI:https://doi.org/10.1145/3133956.3134072Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Thomas Maillart, Mingyi Zhao, Jens Grossklags, and John Chuang. 2017. Given enough eyeballs, all bugs are shallow? Revisiting Eric Raymond with bug bounty programs. J. Cybersecur. 3, 2 (2017), 81--90. DOI:https://doi.org/10.1093/cybsec/tyx008Google ScholarGoogle ScholarCross RefCross Ref
  26. Antonio Nappa, Richard Johnson, Leyla Bilge, Juan Caballero, and Tudor Dumitras. 2015. The attack of the clones: A study of the impact of shared code on vulnerability patching. In Proceedings of the IEEE Symposium on Security and Privacy (SP’15). 692--708. DOI:https://doi.org/10.1109/SP.2015.48Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Matús Nemec, Dusan Klinec, Petr Svenda, Peter Sekan, and Vashek Matyas. 2017. Measuring popularity of cryptographic libraries in internet-wide scans. In Proceedings of the 33rd Annual Computer Security Applications Conference. ACM, 162--175. DOI:https://doi.org/10.1145/3134600.3134612Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Andy Ozment and Stuart E. Schechter. 2006. Milk or wine: Does software security improve with age? In Proceedings of the 15th USENIX Security Symposium, Angelos D. Keromytis (Ed.). USENIX Association. https://www.usenix.org/conference/15th-usenix-security-symposium/milk-or-wine-does-software-security-improve-age.Google ScholarGoogle Scholar
  29. James Andrew Ozment. 2007. Vulnerability Discovery 8 Software Security. Ph.D. Dissertation. University of Cambridge, UK. Retrieved from http://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.613340.Google ScholarGoogle Scholar
  30. Jianfeng Pan, Guanglu Yan, and Xiaocao Fan. 2017. Digtool: A virtualization-based framework for detecting kernel vulnerabilities. In Proceedings of the 26th USENIX Security Symposium (USENIX Security’17), Engin Kirda and Thomas Ristenpart (Eds.). USENIX Association, 149--165. Retrieved from https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/pan.Google ScholarGoogle Scholar
  31. Henning Perl, Sergej Dechand, Matthew Smith, Daniel Arp, Fabian Yamaguchi, Konrad Rieck, Sascha Fahl, and Yasemin Acar. 2015. VCCFinder: Finding potential vulnerabilities in open-source projects to assist code audits. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, Indrajit Ray, Ninghui Li, and Christopher Kruegel (Eds.). ACM, 426--437. DOI:https://doi.org/10.1145/2810103.2813604Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Eric Rescorla. 2005. Is finding security holes a good idea? IEEE Secur. Priv. 3, 1 (2005), 14--19. DOI:https://doi.org/10.1109/MSP.2005.17Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Yaman Roumani, Joseph K. Nwankpa, and Yazan F. Roumani. 2015. Time series modeling of vulnerabilities. Comput. Secur. 51 (2015), 32--40. DOI:https://doi.org/10.1016/j.cose.2015.03.003Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Sergej Schumilo, Cornelius Aschermann, Robert Gawlik, Sebastian Schinzel, and Thorsten Holz. 2017. kAFL: Hardware-assisted feedback fuzzing for OS kernels. In Proceedings of the 26th USENIX Security Symposium (USENIX Security’17), Engin Kirda and Thomas Ristenpart (Eds.). USENIX Association, 167--182. Retrieved from https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/schumilo.Google ScholarGoogle Scholar
  35. Konstantin Serebryany, Derek Bruening, Alexander Potapenko, and Dmitriy Vyukov. 2012. AddressSanitizer: A fast address sanity checker. In Proceedings of the USENIX Annual Technical Conference, Gernot Heiser and Wilson C. Hsieh (Eds.). USENIX Association, 309--318. Retrieved from https://www.usenix.org/conference/atc12/technical-sessions/presentation/serebryany.Google ScholarGoogle Scholar
  36. Muhammad Shahzad, Muhammad Zubair Shafiq, and Alex X. Liu. 2012. A large scale exploratory analysis of software vulnerability life cycles. In Proceedings of the 34th International Conference on Software Engineering (ICSE’12), Martin Glinz, Gail C. Murphy, and Mauro Pezzè (Eds.). IEEE Computer Society, 771--781. DOI:https://doi.org/10.1109/ICSE.2012.6227141Google ScholarGoogle Scholar
  37. Richard Stallman et al. 1991. Gnu General Public License. Technical Report. Free Software Foundation, Inc.Google ScholarGoogle Scholar
  38. Nick Stephens, John Grosen, Christopher Salls, Andrew Dutcher, Ruoyu Wang, Jacopo Corbetta, Yan Shoshitaishvili, Christopher Kruegel, and Giovanni Vigna. 2016. Driller: Augmenting fuzzing through selective symbolic execution. In Proceedings of the 23rd Annual Network and Distributed System Security Symposium (NDSS’16). The Internet Society. Retrieved from http://wp.internetsociety.org/ndss/wp-content/uploads/sites/25/2017/09/driller-augmenting-fuzzing-through-selective-symbolic-execution.pdf.Google ScholarGoogle ScholarCross RefCross Ref
  39. Lin Tan, Chen Liu, Zhenmin Li, Xuanhui Wang, Yuanyuan Zhou, and ChengXiang Zhai. 2014. Bug characteristics in open source software. Empir. Softw. Eng. 19, 6 (2014), 1665--1705. DOI:https://doi.org/10.1007/s10664-013-9258-8Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Anjo Vahldiek-Oberwagner, Eslam Elnikety, Nuno O. Duarte, Michael Sammler, Peter Druschel, and Deepak Garg. 2019. ERIM: Secure, efficient in-process isolation with protection keys (MPK). In Proceedings of the 28th USENIX Security Symposium (USENIX Security’19), Nadia Heninger and Patrick Traynor (Eds.). USENIX Association, 1221--1238. Retrieved from https://www.usenix.org/conference/usenixsecurity19/presentation/vahldiek-oberwagner.Google ScholarGoogle Scholar
  41. Cynthia Wagner, Alexandre Dulaunoy, Gérard Wagener, and Andras Iklody. 2016. MISP: The design and implementation of a collaborative threat intelligence sharing platform. In Proceedings of the ACM Workshop on Information Sharing and Collaborative Security (WISCS’16), Stefan Katzenbeisser, Edgar R. Weippl, Erik-Oliver Blass, and Florian Kerschbaum (Eds.). ACM, 49--56. Retrieved from http://dl.acm.org/citation.cfm?id=2994542.Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Walter Willinger, Vern Paxson, and Murad S. Taqqu. 1998. Self-similarity and heavy tails: Structural modeling of network traffic. In A Practical Guide to Heavy Tails: Statistical Techniques and Applications, Vol. 23. Birkhäuser, 27--53.Google ScholarGoogle Scholar
  43. Chaowei Xiao, Armin Sarabi, Yang Liu, Bo Li, Mingyan Liu, and Tudor Dumitras. 2018. From patching delays to infection symptoms: Using risk profiles for an early discovery of vulnerabilities exploited in the wild. In Proceedings of the 27th USENIX Security Symposium (USENIX Security’18). 903--918. Retrieved from https://www.usenix.org/conference/usenixsecurity18/presentation/xiao.Google ScholarGoogle Scholar
  44. Fabian Yamaguchi, Markus Lottmann, and Konrad Rieck. 2012. Generalized vulnerability extrapolation using abstract syntax trees. In Proceedings of the 28th Annual Computer Security Applications Conference (ACSAC’12), Robert H’obbes’ Zakon (Ed.). ACM, 359--368. DOI:https://doi.org/10.1145/2420950.2421003Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Mingyi Zhao, Jens Grossklags, and Peng Liu. 2015. An empirical study of web vulnerability discovery ecosystems. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, Indrajit Ray, Ninghui Li, and Christopher Kruegel (Eds.). ACM, 1105--1117. DOI:https://doi.org/10.1145/2810103.2813704Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. The Tip of the Iceberg: On the Merits of Finding Security Bugs

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Privacy and Security
        ACM Transactions on Privacy and Security  Volume 24, Issue 1
        February 2021
        191 pages
        ISSN:2471-2566
        EISSN:2471-2574
        DOI:10.1145/3426975
        Issue’s Table of Contents

        Copyright © 2020 ACM

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 28 September 2020
        • Accepted: 1 June 2020
        • Revised: 1 December 2019
        • Received: 1 June 2019
        Published in tops Volume 24, Issue 1

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Research
        • Refereed

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!