Abstract
In this article, we investigate a fundamental question regarding software security: Is the security of SW releases increasing over time? We approach this question with a detailed analysis of the large body of open-source software packaged in the popular Debian GNU/Linux distribution. Contrary to common intuition, we find no clear evidence that the vulnerability rate of widely used software decreases over time: Even in popular and “stable” releases, the fixing of bugs does not seem to reduce the rate of newly identified vulnerabilities. The intuitive conclusion is worrisome: Commonly employed development and validation procedures do not seem to scale with the increase of features and complexity—they are only chopping pieces off the top of an iceberg of vulnerabilities.
To the best of our knowledge, this is the first investigation into the problem that studies a complete distribution of software, spanning multiple versions. Although we can not give a definitive answer, we show that several popular beliefs also cannot be confirmed given our dataset. We publish our Debian Vulnerability Analysis Framework (DVAF), an automated dataset creation and analysis process, to enable reproduction and further analysis of our results. Overall, we hope our contributions provide important insights into the vulnerability discovery process and help in identifying effective techniques for vulnerability analysis and prevention.
- Debian Project. 2016. Debian security FAQ. Retrieved from https://www.debian.org/security/faq.Google Scholar
- O. H. Alhazmi and Y. K. Malaiya. 2005. Quantitative vulnerability assessment of systems software. In Proceedings of the Annual Reliability and Maintainability Symposium. IEEE, 615--620.Google Scholar
- Luca Allodi. 2017. Economic factors of vulnerability trade and exploitation. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security (CCS’17), Bhavani M. Thuraisingham, David Evans, Tal Malkin, and Dongyan Xu (Eds.). ACM, 1483--1499. DOI:https://doi.org/10.1145/3133956.3133960Google Scholar
Digital Library
- Jeff Alstott, Ed Bullmore, and Dietmar Plenz. 2014. Powerlaw: A Python package for analysis of heavy-tailed distributions. PloS One 9, 1 (2014), e85777.Google Scholar
Cross Ref
- Juan José Amor, Gregorio Robles, Jesus M. González-Barahona, and Francisco Rivas. 2009. Measuring Lenny: The size of Debian 5.0. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.464.51088rep=rep18type=pdf.Google Scholar
- Jason Bau, Elie Bursztein, Divij Gupta, and John C. Mitchell. 2010. State of the art: Automated black-box web application vulnerability testing. In Proceedings of the 31st IEEE Symposium on Security and Privacy (S8P’10). IEEE Computer Society, 332--345. DOI:https://doi.org/10.1109/SP.2010.27Google Scholar
- Leyla Bilge and Tudor Dumitras. 2012. Before we knew it: An empirical study of zero-day attacks in the real world. In Proceedings of the ACM Conference on Computer and Communications Security (CCS’12), Ting Yu, George Danezis, and Virgil D. Gligor (Eds.). ACM, 833--844. DOI:https://doi.org/10.1145/2382196.2382284Google Scholar
Digital Library
- Priyam Biswas, Alessandro Di Federico, Scott A. Carr, Prabhu Rajasekaran, Stijn Volckaert, Yeoul Na, Michael Franz, and Mathias Payer. 2017. Venerable variadic vulnerabilities vanquished. In Proceedings of the 26th USENIX Security Symposium (USENIX Security’17), Engin Kirda and Thomas Ristenpart (Eds.). USENIX Association, 186--198. Retrieved from https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/biswas.Google Scholar
- Matteo Bortolozzo, Matteo Centenaro, Riccardo Focardi, and Graham Steel. 2010. Attacking and fixing PKCS#11 security tokens. In Proceedings of the 17th ACM Conference on Computer and Communications Security (CCS’10), Ehab Al-Shaer, Angelos D. Keromytis, and Vitaly Shmatikov (Eds.). ACM, 260--269. DOI:https://doi.org/10.1145/1866307.1866337Google Scholar
Digital Library
- Fraser Brown, Shravan Narayan, Riad S. Wahby, Dawson R. Engler, Ranjit Jhala, and Deian Stefan. 2017. Finding and preventing bugs in JavaScript bindings. In Proceedings of the IEEE Symposium on Security and Privacy (SP’17). IEEE Computer Society, 559--578. DOI:https://doi.org/10.1109/SP.2017.68Google Scholar
Cross Ref
- Sven Bugiel, Lucas Vincenzo Davi, and Steffen Schulz. 2011. Scalable trust establishment with software reputation. In Proceedings of the 6th ACM Workshop on Scalable Trusted Computing ([email protected]’11), Yan Chen, Shouhuai Xu, Ahmad-Reza Sadeghi, and Xinwen Zhang (Eds.). ACM, 15--24. DOI:https://doi.org/10.1145/2046582.2046587Google Scholar
Digital Library
- Sandy Clark, Stefan Frei, Matt Blaze, and Jonathan M. Smith. 2010. Familiarity breeds contempt: The honeymoon effect and the role of legacy code in zero-day vulnerabilities. In Proceedings of the 26th Annual Computer Security Applications Conference (ACSAC’10), Carrie Gates, Michael Franz, and John P. McDermott (Eds.). ACM, 251--260. DOI:https://doi.org/10.1145/1920261.1920299Google Scholar
- Aaron Clauset, Cosma Rohilla Shalizi, and Mark E. J. Newman. 2009. Power-law distributions in empirical data. SIAM Rev. 51, 4 (2009), 661--703. DOI:https://doi.org/10.1137/070710111Google Scholar
Digital Library
- CVE Details. 2020. Browse Vulnerabilities by Date. Retrieved from https://www.cvedetails.com/browse-by-date.php.Google Scholar
- Nigel Edwards and Liqun Chen. 2012. An historical examination of open source releases and their vulnerabilities. In Proceedings of the ACM Conference on Computer and Communications Security (CCS’12), Ting Yu, George Danezis, and Virgil D. Gligor (Eds.). ACM, 183--194. DOI:https://doi.org/10.1145/2382196.2382218Google Scholar
Digital Library
- Jim Finkle and Supriya Kurane. 2014. U.S. hospital breach biggest yet to exploit Heartbleed bug: Expert. Reuters (2014). Retrieved from https://www.reuters.com/article/us-community-health-cybersecurity/u-s-hospital-breach-biggest-yet-to-exploit-heartbleed-bug-expert-idUSKBN0GK0H420140820.Google Scholar
- Andy Greenberg. 2014. Hackers are already using the Shellshock bug to launch botnet attacks. Wired (2014). Retrieved from https://www.wired.com/2014/09/hackers-already-using-shellshock-bug-create-botnets-ddos-attacks/.Google Scholar
- HackerOne. 2017. The Hacker-powered security report 2017. Retrieved from https://www.hackerone.com/resources/hacker-powered-security-report.Google Scholar
- Munawar Hafiz and Ming Fang. 2016. Game of detections: How are security vulnerabilities discovered in the wild?Empir. Softw. Eng. 21, 5 (2016), 1920--1959. DOI:https://doi.org/10.1007/s10664-015-9403-7Google Scholar
Digital Library
- Jinyoo Kim, Yashwant K. Malaiya, and Indrakshi Ray. 2007. Vulnerability discovery in multi-version software systems. In Proceedings of the 10th IEEE International Symposium on High Assurance Systems Engineering (HASE’07). IEEE Computer Society, 141--148. DOI:https://doi.org/10.1109/HASE.2007.55Google Scholar
Digital Library
- Meir M. Lehman. 1980. Programs, life cycles, and laws of software evolution. Proc. IEEE 68, 9 (1980), 1060--1076.Google Scholar
Cross Ref
- Meir M. Lehman, Juan F. Ramil, Paul Wernick, Dewayne E. Perry, and Wladyslaw M. Turski. 1997. Metrics and laws of software evolution—The nineties view. In Proceedings of the 4th IEEE International Software Metrics Symposium (METRICS’97). IEEE Computer Society, 20. DOI:https://doi.org/10.1109/METRIC.1997.637156Google Scholar
Cross Ref
- Nancy G. Leveson. 2009. Software challenges in achieving space safety. J. Brit. Interplan. Soc. 62 (2009), 265–272.Google Scholar
- Frank Li and Vern Paxson. 2017. A large-scale empirical study of security patches. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security (CCS’17), Bhavani M. Thuraisingham, David Evans, Tal Malkin, and Dongyan Xu (Eds.). ACM, 2201--2215. DOI:https://doi.org/10.1145/3133956.3134072Google Scholar
Digital Library
- Thomas Maillart, Mingyi Zhao, Jens Grossklags, and John Chuang. 2017. Given enough eyeballs, all bugs are shallow? Revisiting Eric Raymond with bug bounty programs. J. Cybersecur. 3, 2 (2017), 81--90. DOI:https://doi.org/10.1093/cybsec/tyx008Google Scholar
Cross Ref
- Antonio Nappa, Richard Johnson, Leyla Bilge, Juan Caballero, and Tudor Dumitras. 2015. The attack of the clones: A study of the impact of shared code on vulnerability patching. In Proceedings of the IEEE Symposium on Security and Privacy (SP’15). 692--708. DOI:https://doi.org/10.1109/SP.2015.48Google Scholar
Digital Library
- Matús Nemec, Dusan Klinec, Petr Svenda, Peter Sekan, and Vashek Matyas. 2017. Measuring popularity of cryptographic libraries in internet-wide scans. In Proceedings of the 33rd Annual Computer Security Applications Conference. ACM, 162--175. DOI:https://doi.org/10.1145/3134600.3134612Google Scholar
Digital Library
- Andy Ozment and Stuart E. Schechter. 2006. Milk or wine: Does software security improve with age? In Proceedings of the 15th USENIX Security Symposium, Angelos D. Keromytis (Ed.). USENIX Association. https://www.usenix.org/conference/15th-usenix-security-symposium/milk-or-wine-does-software-security-improve-age.Google Scholar
- James Andrew Ozment. 2007. Vulnerability Discovery 8 Software Security. Ph.D. Dissertation. University of Cambridge, UK. Retrieved from http://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.613340.Google Scholar
- Jianfeng Pan, Guanglu Yan, and Xiaocao Fan. 2017. Digtool: A virtualization-based framework for detecting kernel vulnerabilities. In Proceedings of the 26th USENIX Security Symposium (USENIX Security’17), Engin Kirda and Thomas Ristenpart (Eds.). USENIX Association, 149--165. Retrieved from https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/pan.Google Scholar
- Henning Perl, Sergej Dechand, Matthew Smith, Daniel Arp, Fabian Yamaguchi, Konrad Rieck, Sascha Fahl, and Yasemin Acar. 2015. VCCFinder: Finding potential vulnerabilities in open-source projects to assist code audits. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, Indrajit Ray, Ninghui Li, and Christopher Kruegel (Eds.). ACM, 426--437. DOI:https://doi.org/10.1145/2810103.2813604Google Scholar
Digital Library
- Eric Rescorla. 2005. Is finding security holes a good idea? IEEE Secur. Priv. 3, 1 (2005), 14--19. DOI:https://doi.org/10.1109/MSP.2005.17Google Scholar
Digital Library
- Yaman Roumani, Joseph K. Nwankpa, and Yazan F. Roumani. 2015. Time series modeling of vulnerabilities. Comput. Secur. 51 (2015), 32--40. DOI:https://doi.org/10.1016/j.cose.2015.03.003Google Scholar
Digital Library
- Sergej Schumilo, Cornelius Aschermann, Robert Gawlik, Sebastian Schinzel, and Thorsten Holz. 2017. kAFL: Hardware-assisted feedback fuzzing for OS kernels. In Proceedings of the 26th USENIX Security Symposium (USENIX Security’17), Engin Kirda and Thomas Ristenpart (Eds.). USENIX Association, 167--182. Retrieved from https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/schumilo.Google Scholar
- Konstantin Serebryany, Derek Bruening, Alexander Potapenko, and Dmitriy Vyukov. 2012. AddressSanitizer: A fast address sanity checker. In Proceedings of the USENIX Annual Technical Conference, Gernot Heiser and Wilson C. Hsieh (Eds.). USENIX Association, 309--318. Retrieved from https://www.usenix.org/conference/atc12/technical-sessions/presentation/serebryany.Google Scholar
- Muhammad Shahzad, Muhammad Zubair Shafiq, and Alex X. Liu. 2012. A large scale exploratory analysis of software vulnerability life cycles. In Proceedings of the 34th International Conference on Software Engineering (ICSE’12), Martin Glinz, Gail C. Murphy, and Mauro Pezzè (Eds.). IEEE Computer Society, 771--781. DOI:https://doi.org/10.1109/ICSE.2012.6227141Google Scholar
- Richard Stallman et al. 1991. Gnu General Public License. Technical Report. Free Software Foundation, Inc.Google Scholar
- Nick Stephens, John Grosen, Christopher Salls, Andrew Dutcher, Ruoyu Wang, Jacopo Corbetta, Yan Shoshitaishvili, Christopher Kruegel, and Giovanni Vigna. 2016. Driller: Augmenting fuzzing through selective symbolic execution. In Proceedings of the 23rd Annual Network and Distributed System Security Symposium (NDSS’16). The Internet Society. Retrieved from http://wp.internetsociety.org/ndss/wp-content/uploads/sites/25/2017/09/driller-augmenting-fuzzing-through-selective-symbolic-execution.pdf.Google Scholar
Cross Ref
- Lin Tan, Chen Liu, Zhenmin Li, Xuanhui Wang, Yuanyuan Zhou, and ChengXiang Zhai. 2014. Bug characteristics in open source software. Empir. Softw. Eng. 19, 6 (2014), 1665--1705. DOI:https://doi.org/10.1007/s10664-013-9258-8Google Scholar
Digital Library
- Anjo Vahldiek-Oberwagner, Eslam Elnikety, Nuno O. Duarte, Michael Sammler, Peter Druschel, and Deepak Garg. 2019. ERIM: Secure, efficient in-process isolation with protection keys (MPK). In Proceedings of the 28th USENIX Security Symposium (USENIX Security’19), Nadia Heninger and Patrick Traynor (Eds.). USENIX Association, 1221--1238. Retrieved from https://www.usenix.org/conference/usenixsecurity19/presentation/vahldiek-oberwagner.Google Scholar
- Cynthia Wagner, Alexandre Dulaunoy, Gérard Wagener, and Andras Iklody. 2016. MISP: The design and implementation of a collaborative threat intelligence sharing platform. In Proceedings of the ACM Workshop on Information Sharing and Collaborative Security (WISCS’16), Stefan Katzenbeisser, Edgar R. Weippl, Erik-Oliver Blass, and Florian Kerschbaum (Eds.). ACM, 49--56. Retrieved from http://dl.acm.org/citation.cfm?id=2994542.Google Scholar
Digital Library
- Walter Willinger, Vern Paxson, and Murad S. Taqqu. 1998. Self-similarity and heavy tails: Structural modeling of network traffic. In A Practical Guide to Heavy Tails: Statistical Techniques and Applications, Vol. 23. Birkhäuser, 27--53.Google Scholar
- Chaowei Xiao, Armin Sarabi, Yang Liu, Bo Li, Mingyan Liu, and Tudor Dumitras. 2018. From patching delays to infection symptoms: Using risk profiles for an early discovery of vulnerabilities exploited in the wild. In Proceedings of the 27th USENIX Security Symposium (USENIX Security’18). 903--918. Retrieved from https://www.usenix.org/conference/usenixsecurity18/presentation/xiao.Google Scholar
- Fabian Yamaguchi, Markus Lottmann, and Konrad Rieck. 2012. Generalized vulnerability extrapolation using abstract syntax trees. In Proceedings of the 28th Annual Computer Security Applications Conference (ACSAC’12), Robert H’obbes’ Zakon (Ed.). ACM, 359--368. DOI:https://doi.org/10.1145/2420950.2421003Google Scholar
Digital Library
- Mingyi Zhao, Jens Grossklags, and Peng Liu. 2015. An empirical study of web vulnerability discovery ecosystems. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, Indrajit Ray, Ninghui Li, and Christopher Kruegel (Eds.). ACM, 1105--1117. DOI:https://doi.org/10.1145/2810103.2813704Google Scholar
Digital Library
Index Terms
The Tip of the Iceberg: On the Merits of Finding Security Bugs
Recommendations
Adapting Linux for mobile platforms: An empirical study of Android
ICSM '12: Proceedings of the 2012 IEEE International Conference on Software Maintenance (ICSM)To deliver a high quality software system in a short release cycle time, many software organizations chose to reuse existing mature software systems. Google has adapted one of the most reused computer operating systems (i.e., Linux) into an operating ...
Empirical Study of Tolerating Denial-of-Service Attacks with the Fosel Architecture
NCA '09: Proceedings of the 2009 Eighth IEEE International Symposium on Network Computing and ApplicationsFiltering techniques are one of the main approaches to protect applications from Denial of Service Attacks (DoS).However filtering techniques suffer from two main challenges:a) the accuracy detection of DoS traffic and b) processing time.Fosel (...
Discovering buffer overflow vulnerabilities in the wild: an empirical study
ESEM '14: Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and MeasurementContext: Reporters of security vulnerabilities possess rich information about the security engineering process. Goal: We performed an empirical study on reporters of buffer overflow vulnerabilities to understand the methods and tools used during the ...






Comments