Abstract
Developers frequently use inefficient code sequences that could be fixed by simple patches. These inefficient code sequences can cause significant performance degradation and resource waste, referred to as performance bugs. Meager increases in single threaded performance in the multi-core era and increasing emphasis on energy efficiency call for more effort in tackling performance bugs.
This paper conducts a comprehensive study of 110 real-world performance bugs that are randomly sampled from five representative software suites (Apache, Chrome, GCC, Mozilla, and MySQL). The findings of this study provide guidance for future work to avoid, expose, detect, and fix performance bugs.
Guided by our characteristics study, efficiency rules are extracted from 25 patches and are used to detect performance bugs. 332 previously unknown performance problems are found in the latest versions of MySQL, Apache, and Mozilla applications, including 219 performance problems found by applying rules across applications.
- M. K. Aguilera, J. C. Mogul, J. L. Wiener, P. Reynolds, and A. Muthitacharoen. Performance debugging for distributed systems of black boxes. In SOSP, 2003. Google Scholar
Digital Library
- M. Attariyan and J. Flinn. Automating configuration troubleshooting with dynamic information flow analysis. In OSDI, 2010. Google Scholar
Digital Library
- W. Baek and T. M. Chilimbi. Green: a framework for supporting energy-conscious programming using controlled approximation. In PLDI, 2010. Google Scholar
Digital Library
- C. Cadar, D. Dunbar, and D. Engler. Klee: Unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI, 2008. Google Scholar
Digital Library
- V. Chipounov, V. Kuznetsov, and G. Candea. S2E: a platform for in-vivo multi-path analysis of software systems. In ASPLOS, 2011. Google Scholar
Digital Library
- A. Chou, B. Chelf, D. Engler, and M. Heinrich. Using meta-level compilation to check FLASH protocol code. In ASPLOS, 2000. Google Scholar
Digital Library
- A. Chou, J. Yang, B. Chelf, S. Hallem, and D. R. Engler. An empirical study of operating system errors. In SOSP, 2001. Google Scholar
Digital Library
- I. Cohen, M. Goldszmidt, T. Kelly, J. Symons, and J. S. Chase. Correlating instrumentation data to system states: a building block for automated diagnosis and control. In OSDI, 2004. Google Scholar
Digital Library
- M. Corporation. Performance Testing Guidance for Web Applications. Microsoft Press, 2007. Google Scholar
Digital Library
- A. Diwan, M. Hauswirth, T. Mytkowicz, and P. F. Sweeney. Traceanalyzer: a system for processing performance traces. Softw. Pract. Exper., March 2011. Google Scholar
Digital Library
- B. Dufour, B. G. Ryder, and G. Sevitsky. A scalable technique for characterizing the usage of temporaries in framework-intensive java applications. In FSE, 2008. Google Scholar
Digital Library
- R. F. Dugan. Performance lies my professor told me: the case for teaching software performance engineering to undergraduates. In WOSP, 2004. Google Scholar
Digital Library
- D. Engler, D. Y. Chen, S. Hallem, A. Chou, and B. Chelf. Bugs as deviant behavior: A general approach to inferring errors in systems code. In SOSP, 2001. Google Scholar
Digital Library
- M. Ernst, A. Czeisler, W. G. Griswold, and D. Notkin. Quickly detecting relevant program invariants. In ICSE, 2000. Google Scholar
Digital Library
- Fortify. HP Fortify Static Code Analyzer (SCA). https://www.fortify.com/products/hpfssc/source-code-analyzer.html.Google Scholar
- M. Gabel, J. Yang, Y. Yu, M. Goldszmidt, and Z. Su. Scalable and systematic detection of buggy inconsistencies in source code. In OOPSLA, 2010. Google Scholar
Digital Library
- P. Godefroid, N. Klarlund, and K. Sen. Dart: directed automated random testing. In PLDI, 2005. Google Scholar
Digital Library
- S. Gulwani, K. K. Mehra, and T. Chilimbi. Speed: precise and efficient static estimation of program computational complexity. In POPL, 2009. Google Scholar
Digital Library
- M. Hauswirth, A. Diwan, P. F. Sweeney, and M. C. Mozer. Automating vertical profiling. In OOPSLA, 2005. Google Scholar
Digital Library
- J. K. Hollingsworth, R. B. Irvin, and B. P. Miller. The integration of application and system based metrics in a parallel program performance tool. In PPOPP, 1991. Google Scholar
Digital Library
- D. Hovemeyer and W. Pugh. Finding bugs is easy. In OOPSLA, 2004. Google Scholar
Digital Library
- InfoWorld. Top 10 open source hall of famers. http://www.infoworld.com/d/open-source/top-10-open-source-hall-famers-848.Google Scholar
- R. B. Irvin and B. P. Miller. Mapping performance data for high-level and data views of parallel program performance. In ICS, 1996. Google Scholar
Digital Library
- JMeter. Java desktop application designed for load tests. http://jakarta.apache.org/jmeter.Google Scholar
- M. Jovic, A. Adamoli, and M. Hauswirth. Catch me if you can: performance bug detection in the wild. In OOPSLA, 2011. Google Scholar
Digital Library
- C. Killian, K. Nagaraj, S. Pervez, R. Braud, J. W. Anderson, and R. Jhala. Finding latent performance bugs in systems implementations. In FSE, 2010. Google Scholar
Digital Library
- C. Lattner and V. Adve. LLVM: A compilation framework for lifelong program analysis & transformation. In CGO, 2004. Google Scholar
Digital Library
- Z. Li, S. Lu, S. Myagmar, and Y. Zhou. CP-Miner: A Tool for Finding Copy-paste and Related Bugs in Operating System Code. In OSDI, 2004. Google Scholar
Digital Library
- Z. Li, L. Tan, X. Wang, Y. Zhou, and C. Zhai. Have things changed now?: an empirical study of bug characteristics in modern open source software. In ASID, 2006. Google Scholar
Digital Library
- Z. Li and Y. Zhou. PR-Miner: Automatically extracting implicit programming rules and detecting violations in large software code. In FSE, Sept 2005. Google Scholar
Digital Library
- S. Liu, K. Pattabiraman, T. Moscibroda, and B. G. Zorn. Flikker: saving DRAM refresh-power through critical data partitioning. In ASPLOS, 2011. Google Scholar
Digital Library
- T. Liu and E. D. Berger. Precise detection and automatic mitigation of false sharing. In OOPSLA, 2011. Google Scholar
Digital Library
- V. B. Livshits and T. Zimmermann. Dynamine: Finding common error patterns by mining software revision histories. In FSE, 2005. Google Scholar
Digital Library
- S. Lu, S. Park, E. Seo, and Y. Zhou. Learning from mistakes -- a comprehensive study of real world concurrency bug characteristics. In ASPLOS, 2008. Google Scholar
Digital Library
- J. Mellor-Crummey, R. J. Fowler, G. Marin, and N. Tallent. Hpcview: A tool for top-down analysis of node performance. J. Supercomput., 23(1):81--104, 2002. Google Scholar
Digital Library
- N. Meng, M. Kim, and K. S. McKinley. Systematic editing: generating program transformations from an example. In PLDI, 2011. Google Scholar
Digital Library
- Microsoft. MSDN SAL annotations. http://msdn2.microsoft.com/en-us/library/ms235402.aspx.Google Scholar
- B. P. Miller, M. D. Callaghan, J. M. Cargille, J. K. Hollingsworth, R. B. Irvin, K. L. Karavanic, K. Kunchithapadam, and T. Newhall. The paradyn parallel performance measurement tool. Computer, 28(11), 1995. Google Scholar
Digital Library
- N. Mitchell. The diary of a datum: an approach to modeling runtime complexity in framework-based applications. In ECOOP, 2006. Google Scholar
Digital Library
- I. Molyneaux. The Art of Application Performance Testing: Help for Programmers and Quality Assurance. O'Reilly Media, 2009. Google Scholar
Digital Library
- G. E. Morris. Lessons from the colorado benefits management system disaster. www.ad-mkt-review.com/publichtml/air/ai200411.html, 2004.Google Scholar
- G. Muller, Y. Padioleau, J. L. Lawall, and R. R. Hansen. Semantic patches considered helpful. Operating Systems Review, 40(3):90--92, 2006. Google Scholar
Digital Library
- N. Palix, G. Thomas, S. Saha, C. Calvès, J. Lawall, and G. Muller. Faults in linux: ten years later. In ASPLOS, 2011. Google Scholar
Digital Library
- S. E. Perl and W. E. Weihl. Performance assertion checking. In SOSP, 1993. Google Scholar
Digital Library
- T. Richardson. 1901 census site still down after six months. http://www.theregister.co.uk/2002/07/03/1901_census_site_still_down/.Google Scholar
- C. J. Rossbach, O. S. Hofmann, and E. Witchel. Is transactional programming actually easier? In PPoPP, 2010. Google Scholar
Digital Library
- A. Shankar, M. Arnold, and R. Bodik. Jolt: lightweight dynamic analysis and removal of object churn. In OOPSLA, 2008. Google Scholar
Digital Library
- K. Shen, C. Stewart, C. Li, and X. Li. Reference-driven performance anomaly identification. In SIGMETRICS, 2009. Google Scholar
Digital Library
- R. L. Sites. Identifying dark latency.Google Scholar
- C. U. Smith and L. G. Williams. Software performance antipatterns. In Proceedings of the 2nd international workshop on Software and performance, 2000. Google Scholar
Digital Library
- Stefan Bodewig. Bug 45396: There is no hint in the javadocs. https://issues.apache.org/ bugzilla/show_bug.cgi?id=45396#c4.Google Scholar
- C. Stewart, K. Shen, A. Iyengar, and J. Yin. Entomomodel: Understanding and avoiding performance anomaly manifestations. In MASCOTS, 2010. Google Scholar
Digital Library
- M. Sullivan and R. Chillarege. A comparison of software defects in database management systems and operating systems. In FTCS, 1992.Google Scholar
Cross Ref
- L. Torvalds. Sparse - a semantic parser for c. http://www.kernel.org/pub/software/devel/sparse/.Google Scholar
- J. S. Vetter and P. H. Worley. Asserting performance expectations. In Supercomputing, 2002. Google Scholar
Digital Library
- wikipedia. Chi-squared test. http://en.wikipedia.org/wiki/Chi-squared_test.Google Scholar
- G. Xu, M. Arnold, N. Mitchell, A. Rountev, and G. Sevitsky. Go with the flow: profiling copies to find runtime bloat. In PLDI '09, 2009. Google Scholar
Digital Library
- G. Xu, N. Mitchell, M. Arnold, A. Rountev, E. Schonberg, and G. Sevitsky. Finding low-utility data structures. In PLDI, 2010. Google Scholar
Digital Library
- W. Xu, L. Huang, A. Fox, D. Patterson, and M. I. Jordan. Detecting large-scale system problems by mining console logs. In SOSP, 2009. Google Scholar
Digital Library
- A. Zeller. Isolating cause-effect chains from computer programs. In FSE, 2002. Google Scholar
Digital Library
Index Terms
Understanding and detecting real-world performance bugs
Recommendations
Understanding and detecting real-world performance bugs
PLDI '12: Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and ImplementationDevelopers frequently use inefficient code sequences that could be fixed by simple patches. These inefficient code sequences can cause significant performance degradation and resource waste, referred to as performance bugs. Meager increases in single ...
A qualitative study on performance bugs
MSR '12: Proceedings of the 9th IEEE Working Conference on Mining Software RepositoriesSoftware performance is one of the important qualities that makes software stand out in a competitive market. However, in earlier work we found that performance bugs take more time to fix, need to be fixed by more experienced developers and require ...
Security versus performance bugs: a case study on Firefox
MSR '11: Proceedings of the 8th Working Conference on Mining Software RepositoriesA good understanding of the impact of different types of bugs on various project aspects is essential to improve software quality research and practice. For instance, we would expect that security bugs are fixed faster than other types of bugs due to ...







Comments