Abstract
Design and implementation defects that lead to inefficient computation widely exist in software. These defects are difficult to avoid and discover. They lead to severe performance degradation and energy waste during production runs, and are becoming increasingly critical with the meager increase of single-core hardware performance and the increasing concerns about energy constraints. Effective tools that diagnose performance problems and point out the inefficiency root cause are sorely needed.
The state of the art of performance diagnosis is preliminary. Profiling can identify the functions that consume the most computation resources, but can neither identify the ones that waste the most resources nor explain why. Performance-bug detectors can identify specific type of inefficient computation, but are not suited for diagnosing general performance problems. Effective failure diagnosis techniques, such as statistical debugging, have been proposed for functional bugs. However, whether they work for performance problems is still an open question.
In this paper, we first conduct an empirical study to understand how performance problems are observed and reported by real-world users. Our study shows that statistical debugging is a natural fit for diagnosing performance problems, which are often observed through comparison-based approaches and reported together with both good and bad inputs. We then thoroughly investigate different design points in statistical debugging, including three different predicates and two different types of statistical models, to understand which design point works the best for performance diagnosis. Finally, we study how some unique nature of performance bugs allows sampling techniques to lower the overhead of run-time performance diagnosis without extending the diagnosis latency.
- http://sourceware.org/binutils/docs/gprof/.Google Scholar
- M. K. Aguilera, J. C. Mogul, J. L. Wiener, P. Reynolds, and A. Muthitacharoen. Performance debugging for distributed systems of black boxes. In SOSP, 2003. Google Scholar
Digital Library
- E. Altman, M. Arnold, S. Fink, and N. Mitchell. Performance analysis of idle programs. In Proceedings of the ACM international conference on Object oriented programming systems languages and applications, OOPSLA '10, 2010. Google Scholar
Digital Library
- E. Alves, M. Gligoric, V. Jagannath, and M. d'Amorim. Fault-localization using dynamic slicing and change impact analysis. In ASE, 2011. Google Scholar
Digital Library
- D. Andrzejewski, A. Mulhern, B. Liblit, and X. Zhu. Statistical debugging using latent topic models. In Proceedings of the 18th European conference on Machine Learning, 2007. Google Scholar
Digital Library
- J. Arulraj, P.-C. Chang, G. Jin, and S. Lu. Production-run software failure diagnosis via hardware performance counters. In ASPLOS, 2013. Google Scholar
Digital Library
- M. Attariyan, M. Chow, and J. Flinn. X-ray: automating root-cause diagnosis of performance anomalies in production software. In OSDI, 2012. Google Scholar
Digital Library
- D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet allocation. J. Mach. Learn. Res., 3:993--1022, Mar. 2003. ISSN 1532-4435. Google Scholar
Digital Library
- T.-H. Chen, W. Shang, Z. M. Jiang, A. E. Hassan, M. Nasser, and P. Flora. Detecting performance anti-patterns for applications developed using object-relational mapping. In ICSE, 2014. Google Scholar
Digital Library
- V. Chipounov, V. Kuznetsov, and G. Candea. S2E: a platform for in-vivo multi-path analysis of software systems. In ASPLOS, 2011. Google Scholar
Digital Library
- A. Diwan, M. Hauswirth, T. Mytkowicz, and P. F. Sweeney. Traceanalyzer: a system for processing performance traces. Softw., Pract. Exper., 41(3):267--282, 2011. Google Scholar
Digital Library
- B. Dufour, B. G. Ryder, and G. Sevitsky. A scalable technique for characterizing the usage of temporaries in framework-intensive java applications. In FSE, 2008. Google Scholar
Digital Library
- D. Engler, D. Y. Chen, S. Hallem, A. Chou, and B. Chelf. Bugs as deviant behavior: A general approach to inferring errors in systems code. In SOSP, pages 57--72, 2001. Google Scholar
Digital Library
- R. Fonseca, M. J. Freedman, and G. Porter. Experiences with tracing causality in networked services. In Internet network management conference on Research on enterprise networking, 2010. Google Scholar
Digital Library
- K. Glerum, K. Kinshumann, S. Greenberg, G. Aul, V. Orgovan, G. Nichols, D. Grant, G. Loihle, and G. C. Hunt. Debugging in the (very) large: ten years of implementation and experience. In SOSP, 2009. Google Scholar
Digital Library
- N. Gupta, H. He, X. Zhang, and R. Gupta. Locating faulty code using failure-inducing chops. In ASE, 2005. Google Scholar
Digital Library
- S. Han, Y. Dang, S. Ge, D. Zhang, and T. Xie. Performance debugging in the large via mining millions of stack traces. In ICSE, 2012. Google Scholar
Digital Library
- S. Hangal and M. S. Lam. Tracking down software bugs using automatic anomaly detection. In ICSE, 2002. Google Scholar
Digital Library
- S. Horwitz, T. Reps, and D. Binkley. Interprocedural slicing using dependence graphs. ACM Trans. Program. Lang. Syst., 12(1):26--60, Jan. 1990. Google Scholar
Digital Library
- G. Jin, A. Thakur, B. Liblit, and S. Lu. Instrumentation and sampling strategies for cooperative concurrency bug isolation. In OOPSLA, 2010. Google Scholar
Digital Library
- G. Jin, L. Song, X. Shi, J. Scherpelz, and S. Lu. Understanding and detecting real-world performance bugs. In PLDI, 2012. Google Scholar
Digital Library
- J. A. Jones, M. J. Harrold, and J. Stasko. Visualization of test information to assist fault localization. In ICSE, 2002. Google Scholar
Digital Library
- M. P. Kasick, J. Tan, R. Gandhi, and P. Narasimhan. Blackbox problem diagnosis in parallel file systems. In FAST, 2010. Google Scholar
Digital Library
- C. Killian, K. Nagaraj, S. Pervez, R. Braud, J. W. Anderson, and R. Jhala. Finding latent performance bugs in systems implementations. In FSE, 2010. Google Scholar
Digital Library
- C. H. Kim, J. Rhee, H. Zhang, N. Arora, G. Jiang, X. Zhang, and D. Xu. Introperf: Transparent context-sensitive multilayer performance inference using system stack traces. SIGMETRICS, 2014. Google Scholar
Digital Library
- T. Kremenek, P. Twohey, G. Back, A. Ng, and D. Engler. From uncertainty to belief: Inferring the specification within. In OSDI, Nov 2006. Google Scholar
Digital Library
- Z. Li, S. Lu, S. Myagmar, and Y. Zhou. CP-Miner: A Tool for Finding Copy-paste and Related Bugs in Operating System Code. In OSDI, 2004. Google Scholar
Digital Library
- Z. Li, L. Tan, X. Wang, S. Lu, Y. Zhou, and C. Zhai. Have things changed now?: an empirical study of bug characteristics in modern open source software. In ASID, 2006. Google Scholar
Digital Library
- B. Liblit, A. Aiken, A. X. Zheng, and M. I. Jordan. Bug isolation via remote program sampling. In PLDI, 2003. Google Scholar
Digital Library
- B. Liblit, M. Naik, A. X. Zheng, A. Aiken, and M. I. Jordan. Scalable statistical bug isolation. In PLDI, 2005. Google Scholar
Digital Library
- T. Liu and E. D. Berger. Sheriff: precise detection and automatic mitigation of false sharing. In OOPSLA, 2011. Google Scholar
Digital Library
- Y. Liu, C. Xu, and S.-C. Cheung. Characterizing and detecting performance bugs for smartphone applications. In ICSE, 2014. Google Scholar
Digital Library
- S. Lu, S. Park, E. Seo, and Y. Zhou. Learning from mistakes - a comprehensive study of real world concurrency bug characteristics. In ASPLOS, 2008. Google Scholar
Digital Library
- C.-K. Luk, R. Cohn, R. Muth, H. Patil, A. Klauser, G. Lowney, S. Wallace, V. J. Reddi, and K. Hazelwood. Pin: building customized program analysis tools with dynamic instrumentation. In PLDI, 2005. Google Scholar
Digital Library
- I. Molyneaux. The Art of Application Performance Testing: Help for Programmers and Quality Assurance. O'Reilly Media, 2009. Google Scholar
Digital Library
- A. Nistor, T. Jiang, and L. Tan. Discovering, reporting, and fixing performance bugs. In The 10th Working Conference on Mining Software Repositories, 2013. Google Scholar
Digital Library
- A. Nistor, L. Song, D. Marinov, and S. Lu. Toddler: Detecting performance problems via similar memory-access patterns. In ICSE, 2013. Google Scholar
Digital Library
- OProfile. OProfile - A System Profiler for Linux. http://oprofile.sourceforge.net.Google Scholar
- R. R. Sambasivan, A. X. Zheng, M. De Rosa, E. Krevat, S. Whitman, M. Stroucken, W. Wang, L. Xu, and G. R. Ganger. Diagnosing performance changes by comparing request flows. In NSDI, 2011. Google Scholar
Digital Library
- R. Santelices, J. A. Jones, Y. Yu, and M. J. Harrold. Lightweight fault-localization using multiple coverage types. In ICSE, 2009. Google Scholar
Digital Library
- C. U. Smith and L. G. Williams. Software performance antipatterns. In Proceedings of the 2nd international workshop on Software and performance, 2000. Google Scholar
Digital Library
- Wikipedia. Z-test. http://en.wikipedia.org/wiki/Z-test.Google Scholar
- X. Xiao, S. Han, T. Xie, and D. Zhang. Context-sensitive delta inference for identifying workload-dependent performance bottlenecks. In ISSTA, 2013. Google Scholar
Digital Library
- G. Xu and A. Rountev. Detecting inefficiently-used containers to avoid bloat. In PLDI, 2010. Google Scholar
Digital Library
- G. Xu, M. Arnold, N. Mitchell, A. Rountev, and G. Sevitsky. Go with the flow: profiling copies to find runtime bloat. In PLDI, 2009. Google Scholar
Digital Library
- G. Xu, N. Mitchell, M. Arnold, A. Rountev, E. Schonberg, and G. Sevitsky. Finding low-utility data structures. In PLDI, 2010. Google Scholar
Digital Library
- W. Xu, L. Huang, A. Fox, D. Patterson, and M. I. Jordan. Detecting large-scale system problems by mining console logs. In SOSP, 2009. Google Scholar
Digital Library
- X. Yu, S. Han, D. Zhang, and T. Xie. Comprehending performance from real-world execution traces: A device-driver case. In ASPLOS, 2014. Google Scholar
Digital Library
- S. Zaman, B. Adams, and A. E. Hassan. A qualitative study on performance bugs. In The 9th Working Conference on Mining Software Repositories, 2012. Google Scholar
Digital Library
- A. Zeller. Isolating cause-effect chains from computer programs. In FSE, 2002. Google Scholar
Digital Library
Index Terms
Statistical debugging for real-world performance problems
Recommendations
Understanding and detecting real-world performance bugs
PLDI '12Developers frequently use inefficient code sequences that could be fixed by simple patches. These inefficient code sequences can cause significant performance degradation and resource waste, referred to as performance bugs. Meager increases in single ...
Statistical debugging for real-world performance problems
OOPSLA '14: Proceedings of the 2014 ACM International Conference on Object Oriented Programming Systems Languages & ApplicationsDesign and implementation defects that lead to inefficient computation widely exist in software. These defects are difficult to avoid and discover. They lead to severe performance degradation and energy waste during production runs, and are becoming ...
Model-Based relative performance diagnosis of wavefront parallel computations
HPCC'06: Proceedings of the Second international conference on High Performance Computing and CommunicationsParallel performance diagnosis can be improved with the use of performance knowledge about parallel computation models. The Hercule diagnosis system applies model-based methods to automate performance diagnosis processes and explain performance problems ...







Comments