Abstract
Diagnosing software failures in the field is notoriously difficult, in part due to the fundamental complexity of trouble-shooting any complex software system, but further exacerbated by the paucity of information that is typically available in the production setting. Indeed, for reasons of both overhead and privacy, it is common that only the run-time log generated by a system (e.g., syslog) can be shared with the developers. Unfortunately, the ad-hoc nature of such reports are frequently insufficient for detailed failure diagnosis. This paper seeks to improve this situation within the rubric of existing practice. We describe a tool, LogEnhancer that automatically "enhances" existing logging code to aid in future post-failure debugging. We evaluate LogEnhancer on eight large, real-world applications and demonstrate that it can dramatically reduce the set of potential root failure causes that must be considered during diagnosis while imposing negligible overheads.
- Cisco system log management.Google Scholar
- EMC seen collecting and managing log as key driver for 94 percent of customers.Google Scholar
- M. K. Aguilera, J. C. Mogul, J. L. Wiener, P. Reynolds, and A. Muthitacharoen. Performance debugging for distributed systems of black boxes. In SOSP'03. Google Scholar
Digital Library
- A. V. Aho, M. S. Lam, R. Sethi, and J. D. Ullman. Compilers: Principles, Techniques, and Tools (2nd Edition), Page 528. Google Scholar
Digital Library
- A. Aiken, S. Bugrara, I. Dillig, T. Dillig, P. Hawkins, and B. Hackett. An overview of the Saturn project. In PASTE'07. Google Scholar
Digital Library
- Apple Inc., CrashReport. Technical Report TN2123, 2004.Google Scholar
- A. Ayers, R. Schooler, C. Metcalf, A. Agarwal, J. Rhee, and E. Witchel. Traceback: First fault diagnosis by reconstruction of distributed control flow. In PLDI'05. Google Scholar
Digital Library
- P. Barham, A. Donnelly, R. Isaacs, and R. Mortier. Using magpie for request extraction and workload modelling. In OSDI'04. Google Scholar
Digital Library
- E. D. Berger and B. G. Zorn. Diehard: probabilistic memory safety for unsafe languages. In PLDI'06. Google Scholar
Digital Library
- S. Bhatia, A. Kumar, M. Fiuczynski, and L. Peterson. Lightweight, high-resolution monitoring for troubleshooting production systems. In OSDI'08. Google Scholar
Digital Library
- C. Cadar, D. Dunbar, and D. R. Engler. Klee: Unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI'08. Google Scholar
Digital Library
- M. Castro, M. Costa, and J.-P. Martin. Better bug reporting with better privacy. In ASPLOS'08. Google Scholar
Digital Library
- S. Chen, M. Kozuch, T. Strigkos, B. Falsafi, P. B. Gibbons, T. C. Mowry, V. Ramachandran, O. Ruwase, M. Ryan, and E. Vlachos. Flexible hardware acceleration for instruction-grain program monitoring. In ISCA'08. Google Scholar
Digital Library
- L. Chew and D. Lie. Kivati: fast detection and prevention of atomicity violations. In EuroSys'10. Google Scholar
Digital Library
- T. M. Chilimbi, B. Liblit, K. Mehra, A. V. Nori, and K. Vaswani. HOLMES: Effective statistical debugging via efficient path profiling. In ICSE'09. Google Scholar
Digital Library
- I. Cohen, S. Zhang, M. Goldszmidt, J. Symons, T. Kelly, and A. Fox. Capturing, indexing, clustering, and retrieving system history. In SOSP'05. Google Scholar
Digital Library
- M. Costa, M. Castro, L. Zhou, L. Zhang, and M. Peinado. Bouncer: securing software by blocking bad input. In SOSP'07. Google Scholar
Digital Library
- Dell. Streamlined Troubleshooting with the Dell system E-Support tool. Dell Power Solutions, 2008.Google Scholar
- D. L. Detlefs, K. R. M. Leino, K. Rustan, M. Leino, G. Nelson, and J. B. Saxe. Extended static checking. In TR SRC-159, COMPAQ SRC, 1998. Google Scholar
Digital Library
- J. Devietti, B. Lucia, M. Oskin, and L. Ceze. Dmp: Deterministic shared-memory multiprocessing. In ASPLOS'09. Google Scholar
Digital Library
- G. Dunlap, D. Lucchetti, M. Fetterman, and P. Chen. Execution replay of multiprocessor virtual machines. In VEE, 2008. Google Scholar
Digital Library
- The DWARF Debugging Format. http://dwarfstd.org.Google Scholar
- D. Engler and K. Ashcraft. Racerx: effective, static detection of race conditions and deadlocks. In SOSP'03. Google Scholar
Digital Library
- C. Flanagan, K. R. M. Leino, M. Lillibridge, G. Nelson, J. B. Saxe, and R. Stata. Extended static checking for java. In PLDI'02. Google Scholar
Digital Library
- Man page for gcore (Linux section 1).Google Scholar
- K. Glerum, K. Kinshumann, S. Greenberg, G. Aul, V. Orgovan, G. Nichols, D. Grant, G. Loihle, and G. Hunt. Debugging in the (very) large: ten years of implementation and experience. In SOSP'09. Google Scholar
Digital Library
- Google Inc., Breakpad.Google Scholar
- Z. Guo, X. Wang, J. Tang, X. Liu, Z. Xu, M. Wu, M. F. Kaashoek, and Z. Zhang. R2: An application-level kernel for record and replay. In OSDI'08. Google Scholar
Digital Library
- J. Ha, C. J. Rossbach, J. V. Davis, I. Roy, H. E. Ramadan, D. E. Porter, D. L. Chen, and E. Witchel. Improved error reporting for software that uses black-box components. In PLDI'07. Google Scholar
Digital Library
- B. Hackett and A. Aiken. How is aliasing used in systems software? In FSE'06. Google Scholar
Digital Library
- A. Kadav, M. J. Renzelmann, and M. M. Swift. Tolerating hardware device failures in software. In SOSP'09. Google Scholar
Digital Library
- B. W. Kernighan and R. Pike. The Practice of Programming. Addison-Wesley, 1999. Google Scholar
Digital Library
- S. T. King, G. W. Dunlap, and P. M. Chen. Debugging operating systems with time-traveling virtual machines. In USENIX ATC'05. Google Scholar
Digital Library
- T. J. LeBlanc and J. M. Mellor-Crummey. Debugging parallel programs with instant replay. IEEE Trans. Comput., 36(4), 1987. Google Scholar
Digital Library
- D. Lee, B. Wester, K. Veeraraghavan, S. Narayanasamy, P. M. Chen, and J. Flinn. Respec: efficient online multiprocessor replay via speculation and external determinism. In ASPLOS'10. Google Scholar
Digital Library
- Z. Li, L. Tan, X. Wang, S. Lu, Y. Zhou, and C. Zhai. Have things changed now? An empirical study of bug characteristics in modern open source software. In ASID '06: Proceedings of the 1st workshop on Architectural and system support for improving software dependability, October 2006. Google Scholar
Digital Library
- B. Liblit, A. Aiken, A. X. Zheng, and M. I. Jordan. Bug isolation via remote program sampling. In PLDI'03. Google Scholar
Digital Library
- B. Lucia, L. Ceze, and K. Strauss. Colorsafe: architectural support for debugging and dynamically avoiding multi-variable atomicity violations. In ISCA'10. Google Scholar
Digital Library
- R. Manevich, M. Sridharan, S. Adams, M. Das, and Z. Yang. PSE: Explaining program failures via postmortem static analysis. SIGSOFT Softw. Eng. Notes, 29(6):63--72, 2004. Google Scholar
Digital Library
- P. Montesinos, L. Ceze, and J. Torrellas. Delorean: Recording and deterministically replaying shared-memory multiprocessor execution efficiently. In ISCA'08. Google Scholar
Digital Library
- Mozilla Quality Feedback Agent. http://support.mozilla.com/en-US/kb/quality feedback agent.Google Scholar
- M. Naik and A. Aiken. Conditional must not aliasing for static race detection. In POPL'07. Google Scholar
Digital Library
- M. Naik, A. Aiken, and J. Whaley. Effective static race detection for java. In PLDI'06. Google Scholar
Digital Library
- S. Narayanasamy, G. Pokam, and B. Calder. Bugnet: Continuously recording program execution for deterministic replay debugging. In ISCA'05. Google Scholar
Digital Library
- G. C. Necula, S. McPeak, S. P. Rahul, and W. Weimer. Cil: Intermediate language and tools for analysis and transformation of c programs. In CC'02. Google Scholar
Digital Library
- NetApp Inc., Savecore. ONTAP 7.3 Manual Page Reference, Volume 1, Pages 471--472.Google Scholar
- NetApp. Proactive health management with auto-support. NetApp White Paper, 2007.Google Scholar
- G. Novark, E. D. Berger, and B. G. Zorn. Exterminator: automatically correcting memory errors with high probability. In PLDI'07. Google Scholar
Digital Library
- M. Olszewski, J. Ansel, and S. Amarasinghe. Kendo: Efficient Deterministic Multithreading in software. In ASPLOS'09. Google Scholar
Digital Library
- S. Park, W. Xiong, Z. Yin, R. Kaushik, K. H.Lee, S. Lu, and Y. Zhou. Pres:probabilistic replay with execution sketching on multiprocessors. In SOSP, 2009. Google Scholar
Digital Library
- S. Schmidt. 7 more good tips on logging. http://codemonkeyism.com/7-more-good-tips-on-logging/.Google Scholar
- E. Vlachos, M. L. Goodstein, M. A. Kozuch, S. Chen, B. Falsafi, P. B. Gibbons, and T. C. Mowry. Paralog: enabling and accelerating online parallel monitoring of multithreaded applications. In ASPLOS'10. Google Scholar
Digital Library
- VMware. Using the integrated virtual debugger for visual studio. http://www.vmware.com/pdf/ws65_manual.pdf.Google Scholar
- D. Weeratunge, X. Zhang, and S. Jagannathan. Analyzing multicore dumps to facilitate concurrency bug reproduction. SIGARCH Comput. Archit. News, 38(1):155--166, 2010. Google Scholar
Digital Library
- M. Xu, R. Bodik, and M. Hill. A "flight data recorder" for enabling full-system multiprocessor deterministic replay. In ISCA'03. Google Scholar
Digital Library
- W. Xu, L. Huang, M. Jordan, D. Patterson, and A. Fox. Mining console logs for large-scale system problem detection. In SOSP'09.Google Scholar
- D. Yuan, H. Mai, W. Xiong, L. Tan, Y. Zhou, and S. Pasupathy. SherLog: Error diagnosis by connecting clues from run-time logs. In ASPLOS'10. Google Scholar
Digital Library
- C. Zamfir and G. Candea. Execution synthesis: a technique for automated software debugging. In EuroSys'10. Google Scholar
Digital Library
- C. Zamfir and G. Candea. Low-overhead bug fingerprinting for fast debugging. In Runtime Verification, volume 6418 of Lecture Notes in Computer Science, pages 460--468. 2010. Google Scholar
Digital Library
- Q. Zhao, R. Rabbah, S. Amarasinghe, L. Rudolph, and W.-F. Wong. How to do a million watchpoints: efficient debugging using dynamic instrumentation. In CC'08. Google Scholar
Digital Library
Index Terms
Improving software diagnosability via log enhancement
Recommendations
Improving software diagnosability via log enhancement
ASPLOS '11Diagnosing software failures in the field is notoriously difficult, in part due to the fundamental complexity of trouble-shooting any complex software system, but further exacerbated by the paucity of information that is typically available in the ...
Improving software diagnosability via log enhancement
ASPLOS XVI: Proceedings of the sixteenth international conference on Architectural support for programming languages and operating systemsDiagnosing software failures in the field is notoriously difficult, in part due to the fundamental complexity of trouble-shooting any complex software system, but further exacerbated by the paucity of information that is typically available in the ...
Improving Software Diagnosability via Log Enhancement
Special Issue APLOS 2011Diagnosing software failures in the field is notoriously difficult, in part due to the fundamental complexity of troubleshooting any complex software system, but further exacerbated by the paucity of information that is typically available in the ...







Comments