skip to main content
research-article

Improving Software Diagnosability via Log Enhancement

Published:01 February 2012Publication History
Skip Abstract Section

Abstract

Diagnosing software failures in the field is notoriously difficult, in part due to the fundamental complexity of troubleshooting any complex software system, but further exacerbated by the paucity of information that is typically available in the production setting. Indeed, for reasons of both overhead and privacy, it is common that only the run-time log generated by a system (e.g., syslog) can be shared with the developers. Unfortunately, the ad-hoc nature of such reports are frequently insufficient for detailed failure diagnosis. This paper seeks to improve this situation within the rubric of existing practice. We describe a tool, LogEnhancer that automatically “enhances” existing logging code to aid in future post-failure debugging. We evaluate LogEnhancer on eight large, real-world applications and demonstrate that it can dramatically reduce the set of potential root failure causes that must be considered while imposing negligible overheads.

References

  1. Aguilera, M. K., Mogul, J. C., Wiener, J. L., Reynolds, P., and Muthitacharoen, A. 2003. Performance debugging for distributed systems of black boxes. In Proceedings of the 19th ACM Symposium on Operating Systems Principles (SOSP’03). ACM, New York, 74--89. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Aho, A. V., Lam, M. S., Sethi, R., and Ullman, J. D. 2006. Compilers: Principles, Techniques, and Tools 2nd Ed. Addison-Wesley Longman Publishing Co., Inc., Boston, MA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Aiken, A., Bugrara, S., Dillig, I., Dillig, T., Hackett, B., and Hawkins, P. 2007. An overview of the saturn project. In Proceedings of the 7th ACM SIGPLAN-SIGSOFT Workshop on Program Analysis for Software Tools and Engineering (PASTE’07). ACM, New York, NY, 43--48. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Apple. 2004. Apple Inc., CrashReport. Tech. rep. TN2123.Google ScholarGoogle Scholar
  5. Ayers, A., Schooler, R., Metcalf, C., Agarwal, A., Rhee, J., and Witchel, E. 2005. Traceback: First fault diagnosis by reconstruction of distributed control flow. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI’05). ACM, New York, NY, 201--212. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Barham, P., Donnelly, A., Isaacs, R., and Mortier, R. 2004. Using magpie for request extraction and workload modelling. In Proceedings of the 6th Conference on Symposium on Opearting Systems Design and Implementation. USENIX Association, Berkeley, CA, 18--18. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Bhatia, S., Kumar, A., Fiuczynski, M. E., and Peterson, L. 2008. Lightweight, high-resolution monitoring for troubleshooting production systems. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation (OSDI’08). USENIX Association, Berkeley, CA, 103--116. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Cadar, C., Dunbar, D., and Engler, D. 2008. Klee: Unassisted and automatic generation of high-coverage tests for complex systems programs. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation (OSDI’08). USENIX Association, Berkeley, CA, 209--224. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Castro, M., Costa, M., and Martin, J.-P. 2008. Better bug reporting with better privacy. In Proceedings of the 13th International Conference on Architectural Support for Programming Languages and Operating Systems. ACM, New York, NY, 319--328. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Chen, S., Kozuch, M., Strigkos, T., Falsafi, B., Gibbons, P. B., Mowry, T. C., Ramachandran, V., Ruwase, O., Ryan, M., and Vlachos, E. 2008. Flexible hardware acceleration for instruction-grain program monitoring. In Proceedings of the 35th Annual International Symposium on Computer Architecture (ISCA’08). IEEE Computer Society, Los Alamitos, CA, 377--388. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Chilimbi, T. M., Liblit, B., Mehra, K., Nori, A. V., and Vaswani, K. 2009. HOLMES: Effective statistical debugging via efficient path profiling. In Proceedings of the 31st International Conference on Software Engineering (ICSE’09). IEEE Computer Society, Los Alamitos, CA, 34--44. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Cisco. Cisco system log management.Google ScholarGoogle Scholar
  13. Cohen, I., Zhang, S., Goldszmidt, M., Symons, J., Kelly, T., and Fox, A. 2005. Capturing, indexing, clustering, and retrieving system history. In Proceedings of the 20th ACM Symposium on Operating Systems Principles (SOSP’05). ACM, New York, NY, 105--118. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Costa, M., Castro, M., Zhou, L., Zhang, L., and Peinado, M. 2007. Bouncer: securing software by blocking bad input. In Proceedings of 21st ACM SIGOPS Symposium on Operating Systems Principles (SOSP’07). ACM, New York, NY, 117--130. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Crameri, O., Bianchini, R., and Zwaenepoel, W. 2011. Striking a new balance between program instrumentation and debugging time. In Proceedings of the 6th Conference on Computer Systems (EuroSys’11). ACM, New York, NY, 199--214. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Dell. 2008. Streamlined troubleshooting with the Dell system E-Support tool. Dell Power Solutions.Google ScholarGoogle Scholar
  17. Detlefs, D. L., Leino, K. R. M., Rustan, K., Leino, M., Nelson, G., and Saxe, J. B. 1998. Extended static checking. Compac SRC Research rep. 159.Google ScholarGoogle Scholar
  18. Devietti, J., Lucia, B., Ceze, L., and Oskin, M. 2009. Dmp: deterministic shared memory multiprocessing. In Proceeding of the 14th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’09). ACM, New York, NY, 85--96. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Dunlap, G. W., Lucchetti, D. G., Fetterman, M. A., and Chen, P. M. 2008. Execution replay of multiprocessor virtual machines. In Proceedings of the 4th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE’08). ACM, New York, NY, 121--130. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. DWARF. The DWARF Debugging Format. http://dwarfstd.org.Google ScholarGoogle Scholar
  21. EMC. 2005. EMC seen collecting and managing log as key driver for 94 percent of customers.Google ScholarGoogle Scholar
  22. Engler, D. and Ashcraft, K. 2003. Racerx: Effective, static detection of race conditions and deadlocks. In Proceedings of the 19th ACM Symposium on Operating Systems Principles (SOSP’03). ACM, New York, NY, 237--252. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Flanagan, C., Leino, K. R. M., Lillibridge, M., Nelson, G., Saxe, J. B., and Stata, R. 2002. Extended static checking for java. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI’02). ACM, New York, NY, 234--245. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. GCORE. Man page for gcore (Linux section 1).Google ScholarGoogle Scholar
  25. Glerum, K., Kinshumann, K., Greenberg, S., Aul, G., Orgovan, V., Nichols, G., Grant, D., Loihle, G., and Hunt, G. 2009. Debugging in the (very) large: Ten years of implementation and experience. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles (SOSP’09). ACM, New York, NY, 103--116. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. GoogleBreakpad. Google Inc., Breakpad. http://code.google.com/p/google-breakpad/.Google ScholarGoogle Scholar
  27. Guo, Z., Wang, X., Tang, J., Liu, X., Xu, Z., Wu, M., Kaashoek, M. F., and Zhang, Z. 2008. R2: an application-level kernel for record and replay. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation (OSDI’08). USENIX Association, Berkeley, CA, 193--208. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Ha, J., Rossbach, C. J., Davis, J. V., Roy, I., Ramadan, H. E., Porter, D. E., Chen, D. L., and Witchel, E. 2007. Improved error reporting for software that uses black-box components. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI’07). ACM, New York, NY, 101--111. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Hackett, B. and Aiken, A. 2006. How is aliasing used in systems software? In Proceedings of the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT’06/FSE-14). ACM, New York, NY, 69--80. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Kadav, A., Renzelmann, M. J., and Swift, M. M. 2009. Tolerating hardware device failures in software. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles (SOSP’09). ACM, New York, NY, 59--72. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Kernighan, B. W. and Pike, R. 1999. The Practice of Programming. Addison-Wesley Longman Publishing Co., Inc., Boston, MA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. King, S. T., Dunlap, G. W., and Chen, P. M. 2005. Debugging operating systems with time-traveling virtual machines. In Proceedings of the USENIX Annual Technical Conference (ATEC’05). USENIX Association, Berkeley, CA, 1--1. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. LeBlanc, T. J. and Mellor-Crummey, J. M. 1987. Debugging parallel programs with instant replay. IEEE Trans. Comput. 36, 471--482. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Lee, D., Wester, B., Veeraraghavan, K., Narayanasamy, S., Chen, P. M., and Flinn, J. 2010. Respec: Efficient online multiprocessor replayvia speculation and external determinism. In Proceedings of the 15th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’10). ACM, New York, NY, 77--90. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Li, Z., Tan, L., Wang, X., Lu, S., Zhou, Y., and Zhai, C. 2006. Have things changed now?: An empirical study of bug characteristics in modern open source software. In Proceedings of the 1st Workshop on Architectural and System Support for Improving Software Dependability (ASID’06). ACM, New York, NY, 25--33. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Liblit, B., Aiken, A., Zheng, A. X., and Jordan, M. I. 2003. Bug isolation via remote program sampling. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI’03). ACM, New York, NY, 141--154. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Manevich, R., Sridharan, M., Adams, S., Das, M., and Yang, Z. 2004. PSE: explaining program failures via postmortem static analysis. In Proceedings of the 12th International Symposium on the Foundations of Software Engineering. 63--72. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Montesinos, P., Ceze, L., and Torrellas, J. 2008. Delorean: Recording and deterministically replaying shared-memory multiprocessor execution efficiently. In Proceedings of the 35th Annual International Symposium on Computer Architecture (ISCA’08). IEEE Computer Society, Los Alamitos, CA, 289--300. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Mozilla QFA. Mozilla Quality Feedback Agent. http://kb.mozillazine.org/Quality_Feedback_Agent.Google ScholarGoogle Scholar
  40. Naik, M. and Aiken, A. 2007. Conditional must not aliasing for static race detection. In Proceedings of the 34th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL’07). ACM, New York, NY, 327--338. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Naik, M., Aiken, A., and Whaley, J. 2006. Effective static race detection for java. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI’06). ACM, New York, NY, 308--319. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Narayanasamy, S., Pokam, G., and Calder, B. 2005. Bugnet: Continuously recording program execution for deterministic replay debugging. In Proceedings of the 32nd Annual International Symposium on Computer Architecture (ISCA’05). IEEE Computer Society, Los Alamitos, CA, 284--295. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Necula, G. C., McPeak, S., Rahul, S. P., and Weimer, W. 2002. CIL: Intermediate language and tools for analysis and transformation of c programs. In Proceedings of the 11th International Conference on Compiler Construction (CC’02). Springer-Verlag, Berlin, 213--228. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. NetApp. 2007. Proactive health management with auto-support. NetApp white paper.Google ScholarGoogle Scholar
  45. NetAppSavecore. NetApp Inc., Savecore. ONTAP 7.3 Manual Page Reference, Volume 1, 471--472.Google ScholarGoogle Scholar
  46. Olszewski, M., Ansel, J., and Amarasinghe, S. 2009. Kendo: Efficient deterministic multithreading in software. In Proceeding of the 14th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’09). ACM, New York, NY, 97--108. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Park, S., Zhou, Y., Xiong, W., Yin, Z., Kaushik, R., Lee, K. H., and Lu, S. 2009. Pres: Probabilistic replay with execution sketching on multiprocessors. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles (SOSP’09). ACM, New York, NY, 177--192. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Schmidt, S. 2009. 7 more good tips on logging. http://codemonkeyism.com/7-more-good-tips-on-logging/.Google ScholarGoogle Scholar
  49. SLOCCount. Sloccount. http://www.dwheeler.com/sloccount/.Google ScholarGoogle Scholar
  50. Subhraveti, D. and Nieh, J. 2011. Record and transplay: Partial checkpointing for replay debugging across heterogeneous systems. In Proceedings of the ACM SIGMETRICS Joint International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS’11). ACM, New York, NY, 109--120. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Tucek, J., Lu, S., Huang, C., Xanthos, S., and Zhou, Y. 2007. Triage: Diagnosing production run failures at the user’s site. In Proceedings of 21st ACM SIGOPS Symposium on Operating Systems Principles (SOSP’07). ACM, New York, NY, 131--144. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Veeraraghavan, K., Lee, D., Wester, B., Ouyang, J., Chen, P. M., Flinn, J., and Narayanasamy, S. 2011. Doubleplay: Parallelizing sequential logging and replay. In Proceedings of the 16th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’11). ACM, New York, NY, 15--26. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Vlachos, E., Goodstein, M. L., Kozuch, M. A., Chen, S., Falsafi, B., Gibbons, P. B., and Mowry, T. C. 2010. Paralog: Enabling and accelerating online parallel monitoring of multithreaded applications. In Proceedings of the 15th Edition of ASPLOS on Architectural Support for Programming Languages and Operating Systems (ASPLOS’10). ACM, New York, NY, 271--284. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. VMWare. Using the intergrated virtual debugger for visual studio. http://www.vmware.com/pdf/ws65_manual.pdf.Google ScholarGoogle Scholar
  55. Weeratunge, D., Zhang, X., and Jagannathan, S. 2010. Analyzing multicore dumps to facilitate concurrency bug reproduction. In Proceedings of the 15th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’10). ACM, New York, NY, 155--166. Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. Xu, M., Bodik, R., and Hill, M. D. 2003. A “flight data recorder” for enabling full-system multiprocessor deterministic replay. In Proceedings of the 30th Annual International Symposium on Computer Architecture (ISCA’03). ACM, New York, NY, 122--135. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Xu, W., Huang, L., Fox, A., Patterson, D., and Jordan, M. I. 2009. Detecting large-scale system problems by mining console logs. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles (SOSP’09). ACM, New York, NY, 117--132. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Yuan, D., Mai, H., Xiong, W., Tan, L., Zhou, Y., and Pasupathy, S. 2010. Sherlog: Error diagnosis by connecting clues from run-time logs. In Proceedings of the 15th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’10). ACM, New York, NY, 143--154. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Zamfir, C. and Candea, G. 2010. Execution synthesis: A technique for automated software debugging. In Proceedings of the 5th European Conference on Computer Systems (EuroSys’10). ACM, New York, NY, 321--334. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. Zhang, X., Tallam, S., and Gupta, R. 2006. Dynamic slicing long running programs through execution fast forwarding. In Proceedings of the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT’06/FSE-14). ACM, New York, NY, 81--91. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. Zhao, Q., Rabbah, R., Amarasinghe, S., Rudolph, L., and Wong, W.-F. 2008. How to do a million watchpoints: Efficient debugging using dynamic instrumentation. In Proceedings of the International Conference on Compiler Construction. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Improving Software Diagnosability via Log Enhancement

        Recommendations

        Reviews

        T.H. Tse

        When a software system fails at a user site, information is often automatically sent to the developer for diagnostics. Usually, only a system log is included, with all the confidential data such as identities and financial information stripped off. Such a log message is found to be insufficient for debugging. In the present paper, the authors set out to enhance the log information, using causal relationships to help narrow down the amount of additional information required for diagnostics. They propose the LogEnhancer tool to support the automation of this approach. The tool performs a delayed collection, which captures all the causally related data still live at the time of the failure, and an in-time collection, which includes all the relevant historical data. The paper provides excellent details for the proposed techniques and tool. An empirical study validates that the approach is feasible. The work provides very useful insight and information for system vendors and debuggers, who rely on automatic bug reports from their users for the diagnosis of software faults. In general, however, the root cause of a failure may be very subtle and may not be supported by the causal relationships conceived by the developer. Thus, even the enhanced log messages may not be sufficient for diagnostics. Rather than using a simple controlled experiment to verify whether the enhanced log messages are truly effective, it may be useful to conduct mutation analysis to study the types of faults that are more easily discovered and the types that are more problematic. To address the latter kinds of faults, it would be helpful to complement the proposed tool using statistical fault-localization techniques that do not rely on causal relationships. Online Computing Reviews Service

        Access critical reviews of Computing literature here

        Become a reviewer for Computing Reviews.

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Computer Systems
          ACM Transactions on Computer Systems  Volume 30, Issue 1
          Special Issue APLOS 2011
          February 2012
          137 pages
          ISSN:0734-2071
          EISSN:1557-7333
          DOI:10.1145/2110356
          Issue’s Table of Contents

          Copyright © 2012 ACM

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 1 February 2012
          • Accepted: 1 October 2011
          • Received: 1 July 2011
          Published in tocs Volume 30, Issue 1

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Research
          • Refereed

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!