skip to main content
article

Fine grained kernel logging with KLogger: experience and insights

Authors Info & Claims
Published:21 March 2007Publication History
Skip Abstract Section

Abstract

Understanding the detailed behavior of an operating system is crucial for making informed design decisions. But such an understanding is very hard to achieve, due to the increasing complexity of such systems and the fact that they are implemented and maintained by large and diverse groups of developers. Tools like KLogger --- presented in this paper --- can help by enabling fine-grained logging of system events and the sharing of a logging infrastructure between multiple developers and researchers, facilitating a methodology where design evaluation can be an integral part of kernel development. We demonstrate the need for such methodology by a host of case studies, using KLogger to better understand various subsystems in the Linux kernel, and pinpointing overheads and problems therein.

References

  1. J. M. Anderson, W. E. Weihl, L. M. Berc, J. Dean, S. Ghemawat, M. R. Henzinger, S.-T. A. Leung, R. L. Sites, M. T. Vandevoorde, and C. A. Waldspurger. Continuous profiling: where have all the cycles gone? In ACM Symp. on Operating Systems Principles, pages 1--14, 1997. ACM Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. A. C. Arpaci-Dusseau and R. H. Arpaci-Dusseau. Information and control in gray-box systems. In ACM Symp. on Operating Systems Principles, 2001. ACM Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. D. L. Black, A. Tevanian, D. B. Golub, and M. W. Young. Locking and reference counting in the Mach kernel. In Intl. Conf. on Parallel Processing, volume 2, pages 167--173, 1991.Google ScholarGoogle Scholar
  4. D. P. Bovet and M. Cesati. Understanding the Linux Kernel. O'Reilly & Associates, Sebastopol, CA, USA, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Z. Brown. What's new in kernel developement. Linux Journal, page 10, Mar 2005.Google ScholarGoogle Scholar
  6. R. Bryant, R. Forester, and J. Hawkes. Filesystem performance and scalability in Linux 2.4.17. In Usenix Annual Technical Conf. (FreeNix Track), 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. R. Bryant and J. Hawkes. Lockmeter: Highly-informative instrumentation for spin locks in the Linux kernel. In 4th Annual Linux Showcase & Conf., 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. B. M. Cantrill, M. W. Shapiro, and A. H. Leventhal. Dynamic instrumentation of production systems. In Usenix Annual Technical Conf., June 2004. The Usenix Association. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. J. Casmira, D. Kaeli, and D. Hunter. Tracing and characterization of NT-based system workloads. Digital Tech. J., 10(1):6--21, Dec 1998.Google ScholarGoogle Scholar
  10. Y. Endo and M. Seltzer. Improving interactive performance using TIPME. In Intl. Conf. on Measurement & Modeling of Computer Systems, pages 240--251, June 2000. ACM Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Y. Etsion, D. Tsafrir, and D. G. Feitelson. Effects of clock resolution on the scheduling of interactive and soft real-time processes. In Intl. Conf. on Measurement & Modeling of Computer Systems, pages 172--183, Jun 2003. ACM Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. E. Frachtenberg and Y. Etsion. Hardware parallelism: Are operating systems ready? (case studies in mis-scheduling). In Workshop on the Interaction between Operating System and Computer Architecture, pages 7--15, June 2006. http://www.ideal.ece.ufl.edu/workshops/wiosca06/final.pdf.Google ScholarGoogle Scholar
  13. R. Gioiosa, F. Petrini, K. Davis, and F. Lebaillif-Delamare. Analysis of system overhead on parallel computers. In IEEE Intl. Symp. on Signal Processing and Information Technology, Dec 2004. IEEE Computer Society Press.Google ScholarGoogle ScholarCross RefCross Ref
  14. Intel Corp. IA-32 Intel Achitecture Software Developr's Manual. Vol. 3: System Programming Guide.Google ScholarGoogle Scholar
  15. M. B. Jones and J. Regehr. The problems you're having may not be the problems you think you're having: Results from a latency study of windows NT. In 7th Workshop on Hot Topics in Operating Systems, page 96, March 1999. The Usenix Association. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. M. Kravetz and H. Franke. Enhancing the Linux scheduler. In Ottawa Linux Symp., 2001.Google ScholarGoogle Scholar
  17. J. Lions. Lions' Commentary on UNIX 6th Edition. Annabooks, 1996. (reprint). Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. R. Love. Linux Kernel Development. Novell Press, Indianapolis, IN, USA, 2nd edition, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. R. Love, L. Torvalds, A. Cox, and various kernel developers. LKML Thread: Improving interactivity. http://kerneltrap.org/node/603, Mar 2003.Google ScholarGoogle Scholar
  20. J. M. Mellor-Crummey and M. L. Scott. Algorithms for scalable synchronization on shared-memory multiprocessors. ACM Trans. on Computer Systems, 9(1):21--65, Feb 1991. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. S. W. Melvin and Y. N. Patt. The use of microcode instrumentation for development, debugging and tuning of operating system kernels. In Intl. Conf. on Measurement & Modeling of Computer Systems, pages 207--214, 1988. ACM Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. J. K. Ousterhout. Why aren't operating systems getting faster as fast as hardware? In Usenix Annual Technical Conf. (Summer), Jun 1990. The Usenix Association.Google ScholarGoogle Scholar
  23. P. S. Panchamukhi. Kernel debugging with Kprobes. http://www-106.ibm.com/developerworks/linux/library/l-kprobes.html, Aug 2004.Google ScholarGoogle Scholar
  24. F. Petrini, D. J. Kerbyson, and S. Pakin. The case of missing supercomputer performance: Achieving optimal performance on the 8,192 processors of ASCI Q. In Supercomputing, Nov 2003. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. M. Rosenblum, E. Bugnion, S. Devine, and S. A. Herrod. Using the simos machine simulator to study complex computer systems. ACM Trans. on Modelling & Computer Simulation, 7(1):78--103, 1997. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. M. Rosenblum, E. Bugnion, S. A. Herrod, E. Witchel, and A. Gupta. The impact of architectural trends on operating system performance. In ACM Symp. on Operating Systems Principles, 1995. ACM Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. C. Schimmel. UNIX Systems for Modern Architectures. Addison Wesley, Reading, MA, USA, 1994. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. B. A. Schroeder. On line monitoring: A tutorial. IEEE Computer, 28(6):72--78, June 1995. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. D. A. Solomon and M. E. Russinovich. Inside Windows 2000. Microsoft Press, Redmond, WA, USA, 3rd ed., 2000.Google ScholarGoogle Scholar
  30. A. Tamches and B. P. Miller. Fine-grained dynamic instrumentation of commodity operating system kernels. In 3rd Symp. on Operating Systems Design & Impl., Feb 1999. The Usenix Association. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. J. Torrellas, A. Gupta, and J. Hennessy. Characterizing the caching and synchronization performance of a multiprocessor operating system. In Arch. Support for Programming Languages & Operating Systems, pages 162--174, Oct. 1992. ACM Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. R. Unrau, O. Krieger, B. Gamsa, and M. Stumm. Experiences with locking in a NUMA multiprocessor operating system kernel. In Symp. on Operating Systems Design & Impl., Nov 1994. The Usenix Association. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Various Kernel Developers. LKML Thread: My thoughts on the "new development model". http://www.lkml.org, Oct 2004.Google ScholarGoogle Scholar
  34. R. W. Wisniewski and B. Rosenburg. Efficient, unified, and scalable performance monitoring for multiprocessor operating systems. In Supercomputing, Nov 2003. IEEE Computer Society. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. K. Yaghmour and M. R. Dagenais. Measuring and characterizing system behavior using kernel-level event logging. In Usenix Annual Technical Conf., June 2000. The Usenix Association. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. T. Zanussi, K. Yaghmour, R. W. Wisniewski, M. Dagenais, and R. Moore. An efficient unified approach for trasmitting data from kernel to user space. In Ottawa Linux Symp., Jul 2003.Google ScholarGoogle Scholar

Index Terms

  1. Fine grained kernel logging with KLogger: experience and insights

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          • Published in

            cover image ACM SIGOPS Operating Systems Review
            ACM SIGOPS Operating Systems Review  Volume 41, Issue 3
            EuroSys'07 Conference Proceedings
            June 2007
            386 pages
            ISSN:0163-5980
            DOI:10.1145/1272998
            Issue’s Table of Contents
            • cover image ACM Conferences
              EuroSys '07: Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007
              March 2007
              431 pages
              ISBN:9781595936363
              DOI:10.1145/1272996

            Copyright © 2007 ACM

            Publisher

            Association for Computing Machinery

            New York, NY, United States

            Publication History

            • Published: 21 March 2007

            Check for updates

            Qualifiers

            • article

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!