skip to main content
research-article

Analyzing blocking to debug performance problems on multi-core systems

Published:14 April 2010Publication History
Skip Abstract Section

Abstract

Multi-core systems are rapidly becoming more prevalent. Consequently, developers frequently face performance bugs caused by unexpected interactions between parallel software components. The location of these bugs is difficult to identify with current tools. Indeed, the process exhibiting the slowness may be separated from the root cause of the problem by a blocking chain involving several other processes.

This article introduces a new approach for analyzing blocking on multi-core systems and reports on its implementation in the LTTV Delay Analyzer. It enables developers to quickly understand the dependencies among processes and see how the total elapsed time is divided into its main components. The LTTV Delay Analyzer was used to analyze and rapidly correct complex performance problems, something not possible with the existing tools. The Linux Trace Toolkit, LTTng, is used for most of the instrumentation and the trace recording, allowing the tracing of production systems with great accuracy and minimal impact. This approach uses solely kernel instrumentation and does not require the instrumentation or recompilation of processes. The analysis time is linear with respect to trace size.

References

  1. K Desktop Environment (KDE). http://www.kde.org. Verified 2009/01/05.Google ScholarGoogle Scholar
  2. LTTV. http://ltt.polymtl.ca. Verified 2009/01/05.Google ScholarGoogle Scholar
  3. MediaWiki. http://www.mediawiki.org. Verified 2009/01/05.Google ScholarGoogle Scholar
  4. memcached. http://www.danga.com/memcached/. Verified 2009/01/05.Google ScholarGoogle Scholar
  5. QNX Momentics. http://www.qnx.com. Verified 2009/01/05.Google ScholarGoogle Scholar
  6. tbench. http://samba.org/ftp/tridge/dbench/README. Verified 2009/01/05.Google ScholarGoogle Scholar
  7. Tracing Wiki. http://ltt.polymtl.ca/tracingwiki. Verified 2009/01/05.Google ScholarGoogle Scholar
  8. Wind River Workbench. http://www.windriver.com/products/workbench/. Verified 2009/01/05.Google ScholarGoogle Scholar
  9. P. Barham, A. Donnelly, R. Isaacs, and R. Mortier. Using Magpie for request extraction and workload modelling. In Symposium on Operating Systems Design and Implementation, pages 259--272, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. B. Cantrill, M. Shapiro, and A. Leventhal. Dynamic instrumentation of production systems. pages 15--28, Boston, MA, USA, 2004.Google ScholarGoogle Scholar
  11. J. Corbet. Kernel markers. LWN.net, Aug. 2007. http://lwn.net/Articles/245671/. Verified 2009/01/05.Google ScholarGoogle Scholar
  12. M. Desnoyers and M. R. Dagenais. The LTTng tracer: A low impact performance and behavior monitor for GNU/Linux. In Linux Symposium, Ottawa, Ontario, Canada, June 2006.Google ScholarGoogle Scholar
  13. M. Desnoyers and M. R. Dagenais. LTTng: Tracing across execution layers, from the hypervisor to user-space. In Linux Symposium, 2008.Google ScholarGoogle Scholar
  14. Y. Endo and M. Seltzer. Improving interactive performance using TIPME. SIGMETRICS Perform. Eval. Rev., 28(1):240--251, 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. S. Graham, P. Kessler, and M. McKusick. gprof: a call graph execution profiler. volume 17, pages 120--6, Boston, MA, USA, 1982. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. J. Levon and P. Elie. Oprofile: A system profiler for Linux, 2005.Google ScholarGoogle Scholar
  17. P. Miller Barton, D. Callaghan Mark, M. Cargille Jonathan, et al. The Paradyn Parallel Performance Measurement Tool. IEEE Computer, 28(11):37--46, 1995. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. P. Reynolds, C. Killian, J. Wiener, J. Mogul, M. Shah, and A. Vahdat. Pip: Detecting the unexpected in distributed systems. In Symposium on Networked Systems Design and Implementation, pages 115--128, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Y. Ruan and V. Pai. Making the "box" transparent: system call performance as a first-class result. In Proceedings of the annual conference on USENIX Annual Technical Conference. USENIX Association Berkeley, CA, USA, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. S. Sandmann. Sysprof-a system-wide linux profiler. http://www.daimi.au.dk/~sandmann/sysprof/. Verified 2009/01/05.Google ScholarGoogle Scholar

Index Terms

  1. Analyzing blocking to debug performance problems on multi-core systems

            Recommendations

            Comments

            Login options

            Check if you have access through your login credentials or your institution to get full access on this article.

            Sign in

            Full Access

            • Published in

              cover image ACM SIGOPS Operating Systems Review
              ACM SIGOPS Operating Systems Review  Volume 44, Issue 2
              April 2010
              92 pages
              ISSN:0163-5980
              DOI:10.1145/1773912
              Issue’s Table of Contents

              Copyright © 2010 Authors

              Publisher

              Association for Computing Machinery

              New York, NY, United States

              Publication History

              • Published: 14 April 2010

              Check for updates

              Qualifiers

              • research-article

            PDF Format

            View or Download as a PDF file.

            PDF

            eReader

            View online with eReader.

            eReader
            About Cookies On This Site

            We use cookies to ensure that we give you the best experience on our website.

            Learn more

            Got it!