Abstract
This paper presents an approach for performance analysis of modern enterprise-class server applications. In our experience, performance bottlenecks in these applications differ qualitatively from bottlenecks in smaller, stand-alone systems. Small applications and benchmarks often suffer from CPU-intensive hot spots. In contrast, enterprise-class multi-tier applications often suffer from problems that manifest not as hot spots, but as idle time, indicating a lack of forward motion. Many factors can contribute to undesirable idle time, including locking problems, excessive system-level activities like garbage collection, various resource constraints, and problems driving load.
We present the design and methodology for WAIT, a tool to diagnosis the root cause of idle time in server applications. Given lightweight samples of Java activity on a single tier, the tool can often pinpoint the primary bottleneck on a multi-tier system. The methodology centers on an informative abstraction of the states of idleness observed in a running program. This abstraction allows the tool to distinguish, for example, between hold-ups on a database machine, insufficient load, lock contention in application code, and a conventional bottleneck due to a hot method. To compute the abstraction, we present a simple expert system based on an extensible set of declarative rules.
WAIT can be deployed on the fly, without modifying or even restarting the application. Many groups in IBM have applied the tool to diagnosis performance problems in commercial systems, and we present a number of examples as case studies.
- }}M. K. Aguilera, J. C. Mogul, J. L. Wiener, P. Reynolds, and A. Muthitacharoen. Performance debugging for distributed systems of black boxes. In Symposium on Operating System Principles. ACM, 2003. Google Scholar
Digital Library
- }}W. P. Alexander, R. F. Berry, F. E. Levine, and R. J. Urquhart. A unifying approach to performance analysis in the java environment. IBM Systems Journal, 39(1), 2000. Google Scholar
Digital Library
- }}G. Ammons, J.-D. Choi, M. Gupta, and N. Swamy. Finding and removing performance bottlenecks in large systems. In The European Conference on Object-Oriented Programming. Springer, 2004.Google Scholar
Cross Ref
- }}B. Darmawan, R. Gummadavelli, S. Kuppusamy, C. Tan, D. Rintoul, H. Anglin, H. Chuan, A. Subhedar, A. Firtiyan, P. Nambiar, P. Lall, R. Dhall, and R. Pires. IBM Tivoli Composite Application Manager Family: Installation, Configuration, and Basic Usage. http://www.redbooks.ibm.com/abstracts/sg247151.html?Open.Google Scholar
- }}W. De Pauw, E. Jensen, N. Mitchell, G. Sevitsky, J. Vlissides, and J. Yang. Visualizing the execution of Java programs. In Software Visualization, State-of-the-art Survey, volume 2269 of Lecture Notes in Computer Science. Springer-Verlag, 2002. Google Scholar
Digital Library
- }}R. J. Hall. Cpprofj: Aspect-capable call path profiling of multi-threaded java applications. In Automated Software Engineering, pages 107--116. IEEE Computer Society Press, 2002. Google Scholar
Digital Library
- }}J. K. Hollingsworth. Finding Bottlenecks in Large-scale Parallel Programs. PhD thesis, University of Wisconsin, Aug. 1994. Google Scholar
Digital Library
- }}E. W. Host and B. M. Ostvold. Debugging method names. In The European Conference on Object-Oriented Programming. Springer, 2009. Google Scholar
Digital Library
- }}N. Mitchell, G. Sevitsky, and H. Srinivasan. Modeling runtime behavior in framework-based applications. The European Conference on Object-Oriented Programming, 2006. Google Scholar
Digital Library
- }}K. Srinivas and H. Srinivasan. Summarizing application performance from a components perspective. Foundations of Software Engineering, 30(5):136--145, 2005. Google Scholar
Digital Library
- }}Borland Software Corporation. OptimizeItTM Enterprise Suite. http://www.borland.com/us/products/optimizeit, 2005.Google Scholar
- }}Compuware. Compuware Vantage Analyzer. http://www.compuware.com/solutions/e2e brochures factsheets.asp.Google Scholar
- }}Eclipse. Eclipse Test & Performance Tools Platform Project. http://www.eclipse.org/tptp.Google Scholar
- }}HP. HP Diagnostics for J2EE.Google Scholar
- }}IBM. Compuware Vantage Analyzer. http://alphaworks.ibm.com/tech/dcva4j/download.Google Scholar
- }}IBM. IBM OMEGAMON XE for WebSphere. http://www-01.ibm.com/software/tivoli/products/omegamon-xe-was.Google Scholar
- }}IBM. Thread and Monitor Dump Analyzer for Java. http://www.alphaworks.ibm.com/tech/jca.Google Scholar
- }}IBM. Tivoli Monitoring for Transaction Performance. http://www-01.ibm.com/software/tivoli/products/monitor-transaction.Google Scholar
- }}Sun Microsystems. HPROF JVM profiler. http://java.sun.com/developer/technicalArticles/Programming/HPROF.html, 2005.Google Scholar
- }}Yourkit LLC. Yourkit profiler. http://www.yourkit.com.Google Scholar
- }}G. Xu, M. Arnold, N. Mitchell, A. Rountev, and G. Sevitsky. Go with the flow: profiling copies to find runtime bloat. In PLDI '09: Proceedings of the 2009 ACM SIGPLAN conference on Programming language design and implementation, pages 419--430, New York, NY, USA, 2009. ACM. Google Scholar
Digital Library
- }}W. Xu, L. Huang, A. Fox, D. Patterson, and M. I. Jordan. Detecting large-scale system problems by mining console logs. In Symposium on Operating System Principles. ACM, 2009. Google Scholar
Digital Library
- }}D. Yuan, H. Mai, W. Xiong, L. Tan, Y. Zhou, and S. Pasupathy. Sherlog: Error diagnosis by connecting clues from run-time logs. In Architectural Support for Programming Languages and Operating Systems, Mar. 2010. Google Scholar
Digital Library
- }}Y. Zhao, J. Shi, K. Zheng, H. Wang, H. Lin, and L. Shao. Allocation wall: a limiting factor of java applications on emerging multi-core platforms. In OOPSLA '09: Proceeding of the 24th ACM SIGPLAN conference on Object oriented programming systems languages and applications, pages 361--376, New York, NY, USA, 2009. ACM. Google Scholar
Digital Library
Index Terms
Performance analysis of idle programs
Recommendations
Performance analysis of idle programs
OOPSLA '10: Proceedings of the ACM international conference on Object oriented programming systems languages and applicationsThis paper presents an approach for performance analysis of modern enterprise-class server applications. In our experience, performance bottlenecks in these applications differ qualitatively from bottlenecks in smaller, stand-alone systems. Small ...
Diagnosing performance overheads in the xen virtual machine environment
VEE '05: Proceedings of the 1st ACM/USENIX international conference on Virtual execution environmentsVirtual Machine (VM) environments (e.g., VMware and Xen) are experiencing a resurgence of interest for diverse uses including server consolidation and shared hosting. An application's performance in a virtual machine environment can differ markedly from ...
Performance Analysis of Large Receive Offload in a Xen Virtualized System
ICCET '09: Proceedings of the 2009 International Conference on Computer Engineering and Technology - Volume 01System-level virtualization provides us various advantages including independent and isolated computing environments on which multiple operating systems can be executed, and improved resource utilization. These benefits come with performance overheads ...







Comments