skip to main content
research-article

DoublePlay: parallelizing sequential logging and replay

Published:05 March 2011Publication History
Skip Abstract Section

Abstract

Deterministic replay systems record and reproduce the execution of a hardware or software system. In contrast to replaying execution on uniprocessors, deterministic replay on multiprocessors is very challenging to implement efficiently because of the need to reproduce the order or values read by shared memory operations performed by multiple threads. In this paper, we present DoublePlay, a new way to efficiently guarantee replay on commodity multiprocessors. Our key insight is that one can use the simpler and faster mechanisms of single-processor record and replay, yet still achieve the scalability offered by multiple cores, by using an additional execution to parallelize the record and replay of an application. DoublePlay timeslices multiple threads on a single processor, then runs multiple time intervals (epochs) of the program concurrently on separate processors. This strategy, which we call uniparallelism, makes logging much easier because each epoch runs on a single processor (so threads in an epoch never simultaneously access the same memory) and different epochs operate on different copies of the memory. Thus, rather than logging the order of shared-memory accesses, we need only log the order in which threads in an epoch are timesliced on the processor. DoublePlay runs an additional execution of the program on multiple processors to generate checkpoints so that epochs run in parallel. We evaluate DoublePlay on a variety of client, server, and scientific parallel benchmarks; with spare cores, DoublePlay reduces logging overhead to an average of 15% with two worker threads and 28% with four threads.

References

  1. ALTEKAR, G., AND STOICA, I. ODR: Output-deterministic replay for multicore debugging. In Proceedings of the 22nd ACM Symposium on Operating Systems Principles (October 2009), pp. 193--206. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. BACON, D. F., AND GOLDSTEIN, S. C. Hardware assisted replay of multiprocessor programs. In Proceedings of the 1991 ACM/ONR Workshop on Parallel and Distributed Debugging (1991), ACM Press, pp. 194--206. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. BERGER, E. D., YANG, T., LIU, T., AND NOVARK, G. Grace: safe multithreaded programming for c/c++. In Proceeding of the 24th ACM SIGPLAN conference on Object oriented programming systems languages and applications (Orlando, Florida, USA, 2009), pp. 81--96. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. BHANSALI, S., CHEN, W., DE JONG, S., EDWARDS, A., AND DRINIC, M. Framework for instruction-level tracing and analysis of programs. In Second International Conference on Virtual Execution Environments (June 2006), pp. 154--163. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. BRESSOUD, T. C., AND SCHNEIDER, F. B. Hypervisor-based fault tolerance. ACM Transactions on Computer Systems 14, 1 (February 1996), 80--107. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. CHOI, J. D., ALPERN, B., NGO, T., AND SRIDHARAN, M. A perturbation free replay platform for cross-optimized multithreaded applications. In Proceedings of the 15th International Parallel and Distributed Processing Symposium (April 2001). Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. CHOW, J., GARFINKEL, T., AND CHEN, P. M. Decoupling dynamic program analysis from execution in virtual environments. In Proceedings of the 2008 USENIX Technical Conference (June 2008), pp. 1--14. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. DEVIETTI, J., LUCIA, B., CEZE, L., AND OSKIN, M. DMP: Deterministic shared memory multiprocessing. In Proceedings of the 2009 International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) (March 2009), pp. 85--96. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. DUNLAP, G. W., KING, S. T., CINAR, S., BASRAI, M. A., AND CHEN, P. M. ReVirt: Enabling intrusion analysis through virtualmachine logging and replay. In Proceedings of the 5th Symposium on Operating Systems Design and Implementation (Boston, MA, December 2002), pp. 211--224. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. DUNLAP, G. W., LUCCHETTI, D. G., FETTERMAN, M., AND CHEN, P. M. Execution replay on multiprocessor virtual machines. In Proceedings of the 2008 ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE) (March 2008), pp. 121--130. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. FELDMAN, S. I., AND BROWN, C. B. IGOR: A system for program debugging via reversible execution. In PADD '88: Proceedings of the 1988 ACM SIGPLAN and SIGOPS Workshop on Parallel and Distributed Debugging (1988), pp. 112--123. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. HOWER, D. R., AND HILL, M. D. Rerun: Exploiting episodes for lightweight memory race recording. In Proceedings of the 2008 International Symposium on Computer Architecture (June 2008), pp. 265--276. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. JR., R. L. B., ADVE, V. S., DIG, D., ADVE, S. V., HEUMANN, S., KOMURAVELLI, R., OVERBEY, J., SIMMONS, P., SUNG, H., AND VAKILIAN, M. A type and effect system for deterministic parallel java. In OOPSLA (2009), pp. 97--116. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. KELSEY, K., BAI, T., DING, C., AND ZHANG, C. Fast Track: A software system for speculative program optimization. In Proceedings of the 2009 International Symposium on Code Generation and Optimization (CGO) (March 2009), pp. 157--168. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. KING, S. T., DUNLAP, G. W., AND CHEN, P. M. Debugging operating systems with time-traveling virtual machines. In Proceedings of the 2005 USENIX Technical Conference (April 2005), pp. 1--15. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. LEBLANC, T. J., AND MELLOR-CRUMMEY, J. M. Debugging parallel programs with instant replay. IEEE Transaction on Computers 36, 4 (1987), 471--482. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. LEE, D., SAID, M., NARAYANASAMY, S., YANG, Z. J., AND PEREIRA, C. Offline symbolic analysis for multi-processor execution replay. In International Symposium on Microarchitecture (MICRO) (2009). Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. LEE, D., WESTER, B., VEERARAGHAVAN, K., CHEN, P. M., FLINN, J., AND NARAYANASAMY, S. Respec: Efficient online multiprocessor replay via speculation and external determinism. In Proceedings of the 15th International Conference on Architectural Support for Programming Languages and Operating Systems (Pittsburgh, PA, March 2010), pp. 77--89. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. MELLOR-CRUMMEY, J. M., AND LEBLANC, T. J. A Software Instruction Counter. In Proceedings of the 1989 International Conference on Architectural Support for Programming Languages and Operating Systems (April 1989), pp. 78--86. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. MONTESINOS, P., CEZE, L., AND TORRELLAS, J. DeLorean: Recording and deterministically replaying shared-memory multiprocessor execution efficiently. In Proceedings of the 2008 International Symposium on Computer Architecture (June 2008), pp. 289--300. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. NARAYANASAMY, S., PEREIRA, C., AND CALDER, B. Recording shared memory dependencies using Strata. In ASPLOS-XII: Proceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems (2006), pp. 229--240. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. NARAYANASAMY, S., PEREIRA, C., PATIL, H., COHN, R., AND CALDER, B. Automatic logging of operating system effects to guide application-level architecture simulation. In International Conference on Measurements and Modeling of Computer Systems (SIGMETRICS) (June 2006), pp. 216--227. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. NARAYANASAMY, S., POKAM, G., AND CALDER, B. BugNet: Continuously recording program execution for deterministic replay debugging. In Proceedings of the 32nd Annual International Symposium on Computer Architecture (ISCA) (June 2005), pp. 284--295. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. NETZER, R. H. B. Optimal tracing and replay for debugging sharedmemory parallel programs. In Proceedings of the ACM/ONR Workshop on Parallel and Distributed Debugging (1993), pp. 1--11. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. NIGHTINGALE, E. B., CHEN, P. M., AND FLINN, J. Speculative execution in a distributed file system. In Proceedings of the 20th ACM Symposium on Operating Systems Principles (Brighton, United Kingdom, October 2005), pp. 191--205. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. NIGHTINGALE, E. B., PEEK, D., CHEN, P. M., AND FLINN, J. Parallelizing security checks on commodity hardware. In Proceedings of the 13th International Conference on Architectural Support for Programming Languages and Operating Systems (Seattle,WA,March 2008), pp. 308--318. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. NIGHTINGALE, E. B., VEERARAGHAVAN, K., CHEN, P. M., AND FLINN, J. Rethink the sync. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation (Seattle, WA, October 2006), pp. 1--14. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. OLSZEWSKI, M., ANSEL, J., AND AMARASINGHE, S. Kendo: efficient deterministic multithreading in software. In Proceedings of the 2009 International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) (March 2009), pp. 97--108. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. OPLINGER, J., AND LAM, M. S. Enhancing software reliability using speculative threads. In Proceedings of the 2002 InternationalConference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) (October 2002), pp. 184--196. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. PARK, S., ZHOU, Y., XIONG, W., YIN, Z., KAUSHIK, R., LEE, K. H., AND LU, S. PRES: Probabilistic replay with execution sketching on multiprocessors. In Proceedings of the 22nd ACM Symposium on Operating Systems Principles (October 2009), pp. 177--191. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. PETERSON, Z. N. J., AND BURNS, R. Ext3cow: A time-shifting file system for regulatory compliance. ACM Transacations on Storage 1, 2 (2005), 190--212. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. RONSSE, M., AND BOSSCHERE, K. D. RecPlay: A full integrated practical record/replay system. ACM Transactions on Computer Systems 17, 2 (May 1999), 133--152. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. RUSSINOVICH, M., AND COGSWELL, B. Replay for concurrent nondeterministic shared-memory applications. In Proceedings of the ACM SIGPLAN 1996 Conference on Programming Language Design and Implementation (1996), pp. 258--266. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. SANTRY, D. S., FEELEY, M. J., HUTCHINSON, N. C., VEITCH, A. C., CARTON, R. W., AND OFIR, J. Deciding when to forget in the Elephant file system. SIGOPS Operating Systems Review 33, 5 (1999), 110--123. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. SOHI, G. S., BREACH, S. E., AND VIJAYKUMAR, T. N. Multiscalar processors. In Proceedings of the 1995 International Symposium on Computer Architecture (June 1995), pp. 414--425. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. SRINIVASAN, S., ANDREWS, C., KANDULA, S., AND ZHOU, Y. Flashback: A light-weight extension for rollback and deterministic replay for software debugging. In Proceedings of the 2004 USENIX Technical Conference (Boston, MA, June 2004), pp. 29--44. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. STEFFAN, J. G., AND MOWRY, T. C. The potential for using threadlevel data speculation to facilitate automatic parallelization. In Proceedings of the 1998 Symposium on High Performance Computer Architecture (February 1998), pp. 2--13. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. SUNDARAMOORTHY, K., PURSER, Z., AND ROTENBERG, E. Slipstream processors: improving both performance and fault tolerance. In Proceedings of the 9th International Conference on Architectural Support for Programming Languages and Operating Systems (November 2000), pp. 257--268. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. SÜ SSKRAUT, M., KNAUTH, T., WEIGERT, S., SCHIFFEL, U., MEINHOLD, M., FETZER, C., BAI, T., DING, C., AND ZHANG, C. Prospect: A compiler framework for speculative parallelization. In Proceedings of the 2010 International Symposium on Code Generation and Optimization (CGO) (April 2010), pp. 131--140. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. TUCEK, J., LU, S., HUANG, C., XANTHOS, S., AND ZHOU, Y. Triage: Diagnosing production run failures at the user's site. In Proceedings of the 21st ACMSymposium on Operating Systems Principles (October 2007), pp. 131--144. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. VEERARAGHAVAN, K., FLINN, J., NIGHTINGALE, E. B., AND NOBLE, B. quFiles: The right file at the right time. In Proceedings of the 8th USENIX Conference on File and Storage Technologies (San Jose, CA, February 2010), pp. 1--14. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. VLACHOS, E., GOODSTEIN, M. L., KOZUCH, M. A., CHEN, S., FALSAFI, B., GIBBONS, P. B., AND MOWRY, T. C. ParaLog: Enabling and accelerating online parallel monitoring of multithreaded applications. In Proceedings of the 15th International Conference on Architectural Support for Programming Languages and Operating Systems (Pittsburgh, PA, March 2010), pp. 271--284. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. WEERATUNGE, D., ZHANG, X., AND JAGANNATHAN, S. Analyzing multicore dumps to facilitate concurrency bug reproduction. In Proceedings of the 2010 International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) (March 2010), pp. 155--166. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. WOO, S. C., OHARA, M., TORRIE, E., SINGH, J. P., AND GUPTA, A. The SPLASH-2 programs: Characterization and methodological considerations. In Proceedings of the 22nd International Symposium on Computer Architecture (June 1995), pp. 24--36. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. XU, M., BODIK, R., AND HILL, M. D. A "flight data recorder" for enabling full-system multiprocessor deterministic replay. In Proceedings of the 2003 International Symposium on Computer Architecture (June 2003), pp. 122--135. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. XU, M., MALYUGIN, V., SHELDON, J., VENKITACHALAM, G., AND WEISSMAN, B. ReTrace: Collecting execution trace with virtual machine deterministic replay. In Proceedings of the 2007 Workshop on Modeling, Benchmarking and Simulation (MoBS) (June 2007).Google ScholarGoogle Scholar
  47. ZAMFIR, C., AND CANDEA, G. Execution synthesis: A technique for automated software debugging. In Proceedings of the 2010 European Conference on Computer Systems (EuroSys) (April 2010), pp. 321--334. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. ZILLES, C., AND SOHI, G. Master/slave speculative parallelization. In Proceedings of the 35th Annual ACM/IEEE International Symposium on Microarchitecture (MICRO) (2002), pp. 85--96. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. DoublePlay: parallelizing sequential logging and replay

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          • Published in

            cover image ACM SIGARCH Computer Architecture News
            ACM SIGARCH Computer Architecture News  Volume 39, Issue 1
            ASPLOS '11
            March 2011
            407 pages
            ISSN:0163-5964
            DOI:10.1145/1961295
            Issue’s Table of Contents
            • cover image ACM Conferences
              ASPLOS XVI: Proceedings of the sixteenth international conference on Architectural support for programming languages and operating systems
              March 2011
              432 pages
              ISBN:9781450302661
              DOI:10.1145/1950365

            Copyright © 2011 ACM

            Publisher

            Association for Computing Machinery

            New York, NY, United States

            Publication History

            • Published: 5 March 2011

            Check for updates

            Qualifiers

            • research-article

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!