skip to main content
research-article

Concurrency testing using schedule bounding: an empirical study

Published:06 February 2014Publication History
Skip Abstract Section

Abstract

We present the first independent empirical study on schedule bounding techniques for systematic concurrency testing (SCT). We have gathered 52 buggy concurrent software benchmarks, drawn from public code bases, which we call SCTBench. We applied a modified version of an existing concurrency testing tool to SCTBench to attempt to answer several research questions, including: How effective are the two main schedule bounding techniques, preemption bounding and delay bounding, at bug finding? What challenges are associated with applying SCT to existing code? How effective is schedule bounding compared to a naive random scheduler at finding bugs? Our findings confirm that delay bounding is superior to preemption bounding and that schedule bounding is more effective at finding bugs than unbounded depth-first search. The majority of bugs in SCTBench can be exposed using a small bound (1-3), supporting previous claims, but there is at least one benchmark that requires 5 preemptions. Surprisingly, we found that a naive random scheduler is at least as effective as schedule bounding for finding bugs. We have made SCTBench and our tools publicly available for reproducibility and use in future work.

References

  1. A. Bessey et al. A few billion lines of code later: using static analysis to find bugs in the real world. Commun. ACM, 53(2): 66--75, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. C. Bienia. Benchmarking Modern Multiprocessors. PhD thesis, Princeton University, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. H.-J. Boehm. How to miscompile programs with "benign" data races. In HotPar, pages 1--6, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. S. Burckhardt, P. Kothari, M. Musuvathi, and S. Nagarakatte. A randomized scheduler with probabilistic guarantees of finding bugs. In ASPLOS, pages 167--178, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. K. E. Coons, M. Musuvathi, and K. S. McKinley. Bounded partial-order reduction. In OOPSLA, pages 833--848, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. L. Cordeiro and B. Fischer. Verifying multi-threaded software using SMT-based context-bounded model checking. In ICSE, pages 331--340, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. M. Emmi, S. Qadeer, and Z. Rakamaric. Delay-bounded scheduling. In POPL, pages 411--422, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. C. Flanagan and S. N. Freund. FastTrack: efficient and precise dynamic race detection. In PLDI, pages 121--133, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. C. Flanagan and P. Godefroid. Dynamic partial-order reduction for model checking software. In POPL, pages 110--121, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. M. Frigo, C. E. Leiserson, and K. H. Randall. The implementation of the Cilk-5 multithreaded language. In PLDI, pages 212--223, 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. P. Godefroid. Partial-Order Methods for the Verification of Concurrent Systems. Springer, 1996. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. P. Godefroid. Model checking for programming languages using VeriSoft. In POPL, pages 174--186, 1997. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. G. J. Holzmann. On limits and possibilities of automated protocol analysis. In PSTV, pages 339--344, 1987. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. G. J. Holzmann and M. Florian. Model checking with bounded context switching. Formal Asp. Comput., 23(3):365--389, 2011. Google ScholarGoogle ScholarCross RefCross Ref
  15. J. Huang and C. Zhang. An efficient static trace simplification technique for debugging concurrent programs. In SAS, pages 163--179, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. N. Jalbert and K. Sen. A trace simplification technique for effective debugging of concurrent programs. In FSE, FSE '10, pages 57--66, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. N. Jalbert, C. Pereira, G. Pokam, and K. Sen. RADBench: a concurrency bug benchmark suite. In HotPar, pages 1--6, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. B. Kasikci, C. Zamfir, and G. Candea. Data races vs. data race bugs: telling the difference with Portend. In ASPLOS, pages 185--198, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. B. Krena, Z. Letko, T. Vojnar, and S. Ur. A platform for search-based testing of concurrent software. In PADTAD, pages 48--58, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. A. Lal and T. W. Reps. Reducing concurrent analysis under a context bound to sequential analysis. Formal Methods in System Design, 35(1):73--97, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. B. Lewis and D. J. Berg. Multithreaded programming with Pthreads. Prentice-Hall, 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. C.-K. Luk et al. Pin: building customized program analysis tools with dynamic instrumentation. In PLDI, pages 190--200, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. M. Musuvathi and S. Qadeer. Iterative context bounding for systematic testing of multithreaded programs. In PLDI, pages 446--455, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. M. Musuvathi and S. Qadeer. Partial-order reduction for context-bounded state exploration. Technical Report MSR- TR-2007-12, Microsoft Research, 2007.Google ScholarGoogle Scholar
  25. M. Musuvathi and S. Qadeer. Fair stateless model checking. In PLDI, pages 362--371, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. M. Musuvathi et al. Finding and reproducing Heisenbugs in concurrent programs. In OSDI, pages 267--280, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. S. Nagarakatte, S. Burckhardt, M. M. Martin, and M. Musu- vathi. Multicore acceleration of priority-based schedulers for concurrency bug detection. In PLDI, pages 543--554, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. S. Narayanasamy et al. Automatically classifying benign and harmful data races using replay analysis. In PLDI, pages 22--31, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. S. Park, S. Lu, and Y. Zhou. CTrigger: exposing atomicity violation bugs from their hiding places. In ASPLOS, pages 25--36, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. K. Sen. Race directed random testing of concurrent programs. In PLDI, pages 11--21, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. H. Sutter and J. Larus. Software and the concurrency revolution. ACM Queue, 3(7):54--62, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. C. Wang, M. Said, and A. Gupta. Coverage guided systematic concurrency testing. In ICSE, pages 221--230, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. S. C. Woo et al. The SPLASH-2 programs: characterization and methodological considerations. In ISCA, pages 24--36, 1995. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Y. Yang, X. Chen, and G. Gopalakrishnan. Inspect: A run- time model checker for multithreaded C programs. Technical Report UUCS-08-004, University of Utah, 2008.Google ScholarGoogle Scholar
  35. J. Yu and S. Narayanasamy. A case for an interleaving constrained shared-memory multi-processor. In ISCA, pages 325--336, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. J. Yu, S. Narayanasamy, C. Pereira, and G. Pokam. Maple: a coverage-driven testing tool for multithreaded programs. In OOPSLA, pages 485--502, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Concurrency testing using schedule bounding: an empirical study

            Recommendations

            Comments

            Login options

            Check if you have access through your login credentials or your institution to get full access on this article.

            Sign in

            Full Access

            PDF Format

            View or Download as a PDF file.

            PDF

            eReader

            View online with eReader.

            eReader
            About Cookies On This Site

            We use cookies to ensure that we give you the best experience on our website.

            Learn more

            Got it!