Abstract
We present the first independent empirical study on schedule bounding techniques for systematic concurrency testing (SCT). We have gathered 52 buggy concurrent software benchmarks, drawn from public code bases, which we call SCTBench. We applied a modified version of an existing concurrency testing tool to SCTBench to attempt to answer several research questions, including: How effective are the two main schedule bounding techniques, preemption bounding and delay bounding, at bug finding? What challenges are associated with applying SCT to existing code? How effective is schedule bounding compared to a naive random scheduler at finding bugs? Our findings confirm that delay bounding is superior to preemption bounding and that schedule bounding is more effective at finding bugs than unbounded depth-first search. The majority of bugs in SCTBench can be exposed using a small bound (1-3), supporting previous claims, but there is at least one benchmark that requires 5 preemptions. Surprisingly, we found that a naive random scheduler is at least as effective as schedule bounding for finding bugs. We have made SCTBench and our tools publicly available for reproducibility and use in future work.
- A. Bessey et al. A few billion lines of code later: using static analysis to find bugs in the real world. Commun. ACM, 53(2): 66--75, 2010. Google Scholar
Digital Library
- C. Bienia. Benchmarking Modern Multiprocessors. PhD thesis, Princeton University, 2011. Google Scholar
Digital Library
- H.-J. Boehm. How to miscompile programs with "benign" data races. In HotPar, pages 1--6, 2011. Google Scholar
Digital Library
- S. Burckhardt, P. Kothari, M. Musuvathi, and S. Nagarakatte. A randomized scheduler with probabilistic guarantees of finding bugs. In ASPLOS, pages 167--178, 2010. Google Scholar
Digital Library
- K. E. Coons, M. Musuvathi, and K. S. McKinley. Bounded partial-order reduction. In OOPSLA, pages 833--848, 2013. Google Scholar
Digital Library
- L. Cordeiro and B. Fischer. Verifying multi-threaded software using SMT-based context-bounded model checking. In ICSE, pages 331--340, 2011. Google Scholar
Digital Library
- M. Emmi, S. Qadeer, and Z. Rakamaric. Delay-bounded scheduling. In POPL, pages 411--422, 2011. Google Scholar
Digital Library
- C. Flanagan and S. N. Freund. FastTrack: efficient and precise dynamic race detection. In PLDI, pages 121--133, 2009. Google Scholar
Digital Library
- C. Flanagan and P. Godefroid. Dynamic partial-order reduction for model checking software. In POPL, pages 110--121, 2005. Google Scholar
Digital Library
- M. Frigo, C. E. Leiserson, and K. H. Randall. The implementation of the Cilk-5 multithreaded language. In PLDI, pages 212--223, 1998. Google Scholar
Digital Library
- P. Godefroid. Partial-Order Methods for the Verification of Concurrent Systems. Springer, 1996. Google Scholar
Digital Library
- P. Godefroid. Model checking for programming languages using VeriSoft. In POPL, pages 174--186, 1997. Google Scholar
Digital Library
- G. J. Holzmann. On limits and possibilities of automated protocol analysis. In PSTV, pages 339--344, 1987. Google Scholar
Digital Library
- G. J. Holzmann and M. Florian. Model checking with bounded context switching. Formal Asp. Comput., 23(3):365--389, 2011. Google Scholar
Cross Ref
- J. Huang and C. Zhang. An efficient static trace simplification technique for debugging concurrent programs. In SAS, pages 163--179, 2011. Google Scholar
Digital Library
- N. Jalbert and K. Sen. A trace simplification technique for effective debugging of concurrent programs. In FSE, FSE '10, pages 57--66, 2010. Google Scholar
Digital Library
- N. Jalbert, C. Pereira, G. Pokam, and K. Sen. RADBench: a concurrency bug benchmark suite. In HotPar, pages 1--6, 2011. Google Scholar
Digital Library
- B. Kasikci, C. Zamfir, and G. Candea. Data races vs. data race bugs: telling the difference with Portend. In ASPLOS, pages 185--198, 2012. Google Scholar
Digital Library
- B. Krena, Z. Letko, T. Vojnar, and S. Ur. A platform for search-based testing of concurrent software. In PADTAD, pages 48--58, 2010. Google Scholar
Digital Library
- A. Lal and T. W. Reps. Reducing concurrent analysis under a context bound to sequential analysis. Formal Methods in System Design, 35(1):73--97, 2009. Google Scholar
Digital Library
- B. Lewis and D. J. Berg. Multithreaded programming with Pthreads. Prentice-Hall, 1998. Google Scholar
Digital Library
- C.-K. Luk et al. Pin: building customized program analysis tools with dynamic instrumentation. In PLDI, pages 190--200, 2005. Google Scholar
Digital Library
- M. Musuvathi and S. Qadeer. Iterative context bounding for systematic testing of multithreaded programs. In PLDI, pages 446--455, 2007. Google Scholar
Digital Library
- M. Musuvathi and S. Qadeer. Partial-order reduction for context-bounded state exploration. Technical Report MSR- TR-2007-12, Microsoft Research, 2007.Google Scholar
- M. Musuvathi and S. Qadeer. Fair stateless model checking. In PLDI, pages 362--371, 2008. Google Scholar
Digital Library
- M. Musuvathi et al. Finding and reproducing Heisenbugs in concurrent programs. In OSDI, pages 267--280, 2008. Google Scholar
Digital Library
- S. Nagarakatte, S. Burckhardt, M. M. Martin, and M. Musu- vathi. Multicore acceleration of priority-based schedulers for concurrency bug detection. In PLDI, pages 543--554, 2012. Google Scholar
Digital Library
- S. Narayanasamy et al. Automatically classifying benign and harmful data races using replay analysis. In PLDI, pages 22--31, 2007. Google Scholar
Digital Library
- S. Park, S. Lu, and Y. Zhou. CTrigger: exposing atomicity violation bugs from their hiding places. In ASPLOS, pages 25--36, 2009. Google Scholar
Digital Library
- K. Sen. Race directed random testing of concurrent programs. In PLDI, pages 11--21, 2008. Google Scholar
Digital Library
- H. Sutter and J. Larus. Software and the concurrency revolution. ACM Queue, 3(7):54--62, 2005. Google Scholar
Digital Library
- C. Wang, M. Said, and A. Gupta. Coverage guided systematic concurrency testing. In ICSE, pages 221--230, 2011. Google Scholar
Digital Library
- S. C. Woo et al. The SPLASH-2 programs: characterization and methodological considerations. In ISCA, pages 24--36, 1995. Google Scholar
Digital Library
- Y. Yang, X. Chen, and G. Gopalakrishnan. Inspect: A run- time model checker for multithreaded C programs. Technical Report UUCS-08-004, University of Utah, 2008.Google Scholar
- J. Yu and S. Narayanasamy. A case for an interleaving constrained shared-memory multi-processor. In ISCA, pages 325--336, 2009. Google Scholar
Digital Library
- J. Yu, S. Narayanasamy, C. Pereira, and G. Pokam. Maple: a coverage-driven testing tool for multithreaded programs. In OOPSLA, pages 485--502, 2012. Google Scholar
Digital Library
Index Terms
Concurrency testing using schedule bounding: an empirical study
Recommendations
Concurrency testing using schedule bounding: an empirical study
PPoPP '14: Proceedings of the 19th ACM SIGPLAN symposium on Principles and practice of parallel programmingWe present the first independent empirical study on schedule bounding techniques for systematic concurrency testing (SCT). We have gathered 52 buggy concurrent software benchmarks, drawn from public code bases, which we call SCTBench. We applied a ...
Concurrency Testing Using Controlled Schedulers: An Empirical Study
Special Issue on PPOPP 2014We present an independent empirical study on concurrency testing using controlled schedulers. We have gathered 49 buggy concurrent software benchmarks, drawn from public code bases, which we call SCTBench. We applied a modified version of an existing ...
Déjà Fu: a concurrency testing library for Haskell
Haskell '15Systematic concurrency testing (SCT) is an approach to testing potentially nondeterministic concurrent programs. SCT avoids potentially unrepeatable results that may arise from unit testing concurrent programs. It seems to have received little ...







Comments