Abstract
Bounded-exhaustive testing (BET), which exercises a program under test for all inputs up to some bounds, is an effective method for detecting software bugs. Systematic property-based testing is a BET approach where developers write test generation programs that describe properties of test inputs. Hybrid test generation programs offer the most expressive way to write desired properties by freely combining declarative filters and imperative generators. However, exploring hybrid test generation programs, to obtain test inputs, is both computationally demanding and challenging to parallelize. We present the first programming and execution models, dubbed Tempo, for parallel exploration of hybrid test generation programs. We describe two different strategies for mapping the computation to parallel hardware and implement them both for GPUs and CPUs. We evaluated Tempo by generating instances of various data structures commonly used for benchmarking in the BET domain. Additionally, we generated CUDA programs to stress test CUDA compilers, finding four bugs confirmed by the developers.
Supplemental Material
- Michael Bauer, Sean Treichler, Elliott Slaughter, and Alex Aiken. 2012. Legion: Expressing locality and independence with logical regions. International Conference on High Performance Computing, Networking, Storage and Analysis, 1–11. Google Scholar
Digital Library
- Chandrasekhar Boyapati, Sarfraz Khurshid, and Darko Marinov. 2002. Korat: automated testing based on Java predicates. In International Symposium on Software Testing and Analysis. 123–133. Google Scholar
Digital Library
- Ahmet Celik, Pengyu Nie, Christopher J. Rossbach, and Milos Gligoric. 2019. Design, Implementation, and Application of GPU-based Java Bytecode Interpreters. In Conference on Object-Oriented Programming, Systems, Languages, and Applications. 177:1–177:28.Google Scholar
- Ahmet Celik, Sreepathi Pai, Sarfraz Khurshid, and Milos Gligoric. 2017. Bounded Exhaustive Test-Input Generation on GPUs. In Conference on Object-Oriented Programming, Systems, Languages, and Applications. 94:1–94:25.Google Scholar
- Junjie Chen, Jibesh Patra, Michael Pradel, Yingfei Xiong, Hongyu Zhang, Dan Hao, and Lu Zhang. 2020. A Survey of Compiler Testing. ACM Computing Survey, 53, 1 (2020). Google Scholar
Digital Library
- Artem Chikin, Jose Nelson Amaral, Karim Ali, and Ettore Tiotto. 2019. Toward an Analytical Performance Model to Select between GPU and CPU Execution. In International Parallel and Distributed Processing Symposium Workshops. 353–362.Google Scholar
Cross Ref
- 2020. NVCC. https://llvm.org/docs/CompileCudaWithLLVM.htmlGoogle Scholar
- 2020. Clang: a C language family frontend for LLVM. https://clang.llvm.org/Google Scholar
- James Clarkson, Juan Fumero, Michail Papadimitriou, Foivos S. Zakkak, Maria Xekalaki, Christos Kotselidis, and Mikel Luján. 2018. Exploiting High-performance Heterogeneous Hardware for Java Programs Using Graal. In International Conference on Managed Languages & Runtimes. 4:1–4:13. Google Scholar
Digital Library
- 2020. constexpr specifier. https://en.cppreference.com/w/cpp/language/constexprGoogle Scholar
- 2020. CUDA C++ Programming Guide. https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.htmlGoogle Scholar
- 2020. CUDA Zone. https://developer.nvidia.com/cuda-zoneGoogle Scholar
- Brett Daniel, Danny Dig, Kely Garcia, and Darko Marinov. 2007. Automated Testing of Refactoring Engines. In International Symposium on Foundations of Software Engineering. 185–194. Google Scholar
Digital Library
- Kyle Dewey, Lawton Nichols, and Ben Hardekopf. 2015. Automated Data Structure Generation: Refuting Common Wisdom. In International Conference on Software Engineering. 32–43. Google Scholar
Digital Library
- Kyle Dewey, Jared Roesch, and Ben Hardekopf. 2015. Fuzzing the Rust Typechecker Using CLP. In Automated Software Engineering. 482–493. Google Scholar
Digital Library
- Chucky Ellison and Grigore Roşu. 2012. An Executable Formal Semantics of C with Applications. In Symposium on Principles of Programming Languages. 533–544. Google Scholar
Digital Library
- Juliana Franco, Martin Hagelin, Tobias Wrigstad, Sophia Drossopoulou, and Susan Eisenbach. 2017. You Can Have It All: Abstraction and Good Cache Performance. In International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software. 148–167. Google Scholar
Digital Library
- Juan Pablo Galeotti, Nicolás Rosner, Carlos Gustavo López Pombo, and Marcelo Fabian Frias. 2010. Analysis of Invariants for Efficient Bounded Verification. In International Symposium on Software Testing and Analysis. 25–36. Google Scholar
Digital Library
- 2020. GCC, the GNU Compiler Collection. https://gcc.gnu.org/Google Scholar
- Milos Gligoric, Tihomir Gvero, Vilas Jagannath, Sarfraz Khurshid, Viktor Kuncak, and Darko Marinov. 2010. Test generation through programming in UDITA. In International Conference on Software Engineering. 225–234. Google Scholar
Digital Library
- Daniel Jackson and Craig A. Damon. 1996. Elements of Style: Analyzing a Software Design Feature with a Counterexample Detector. In International Symposium on Software Testing and Analysis. 239–249. Google Scholar
Digital Library
- Timotej Kapus and Cristian Cadar. 2017. Automatic Testing of Symbolic Execution Engines via Program Generation and Differential Testing. In Automated Software Engineering. 590–600. Google Scholar
Digital Library
- Ivan Kuraj, Viktor Kuncak, and Daniel Jackson. 2015. Programming with enumerable sets of structures. In Conference on Object-Oriented Programming, Systems, Languages, and Applications. 37–56. Google Scholar
Digital Library
- Christopher Lidbury, Andrei Lascu, Nathan Chong, and Alastair F. Donaldson. 2015. Many-Core Compiler Fuzzing. In Conference on Programming Language Design and Implementation. 65–76. Google Scholar
Digital Library
- Deepak Majeti, Kuldeep S. Meel, Rajkishore Barik, and Vivek Sarkar. 2016. Automatic Data Layout Generation and Kernel Mapping for CPU+GPU Architectures. In Proceedings of the 25th International Conference on Compiler Construction. 240–250. Google Scholar
Digital Library
- William M. McKeeman. 1998. Differential Testing for Software. Digital Technical Journal, 10, 1 (1998), 100–107.Google Scholar
- José Meseguer and Grigore Roşu. 2013. The Rewriting Logic Semantics Project: A Progress Report. Information and Computation, 231 (2013), 38–69. Google Scholar
Digital Library
- Sasa Misailovic, Aleksandar Milicevic, Nemanja Petrovic, Sarfraz Khurshid, and Darko Marinov. 2007. Parallel Test Generation and Execution with Korat. In International Symposium on Foundations of Software Engineering. 135–144. Google Scholar
Digital Library
- 2020. Nsight Compute CLI. https://docs.nvidia.com/nsight-compute/NsightComputeCli/index.htmlGoogle Scholar
- 2019. NVCC. https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.htmlGoogle Scholar
- 2020. OpenMPI - A High Performance Message Passing Library. https://www.open-mpi.orgGoogle Scholar
- 2020. OpenMP. https://www.openmp.orgGoogle Scholar
- 2020. C++ Operator Precedence. https://en.cppreference.com/w/cpp/language/operator_precedenceGoogle Scholar
- Hila Peleg, Dan Rasin, and Eran Yahav. 2018. Generating Tests by Example. In Verification, Model Checking, and Abstract Interpretation. 406–429.Google Scholar
- Talia Ringer, Dan Grossman, Daniel Schwartz-Narbonne, and Serdar Tasiran. 2017. A Solver-Aided Language for Test Input Generation. Proc. ACM Program. Lang., 1, Conference on Object-Oriented Programming, Systems, Languages, and Applications. Google Scholar
Digital Library
- Nicolás Rosner, Valeria Bengolea, Pablo Ponzio, Shadi Abdul Khalek, Nazareno Aguirre, Marcelo F. Frias, and Sarfraz Khurshid. 2014. Bounded Exhaustive Test Input Generation from Hybrid Invariants. In Conference on Object-Oriented Programming, Systems, Languages, and Applications. 655–674. Google Scholar
Digital Library
- Marija Selakovic, Michael Pradel, Rezwana Karim, and Frank Tip. 2018. Test Generation for Higher-Order Functions in Dynamic Languages. Proc. ACM Program. Lang., 2, Conference on Object-Oriented Programming, Systems, Languages, and Applications (2018). Google Scholar
Digital Library
- Rohan Sharma, Milos Gligoric, Andrea Arcuri, Gordon Fraser, and Darko Marinov. 2011. Testing container classes: Random or Systematic? In Fundamental Approaches to Software Engineering. 262–277. Google Scholar
Digital Library
- Rohan Sharma, Milos Gligoric, Vilas Jagannath, and Darko Marinov. 2010. A comparison of constraint-based and sequence-based generation of complex input data structures. In Workshop on Constraints in Software Testing, Verification and Analysis. 337–342. Google Scholar
Digital Library
- Aaron Tomb, Stuart Pernsteiner, and Mike Dodds. 2020. Symbolic Testing for C and Rust. In 2020 IEEE Secure Development (SecDev). 33–33.Google Scholar
- [n. d.]. UDITA Home Page. http://mir.cs.illinois.edu/uditaGoogle Scholar
- Willem Visser, Corina S. Pǎsǎreanu, and Radek Pelánek. 2006. Test Input Generation for Java Containers Using State Matching. In International Symposium on Software Testing and Analysis. 37–48. Google Scholar
Digital Library
- Tao Xie, Darko Marinov, Wolfram Schulte, and David Notkin. 2005. Symstra: A Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution. In Tools and Algorithms for the Construction and Analysis of Systems. Springer Berlin Heidelberg, 365–381. Google Scholar
Digital Library
- Vanya Yaneva, Ajitha Rajan, and Christophe Dubach. 2017. Compiler-assisted Test Acceleration on GPUs for Embedded Software. In International Symposium on Software Testing and Analysis. 35–45. Google Scholar
Digital Library
- Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and Understanding Bugs in C Compilers. In Conference on Programming Language Design and Implementation. 283–294. Google Scholar
Digital Library
- Andreas Zeller, Rahul Gopinath, Marcel Böhme, Gordon Fraser, and Christian Holler. 2019. The Fuzzing Book. Saarland University.Google Scholar
- Qirun Zhang, Chengnian Sun, and Zhendong Su. 2017. Skeletal Program Enumeration for Rigorous Compiler Testing. In Conference on Programming Language Design and Implementation. 347–361. Google Scholar
Digital Library
Index Terms
Programming and execution models for parallel bounded exhaustive testing
Recommendations
RepOK-based reduction of bounded exhaustive testing
While the effectiveness of bounded exhaustive test suites increases as one increases the scope for the bounded exhaustive generation, both the time for test generation and the time for test execution grow exponentially with respect to the scope. In this ...
Software assurance by bounded exhaustive testing
The contribution of this paper is an experiment that shows the potential value of a combination of selective reverse engineering to formal specifications and bounded exhaustive testing to improve the assurance levels of complex software. A key problem ...
Using coverage criteria on RepOK to reduce bounded-exhaustive test suites
TAP'12: Proceedings of the 6th international conference on Tests and ProofsBounded-exhaustive exploration of test case candidates is a commonly employed approach for test generation in some contexts. Even when small bounds are used for test generation, executing the obtained tests may become prohibitive, despite the time for ...






Comments