skip to main content
research-article
Open Access

Programming and execution models for parallel bounded exhaustive testing

Published:15 October 2021Publication History
Skip Abstract Section

Abstract

Bounded-exhaustive testing (BET), which exercises a program under test for all inputs up to some bounds, is an effective method for detecting software bugs. Systematic property-based testing is a BET approach where developers write test generation programs that describe properties of test inputs. Hybrid test generation programs offer the most expressive way to write desired properties by freely combining declarative filters and imperative generators. However, exploring hybrid test generation programs, to obtain test inputs, is both computationally demanding and challenging to parallelize. We present the first programming and execution models, dubbed Tempo, for parallel exploration of hybrid test generation programs. We describe two different strategies for mapping the computation to parallel hardware and implement them both for GPUs and CPUs. We evaluated Tempo by generating instances of various data structures commonly used for benchmarking in the BET domain. Additionally, we generated CUDA programs to stress test CUDA compilers, finding four bugs confirmed by the developers.

Skip Supplemental Material Section

Supplemental Material

Auxiliary Presentation Video

This is a presentation video of my talk at OOPSLA 2021 on our paper accepted in the research track. The paper is on Bounded-exhaustive testing (BET), which exercises a program under test for all inputs up to some bounds, is an effective method for detecting software bugs. Systematic property-based testing is a BET approach where developers write test generation programs that describe properties of test inputs. Hybrid test generation programs offer the most expressive way to write desired properties by freely combining declarative filters and imperative generators. However, exploring hybrid test generation programs, to obtain test inputs, is both computationally demanding and challenging to parallelize. We present the first programming and execution models, dubbed Tempo, for parallel exploration of hybrid test generation programs. We describe two different strategies for mapping the computation to parallel hardware and implement them both for GPUs and CPUs.

References

  1. Michael Bauer, Sean Treichler, Elliott Slaughter, and Alex Aiken. 2012. Legion: Expressing locality and independence with logical regions. International Conference on High Performance Computing, Networking, Storage and Analysis, 1–11. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Chandrasekhar Boyapati, Sarfraz Khurshid, and Darko Marinov. 2002. Korat: automated testing based on Java predicates. In International Symposium on Software Testing and Analysis. 123–133. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Ahmet Celik, Pengyu Nie, Christopher J. Rossbach, and Milos Gligoric. 2019. Design, Implementation, and Application of GPU-based Java Bytecode Interpreters. In Conference on Object-Oriented Programming, Systems, Languages, and Applications. 177:1–177:28.Google ScholarGoogle Scholar
  4. Ahmet Celik, Sreepathi Pai, Sarfraz Khurshid, and Milos Gligoric. 2017. Bounded Exhaustive Test-Input Generation on GPUs. In Conference on Object-Oriented Programming, Systems, Languages, and Applications. 94:1–94:25.Google ScholarGoogle Scholar
  5. Junjie Chen, Jibesh Patra, Michael Pradel, Yingfei Xiong, Hongyu Zhang, Dan Hao, and Lu Zhang. 2020. A Survey of Compiler Testing. ACM Computing Survey, 53, 1 (2020). Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Artem Chikin, Jose Nelson Amaral, Karim Ali, and Ettore Tiotto. 2019. Toward an Analytical Performance Model to Select between GPU and CPU Execution. In International Parallel and Distributed Processing Symposium Workshops. 353–362.Google ScholarGoogle ScholarCross RefCross Ref
  7. 2020. NVCC. https://llvm.org/docs/CompileCudaWithLLVM.htmlGoogle ScholarGoogle Scholar
  8. 2020. Clang: a C language family frontend for LLVM. https://clang.llvm.org/Google ScholarGoogle Scholar
  9. James Clarkson, Juan Fumero, Michail Papadimitriou, Foivos S. Zakkak, Maria Xekalaki, Christos Kotselidis, and Mikel Luján. 2018. Exploiting High-performance Heterogeneous Hardware for Java Programs Using Graal. In International Conference on Managed Languages & Runtimes. 4:1–4:13. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. 2020. constexpr specifier. https://en.cppreference.com/w/cpp/language/constexprGoogle ScholarGoogle Scholar
  11. 2020. CUDA C++ Programming Guide. https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.htmlGoogle ScholarGoogle Scholar
  12. 2020. CUDA Zone. https://developer.nvidia.com/cuda-zoneGoogle ScholarGoogle Scholar
  13. Brett Daniel, Danny Dig, Kely Garcia, and Darko Marinov. 2007. Automated Testing of Refactoring Engines. In International Symposium on Foundations of Software Engineering. 185–194. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Kyle Dewey, Lawton Nichols, and Ben Hardekopf. 2015. Automated Data Structure Generation: Refuting Common Wisdom. In International Conference on Software Engineering. 32–43. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Kyle Dewey, Jared Roesch, and Ben Hardekopf. 2015. Fuzzing the Rust Typechecker Using CLP. In Automated Software Engineering. 482–493. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Chucky Ellison and Grigore Roşu. 2012. An Executable Formal Semantics of C with Applications. In Symposium on Principles of Programming Languages. 533–544. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Juliana Franco, Martin Hagelin, Tobias Wrigstad, Sophia Drossopoulou, and Susan Eisenbach. 2017. You Can Have It All: Abstraction and Good Cache Performance. In International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software. 148–167. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Juan Pablo Galeotti, Nicolás Rosner, Carlos Gustavo López Pombo, and Marcelo Fabian Frias. 2010. Analysis of Invariants for Efficient Bounded Verification. In International Symposium on Software Testing and Analysis. 25–36. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. 2020. GCC, the GNU Compiler Collection. https://gcc.gnu.org/Google ScholarGoogle Scholar
  20. Milos Gligoric, Tihomir Gvero, Vilas Jagannath, Sarfraz Khurshid, Viktor Kuncak, and Darko Marinov. 2010. Test generation through programming in UDITA. In International Conference on Software Engineering. 225–234. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Daniel Jackson and Craig A. Damon. 1996. Elements of Style: Analyzing a Software Design Feature with a Counterexample Detector. In International Symposium on Software Testing and Analysis. 239–249. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Timotej Kapus and Cristian Cadar. 2017. Automatic Testing of Symbolic Execution Engines via Program Generation and Differential Testing. In Automated Software Engineering. 590–600. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Ivan Kuraj, Viktor Kuncak, and Daniel Jackson. 2015. Programming with enumerable sets of structures. In Conference on Object-Oriented Programming, Systems, Languages, and Applications. 37–56. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Christopher Lidbury, Andrei Lascu, Nathan Chong, and Alastair F. Donaldson. 2015. Many-Core Compiler Fuzzing. In Conference on Programming Language Design and Implementation. 65–76. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Deepak Majeti, Kuldeep S. Meel, Rajkishore Barik, and Vivek Sarkar. 2016. Automatic Data Layout Generation and Kernel Mapping for CPU+GPU Architectures. In Proceedings of the 25th International Conference on Compiler Construction. 240–250. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. William M. McKeeman. 1998. Differential Testing for Software. Digital Technical Journal, 10, 1 (1998), 100–107.Google ScholarGoogle Scholar
  27. José Meseguer and Grigore Roşu. 2013. The Rewriting Logic Semantics Project: A Progress Report. Information and Computation, 231 (2013), 38–69. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Sasa Misailovic, Aleksandar Milicevic, Nemanja Petrovic, Sarfraz Khurshid, and Darko Marinov. 2007. Parallel Test Generation and Execution with Korat. In International Symposium on Foundations of Software Engineering. 135–144. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. 2020. Nsight Compute CLI. https://docs.nvidia.com/nsight-compute/NsightComputeCli/index.htmlGoogle ScholarGoogle Scholar
  30. 2019. NVCC. https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.htmlGoogle ScholarGoogle Scholar
  31. 2020. OpenMPI - A High Performance Message Passing Library. https://www.open-mpi.orgGoogle ScholarGoogle Scholar
  32. 2020. OpenMP. https://www.openmp.orgGoogle ScholarGoogle Scholar
  33. 2020. C++ Operator Precedence. https://en.cppreference.com/w/cpp/language/operator_precedenceGoogle ScholarGoogle Scholar
  34. Hila Peleg, Dan Rasin, and Eran Yahav. 2018. Generating Tests by Example. In Verification, Model Checking, and Abstract Interpretation. 406–429.Google ScholarGoogle Scholar
  35. Talia Ringer, Dan Grossman, Daniel Schwartz-Narbonne, and Serdar Tasiran. 2017. A Solver-Aided Language for Test Input Generation. Proc. ACM Program. Lang., 1, Conference on Object-Oriented Programming, Systems, Languages, and Applications. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Nicolás Rosner, Valeria Bengolea, Pablo Ponzio, Shadi Abdul Khalek, Nazareno Aguirre, Marcelo F. Frias, and Sarfraz Khurshid. 2014. Bounded Exhaustive Test Input Generation from Hybrid Invariants. In Conference on Object-Oriented Programming, Systems, Languages, and Applications. 655–674. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Marija Selakovic, Michael Pradel, Rezwana Karim, and Frank Tip. 2018. Test Generation for Higher-Order Functions in Dynamic Languages. Proc. ACM Program. Lang., 2, Conference on Object-Oriented Programming, Systems, Languages, and Applications (2018). Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Rohan Sharma, Milos Gligoric, Andrea Arcuri, Gordon Fraser, and Darko Marinov. 2011. Testing container classes: Random or Systematic? In Fundamental Approaches to Software Engineering. 262–277. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Rohan Sharma, Milos Gligoric, Vilas Jagannath, and Darko Marinov. 2010. A comparison of constraint-based and sequence-based generation of complex input data structures. In Workshop on Constraints in Software Testing, Verification and Analysis. 337–342. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Aaron Tomb, Stuart Pernsteiner, and Mike Dodds. 2020. Symbolic Testing for C and Rust. In 2020 IEEE Secure Development (SecDev). 33–33.Google ScholarGoogle Scholar
  41. [n. d.]. UDITA Home Page. http://mir.cs.illinois.edu/uditaGoogle ScholarGoogle Scholar
  42. Willem Visser, Corina S. Pǎsǎreanu, and Radek Pelánek. 2006. Test Input Generation for Java Containers Using State Matching. In International Symposium on Software Testing and Analysis. 37–48. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Tao Xie, Darko Marinov, Wolfram Schulte, and David Notkin. 2005. Symstra: A Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution. In Tools and Algorithms for the Construction and Analysis of Systems. Springer Berlin Heidelberg, 365–381. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Vanya Yaneva, Ajitha Rajan, and Christophe Dubach. 2017. Compiler-assisted Test Acceleration on GPUs for Embedded Software. In International Symposium on Software Testing and Analysis. 35–45. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and Understanding Bugs in C Compilers. In Conference on Programming Language Design and Implementation. 283–294. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Andreas Zeller, Rahul Gopinath, Marcel Böhme, Gordon Fraser, and Christian Holler. 2019. The Fuzzing Book. Saarland University.Google ScholarGoogle Scholar
  47. Qirun Zhang, Chengnian Sun, and Zhendong Su. 2017. Skeletal Program Enumeration for Rigorous Compiler Testing. In Conference on Programming Language Design and Implementation. 347–361. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Programming and execution models for parallel bounded exhaustive testing

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Article Metrics

        • Downloads (Last 12 months)75
        • Downloads (Last 6 weeks)6

        Other Metrics

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!