Abstract
Bounded exhaustive testing is an effective methodology for detecting bugs in a wide range of applications. A well-known approach for bounded exhaustive testing is Korat. It generates all test inputs, up to a given small size, based on a formal specification that is written as an executable predicate and characterizes properties of desired inputs. Korat uses the predicate's executions on candidate inputs to implement a backtracking search based on pruning to systematically explore the space of all possible inputs and generate only those that satisfy the specification.
This paper presents a novel approach for speeding up test generation for bounded exhaustive testing using Korat. The novelty of our approach is two-fold. One, we introduce a new technique for writing the specification predicate based on an abstract representation of candidate inputs, so that the predicate executes directly on these abstract structures and each execution has a lower cost. Two, we use the abstract representation as the basis to define the first technique for utilizing GPUs for systematic test generation using executable predicates. Moreover, we present a suite of optimizations that enable effective utilization of the computational resources offered by modern GPUs. We use our prototype tool KoratG to experimentally evaluate our approach using a suite of 7 data structures that were used in prior studies on bounded exhaustive testing. Our results show that our abstract representation can speed up test generation by 5.68 times on a standard CPU, while execution on a GPU speeds up the execution, on average, by 17.46 times.
- Paul Ammann and Jeff Offutt. 2008. Introduction to Software Testing. Cambridge University Press. Google Scholar
Cross Ref
- Tal Ben-Nun, Michael Sutton, Sreepathi Pai, and Keshav Pingali. 2017. Groute: An Asynchronous Multi-GPU Programming Model for Irregular Computations. In Symposium on Principles and Practice of Parallel Programming. 235–248. Google Scholar
Digital Library
- Antonia Bertolino. 2007. Software Testing Research: Achievements, Challenges, Dreams. In Future of Software Engineering. 85–103.Google Scholar
- Adam Betts, Nathan Chong, Alastair F. Donaldson, Jeroen Ketema, Shaz Qadeer, Paul Thomson, and John Wickerson. 2015. The Design and Implementation of a Verification Technique for GPU Kernels. ACM Trans. Program. Lang. Syst. 37, 3 (2015), 10:1–10:49.Google Scholar
Digital Library
- Chandrasekhar Boyapati, Sarfraz Khurshid, and Darko Marinov. 2002. Korat: Automated testing based on Java predicates. In International Symposium on Software Testing and Analysis. 123–133. Google Scholar
Digital Library
- Michael Boyer, Kevin Skadron, and Westley Weimer. 2008. Automated dynamic analysis of CUDA programs. In Workshop on Software Tools for MultiCore Systems .Google Scholar
- Federico Campeotto, Alessandro Palù, Agostino Dovier, Ferdinando Fioretto, and Enrico Pontelli. 2014. Exploring the Use of GPUs in Constraint Solving. In International Symposium on Practical Aspects of Declarative Languages. 152–167. Google Scholar
Digital Library
- Alessandro Dal Palù, Agostino Dovier, Andrea Formisano, and Enrico Pontelli. 2015. [email protected]: SAT solving on GPUs. Journal of Experimental & Theoretical Artificial Intelligence 27, 3 (2015), 293–316. Google Scholar
Cross Ref
- Brett Daniel, Danny Dig, Kely Garcia, and Darko Marinov. 2007. Automated Testing of Refactoring Engines. In International Symposium on Foundations of Software Engineering . 185–194. Google Scholar
Digital Library
- Peter J Denning. 1968. Thrashing: Its causes and prevention. In Proceedings of the December 9-11, 1968, fall joint computer conference, part I . ACM, 915–922.Google Scholar
- Nima Dini. 2016. MKorat: A Novel Approach for Memoizing the Korat Search and Some Potential Applications. Master’s thesis. University of Texas at Austin.Google Scholar
- Nima Dini, Cagdas Yelen, and Sarfraz Khurshid. 2017. Optimizing Parallel Korat Using Invalid Ranges. In International SPIN Symposium on Model Checking of Software . 182–191. Google Scholar
Digital Library
- A. G. Duncan and J. S. Hutchison. 1981. Using Attributed Grammars to Test Designs and Implementations. In International Conference on Software Engineering . 170–178.Google Scholar
- Jianbin Fang, Ana Lucia Varbanescu, and Henk Sips. 2011. A Comprehensive Performance Comparison of CUDA and OpenCL. In International Conference on Parallel Processing. 216–225. Google Scholar
Digital Library
- Gordon Fraser and Andrea Arcuri. 2013. Whole Test Suite Generation. Transactions on Software Engineering 39, 2 (2013), 276–291. Google Scholar
Digital Library
- Milos Gligoric, Tihomir Gvero, Vilas Jagannath, Sarfraz Khurshid, Viktor Kuncak, and Darko Marinov. 2010. Test generation through programming in UDITA. In International Conference on Software Engineering. 225–234. Google Scholar
Digital Library
- Daniel Jackson. 2006. Software Abstractions: Logic, language, and analysis. MIT Press.Google Scholar
Digital Library
- Khronos Group. 2017. OpenCL - The Open Standard for Parallel Programming of Heterogeneous Systems. (2017). https://www.khronos.org/opencl .Google Scholar
- Korat Home Page 2017. (2017). http://korat.sourceforge.net/index.html .Google Scholar
- Milind Kulkarni, Keshav Pingali, Bruce Walter, Ganesh Ramanarayanan, Kavita Bala, and L Paul Chew. 2007. Optimistic parallelism requires abstractions. ACM SIGPLAN Notices 42, 6 (2007), 211–222. Google Scholar
Digital Library
- Ivan Kuraj, Viktor Kuncak, and Daniel Jackson. 2015. Programming with enumerable sets of structures. In Conference on Object-Oriented Programming, Systems, Languages, and Applications . 37–56. Google Scholar
Digital Library
- Victor W. Lee, Changkyu Kim, Jatin Chhugani, Michael Deisher, Daehyun Kim, Anthony D. Nguyen, Nadathur Satish, Mikhail Smelyanskiy, Srinivas Chennupaty, Per Hammarlund, Ronak Singhal, and Pradeep Dubey. 2010. Debunking the 100X GPU vs. CPU Myth: An Evaluation of Throughput Computing on CPU and GPU. In International Symposium on Computer Architecture . 451–460. Google Scholar
Digital Library
- Alan Leung, Manish Gupta, Yuvraj Agarwal, Rajesh Gupta, Ranjit Jhala, and Sorin Lerner. 2012. Verifying GPU kernels by test amplification. ACM SIGPLAN Notices 47, 6 (2012), 383–394. Google Scholar
Digital Library
- Guodong Li and Ganesh Gopalakrishnan. 2010. Scalable SMT-based verification of GPU kernel functions. In International Symposium on Foundations of Software Engineering . 187–196. Google Scholar
Digital Library
- Barbara Liskov and John Guttag. 2000. Program Development in Java: Abstraction, Specification, and Object-Oriented Design (1st ed.). Addison-Wesley Longman Publishing Co., Inc.Google Scholar
Digital Library
- Darko Marinov and Sarfraz Khurshid. 2001. TestEra: A Novel Framework for Automated Testing of Java Programs. In Automated Software Engineering . 22–31. Google Scholar
Cross Ref
- Phil McMinn. 2011. Search-Based Software Testing: Past, Present and Future. In International Conference on Software Testing, Verification and Validation Workshops . 153–163.Google Scholar
- Mario Mendez-Lojo, Martin Burtscher, and Keshav Pingali. 2012. A GPU Implementation of Inclusion-based Points-to Analysis. In Symposium on Principles and Practice of Parallel Programming. 107–116. Google Scholar
Digital Library
- Duane Merrill, Michael Garland, and Andrew Grimshaw. 2015. High-Performance and Scalable GPU Graph Traversal. ACM Trans. Parallel Comput. 1, 2 (2015), 30. Google Scholar
Digital Library
- Aleksandar Milicevic, Sasa Misailovic, Darko Marinov, and Sarfraz Khurshid. 2007. Korat: A Tool for Generating Structurally Complex Test Inputs. In International Conference on Software Engineering, Demo. 771–774. Google Scholar
Digital Library
- Sasa Misailovic, Aleksandar Milicevic, Nemanja Petrovic, Sarfraz Khurshid, and Darko Marinov. 2007. Parallel Test Generation and Execution with Korat. In International Symposium on Foundations of Software Engineering. 135–144. Google Scholar
Digital Library
- Razieh Nokhbeh Zaeem and Sarfraz Khurshid. 2012. Test input generation using dynamic programming. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering . ACM, 34.Google Scholar
Digital Library
- NVIDIA. September 2016. CUDA C Programming Guide v8.0. NVIDIA.Google Scholar
- Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. 2007. Feedback-Directed Random Test Generation. In International Conference on Software Engineering. 75–84. Google Scholar
Digital Library
- Sreepathi Pai and Keshav Pingali. 2016. A Compiler for Throughput Optimization of Graph Algorithms on GPUs. In International Conference on Object-Oriented Programming, Systems, Languages, and Applications . 1–19. Google Scholar
Digital Library
- Pablo Ponzio, Nazareno Aguirre, Marcelo F. Frias, and Willem Visser. 2016. Field-exhaustive Testing. In International Symposium on Foundations of Software Engineering . 908–919. Google Scholar
Digital Library
- Tarun Prabhu, Shreyas Ramalingam, Matthew Might, and Mary Hall. 2011. EigenCFA: Accelerating Flow Analysis with GPUs. In Symposium on Principles of Programming Languages. 511–522. Google Scholar
Digital Library
- Randoop Home Page 2017. (2017). https://github.com/randoop/randoop .Google Scholar
- Nicolás Rosner, Valeria Bengolea, Pablo Ponzio, Shadi Abdul Khalek, Nazareno Aguirre, Marcelo F. Frias, and Sarfraz Khurshid. 2014. Bounded Exhaustive Test Input Generation from Hybrid Invariants. In Conference on Object-Oriented Programming, Systems, Languages, and Applications . 655–674. Google Scholar
Digital Library
- Savija T. V. 2011. Oracle JRockit Introduction, Release R28. http://docs.oracle.com/cd/E15289_01/doc.40/e15058/underst_jit. htm .Google Scholar
- Rohan Sharma, Milos Gligoric, Andrea Arcuri, Gordon Fraser, and Darko Marinov. 2011. Testing container classes: Random or Systematic?. In Fundamental Approaches to Software Engineering. 262–277.Google Scholar
- Junaid Haroon Siddiqui and Sarfraz Khurshid. 2009. PKorat: Parallel Generation of Structurally Complex Test Inputs. In International Conference on Software Testing, Verification, and Validation . 250–259. Google Scholar
Digital Library
- Erich Strohmaier, Jack Dongarra, Horst Simon, and Martin Meuer. November 2016. The Top500 List Statistics – Accelerator/Co-processor Share. (November 2016). https://www.top500.org/statistics/list/ .Google Scholar
- Kevin Sullivan, Jinlin Yang, David Coppit, Sarfraz Khurshid, and Daniel Jackson. 2004. Software Assurance by Bounded Exhaustive Testing. In International Symposium on Software Testing and Analysis. 133–142. Google Scholar
Digital Library
- Kuo-Chung Tai and Yu Lei. 2002. A test generation strategy for pairwise testing. Transactions on Software Engineering 28, 1 (2002), 109–111. Google Scholar
Digital Library
- Paolo Tonella. 2004. Evolutionary Testing of Classes. In International Symposium on Software Testing and Analysis. 119–128. Google Scholar
Digital Library
- Willem Visser, Corina S. Pˇasˇareanu, and Radek Pelánek. 2006. Test Input Generation for Java Containers Using State Matching. In International Symposium on Software Testing and Analysis. 37–48. Google Scholar
Digital Library
- Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and understanding bugs in C compilers. In Conference on Programming Language Design and Implementation . 283–294. Google Scholar
Digital Library
- Mai Zheng, Vignesh T Ravi, Feng Qin, and Gagan Agrawal. 2011. GRace: A low-overhead mechanism for detecting data races in GPU programs. In ACM SIGPLAN Notices, Vol. 46. 135–146.Google Scholar
Digital Library
Index Terms
Bounded exhaustive test-input generation on GPUs
Recommendations
RepOK-based reduction of bounded exhaustive testing
While the effectiveness of bounded exhaustive test suites increases as one increases the scope for the bounded exhaustive generation, both the time for test generation and the time for test execution grow exponentially with respect to the scope. In this ...
Software assurance by bounded exhaustive testing
The contribution of this paper is an experiment that shows the potential value of a combination of selective reverse engineering to formal specifications and bounded exhaustive testing to improve the assurance levels of complex software. A key problem ...
Software assurance by bounded exhaustive testing
ISSTA '04: Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysisThe contribution of this paper is an experiment that shows the potential value of a combination of selective reverse engineering to formal specifications and bounded exhaustive testing to improve the assurance levels of complex software. A key problem ...






Comments