skip to main content
research-article
Open Access

Bounded exhaustive test-input generation on GPUs

Published:12 October 2017Publication History
Skip Abstract Section

Abstract

Bounded exhaustive testing is an effective methodology for detecting bugs in a wide range of applications. A well-known approach for bounded exhaustive testing is Korat. It generates all test inputs, up to a given small size, based on a formal specification that is written as an executable predicate and characterizes properties of desired inputs. Korat uses the predicate's executions on candidate inputs to implement a backtracking search based on pruning to systematically explore the space of all possible inputs and generate only those that satisfy the specification.

This paper presents a novel approach for speeding up test generation for bounded exhaustive testing using Korat. The novelty of our approach is two-fold. One, we introduce a new technique for writing the specification predicate based on an abstract representation of candidate inputs, so that the predicate executes directly on these abstract structures and each execution has a lower cost. Two, we use the abstract representation as the basis to define the first technique for utilizing GPUs for systematic test generation using executable predicates. Moreover, we present a suite of optimizations that enable effective utilization of the computational resources offered by modern GPUs. We use our prototype tool KoratG to experimentally evaluate our approach using a suite of 7 data structures that were used in prior studies on bounded exhaustive testing. Our results show that our abstract representation can speed up test generation by 5.68 times on a standard CPU, while execution on a GPU speeds up the execution, on average, by 17.46 times.

References

  1. Paul Ammann and Jeff Offutt. 2008. Introduction to Software Testing. Cambridge University Press. Google ScholarGoogle ScholarCross RefCross Ref
  2. Tal Ben-Nun, Michael Sutton, Sreepathi Pai, and Keshav Pingali. 2017. Groute: An Asynchronous Multi-GPU Programming Model for Irregular Computations. In Symposium on Principles and Practice of Parallel Programming. 235–248. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Antonia Bertolino. 2007. Software Testing Research: Achievements, Challenges, Dreams. In Future of Software Engineering. 85–103.Google ScholarGoogle Scholar
  4. Adam Betts, Nathan Chong, Alastair F. Donaldson, Jeroen Ketema, Shaz Qadeer, Paul Thomson, and John Wickerson. 2015. The Design and Implementation of a Verification Technique for GPU Kernels. ACM Trans. Program. Lang. Syst. 37, 3 (2015), 10:1–10:49.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Chandrasekhar Boyapati, Sarfraz Khurshid, and Darko Marinov. 2002. Korat: Automated testing based on Java predicates. In International Symposium on Software Testing and Analysis. 123–133. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Michael Boyer, Kevin Skadron, and Westley Weimer. 2008. Automated dynamic analysis of CUDA programs. In Workshop on Software Tools for MultiCore Systems .Google ScholarGoogle Scholar
  7. Federico Campeotto, Alessandro Palù, Agostino Dovier, Ferdinando Fioretto, and Enrico Pontelli. 2014. Exploring the Use of GPUs in Constraint Solving. In International Symposium on Practical Aspects of Declarative Languages. 152–167. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Alessandro Dal Palù, Agostino Dovier, Andrea Formisano, and Enrico Pontelli. 2015. [email protected]: SAT solving on GPUs. Journal of Experimental & Theoretical Artificial Intelligence 27, 3 (2015), 293–316. Google ScholarGoogle ScholarCross RefCross Ref
  9. Brett Daniel, Danny Dig, Kely Garcia, and Darko Marinov. 2007. Automated Testing of Refactoring Engines. In International Symposium on Foundations of Software Engineering . 185–194. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Peter J Denning. 1968. Thrashing: Its causes and prevention. In Proceedings of the December 9-11, 1968, fall joint computer conference, part I . ACM, 915–922.Google ScholarGoogle Scholar
  11. Nima Dini. 2016. MKorat: A Novel Approach for Memoizing the Korat Search and Some Potential Applications. Master’s thesis. University of Texas at Austin.Google ScholarGoogle Scholar
  12. Nima Dini, Cagdas Yelen, and Sarfraz Khurshid. 2017. Optimizing Parallel Korat Using Invalid Ranges. In International SPIN Symposium on Model Checking of Software . 182–191. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. A. G. Duncan and J. S. Hutchison. 1981. Using Attributed Grammars to Test Designs and Implementations. In International Conference on Software Engineering . 170–178.Google ScholarGoogle Scholar
  14. Jianbin Fang, Ana Lucia Varbanescu, and Henk Sips. 2011. A Comprehensive Performance Comparison of CUDA and OpenCL. In International Conference on Parallel Processing. 216–225. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Gordon Fraser and Andrea Arcuri. 2013. Whole Test Suite Generation. Transactions on Software Engineering 39, 2 (2013), 276–291. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Milos Gligoric, Tihomir Gvero, Vilas Jagannath, Sarfraz Khurshid, Viktor Kuncak, and Darko Marinov. 2010. Test generation through programming in UDITA. In International Conference on Software Engineering. 225–234. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Daniel Jackson. 2006. Software Abstractions: Logic, language, and analysis. MIT Press.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Khronos Group. 2017. OpenCL - The Open Standard for Parallel Programming of Heterogeneous Systems. (2017). https://www.khronos.org/opencl .Google ScholarGoogle Scholar
  19. Korat Home Page 2017. (2017). http://korat.sourceforge.net/index.html .Google ScholarGoogle Scholar
  20. Milind Kulkarni, Keshav Pingali, Bruce Walter, Ganesh Ramanarayanan, Kavita Bala, and L Paul Chew. 2007. Optimistic parallelism requires abstractions. ACM SIGPLAN Notices 42, 6 (2007), 211–222. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Ivan Kuraj, Viktor Kuncak, and Daniel Jackson. 2015. Programming with enumerable sets of structures. In Conference on Object-Oriented Programming, Systems, Languages, and Applications . 37–56. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Victor W. Lee, Changkyu Kim, Jatin Chhugani, Michael Deisher, Daehyun Kim, Anthony D. Nguyen, Nadathur Satish, Mikhail Smelyanskiy, Srinivas Chennupaty, Per Hammarlund, Ronak Singhal, and Pradeep Dubey. 2010. Debunking the 100X GPU vs. CPU Myth: An Evaluation of Throughput Computing on CPU and GPU. In International Symposium on Computer Architecture . 451–460. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Alan Leung, Manish Gupta, Yuvraj Agarwal, Rajesh Gupta, Ranjit Jhala, and Sorin Lerner. 2012. Verifying GPU kernels by test amplification. ACM SIGPLAN Notices 47, 6 (2012), 383–394. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Guodong Li and Ganesh Gopalakrishnan. 2010. Scalable SMT-based verification of GPU kernel functions. In International Symposium on Foundations of Software Engineering . 187–196. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Barbara Liskov and John Guttag. 2000. Program Development in Java: Abstraction, Specification, and Object-Oriented Design (1st ed.). Addison-Wesley Longman Publishing Co., Inc.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Darko Marinov and Sarfraz Khurshid. 2001. TestEra: A Novel Framework for Automated Testing of Java Programs. In Automated Software Engineering . 22–31. Google ScholarGoogle ScholarCross RefCross Ref
  27. Phil McMinn. 2011. Search-Based Software Testing: Past, Present and Future. In International Conference on Software Testing, Verification and Validation Workshops . 153–163.Google ScholarGoogle Scholar
  28. Mario Mendez-Lojo, Martin Burtscher, and Keshav Pingali. 2012. A GPU Implementation of Inclusion-based Points-to Analysis. In Symposium on Principles and Practice of Parallel Programming. 107–116. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Duane Merrill, Michael Garland, and Andrew Grimshaw. 2015. High-Performance and Scalable GPU Graph Traversal. ACM Trans. Parallel Comput. 1, 2 (2015), 30. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Aleksandar Milicevic, Sasa Misailovic, Darko Marinov, and Sarfraz Khurshid. 2007. Korat: A Tool for Generating Structurally Complex Test Inputs. In International Conference on Software Engineering, Demo. 771–774. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Sasa Misailovic, Aleksandar Milicevic, Nemanja Petrovic, Sarfraz Khurshid, and Darko Marinov. 2007. Parallel Test Generation and Execution with Korat. In International Symposium on Foundations of Software Engineering. 135–144. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Razieh Nokhbeh Zaeem and Sarfraz Khurshid. 2012. Test input generation using dynamic programming. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering . ACM, 34.Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. NVIDIA. September 2016. CUDA C Programming Guide v8.0. NVIDIA.Google ScholarGoogle Scholar
  34. Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. 2007. Feedback-Directed Random Test Generation. In International Conference on Software Engineering. 75–84. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Sreepathi Pai and Keshav Pingali. 2016. A Compiler for Throughput Optimization of Graph Algorithms on GPUs. In International Conference on Object-Oriented Programming, Systems, Languages, and Applications . 1–19. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Pablo Ponzio, Nazareno Aguirre, Marcelo F. Frias, and Willem Visser. 2016. Field-exhaustive Testing. In International Symposium on Foundations of Software Engineering . 908–919. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Tarun Prabhu, Shreyas Ramalingam, Matthew Might, and Mary Hall. 2011. EigenCFA: Accelerating Flow Analysis with GPUs. In Symposium on Principles of Programming Languages. 511–522. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Randoop Home Page 2017. (2017). https://github.com/randoop/randoop .Google ScholarGoogle Scholar
  39. Nicolás Rosner, Valeria Bengolea, Pablo Ponzio, Shadi Abdul Khalek, Nazareno Aguirre, Marcelo F. Frias, and Sarfraz Khurshid. 2014. Bounded Exhaustive Test Input Generation from Hybrid Invariants. In Conference on Object-Oriented Programming, Systems, Languages, and Applications . 655–674. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Savija T. V. 2011. Oracle JRockit Introduction, Release R28. http://docs.oracle.com/cd/E15289_01/doc.40/e15058/underst_jit. htm .Google ScholarGoogle Scholar
  41. Rohan Sharma, Milos Gligoric, Andrea Arcuri, Gordon Fraser, and Darko Marinov. 2011. Testing container classes: Random or Systematic?. In Fundamental Approaches to Software Engineering. 262–277.Google ScholarGoogle Scholar
  42. Junaid Haroon Siddiqui and Sarfraz Khurshid. 2009. PKorat: Parallel Generation of Structurally Complex Test Inputs. In International Conference on Software Testing, Verification, and Validation . 250–259. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Erich Strohmaier, Jack Dongarra, Horst Simon, and Martin Meuer. November 2016. The Top500 List Statistics – Accelerator/Co-processor Share. (November 2016). https://www.top500.org/statistics/list/ .Google ScholarGoogle Scholar
  44. Kevin Sullivan, Jinlin Yang, David Coppit, Sarfraz Khurshid, and Daniel Jackson. 2004. Software Assurance by Bounded Exhaustive Testing. In International Symposium on Software Testing and Analysis. 133–142. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Kuo-Chung Tai and Yu Lei. 2002. A test generation strategy for pairwise testing. Transactions on Software Engineering 28, 1 (2002), 109–111. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Paolo Tonella. 2004. Evolutionary Testing of Classes. In International Symposium on Software Testing and Analysis. 119–128. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Willem Visser, Corina S. Pˇasˇareanu, and Radek Pelánek. 2006. Test Input Generation for Java Containers Using State Matching. In International Symposium on Software Testing and Analysis. 37–48. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and understanding bugs in C compilers. In Conference on Programming Language Design and Implementation . 283–294. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Mai Zheng, Vignesh T Ravi, Feng Qin, and Gagan Agrawal. 2011. GRace: A low-overhead mechanism for detecting data races in GPU programs. In ACM SIGPLAN Notices, Vol. 46. 135–146.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Bounded exhaustive test-input generation on GPUs

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!