skip to main content
research-article

Bounded exhaustive test input generation from hybrid invariants

Published:15 October 2014Publication History
Skip Abstract Section

Abstract

We present a novel technique for producing bounded exhaustive test suites from hybrid invariants, i.e., invariants that are expressed imperatively, declaratively, or as a combination of declarative and imperative predicates. Hybrid specifications are processed using known mechanisms for the imperative and declarative parts, but combined in a way that enables us to exploit information from the declarative side, such as tight bounds computed from the declarative specification, to improve the search both on the imperative and declarative sides. Moreover, our technique automatically evaluates different possible ways of processing the imperative side, and the alternative settings (imperative or declarative) for parts of the invariant available both declaratively and imperatively, to decide the most convenient invariant configuration with respect to efficiency in test generation. This is achieved by transcoping, i.e., by assessing the efficiency of the different alternatives on small scopes (where generation times are negligible), and then extrapolating the results to larger scopes.

We also show experiments involving collection classes that support the effectiveness of our technique, by demonstrating that (i) bounded exhaustive suites can be computed from hybrid invariants significantly more efficiently than doing so using state-of-the-art purely imperative and purely declarative approaches, and (ii) our technique is able to automatically determine efficient hybrid invariants, in the sense that they lead to an efficient computation of bounded exhaustive suites, using transcoping.

References

  1. P. Abad, N. Aguirre, V. S. Bengolea, D. Ciolek, M. F. Frias, J. P. Galeotti, T. Maibaum, M. Moscato, N. Rosner and I. Vissani, Improving Test Generation under Rich Contracts by Tight Bounds and Incremental SAT Solving, in ICST 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. C. Boyapati, S. Khurshid and D. Marinov, Korat: Automated Testing based on Java Predicates, in ISSTA 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. L. Burdy, Y. Cheon, D. R. Cok, M. D. Ernst, J. R. Kiniry, G. T. Leavens, K. Rustan M. Leino and E. Poll, An overview of JML tools and applications, in STTT 7(3), Springer, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. F. S.-H. Chang and D. Jackson. Symbolic model checking of declarative relational models. In ICSE 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. K. Claessen and J. Hughes, QuickCheck: a lightweight tool for random testing of Haskell programs, in ICFP 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. L. De Moura and N. Bjørner. Z3: An efficient SMT solver. In TACAS 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. G. Dennis, F. Chang, D. Jackson. Verification of Code with SAT, in ISSTA 2006, ACM, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. B. Dutertre and L. D. Moura. The Yices SMT solver. Technical report, 2006.Google ScholarGoogle Scholar
  9. N. Een and N. Sorensson. An Extensible SAT-solver. In SAT 2003.Google ScholarGoogle Scholar
  10. M. Frias, J. Galeotti, C. Pombo, and N. Aguirre. DynAlloy: Upgrading Alloy with actions. In ICSE 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. J.P. Galeotti, N. Rosner, C. López Pombo, M. Frias, TACO: Efficient SAT-Based Bounded Verification Using Symmetry Breaking and Tight Bounds. IEEE TSE 39(9): 1283--1307 (2013). Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. S. Ganov, S. Khurshid, D. E. Perry. Annotations for Alloy: Automated Incremental Analysis Using Domain Specific Solvers. In ICFEM 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. J. Geldenhuys, N. Aguirre, M. F. Frias and W. Visser, Bounded Lazy Initialization, in NFM 2013, LNCS, Springer, 2013.Google ScholarGoogle ScholarCross RefCross Ref
  14. M. Gligoric, T. Gvero, V. Jagannath, S. Khurshid, V. Kuncak and D. Marinov, Test generation through programming in UDITA, in ICSE 2010, Cape Town, South Africa. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. G. J. Holzmann. The model checker SPIN. IEEE TSE 1997. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. D. Jackson, Software Abstractions: Logic, Language and Analysis, The MIT Press, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. C. Kaner, J. Bach and B. Pettichord, Lessons Learned in Software Testing, Wiley, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. S. A. Khalek. Systematic Testing Using Test Summaries: Effective and Efficient Testing of Relational Applications. Ph.D. Thesis. University of Texas at Austin, 2011.Google ScholarGoogle Scholar
  19. S. A. Khalek, V. P. Narayanan, and S. Khurshid. Mixed constraints for test input generation - An initial exploration. In ASE 2011 (Short paper). Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. S. A. Khalek, G. Yang, L. Zhang, D. Marinov and S. Khurshid, TestEra: A Tool for Testing Java Programs using Alloy Specifications, in ASE 2011, IEEE, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. S. Khurshid and D. Marinov, TestEra: Specification-Based Testing of Java Programs Using SAT, Automated Software Engineering 11(4), Springer, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. A. S. Koksal, V. Kuncak, and P. Suter. Constraints as Control. In POPL 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. B. Liskov and J. Guttag, Program Development in Java: Abstraction, Specification, and Object-Oriented Design, Addison-Wesley, 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. A. Milicevic, S. Misailovic, D. Marinov and S. Khurshid, Korat: A Tool for Generating Structurally Complex Test Inputs, in ICSE 2007, IEEE Press, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. A. Milicevic, D. Rayside, K. Yessenov, and D. Jackson. Unifying execution of imperative and declarative code. In ICSE 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. C. Pacheco, S. K. Lahiri, M. D. Ernst and T. Ball, Feedback-Directed Random Test Generation, in ICSE 2007, IEEE, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. N. Rosner, C. G. López Pombo, N. Aguirre, A. Jaoua, A. Mili and M. F. Frias, Parallel Bounded Verification of Alloy Models by TranScoping, in VSTTE 2014, LNCS, Springer, 2013.Google ScholarGoogle Scholar
  28. H. Samimi, E. D. Aung, and T. D. Millstein. Falling back on executable specifications. In ECOOP 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. J. Siddiqui and S. Khurshid, An Empirical Study of Structural Constraint Solving Techniques, in ICFEM 2009, LNCS, Springer, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. E. Uzuncaova. Efficient specification-based testing using incremental techniques. PhD thesis, [email protected], 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. E. Uzuncaova and S. Khurshid. Constraint prioritization for efficient analysis of declarative models. In FM 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. W. Visser, C. Pasareanu and S. Khurshid, Test input generation with java PathFinder, in ISSTA 2004, ACM, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. T. Xie, D. Marinov and D. Notkin, Rostra: A Framework for Detecting Redundant Object-Oriented Unit Tests, in ASE 2004, IEEE, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. R. N. Zaeem and S. Khurshid. Contract-based data structure repair using Alloy. In ECOOP 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. The HyTeK distribution can be downloaded from http://bonnie.exp.dc.uba.ar/hytek-oopsla.tar.gzGoogle ScholarGoogle Scholar
  36. http://alloy.mit.edu/Google ScholarGoogle Scholar

Index Terms

  1. Bounded exhaustive test input generation from hybrid invariants

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM SIGPLAN Notices
      ACM SIGPLAN Notices  Volume 49, Issue 10
      OOPSLA '14
      October 2014
      907 pages
      ISSN:0362-1340
      EISSN:1558-1160
      DOI:10.1145/2714064
      • Editor:
      • Andy Gill
      Issue’s Table of Contents
      • cover image ACM Conferences
        OOPSLA '14: Proceedings of the 2014 ACM International Conference on Object Oriented Programming Systems Languages & Applications
        October 2014
        946 pages
        ISBN:9781450325851
        DOI:10.1145/2660193

      Copyright © 2014 ACM

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 15 October 2014

      Check for updates

      Qualifiers

      • research-article

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!