skip to main content
research-article
Open Access

Synthesizing contracts correct modulo a test generator

Published:15 October 2021Publication History
Skip Abstract Section

Abstract

We present an approach to learn contracts for object-oriented programs where guarantees of correctness of the contracts are made with respect to a test generator. Our contract synthesis approach is based on a novel notion of tight contracts and an online learning algorithm that works in tandem with a test generator to synthesize tight contracts. We implement our approach in a tool called Precis and evaluate it on a suite of programs written in C#, studying the safety and strength of the synthesized contracts, and compare them to those synthesized by Daikon.

References

  1. Rajeev Alur, Rastislav Bodík, Eric Dallal, Dana Fisman, Pranav Garg, Garvit Juniwal, Hadas Kress-Gazit, P. Madhusudan, Milo M. K. Martin, Mukund Raghothaman, Shambwaditya Saha, Sanjit A. Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, and Abhishek Udupa. 2015. Syntax-guided synthesis. In Dependable Software Systems Engineering 2015.Google ScholarGoogle Scholar
  2. Rajeev Alur, Arjun Radhakrishna, and Abhishek Udupa. 2017. Scaling enumerative program synthesis via divide and conquer. In Tools and Algorithms for the Construction and Analysis of Systems.Google ScholarGoogle Scholar
  3. Rajeev Alur, Pavol Černý, P. Madhusudan, and Wonhong Nam. 2005. Synthesis of interface specifications for Java classes. In Proceedings of the 32nd ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages. https://doi.org/10.1145/1040305.1040314 Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Glenn Ammons, Rastislav Bodík, and James R. Larus. 2002. Mining Specifications. In Proceedings of the 29th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’02). Association for Computing Machinery, New York, NY, USA. isbn:1581134509 https://doi.org/10.1145/503272.503275 Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Angello Astorga, P. Madhusudan, Shambwaditya Saha, Shiyu Wang, and Tao Xie. 2019. Learning Stateful Preconditions modulo a Test Generator. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2019). Association for Computing Machinery, New York, NY, USA. isbn:9781450367127 https://doi.org/10.1145/3314221.3314641 Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. A. Astorga, S. Srisakaokul, X. Xiao, and T. Xie. 2018. PreInfer: Automatic Inference of Preconditions via Symbolic Analysis. In 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN). 678–689. https://doi.org/10.1109/DSN.2018.00074 Google ScholarGoogle ScholarCross RefCross Ref
  7. Mike Barnett, K. Rustan M. Leino, and Wolfram Schulte. 2005. The Spec# Programming System: An Overview. In Construction and Analysis of Safe, Secure, and Interoperable Smart Devices.Google ScholarGoogle Scholar
  8. Adam Betts, Nathan Chong, Alastair F. Donaldson, Shaz Qadeer, and Paul Thomson. 2012. GPUVerify: a verifier for GPU kernels. In Proceedings of the 27th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2012, part of SPLASH 2012, Tucson, AZ, USA, October 21-25, 2012, Gary T. Leavens and Matthew B. Dwyer (Eds.). ACM, 113–132. https://doi.org/10.1145/2384616.2384625 Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Feng Chen and Grigore Roşu. 2007. Mop: An Efficient and Generic Runtime Verification Framework. SIGPLAN Not., 42, 10 (2007), Oct., issn:0362-1340 https://doi.org/10.1145/1297105.1297069 Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Berkeley Churchill, Oded Padon, Rahul Sharma, and Alex Aiken. 2019. Semantic Program Alignment for Equivalence Checking. In PLDI 2019.Google ScholarGoogle Scholar
  11. Patrick Cousot, Radhia Cousot, Manuel Fähndrich, and Francesco Logozzo. 2013. Automatic Inference of Necessary Preconditions. In Verification, Model Checking, and Abstract Interpretation, Roberto Giacobazzi, Josh Berdine, and Isabella Mastroeni (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg. isbn:978-3-642-35873-9Google ScholarGoogle Scholar
  12. Christoph Csallner, Nikolai Tillmann, and Yannis Smaragdakis. 2008. DySy: Dynamic symbolic execution for invariant inference. In ICSE 2008.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An Efficient SMT Solver. In TACAS 2008.Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Daniel DeFreez, Haaken Martinson Baldwin, Cindy Rubio-González, and Aditya V. Thakur. 2019. Effective error-specification inference via domain-knowledge expansion. In FSE 2019.Google ScholarGoogle Scholar
  15. Isil Dillig, Thomas Dillig, Boyang Li, and Ken McMillan. 2013. Inductive invariant generation via abductive inference. In OOPSLA 2013.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Nii Dodoo, Lin Li, and Michael Ernst. 2003. Selecting, Refining, and Evaluating Predicates for Program Analysis.Google ScholarGoogle Scholar
  17. Michael D. Ernst. 2000. Dynamically Discovering Likely Program Invariants. University of Washington Department of Computer Science and Engineering. Seattle, Washington.Google ScholarGoogle Scholar
  18. Michael D. Ernst, Jake Cockrell, William G. Griswold, and David Notkin. 1999. Dynamically Discovering Likely Program Invariants to Support Program Evolution. In ICSE 1999.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. P. Ezudheen, Daniel Neider, Deepak D’Souza, Pranav Garg, and P. Madhusudan. 2018. Horn-ICE learning for synthesizing invariants and contracts. In OOPSLA 2018.Google ScholarGoogle Scholar
  20. Manuel Fähndrich. 2010. Static Verification for Code Contracts. In SAS 2010.Google ScholarGoogle Scholar
  21. Robert W. Floyd. 1960. An Algorithm Defining ALGOL Assignment Statements. Commun. ACM, 3, 3 (1960), 170–171. https://doi.org/10.1145/367149.367170 Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Gordon Fraser and Andreas Zeller. 2011. Generating Parameterized Unit Tests. In Proceedings of the 2011 International Symposium on Software Testing and Analysis (ISSTA ’11). Association for Computing Machinery, New York, NY, USA. isbn:9781450305624 https://doi.org/10.1145/2001420.2001464 Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Pranav Garg, Daniel Neider, P. Madhusudan, and Dan Roth. 2016. Learning Invariants Using Decision Trees and Implication Counterexamples. In Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’16). Association for Computing Machinery, New York, NY, USA. isbn:9781450335492 https://doi.org/10.1145/2837614.2837664 Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Milos Gligoric, Tihomir Gvero, Vilas Jagannath, Sarfraz Khurshid, Viktor Kuncak, and Darko Marinov. 2010. Test Generation through Programming in UDITA. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1. https://doi.org/10.1145/1806799.1806835 Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Thomas A. Henzinger, Ranjit Jhala, and Rupak Majumdar. 2005. Permissive Interfaces. In Proceedings of the 10th European Software Engineering Conference Held Jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE-13). https://doi.org/10.1145/1081706.1081713 Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Charles Antony Richard Hoare. 1969. An axiomatic basis for computer programming. Commun. ACM, 12, 10 (1969), 576–580.Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Gunel Jahangirova, David Clark, Mark Harman, and Paolo Tonella. 2016. Test Oracle Assessment and Improvement. In ISSTA 2016.Google ScholarGoogle Scholar
  28. Ton Chanh Le, Guolong Zheng, and ThanhVu Nguyen. 2019. SLING: Using Dynamic Analysis to Infer Program Invariants in Separation Logic. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2019). https://doi.org/10.1145/3314221.3314634 Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Gary T. Leavens, Albert L. Baker, and Clyde Ruby. 2006. Preliminary Design of JML: A Behavioral Interface Specification Language for Java. SIGSOFT Softw. Eng. Notes.Google ScholarGoogle Scholar
  30. Bertrand Meyer. 1988. Object-Oriented Software Construction (1st ed.). Prentice-Hall, Inc., USA. isbn:0136290493Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Thomas M. Mitchell. 1997. Machine Learning (1 ed.).Google ScholarGoogle Scholar
  32. Facundo Molina, Pablo Ponzio, Nazareno Aguirre, and Marcelo F. Frias. 2021. EvoSpex: An Evolutionary Algorithm for Learning Postconditions. arxiv:2102.13569.Google ScholarGoogle Scholar
  33. Mary M. Moya and Don R. Hush. 1996. Network constraints and multi-objective optimization for one-class classification.. Neural Networks, 9, 3 (1996), 463–474. http://dblp.uni-trier.de/db/journals/nn/nn9.html##MoyaH96Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Daniel Neider, Shambwaditya Saha, and P. Madhusudan. 2016. Synthesizing Piece-Wise Functions by Learning Classifiers. In Proceedings of the 22nd International Conference on Tools and Algorithms for the Construction and Analysis of Systems - Volume 9636. https://doi.org/10.1007/978-3-662-49674-9_11 Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Julie L. Newcomb, Andrew Adams, Steven Johnson, Rastislav Bodik, and Shoaib Kamil. 2020. Verifying and Improving Halide’s Term Rewriting System with Program Synthesis. Proc. ACM Program. Lang., 4, OOPSLA (2020), Article 166, Nov., https://doi.org/10.1145/3428234 Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Pengyu Nie, Marinela Parovic, Zhiqiang Zang, Sarfraz Khurshid, Aleksandar Milicevic, and Milos Gligoric. 2020. Unifying Execution of Imperative Generators and Declarative Specifications. Proc. ACM Program. Lang., 4, OOPSLA (2020), Article 217, Nov., https://doi.org/10.1145/3428285 Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Carlos Pacheco and Michael D. Ernst. 2007. Randoop: Feedback-Directed Random Testing for Java. In Companion to the 22nd ACM SIGPLAN Conference on Object-Oriented Programming Systems and Applications Companion (OOPSLA ’07). https://doi.org/10.1145/1297846.1297902 Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Saswat Padhi, Rahul Sharma, and Todd Millstein. 2016. Data-Driven Precondition Inference with Learned Features. In Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’16). Association for Computing Machinery, New York, NY, USA. isbn:9781450342612 https://doi.org/10.1145/2908080.2908099 Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Rahul Pandita, Xusheng Xiao, Hao Zhong, Tao Xie, Stephen Oney, and Amit Paradkar. 2012. Inferring Method Specifications from Natural Language API Descriptions. In 2012 34th International Conference on Software Engineering (ICSE).Google ScholarGoogle ScholarCross RefCross Ref
  40. Nadia Polikarpova, Ilinca Ciupa, and Bertrand Meyer. 2009. A Comparative Study of Programmer-Written and Automatically Inferred Contracts. In Proceedings of the Eighteenth International Symposium on Software Testing and Analysis (ISSTA ’09). Association for Computing Machinery, New York, NY, USA. isbn:9781605583389 https://doi.org/10.1145/1572272.1572284 Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Andrew Reynolds, Haniel Barbosa, Andres Nötzli, Clark Barrett, and Cesare Tinelli. 2019. cvc4sy: Smart and Fast Term Enumeration for Syntax-Guided Synthesis. In Computer Aided Verification.Google ScholarGoogle Scholar
  42. Todd W. Schiller, Kellen Donohue, Forrest Coward, and Michael D. Ernst. 2014. Case Studies and Tools for Contract Specifications. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). https://doi.org/10.1145/2568225.2568285 Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. J. M. Spivey. 1988. Understanding Z: A Specification Language and Its Formal Semantics. Cambridge University Press, USA. isbn:0521334292Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Valerio Terragni, Gunel Jahangirova, Paolo Tonella, and Mauro Pezzè. 2020. Evolutionary Improvement of Assertion Oracles. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2020). https://doi.org/10.1145/3368089.3409758 Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Nikolai Tillmann and Jonathan De Halleux. 2008. Pex: White Box Test Generation for .NET. In Tests and Proofs.Google ScholarGoogle Scholar
  46. John Whaley, Michael C. Martin, and Monica S. Lam. 2002. Automatic Extraction of Object-Oriented Component Interfaces. In Proceedings of the 2002 ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA ’02). https://doi.org/10.1145/566172.566212 Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Tao Xie, Evan Martin, and Hai Yuan. 2006. Automatic Extraction of Abstract-Object-State Machines from Unit-Test Executions. In Proceedings of the 28th International Conference on Software Engineering (ICSE ’06). Association for Computing Machinery, New York, NY, USA. isbn:1595933751 https://doi.org/10.1145/1134285.1134427 Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Juan Zhai, Yu Shi, Minxue Pan, Guian Zhou, Yongxiang Liu, Chunrong Fang, Shiqing Ma, Lin Tan, and Xiangyu Zhang. 2020. C2S: Translating Natural Language Comments to Formal Program Specifications. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2020). https://doi.org/10.1145/3368089.3409716 Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. He Zhu, Stephen Magill, and Suresh Jagannathan. 2018. A Data-Driven CHC Solver. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2018). https://doi.org/10.1145/3192366.3192416 Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Synthesizing contracts correct modulo a test generator

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!