skip to main content
research-article
Open Access

Augmented example-based synthesis using relational perturbation properties

Published:20 December 2019Publication History
Skip Abstract Section

Abstract

Example-based specifications for program synthesis are inherently ambiguous and may cause synthesizers to generate programs that do not exhibit intended behavior on unseen inputs. Existing synthesis techniques attempt to address this problem by either placing a domain-specific syntactic bias on the hypothesis space or heavily relying on user feedback to help resolve ambiguity.

We present a new framework to address the ambiguity/generalizability problem in example-based synthesis. The key feature of our framework is that it places a semantic bias on the hypothesis space using "relational perturbation properties" that relate the perturbation/change in a program output to the perturbation/change in a program input. An example of such a property is permutation invariance: the program output does not change when the elements of the program input (array) are permuted. The framework is portable across multiple domains and synthesizers and is based on two core steps: (1) automatically augment the set of user-provided examples by "applying" relational perturbation properties and (2) use a generic example-based synthesizer to generate a program consistent with the augmented set of examples. Our framework can be instantiated with three different user interfaces, with varying degrees of user engagement to help infer relevant relational perturbation properties. This includes an interface in which the user only provides examples and our framework automatically infers relevant properties. We implement our framework in a tool SKETCHAX specialized to the SKETCH synthesizer and demonstrate that SKETCHAX is effective in significantly boosting the performance of SKETCH for all three user interfaces.

Skip Supplemental Material Section

Supplemental Material

a56-an.webm

References

  1. 2019. Syntax-Guided Synthesis Competition. https://sygus.org/ .Google ScholarGoogle Scholar
  2. 2019. Sketch Benchmarks Repositories. https://bitbucket.org/gatoatigrado/sketch-frontend/src .Google ScholarGoogle Scholar
  3. Rajeev Alur, Rastislav Bodík, Garvit Juniwal, Milo M. K. Martin, Mukund Raghothaman, Sanjit A. Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, and Abhishek Udupa. 2013. Syntax-Guided Synthesis. In Formal Methods in Computer-Aided Design (FMCAD) . 1–8.Google ScholarGoogle Scholar
  4. Matej Balog, Alexander L. Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2017. DeepCoder: Learning to Write Programs. In International Conference on Learning Representations (ICLR).Google ScholarGoogle Scholar
  5. Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, T. H. Tse, and Zhi Quan Zhou. 2018. Metamorphic Testing: A Review of Challenges and Opportunities. Comput. Surveys 51, 1 (2018), 4:1–4:27.Google ScholarGoogle Scholar
  6. Alessandro Cimatti, Anders Franzén, Alberto Griggio, Roberto Sebastiani, and Cristian Stenico. 2010. Satisfiability Modulo the Theory of Costs: Foundations and Applications. In Tools and Algorithms for the Construction and Analysis of Systems (TACAS) . 99–113.Google ScholarGoogle Scholar
  7. Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. 2017. RobustFill: Neural Program Learning under Noisy I/O. In International Conference on Machine Learning (ICML). 990–998.Google ScholarGoogle Scholar
  8. Dana Drachsler-Cohen, Sharon Shoham, and Eran Yahav. 2017. Synthesis with Abstract Examples. In Computer Aided Verification (CAV) . 254–278.Google ScholarGoogle Scholar
  9. John K. Feser, Swarat Chaudhuri, and Isil Dillig. 2015. Synthesizing Data Structure Transformations from Input-Output Examples. In Programming Language Design and Implementation (PLDI). 229–239.Google ScholarGoogle Scholar
  10. Sumit Gulwani. 2011. Automating String Processing in Spreadsheets using Input-Output Examples. In Principles of Programming Languages (POPL) . 317–330.Google ScholarGoogle Scholar
  11. Sumit Gulwani. 2016. Programming by Examples: Applications, Algorithms, and Ambiguity Resolution. In International Joint Conference on Automated Reasoning (IJCAR) . 9–14.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Sumit Gulwani, William R. Harris, and Rishabh Singh. 2012. Spreadsheet Data Manipulation using Examples. Commun. ACM 55, 8 (2012), 97–105.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Sumit Gulwani, Oleksandr Polozov, and Rishabh Singh. 2017. Program Synthesis. Foundations and Trends in Programming Languages 4, 1-2 (2017), 1–119.Google ScholarGoogle ScholarCross RefCross Ref
  14. Jeevana Priya Inala and Rishabh Singh. 2017. WebRelate: Integrating Web Data with Spreadsheets using Examples. Proceedings of the ACM on Programming Languages (PACMPL) 2, POPL (2017), 2:1–2:28.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Susmit Jha, Sumit Gulwani, Sanjit A. Seshia, and Ashish Tiwari. 2010. Oracle-guided Component-based Program Synthesis. In International Conference on Software Engineering (ICSE). 215–224.Google ScholarGoogle Scholar
  16. Upulee Kanewala, James M Bieman, and Asa Ben-Hur. 2016. Predicting Metamorphic Relations for Testing Scientific Software: A Machine Learning Approach using Graph Kernels. Software Testing, Verification & Reliability 26, 3 (2016), 245–269.Google ScholarGoogle Scholar
  17. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems (NIPS) 25. 1097–1105.Google ScholarGoogle Scholar
  18. Alan Leung, John Sarracino, and Sorin Lerner. 2015. Interactive Parser Synthesis by Example. In Programming Language Design and Implementation (PLDI) . 565–574.Google ScholarGoogle Scholar
  19. Henry Lieberman. 2000. Programming by Example: Introduction. Commun. ACM 43, 3 (2000), 72–74.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Henry Lieberman. 2001. Your Wish is My Command: Programming by Example. Morgan Kaufmann.Google ScholarGoogle Scholar
  21. Mikaël Mayer, Gustavo Soares, Maxim Grechkin, Vu Le, Mark Marron, Oleksandr Polozov, Rishabh Singh, Benjamin G. Zorn, and Sumit Gulwani. 2015. User Interaction Models for Disambiguation in Programming by Example. In Symposium on User Interface Software & Technology (UIST) . 291–301.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Emilio Parisotto, Abdel-rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli. 2017. NeuroSymbolic Program Synthesis. In International Conference on Learning Representations (ICLR).Google ScholarGoogle Scholar
  23. Hila Peleg, Sharon Shoham, and Eran Yahav. 2018. Programming Not Only by Example. In International Conference on Software Engineering (ICSE) . 1114–1124.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Veselin Raychev, Pavol Bielik, Martin Vechev, and Andreas Krause. 2016. Learning Programs from Noisy Data. In Principles of Programming Languages (POPL) . 761–774.Google ScholarGoogle Scholar
  25. Patrice Y. Simard, David Steinkraus, and John C. Platt. 2003. Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis. In International Conference on Document Analysis and Recognition (ICDAR). 958–962.Google ScholarGoogle Scholar
  26. Rishabh Singh. 2016. BlinkFill: Semi-supervised Programming by Example for Syntactic String Transformations. Proceedings of the VLDB Endowment 9, 10 (2016), 816–827.Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Rishabh Singh and Sumit Gulwani. 2012. Synthesizing Number Transformations from Input-Output Examples. In Computer Aided Verification (CAV) . 634–651.Google ScholarGoogle Scholar
  28. Rishabh Singh and Sumit Gulwani. 2015. Predicting a Correct Program in Programming by Example. In Computer Aided Verificationi (CAV) . 398–414.Google ScholarGoogle Scholar
  29. Rishabh Singh and Armando Solar-Lezama. 2011. Synthesizing Data Structure Manipulations from Storyboards. In Foundations of Software Engineeringi (ESEC/FSE) . 289–299.Google ScholarGoogle Scholar
  30. Calvin Smith and Aws Albarghouthi. 2016. MapReduce Program Synthesis. In Programming Language Design and Implementation (PLDI) . 326–340.Google ScholarGoogle Scholar
  31. Armando Solar-Lezama, Liviu Tancau, Rastislav Bodik, Sanjit Seshia, and Vijay Saraswat. 2006. Combinatorial Sketching for Finite Programs. In Architectural Support for Programming Languages and Operating Systems (ASPLOS). 404–415.Google ScholarGoogle Scholar
  32. Chenglong Wang, Alvin Cheung, and Rastislav Bodík. 2017. Synthesizing Highly Expressive SQL Queries from Input-Output Examples. In Programming Language Design and Implementation (PLDI). 452–466.Google ScholarGoogle Scholar
  33. Yuepeng Wang, Xinyu Wang, and Isil Dillig. 2018. Relational Program Synthesis. Proceedings of the ACM on Programming Languages (PACMPL) 2, OOPSLA (2018), 155:1–155:27.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Jie Zhang, Junjie Chen, Dan Hao, Yingfei Xiong, Bing Xie, Lu Zhang, and Hong Mei. 2014. Search-based Inference of Polynomial Metamorphic Relations. In Automated Software Engineering (ASE). 701–712.Google ScholarGoogle Scholar

Index Terms

  1. Augmented example-based synthesis using relational perturbation properties

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!