Abstract
Example-based specifications for program synthesis are inherently ambiguous and may cause synthesizers to generate programs that do not exhibit intended behavior on unseen inputs. Existing synthesis techniques attempt to address this problem by either placing a domain-specific syntactic bias on the hypothesis space or heavily relying on user feedback to help resolve ambiguity.
We present a new framework to address the ambiguity/generalizability problem in example-based synthesis. The key feature of our framework is that it places a semantic bias on the hypothesis space using "relational perturbation properties" that relate the perturbation/change in a program output to the perturbation/change in a program input. An example of such a property is permutation invariance: the program output does not change when the elements of the program input (array) are permuted. The framework is portable across multiple domains and synthesizers and is based on two core steps: (1) automatically augment the set of user-provided examples by "applying" relational perturbation properties and (2) use a generic example-based synthesizer to generate a program consistent with the augmented set of examples. Our framework can be instantiated with three different user interfaces, with varying degrees of user engagement to help infer relevant relational perturbation properties. This includes an interface in which the user only provides examples and our framework automatically infers relevant properties. We implement our framework in a tool SKETCHAX specialized to the SKETCH synthesizer and demonstrate that SKETCHAX is effective in significantly boosting the performance of SKETCH for all three user interfaces.
Supplemental Material
- 2019. Syntax-Guided Synthesis Competition. https://sygus.org/ .Google Scholar
- 2019. Sketch Benchmarks Repositories. https://bitbucket.org/gatoatigrado/sketch-frontend/src .Google Scholar
- Rajeev Alur, Rastislav Bodík, Garvit Juniwal, Milo M. K. Martin, Mukund Raghothaman, Sanjit A. Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, and Abhishek Udupa. 2013. Syntax-Guided Synthesis. In Formal Methods in Computer-Aided Design (FMCAD) . 1–8.Google Scholar
- Matej Balog, Alexander L. Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2017. DeepCoder: Learning to Write Programs. In International Conference on Learning Representations (ICLR).Google Scholar
- Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, T. H. Tse, and Zhi Quan Zhou. 2018. Metamorphic Testing: A Review of Challenges and Opportunities. Comput. Surveys 51, 1 (2018), 4:1–4:27.Google Scholar
- Alessandro Cimatti, Anders Franzén, Alberto Griggio, Roberto Sebastiani, and Cristian Stenico. 2010. Satisfiability Modulo the Theory of Costs: Foundations and Applications. In Tools and Algorithms for the Construction and Analysis of Systems (TACAS) . 99–113.Google Scholar
- Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. 2017. RobustFill: Neural Program Learning under Noisy I/O. In International Conference on Machine Learning (ICML). 990–998.Google Scholar
- Dana Drachsler-Cohen, Sharon Shoham, and Eran Yahav. 2017. Synthesis with Abstract Examples. In Computer Aided Verification (CAV) . 254–278.Google Scholar
- John K. Feser, Swarat Chaudhuri, and Isil Dillig. 2015. Synthesizing Data Structure Transformations from Input-Output Examples. In Programming Language Design and Implementation (PLDI). 229–239.Google Scholar
- Sumit Gulwani. 2011. Automating String Processing in Spreadsheets using Input-Output Examples. In Principles of Programming Languages (POPL) . 317–330.Google Scholar
- Sumit Gulwani. 2016. Programming by Examples: Applications, Algorithms, and Ambiguity Resolution. In International Joint Conference on Automated Reasoning (IJCAR) . 9–14.Google Scholar
Digital Library
- Sumit Gulwani, William R. Harris, and Rishabh Singh. 2012. Spreadsheet Data Manipulation using Examples. Commun. ACM 55, 8 (2012), 97–105.Google Scholar
Digital Library
- Sumit Gulwani, Oleksandr Polozov, and Rishabh Singh. 2017. Program Synthesis. Foundations and Trends in Programming Languages 4, 1-2 (2017), 1–119.Google Scholar
Cross Ref
- Jeevana Priya Inala and Rishabh Singh. 2017. WebRelate: Integrating Web Data with Spreadsheets using Examples. Proceedings of the ACM on Programming Languages (PACMPL) 2, POPL (2017), 2:1–2:28.Google Scholar
Digital Library
- Susmit Jha, Sumit Gulwani, Sanjit A. Seshia, and Ashish Tiwari. 2010. Oracle-guided Component-based Program Synthesis. In International Conference on Software Engineering (ICSE). 215–224.Google Scholar
- Upulee Kanewala, James M Bieman, and Asa Ben-Hur. 2016. Predicting Metamorphic Relations for Testing Scientific Software: A Machine Learning Approach using Graph Kernels. Software Testing, Verification & Reliability 26, 3 (2016), 245–269.Google Scholar
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems (NIPS) 25. 1097–1105.Google Scholar
- Alan Leung, John Sarracino, and Sorin Lerner. 2015. Interactive Parser Synthesis by Example. In Programming Language Design and Implementation (PLDI) . 565–574.Google Scholar
- Henry Lieberman. 2000. Programming by Example: Introduction. Commun. ACM 43, 3 (2000), 72–74.Google Scholar
Digital Library
- Henry Lieberman. 2001. Your Wish is My Command: Programming by Example. Morgan Kaufmann.Google Scholar
- Mikaël Mayer, Gustavo Soares, Maxim Grechkin, Vu Le, Mark Marron, Oleksandr Polozov, Rishabh Singh, Benjamin G. Zorn, and Sumit Gulwani. 2015. User Interaction Models for Disambiguation in Programming by Example. In Symposium on User Interface Software & Technology (UIST) . 291–301.Google Scholar
Digital Library
- Emilio Parisotto, Abdel-rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli. 2017. NeuroSymbolic Program Synthesis. In International Conference on Learning Representations (ICLR).Google Scholar
- Hila Peleg, Sharon Shoham, and Eran Yahav. 2018. Programming Not Only by Example. In International Conference on Software Engineering (ICSE) . 1114–1124.Google Scholar
Digital Library
- Veselin Raychev, Pavol Bielik, Martin Vechev, and Andreas Krause. 2016. Learning Programs from Noisy Data. In Principles of Programming Languages (POPL) . 761–774.Google Scholar
- Patrice Y. Simard, David Steinkraus, and John C. Platt. 2003. Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis. In International Conference on Document Analysis and Recognition (ICDAR). 958–962.Google Scholar
- Rishabh Singh. 2016. BlinkFill: Semi-supervised Programming by Example for Syntactic String Transformations. Proceedings of the VLDB Endowment 9, 10 (2016), 816–827.Google Scholar
Digital Library
- Rishabh Singh and Sumit Gulwani. 2012. Synthesizing Number Transformations from Input-Output Examples. In Computer Aided Verification (CAV) . 634–651.Google Scholar
- Rishabh Singh and Sumit Gulwani. 2015. Predicting a Correct Program in Programming by Example. In Computer Aided Verificationi (CAV) . 398–414.Google Scholar
- Rishabh Singh and Armando Solar-Lezama. 2011. Synthesizing Data Structure Manipulations from Storyboards. In Foundations of Software Engineeringi (ESEC/FSE) . 289–299.Google Scholar
- Calvin Smith and Aws Albarghouthi. 2016. MapReduce Program Synthesis. In Programming Language Design and Implementation (PLDI) . 326–340.Google Scholar
- Armando Solar-Lezama, Liviu Tancau, Rastislav Bodik, Sanjit Seshia, and Vijay Saraswat. 2006. Combinatorial Sketching for Finite Programs. In Architectural Support for Programming Languages and Operating Systems (ASPLOS). 404–415.Google Scholar
- Chenglong Wang, Alvin Cheung, and Rastislav Bodík. 2017. Synthesizing Highly Expressive SQL Queries from Input-Output Examples. In Programming Language Design and Implementation (PLDI). 452–466.Google Scholar
- Yuepeng Wang, Xinyu Wang, and Isil Dillig. 2018. Relational Program Synthesis. Proceedings of the ACM on Programming Languages (PACMPL) 2, OOPSLA (2018), 155:1–155:27.Google Scholar
Digital Library
- Jie Zhang, Junjie Chen, Dan Hao, Yingfei Xiong, Bing Xie, Lu Zhang, and Hong Mei. 2014. Search-based Inference of Polynomial Metamorphic Relations. In Automated Software Engineering (ASE). 701–712.Google Scholar
Index Terms
Augmented example-based synthesis using relational perturbation properties
Recommendations
Can reactive synthesis and syntax-guided synthesis be friends?
PLDI 2022: Proceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and ImplementationWhile reactive synthesis and syntax-guided synthesis (SyGuS) have seen enormous progress in recent years, combining the two approaches has remained a challenge. In this work, we present the synthesis of reactive programs from Temporal Stream Logic ...
Optimizing synthesis with metasketches
POPL '16Many advanced programming tools---for both end-users and expert developers---rely on program synthesis to automatically generate implementations from high-level specifications. These tools often need to employ tricky, custom-built synthesis algorithms ...
Can reactive synthesis and syntax-guided synthesis be friends?
SPLASH Companion 2021: Companion Proceedings of the 2021 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for HumanityWhile reactive synthesis and syntax-guided synthesis (SyGuS) have seen enormous progress in recent years, combining the two approaches has remained a challenge. In this work, we present the synthesis of reactive programs from Temporal Stream Logic ...






Comments