Abstract
Motivated by applications in robotics, we consider the task of synthesizing linear temporal logic (LTL) specifications based on examples and natural language descriptions. While LTL is a flexible, expressive, and unambiguous language to describe robotic tasks, it is often challenging for non-expert users. In this paper, we present an interactive method for synthesizing LTL specifications from a single example trace and a natural language description. The interaction is limited to showing a small number of behavioral examples to the user who decides whether or not they exhibit the original intent. Our approach generates candidate LTL specifications and distinguishing examples using an encoding into optimization modulo theories problems. Additionally, we use a grammar extension mechanism and a semantic parser to generalize synthesized specifications to parametric task descriptions for subsequent use. Our implementation in the tool LtlTalk starts with a domain-specific language that maps to a fragment of LTL and expands it through example-based user interactions, thus enabling natural language-like robot programming, while maintaining the expressive power and precision of a formal language. Our experiments show that the synthesis method is precise, quick, and asks only a few questions to the users, and we demonstrate in a case study how LtlTalk generalizes from the synthesized tasks to other, yet unseen, tasks.
Supplemental Material
- Rajeev Alur, Rastislav Bodík, Eric Dallal, Dana Fisman, Pranav Garg, Garvit Juniwal, Hadas Kress-Gazit, P. Madhusudan, Milo M. K. Martin, Mukund Raghothaman, Shambwaditya Saha, Sanjit A. Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, and Abhishek Udupa. 2015. Syntax-Guided Synthesis. In Dependable Software Systems Engineering. NATO Science for Peace and Security Series, D: Information and Communication Security, Vol. 40. IOS Press, 1-25. https://doi.org/10.3233/978-1-61499-495-4-1 Google Scholar
Cross Ref
- Shengwei An, Rishabh Singh, Sasa Misailovic, and Roopsha Samanta. 2020. Augmented example-based synthesis using relational perturbation properties. PACMPL 4, POPL ( 2020 ). https://doi.org/10.1145/3371124 Google Scholar
Digital Library
- Dana Angluin. 1987. Learning Regular Sets from Queries and Counterexamples. Inf. Comput. 75, 2 ( 1987 ). https: //doi.org/10.1016/ 0890-5401 ( 87 ) 90052-6 Google Scholar
Digital Library
- Yoav Artzi. 2016. Cornell SPF : Cornell Semantic Parsing Framework. arXiv:arXiv:1311.3011Google Scholar
- Matej Balog, Alexander L. Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2017. DeepCoder: Learning to Write Programs. In International Conference on Learning Representations (ICLR).Google Scholar
- Ezio Bartocci, Luca Bortolussi, and Guido Sanguinetti. 2013. Learning Temporal Logical Properties Discriminating ECG models of Cardiac Arrhytmias. CoRR abs/1312.7523 ( 2013 ). arXiv: 1312. 7523Google Scholar
- I. Beltagy and Chris Quirk. 2016. Improved Semantic Parsers For If-Then Statements. In Annual Meeting of the Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/ P16-1069 Google Scholar
Cross Ref
- Armin Biere, Keijo Heljanko, Tommi A. Junttila, Timo Latvala, and Viktor Schuppan. 2006. Linear Encodings of Bounded LTL Model Checking. Logical Methods in Computer Science 2, 5 ( 2006 ). https://doi.org/10.2168/LMCS-2( 5 :5) 2006 Google Scholar
Cross Ref
- Nikolaj Bjørner, Anh-Dung Phan, and Lars Fleckenstein. 2015. ν Z-An Optimizing SMT Solver. In Tools and Algorithms for the Construction and Analysis of Systems (TACAS). https://doi.org/10.1007/978-3-662-46681-0_14 Google Scholar
Digital Library
- Alberto Camacho and Sheila A. McIlraith. 2019. Learning Interpretable Models Expressed in Linear Temporal Logic. In International Conference on Automated Planning and Scheduling (ICAPS).Google Scholar
- Sofia Cassel, Falk Howar, Bengt Jonsson, and Bernhard Stefen. 2016. Active learning for extended finite state machines. Formal Asp. Comput. 28, 2 ( 2016 ). https://doi.org/10.1007/s00165-016-0355-5 Google Scholar
Digital Library
- Qiaochu Chen, Xinyu Wang, Xi Ye, Greg Durrett, and Isil Dillig. 2020. Multi-modal Synthesis of Regular Expressions. In Programming Language Design and Implementation (PLDI).Google Scholar
- Yanju Chen, Ruben Martins, and Yu Feng. 2019. Maximal Multi-layer Specification Synthesis. In Foundations of Software Engineering (FSE). https://doi.org/10.1145/3338906.3338951 Google Scholar
Digital Library
- Jamieson M. Cobleigh, Dimitra Giannakopoulou, and Corina S. Pasareanu. 2003. Learning Assumptions for Compositional Verification. In Tools and Algorithms for the Construction and Analysis of Systems (TACAS). https://doi.org/10.1007/3-540-36577-X_24 Google Scholar
- Giuseppe De Giacomo and Moshe Y. Vardi. 2013. Linear Temporal Logic and Linear Dynamic Logic on Finite Traces. In International Joint Conference on Artificial Intelligence (IJCAI).Google Scholar
- Giuseppe De Giacomo and Moshe Y. Vardi. 2015. Synthesis for LTL and LDL on Finite Traces. In International Joint Conference on Artificial Intelligence (IJCAI).Google Scholar
- Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An Eficient SMT Solver. In Tools and Algorithms for the Construction and Analysis of Systems (TACAS). https://doi.org/10.1007/978-3-540-78800-3_24 Google Scholar
Cross Ref
- Aditya Desai, Sumit Gulwani, Vineet Hingorani, Nidhi Jain, Amey Karkare, Mark Marron, Sailesh R, and Subhajit Roy. 2016. Program synthesis using natural language. In International Conference on Software Engineering (ICSE). https: //doi.org/10.1145/2884781.2884786 Google Scholar
Digital Library
- Dana Drachsler-Cohen, Sharon Shoham, and Eran Yahav. 2017. Synthesis with Abstract Examples. In Computer Aided Verification (CAV). https://doi.org/10.1007/978-3-319-63387-9_13 Google Scholar
Cross Ref
- Matthew B. Dwyer, George S. Avrunin, and James C. Corbett. 1999. Patterns in Property Specifications for Finite-State Verification. In International Conference on Software Engineering (ICSE). https://doi.org/10.1145/302405.302672 Google Scholar
Digital Library
- R. Fikes and N. J. Nilsson. 1971. STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving. Artif. Intell. 2, 3 /4 ( 1971 ). https://doi.org/10.1016/ 0004-3702 ( 71 ) 90010-5 Google Scholar
Cross Ref
- Ivan Gavran, Rupak Majumdar, and Indranil Saha. 2017. Antlab: A Multi-Robot Task Server. ACM Trans. Embed. Comput. Syst. 16, 5s, Article 190 ( 2017 ). https://doi.org/10.1145/3126513 Google Scholar
Digital Library
- M. Ghallab, C. Aeronautiques, C. K. Isi, and D. Wilkins. 1998. PDDL: The Planning Domain Definition Language. Technical Report CVC TR98003/DCS TR1165. Yale Center for Computational Vision and Control.Google Scholar
- E. Mark Gold. 1967. Language Identification in the Limit. Information and Control 10, 5 ( 1967 ). https://doi.org/10.1016/S0019-9958 ( 67 ) 91165-5 Google Scholar
Cross Ref
- Sumit Gulwani and Mark Marron. 2014. NLyze: Interactive Programming by Natural Language for Spreadsheet Data Analysis and Manipulation. In Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data. https://doi.org/10.1145/2588555.2612177 Google Scholar
Digital Library
- Gerard J. Holzmann. 2002. The logic of bugs. In Symposium on Foundations of Software Engineering (FSE). https://doi.org/10. 1145/587051.587064 Google Scholar
Digital Library
- Xiaoqing Jin, Alexandre Donzé, Jyotirmoy V. Deshmukh, and Sanjit A. Seshia. 2015. Mining Requirements From Closed-Loop Control Models. IEEE Trans. on CAD of Integrated Circuits and Systems 34, 11 ( 2015 ). https://doi.org/10.1109/TCAD. 2015. 2421907 Google Scholar
Digital Library
- Joseph Kim, Christian Muise, Ankit Shah, Shubham Agarwal, and Julie Shah. 2019. Bayesian Inference of Linear Temporal Logic Specifications for Contrastive Explanations. In International Joint Conference on Artificial Intelligence (IJCAI). https://doi.org/10.24963/ijcai. 2019 /776 Google Scholar
Cross Ref
- Thomas Kollar, Vittorio Perera, Daniele Nardi, and Manuela M. Veloso. 2013. Learning Environmental Knowledge from Task-Based Human-Robot Dialog. In International Conference on Robotics and Automation (ICRA). https://doi.org/10. 1109/ICRA. 2013.6631186 Google Scholar
Cross Ref
- Zhaodan Kong, Austin Jones, Ana Medina Ayala, Ebru Aydin Gol, and Calin Belta. 2014. Temporal logic inference for classification and prediction from data. In Hybrid Systems: Computation and Control (HSCC). https://doi.org/10.1145/ 2562059.2562146 Google Scholar
Digital Library
- Zhaodan Kong, Austin Jones, and Calin Belta. 2017. Temporal Logics for Learning and Detection of Anomalous Behavior. IEEE Trans. Automat. Contr. 62, 3 ( 2017 ). https://doi.org/10.1109/TAC. 2016.2585083 Google Scholar
Cross Ref
- Hadas Kress-Gazit, Georgios E. Fainekos, and George J. Pappas. 2008. Translating Structured English to Robot Controllers. Advanced Robotics 22, 12 ( 2008 ). https://doi.org/10.1163/156855308X344864 Google Scholar
Cross Ref
- H. Kress-Gazit, G. E. Fainekos, and G. J. Pappas. 2009. Temporal-Logic-Based Reactive Mission and Motion Planning. IEEE Transactions on Robotics ( 2009 ). https://doi.org/10.1109/TRO. 2009.2030225 Google Scholar
Digital Library
- H.J. Levesque, R. Reiter, Y. Lespérance, F. Lin, and R.B. Scherl. 1997. GOLOG: A Logic Programming Language for Dynamic Domains. J. Log. Program. 31, 1-3 ( 1997 ). https://doi.org/10.1016/S0743-1066 ( 96 ) 00121-5 Google Scholar
Cross Ref
- Fei Li and H. V. Jagadish. 2014. Constructing an Interactive Natural Language Interface for Relational Databases. PVLDB 8, 1 ( 2014 ). https://doi.org/10.14778/2735461.2735468 Google Scholar
Digital Library
- Constantine Lignos, Vasumathi Raman, Cameron Finucane, Mitchell P. Marcus, and Hadas Kress-Gazit. 2015. Provably Correct Reactive Control from Natural Language. Auton. Robots 38, 1 ( 2015 ). https://doi.org/10.1007/s10514-014-9418-8 Google Scholar
Digital Library
- Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. In Association for Computational Linguistics (ACL) System Demonstrations. https://doi.org/10.3115/v1/ P14-5010 Google Scholar
Cross Ref
- Mehdi Hafezi Manshadi, Daniel Gildea, and James F. Allen. 2013. Integrating Programming by Example and Natural Language Programming. In Conference on Artificial Intelligence (AAAI).Google Scholar
- Mikaël Mayer, Gustavo Soares, Maxim Grechkin, Vu Le, Mark Marron, Oleksandr Polozov, Rishabh Singh, Benjamin Zorn, and Sumit Gulwani. 2015. User Interaction Models for Disambiguation in Programming by Example. In User Interface Software & Technology (UIST). https://doi.org/10.1145/2807442.2807459 Google Scholar
Digital Library
- Çetin Meriçli, Steven D. Klee, Jack Paparian, and Manuela M. Veloso. 2014. An interactive approach for situated task specification through verbal instructions. In Autonomous Agents and Multi-Agent Systems (AAMAS).Google Scholar
- George A. Miller. 1995. WordNet: A Lexical Database for English. Commun. ACM 38, 11 ( 1995 ). https://doi.org/10.1145/ 219717.219748 Google Scholar
Digital Library
- Sara Mohammadinejad, Jyotirmoy V. Deshmukh, Aniruddh G. Puranic, Marcell Vazquez-Chanlatte, and Alexandre Donzé. 2020. Interpretable classification of time-series data using eficient enumerative techniques. In Hybrid Systems: Computation and Control (HSCC). https://doi.org/10.1145/3365365.3382218 Google Scholar
Digital Library
- Daniel Neider and Ivan Gavran. 2018. Learning Linear Temporal Properties. In FMCAD (Formal Methods in Computer Aided Design). https://doi.org/10.23919/FMCAD. 2018.8603016 Google Scholar
Cross Ref
- Maxwell I. Nye, Luke B. Hewitt, Joshua B. Tenenbaum, and Armando Solar-Lezama. 2019. Learning to Infer Program Sketches. In International Conference on Machine Learning (ICML).Google Scholar
- Hila Peleg, Sharon Shoham, and Eran Yahav. 2018. Programming Not Only by Example. In International Conference on Software Engineering (ICSE). https://doi.org/10.1145/3180155.3180189 Google Scholar
Digital Library
- Vittorio Perera and Manuela M. Veloso. 2015. Handling Complex Commands as Service Robot Task Requests. In International Joint Conference on Artificial Intelligence (IJCAI).Google Scholar
- Amir Pnueli. 1977. The temporal logic of programs. In Foundations of Computer Science, 1977., 18th Annual Symposium on. IEEE. https://doi.org/10.1109/SFCS. 1977.32 Google Scholar
Digital Library
- Illia Polosukhin and Alexander Skidanov. 2018. Neural Program Search: Solving Programming Tasks from Description and Examples. In International Conference on Learning Representations (ICLR).Google Scholar
- Mohammad Raza, Sumit Gulwani, and Natasa Milic-Frayling. 2015. Compositional Program Synthesis from Natural Language and Examples. In International Joint Conference on Artificial Intelligence (IJCAI).Google Scholar
- Indranil Saha, Rattanachai Ramaithitima, Vijay Kumar, George J. Pappas, and Sanjit A. Seshia. 2016. Implan: Scalable Incremental Motion Planning for Multi-Robot Systems. In International Conference on Cyber-Physical Systems (ICCPS). https://doi.org/10.1109/ICCPS. 2016.7479105 Google Scholar
Cross Ref
- Rishabh Singh and Sumit Gulwani. 2015. Predicting a Correct Program in Programming by Example. In Computer Aided Verification (CAV). https://doi.org/10.1007/978-3-319-21690-4_23 Google Scholar
Cross Ref
- Jesse Thomason, Shiqi Zhang, Raymond J. Mooney, and Peter Stone. 2015. Learning to Interpret Natural Language Commands through Human-Robot Dialog. In International Joint Conference on Artificial Intelligence (IJCAI).Google Scholar
- Marcell Vazquez-Chanlatte, Susmit Jha, Ashish Tiwari, Mark K. Ho, and Sanjit A. Seshia. 2018. Learning Task Specifications from Demonstrations. In Neural Information Processing Systems (NeurIPS).Google Scholar
- Chenglong Wang, Alvin Cheung, and Rastislav Bodík. 2017a. Interactive Query Synthesis from Input-Output Examples. In SIGMOD Conference. https://doi.org/10.1145/3035918.3058738 Google Scholar
Digital Library
- Chenglong Wang, Alvin Cheung, and Rastislav Bodík. 2017b. Synthesizing highly expressive SQL queries from input-output examples. In Programming Language Design and Implementation (PLDI). https://doi.org/10.1145/3140587.3062365 Google Scholar
Digital Library
- Sida I. Wang, Samuel Ginn, Percy Liang, and Christopher D. Manning. 2017c. Naturalizing a Programming Language via Interactive Learning. In Annual Meeting of the Association for Computational Linguistics (ACL). https://doi.org/10.18653/ v1/ P17-1086 Google Scholar
Cross Ref
- Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. 2017. SQLizer: query synthesis from natural language. PACMPL 1, OOPSLA ( 2017 ). https://doi.org/10.1145/3133887 Google Scholar
Digital Library
Index Terms
Interactive synthesis of temporal specifications from examples and natural language
Recommendations
Extracting design information from natural language specifications
DAC '12: Proceedings of the 49th Annual Design Automation ConferenceNatural language specifications are the first concrete behavioral description which is the basis for any manually generated formal behavioral model. Natural language is preferred as the initial description method mainly because it is much simpler for a ...
Breaking Down High-Level Robot Path-Finding Abstractions in Natural Language Programming
AIxIA 2020 – Advances in Artificial IntelligenceAbstractNatural language programming (NLPr) allows people to program in natural language (NL) for specific domains. It poses great potential since it gives non-experts the ability to develop projects without exhaustive training. However, complex ...
Algorithmic program synthesis: introduction
Program synthesis is a process of producing an executable program from a specification. Algorithmic synthesis produces the program automatically, without an intervention from an expert. While classical compilation falls under the definition of ...






Comments