skip to main content

Interactive synthesis of temporal specifications from examples and natural language

Published:13 November 2020Publication History
Skip Abstract Section

Abstract

Motivated by applications in robotics, we consider the task of synthesizing linear temporal logic (LTL) specifications based on examples and natural language descriptions. While LTL is a flexible, expressive, and unambiguous language to describe robotic tasks, it is often challenging for non-expert users. In this paper, we present an interactive method for synthesizing LTL specifications from a single example trace and a natural language description. The interaction is limited to showing a small number of behavioral examples to the user who decides whether or not they exhibit the original intent. Our approach generates candidate LTL specifications and distinguishing examples using an encoding into optimization modulo theories problems. Additionally, we use a grammar extension mechanism and a semantic parser to generalize synthesized specifications to parametric task descriptions for subsequent use. Our implementation in the tool LtlTalk starts with a domain-specific language that maps to a fragment of LTL and expands it through example-based user interactions, thus enabling natural language-like robot programming, while maintaining the expressive power and precision of a formal language. Our experiments show that the synthesis method is precise, quick, and asks only a few questions to the users, and we demonstrate in a case study how LtlTalk generalizes from the synthesized tasks to other, yet unseen, tasks.

Skip Supplemental Material Section

Supplemental Material

Auxiliary Presentation Video

This is a presentation video for the paper on instructing robots in natural language with the help of specification synthesis. (The paper appeared at OOPSLA 2020.) The demo-implementation is available at ltltalk.mpi-sws.org

References

  1. Rajeev Alur, Rastislav Bodík, Eric Dallal, Dana Fisman, Pranav Garg, Garvit Juniwal, Hadas Kress-Gazit, P. Madhusudan, Milo M. K. Martin, Mukund Raghothaman, Shambwaditya Saha, Sanjit A. Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, and Abhishek Udupa. 2015. Syntax-Guided Synthesis. In Dependable Software Systems Engineering. NATO Science for Peace and Security Series, D: Information and Communication Security, Vol. 40. IOS Press, 1-25. https://doi.org/10.3233/978-1-61499-495-4-1 Google ScholarGoogle ScholarCross RefCross Ref
  2. Shengwei An, Rishabh Singh, Sasa Misailovic, and Roopsha Samanta. 2020. Augmented example-based synthesis using relational perturbation properties. PACMPL 4, POPL ( 2020 ). https://doi.org/10.1145/3371124 Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Dana Angluin. 1987. Learning Regular Sets from Queries and Counterexamples. Inf. Comput. 75, 2 ( 1987 ). https: //doi.org/10.1016/ 0890-5401 ( 87 ) 90052-6 Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Yoav Artzi. 2016. Cornell SPF : Cornell Semantic Parsing Framework. arXiv:arXiv:1311.3011Google ScholarGoogle Scholar
  5. Matej Balog, Alexander L. Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2017. DeepCoder: Learning to Write Programs. In International Conference on Learning Representations (ICLR).Google ScholarGoogle Scholar
  6. Ezio Bartocci, Luca Bortolussi, and Guido Sanguinetti. 2013. Learning Temporal Logical Properties Discriminating ECG models of Cardiac Arrhytmias. CoRR abs/1312.7523 ( 2013 ). arXiv: 1312. 7523Google ScholarGoogle Scholar
  7. I. Beltagy and Chris Quirk. 2016. Improved Semantic Parsers For If-Then Statements. In Annual Meeting of the Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/ P16-1069 Google ScholarGoogle ScholarCross RefCross Ref
  8. Armin Biere, Keijo Heljanko, Tommi A. Junttila, Timo Latvala, and Viktor Schuppan. 2006. Linear Encodings of Bounded LTL Model Checking. Logical Methods in Computer Science 2, 5 ( 2006 ). https://doi.org/10.2168/LMCS-2( 5 :5) 2006 Google ScholarGoogle ScholarCross RefCross Ref
  9. Nikolaj Bjørner, Anh-Dung Phan, and Lars Fleckenstein. 2015. ν Z-An Optimizing SMT Solver. In Tools and Algorithms for the Construction and Analysis of Systems (TACAS). https://doi.org/10.1007/978-3-662-46681-0_14 Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Alberto Camacho and Sheila A. McIlraith. 2019. Learning Interpretable Models Expressed in Linear Temporal Logic. In International Conference on Automated Planning and Scheduling (ICAPS).Google ScholarGoogle Scholar
  11. Sofia Cassel, Falk Howar, Bengt Jonsson, and Bernhard Stefen. 2016. Active learning for extended finite state machines. Formal Asp. Comput. 28, 2 ( 2016 ). https://doi.org/10.1007/s00165-016-0355-5 Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Qiaochu Chen, Xinyu Wang, Xi Ye, Greg Durrett, and Isil Dillig. 2020. Multi-modal Synthesis of Regular Expressions. In Programming Language Design and Implementation (PLDI).Google ScholarGoogle Scholar
  13. Yanju Chen, Ruben Martins, and Yu Feng. 2019. Maximal Multi-layer Specification Synthesis. In Foundations of Software Engineering (FSE). https://doi.org/10.1145/3338906.3338951 Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Jamieson M. Cobleigh, Dimitra Giannakopoulou, and Corina S. Pasareanu. 2003. Learning Assumptions for Compositional Verification. In Tools and Algorithms for the Construction and Analysis of Systems (TACAS). https://doi.org/10.1007/3-540-36577-X_24 Google ScholarGoogle Scholar
  15. Giuseppe De Giacomo and Moshe Y. Vardi. 2013. Linear Temporal Logic and Linear Dynamic Logic on Finite Traces. In International Joint Conference on Artificial Intelligence (IJCAI).Google ScholarGoogle Scholar
  16. Giuseppe De Giacomo and Moshe Y. Vardi. 2015. Synthesis for LTL and LDL on Finite Traces. In International Joint Conference on Artificial Intelligence (IJCAI).Google ScholarGoogle Scholar
  17. Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An Eficient SMT Solver. In Tools and Algorithms for the Construction and Analysis of Systems (TACAS). https://doi.org/10.1007/978-3-540-78800-3_24 Google ScholarGoogle ScholarCross RefCross Ref
  18. Aditya Desai, Sumit Gulwani, Vineet Hingorani, Nidhi Jain, Amey Karkare, Mark Marron, Sailesh R, and Subhajit Roy. 2016. Program synthesis using natural language. In International Conference on Software Engineering (ICSE). https: //doi.org/10.1145/2884781.2884786 Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Dana Drachsler-Cohen, Sharon Shoham, and Eran Yahav. 2017. Synthesis with Abstract Examples. In Computer Aided Verification (CAV). https://doi.org/10.1007/978-3-319-63387-9_13 Google ScholarGoogle ScholarCross RefCross Ref
  20. Matthew B. Dwyer, George S. Avrunin, and James C. Corbett. 1999. Patterns in Property Specifications for Finite-State Verification. In International Conference on Software Engineering (ICSE). https://doi.org/10.1145/302405.302672 Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. R. Fikes and N. J. Nilsson. 1971. STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving. Artif. Intell. 2, 3 /4 ( 1971 ). https://doi.org/10.1016/ 0004-3702 ( 71 ) 90010-5 Google ScholarGoogle ScholarCross RefCross Ref
  22. Ivan Gavran, Rupak Majumdar, and Indranil Saha. 2017. Antlab: A Multi-Robot Task Server. ACM Trans. Embed. Comput. Syst. 16, 5s, Article 190 ( 2017 ). https://doi.org/10.1145/3126513 Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. M. Ghallab, C. Aeronautiques, C. K. Isi, and D. Wilkins. 1998. PDDL: The Planning Domain Definition Language. Technical Report CVC TR98003/DCS TR1165. Yale Center for Computational Vision and Control.Google ScholarGoogle Scholar
  24. E. Mark Gold. 1967. Language Identification in the Limit. Information and Control 10, 5 ( 1967 ). https://doi.org/10.1016/S0019-9958 ( 67 ) 91165-5 Google ScholarGoogle ScholarCross RefCross Ref
  25. Sumit Gulwani and Mark Marron. 2014. NLyze: Interactive Programming by Natural Language for Spreadsheet Data Analysis and Manipulation. In Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data. https://doi.org/10.1145/2588555.2612177 Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Gerard J. Holzmann. 2002. The logic of bugs. In Symposium on Foundations of Software Engineering (FSE). https://doi.org/10. 1145/587051.587064 Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Xiaoqing Jin, Alexandre Donzé, Jyotirmoy V. Deshmukh, and Sanjit A. Seshia. 2015. Mining Requirements From Closed-Loop Control Models. IEEE Trans. on CAD of Integrated Circuits and Systems 34, 11 ( 2015 ). https://doi.org/10.1109/TCAD. 2015. 2421907 Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Joseph Kim, Christian Muise, Ankit Shah, Shubham Agarwal, and Julie Shah. 2019. Bayesian Inference of Linear Temporal Logic Specifications for Contrastive Explanations. In International Joint Conference on Artificial Intelligence (IJCAI). https://doi.org/10.24963/ijcai. 2019 /776 Google ScholarGoogle ScholarCross RefCross Ref
  29. Thomas Kollar, Vittorio Perera, Daniele Nardi, and Manuela M. Veloso. 2013. Learning Environmental Knowledge from Task-Based Human-Robot Dialog. In International Conference on Robotics and Automation (ICRA). https://doi.org/10. 1109/ICRA. 2013.6631186 Google ScholarGoogle ScholarCross RefCross Ref
  30. Zhaodan Kong, Austin Jones, Ana Medina Ayala, Ebru Aydin Gol, and Calin Belta. 2014. Temporal logic inference for classification and prediction from data. In Hybrid Systems: Computation and Control (HSCC). https://doi.org/10.1145/ 2562059.2562146 Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Zhaodan Kong, Austin Jones, and Calin Belta. 2017. Temporal Logics for Learning and Detection of Anomalous Behavior. IEEE Trans. Automat. Contr. 62, 3 ( 2017 ). https://doi.org/10.1109/TAC. 2016.2585083 Google ScholarGoogle ScholarCross RefCross Ref
  32. Hadas Kress-Gazit, Georgios E. Fainekos, and George J. Pappas. 2008. Translating Structured English to Robot Controllers. Advanced Robotics 22, 12 ( 2008 ). https://doi.org/10.1163/156855308X344864 Google ScholarGoogle ScholarCross RefCross Ref
  33. H. Kress-Gazit, G. E. Fainekos, and G. J. Pappas. 2009. Temporal-Logic-Based Reactive Mission and Motion Planning. IEEE Transactions on Robotics ( 2009 ). https://doi.org/10.1109/TRO. 2009.2030225 Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. H.J. Levesque, R. Reiter, Y. Lespérance, F. Lin, and R.B. Scherl. 1997. GOLOG: A Logic Programming Language for Dynamic Domains. J. Log. Program. 31, 1-3 ( 1997 ). https://doi.org/10.1016/S0743-1066 ( 96 ) 00121-5 Google ScholarGoogle ScholarCross RefCross Ref
  35. Fei Li and H. V. Jagadish. 2014. Constructing an Interactive Natural Language Interface for Relational Databases. PVLDB 8, 1 ( 2014 ). https://doi.org/10.14778/2735461.2735468 Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Constantine Lignos, Vasumathi Raman, Cameron Finucane, Mitchell P. Marcus, and Hadas Kress-Gazit. 2015. Provably Correct Reactive Control from Natural Language. Auton. Robots 38, 1 ( 2015 ). https://doi.org/10.1007/s10514-014-9418-8 Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. In Association for Computational Linguistics (ACL) System Demonstrations. https://doi.org/10.3115/v1/ P14-5010 Google ScholarGoogle ScholarCross RefCross Ref
  38. Mehdi Hafezi Manshadi, Daniel Gildea, and James F. Allen. 2013. Integrating Programming by Example and Natural Language Programming. In Conference on Artificial Intelligence (AAAI).Google ScholarGoogle Scholar
  39. Mikaël Mayer, Gustavo Soares, Maxim Grechkin, Vu Le, Mark Marron, Oleksandr Polozov, Rishabh Singh, Benjamin Zorn, and Sumit Gulwani. 2015. User Interaction Models for Disambiguation in Programming by Example. In User Interface Software & Technology (UIST). https://doi.org/10.1145/2807442.2807459 Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Çetin Meriçli, Steven D. Klee, Jack Paparian, and Manuela M. Veloso. 2014. An interactive approach for situated task specification through verbal instructions. In Autonomous Agents and Multi-Agent Systems (AAMAS).Google ScholarGoogle Scholar
  41. George A. Miller. 1995. WordNet: A Lexical Database for English. Commun. ACM 38, 11 ( 1995 ). https://doi.org/10.1145/ 219717.219748 Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Sara Mohammadinejad, Jyotirmoy V. Deshmukh, Aniruddh G. Puranic, Marcell Vazquez-Chanlatte, and Alexandre Donzé. 2020. Interpretable classification of time-series data using eficient enumerative techniques. In Hybrid Systems: Computation and Control (HSCC). https://doi.org/10.1145/3365365.3382218 Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Daniel Neider and Ivan Gavran. 2018. Learning Linear Temporal Properties. In FMCAD (Formal Methods in Computer Aided Design). https://doi.org/10.23919/FMCAD. 2018.8603016 Google ScholarGoogle ScholarCross RefCross Ref
  44. Maxwell I. Nye, Luke B. Hewitt, Joshua B. Tenenbaum, and Armando Solar-Lezama. 2019. Learning to Infer Program Sketches. In International Conference on Machine Learning (ICML).Google ScholarGoogle Scholar
  45. Hila Peleg, Sharon Shoham, and Eran Yahav. 2018. Programming Not Only by Example. In International Conference on Software Engineering (ICSE). https://doi.org/10.1145/3180155.3180189 Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Vittorio Perera and Manuela M. Veloso. 2015. Handling Complex Commands as Service Robot Task Requests. In International Joint Conference on Artificial Intelligence (IJCAI).Google ScholarGoogle Scholar
  47. Amir Pnueli. 1977. The temporal logic of programs. In Foundations of Computer Science, 1977., 18th Annual Symposium on. IEEE. https://doi.org/10.1109/SFCS. 1977.32 Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Illia Polosukhin and Alexander Skidanov. 2018. Neural Program Search: Solving Programming Tasks from Description and Examples. In International Conference on Learning Representations (ICLR).Google ScholarGoogle Scholar
  49. Mohammad Raza, Sumit Gulwani, and Natasa Milic-Frayling. 2015. Compositional Program Synthesis from Natural Language and Examples. In International Joint Conference on Artificial Intelligence (IJCAI).Google ScholarGoogle Scholar
  50. Indranil Saha, Rattanachai Ramaithitima, Vijay Kumar, George J. Pappas, and Sanjit A. Seshia. 2016. Implan: Scalable Incremental Motion Planning for Multi-Robot Systems. In International Conference on Cyber-Physical Systems (ICCPS). https://doi.org/10.1109/ICCPS. 2016.7479105 Google ScholarGoogle ScholarCross RefCross Ref
  51. Rishabh Singh and Sumit Gulwani. 2015. Predicting a Correct Program in Programming by Example. In Computer Aided Verification (CAV). https://doi.org/10.1007/978-3-319-21690-4_23 Google ScholarGoogle ScholarCross RefCross Ref
  52. Jesse Thomason, Shiqi Zhang, Raymond J. Mooney, and Peter Stone. 2015. Learning to Interpret Natural Language Commands through Human-Robot Dialog. In International Joint Conference on Artificial Intelligence (IJCAI).Google ScholarGoogle Scholar
  53. Marcell Vazquez-Chanlatte, Susmit Jha, Ashish Tiwari, Mark K. Ho, and Sanjit A. Seshia. 2018. Learning Task Specifications from Demonstrations. In Neural Information Processing Systems (NeurIPS).Google ScholarGoogle Scholar
  54. Chenglong Wang, Alvin Cheung, and Rastislav Bodík. 2017a. Interactive Query Synthesis from Input-Output Examples. In SIGMOD Conference. https://doi.org/10.1145/3035918.3058738 Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Chenglong Wang, Alvin Cheung, and Rastislav Bodík. 2017b. Synthesizing highly expressive SQL queries from input-output examples. In Programming Language Design and Implementation (PLDI). https://doi.org/10.1145/3140587.3062365 Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. Sida I. Wang, Samuel Ginn, Percy Liang, and Christopher D. Manning. 2017c. Naturalizing a Programming Language via Interactive Learning. In Annual Meeting of the Association for Computational Linguistics (ACL). https://doi.org/10.18653/ v1/ P17-1086 Google ScholarGoogle ScholarCross RefCross Ref
  57. Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. 2017. SQLizer: query synthesis from natural language. PACMPL 1, OOPSLA ( 2017 ). https://doi.org/10.1145/3133887 Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Interactive synthesis of temporal specifications from examples and natural language

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!