skip to main content
research-article

Test-driven synthesis

Published:09 June 2014Publication History
Skip Abstract Section

Abstract

Programming-by-example technologies empower end-users to create simple programs merely by providing input/output examples. Existing systems are designed around solvers specialized for a specific set of data types or domain-specific language (DSL). We present a program synthesizer which can be parameterized by an arbitrary DSL that may contain conditionals and loops and therefore is able to synthesize programs in any domain. In order to use our synthesizer, the user provides a sequence of increasingly sophisticated input/output examples along with an expert-written DSL definition. These two inputs correspond to the two key ideas that allow our synthesizer to work in arbitrary domains. First, we developed a novel iterative synthesis technique inspired by test-driven development---which also gives our technique the name of test-driven synthesis---where the input/output examples are consumed one at a time as the program is refined. Second, the DSL allows our system to take an efficient component-based approach to enumerating possible programs. We present applications of our synthesis methodology to end-user programming for transformations over strings, XML, and table layouts. We compare our synthesizer on these applications to state-of-the-art DSL-specific synthesizers as well to the general purpose synthesizer Sketch.

References

  1. http://research.microsoft.com/en-us/um/people/sumitg/flashfill.html.Google ScholarGoogle Scholar
  2. R. Alur, R. Bodik, G. Juniwal, M. M. Martin, M. Raghothaman, S. A. Seshia, R. Singh, A. Solar-Lezama, E. Torlak, and A. Udupa. Syntax-guided synthesis. In FMCAD, 2013.Google ScholarGoogle ScholarCross RefCross Ref
  3. S. Chandra, E. Torlak, S. Barman, and R. Bodik. Angelic debugging. ICSE, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. A. Cypher, D. Halbert, D. Kurlander, H. Lieberman, D. Maulsby, B. Myers, and A. Turransky. Watch What I Do: Programming by Demonstration. MIT press, 1993. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. T. Granlund and R. Kenner. Eliminating branches using a superoptimizer and the GNU C compiler. PLDI, 1992. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. S. Gulwani. Automating string processing in spreadsheets using input-output examples. POPL, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. S. Gulwani. Synthesis from examples: Interaction models and algorithms. SYNASC, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. S. Gulwani, W. R. Harris, and R. Singh. Spreadsheet data manipulation using examples. Commun. ACM, 55(8), Aug. 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. S. Gulwani, S. Jha, A. Tiwari, and R. Venkatesan. Synthesis of loop-free programs. PLDI, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. S. Gulwani, V. Korthikanti, and A. Tiwari. Synthesizing geometry constructions. PLDI, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. W. R. Harris and S. Gulwani. Spreadsheet table transformations from examples. PLDI, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. D. Janzen and H. Saiedian. Test-driven development concepts, taxonomy, and future direction. Computer, 38(9), sept. 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. S. Jha, S. Gulwani, S. Seshia, and A. Tiwari. Oracle-guided component-based program synthesis. ICSE, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. S. Katayama. Systematic search for lambda expressions. TFP, 2005.Google ScholarGoogle Scholar
  15. S. Katayama. Efficient exhaustive generation of functional programs using monte-carlo search with iterative deepening. In PRICAI. 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. E. Kitzelmann. Inductive programming: A survey of program synthesis techniques. In U. Schmid, E. Kitzelmann, and R. Plasmeijer, editors, Approaches and Applications of Inductive Programming, volume 5812 of Lecture Notes in Computer Science. 2010.Google ScholarGoogle Scholar
  17. T. Lau et al. Why PBD systems fail: Lessons learned for usable AI. In CHI 2008 Workshop on Usable AI, 2008.Google ScholarGoogle Scholar
  18. T. Lau, S. Wolfman, P. Domingos, and D. Weld. Programming by demonstration using version space algebra. Machine Learning, 53(1), 2003. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer. Genprog: A generic method for automatic software repair. IEEE Trans. Softw. Eng., 38(1), Jan. 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. H. Lieberman. Your wish is my command: Programming by example. Morgan Kaufmann, 2001.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. D. Mandelin, L. Xu, R. Bodík, and D. Kimelman. Jungloid mining: helping to navigate the API jungle. PLDI, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. R. Martin. The transformation priority premise. http://blog.8thlight.com/uncle-bob/2013/05/27/TheTransformationPriorityPremise.html, 2010.Google ScholarGoogle Scholar
  23. R. I. Mckay, N. X. Hoai, P. A. Whigham, Y. Shan, and M. ONeill. Grammar-based genetic programming: a survey. Genetic Programming and Evolvable Machines, 11(3--4):365--396, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. A. Menon, O. Tamuz, S. Gulwani, B. Lampson, and A. Kalai. A machine learning framework for programming by example. ICML, 2013.Google ScholarGoogle Scholar
  25. R. Olsson. Inductive functional programming using incremental program transformation. Artif. Intell., 74(1), Mar. 1995. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. R. Panigrahy and L. Zhang. The mind grows circuits. CoRR, abs/1203.0088, 2012.Google ScholarGoogle Scholar
  27. D. Perelman, S. Gulwani, T. Ball, and D. Grossman. Type-directed completion of partial expressions. PLDI. ACM, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. R. Singh and S. Gulwani. Learning semantic string transformations from examples. PVLDB, 5(8), 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. R. Singh and S. Gulwani. Synthesizing number transformations from input-output examples. In CAV, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. A. Solar Lezama. Program Synthesis By Sketching. PhD thesis, EECS Department, University of California, Berkeley, Dec 2008.Google ScholarGoogle Scholar
  31. N. Tillmann and J. De Halleux. Pex: white box test generation for .NET. TAP, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. N. Tillmann, J. de Halleux, T. Xie, and J. Bishop. Pex4Fun: Teaching and learning computer science via social gaming. CSEE&T, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. E. Torlak and R. Bodik. Growing solver-aided languages with Rosette. In Proceedings of the 2013 ACM international symposium on New ideas, new paradigms, and reflections on programming & software, pages 135--152. ACM, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. W. Weimer, S. Forrest, C. Le Goues, and T. Nguyen. Automatic program repair with evolutionary computation. Commun. ACM, 53(5), May 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. W. Weimer, T. Nguyen, C. Le Goues, and S. Forrest. Automatically finding patches using genetic programming. ICSE, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library

Recommendations

Comments

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Sign in

Full Access

  • Published in

    cover image ACM SIGPLAN Notices
    ACM SIGPLAN Notices  Volume 49, Issue 6
    PLDI '14
    June 2014
    598 pages
    ISSN:0362-1340
    EISSN:1558-1160
    DOI:10.1145/2666356
    • Editor:
    • Andy Gill
    Issue’s Table of Contents
    • cover image ACM Conferences
      PLDI '14: Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation
      June 2014
      619 pages
      ISBN:9781450327848
      DOI:10.1145/2594291

    Copyright © 2014 ACM

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    • Published: 9 June 2014

    Check for updates

    Qualifiers

    • research-article

PDF Format

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader
About Cookies On This Site

We use cookies to ensure that we give you the best experience on our website.

Learn more

Got it!