skip to main content
article

Programmatic and direct manipulation, together at last

Published:02 June 2016Publication History
Skip Abstract Section

Abstract

Direct manipulation interfaces and programmatic systems have distinct and complementary strengths. The former provide intuitive, immediate visual feedback and enable rapid prototyping, whereas the latter enable complex, reusable abstractions. Unfortunately, existing systems typically force users into just one of these two interaction modes. We present a system called Sketch-n-Sketch that integrates programmatic and direct manipulation for the particular domain of Scalable Vector Graphics (SVG). In Sketch-n-Sketch, the user writes a program to generate an output SVG canvas. Then the user may directly manipulate the canvas while the system immediately infers a program update in order to match the changes to the output, a workflow we call live synchronization. To achieve this, we propose (i) a technique called trace-based program synthesis that takes program execution history into account in order to constrain the search space and (ii) heuristics for dealing with ambiguities. Based on our experience with examples spanning 2,000 lines of code and from the results of a preliminary user study, we believe that Sketch-n-Sketch provides a novel workflow that can augment traditional programming systems. Our approach may serve as the basis for live synchronization in other application domains, as well as a starting point for yet more ambitious ways of combining programmatic and direct manipulation.

References

  1. Daniel W. Barowy, Sumit Gulwani, Ted Hart, and Benjamin Zorn. FlashRelate: Extracting Relational Data from Semi-Structured Spreadsheets Using Examples. In Conference on Programming Language Design and Implementation (PLDI), 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Michel Beaudouin-Lafon. Instrumental Interaction: An Interaction Model for Designing Post-WIMP User Interfaces. In Conference on Human Factors in Computing Systems (CHI), 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Ben Fry and Casey Reas. Processing. https:// processing.org/.Google ScholarGoogle Scholar
  4. Gilbert Louis Bernstein and Wilmot Li. Lillicon: Using Transient Widgets to Create Scale Variations of Icons. Transactions on Graphics (TOG), 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Alan Borning. The Programming Language Aspects of ThingLab. Transactions on Programming Languages and Systems (TOPLAS), October 1981. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Alan Borning and Bert Freudenberg. ThingLab. https: //github.com/cdglabs/thinglab.Google ScholarGoogle Scholar
  7. Bret Victor. Drawing Dynamic Visualizations. http:// worrydream.com/.Google ScholarGoogle Scholar
  8. Satish Chandra, Emina Torlak, Shaon Barman, and Rastislav Bodik. Angelic Debugging. In International Conference on Software Engineering (ICSE), 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Salman Cheema, Sumit Gulwani, and Joseph LaViola. Quick-Draw: Improving Drawing Experience for Geometric Diagrams. In Conference on Human Factors in Computing Systems (CHI), 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Salman Cheema, Sarah Buchanan, Sumit Gulwani, and Joseph J. LaViola, Jr. A Practical Framework for Constructing Structured Drawings. In International Conference on Intelligent User Interfaces (IUI), 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Yan Chen, Joshua Dunfield, Matthew A. Hammer, and Umut A. Acar. Implicit Self-Adjusting Computation for Purely Functional Programs. In International Conference on Functional Programming (ICFP), 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Ravi Chugh. Prodirect Manipulation: Bidirectional Programming for the Masses. In International Conference on Software Engineering, Visions of 2025 and Beyond Track (ICSE V2025), 2016. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Ravi Chugh, Brian Hempel, Mitchell Spradlin, and Jacob Albers. Programmatic and Direct Manipulation, Together at Last (Supplementary Appendices). http://arxiv.org/ abs/1507.02988, 2016. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Leonardo de Moura and Nikolaj Bjørner. Z3: An Efficient SMT solver. In Tools and Algorithms for the Construction and Analysis of Systems (TACAS), 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. DrawScript. http://drawscri.pt.Google ScholarGoogle Scholar
  16. Evan Czaplicki. Elm. http://elm-lang.org.Google ScholarGoogle Scholar
  17. GNU Project. The gnu image manipulation program. http: //www.gimp.org/.Google ScholarGoogle Scholar
  18. Sumit Gulwani. Automating String Processing in Spreadsheets Using Input-Output Examples. In Symposium on Principles of Programming Languages (POPL), 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Sumit Gulwani, Vijay Anand Korthikanti, and Ashish Tiwari. Synthesizing Geometry Constructions. In Programming Language Design and Implementation (PLDI), 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Hesam Samimi and Alex Warth. Sketchpad14. http://www. cdglabs.org/sketchpad14.Google ScholarGoogle Scholar
  21. Thibaud Hottelier, Ras Bodik, and Kimiko Ryokai. Programming by Manipulation for Layout. In Symposium on User Interface Software and Technology (UIST), 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Zhenjiang Hu, Andy Schurr, Perdita Stevens, and James F. Terwilliger. Dagstuhl Seminar on Bidirectional Transformations (BX). SIGMOD, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Etienne Kneuss, Viktor Kuncak, Ivan Kuraj, and Philippe Suter. Synthesis Modulo Recursive Functions. In Conference on Object-Oriented Programming Languages, Systems, and Applications (OOPSLA), 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Viktor Kuncak, Mikael Mayer, Ruzica Piskac, and Philippe Suter. Complete Functional Synthesis. In Conference on Programming Language Design and Implementation (PLDI), 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Bum Chul Kwon, Waqas Javed, Niklas Elmqvist, and Ji Soo Yi. Direct Manipulation Through Surrogate Objects. In Conference on Human Factors in Computing Systems (CHI), 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Mika¨el Mayer, Gustavo Soares, Maxim Grechkin, Vu Le, Mark Marron, Oleksandr Polozov, Rishabh Singh, Benjamin Zorn, and Sumit Gulwani. User Interaction Models for Disambiguation in Programming by Example. In Symposium on User Interface Software and Technology (UIST), 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. PaintCode. http://www.paintcodeapp.com.Google ScholarGoogle Scholar
  28. Processing.js. http://processingjs.org/.Google ScholarGoogle Scholar
  29. Ben Shneiderman. Direct Manipulation: A Step Beyond Programming Languages. Computer, August 1983. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Armando Solar-Lezama. Program Synthesis by Sketching. PhD thesis, UC Berkeley, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Saurabh Srivastava, Sumit Gulwani, and Jeffrey S. Foster. From Program Verification to Program Synthesis. In Symposium on Principles of Programming Languages (POPL), 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Ivan Sutherland. Sketchpad, A Man-Machine Graphical Communication System. PhD thesis, MIT, 1963.Google ScholarGoogle Scholar
  33. SVG-edit. https://code.google.com/p/svg-edit/.Google ScholarGoogle Scholar
  34. Toby Schachman. Apparatus. http://aprt.us/.Google ScholarGoogle Scholar

Index Terms

  1. Programmatic and direct manipulation, together at last

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM SIGPLAN Notices
          ACM SIGPLAN Notices  Volume 51, Issue 6
          PLDI '16
          June 2016
          726 pages
          ISSN:0362-1340
          EISSN:1558-1160
          DOI:10.1145/2980983
          • Editor:
          • Andy Gill
          Issue’s Table of Contents
          • cover image ACM Conferences
            PLDI '16: Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation
            June 2016
            726 pages
            ISBN:9781450342612
            DOI:10.1145/2908080
            • General Chair:
            • Chandra Krintz,
            • Program Chair:
            • Emery Berger

          Copyright © 2016 ACM

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 2 June 2016

          Check for updates

          Qualifiers

          • article

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!