Abstract
Direct manipulation interfaces and programmatic systems have distinct and complementary strengths. The former provide intuitive, immediate visual feedback and enable rapid prototyping, whereas the latter enable complex, reusable abstractions. Unfortunately, existing systems typically force users into just one of these two interaction modes. We present a system called Sketch-n-Sketch that integrates programmatic and direct manipulation for the particular domain of Scalable Vector Graphics (SVG). In Sketch-n-Sketch, the user writes a program to generate an output SVG canvas. Then the user may directly manipulate the canvas while the system immediately infers a program update in order to match the changes to the output, a workflow we call live synchronization. To achieve this, we propose (i) a technique called trace-based program synthesis that takes program execution history into account in order to constrain the search space and (ii) heuristics for dealing with ambiguities. Based on our experience with examples spanning 2,000 lines of code and from the results of a preliminary user study, we believe that Sketch-n-Sketch provides a novel workflow that can augment traditional programming systems. Our approach may serve as the basis for live synchronization in other application domains, as well as a starting point for yet more ambitious ways of combining programmatic and direct manipulation.
- Daniel W. Barowy, Sumit Gulwani, Ted Hart, and Benjamin Zorn. FlashRelate: Extracting Relational Data from Semi-Structured Spreadsheets Using Examples. In Conference on Programming Language Design and Implementation (PLDI), 2015. Google Scholar
Digital Library
- Michel Beaudouin-Lafon. Instrumental Interaction: An Interaction Model for Designing Post-WIMP User Interfaces. In Conference on Human Factors in Computing Systems (CHI), 2000. Google Scholar
Digital Library
- Ben Fry and Casey Reas. Processing. https:// processing.org/.Google Scholar
- Gilbert Louis Bernstein and Wilmot Li. Lillicon: Using Transient Widgets to Create Scale Variations of Icons. Transactions on Graphics (TOG), 2015. Google Scholar
Digital Library
- Alan Borning. The Programming Language Aspects of ThingLab. Transactions on Programming Languages and Systems (TOPLAS), October 1981. Google Scholar
Digital Library
- Alan Borning and Bert Freudenberg. ThingLab. https: //github.com/cdglabs/thinglab.Google Scholar
- Bret Victor. Drawing Dynamic Visualizations. http:// worrydream.com/.Google Scholar
- Satish Chandra, Emina Torlak, Shaon Barman, and Rastislav Bodik. Angelic Debugging. In International Conference on Software Engineering (ICSE), 2011. Google Scholar
Digital Library
- Salman Cheema, Sumit Gulwani, and Joseph LaViola. Quick-Draw: Improving Drawing Experience for Geometric Diagrams. In Conference on Human Factors in Computing Systems (CHI), 2012. Google Scholar
Digital Library
- Salman Cheema, Sarah Buchanan, Sumit Gulwani, and Joseph J. LaViola, Jr. A Practical Framework for Constructing Structured Drawings. In International Conference on Intelligent User Interfaces (IUI), 2014. Google Scholar
Digital Library
- Yan Chen, Joshua Dunfield, Matthew A. Hammer, and Umut A. Acar. Implicit Self-Adjusting Computation for Purely Functional Programs. In International Conference on Functional Programming (ICFP), 2011. Google Scholar
Digital Library
- Ravi Chugh. Prodirect Manipulation: Bidirectional Programming for the Masses. In International Conference on Software Engineering, Visions of 2025 and Beyond Track (ICSE V2025), 2016. Google Scholar
Digital Library
- Ravi Chugh, Brian Hempel, Mitchell Spradlin, and Jacob Albers. Programmatic and Direct Manipulation, Together at Last (Supplementary Appendices). http://arxiv.org/ abs/1507.02988, 2016. Google Scholar
Digital Library
- Leonardo de Moura and Nikolaj Bjørner. Z3: An Efficient SMT solver. In Tools and Algorithms for the Construction and Analysis of Systems (TACAS), 2008. Google Scholar
Digital Library
- DrawScript. http://drawscri.pt.Google Scholar
- Evan Czaplicki. Elm. http://elm-lang.org.Google Scholar
- GNU Project. The gnu image manipulation program. http: //www.gimp.org/.Google Scholar
- Sumit Gulwani. Automating String Processing in Spreadsheets Using Input-Output Examples. In Symposium on Principles of Programming Languages (POPL), 2011. Google Scholar
Digital Library
- Sumit Gulwani, Vijay Anand Korthikanti, and Ashish Tiwari. Synthesizing Geometry Constructions. In Programming Language Design and Implementation (PLDI), 2011. Google Scholar
Digital Library
- Hesam Samimi and Alex Warth. Sketchpad14. http://www. cdglabs.org/sketchpad14.Google Scholar
- Thibaud Hottelier, Ras Bodik, and Kimiko Ryokai. Programming by Manipulation for Layout. In Symposium on User Interface Software and Technology (UIST), 2014. Google Scholar
Digital Library
- Zhenjiang Hu, Andy Schurr, Perdita Stevens, and James F. Terwilliger. Dagstuhl Seminar on Bidirectional Transformations (BX). SIGMOD, 2011. Google Scholar
Digital Library
- Etienne Kneuss, Viktor Kuncak, Ivan Kuraj, and Philippe Suter. Synthesis Modulo Recursive Functions. In Conference on Object-Oriented Programming Languages, Systems, and Applications (OOPSLA), 2013. Google Scholar
Digital Library
- Viktor Kuncak, Mikael Mayer, Ruzica Piskac, and Philippe Suter. Complete Functional Synthesis. In Conference on Programming Language Design and Implementation (PLDI), 2010. Google Scholar
Digital Library
- Bum Chul Kwon, Waqas Javed, Niklas Elmqvist, and Ji Soo Yi. Direct Manipulation Through Surrogate Objects. In Conference on Human Factors in Computing Systems (CHI), 2011. Google Scholar
Digital Library
- Mika¨el Mayer, Gustavo Soares, Maxim Grechkin, Vu Le, Mark Marron, Oleksandr Polozov, Rishabh Singh, Benjamin Zorn, and Sumit Gulwani. User Interaction Models for Disambiguation in Programming by Example. In Symposium on User Interface Software and Technology (UIST), 2015. Google Scholar
Digital Library
- PaintCode. http://www.paintcodeapp.com.Google Scholar
- Processing.js. http://processingjs.org/.Google Scholar
- Ben Shneiderman. Direct Manipulation: A Step Beyond Programming Languages. Computer, August 1983. Google Scholar
Digital Library
- Armando Solar-Lezama. Program Synthesis by Sketching. PhD thesis, UC Berkeley, 2008. Google Scholar
Digital Library
- Saurabh Srivastava, Sumit Gulwani, and Jeffrey S. Foster. From Program Verification to Program Synthesis. In Symposium on Principles of Programming Languages (POPL), 2010. Google Scholar
Digital Library
- Ivan Sutherland. Sketchpad, A Man-Machine Graphical Communication System. PhD thesis, MIT, 1963.Google Scholar
- SVG-edit. https://code.google.com/p/svg-edit/.Google Scholar
- Toby Schachman. Apparatus. http://aprt.us/.Google Scholar
Index Terms
Programmatic and direct manipulation, together at last
Recommendations
Sketch-n-Sketch: Output-Directed Programming for SVG
UIST '19: Proceedings of the 32nd Annual ACM Symposium on User Interface Software and TechnologyFor creative tasks, programmers face a choice: Use a GUI and sacrifice flexibility, or write code and sacrifice ergonomics?
To obtain both flexibility and ease of use, a number of systems have explored a workflow that we call output-directed ...
Semi-Automated SVG Programming via Direct Manipulation
UIST '16: Proceedings of the 29th Annual Symposium on User Interface Software and TechnologyDirect manipulation interfaces provide intuitive and interactive features to a broad range of users, but they often exhibit two limitations: the built-in features cannot possibly cover all use cases, and the internal representation of the content is not ...
Bidirectional evaluation with direct manipulation
We present an evaluation update (or simply, update) algorithm for a full-featured functional programming language, which synthesizes program changes based on output changes. Intuitively, the update algorithm retraces the steps of the original evaluation,...







Comments