Abstract
While visualizations play a crucial role in gaining insights from data, generating useful visualizations from a complex dataset is far from an easy task. In particular, besides understanding the functionality provided by existing visualization libraries, generating the desired visualization also requires reshaping and aggregating the underlying data as well as composing different visual elements to achieve the intended visual narrative. This paper aims to simplify visualization tasks by automatically synthesizing the required program from simple visual sketches provided by the user. Specifically, given an input data set and a visual sketch that demonstrates how to visualize a very small subset of this data, our technique automatically generates a program that can be used to visualize the entire data set.
From a program synthesis perspective, automating visualization tasks poses several challenges that are not addressed by prior techniques. First, because many visualization tasks require data wrangling in addition to generating plots from a given table, we need to decompose the end-to-end synthesis task into two separate sub-problems. Second, because the intermediate specification that results from the decomposition is necessarily imprecise, this makes the data wrangling task particularly challenging in our context. In this paper, we address these problems by developing a new compositional visualization-by-example technique that (a) decomposes the end-to-end task into two different synthesis problems over different DSLs and (b) leverages bi-directional program analysis to deal with the complexity that arises from having an imprecise intermediate specification.
We have implemented our visualization-by-example approach in a tool called Viser and evaluate it on over 80 visualization tasks collected from on-line forums and tutorials. Viser can solve 84 of these benchmarks within a 600 second time limit, and, for those tasks that can be solved, the desired visualization is among the top-5 generated by Viser in 70% of the cases.
Supplemental Material
- Michael Bostock, Vadim Ogievetsky, and Jeffrey Heer. 2011. D 3 Data-Driven Documents. IEEE Trans. Vis. Comput. Graph. 17, 12 (2011), 2301–2309. Google Scholar
Digital Library
- H Carr, P Rheingans, H Schumann, Arvind Satyanarayan, and Jeffrey Heer. 2014. Lyra: An Interactive Visualization Design Environment. In Eurographics Conference on Visualization, Vol. 33. 10.Google Scholar
- Satish Chandra, Stephen J. Fink, and Manu Sridharan. 2009. Snugglebug: a powerful approach to weakest preconditions. In Proceedings of the 2009 ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2009, Dublin, Ireland, June 15-21, 2009. 363–374. Google Scholar
Digital Library
- Dinakar Dhurjati, Manuvir Das, and Yue Yang. 2006. Path-Sensitive Dataflow Analysis with Iterative Refinement. In Static Analysis, 13th International Symposium, SAS 2006, Seoul, Korea, August 29-31, 2006, Proceedings. 425–442. Google Scholar
Digital Library
- Yu Feng, Ruben Martins, Osbert Bastani, and Isil Dillig. 2018. Program synthesis using conflict-driven learning. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2018, Philadelphia, PA, USA, June 18-22, 2018. 420–435.Google Scholar
Digital Library
- Yu Feng, Ruben Martins, Jacob Van Geffen, Isil Dillig, and Swarat Chaudhuri. 2017. Component-based synthesis of table consolidation and transformation tasks from examples. In Proc. Conference on Programming Language Design and Implementation. ACM, 422–436.Google Scholar
Digital Library
- John K. Feser, Swarat Chaudhuri, and Isil Dillig. 2015. Synthesizing data structure transformations from input-output examples. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation, Portland, OR, USA, June 15-17, 2015. 229–239. Google Scholar
Digital Library
- Pat Hanrahan. 2006. VizQL: a language for query, analysis and visualization. In Proceedings of the ACM SIGMOD International Conference on Management of Data, Chicago, Illinois, USA, June 27-29, 2006. 721.Google Scholar
Digital Library
- William R Harris and Sumit Gulwani. 2011. Spreadsheet table transformations from examples. In ACM SIGPLAN Notices, Vol. 46. ACM, 317–328.Google Scholar
Digital Library
- Jock D. Mackinlay, Pat Hanrahan, and Chris Stolte. 2007. Show Me: Automatic Presentation for Visual Analysis. IEEE Trans. Vis. Comput. Graph. 13, 6 (2007), 1137–1144. Google Scholar
Digital Library
- Solomon Maina, Anders Miltner, Kathleen Fisher, Benjamin C. Pierce, David Walker, and Steve Zdancewic. 2018. Synthesizing quotient lenses. PACMPL 2, ICFP (2018), 80:1–80:29. Google Scholar
Digital Library
- Ruben Martins, Jia Chen, Yanju Chen, Yu Feng, and Isil Dillig. 2019. Trinity: An Extensible Synthesis Framework for Data Science. https://github.com/fredfeng/Trinity/ .Google Scholar
- Anders Miltner, Kathleen Fisher, Benjamin C. Pierce, David Walker, and Steve Zdancewic. 2018. Synthesizing bijective lenses. PACMPL 2, POPL (2018), 1:1–1:30. Google Scholar
Digital Library
- Dominik Moritz, Chenglong Wang, Greg L. Nelson, Halden Lin, Adam M. Smith, Bill Howe, and Jeffrey Heer. 2019. Formalizing Visualization Design Knowledge as Constraints: Actionable and Extensible Models in Draco. IEEE Trans. Vis. Comput. Graph. 25, 1 (2019), 438–448. Google Scholar
Digital Library
- Phitchaya Mangpo Phothilimthana, Aditya Thakur, Rastislav Bodík, and Dinakar Dhurjati. 2016. Scaling up Superoptimization. In Proceedings of the Twenty-First International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’16, Atlanta, GA, USA, April 2-6, 2016. 297–310. Google Scholar
Digital Library
- Nadia Polikarpova, Ivan Kuraj, and Armando Solar-Lezama. 2016. Program synthesis from polymorphic refinement types. Proc. Conference on Programming Language Design and Implementation (2016), 522–538.Google Scholar
Digital Library
- Oleksandr Polozov and Sumit Gulwani. 2015. FlashMeta: a framework for inductive program synthesis. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2015, part of SPLASH 2015, Pittsburgh, PA, USA, October 25-30, 2015. 107–126. Google Scholar
Digital Library
- Mohammad Raza, Sumit Gulwani, and Natasa Milic-Frayling. 2015. Compositional Program Synthesis from Natural Language and Examples. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015. 792–800. http://ijcai.org/Abstract/15/117Google Scholar
- Thomas W. Reps, Susan Horwitz, and Shmuel Sagiv. 1995. Precise Interprocedural Dataflow Analysis via Graph Reachability. In Conference Record of POPL’95: 22nd ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, San Francisco, California, USA, January 23-25, 1995. 49–61. Google Scholar
Digital Library
- Bahador Saket, Hannah Kim, Eli T. Brown, and Alex Endert. 2017a. Visualization by Demonstration: An Interaction Paradigm for Visual Data Exploration. IEEE Trans. Vis. Comput. Graph. 23, 1 (2017), 331–340. Google Scholar
Digital Library
- Bahador Saket, Hannah Kim, Eli T. Brown, and Alex Endert. 2017b. Visualization by Demonstration: An Interaction Paradigm for Visual Data Exploration. IEEE Trans. Vis. Comput. Graph. 23, 1 (2017), 331–340. Google Scholar
Digital Library
- Arvind Satyanarayan and Jeffrey Heer. 2014. Lyra: An Interactive Visualization Design Environment. Comput. Graph. Forum 33, 3 (2014), 351–360. Google Scholar
Cross Ref
- Arvind Satyanarayan, Dominik Moritz, Kanit Wongsuphasawat, and Jeffrey Heer. 2017. Vega-Lite: A Grammar of Interactive Graphics. IEEE Trans. Vis. Comput. Graph. 23, 1 (2017), 341–350. Google Scholar
Digital Library
- David Schroeder and Daniel F. Keefe. 2016. Visualization-by-Sketching: An Artist’s Interface for Creating Multivariate Time-Varying Data Visualizations. IEEE Trans. Vis. Comput. Graph. 22, 1 (2016), 877–885. Google Scholar
Digital Library
- Chris Stolte, Diane Tang, and Pat Hanrahan. 2008. Polaris: a system for query, analysis, and visualization of multidimensional databases. Commun. ACM 51, 11 (2008), 75–84. Google Scholar
Digital Library
- Quoc Trung Tran, Chee-Yong Chan, and Srinivasan Parthasarathy. 2009. Query by output. In Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2009, Providence, Rhode Island, USA, June 29 - July 2, 2009. 535–548. Google Scholar
Digital Library
- Chenglong Wang, Alvin Cheung, and Rastislav Bodik. 2017a. Synthesizing highly expressive SQL queries from input-output examples. In Proc. Conference on Programming Language Design and Implementation. ACM, 452–466.Google Scholar
Digital Library
- Chenglong Wang, Alvin Cheung, and Rastislav Bodík. 2018a. Speeding up symbolic reasoning for relational queries. PACMPL 2, OOPSLA (2018), 157:1–157:25. Google Scholar
Digital Library
- Xinyu Wang, Isil Dillig, and Rishabh Singh. 2017b. Synthesis of data completion scripts using finite tree automata. PACMPL 1, OOPSLA (2017), 62:1–62:26. Google Scholar
Digital Library
- Xinyu Wang, Isil Dillig, and Rishabh Singh. 2018b. Program Synthesis using Abstraction Refinement. In Proc. Symposium on Principles of Programming Languages. ACM, 63:1–63:30.Google Scholar
Digital Library
- Kanit Wongsuphasawat, Dominik Moritz, Anushka Anand, Jock D. Mackinlay, Bill Howe, and Jeffrey Heer. 2016a. Towards a general-purpose query language for visualization recommendation. In Proceedings of the Workshop on Human-In-the-Loop Data Analytics, [email protected] 2016, San Francisco, CA, USA, June 26 - July 01, 2016. 4. Google Scholar
Digital Library
- Kanit Wongsuphasawat, Dominik Moritz, Anushka Anand, Jock D. Mackinlay, Bill Howe, and Jeffrey Heer. 2016b. Voyager: Exploratory Analysis via Faceted Browsing of Visualization Recommendations. IEEE Trans. Vis. Comput. Graph. 22, 1 (2016), 649–658. Google Scholar
Digital Library
- Kanit Wongsuphasawat, Zening Qu, Dominik Moritz, Riley Chang, Felix Ouk, Anushka Anand, Jock D. Mackinlay, Bill Howe, and Jeffrey Heer. 2017. Voyager 2: Augmenting Visual Analysis with Partial View Specifications. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver, CO, USA, May 06-11, 2017. 2648–2659. Google Scholar
Digital Library
- Sai Zhang and Yuyin Sun. 2013. Automatically synthesizing sql queries from input-output examples. In Proc. International Conference on Automated Software Engineering. IEEE, 224–234.Google Scholar
Digital Library
Index Terms
Visualization by example
Recommendations
TimeLine and visualization of multiple-data sets and the visualization querying challenge
Data in its raw form can potentially contain valuable information, but much of that value is lost if it cannot be presented to a user in a way that is useful and meaningful. Data visualization techniques offer a solution to this issue. Such methods are ...
Real-time Data Visualization of Intelligent Networked Vehicles
CNIOT '20: Proceedings of the 2020 International Conference on Computing, Networks and Internet of ThingsIn recent years, with the advent of the Internet of Things era of "Internet of Everything", the amount of data generated by human society has exploded. Visualization provides an intuitive, vivid, and effective way to visualize complex data in a visual ...
Effective Big Data Visualization
IDEAS '17: Proceedings of the 21st International Database Engineering & Applications SymposiumIn the last several years, big data analytics has found an increasing role in our everyday lives. Data visualization has long been accepted as an integral part of data analytics. However, data visualization systems are not equipped to handle the ...






Comments