skip to main content
research-article
Open Access
Artifacts Available
Artifacts Evaluated & Functional

Visualization by example

Published:20 December 2019Publication History
Skip Abstract Section

Abstract

While visualizations play a crucial role in gaining insights from data, generating useful visualizations from a complex dataset is far from an easy task. In particular, besides understanding the functionality provided by existing visualization libraries, generating the desired visualization also requires reshaping and aggregating the underlying data as well as composing different visual elements to achieve the intended visual narrative. This paper aims to simplify visualization tasks by automatically synthesizing the required program from simple visual sketches provided by the user. Specifically, given an input data set and a visual sketch that demonstrates how to visualize a very small subset of this data, our technique automatically generates a program that can be used to visualize the entire data set.

From a program synthesis perspective, automating visualization tasks poses several challenges that are not addressed by prior techniques. First, because many visualization tasks require data wrangling in addition to generating plots from a given table, we need to decompose the end-to-end synthesis task into two separate sub-problems. Second, because the intermediate specification that results from the decomposition is necessarily imprecise, this makes the data wrangling task particularly challenging in our context. In this paper, we address these problems by developing a new compositional visualization-by-example technique that (a) decomposes the end-to-end task into two different synthesis problems over different DSLs and (b) leverages bi-directional program analysis to deal with the complexity that arises from having an imprecise intermediate specification.

We have implemented our visualization-by-example approach in a tool called Viser and evaluate it on over 80 visualization tasks collected from on-line forums and tutorials. Viser can solve 84 of these benchmarks within a 600 second time limit, and, for those tasks that can be solved, the desired visualization is among the top-5 generated by Viser in 70% of the cases.

Skip Supplemental Material Section

Supplemental Material

a49-wang.webm

References

  1. Michael Bostock, Vadim Ogievetsky, and Jeffrey Heer. 2011. D 3 Data-Driven Documents. IEEE Trans. Vis. Comput. Graph. 17, 12 (2011), 2301–2309. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. H Carr, P Rheingans, H Schumann, Arvind Satyanarayan, and Jeffrey Heer. 2014. Lyra: An Interactive Visualization Design Environment. In Eurographics Conference on Visualization, Vol. 33. 10.Google ScholarGoogle Scholar
  3. Satish Chandra, Stephen J. Fink, and Manu Sridharan. 2009. Snugglebug: a powerful approach to weakest preconditions. In Proceedings of the 2009 ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2009, Dublin, Ireland, June 15-21, 2009. 363–374. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Dinakar Dhurjati, Manuvir Das, and Yue Yang. 2006. Path-Sensitive Dataflow Analysis with Iterative Refinement. In Static Analysis, 13th International Symposium, SAS 2006, Seoul, Korea, August 29-31, 2006, Proceedings. 425–442. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Yu Feng, Ruben Martins, Osbert Bastani, and Isil Dillig. 2018. Program synthesis using conflict-driven learning. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2018, Philadelphia, PA, USA, June 18-22, 2018. 420–435.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Yu Feng, Ruben Martins, Jacob Van Geffen, Isil Dillig, and Swarat Chaudhuri. 2017. Component-based synthesis of table consolidation and transformation tasks from examples. In Proc. Conference on Programming Language Design and Implementation. ACM, 422–436.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. John K. Feser, Swarat Chaudhuri, and Isil Dillig. 2015. Synthesizing data structure transformations from input-output examples. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation, Portland, OR, USA, June 15-17, 2015. 229–239. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Pat Hanrahan. 2006. VizQL: a language for query, analysis and visualization. In Proceedings of the ACM SIGMOD International Conference on Management of Data, Chicago, Illinois, USA, June 27-29, 2006. 721.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. William R Harris and Sumit Gulwani. 2011. Spreadsheet table transformations from examples. In ACM SIGPLAN Notices, Vol. 46. ACM, 317–328.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Jock D. Mackinlay, Pat Hanrahan, and Chris Stolte. 2007. Show Me: Automatic Presentation for Visual Analysis. IEEE Trans. Vis. Comput. Graph. 13, 6 (2007), 1137–1144. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Solomon Maina, Anders Miltner, Kathleen Fisher, Benjamin C. Pierce, David Walker, and Steve Zdancewic. 2018. Synthesizing quotient lenses. PACMPL 2, ICFP (2018), 80:1–80:29. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Ruben Martins, Jia Chen, Yanju Chen, Yu Feng, and Isil Dillig. 2019. Trinity: An Extensible Synthesis Framework for Data Science. https://github.com/fredfeng/Trinity/ .Google ScholarGoogle Scholar
  13. Anders Miltner, Kathleen Fisher, Benjamin C. Pierce, David Walker, and Steve Zdancewic. 2018. Synthesizing bijective lenses. PACMPL 2, POPL (2018), 1:1–1:30. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Dominik Moritz, Chenglong Wang, Greg L. Nelson, Halden Lin, Adam M. Smith, Bill Howe, and Jeffrey Heer. 2019. Formalizing Visualization Design Knowledge as Constraints: Actionable and Extensible Models in Draco. IEEE Trans. Vis. Comput. Graph. 25, 1 (2019), 438–448. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Phitchaya Mangpo Phothilimthana, Aditya Thakur, Rastislav Bodík, and Dinakar Dhurjati. 2016. Scaling up Superoptimization. In Proceedings of the Twenty-First International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’16, Atlanta, GA, USA, April 2-6, 2016. 297–310. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Nadia Polikarpova, Ivan Kuraj, and Armando Solar-Lezama. 2016. Program synthesis from polymorphic refinement types. Proc. Conference on Programming Language Design and Implementation (2016), 522–538.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Oleksandr Polozov and Sumit Gulwani. 2015. FlashMeta: a framework for inductive program synthesis. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2015, part of SPLASH 2015, Pittsburgh, PA, USA, October 25-30, 2015. 107–126. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Mohammad Raza, Sumit Gulwani, and Natasa Milic-Frayling. 2015. Compositional Program Synthesis from Natural Language and Examples. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015. 792–800. http://ijcai.org/Abstract/15/117Google ScholarGoogle Scholar
  19. Thomas W. Reps, Susan Horwitz, and Shmuel Sagiv. 1995. Precise Interprocedural Dataflow Analysis via Graph Reachability. In Conference Record of POPL’95: 22nd ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, San Francisco, California, USA, January 23-25, 1995. 49–61. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Bahador Saket, Hannah Kim, Eli T. Brown, and Alex Endert. 2017a. Visualization by Demonstration: An Interaction Paradigm for Visual Data Exploration. IEEE Trans. Vis. Comput. Graph. 23, 1 (2017), 331–340. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Bahador Saket, Hannah Kim, Eli T. Brown, and Alex Endert. 2017b. Visualization by Demonstration: An Interaction Paradigm for Visual Data Exploration. IEEE Trans. Vis. Comput. Graph. 23, 1 (2017), 331–340. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Arvind Satyanarayan and Jeffrey Heer. 2014. Lyra: An Interactive Visualization Design Environment. Comput. Graph. Forum 33, 3 (2014), 351–360. Google ScholarGoogle ScholarCross RefCross Ref
  23. Arvind Satyanarayan, Dominik Moritz, Kanit Wongsuphasawat, and Jeffrey Heer. 2017. Vega-Lite: A Grammar of Interactive Graphics. IEEE Trans. Vis. Comput. Graph. 23, 1 (2017), 341–350. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. David Schroeder and Daniel F. Keefe. 2016. Visualization-by-Sketching: An Artist’s Interface for Creating Multivariate Time-Varying Data Visualizations. IEEE Trans. Vis. Comput. Graph. 22, 1 (2016), 877–885. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Chris Stolte, Diane Tang, and Pat Hanrahan. 2008. Polaris: a system for query, analysis, and visualization of multidimensional databases. Commun. ACM 51, 11 (2008), 75–84. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Quoc Trung Tran, Chee-Yong Chan, and Srinivasan Parthasarathy. 2009. Query by output. In Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2009, Providence, Rhode Island, USA, June 29 - July 2, 2009. 535–548. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Chenglong Wang, Alvin Cheung, and Rastislav Bodik. 2017a. Synthesizing highly expressive SQL queries from input-output examples. In Proc. Conference on Programming Language Design and Implementation. ACM, 452–466.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Chenglong Wang, Alvin Cheung, and Rastislav Bodík. 2018a. Speeding up symbolic reasoning for relational queries. PACMPL 2, OOPSLA (2018), 157:1–157:25. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Xinyu Wang, Isil Dillig, and Rishabh Singh. 2017b. Synthesis of data completion scripts using finite tree automata. PACMPL 1, OOPSLA (2017), 62:1–62:26. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Xinyu Wang, Isil Dillig, and Rishabh Singh. 2018b. Program Synthesis using Abstraction Refinement. In Proc. Symposium on Principles of Programming Languages. ACM, 63:1–63:30.Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Kanit Wongsuphasawat, Dominik Moritz, Anushka Anand, Jock D. Mackinlay, Bill Howe, and Jeffrey Heer. 2016a. Towards a general-purpose query language for visualization recommendation. In Proceedings of the Workshop on Human-In-the-Loop Data Analytics, [email protected] 2016, San Francisco, CA, USA, June 26 - July 01, 2016. 4. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Kanit Wongsuphasawat, Dominik Moritz, Anushka Anand, Jock D. Mackinlay, Bill Howe, and Jeffrey Heer. 2016b. Voyager: Exploratory Analysis via Faceted Browsing of Visualization Recommendations. IEEE Trans. Vis. Comput. Graph. 22, 1 (2016), 649–658. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Kanit Wongsuphasawat, Zening Qu, Dominik Moritz, Riley Chang, Felix Ouk, Anushka Anand, Jock D. Mackinlay, Bill Howe, and Jeffrey Heer. 2017. Voyager 2: Augmenting Visual Analysis with Partial View Specifications. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver, CO, USA, May 06-11, 2017. 2648–2659. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Sai Zhang and Yuyin Sun. 2013. Automatically synthesizing sql queries from input-output examples. In Proc. International Conference on Automated Software Engineering. IEEE, 224–234.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Visualization by example

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!