Abstract

Data physicalizations (3D printed terrain models, anatomical scans, or even abstract data) can naturally engage both the visual and haptic senses in ways that are difficult or impossible to do with traditional planar touch screens and even immersive digital displays. Yet, the rigid 3D physicalizations produced with today's most common 3D printers are fundamentally limited for data exploration and querying tasks that require dynamic input (e.g., touch sensing) and output (e.g., animation), functions that are easily handled with digital displays. We introduce a novel style of hybrid virtual + physical visualization designed specifically to support interactive data exploration tasks. Working toward a "best of both worlds" solution, our approach fuses immersive AR, physical 3D data printouts, and touch sensing through the physicalization. We demonstrate that this solution can support three of the most common spatial data querying interactions used in scientific visualization (streamline seeding, dynamic cutting places, and world-in-miniature visualization). Finally, we present quantitative performance data and describe a first application to exploratory visualization of an actively studied supercomputer climate simulation data with feedback from domain scientists.
Supplemental Material
- Daniel F Abawi, Joachim Bienwald, and Ralf Dorner. 2004. Accuracy in optical tracking with fiducial markers: an accuracy function for ARToolKit. In Third IEEE and ACM International symposium on mixed and augmented reality. IEEE, 260--261.Google Scholar
Digital Library
- Greg Abram, Francesca Samsel, Mark R. Petersen, Xylar Asay-Davis, Darin Comeau, Stephen F. Price, and Mike Potel. 2021. Antarctic Water Masses and Ice Shelves: Visualizing the Physics. IEEE Computer Graphics and Applications 41, 1 (2021), 35--41. https://doi.org/10.1109/MCG.2020.3044228Google Scholar
Cross Ref
- Daniel Acevedo, Eileen Vote, David H Laidlaw, and Martha S Joukowsky. 2001. Archaeological data visualization in VR: Analysis of lamp finds at the Great Temple of Petra, a case study. In Proceedings Visualization, 2001. VIS'01. IEEE, 493--597.Google Scholar
Cross Ref
- James Ahrens, Berk Geveci, and Charles Law. 2005. Paraview: An end-user tool for large data visualization. The visualization handbook 717, 8 (2005).Google Scholar
- Luis Alonso, Yan Ryan Zhang, Arnaud Grignard, Ariel Noyman, Yasushi Sakai, Markus ElKatsha, Ronan Doorley, and Kent Larson. 2018. Cityscope: a data-driven interactive simulation tool for urban design. Use case volpe. In International conference on complex systems. Springer, 253--261.Google Scholar
Cross Ref
- Benjamin Bach, Ronell Sicat, Johanna Beyer, Maxime Cordeil, and Hanspeter Pfister. 2017. The hologram in my hand: How effective is interactive exploration of 3d visualizations in immersive tangible augmented reality? IEEE transactions on visualization and computer graphics 24, 1 (2017), 457--467.Google Scholar
- Christoph Bader, Dominik Kolb, James C Weaver, Sunanda Sharma, Ahmed Hosny, João Costa, and Neri Oxman. 2018. Making data matter: Voxel printing for the digital fabrication of data across scales and domains. Science advances 4, 5 (2018), eaas8652. Proc. ACM Hum.-Comput. Interact., Vol. 5, No. ISS, Article 497. Publication date: November 2021. Multi-Touch Querying on Data Physicalizations in Immersive AR 497:17Google Scholar
- Francois Berard and Thibault Louis. 2017. The object inside: Assessing 3d examination with a spherical handheld perspective-corrected display. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 4396--4404.Google Scholar
Digital Library
- Mark Billinghurst, Hirokazu Kato, Ivan Poupyrev, et al. 2008. Tangible augmented reality. Acm siggraph asia 7, 2 (2008), 1--10.Google Scholar
- Joachim Bobrich and Steffen Otto. 2002. Augmented maps. International Archives of Photogrammetry Remote Sensing and Spatial Information Sciences 34, 4 (2002), 502--505.Google Scholar
- Gerd Bruder, Frank Steinicke, Phil Wieland, and Markus Lappe. 2011. Tuning self-motion perception in virtual reality with visual illusions. IEEE Transactions on Visualization and Computer Graphics 18, 7 (2011), 1068--1078.Google Scholar
Digital Library
- Steve Bryson. 1996. Virtual reality in scientific visualization. Commun. ACM 39, 5 (1996), 62--71.Google Scholar
Digital Library
- William Buxton, Ralph Hill, and Peter Rowley. 1985. Issues and techniques in touch-sensitive tablet input. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques. 215--224.Google Scholar
Digital Library
- Felix A Carroll and David N Blauch. 2017. 3D printing of molecular models with calculated geometries and p orbital isosurfaces. Journal of Chemical Education 94, 7 (2017), 886--891.Google Scholar
Cross Ref
- Hank Childs. 2012. VisIt: An end-user tool for visualizing and analyzing very large data. (2012).Google Scholar
- Dane Coffey, Nicholas Malbraaten, Trung Le, Iman Borazjani, Fotis Sotiropoulos, and Daniel F Keefe. 2011. Slice WIM: a multi-surface, multi-touch interface for overview+ detail exploration of volume datasets in virtual reality. In Symposium on Interactive 3D Graphics and Games. 191--198.Google Scholar
Digital Library
- Dane Coffey, Nicholas Malbraaten, Trung Bao Le, Iman Borazjani, Fotis Sotiropoulos, Arthur G Erdman, and Daniel F Keefe. 2011. Interactive slice WIM: Navigating and interrogating volume data sets using a multisurface, multitouch VR interface. IEEE Transactions on Visualization and Computer Graphics 18, 10 (2011), 1614--1626.Google Scholar
Digital Library
- Shima Dadkhahfard, Katayoon Etemad, John Brosz, and Faramarz Samavati. 2018. Area Preserving Dynamic Geospatial Visualization on Physical Globe.. In VISIGRAPP (3: IVAPP). 309--318.Google Scholar
- Kurtis Danyluk, Teoman Tomo Ulusoy, Wei Wei, and Wesley Willett. 2020. Touch and Beyond: Comparing Physical and Virtual Reality Visualizations. IEEE Transactions on Visualization and Computer Graphics (2020).Google Scholar
- Hessam Djavaherpour, Ali Mahdavi-Amiri, and Faramarz F Samavati. 2017. Physical visualization of geospatial datasets. IEEE computer graphics and applications 37, 3 (2017), 61--69.Google Scholar
Digital Library
- Hessam Djavaherpour, Faramarz Samavati, Ali Mahdavi-Amiri, Fatemeh Yazdanbakhsh, Samuel Huron, Richard Levy, Yvonne Jansen, and Lora Oehlberg. 2021. Data to Physicalization: A Survey of the Physical Rendering Process. arXiv preprint arXiv:2102.11175 (2021).Google Scholar
- Bin Fang, Di Guo, Fuchun Sun, Huaping Liu, and Yupei Wu. 2015. A robotic hand-arm teleoperation system using human arm/hand with a novel data glove. In 2015 IEEE International Conference on Robotics and Biomimetics (ROBIO). IEEE, 2483--2488.Google Scholar
Cross Ref
- Sean Follmer, Daniel Leithinger, Alex Olwal, Akimitsu Hogge, and Hiroshi Ishii. 2013. inFORM: dynamic physical affordances and constraints through shape and object actuation.. In Uist, Vol. 13.Google Scholar
Digital Library
- Alexandre Gillet, Michel Sanner, Daniel Stoffler, David Goodsell, and Arthur Olson. 2004. Augmented reality with tangible auto-fabricated models for molecular biology applications. In IEEE Visualization 2004. IEEE, 235--241.Google Scholar
Digital Library
- Alexandre Gillet, Michel Sanner, Daniel Stoffler, and Arthur Olson. 2005. Tangible interfaces for structural molecular biology. Structure 13, 3 (2005), 483--491.Google Scholar
- Leah A Groves, Patrick Carnahan, Daniel R Allen, Rankin Adam, Terry M Peters, and Elvis CS Chen. 2019. Accuracy assessment for the co-registration between optical and VIVE head-mounted display tracking. International journal of computer assisted radiology and surgery 14, 7 (2019), 1207--1215.Google Scholar
Cross Ref
- Nicholas R Hedley, Mark Billinghurst, Lori Postner, Richard May, and Hirokazu Kato. 2002. Explorations in the use of augmented reality for geographic visualization. Presence: Teleoperators & Virtual Environments 11, 2 (2002), 119--133.Google Scholar
Digital Library
- Bridger Herman, Francesca Samsel, Annie Bares, Seth Johnson, Greg Abram, and Daniel F Keefe. 2020. Printmaking, Puzzles, and Studio Closets: Using Artistic Metaphors to Reimagine the User Interface for Designing Immersive Visualizations. In 2020 IEEE VIS Arts Program (VISAP). IEEE, 19--28.Google Scholar
- Trevor Hogan and Eva Hornecker. 2016. Towards a design space for multisensory data representation. Interacting with Computers 29, 2 (2016), 147--167.Google Scholar
- Hiroshi Ishii and Brygg Ullmer. 1997. Tangible bits: towards seamless interfaces between people, bits and atoms. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems. 234--241.Google Scholar
Digital Library
- Bret Jackson, Dane Coffey, and Daniel F Keefe. 2012. Force Brushes: Progressive data-driven haptic selection and filtering for multi-variate flow visualizations. Proceedings of EuroVis 2012 (2012), 7--11.Google Scholar
- Bret Jackson and Daniel F Keefe. 2019. From Painting to Widgets, 6-DOF and Bimanual Input Beyond Pointing.Google Scholar
- Bret Jackson, Tung Yuen Lau, David Schroeder, Kimani C Toussaint, and Daniel F Keefe. 2013. A lightweight tangible 3D interface for interactive visualization of thin fiber structures. IEEE transactions on visualization and computer graphics 19, 12 (2013), 2802--2809. Proc. ACM Hum.-Comput. Interact., Vol. 5, No. ISS, Article 497. Publication date: November 2021. 497:18 Bridger Herman et al.Google Scholar
- Bret Jackson, David Schroeder, and Daniel F Keefe. 2012. Nailing down multi-touch: anchored above the surface interaction for 3D modeling and navigation. In Proceedings of Graphics Interface 2012. Citeseer, 181--184.Google Scholar
Digital Library
- Yvonne Jansen, Pierre Dragicevic, and Jean-Daniel Fekete. 2013. Evaluating the efficiency of physical visualizations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2593--2602.Google Scholar
Digital Library
- Yvonne Jansen, Pierre Dragicevic, Petra Isenberg, Jason Alexander, Abhijit Karnik, Johan Kildal, Sriram Subramanian, and Kasper Hornbæk. 2015. Opportunities and challenges for data physicalization. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 3227--3236.Google Scholar
Digital Library
- Hyein Jeong, Xylar S. Asay-Davis, Adrian K. Turner, Darin S. Comeau, et al. 2020. Impacts of Ice-Shelf Melting on Water-Mass Transformation in the Southern Ocean from E3SM Simulations. Journal of Climate 33, 13 (06 2020), 5787--5807. https://doi.org/10.1175/JCLI-D-19-0683.1 arXiv:https://journals.ametsoc.org/jcli/articlepdf/ 33/13/5787/4955899/jclid190683.pdfGoogle Scholar
Cross Ref
- Seth Johnson, Daniel Orban, Hakizumwami Birali Runesha, Lingyu Meng, Bethany Juhnke, Arthur Erdman, Francesca Samsel, and Daniel F Keefe. 2019. Bento box: An interactive and zoomable small multiples technique for visualizing 4d simulation ensembles in virtual reality. Frontiers in Robotics and AI 6 (2019), 61.Google Scholar
Cross Ref
- Seth Johnson, Francesca Samsel, Gregory Abram, Daniel Olson, Andrew J Solis, Bridger Herman, Phillip J Wolfram, Christophe Lenglet, and Daniel F Keefe. 2019. Artifact-based rendering: Harnessing natural and traditional visual media for more expressive and engaging 3D visualizations. IEEE transactions on visualization and computer graphics 26, 1 (2019), 492--502.Google Scholar
- Daniel F Keefe, Robert C Zeleznik, and David H Laidlaw. 2008. Tech-note: Dynamic dragging for input of 3D trajectories. In 2008 IEEE Symposium on 3D User Interfaces. IEEE, 51--54.Google Scholar
Digital Library
- Nurit Kirshenbaum, James Hutchison, Ryan Theriot, Dylan Kobayashi, and Jason Leigh. 2020. Data in Context: Engaging Audiences with 3D Physical Geo-Visualization. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1--9.Google Scholar
Digital Library
- Tijmen Klein, Florimond Guéniat, Luc Pastur, Frédéric Vernier, and Tobias Isenberg. 2012. A design study of direct-touch interaction for exploratory 3D scientific visualization. In Computer Graphics Forum, Vol. 31. Wiley Online Library, 1225--1234.Google Scholar
- Luv Kohli. 2010. Redirected touching: Warping space to remap passive haptics. In 2010 IEEE Symposium on 3D User Interfaces (3DUI). IEEE, 129--130.Google Scholar
Cross Ref
- Joseph LaViola and Robert Zeleznik. 1999. Flex and pinch: A case study of whole hand input design for virtual environment interaction. In Proceedings of the second IASTED international conference on computer graphics and imaging. 221--225.Google Scholar
- Daniel Leithinger, David Lakatos, Anthony DeVincenzi, Matthew Blackshaw, and Hiroshi Ishii. 2011. Direct and gestural interaction with relief: a 2.5 D shape display. In Proceedings of the 24th annual ACM symposium on User interface software and technology. 541--548.Google Scholar
Digital Library
- Loren Madsen. 2018. https://www.facebook.com/loren.madsen.9/posts/10156981297933841. Accessed July 2020.Google Scholar
- Nathalie Miebach. [n.d.]. Changing Waters. http://nathaliemiebach.com/waters.html. Accessed March 2021.Google Scholar
- Jung Who Nam, Krista McCullough, Joshua Tveite, Maria Molina Espinosa, Charles H Perry, Barry T Wilson, and Daniel F Keefe. 2019. Worlds-in-Wedges: Combining Worlds-in-Miniature and Portals to Support Comparative Immersive Visualization of Forestry Data. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 747--755.Google Scholar
Cross Ref
- Gregory Nielson, Hans Hagen, and Heinrich Muller. 1997. Scientific visualization. Institute of Electrical & Electronics Engineers.Google Scholar
- Niels Christian Nilsson, Andre Zenner, and Adalberto L Simeone. 2020. Haptic Proxies for Virtual Reality: Success Criteria and Taxonomy. In Workshop on Everyday Proxy Objects for Virtual Reality at CHI'20.Google Scholar
- Aditya Shekhar Nittala, Nico Li, Stephen Cartwright, Kazuki Takashima, Ehud Sharlin, and Mario Costa Sousa. 2015. PLANWELL: Spatial User Interface for Collaborative PetroleumWell-Planning. In SIGGRAPH Asia 2015 Mobile Graphics and Interactive Applications (Kobe, Japan) (SA '15). Association for Computing Machinery, New York, NY, USA, Article 19, 8 pages. https://doi.org/10.1145/2818427.2818443Google Scholar
Digital Library
- Mark R. Petersen, Xylar S. Asay-Davis, et al. 2019. An Evaluation of the Ocean and Sea Ice Climate of E3SM Using MPAS and Interannual CORE-II Forcing. Journal of Advances in Modeling Earth Systems 11, 5 (2019), 1438--1458. https://doi.org/10.1029/2018MS001373Google Scholar
Cross Ref
- Ben Piper, Carlo Ratti, and Hiroshi Ishii. 2002. Illuminating clay: a 3-D tangible interface for landscape analysis. In Proceedings of the SIGCHI conference on Human factors in computing systems. 355--362.Google Scholar
Digital Library
- PTC. [n.d.]. Vuforia. https://www.ptc.com/en/products/vuforia/vuforia-engineGoogle Scholar
- Daniel Pustka, Jan-Patrick Hülß, Jochen Willneff, Frieder Pankratz, Manuel Huber, and Gudrun Klinker. 2012. Optical outside-in tracking using unmodified mobile phones. In 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 81--89. Proc. ACM Hum.-Comput. Interact., Vol. 5, No. ISS, Article 497. Publication date: November 2021. Multi-Touch Querying on Data Physicalizations in Immersive AR 497:19Google Scholar
Digital Library
- Wolf-Dieter Rase. 2011. Creating physical 3D maps using rapid prototyping techniques. In True-3D in Cartography. Springer, 119--134.Google Scholar
- Kim Sauvé, Dominic Potts, Jason Alexander, and Steven Houben. 2020. A Change of Perspective: How User Orientation Influences the Perception of Physicalizations. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1--12.Google Scholar
Digital Library
- Valkyrie Savage, Xiaohan Zhang, and Björn Hartmann. 2012. Midas: fabricating custom capacitive touch sensors to prototype interactive objects. In Proceedings of the 25th annual ACM symposium on User interface software and technology. 579--588.Google Scholar
Digital Library
- Will J Schroeder, Bill Lorensen, and Ken Martin. 2004. The visualization toolkit: an object-oriented approach to 3D graphics. Kitware.Google Scholar
- Jürgen Schulze-Döbold, Uwe Wössner, Steffen PWalz, and Ulrich Lang. 2001. Volume rendering in a virtual environment. In Immersive Projection Technology and Virtual Environments 2001. Springer, 187--198.Google Scholar
- Adrien Segal. 2015. Grewingk Glacier. https://www.adriensegal.com/grewingk-glacier. Accessed July 2020.Google Scholar
- Sensel, Inc. [n.d.]. The Sensel Morph. https://morph.sensel.com/. Accessed March 2021.Google Scholar
- Orit Shaer and Eva Hornecker. 2010. Tangible user interfaces: past, present, and future directions. Now Publishers Inc.Google Scholar
- Gary Singh. 2018. Wearing Multiple Hats. IEEE Computer Graphics and Applications 38, 4 (2018), 6--8.Google Scholar
Cross Ref
- Jason S Sobel, Andrew S Forsberg, David H Laidlaw, Robert C Zeleznik, Daniel F Keefe, Igor Pivkin, George E Karniadakis, Peter Richardson, and Sharon Swartz. 2004. Particle flurries. IEEE Computer Graphics and Applications 24, 2 (2004), 76--85.Google Scholar
Digital Library
- Hyunyoung Song, Hrvoje Benko, Francois Guimbretiere, Shahram Izadi, Xiang Cao, and Ken Hinckley. 2011. Grips and gestures on a multi-touch pen. In Proceedings of the SIGCHI conference on Human Factors in computing systems. 1323--1332.Google Scholar
Digital Library
- Anthony Steed. 2008. A simple method for estimating the latency of interactive, real-time graphics simulations. In Proceedings of the 2008 ACM symposium on Virtual reality software and technology. 123--129.Google Scholar
Digital Library
- Frank Steinicke, Hrvoje Benko, Antonio Krüger, Daniel Keefe, Jean-Baptiste de la Riviére, Ken Anderson, Jonna Häkkilä, Leena Arhippainen, and Minna Pakanen. 2012. The 3rd dimension of CHI (3DCHI) touching and designing 3D user interfaces. In CHI'12 Extended Abstracts on Human Factors in Computing Systems. 2695--2698.Google Scholar
- Andrew Stevenson, Christopher Perez, and Roel Vertegaal. 2010. An inflatable hemispherical multi-touch display. In Proceedings of the fifth international conference on Tangible, embedded, and embodied interaction. 289--292.Google Scholar
Digital Library
- Richard Stoakley, Matthew J Conway, and Randy Pausch. 1995. Virtual reality on a WIM: interactive worlds in miniature. In Proceedings of the SIGCHI conference on Human factors in computing systems. 265--272.Google Scholar
Digital Library
- Faisal Taher, John Hardy, Abhijit Karnik, Christian Weichel, Yvonne Jansen, Kasper Hornbæk, and Jason Alexander. 2015. Exploring interactions with physically dynamic bar charts. In Proceedings of the 33rd annual acm conference on human factors in computing systems. 3237--3246.Google Scholar
Digital Library
- Sheng Kai Tang, Yusuke Sekikawa, Daniel Leithinger, Sean Follmer, and Hirishi Ishii. [n.d.]. Tangible CityScape. http://tangible.media.mit.edu/project/tangible-cityscape. Accessed March 2021.Google Scholar
- Laura Tateosian, Helena Mitasova, Brendan Harmon, Brent Fogleman, Katherine Weaver, and Russel Harmon. 2010. TanGeoMS: Tangible geospatial modeling system. IEEE transactions on visualization and computer graphics 16, 6 (2010), 1605--1612.Google Scholar
- Russell M Taylor, Thomas C Hudson, Adam Seeger, Hans Weber, Jeffrey Juliano, and Aron T Helser. 2001. VRPN: a device-independent, network-transparent VR peripheral system. In Proceedings of the ACM symposium on Virtual reality software and technology. 55--61.Google Scholar
Digital Library
- Carlos E Tejada, Raf Ramakers, Sebastian Boring, and Daniel Ashbrook. 2020. AirTouch: 3D-printed Touch-Sensitive Objects Using Pneumatic Sensing. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1--10.Google Scholar
Digital Library
- Michael C Thrun and Florian Lerch. 2016. Visualization and 3D printing of multivariate data of biomarkers. WSCG (2016).Google Scholar
- Yun Wang, Adrien Segal, Roberta Klatzky, Daniel F Keefe, Petra Isenberg, Jörn Hurtienne, Eva Hornecker, Tim Dwyer, and Stephen Barrass. 2019. An emotional response to the value of visualization. IEEE computer graphics and applications 39, 5 (2019), 8--17.Google Scholar
Digital Library
- Maximilian Weiß, Katrin Angerbauer, Alexandra Voit, Magdalena Schwarzl, Michael Sedlmair, and Sven Mayer. 2020. Revisited: Comparison of empirical methods to evaluate visualizations supporting crafting and assembly purposes. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2020), 1204--1213.Google Scholar
- Andrew D Wilson. 2010. Using a depth camera as a touch sensor. In ACM international conference on interactive tabletops and surfaces. 69--72.Google Scholar
Digital Library
- Terri L Woods, Sarah Reed, Sherry Hsi, John A Woods, and Michael R Woods. 2016. Pilot study using the augmented reality sandbox to teach topographic maps and surficial processes in introductory geology labs. Journal of Geoscience Proc. ACM Hum.-Comput. Interact., Vol. 5, No. ISS, Article 497. Publication date: November 2021. 497:20 Bridger Herman et al. Education 64, 3 (2016), 199--214.Google Scholar
- Guoyong Zhang, Jianhua Gong, Yi Li, Jun Sun, Bingli Xu, Dong Zhang, Jieping Zhou, Ling Guo, Shen Shen, and Bingxiao Yin. 2020. An efficient flood dynamic visualization approach based on 3D printing and augmented reality. International Journal of Digital Earth 13, 11 (2020), 1302--1320.Google Scholar
Cross Ref
Index Terms
Multi-Touch Querying on Data Physicalizations in Immersive AR
Recommendations
Opportunities and Challenges for Data Physicalization
CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing SystemsPhysical representations of data have existed for thousands of years. Yet it is now that advances in digital fabrication, actuated tangible interfaces, and shape-changing displays are spurring an emerging area of research that we call Data ...
Immersive tangible geospatial modeling
SIGSPACIAL '16: Proceedings of the 24th ACM SIGSPATIAL International Conference on Advances in Geographic Information SystemsTangible Landscape is a tangible interface for geographic information systems (GIS). It interactively couples physical and digital models of a landscape so that users can intuitively explore, model, and analyze geospatial data in a collaborative ...
Visualizing the keyboard in virtual reality for enhancing immersive experience
SIGGRAPH '17: ACM SIGGRAPH 2017 PostersRecently, virtual reality (VR) becomes more and more popular and provides users an immersive experience with a head-mounted display (HMD). However, in some applications, users have to interact with physical objects while immersed in VR. With a non-see-...






Comments