skip to main content
research-article

Supporting Responsive Cohabitation Between Virtual Interfaces and Physical Objects on Everyday Surfaces

Published:30 June 2017Publication History
Skip Abstract Section

Abstract

Systems for providing mixed physical-virtual interaction on desktop surfaces have been proposed for decades, though no such systems have achieved widespread use. One major factor contributing to this lack of acceptance may be that these systems are not designed for the variety and complexity of actual work surfaces, which are often in flux and cluttered with physical objects. In this paper, we use an elicitation study and interviews to synthesize a list of ten interactive behaviors that desk-bound, digital interfaces should implement to support responsive cohabitation with physical objects. As a proof of concept, we implemented these interactive behaviors in a working augmented desk system, demonstrating their imminent feasibility.

References

  1. Anand Agarawala and Ravin Balakrishnan. 2006. Keepin' it real: pushing the desktop metaphor with physics, piles and the pen. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '06). ACM, New York, NY, USA, 1283--1292. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Maneesh Agrawala and Chris Stolte. 2001. Rendering effective route maps: improving usability through generalization. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques (SIGGRAPH '01). ACM, New York, NY, USA, 241--249. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Toshifumi Arai, Kimiyoshi Machii, Soshiro Kuzunuki, and Hiroshi Shojima. 1995. InteractiveDESK: a computer-augmented desk which responds to operations on real objects. In Conference Companion on Human Factors in Computing Systems (CHI '95). ACM, New York, NY, USA, 141--142. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Hrvoje Benko, Ricardo Jota, and Andrew Wilson. 2012. MirageTable: freehand interaction on a projected augmented reality tabletop. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 199--208. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Xiaojun Bi, Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2011. Magic desk: bringing multi-touch surfaces into desktop work. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 2511--2520. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. François Bodart, Anne-Marie Hennebert, Jean-Marie Leheureux, and Jean Vanderdonckt. 1994. Towards a dynamic strategy for computer-aided visual placement. In Proceedings of the workshop on Advanced visual interfaces (AVI '94). ACM, New York, NY, USA, 78--87. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Olha Bondarenko and Ruud Janssen. 2005. Documents at Hand: Learning from Paper to Improve Digital Technologies. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '05). ACM, New York, NY, USA, 121--130. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. J Canny. 1986. A Computational Approach to Edge Detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 6 (June 1986), 679--698. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Xiang Cao, Andrew D.Wilson, Ravin Balakrishnan, Ken Hinckley, and Scott E. Hudson. ShapeTouch: Leveraging Contact Shape on Interactive Surfaces. In 3rd IEEE International Workshop on Horizontal Interactive Human Computer Systems, 2008 (TABLETOP '08), 129--136.Google ScholarGoogle Scholar
  10. Jae Sik Chang, Eun Yi Kim, KeeChul Jung, and Hang Joon Kim. Real time hand tracking based on active contour model. In Proc. ICCSA '05, 999--1006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Jason Cong, Lei He, Cheng-Kok Koh, and Patrick H. Madden. 1996. Performance optimization of VLSI interconnect layout. Integr. VLSI J. 21, 1--2 (November 1996), 1--94. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Franklin C. Crow. 1984. Summed-area tables for texture mapping. In Proceedings of the 11th annual conference on Computer graphics and interactive techniques (SIGGRAPH '84). ACM, New York, NY, USA, 207--212. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Giuseppe Di Battista, Peter Eades, Roberto Tamassia, and Ioannis G. Tollis. 1998. Graph Drawing: Algorithms for the Visualization of Graphs (1st ed.). Prentice Hall PTR, Upper Saddle River, NJ, USA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Jerry Alan Fails and Dan Olsen Jr.. 2002. Light widgets: interacting in every-day spaces. In Proceedings of the 7th international conference on Intelligent user interfaces (IUI '02). ACM, New York, NY, USA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. James Fogarty and Scott E. Hudson. 2003. GADGET: a toolkit for optimization-based approaches to interface and display generation. In Proceedings of the 16th annual ACM symposium on User interface software and technology (UIST '03). ACM, New York, NY, USA, 125--134. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Krzysztof Z. Gajos, Daniel S. Weld, and Jacob O. Wobbrock. 2008. Decision-theoretic user interface generation. In Proceedings of the 23rd national conference on Artificial intelligence - Volume 3 (AAAI'08), Anthony Cohn (Ed.), Vol. 3. AAAI Press 1532--1536. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Christoph Gebhardt, Roman Rädle, and Harald Reiterer. 2014. Integrative workplace: studying the effect of digital desks on users' working practices. In CHI '14 Extended Abstracts on Human Factors in Computing Systems (CHI EA '14). ACM, New York, NY, USA, 2155--2160. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. John Hardy. 2012. Experiences: a year in the life of an interactive desk. In Proceedings of the Designing Interactive Systems Conference (DIS '12). ACM, New York, NY, USA, 679--688. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Chris Harrison, Hrvoje Benko, and Andrew D. Wilson. 2011. OmniTouch: wearable multitouch interaction everywhere. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST '11). ACM, New York, NY, USA, 441--450. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Hiroshi Ishii. 1990. TeamWorkStation: towards a seamless shared workspace. In Proceedings of the 1990 ACM conference on Computer-supported cooperative work (CSCW '90). ACM, New York, NY, USA, 13--26. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, and Andrew Fitzgibbon. 2011. KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST '11). ACM, New York, NY, USA, 559--568. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Sasa Junuzovic, Kori Inkpen, Tom Blank, and Anoop Gupta. 2012. IllumiShare: sharing any surface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 1919--1928. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Shaun K. Kane, Daniel Avrahami, Jacob O. Wobbrock, Beverly Harrison, Adam D. Rea, Matthai Philipose, and Anthony LaMarca. 2009. Bonfire: a nomadic system for hybrid laptop-tabletop interaction. In Proceedings of the 22nd annual ACM symposium on User interface software and technology (UIST '09). ACM, New York, NY, USA, 129--138. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Kourosh Khoshelham and Sander Oude Elberink. Accuracy and resolution of Kinect depth data for indoor mapping applications. Sensors, 12(2), 2012, 437--454.Google ScholarGoogle Scholar
  25. Jeongyun Kim, Jonghoon Seo, and Tack-Don Han. 2014. AR Lamp: interactions on projection-based augmented reality for interactive learning. In Proceedings of the 19th international conference on Intelligent User Interfaces (IUI '14). ACM, New York, NY, USA, 353--358. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. K. C. Knowlton. Computer Displays Optically Superimposed on Input Devices. Bell Systems Technical Journal, 53(3), 1977, 367--383.Google ScholarGoogle Scholar
  27. Hideki Koike, Yoichi Sato, and Yoshinori Kobayashi. 2001. Integrating paper and digital information on EnhancedDesk: a method for realtime finger tracking on an augmented desk system. ACM Trans. Comput.-Hum. Interact. 8, 4 (December 2001), 307--322. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Eric Larson, Gabe Cohn, Sidhant Gupta, Xiaofeng Ren, Beverly Harrison, Dieter Fox, and Shwetak Patel. 2011. HeatWave: thermal imaging for surface user interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 2565--2574. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Joonhwan Lee, Jodi Forlizzi, and Scott E. Hudson. 2005. Studying the effectiveness of MOVE: a contextually optimized in-vehicle navigation system. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '05). ACM, New York, NY, USA, 571--580. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Natan Linder and Pattie Maes. 2010. LuminAR: portable robotic augmented reality interface design and prototype. In Adjunct proceedings of the 23nd annual ACM symposium on User interface software and technology (UIST '10). ACM, New York, NY, USA, 395--396. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Thomas W. Malone. 1983. How do people organize their desks?: Implications for the design of office information systems. ACM Trans. Inf. Syst. 1, 1 (January 1983), 99--112. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Daniel C. McFarlane and Steven M. Wilder. 2009. Interactive dirt: increasing mobile work performance with a wearable projector-camera system. In Proceedings of the 11th international conference on Ubiquitous computing (UbiComp '09). ACM, New York, NY, USA, 205--214. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Jurriaan D. Mulder, Jack Jansen, and Arjen van Rhijn. 2003. An affordable optical head tracking system for desktop VR/AR systems. In Proceedings of the workshop on Virtual environments 2003 (EGVE '03). ACM, New York, NY, USA, 215--223. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. William Newman and Pierre Wellner. 1992. A desk supporting computer-based interaction with paper documents. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '92). ACM, New York, NY, USA, 587--592. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Michael Nielsen, Moritz Störring, Thomas B. Moeslund, and Erik Granum. 2004. A procedure for developing intuitive and ergonomic gesture interfaces for HCI. In Gesture Based Communication in Human-Computer Interaction 2915, 409--420. Springer Berlin Heidelberg.Google ScholarGoogle Scholar
  36. Claudio S. Pinhanez. 2001. The Everywhere Displays Projector: A Device to Create Ubiquitous Graphical Interfaces. In Proceedings of the 3rd international conference on Ubiquitous Computing (UbiComp '01). Springer-Verlag, London, UK, UK, 315--331. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Ramesh Raskar, Jeroen van Baar, Paul Beardsley, Thomas Willwacher, Srinivas Rao, and Clifton Forlines. 2003. iLamps: geometrically aware and self-configuring projectors. In ACM SIGGRAPH 2003 Papers (SIGGRAPH '03). ACM, New York, NY, USA, 809--818. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Ramesh Raskar, Greg Welch, Matt Cutts, Adam Lake, Lev Stesin, and Henry Fuchs. 1998. The office of the future: a unified approach to image-based modeling and spatially immersive displays. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques (SIGGRAPH '98). ACM, New York, NY, USA, 179--188. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Jun Rekimoto. 2002. SmartSkin: an infrastructure for freehand manipulation on interactive surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '02). ACM, New York, NY, USA, 113--120. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Charles Robertson and John Robinson. 1999. Live paper: video augmentation to simulate interactive paper. In Proceedings of the seventh ACM international conference on Multimedia (Part 2) (MULTIMEDIA '99). ACM, New York, NY, USA, 167--170. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. A. Sears. 1993. Layout Appropriateness: A Metric for Evaluating User Interface Widget Layout. IEEE Trans. Softw. Eng. 19, 7 (July 1993), 707--719. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Abigail J. Sellen and Richard H.R. Harper. 2003. The Myth of the Paperless Office. MIT Press, Cambridge, MA, USA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Jürgen Steimle, Mohammadreza Khalilbeigi, Max Mühlhäuser, and James D. Hollan. 2010. Physical and digital media usage patterns on interactive tabletop surfaces. In ACM International Conference on Interactive Tabletops and Surfaces (ITS '10). ACM, New York, NY, USA, 167--176. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Yoshiki Takeoka, Takashi Miyaki, and Jun Rekimoto. 2010. Z-touch: an infrastructure for 3d gesture interaction in the proximity of tabletop surfaces. In ACM International Conference on Interactive Tabletops and Surfaces (ITS '10). ACM, New York, NY, USA, 91--94. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. John Underkoffler and Hiroshi Ishii. 1998. Illuminating light: an optical design tool with a luminous-tangible interface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '98). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 542--549. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. John Underkoffler, Brygg Ullmer, and Hiroshi Ishii. 1999. Emancipated pixels: real-world graphics in the luminous room. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques (SIGGRAPH '99). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 385--392. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Dhaval Vyas and Anton Nijholt. Artful surfaces: an ethnographic study exploring the use of space in design studios. Digital Creativity, 23(1), 2012, 1--20.Google ScholarGoogle Scholar
  48. Pierre Wellner. 1993. Interacting with paper on the DigitalDesk. Commun. ACM 36, 7 (July 1993), 87--96. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Pierre Wellner. 1991. The DigitalDesk calculator: tangible manipulation on a desk top display. In Proceedings of the 4th annual ACM symposium on User interface software and technology (UIST '91). ACM, New York, NY, USA, 27--33. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Andrew D. Wilson and Hrvoje Benko. 2010. Combining multiple depth cameras and projectors for interactions on, above and between surfaces. In Proceedings of the 23nd annual ACM symposium on User interface software and technology (UIST '10). ACM, New York, NY, USA, 273--282. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Andrew D. Wilson. Depth sensing video cameras for 3D tangible tabletop interaction. In Proc. Tabletop '07, 201--204.Google ScholarGoogle Scholar
  52. Raphael Wimmer, Fabian Hennecke, Florian Schulz, Sebastian Boring, Andreas Butz, and Heinrich Hußmann. 2010. Curve: revisiting the digital desk. In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries (NordiCHI '10). ACM, New York, NY, USA, 561--570. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Robert Xiao, Chris Harrison, and Scott E. Hudson. 2013. WorldKit: rapid and easy creation of ad-hoc interactive applications on everyday surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 879--888. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Clemens Zeidler, Christof Lutteroth, Wolfgang Sturzlinger, and Gerald Weber. 2013. The auckland layout editor: an improved GUI layout specification process. In Proceedings of the 26th annual ACM symposium on User interface software and technology (UIST '13). ACM, New York, NY, USA, 343--352. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Supporting Responsive Cohabitation Between Virtual Interfaces and Physical Objects on Everyday Surfaces

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!