Abstract
Systems for providing mixed physical-virtual interaction on desktop surfaces have been proposed for decades, though no such systems have achieved widespread use. One major factor contributing to this lack of acceptance may be that these systems are not designed for the variety and complexity of actual work surfaces, which are often in flux and cluttered with physical objects. In this paper, we use an elicitation study and interviews to synthesize a list of ten interactive behaviors that desk-bound, digital interfaces should implement to support responsive cohabitation with physical objects. As a proof of concept, we implemented these interactive behaviors in a working augmented desk system, demonstrating their imminent feasibility.
- Anand Agarawala and Ravin Balakrishnan. 2006. Keepin' it real: pushing the desktop metaphor with physics, piles and the pen. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '06). ACM, New York, NY, USA, 1283--1292. Google Scholar
Digital Library
- Maneesh Agrawala and Chris Stolte. 2001. Rendering effective route maps: improving usability through generalization. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques (SIGGRAPH '01). ACM, New York, NY, USA, 241--249. Google Scholar
Digital Library
- Toshifumi Arai, Kimiyoshi Machii, Soshiro Kuzunuki, and Hiroshi Shojima. 1995. InteractiveDESK: a computer-augmented desk which responds to operations on real objects. In Conference Companion on Human Factors in Computing Systems (CHI '95). ACM, New York, NY, USA, 141--142. Google Scholar
Digital Library
- Hrvoje Benko, Ricardo Jota, and Andrew Wilson. 2012. MirageTable: freehand interaction on a projected augmented reality tabletop. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 199--208. Google Scholar
Digital Library
- Xiaojun Bi, Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2011. Magic desk: bringing multi-touch surfaces into desktop work. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 2511--2520. Google Scholar
Digital Library
- François Bodart, Anne-Marie Hennebert, Jean-Marie Leheureux, and Jean Vanderdonckt. 1994. Towards a dynamic strategy for computer-aided visual placement. In Proceedings of the workshop on Advanced visual interfaces (AVI '94). ACM, New York, NY, USA, 78--87. Google Scholar
Digital Library
- Olha Bondarenko and Ruud Janssen. 2005. Documents at Hand: Learning from Paper to Improve Digital Technologies. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '05). ACM, New York, NY, USA, 121--130. Google Scholar
Digital Library
- J Canny. 1986. A Computational Approach to Edge Detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 6 (June 1986), 679--698. Google Scholar
Digital Library
- Xiang Cao, Andrew D.Wilson, Ravin Balakrishnan, Ken Hinckley, and Scott E. Hudson. ShapeTouch: Leveraging Contact Shape on Interactive Surfaces. In 3rd IEEE International Workshop on Horizontal Interactive Human Computer Systems, 2008 (TABLETOP '08), 129--136.Google Scholar
- Jae Sik Chang, Eun Yi Kim, KeeChul Jung, and Hang Joon Kim. Real time hand tracking based on active contour model. In Proc. ICCSA '05, 999--1006. Google Scholar
Digital Library
- Jason Cong, Lei He, Cheng-Kok Koh, and Patrick H. Madden. 1996. Performance optimization of VLSI interconnect layout. Integr. VLSI J. 21, 1--2 (November 1996), 1--94. Google Scholar
Digital Library
- Franklin C. Crow. 1984. Summed-area tables for texture mapping. In Proceedings of the 11th annual conference on Computer graphics and interactive techniques (SIGGRAPH '84). ACM, New York, NY, USA, 207--212. Google Scholar
Digital Library
- Giuseppe Di Battista, Peter Eades, Roberto Tamassia, and Ioannis G. Tollis. 1998. Graph Drawing: Algorithms for the Visualization of Graphs (1st ed.). Prentice Hall PTR, Upper Saddle River, NJ, USA. Google Scholar
Digital Library
- Jerry Alan Fails and Dan Olsen Jr.. 2002. Light widgets: interacting in every-day spaces. In Proceedings of the 7th international conference on Intelligent user interfaces (IUI '02). ACM, New York, NY, USA. Google Scholar
Digital Library
- James Fogarty and Scott E. Hudson. 2003. GADGET: a toolkit for optimization-based approaches to interface and display generation. In Proceedings of the 16th annual ACM symposium on User interface software and technology (UIST '03). ACM, New York, NY, USA, 125--134. Google Scholar
Digital Library
- Krzysztof Z. Gajos, Daniel S. Weld, and Jacob O. Wobbrock. 2008. Decision-theoretic user interface generation. In Proceedings of the 23rd national conference on Artificial intelligence - Volume 3 (AAAI'08), Anthony Cohn (Ed.), Vol. 3. AAAI Press 1532--1536. Google Scholar
Digital Library
- Christoph Gebhardt, Roman Rädle, and Harald Reiterer. 2014. Integrative workplace: studying the effect of digital desks on users' working practices. In CHI '14 Extended Abstracts on Human Factors in Computing Systems (CHI EA '14). ACM, New York, NY, USA, 2155--2160. Google Scholar
Digital Library
- John Hardy. 2012. Experiences: a year in the life of an interactive desk. In Proceedings of the Designing Interactive Systems Conference (DIS '12). ACM, New York, NY, USA, 679--688. Google Scholar
Digital Library
- Chris Harrison, Hrvoje Benko, and Andrew D. Wilson. 2011. OmniTouch: wearable multitouch interaction everywhere. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST '11). ACM, New York, NY, USA, 441--450. Google Scholar
Digital Library
- Hiroshi Ishii. 1990. TeamWorkStation: towards a seamless shared workspace. In Proceedings of the 1990 ACM conference on Computer-supported cooperative work (CSCW '90). ACM, New York, NY, USA, 13--26. Google Scholar
Digital Library
- Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, and Andrew Fitzgibbon. 2011. KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST '11). ACM, New York, NY, USA, 559--568. Google Scholar
Digital Library
- Sasa Junuzovic, Kori Inkpen, Tom Blank, and Anoop Gupta. 2012. IllumiShare: sharing any surface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 1919--1928. Google Scholar
Digital Library
- Shaun K. Kane, Daniel Avrahami, Jacob O. Wobbrock, Beverly Harrison, Adam D. Rea, Matthai Philipose, and Anthony LaMarca. 2009. Bonfire: a nomadic system for hybrid laptop-tabletop interaction. In Proceedings of the 22nd annual ACM symposium on User interface software and technology (UIST '09). ACM, New York, NY, USA, 129--138. Google Scholar
Digital Library
- Kourosh Khoshelham and Sander Oude Elberink. Accuracy and resolution of Kinect depth data for indoor mapping applications. Sensors, 12(2), 2012, 437--454.Google Scholar
- Jeongyun Kim, Jonghoon Seo, and Tack-Don Han. 2014. AR Lamp: interactions on projection-based augmented reality for interactive learning. In Proceedings of the 19th international conference on Intelligent User Interfaces (IUI '14). ACM, New York, NY, USA, 353--358. Google Scholar
Digital Library
- K. C. Knowlton. Computer Displays Optically Superimposed on Input Devices. Bell Systems Technical Journal, 53(3), 1977, 367--383.Google Scholar
- Hideki Koike, Yoichi Sato, and Yoshinori Kobayashi. 2001. Integrating paper and digital information on EnhancedDesk: a method for realtime finger tracking on an augmented desk system. ACM Trans. Comput.-Hum. Interact. 8, 4 (December 2001), 307--322. Google Scholar
Digital Library
- Eric Larson, Gabe Cohn, Sidhant Gupta, Xiaofeng Ren, Beverly Harrison, Dieter Fox, and Shwetak Patel. 2011. HeatWave: thermal imaging for surface user interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 2565--2574. Google Scholar
Digital Library
- Joonhwan Lee, Jodi Forlizzi, and Scott E. Hudson. 2005. Studying the effectiveness of MOVE: a contextually optimized in-vehicle navigation system. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '05). ACM, New York, NY, USA, 571--580. Google Scholar
Digital Library
- Natan Linder and Pattie Maes. 2010. LuminAR: portable robotic augmented reality interface design and prototype. In Adjunct proceedings of the 23nd annual ACM symposium on User interface software and technology (UIST '10). ACM, New York, NY, USA, 395--396. Google Scholar
Digital Library
- Thomas W. Malone. 1983. How do people organize their desks?: Implications for the design of office information systems. ACM Trans. Inf. Syst. 1, 1 (January 1983), 99--112. Google Scholar
Digital Library
- Daniel C. McFarlane and Steven M. Wilder. 2009. Interactive dirt: increasing mobile work performance with a wearable projector-camera system. In Proceedings of the 11th international conference on Ubiquitous computing (UbiComp '09). ACM, New York, NY, USA, 205--214. Google Scholar
Digital Library
- Jurriaan D. Mulder, Jack Jansen, and Arjen van Rhijn. 2003. An affordable optical head tracking system for desktop VR/AR systems. In Proceedings of the workshop on Virtual environments 2003 (EGVE '03). ACM, New York, NY, USA, 215--223. Google Scholar
Digital Library
- William Newman and Pierre Wellner. 1992. A desk supporting computer-based interaction with paper documents. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '92). ACM, New York, NY, USA, 587--592. Google Scholar
Digital Library
- Michael Nielsen, Moritz Störring, Thomas B. Moeslund, and Erik Granum. 2004. A procedure for developing intuitive and ergonomic gesture interfaces for HCI. In Gesture Based Communication in Human-Computer Interaction 2915, 409--420. Springer Berlin Heidelberg.Google Scholar
- Claudio S. Pinhanez. 2001. The Everywhere Displays Projector: A Device to Create Ubiquitous Graphical Interfaces. In Proceedings of the 3rd international conference on Ubiquitous Computing (UbiComp '01). Springer-Verlag, London, UK, UK, 315--331. Google Scholar
Digital Library
- Ramesh Raskar, Jeroen van Baar, Paul Beardsley, Thomas Willwacher, Srinivas Rao, and Clifton Forlines. 2003. iLamps: geometrically aware and self-configuring projectors. In ACM SIGGRAPH 2003 Papers (SIGGRAPH '03). ACM, New York, NY, USA, 809--818. Google Scholar
Digital Library
- Ramesh Raskar, Greg Welch, Matt Cutts, Adam Lake, Lev Stesin, and Henry Fuchs. 1998. The office of the future: a unified approach to image-based modeling and spatially immersive displays. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques (SIGGRAPH '98). ACM, New York, NY, USA, 179--188. Google Scholar
Digital Library
- Jun Rekimoto. 2002. SmartSkin: an infrastructure for freehand manipulation on interactive surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '02). ACM, New York, NY, USA, 113--120. Google Scholar
Digital Library
- Charles Robertson and John Robinson. 1999. Live paper: video augmentation to simulate interactive paper. In Proceedings of the seventh ACM international conference on Multimedia (Part 2) (MULTIMEDIA '99). ACM, New York, NY, USA, 167--170. Google Scholar
Digital Library
- A. Sears. 1993. Layout Appropriateness: A Metric for Evaluating User Interface Widget Layout. IEEE Trans. Softw. Eng. 19, 7 (July 1993), 707--719. Google Scholar
Digital Library
- Abigail J. Sellen and Richard H.R. Harper. 2003. The Myth of the Paperless Office. MIT Press, Cambridge, MA, USA. Google Scholar
Digital Library
- Jürgen Steimle, Mohammadreza Khalilbeigi, Max Mühlhäuser, and James D. Hollan. 2010. Physical and digital media usage patterns on interactive tabletop surfaces. In ACM International Conference on Interactive Tabletops and Surfaces (ITS '10). ACM, New York, NY, USA, 167--176. Google Scholar
Digital Library
- Yoshiki Takeoka, Takashi Miyaki, and Jun Rekimoto. 2010. Z-touch: an infrastructure for 3d gesture interaction in the proximity of tabletop surfaces. In ACM International Conference on Interactive Tabletops and Surfaces (ITS '10). ACM, New York, NY, USA, 91--94. Google Scholar
Digital Library
- John Underkoffler and Hiroshi Ishii. 1998. Illuminating light: an optical design tool with a luminous-tangible interface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '98). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 542--549. Google Scholar
Digital Library
- John Underkoffler, Brygg Ullmer, and Hiroshi Ishii. 1999. Emancipated pixels: real-world graphics in the luminous room. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques (SIGGRAPH '99). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 385--392. Google Scholar
Digital Library
- Dhaval Vyas and Anton Nijholt. Artful surfaces: an ethnographic study exploring the use of space in design studios. Digital Creativity, 23(1), 2012, 1--20.Google Scholar
- Pierre Wellner. 1993. Interacting with paper on the DigitalDesk. Commun. ACM 36, 7 (July 1993), 87--96. Google Scholar
Digital Library
- Pierre Wellner. 1991. The DigitalDesk calculator: tangible manipulation on a desk top display. In Proceedings of the 4th annual ACM symposium on User interface software and technology (UIST '91). ACM, New York, NY, USA, 27--33. Google Scholar
Digital Library
- Andrew D. Wilson and Hrvoje Benko. 2010. Combining multiple depth cameras and projectors for interactions on, above and between surfaces. In Proceedings of the 23nd annual ACM symposium on User interface software and technology (UIST '10). ACM, New York, NY, USA, 273--282. Google Scholar
Digital Library
- Andrew D. Wilson. Depth sensing video cameras for 3D tangible tabletop interaction. In Proc. Tabletop '07, 201--204.Google Scholar
- Raphael Wimmer, Fabian Hennecke, Florian Schulz, Sebastian Boring, Andreas Butz, and Heinrich Hußmann. 2010. Curve: revisiting the digital desk. In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries (NordiCHI '10). ACM, New York, NY, USA, 561--570. Google Scholar
Digital Library
- Robert Xiao, Chris Harrison, and Scott E. Hudson. 2013. WorldKit: rapid and easy creation of ad-hoc interactive applications on everyday surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 879--888. Google Scholar
Digital Library
- Clemens Zeidler, Christof Lutteroth, Wolfgang Sturzlinger, and Gerald Weber. 2013. The auckland layout editor: an improved GUI layout specification process. In Proceedings of the 26th annual ACM symposium on User interface software and technology (UIST '13). ACM, New York, NY, USA, 343--352. Google Scholar
Digital Library
Index Terms
Supporting Responsive Cohabitation Between Virtual Interfaces and Physical Objects on Everyday Surfaces
Recommendations
Preliminary Investigation of Text Entry Method with Haptic Feedback from Real Object Surfaces Estimated Using Hand Tracking on HMD
Asian CHI Symposium 2021: Asian CHI Symposium 2021In virtual reality systems, users enter text by selecting virtual keys with the fingers. A virtual keyboard is displayed in mid-air and thus does not provide haptic feedback. To address this problem, we present a text entry method that uses the surfaces ...
A study on touch & hover based interaction for zooming
CHI EA '12: CHI '12 Extended Abstracts on Human Factors in Computing SystemsProximity is a useful medium for interaction with high interactive digital contents. It can be used in different contexts such as for navigation through depth in 3D space in zoomable interfaces. In this paper, we propose hover-based zoom interaction as ...
Bezel swipe: conflict-free scrolling and multiple selection on mobile touch screen devices
CHI '09: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsZooming user interfaces are increasingly popular on mobile devices with touch screens. Swiping and pinching finger gestures anywhere on the screen manipulate the displayed portion of a page, and taps open objects within the page. This makes navigation ...






Comments