skip to main content
research-article

"I simply watched where she was looking at": Coordination in Short-term Synchronous Cooperative Mixed Reality

Published:05 December 2019Publication History
Skip Abstract Section

Abstract

Mixed reality (MR) cooperation scenarios are more and more interesting for business and research as powerful wearable devices like head mounted displays (HMD) become commercially available. A lot of work focuses on remote MR cooperation settings like remote maintenance support and co-located scenarios in which participants cooperate over a longer period time. Despite this, MR also has great potential for real-time co-located cooperation support with the need of short-term decisions and interactions. However, little is known on how this support can be provided. To bridge this gap, we conducted an experiment using a MR visual search task performed by dyads. Based on related work, visual search was chosen to represent typical challenges of short-term cooperative MR tasks. The aim of the experiment was to explore how the participants coordinate their searches and how this influences their performance in a task. We found that participants mainly used embodied and verbal cues to coordinate their searches (rather than virtual cues provided by the HMD) and that less communication worked significantly better, which is in (partial) contrast to existing findings. We discuss potential reasons for and impacts of these findings.

References

  1. Ahish Agrawal, Gourav Acharya, Krishna Balasubramanian, Nehal Agrawal, and Ratnesh Chaturvedi. 2016. A Review on the use of Augmented Reality to Generate Safety Awareness and Enhance Emergency Response. International Journal of Current Engineering and Technology 6, 3 (June 2016), 813--820.Google ScholarGoogle Scholar
  2. David G. Armstrong, Timothy M. Rankin, Nicholas A. Giovinco, Joseph L. Mills, and Yoky Matsuoka. 2014. A Heads-Up Display for Diabetic Limb Salvage Surgery A View Through the Google Looking Glass. Journal of diabetes science and technology 8, 5 (2014), 951--956.Google ScholarGoogle ScholarCross RefCross Ref
  3. Ronald T. Azuma. 1997. A Survey of Augmented Reality. Presence: Teleoperators and Virtual Environments 6, 4 (August 1997), 355--385. DOI:https://doi.org/10.1162/pres.1997.6.4.355Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. L. Bannon and K. Schmidt. 1992. Taking CSCW Seriously: Supporting articulation work. Computer Supported Cooperative Work 1, 1 (1992), 7--40.Google ScholarGoogle ScholarCross RefCross Ref
  5. Patrick Baudisch and Ruth Rosenholtz. 2003. Halo: a technique for visualizing off-screen objects. In Proceedings of the SIGCHI conference on Human factors in computing systems, 481--488. Retrieved June 27, 2017 from http://dl.acm.org/citation.cfm?id=642695Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Martin Bauer, Gerd Kortuem, and Zary Segall. 1999. "Where Are You Pointing At?" A Study of Remote Collaboration in a Wearable Videoconference System. In Proceedings of the 3rd IEEE International Symposium on Wearable Computers (ISWC '99), 151--. Retrieved September 21, 2016 from http://dl.acm.org/citation.cfm?id=519309.856501Google ScholarGoogle ScholarCross RefCross Ref
  7. Michel Beaudouin-Lafon and Alain Karsenty. 1992. Transparency and Awareness in a Real-time Groupware System. In Proceedings of the 5th Annual ACM Symposium on User Interface Software and Technology (UIST '92), 171--180. DOI:https://doi.org/10.1145/142621.142646Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Mark Billinghurst and Hirokazu Kato. 2002. Collaborative Augmented Reality. Commun. ACM 45, 7 (July 2002), 64--70. DOI:https://doi.org/10.1145/514236.514265Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Susan E. Brennan, Xin Chen, Christopher A. Dickinson, Mark B. Neider, and Gregory J. Zelinsky. 2008. Coordinating cognition: the costs and benefits of shared gaze during collaborative search. Cognition 106, 3 (March 2008), 1465--1477. DOI:https://doi.org/10.1016/j.cognition.2007.05.012Google ScholarGoogle ScholarCross RefCross Ref
  10. Nicko R. Caluya, Alexander Plopski, Jayzon F. Ty, Christian Sandor, Takafumi Taketomi, and Hirokazu Kato. 2018. Transferability of Spatial Maps: Augmented Versus Virtual Reality Training. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 387--393. DOI:https://doi.org/10.1109/VR.2018.8447561Google ScholarGoogle Scholar
  11. Jeffrey W. Chastine, Kristine Nagel, Ying Zhu, and Luca Yearsovich. 2007. Understanding the Design Space of Referencing in Collaborative Augmented Reality Environments. In Proceedings of Graphics Interface 2007 (GI '07), 207--214. DOI:https://doi.org/10.1145/1268517.1268552Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Drago Datcu, Stephan G. Lukosch, and Heide K. Lukosch. 2016. Handheld Augmented Reality for Distributed Collaborative Crime Scene Investigation. In Proceedings of the 19th International Conference on Supporting Group Work (GROUP '16), 267--276. DOI:https://doi.org/10.1145/2957276.2957302Google ScholarGoogle Scholar
  13. Arturo Deza, Jeffrey R. Peters, Grant S. Taylor, Amit Surana, and Miguel P. Eckstein. 2017. Attention Allocation Aid for Visual Search. arXiv:1701.03968 [cs] (2017), 220--231. DOI:https://doi.org/10.1145/3025453.3025834Google ScholarGoogle Scholar
  14. Paul Dourish and Victoria Bellotti. 1992. Awareness and coordination in shared workspaces. In CSCW '92: Proceedings of the 1992 ACM conference on Computer-supported cooperative work, 107--114. DOI:http://doi.acm.org/10.1145/143457.143468Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Omid Fakourfar, Kevin Ta, Richard Tang, Scott Bateman, and Anthony Tang. 2016. Stabilized Annotations for Mobile Remote Assistance. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 1548--1560. DOI:https://doi.org/10.1145/2858036.2858171Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Steffen Gauglitz, Benjamin Nuernberger, Matthew Turk, and Tobias Höllerer. 2014. World-stabilized Annotations and Virtual Scene Navigation for Remote Collaboration. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14), 449--459. DOI:https://doi.org/10.1145/2642918.2647372Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. K. Gupta, G. A. Lee, and M. Billinghurst. 2016. Do You See What I See? The Effect of Gaze Tracking on Task Space Remote Collaboration. IEEE Transactions on Visualization and Computer Graphics 22, 11 (November 2016), 2413--2422. DOI:https://doi.org/10.1109/TVCG.2016.2593778Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Carl Gutwin, Mutasem Barjawi, and David Pinelle. 2016. The Emergence of High-Speed Interaction and Coordination in a (Formerly) Turn-based Groupware Game. In Proceedings of the 19th International Conference on Supporting Group Work, 277--286.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Carl Gutwin and Saul Greenberg. 2002. A descriptive framework of workspace awareness for real-time groupware. Computer Supported Cooperative Work (CSCW) 11, 3--4 (2002), 411--446.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. J. Hagler, M. Lankes, and J. Diephuis. 2018. Animating Participants in Co-located Playful Mixed-Reality Installations. In 2018 IEEE 1st Workshop on Animation in Virtual and Augmented Environments (ANIVAE), 1--4. DOI:https://doi.org/10.1109/ANIVAE.2018.8587269Google ScholarGoogle Scholar
  21. Jon Hindmarsh, Mike Fraser, Christian Heath, Steve Benford, and Chris Greenhalgh. 1998. Fragmented Interaction: Establishing Mutual Orientation in Virtual Environments. In Proceedings of the 1998 ACM Conference on Computer Supported Cooperative Work (CSCW '98), 217--226. DOI:https://doi.org/10.1145/289444.289496Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Jon Hindmarsh, Mike Fraser, Christian Heath, Steve Benford, and Chris Greenhalgh. 1998. Fragmented interaction: establishing mutual orientation in virtual environments. In Proceedings of the 1998 ACM conference on Computer supported cooperative work, 217--226.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Hossein Mousavi Hondori, Maryam Khademi, Lucy Dodakian, Steven C Cramer, and Cristina Videira Lopes. 2013. A Spatial Augmented Reality rehab system for post-stroke hand rehabilitation. In Medicine Meets Virtual Reality, 279--285.Google ScholarGoogle Scholar
  24. Ke Huo, Tianyi Wang, Luis Paredes, Ana M. Villanueva, Yuanzhi Cao, and Karthik Ramani. 2018. SynchronizAR: Instant Synchronization for Spontaneous and Spatial Collaborations in Augmented Reality. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST '18), 19--30. DOI:https://doi.org/10.1145/3242587.3242595Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Steven Johnson, Madeleine Gibson, and Bilge Mutlu. 2015. Handheld or Handsfree?: Remote Collaboration via Lightweight Head-Mounted Displays and Handheld Devices. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '15), 1825--1836. DOI:https://doi.org/10.1145/2675133.2675176Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Kiyoshi Kiyokawa, Mark Billinghurst, Sohan E. Hayes, Anoop Gupta, Yuki Sannohe, and Hirokazu Kato. 2002. Communication behaviors of co-located users in collaborative AR interfaces. In Proceedings of the 1st International Symposium on Mixed and Augmented Reality, 139.Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Chris L. Kleinke. 1986. Gaze and eye contact: a research review. Psychological bulletin 100, 1 (1986), 78.Google ScholarGoogle Scholar
  28. Mikko Kytö, Barrett Ens, Thammathip Piumsomboon, Gun A. Lee, and Mark Billinghurst. 2018. Pinpointing: Precise Head- and Eye-Based Target Selection for Augmented Reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18), 81:1--81:14. DOI:https://doi.org/10.1145/3173574.3173655Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Cha Lee, Gustavo A. Rincon, Greg Meyer, Tobias Höllerer, and Doug A. Bowman. 2013. The effects of visual realism on search tasks in mixed reality simulation. IEEE transactions on visualization and computer graphics 19, 4 (2013), 547--556.Google ScholarGoogle Scholar
  30. Jerry Li, Mia Manavalan, Sarah D'Angelo, and Darren Gergle. 2016. Designing Shared Gaze Awareness for Remote Collaboration. In Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion (CSCW '16 Companion), 325--328. DOI:https://doi.org/10.1145/2818052.2869097Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Kristine Lund. 2007. The importance of gaze and gesture in interactive multimodal explanation. Language Resources and Evaluation 41, 3--4 (2007), 289--303.Google ScholarGoogle ScholarCross RefCross Ref
  32. Paul Milgram and Fumio Kishino. 1994. A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems 77, 12 (1994), 1321--1329.Google ScholarGoogle Scholar
  33. Robert J. Moore, Nicolas Ducheneaut, and Eric Nickell. 2007. Doing virtually nothing: Awareness and accountability in massively multiplayer online worlds. Computer Supported Cooperative Work (CSCW) 16, 3 (2007), 265--305.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Hermann J. Müller and Joseph Krummenacher. 2006. Visual search and selective attention. Visual Cognition 14, 4--8 (August 2006), 389--410. DOI:https://doi.org/10.1080/13506280500527676Google ScholarGoogle Scholar
  35. Jens Müller, Roman Rädle, and Harald Reiterer. 2016. Virtual Objects As Spatial Cues in Collaborative Mixed Reality Environments: How They Shape Communication Behavior and User Task Load. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 1245--1249. DOI:https://doi.org/10.1145/2858036.2858043Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Romy Müller, Jens R. Helmert, Sebastian Pannasch, and Boris M. Velichkovsky. 2013. Gaze transfer in remote cooperation: is it always helpful to see what your partner is attending to? Q J Exp Psychol (Hove) 66, 7 (2013), 1302--1316. DOI:https://doi.org/10.1080/17470218.2012.737813Google ScholarGoogle ScholarCross RefCross Ref
  37. Alaeddin Nassani, Hyungon Kim, Gun Lee, Mark Billinghurst, Tobias Langlotz, and Robert W. Lindeman. 2016. Augmented Reality Annotation for Social Video Sharing. In SIGGRAPH ASIA 2016 Mobile Graphics and Interactive Applications (SA '16), 9:1--9:5. DOI:https://doi.org/10.1145/2999508.2999529Google ScholarGoogle Scholar
  38. Thammathip Piumsomboon, Arindam Dey, Barrett Ens, Gun Lee, and Mark Billinghurst. 2019. The Effects of Sharing Awareness Cues in Collaborative Mixed Reality. Front. Robot. AI 6, (2019). DOI:https://doi.org/10.3389/frobt.2019.00005Google ScholarGoogle Scholar
  39. Ronald Poelman, Oytun Akman, Stephan Lukosch, and Pieter Jonker. 2012. As if Being There: Mediated Reality for Crime Scene Investigation. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW '12), 1267--1276. DOI:https://doi.org/10.1145/2145204.2145394Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Marc Pomplun, Tyler W Garaas, and Marisa Carrasco. 2013. The effects of task difficulty on visual search strategy in virtual 3D displays. Journal of vision 13, 3 (2013), 24--24.Google ScholarGoogle ScholarCross RefCross Ref
  41. Dieter Schmalstieg, Anton Fuhrmann, Gerd Hesina, Zsolt Szalavári, L Miguel Encarnaçao, Michael Gervautz, and Werner Purgathofer. 2002. The studierstube augmented reality project. Presence: Teleoperators and Virtual Environments 11, 1 (2002), 33--54.Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Dieter Schmalstieg and Tobias Hollerer. 2016. Augmented Reality: Principles and Practice (01 ed.). Addison Wesley, Boston.Google ScholarGoogle Scholar
  43. Christian Schnier, Karola Pitsch, Angelika Dierker, and Thomas Hermann. 2011. Collaboration in Augmented Reality: How to establish coordination and joint attention? In ECSCW 2011: Proceedings of the 12th European Conference on Computer Supported Cooperative Work, 24--28 September 2011, Aarhus Denmark, 405--416.Google ScholarGoogle ScholarCross RefCross Ref
  44. Bari Serim, Ken Pfeuffer, Hans Gellersen, and Giulio Jacucci. 2018. Visual Attention-Based Access: Granting Access Based on Users' Joint Attention on Shared Workspaces. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (2018), 133.Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Harri Siirtola, Oleg pakov, Howell Istance, and Kari-Jouko Räihä. 2019. Shared Gaze in Collaborative Visual Search. International Journal of Human--Computer Interaction 0, 0 (February 2019), 1--13. DOI:https://doi.org/10.1080/10447318.2019.1565746Google ScholarGoogle Scholar
  46. Randy Stein and Susan E. Brennan. 2004. Another Person's Eye Gaze As a Cue in Solving Programming Problems. In Proceedings of the 6th International Conference on Multimodal Interfaces (ICMI '04), 9--15. DOI:https://doi.org/10.1145/1027933.1027936Google ScholarGoogle Scholar
  47. L. Suchman. 1996. Supporting articulation work. 407--423. Retrieved from http://books.google.com/books?hl=de&lr=&id=9wlN9eOomacC&oi=fnd&pg=PA407&dq=Supporting+articulation+work&ots=fgGvj8yL3W&sig=t1qgKME9J04GkzS7Haw1MbytSJ4#v=onepage&q=Supporting%20articulation%20work&f=falseGoogle ScholarGoogle Scholar
  48. Anne M. Treisman and Garry Gelade. 1980. A feature-integration theory of attention. Cognitive Psychology 12, 1 (January 1980), 97--136. DOI:https://doi.org/10.1016/0010-0285(80)90005--5Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Roel Vertegaal. 1999. The GAZE Groupware System: Mediating Joint Attention in Multiparty Communication and Collaboration. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '99), 294--301. DOI:https://doi.org/10.1145/302979.303065Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Benjamin Volmer, James Baumeister, Stewart Von Itzstein, Ina Bornkessel-Schlesewsky, Matthias Schlesewsky, Mark Billinghurst, and Bruce H. Thomas. 2018. A Comparison of Predictive Spatial Augmented Reality Cues for Procedural Tasks. IEEE transactions on visualization and computer graphics (2018).Google ScholarGoogle Scholar
  51. Jeremy M. Wolfe. 2007. Guided search 4.0. In Integrated models of cognitive systems, W. Gray (ed.). Oxford, New York, 99--119. Retrieved March 7, 2017 from https://books.google.com/books?hl=en&lr=&id=eB9nDAAAQBAJ&oi=fnd&pg=PA99&dq=%22attention.+Search+tasks+could+be+divided%22+%22VISUAL+ATTENTION+AND%22+%22color+processes+could+guide+the%22+%22among+green),+attention+would+be+directed%22+%2220%E2%80%9330+ms/item+on+target-present+trials+and+a+bit%22+&ots=zSV_aFeqFI&sig=HXB0dHMmamx0mTE4dxDgOVdIkoEGoogle ScholarGoogle Scholar
  52. Nancy Yao, Jeff Brewer, Sarah D'Angelo, Mike Horn, and Darren Gergle. 2018. Visualizing Gaze Information from Multiple Students to Support Remote Instruction. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA '18), LBW051:1--LBW051:6. DOI:https://doi.org/10.1145/3170427.3188453Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Yanxia Zhang, Ken Pfeuffer, Ming Ki Chong, Jason Alexander, Andreas Bulling, and Hans Gellersen. 2017. Look together: using gaze for assisting co-located collaborative search. Pers Ubiquit Comput 21, 1 (February 2017), 173--186. DOI:https://doi.org/10.1007/s00779-016-0969-xGoogle ScholarGoogle ScholarDigital LibraryDigital Library
  54. Egui Zhu, Arash Hadadgar, Italo Masiello, and Nabil Zary. 2014. Augmented reality in healthcare education: an integrative review. PeerJ 2, (2014), e469.Google ScholarGoogle Scholar

Index Terms

  1. "I simply watched where she was looking at": Coordination in Short-term Synchronous Cooperative Mixed Reality

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!