RoomShift: Room-scale Dynamic Haptics for VR with Furniture-moving Swarm Robots

ABSTRACT

RoomShift is a room-scale dynamic haptic environment for virtual reality, using a small swarm of robots that can move furniture. RoomShift consists of nine shape-changing robots: Roombas with mechanical scissor lifts. These robots drive beneath a piece of furniture to lift, move and place it. By augmenting virtual scenes with physical objects, users can sit on, lean against, place and otherwise interact with furniture with their whole body; just as in the real world. When the virtual scene changes or users navigate within it, the swarm of robots dynamically reconfigures the physical environment to match the virtual content. We describe the hardware and software implementation, applications in virtual tours and architectural design and interaction techniques.

References

1. Parastoo Abtahi and Sean Follmer. 2018. Visuo-Haptic Illusions for Improving the Perceived Performance of Shape Displays. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 150.
2. Parastoo Abtahi, Benoit Landry, Jackie (Junrui) Yang, Marco Pavone, Sean Follmer, and James A. Landay. 2019. Beyond The Force: Using Quadcopters to Appropriate Objects and the Environment for Haptics in Virtual Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, NY, NY, USA, Article 359, 13 pages. DOI: http://dx.doi.org/10.1145/3290605.3300589
3. Bruno Araujo, Ricardo Jota, Varun Perumal, Jia Xian Yao, Karan Singh, and Daniel Wigdor. 2016. Snake Charmer: Physically Enabling Virtual Objects. In Proceedings of the TEI'16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction. ACM, 218--226.
4. Frederico Augugliaro, Sergei Lupashin, Michael Hamer, Cason Male, Markus Hehn, Mark W Mueller, Jan Sebastian Willmann, Fabio Gramazio, Matthias Kohler, and Raffaello D'Andrea. 2014. The flight assembled architecture installation: Cooperative construction with flying machines. IEEE Control Systems Magazine 34, 4 (2014), 46--64.
5. Mahdi Azmandian, Mark Hancock, Hrvoje Benko, Eyal Ofek, and Andrew D Wilson. 2016. Haptic retargeting: Dynamic repurposing of passive haptics for enhanced virtual reality experiences. In Proceedings of the 2016 chi conference on human factors in computing systems. ACM, 1968--1979.
6. Hrvoje Benko, Christian Holz, Mike Sinclair, and Eyal Ofek. 2016. Normaltouch and texturetouch: High-fidelity 3d haptic shape rendering on handheld virtual reality controllers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 717--728.
7. Lung-Pan Cheng, Li Chang, Sebastian Marwecki, and Patrick Baudisch. 2018. iTurk: Turning Passive Haptics into Active Haptics by Making Users Reconfigure Props in Virtual Reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 89.
8. Lung-Pan Cheng, Patrick Lühne, Pedro Lopes, Christoph Sterz, and Patrick Baudisch. 2014. Haptic turk: a motion platform based on people. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 3463--3472.
9. Lung-Pan Cheng, Sebastian Marwecki, and Patrick Baudisch. 2017. Mutual human actuation. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. ACM, 797--805.
10. Lung-Pan Cheng, Thijs Roumen, Hannes Rantzsch, Sven Köhler, Patrick Schmidt, Robert Kovacs, Johannes Jasper, Jonas Kemper, and Patrick Baudisch. 2015. Turkdeck: Physical virtual reality based on people. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, 417--426.
11. Inrak Choi, Heather Culbertson, Mark R Miller, Alex Olwal, and Sean Follmer. 2017. Grabity: A wearable haptic interface for simulating weight and grasping in virtual reality. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. ACM, 119--130.
12. Inrak Choi, Elliot W Hawkes, David L Christensen, Christopher J Ploch, and Sean Follmer. 2016. Wolverine: A wearable haptic interface for grasping in virtual reality. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 986--993.
13. Martin Dekan, F Duchon, and others. 2013. iRobot create used in education. Journal of Mechanics Engineering and Automation 3, 4 (2013), 197--202.
14. Alexandra Delazio, Ken Nakagaki, Roberta L Klatzky, Scott E Hudson, Jill Fain Lehman, and Alanson P Sample. 2018. Force jacket: Pneumatically-actuated jacket for embodied haptic experiences. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 320.
15. Darren Guinness, Annika Muehlbradt, Daniel Szafir, and Shaun K Kane. 2018. The Haptic Video Player: Using Mobile Robots to Create Tangible Video Annotations. In Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces. ACM, 203--211.
16. Eric Guizzo. 2008. Three engineers, hundreds of robots, one warehouse. IEEE spectrum 45, 7 (2008), 26--34.
17. Zachary M Hammond, Nathan S Usevitch, Elliot W Hawkes, and Sean Follmer. 2017. Pneumatic reel actuator: Design, modeling, and implementation. In Robotics and Automation (ICRA), 2017 IEEE International Conference on. IEEE, 626--633.
18. Zhenyi He, Fengyuan Zhu, and Ken Perlin. 2017. Physhare: sharing physical interaction in virtual reality. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. ACM, 17--19.
19. Seongkook Heo, Christina Chung, Geehyuk Lee, and Daniel Wigdor. 2018. Thor's hammer: An ungrounded force feedback device utilizing propeller-induced propulsive force. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 525.
20. Anuruddha Hettiarachchi and Daniel Wigdor. 2016. Annexing reality: Enabling opportunistic use of everyday objects as tangible proxies in augmented reality. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 1957--1967.
21. Matthias Hoppe, Pascal Knierim, Thomas Kosch, Markus Funk, Lauren Futami, Stefan Schneegass, Niels Henze, Albrecht Schmidt, and Tonja Machulla. 2018. VRHapticDrones: Providing Haptics in Virtual Reality through Quadcopters. In Proceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia. ACM, 7--18.
22. Hikaru Ibayashi, Yuta Sugiura, Daisuke Sakamoto, Natsuki Miyata, Mitsunori Tada, Takashi Okuma, Takeshi Kurata, Masaaki Mochimaru, and Takeo Igarashi. 2015. Dollhouse vr: a multi-view, multi-user collaborative design workspace with vr technology. In SIGGRAPH Asia 2015 Emerging Technologies. ACM, 8.
23. Brent Edward Insko, M Meehan, M Whitton, and F Brooks. 2001. Passive haptics significantly enhances virtual environments. Ph.D. Dissertation. University of North Carolina at Chapel Hill.
24. Hiroo Iwata. 1999. Walking about virtual environments on an infinite floor. In Proceedings IEEE Virtual Reality (Cat. No. 99CB36316). IEEE, 286--293.
25. Hiroo Iwata, Hiroaki Yano, Fumitaka Nakaizumi, and Ryo Kawamura. 2001. Project FEELEX: adding haptic surface to graphics. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, 469--476.
26. Seungwoo Je, Myung Jin Kim, Woojin Lee, Byungjoo Lee, Xing-Dong Yang, Pedro Lopes, and Andrea Bianchi. 2019. Aero-plane: A Handheld Force-Feedback Device that Renders Weight Motion Illusion on a Virtual 2D Plane. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. ACM, 763--775.
27. Ben Jenett and Kenneth Cheung. 2017. Bill-e: Robotic platform for locomotion and manipulation of lightweight space structures. In 25th AIAA/AHS Adaptive Structures Conference. 1876.
28. F Jensen and S Pellegrino. 2001. Arm development review of existing technologies. Technical Report. Cambridge Univ.(United Kingdom). Dept. of Engineering.
29. B Kent Joosten. 2007. Preliminary assessment of artificial gravity impacts to deep-space vehicle design. (2007).
30. Lawrence H Kim and Sean Follmer. 2017. Ubiswarm: Ubiquitous robotic interfaces and investigation of abstract motion as a display. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 66.
31. Lawrence H Kim and Sean Follmer. 2019. SwarmHaptics: Haptic Display with Swarm Robots. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 688.
32. Luv Kohli, Eric Burns, Dorian Miller, and Henry Fuchs. 2005. Combining passive haptics with redirected walking. In Proceedings of the 2005 international conference on Augmented tele-existence. ACM, 253--254.
33. Eike Langbehn, Paul Lubos, and Frank Steinicke. 2018. Evaluation of locomotion techniques for room-scale vr: Joystick, teleportation, and redirected walking. In Proceedings of the Virtual Reality International Conference-Laval Virtual. ACM, 4.
34. Mathieu Le Goc, Lawrence H Kim, Ali Parsaei, Jean-Daniel Fekete, Pierre Dragicevic, and Sean Follmer. 2016. Zooids: Building blocks for swarm user interfaces. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 97--109.
35. Robert W Lindeman, Yasuyuki Yanagida, Haruo Noma, and Kenichi Hosaka. 2006. Wearable vibrotactile systems for virtual contact and information display. Virtual Reality 9, 2--3 (2006), 203--213.
36. Pedro Lopes, Alexandra Ion, and Patrick Baudisch. 2015. Impacto: Simulating physical impact by combining tactile stimulation with electrical muscle stimulation. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, 11--19.
37. William A McNeely. 1993. Robotic graphics: a new approach to force feedback for virtual reality. In Proceedings of IEEE Virtual Reality Annual International Symposium. IEEE, 336--341.
38. Kazuki Nagai, Soma Tanoue, Katsuhito Akahane, and Makoto Sato. 2015. Wearable 6-DoF wrist haptic device SPIDAR-W. In SIGGRAPH Asia 2015 Haptic Media And Contents Design. ACM, 19.
39. Sharif Razzaque, Zachariah Kohn, and Mary C Whitton. 2005. Redirected walking. Citeseer.
40. Tomoya Sasaki, Richard Sahala Hartanto, Kao-Hua Liu, Keitarou Tsuchiya, Atsushi Hiyama, and Masahiko Inami. 2018. Leviopole: mid-air haptic interactions using multirotor. In ACM SIGGRAPH 2018 Emerging Technologies. ACM, 12.
41. Makoto Sato. 2002. Development of string-based force display: SPIDAR. In 8th international conference on virtual systems and multimedia. Citeseer.
42. Dominik Schmidt, Rob Kovacs, Vikram Mehta, Udayan Umapathi, Sven Köhler, Lung-Pan Cheng, and Patrick Baudisch. 2015. Level-ups: Motorized stilts that simulate stair steps in virtual reality. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 2157--2160.
43. Jotaro Shigeyama, Takeru Hashimoto, Shigeo Yoshida, Takuji Narumi, Tomohiro Tanikawa, and Michitaka Hirose. 2019. Transcalibur: A Weight Shifting Virtual Reality Controller for 2D Shape Rendering based on Computational Perception Model. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 11.
44. David Sirkin, Brian Mok, Stephen Yang, and Wendy Ju. 2015. Mechanical ottoman: how robotic furniture offers and withdraws support. In Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction. ACM, 11--18.
45. Alexa F Siu, Eric J Gonzalez, Shenli Yuan, Jason B Ginsberg, and Sean Follmer. 2018. Shapeshift: 2D spatial manipulation and self-actuation of tabletop shape displays for tangible and haptic interaction. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 291.
46. Ryo Suzuki, Ryosuke Nakayama, Dan Liu, Yasuaki Kakehi, Mark D. Gross, and Daniel Leithinger. 2020. LiftTiles: Constructive Building Blocks for Prototyping Room-scale Shape-changing Interfaces. In Proceedings of the Fourteenth International Conference on Tangible, Embedded, and Embodied Interaction. ACM.
47. Ryo Suzuki, Abigale Stangl, Mark D Gross, and Tom Yeh. 2017. FluxMarker: Enhancing Tactile Graphics with Dynamic Tactile Markers. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, 190--199.
48. Ryo Suzuki, Yasuaki Zheng, Clement Kakehi, Tom Yeh, Ellen Yi-Luen Do, Mark D Gross, and Daniel Leithinger. 2019. ShapeBots: Shape-changing Swarm Robots. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. ACM.
49. Kazuki Takashima, Takafumi Oyama, Yusuke Asari, Ehud Sharlin, Saul Greenberg, and Yoshifumi Kitamura. 2016. Study and design of a shape-shifting wall display. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems. ACM, 796--806.
50. Shohei Takei, Makoto Iida, and Takeshi Naemura. 2011. Kinereels: extension actuators for dynamic 3d shape. In ACM SIGGRAPH 2011 Posters. ACM, 84.
51. Shan-Yuan Teng, Cheng-Lung Lin, Chi-huan Chiang, Tzu-Sheng Kuo, Liwei Chan, Da-Yuan Huang, and Bing-Yu Chen. 2019. TilePoP: Tile-type Pop-up Prop for Virtual Reality. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. ACM.
52. Jur Van den Berg, Ming Lin, and Dinesh Manocha. 2008. Reciprocal velocity obstacles for real-time multi-agent navigation. In 2008 IEEE International Conference on Robotics and Automation. IEEE, 1928--1935.
53. Emanuel Vonach, Clemens Gatterer, and Hannes Kaufmann. 2017. VRRobot: Robot actuated props in an infinite virtual environment. In 2017 IEEE Virtual Reality (VR). IEEE, 74--83.
54. Justin Werfel, Kirstin Petersen, and Radhika Nagpal. 2014. Designing collective behavior in a termite-inspired robot construction team. Science 343, 6172 (2014), 754--758.
55. Highland Woodworking. 2019. Standard Furniture Dimensions. (2019). http://highlandwoodworking.com/library/furnituredimensions.pdf
56. Peter R Wurman, Raffaello D'Andrea, and Mick Mountz. 2008. Coordinating hundreds of cooperative, autonomous vehicles in warehouses. AI magazine 29, 1 (2008), 9--9.
57. Yiwei Zhao, Lawrence H Kim, Ye Wang, Mathieu Le Goc, and Sean Follmer. 2017. Robotic assembly of haptic proxy objects for tangible interaction and virtual reality. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces. ACM, 82--91.

Index Terms

• Human-centered computing