Tactile hand motion and pose guidance for 3D interaction

ABSTRACT

We present a novel forearm-and-glove tactile interface that can enhance 3D interaction by guiding hand motor planning and coordination. In particular, we aim to improve hand motion and pose actions related to selection and manipulation tasks. Through our user studies, we illustrate how tactile patterns can guide the user, by triggering hand pose and motion changes, for example to grasp (select) and manipulate (move) an object. We discuss the potential and limitations of the interface, and outline future work.

References

1. C. Afonso and S. Beckhaus. 2011. How to Not Hit a Virtual Wall: Aural Spatial Awareness for Collision Avoidance in Virtual Environments. In Proceedings of the 6th Audio Mostly Conference: A Conference on Interaction with Sound (AM '11). ACM, 101--108.
2. R. B. Ammons. 1956. Effects of Knowledge of Performance: A Survey and Tentative Theoretical Formulation. The Journal of General Psychology 54, 2 (1956), 279--299.
3. F. Argelaguet, A. Kulik, A. Kunert, C. Andujar, and B. Froehlich. 2011. See-through techniques for referential awareness in collaborative virtual reality. International Journal of Human-Computer Studies 69, 6 (2011), 387--400.
4. R. Bane and T. Hollerer. 2004. Interactive Tools for Virtual X-Ray Vision in Mobile Augmented Reality. In Proceedings of the 3rd IEEE/ACM International Symposium on Mixed and Augmented Reality (ISMAR '04). IEEE, 231--239.
5. K. Bark, P. Khanna, R. Irwin, P. Kapur, S. A. Jax, L. Buxbaum, and K. Kuchenbecker. 2011. Lessons in using vibrotactile feedback to guide fast arm motions. In World Haptics Conference (WHC), 2011 IEEE. IEEE, 355--360.
6. P. W. Battaglia, M. Di Luca, M. Ernst, P. R. Schrater, T. Machulla, and D. Kersten. 2010. Within- and Cross-Modal Distance Information Disambiguate Visual Size-Change Perception. PLOS Computational Biology 6, 3 (03 2010), 1--10.
7. S. Beckhaus, F. Ritter, and T. Strothotte. 2000. CubicalPath-dynamic potential fields for guided exploration in virtual environments. In Proceedings the Eighth Pacific Conference on Computer Graphics and Applications. 387--459.
8. H. Benko, C. Holz, M. Sinclair, and E. Ofek. 2016. NormalTouch and TextureTouch: High-fidelity 3D Haptic Shape Rendering on Handheld Virtual Reality Controllers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, 717--728.
9. J. Blake and H. B. Gurocak. 2009. Haptic Glove With MR Brakes for Virtual Reality. IEEE/ASME Transactions on Mechatronics 14, 5 (2009), 606--615.
10. A. Bloomfield and N. Badler. 2008. Virtual training via vibrotactile arrays. Presence: Teleoperators and Virtual Environments 17, 2 (2008), 103--120.
11. A. Bloomfield, Y. Deng, J. Wampler, P. Rondot, M. Harth, D. and McManus, and N. Badler. 2003. A taxonomy and comparison of haptic actions for disassembly tasks. In Virtual Reality, 2003. Proceedings. IEEE. IEEE, 225--231.
12. G. C. Burdea. 1996. Force and Touch Feedback for Virtual Reality. John Wiley & Sons, Inc.
13. C. Chen, Y. Chen, Y. Chung, and N. Yu. 2016. Motion Guidance Sleeve: Guiding the Forearm Rotation Through External Artificial Muscles. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, 3272--3276.
14. D. Drobny and J. O. Borchers. 2010. Learning basic dance choreographies with different augmented feedback modalities. In Proceedings of the 28th International Conference on Human Factors in Computing Systems, CHI 2010, Extended Abstracts Volume, 2010. 3793--3798.
15. M. Ernst and M. Banks. 2002. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 6870 (2002), 429.
16. P. Gallotti, A. Raposo, and L. Soares. 2011. v-Glove: A 3D Virtual Touch Interface. In 2011 XIII Symposium on Virtual Reality. 242--251.
17. U. Gollner, T. Bieling, and G. Joost. 2012. Mobile Lorm Glove: Introducing a Communication Device for Deaf-blind People. In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction (TEI '12). ACM, 127--130.
18. P. Green and L. Wei-Haas. 1985. The rapid development of user interfaces: Experience with the Wizard of Oz method. In Proceedings of the Human Factors Society Annual Meeting, Vol. 29. SAGE Publications Sage CA: Los Angeles, CA, 470--474.
19. C. Hatzfeld and T.A. Kern. 2014. Engineering Haptic Devices: A Beginner's Guide. Springer London.
20. B. Holbert. 2007. Enhanced Targeting in a Haptic User Interface for the Physically Disabled Using a Force Feedback Mouse. Ph.D. Dissertation. Advisor(s) Huber, M. AAI3277666.
21. H. Jin, Q. Chen, Z. Chen, Y. Hu, and J. Zhang. 2016. Multi-LeapMotion sensor based demonstration for robotic refine tabletop object manipulation task. CAAI Transactions on Intelligence Technology 1, 1 (2016), 104 -- 113.
22. R. Johansson and R. Flanagan. 2009. Coding and use of tactile signals from the fingertips in object manipulation tasks. Nature reviews. Neuroscience 10, 5 (2009), 345.
23. R. S Johansson and Å B Vallbo. 1979. Tactile sensibility in the human hand: relative and absolute densities of four types of mechanoreceptive units in glabrous skin. The Journal of physiology 286, 1 (1979), 283--300.
24. K. Kaczmarek, J. Webster, P. Bach-y Rita, and W. Tompkins. 1991. Electrotactile and vibrotactile displays for sensory substitution systems. IEEE Transactions on Biomedical Engineering 38, 1 (1991), 1--16.
25. M. Klapdohr, B. Wöldecke, D. Marinos, J. Herder, C. Geiger, and W. Vonolfen. 2010. Vibrotactile Pitfalls: Arm Guidance for Moderators in Virtual TV Studios. In Proceedings of the 13th International Conference on Humans and Computers (HC '10). University of Aizu Press, 72--80.
26. E. Kruijff, A. Marquardt, C. Trepkowski, R. W. Lindeman, A. Hinkenjann, J. Maiero, and B. E. Riecke. 2016. On Your Feet!: Enhancing Vection in Leaning-Based Interfaces Through Multisensory Stimuli. In Proceedings of the 2016 Symposium on Spatial User Interaction (SUI '16). ACM, 149--158.
27. E. Kruijff, A. Marquardt, C. Trepkowski, J. Schild, and A. Hinkenjann. 2017. Designed Emotions: Challenges and Potential Methodologies for Improving Multisensory Cues to Enhance User Engagement in Immersive Systems. Vis. Comput. 33, 4 (April 2017), 471--488.
28. E. Kruijff, K. Wesche, G. and Riege, G. Goebbels, M. Kunstman, and D. Schmalstieg. 2006. Tactylus, a Pen-input Device Exploring Audiotactile Sensory Binding. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (VRST '06). ACM, 312--315.
29. J.J. LaViola, E. Kruijff, R.P. McMahan, D. Bowman, and I.P. Poupyrev. 2017. 3D User Interfaces: Theory and Practice. Pearson Education.
30. D. Levac and H. Sveistrup. 2014. Motor Learning and Virtual Reality., 25--46 pages.
31. J. Lieberman and C. Breazeal. 2007. TIKL: Development of a Wearable Vibrotactile Feedback Suit for Improved Human Motor Learning. (2007).
32. P. Lopes, D. Yüksel, F. Guimbretière, and P. Baudisch. 2016. Muscle-plotter: An Interactive System Based on Electrical Muscle Stimulation That Produces Spatial Output. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, 207--217.
33. V. Maheshwari and R. Saraf. 2008. Tactile Devices To Sense Touch on a Par with a Human Finger. Angewandte Chemie International Edition 47, 41 (2008), 7808--7826.
34. A. Marquardt, E. Kruijff, C. Trepkowski, J. Maiero, A. Schwandt, A. Hinkenjann, W. Stuerzlinger, and J. Schoening. 2018. Audio-Tactile Feedback for Enhancing 3D Manipulation. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (VRST '18). ACM.
35. J. Martinez, A. Garcia, M. Oliver, J. P. Molina, and P. Gonzalez. 2016. Identifying Virtual 3D Geometric Shapes with a Vibrotactile Glove. IEEE Computer Graphics and Applications 36, 1 (Jan 2016), 42--51.
36. T. H. Massie, K. Salisbury, et al. 1994. The phantom haptic interface: A device for probing virtual objects. In Proceedings of the ASME winter annual meeting, symposium on haptic interfaces for virtual environment and teleoperator systems, Vol. 55. 295--300.
37. T. McDaniel, D. Villanueva, S. Krishna, and S. Panchanathan. 2010. MOVeMENT: A framework for systematically mapping vibrotactile stimulations to fundamental body movements. In Haptic Audio-Visual Environments and Games (HAVE), 2010 IEEE International Symposium on. IEEE, 1--6.
38. R. P McMahan, D. A Bowman, D. J Zielinski, and R. B Brady. 2012. Evaluating display fidelity and interaction fidelity in a virtual reality game. IEEE transactions on visualization and computer graphics 18, 4 (2012), 626--633.
39. K. Nosaka, A. Aldayel, M. Jubeau, and T. C. Chen. 2011. Muscle damage induced by electrical stimulation. European Journal of Applied Physiology 111, 10 (03 Aug 2011), 2427.
40. D. Pai. 2005. Multisensory interaction: Real and virtual. In Robotics Research. The Eleventh International Symposium. Springer, 489--498.
41. E. Piateski and L. Jones. 2005. Vibrotactile pattern recognition on the arm and torso. In Eurohaptics Conference, 2005 and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2005. World Haptics 2005. First Joint. IEEE, 90--95.
42. H. Regenbrecht, J. Hauber, R. Schoenfelder, and A. Maegerlein. 2005. Virtual Reality Aided Assembly with Directional Vibro-tactile Feedback. In Proceedings of the 3rd International Conference on Computer Graphics and Interactive Techniques in Australasia and South East Asia (GRAPHITE '05). ACM, 381--387.
43. E. Ruffaldi, A. Filippeschi, A. Frisoli, O. Sandoval, C. A. Avizzano, and M. Bergamasco. 2009. Vibrotactile perception assessment for a rowing training system. In World Haptics 2009 - Third Joint EuroHaptics conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. 350--355.
44. K. Sato, K. Minamizawa, N. Kawakami, and S. Tachi. 2007. Haptic Telexistence. In ACM SIGGRAPH 2007 Emerging Technologies (SIGGRAPH '07). ACM, Article 10.
45. R.A. Schmidt and C.A. Wrisberg. 2004. Motor Learning and Performance. Human Kinetics.
46. C. Schönauer, K. Fukushi, A. Olwal, H. Kaufmann, and R. Raskar. 2012. Multimodal Motion Guidance: Techniques for Adaptive and Dynamic Feedback. In Proceedings of the 14th ACM International Conference on Multimodal Interaction (ICMI '12). ACM, 133--140.
47. D. Spelmezan, M. Jacobs, A. Hilgers, and J. Borchers. 2009. Tactile Motion Instructions for Physical Activities. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). ACM, 2243--2252.
48. C. Spence and S. Squire. 2003. Multisensory integration: maintaining the perception of synchrony. Current Biology 13, 13 (2003), R519--R521.
49. A. A. Stanley and K. J. Kuchenbecker. 2012. Evaluation of Tactile Feedback Methods for Wrist Rotation Guidance. EEE Trans. Haptics 5, 3 (Jan. 2012), 240--251.
50. E. Tamaki, T. Miyaki, and J. Rekimoto. 2011. PossessedHand: Techniques for Controlling Human Hands Using Electrical Muscles Stimuli. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, 543--552.
51. H. Uematsu, D. Ogawa, R. Okazaki, T. Hachisu, and H. Kajimoto. 2016. HALUX: projection-based interactive skin for digital sports. In SIGGRAPH Emerging Technologies.
52. G. H. VanDoorn, V. Dubaj, D. B. Wuillemin, B. L. Richardson, and M. A. Symmons. 2012. Cognitive Load Can Explain Differences in Active and Passive Touch. In Haptics: Perception, Devices, Mobility, and Communication, P. Isokoski and J. Springare (Eds.). Springer Berlin Heidelberg, 91--102.
53. S. Vishniakou, B. W. Lewis, X. Niu, A. Kargar, K. Sun, M. Kalajian, N. Park, M. Yang, Y. Jing, P. Brochu, et al. 2013. Tactile Feedback Display with Spatial and Temporal Resolutions. Scientific reports 3 (2013), 2521.
54. H. S. Vitense, J. A. Jacko, and V. K. Emery. 2002. Multimodal Feedback: Establishing a Performance Baseline for Improved Access by Individuals with Visual Impairments. In Proceedings of the Fifth International ACM Conference on Assistive Technologies (Assets '02). ACM, 49--56.
55. J. Zheng, Y. and Morrell. 2010. A vibrotactile feedback approach to posture guidance. In Haptics Symposium, 2010 IEEE. IEEE, 351--358.
56. M. Zhou, D.B. Jones, S.D. Schwaitzberg, and C.G.L. Cao. 2007. Role of Haptic Feedback and Cognitive Load in Surgical Skill Acquisition. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 51, 11 (2007),631--635.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods

    • Interaction devices

      • Haptic devices

    • Interaction techniques