Evaluation of finger position estimation with a small ranging sensor array

ABSTRACT

We implemented the input interface named 'Novest' which can estimate the finger position and classify the finger state (out-of-range/hovering/touching) on the back of a hand with a small ranging sensor array attached on the side of a smartwatch, each of the small ranging sensors gets distance data and strength of signal data. With a prototype of Novest, we conducted the first evaluation to assess the basic performance. In this paper, we conducted the formal user study with two different conditions: the sitting/standing conditions to evaluate the practical performance of Novest as an input interface. The results show that the finger position estimation accuracies in the sitting and standing conditions are 4.2 mm and 5.0 mm, respectively. Additionally, the finger state classification accuracies with parameters adjusted by grid search in the sitting and standing conditions are 97.2% and 97.9%, respectively.

References

1. Gierad Laput, Robert Xiao, Xiang 'Anthony' Chen, Scott E. Hudson, and Chris Harrison. 2014. Skin Buttons: Cheap, Small, Low-Powered and Clickable Fixed-Icon Laser Projectors. In: Proceedings of the 27th annual ACM symposium on User interface software and technology (UIST '15), 389--394.
2. Niloofar Dezfuli, Mohammadreza Khalilbeigi, Jochen Huber, Florian Müller, and Max Mühlhäuser. 2012. PalmRC: Imaginary Palm-based Remote Control for Eyes-free Television Interaction. In: Proceedings of the 10th European Conference on Interactive TV and Video (EuroITV '12), 27--34.
3. Chris Harrison, Hrvoje Benko, and Andrew D. Wilson. 2011. OmniTouch: Wearable Multitouch Interaction Everywhere. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11), 441--450.
4. Kei Nakatsuma, Hiroyuki Shinoda, Yasutoshi Makino, Katsunari Sato, and Takashi Maeno. 2011. Touch Interface on Back of the Hand. In: ACM SIGGRAPH 2011 Emerging Technologies (SIGGRAPH '11), ACM, 19.
5. Masa Ogata, Yuta Sugiura, Yasutoshi Makino, Masahiko Inami, and Michita Imai. 2013. SenSkin: Adapting Skin as a Soft Interface. In: Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13), 539--544.
6. Chris Harrison, Desney Tan, and Dan Morris. 2010. Skinput: Appropriating the Body as an Input Surface. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10), 453--462.
7. Adiyan Mujibiya, Xiang Cao, Desney S. Tan, Dan Morris, Shwetak N. Patel, and Jun Rekimoto. 2013. The Sound of Touch: On-body Touch and Gesture Sensing Based on Transdermal Ultrasound Propagation. In: Proceedings of the 2013 ACM international Conference on Interactive Tabletops and Surfaces (ITS '13), 189--198.
8. Cheng Zhang, AbdelKareem Bedri, Gabriel Reyes, Bailey Bercik, Omer T. Inan, Thad E. Starner, and Gregory D. Abowd. 2016. TapSkin: Recognizing On-Skin Input for Smartwatches. In: Proceedings of the 2016 ACM on Interactive Surfaces and Spaces (ISS '16), 13--22.
9. Rong-Hao Liang, Shu-Yang Lin, Chao-Huai Su, Kai-Yin Cheng, Bing-Yu Chen, and De-Nian Yang. 2001. SonarWatch: Appropriating the Forearm as a Slider Bar. In: SIGGRAPH Asia 2011 Emerging Technologies (SA '11), 5.
10. Martin Weigel, Aditya Shekhar Nittala, Alex Olwal, and Jürgen Steimle. 2017. SkinMarks: Enabling Interactions on Body Landmarks Using Conformal Skin Electronics. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 3095--3105.
11. Yang Zhang, Junhan Zhou, Gierad Laput, and Chris Harrison. 2016. Skin-Track: Using the Body as an Electrical Waveguide for Continuous Finger Tracking on the Skin. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 1491--1503.
12. Denys J.C. Matthies, Simon T. Perrault, Bodo Urban, and Shengdong Zhao. 2015. Botential: Localizing On-Body Gestures by Measuring Electrical Signatures on the Human Skin. In: Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '15), 207--216.
13. Liwei Chan, Rong-Hao Liang, Ming-Chang Tsai, Kai-Yin Cheng, Chao-Huai Su, Mike Y. Chen, Wen-Huang Cheng, and Bing-Yu Chen. 2013. FingerPad: Private and Subtle Interaction Using Fingertips. In: Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13), 255--260.
14. Chris Harrison, and Scott E. Hudson. 2009. Abracadabra: Wireless, High-Precision, and Unpowered Finger Input for Very Small Mobile Devices. In: Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (UIST '09), 121--124.
15. Chen Zhao, Ke-Yu Chen, Md Tanvir Islam Aumi, Shwetak Patel, and Matthew S. Reynolds. 2014. SideSwipe: Detecting In-air Gestures Around Mobile Devices Using Actual GSM Signals. In: Proceedings of the 27th Annual ACM symposium on User Interface Software and Technology (UIST '14), 527--534.
16. Mathieu Le Goc, Stuart Taylor, Shahram Izadi, and Cem Keskin. 2014. A low-cost Transparent Electric Field Sensor for 3D Interaction on Mobile Devices. In: Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems (CHI '14), 3167--3170.
17. Junhan Zhou, Yang Zhang, Gierad Laput, and Chris Harrison. 2016. AuraSense: Enabling Expressive Around-Smartwatch Interactions with Electric Field Sensing. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16), 81--86.
18. Jaime Lien, Nicholas Gillian, M. Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous Gesture Sensing with Millimeter Wave Radar. ACM Transactions on Graphics (TOG), 35.4: No.142.
19. Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. FingerIO: Using Active Sonar for Fine-Grained Finger Tracking. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 1515--1525.
20. Wei Wang, Alex X. Liu, and Ke Sun. 2016. Device-Free Gesture Tracking Using Acoustic Signals. In: Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking (MobiCom '16), 82--94.
21. Elliott Wen, Winston Seah, Bryan Ng, Xuefeng Liu, and Jiannong Cao. 2016. UbiTouch: Ubiquitous Smartphone Touchpads using Built-in Proximity and Ambient Light Sensors. In: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '16), 286--297.
22. Junjue Wang, Kaichen Zhao, Xinyu Zhang, and Chunyi Peng. 2014. Ubiquitous Keyboard for Small Mobile Devices: Harnessing Multipath Fading for Fine-Grained Keystroke Localization. In: Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '14), 14--27.
23. Chi Zhang, Josh Tabor, Jialiang Zhang, and Xinyu Zhang. 2015. Extending Mobile Interaction Through Near-Field Visible Light Sensing. In: Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (MobiCom '15), 345--357.
24. Alex Butler, Shahram Izadi, and Steve Hodges. 2008. SideSight: Multi-touch Interaction Around Small Devices. In: Proceedings of the 21st Annual ACM symposium on User Interface Software and Technology (UIST '08), 201--204.
25. Shenwei Liu, and François Guimbretière. 2012. FlexAura: A Flexible Near Surface Range Sensor. In: Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12), 327--330.
26. Jungsoo Kim, Jiasheng He, Kent Lyons, and Thad Starner. 2007. The Gesture Watch: A Wireless Contact-Free Gesture based Wrist Interface. In: Wearable Computers, 2007 11th IEEE International Symposium on. IEEE (ISWC '07), 15--22.
27. Sven Kratz, and Michael Rohs. 2009. HoverFlow: Expanding the Design Space of Around-Device Interaction. In: Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '09), 4.
28. Srinath Sridhar, Anders Markussen, Antti Oulasvirta, Christian Theobalt, and Sebastian Boring. 2017. WatchSense: On-and Above-Skin Input Sensing through a Wearable Depth Sensor. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 3891--3902.
29. Daniel Wigdor, Clifton Forlines, Patrick Baudisch, John Barnwell, and Chia Shen. 2007. Lucid Touch: A See-Through Mobile Device. In: Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (UIST '07), 269--278.
30. Patrick Baudisch, and Gerry Chu. 2009. Back-of-Device Interaction Allows Creating Very Small Touch Devices. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09), 1923--1932.
31. Jon Moeller, and Andruid Kerne. 2012. ZeroTouch: An Optical Multi-Touch and Free-Air Interaction Architecture. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12), 2165--2174.
32. Jue Wang, Deepak Vasisht, and Dina Katabi. 2014. RF-IDraw: Virtual Touch Screen in the Air Using RF Signals. In: Proceedings of the 2014 ACM conference on SIGCOMM (SIGCOMM '14), 235--246.
33. Hsin-Liu (Cindy) Kao, Artem Dementyev, Joseph A. Paradiso, and Chris Schmandt. 2015. NailO: Fingernails as an Input Surface. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 3015--3018.
34. Ke-Yu Chen, Kent Lyons, Sean White, and Shwetak Patel. 2013. uTrack: 3D Input Using Two Magnetic Sensors. In: Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13), 237--244.
35. Xiang 'Anthony' Chen, Julia Schwarz, Chris Harrison, Jennifer Mankoff, and Scott E. Hudson. 2014. Air+ touch: interweaving touch & in-air gestures. In Proceedings of the 27th annual ACM symposium on User Interface Software and Technology (UIST '14), 519--525.
36. Sony Mobile Communications Inc. : Smart Watch 3 SWR50, Sony Mobile Communications Inc. Retrieved August 26, 2016 from https://www.sonymobile.com/global-en/products/smart-products/smartwatch-3-swr50/
37. Chih-Chung Chang, and Chih-Jen Lin. 2011. LIBSVM: A Library for Support Vector Machines. ACM Transactions on Intelligent Systems and Technology (TIST), Vol. 2, No. 3, pp. 27:1--27:27.
38. Christian Holz, and Patrick Baudisch. 2010. The Generalized Perceived Input Point Model and How to Double Touch Accuracy by Extracting Fingerprints. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10), 581--590.
39. Marcos Serrano, Barrett M. Ens, and Pourang P. Irani. 2014. Exploring the Use of Hand-To-Face Input for Interacting with Head-Worn Displays. In: Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems (CHI '14), 3181--3190.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction devices

      • Pointing devices