EarphoneTrack: involving earphones into the ecosystem of acoustic motion tracking

ABSTRACT

Acoustic motion tracking is an exciting new research area with promising progress in the last few years. Due to the inherent low propagation speed in the air, acoustic signals have the unique advantage of fine sensing granularity compared to RF signals. Speakers and microphones nowadays are pervasively available in devices surrounding us, such as smartphones and voice-controlled smart speakers. Though promising, one fundamental issue hindering the adoption of acoustic-based motion tracking is that the positions of microphones and speakers inside a device are fixed, which greatly limits the flexibility of acoustic motion tracking. In this work, we propose a new modality of acoustic motion tracking using earphones. Earphone-based tracking mitigates the constraints associated with traditional smartphone-based tracking. With novel designs and comprehensive experiments, we show earphone-based motion tracking can achieve a great flexibility and a high accuracy at the same time. We believe this is an important step towards "earable" sensing.

References

1. Fadel Adib, Zachary Kabelac, Dina Katabi, and Rob Miller. 2014. WiTrack: motion tracking via radio reflections off the body. In Proc. of NSDI.
2. Sandip Agrawal, Ionut Constandache, Shravan Gaonkar, Romit Roy Choudhury, Kevin Caves, and Frank DeRuyter. 2011. Using mobile phones to write in air. In Proceedings of the 9th international conference on Mobile systems, applications, and services. 15--28.
3. John R. Buck Alan V. Oppenheim, Ronald W. Schafer. 1989. Discrete-time signal processing. Vol. volume 2. Prentice-Hall, Inc.
4. Inc Amazon.com. 2019. Amazon Alexa. https://developer.amazon.com/en-US/alexa.
5. Inc Apple. 2019. AirPods. https://www.apple.com/shop/product/MRXJ2AM/A/airpods-with-wireless-charging-case.
6. Inc Arduino.com. 2019. Arduino. https://www.arduino.cc/.
7. Inc Audio-Technica. 2019. ATH-CKL220iS. https://audio-technica.com.au/products/ath-ckl220is/.
8. Inc Bela.com. 2019. Bela. https://www.blta.com.
9. Kongyang Chen and Guang Tan. 2018. BikeGPS: Accurate Localization of Shared Bikes in Street Canyons via Low-Level GPS Cooperation. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 150--162.
10. Zicheng Chi, Yao Yao, Tiantian Xie, Xin Liu, Zhichuan Huang, Wei Wang, and Ting Zhu. 2018. EAR: Exploiting uncontrollable ambient RF signals in heterogeneous networks for gesture recognition. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems. 237--249.
11. Audio-Technica Corporation. 2019. ATH-CK350IS. http://sea.audio-technica.com/happenings/2018-05/audio-technica-unveils-ath-ck350is-new-in-ear-headphones-for-smartphones.
12. HUAWEI Corporation. 2019. Flypods Lite. https://www.hihonor.com/global/products/accessories/honor-flypods-lite/.
13. HTC Corporation. 2019. HTC VIVE. https://www.vive.com/us/product/vive-virtual-reality-system/.
14. HUAWEI Corporation. 2019. HUAWEI P20. https://consumer.huawei.com/en/support/phones/p20/.
15. Lenovo Corporation. 2019. ThinkPad P1 Gen2. https://www.lenovo.com/us/en/laptops/thinkpad/thinkpad-p/P1-Gen-2/p/22WS2WPP102.
16. Moshi Corporation. 2019. Mythro Earbuds. https://www.moshi.com/en/product/audio-earbuds-mythro-mic/gray.
17. Philips Corporation. 2019. Philips PRO6105BK. https://www.usa.philips.com/cp/PRO6105BK_00/6000-series-in-ear-headphones-with-mic.
18. SaLusTex Corporation. 2019. Exhibit at Healthcare Next Generation. https://salustek.com/en/25/.
19. Sony Corporation. 2019. Sony MDR-XB75AP. https://www.sony.com.my/electronics/in-ear-headphones/mdr-xb75ap.
20. Sony Corporation. 2019. WF-1000XM3. https://helpguide.sony.net/mdr/wf1000xm3/v1/en/contents/TP0002289865.html.
21. Samsung Corporation. 2020. Samsung Galaxy Buds+. https://www.samsung.com/us/mobile/audio/galaxy-buds-plus/.
22. Inc Google.com. 2019. Google Home. https://store.google.com/gb/product/google_home.
23. Apple Inc. 2019. AirPods Pro. https://support.apple.com/kb/SP811?viewlocale=en_US&locale=en_US.
24. AVTech Media Americas Inc. 2019. Headphone measures. https://www.innerfidelity.com/headphone-measurements.
25. Google Inc. 2019. Google Daydream. https://arvr.google.com/daydream/.
26. Quebec Inc. 2019. Headphones Frequency Response. https://www.thephonograph.net/headphones-frequency-response/.
27. Quebec Inc. 2019. Raw Frequency Response. https://www.rtings.com/headphones/tests/sound-quality/raw-frequency-response.
28. Sony Interactive Entertainment Inc. 2019. PlayStation VR. https://www.playstation.com/.
29. Pravein Govindan Kannan, Seshadri Padmanabha Venkatagiri, Mun Choon Chan, Akhihebbal L Ananda, and Li-Shiuan Peh. 2012. Low cost crowd counting using audio tones. In Proceedings of the 10th ACM Conference on Embedded Network Sensor Systems. 155--168.
30. Samuel J. Ling, Jeff Sanny, and William Moebs. 2016. University Physics - Volume 2 (OpenStax). (2016).
31. Yang Liu, Wuxiong Zhang, Yang Yang, Weidong Fang, Fei Qin, and Xuewu Dai. 2019. PAMT: Phase-based Acoustic Motion Tracking in Multipath Fading Environments. In IEEE INFOCOM 2019-IEEE Conference on Computer Communications. IEEE, 2386--2394.
32. Hiroyuki Manabe and Masaaki Fukumoto. 2012. Headphone taps: A simple technique to add input function to regular headphones. In Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services companion.
33. Wenguang Mao, Jian He, and Lili Qiu. 2016. CAT: high-precision acoustic motion tracking. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. ACM, 69--81.
34. Robert A. Meyers. 1987. Encyclopedia of physical science and technology.
35. Chulhong Min, Akhil Mathur, and Fahim Kawsar. 2018. Exploring audio and kinetic sensing on earable devices. In Proceedings of the 4th ACM Workshop on Wearable Systems and Applications. ACM, 5--10.
36. Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. FingerIO: Using Active Sonar for Fine-Grained Finger Tracking. (2016).
37. Hiroki Ota, Minghan Chao, Yuji Gao, Eric Wu, Li-Chia Tai, Kevin Chen, Yasutomo Matsuoka, Kosuke Iwai, Hossain M Fahad, Wei Gao, et al. 2017. 3d printed "earable" smart devices for real-time detection of core body temperature. ACS sensors 2, 7 (2017), 990--997.
38. Chunyi Peng, Guobin Shen, Yongguang Zhang, Yanlin Li, and Kun Tan. 2007. Beepbeep: a high accuracy acoustic ranging system using cots mobile devices. In Proceedings of the 5th international conference on Embedded networked sensor systems. ACM, 1--14.
39. Alexander D. Poularikas. 1998. The handbook of formulas and tables for signal processing.
40. Swadhin Pradhan, Eugene Chai, Karthikeyan Sundaresan, Lili Qiu, Mohammad A Khojastepour, and Sampath Rangarajan. 2017. Rio: A pervasive rfid-based touch gesture interface. In Proceedings of the 23rd Annual International Conference on Mobile Computing and Networking. 261--274.
41. Nissanka B Priyantha, Anit Chakraborty, and Hari Balakrishnan. 2000. The cricket location-support system. In Proceedings of the 6th annual international conference on Mobile computing and networking. ACM, 32--43.
42. Wenjie Ruan, Quan Z Sheng, Lei Yang, Tao Gu, Peipei Xu, and Longfei Shangguan. 2016. AudioGest: enabling fine-grained hand gesture detection by decoding echo signal. In Proceedings of the 2016 ACM international joint conference on pervasive and ubiquitous computing. 474--485.
43. Akane Sano, Takashi Tomita, and Haruo Oba. 2010. Applications using earphone with biosignal sensors. In Human Interface Society Meeting, Vol. 12. 1--6.
44. Sheng Shen, Mahanth Gowda, and Romit Roy Choudhury. 2018. Closing the gaps in inertial motion tracking. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. ACM, 429--444.
45. Ke Sun, Ting Zhao, Wei Wang, and Lei Xie. 2018. Vskin: Sensing touch gestures on surfaces of mobile devices using acoustic signals. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. ACM, 591--605.
46. Li Sun, Souvik Sen, Dimitrios Koutsonikolas, and Kyu-Han Kim. 2015. Widraw: Enabling hands-free drawing in the air on commodity wifi devices. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. ACM, 77--89.
47. Kazuhiro Taniguchi, Hisashi Kondo, Toshiya Tanaka, and Atsushi Nishikawa. 2018. Earable RCC: development of an earphone-type reliable chewing-count measurement device. Journal of healthcare engineering 2018 (2018).
48. Deepak Vasisht, Swarun Kumar, and Dina Katabi. 2016. Decimeter-level localization with a single WiFi access point. In 13th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 16). 165--178.
49. Aditya Virmani and Muhammad Shahzad. 2017. Position and orientation agnostic gesture recognition using wifi. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services. 252--264.
50. Anran Wang and Shyamnath Gollakota. 2019. MilliSonic: Pushing the Limits of Acoustic Motion Tracking. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 18.
51. Chuyu Wang, Jian Liu, Yingying Chen, Hongbo Liu, Lei Xie, Wei Wang, Bingbing He, and Sanglu Lu. 2018. Multi-touch in the air: Device-free finger tracking and gesture recognition via COTS RFID. In IEEE INFOCOM 2018-IEEE Conference on Computer Communications. IEEE, 1691--1699.
52. Hao Wang, Daqing Zhang, Junyi Ma, Yasha Wang, Yuxiang Wang, Dan Wu, Tao Gu, and Bing Xie. 2016. Human respiration detection with commodity wifi devices: do user location and body orientation matter?. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 25--36.
53. Jue Wang, Deepak Vasisht, and Dina Katabi. 2015. RF-IDraw: virtual touch screen in the air using RF signals. ACM SIGCOMM Computer Communication Review 44, 4 (2015), 235--246.
54. Wang Wei, Alex X. Liu, and Sun Ke. 2016. Device-free gesture tracking using acoustic signals. In International Conference on Mobile Computing & Networking.
55. Ning Xiao, Panlong Yang, Xiang-Yang Li, Yanyong Zhang, Yubo Yan, and Hao Zhou. 2019. MilliBack: Real-Time Plug-n-Play Millimeter Level Tracking Using Wireless Backscattering. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 3 (2019), 112.
56. Jie Xiong and Kyle Jamieson. 2013. Arraytrack: A fine-grained indoor location system. In Presented as part of the 10th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 13). 71--84.
57. Jie Yang, Simon Sidhom, Gayathri Chandrasekaran, Tam Vu, Hongbo Liu, Nicolae Cecan, Yingying Chen, Marco Gruteser, and Richard P Martin. 2011. Detecting driver phone use leveraging car speakers. In Proceedings of the 17th annual international conference on Mobile computing and networking. ACM, 97--108.
58. Lei Yang, Yekui Chen, Xiang-Yang Li, Chaowei Xiao, Mo Li, and Yunhao Liu. 2014. Tagoram: Real-time tracking of mobile RFID tags to high precision using COTS devices. In Proceedings of the 20th annual international conference on Mobile computing and networking. ACM, 237--248.
59. Shichao Yue, Hao He, Hao Wang, Hariharan Rahul, and Dina Katabi. 2018. Extracting multi-person respiration from entangled RF signals. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 2 (2018), 1--22.
60. Sangki Yun, Yi-Chao Chen, and Lili Qiu. 2015. Turning a mobile device into a mouse in the air. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 15--29.
61. Sangki Yun, Yi-Chao Chen, Huihuang Zheng, Lili Qiu, and Wenguang Mao. 2017. Strata: Fine-grained acoustic-based device-free tracking. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 15--28.
62. Chi Zhang, Josh Tabor, Jialiang Zhang, and Xinyu Zhang. 2015. Extending mobile interaction through near-field visible light sensing. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. ACM, 345--357.
63. Yunting Zhang, Jiliang Wang, Weiyi Wang, Zhao Wang, and Yunhao Liu. 2018. Vernier: Accurate and Fast Acoustic Motion Tracking Using Mobile Devices. In IEEE INFOCOM 2018-IEEE Conference on Computer Communications. IEEE, 1709--1717.
64. Zengbin Zhang, David Chu, Xiaomeng Chen, and Thomas Moscibroda. 2012. Swordfight: Enabling a new class of phone-to-phone action games on commodity phones. In Proceedings of the 10th international conference on Mobile systems, applications, and services. ACM, 1--14.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction devices

      • Sound-based input / output