FM-track: pushing the limits of contactless multi-target tracking using acoustic signals

ABSTRACT

Contactless acoustic motion tracking enables new opportunities to interact with smart devices, such as smartphones and voice-controlled smart assistants. The speakers and microphones integrated in these devices provide unique opportunities to simultaneously track multiple targets in a fine-grained manner. To this end, we propose a system, namely FM-Track, that enables contactless multi-target tracking using acoustic signals. We first introduce a signal model to characterize the location and motion status of targets by fusing the information from multiple dimensions (i.e., range, velocity, and angle of targets). Then we develop a series of techniques to separate signals reflected from multiple targets and accurately track each individual target. We implement and evaluate FM-Track on both research-purpose hardware platform (i.e., Bela) and commercial devices (i.e., smartphones and smart speakers). Extensive experiments show that FM-Track can successfully differentiate two targets with a spacing as small as 1 cm, and achieve a median tracking accuracy of 0.86 cm and 0.11 cm for absolute range and displacement estimates respectively. For multi-target tracking, FM-Track can accurately track four targets and the tracking range can be up to 3 m.

References

1. Fadel Adib, Zachary Kabelac, and Dina Katabi. 2015. Multi-person localization via RF body reflections. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15). 279--292.
2. Amazon. 2020. Amazon Echo Dot 2nd Generation. https://www.amazon.com/All-New-Amazon-Echo-Dot-Add-Alexa-To-Any-Room/dp/B01DFKC2SO
3. Apple. 2020. iPhone 5c. https://support.apple.com/kb/sp684?locale=en_US
4. Francesco Belfiori, Wim van Rossum, and Peter Hoogeboom. 2012. 2D-MUSIC technique applied to a coherent FMCW MIMO radar. (2012).
5. Jagmohan Chauhan, Yining Hu, Suranga Seneviratne, Archan Misra, Aruna Seneviratne, and Youngki Lee. 2017. BreathPrint: Breathing acoustics-based user authentication. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 278--291.
6. Huijie Chen, Fan Li, and Yu Wang. 2017. EchoTrack: Acoustic device-free hand tracking on smartphones. In IEEE INFOCOM 2017-IEEE Conference on Computer Communications. IEEE, 1--9.
7. EARISE. 2020. AL-202 High-fidelity USB Acoustics System. https://www.sears.com/gemini-doctor-al-202-high-fidelity-usb-acoustics/p-SPM11706412330
8. SparkFun Electronics. 2020. ADMP401 MEMS microphones. https://www.sparkfun.com/products/9868
9. Doug Grimmett and Cherry Wakayama. 2013. Multistatic tracking for continous active sonar using Doppler-bearing measurements. In Proceedings of the 16th International Conference on Information Fusion. IEEE, 258--265.
10. Yalin Guercan and Alexander Yarovoy. 2017. Super-resolution algorithm for joint range-azimuth-Doppler estimation in automotive radars. In 2017 European Radar Conference (EURAD). IEEE, 73--76.
11. Sidhant Gupta, Daniel Morris, Shwetak Patel, and Desney Tan. 2012. Soundwave: using the doppler effect to sense gestures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1911--1914.
12. Yaxiong Huang, Simon C Busbridge, and Deshinder S Gill. 2001. Distortion and directivity in a digital transducer array loudspeaker. Journal of the Audio Engineering Society 49, 5 (2001), 337--352.
13. Qualisys Inc. 2020. Qualisys motion capture systems. https://www.qualisys.com/hardware/miqus/
14. Kiran Joshi, Dinesh Bharadia, Manikanta Kotaru, and Sachin Katti. 2015. WiDeo: Fine-grained Device-free Motion Tracing using RF Backscatter. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15). 189--204.
15. Chitra R Karanam, Belal Korany, and Yasamin Mostofi. 2019. Tracking from one side: multi-person passive tracking with WiFi magnitude measurements. In Proceedings of the 18th International Conference on Information Processing in Sensor Networks. 181--192.
16. Vladimir Katkovnik, Moon-Sik Lee, and Yong-Hoon Kim. 2002. High-resolution signal processing for a switch antenna array FMCW radar with a single channel receiver. In Sensor Array and Multichannel Signal Processing Workshop Proceedings, 2002. IEEE, 543--547.
17. Hyosu Kim, Anish Byanjankar, Yunxin Liu, Yuanchao Shu, and Insik Shin. 2018. UbiTap: Leveraging acoustic dispersion for ubiquitous touch interface on solid surfaces. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems. 211--223.
18. Manikanta Kotaru, Kiran Joshi, Dinesh Bharadia, and Sachin Katti. 2015. Spotfi: Decimeter level localization using wifi. In ACM SIGCOMM computer communication review, Vol. 45. ACM, 269--282.
19. Harold W Kuhn. 1955. The Hungarian method for the assignment problem. Naval research logistics quarterly 2, 1--2 (1955), 83--97.
20. Pei H Leong, Thushara D Abhayapala, and Tharaka A Lamahewa. 2013. Multiple target localization using wideband echo chirp signals. IEEE Transactions on Signal Processing 61, 16 (2013), 4077--4089.
21. Xiang Li, Daqing Zhang, Qin Lv, Jie Xiong, Shengjie Li, Yue Zhang, and Hong Mei. 2017. IndoTrack: Device-free indoor human tracking with commodity Wi-Fi. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 1--22.
22. Jaime Lien, Nicholas Gillian, M Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous gesture sensing with millimeter wave radar. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1--19.
23. Wenguang Mao, Jian He, and Lili Qiu. 2016. CAT: high-precision acoustic motion tracking. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. ACM, 69--81.
24. Wenguang Mao, Mei Wang, Wei Sun, Lili Qiu, Swadhin Pradhan, and Yi-Chao Chen. 2019. RNN-Based Room Scale Hand Motion Tracking. In The 25th Annual International Conference on Mobile Computing and Networking. ACM, 38.
25. Wenguang Mao, Zaiwei Zhang, Lili Qiu, Jian He, Yuchen Cui, and Sangki Yun. 2017. Indoor follow me drone. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 345--358.
26. MiniDSP. 2020. UMA-8-SP USB mic array. https://www.minidsp.com/products/usb-audio-interface/uma-8-sp-detail
27. Rajalakshmi Nandakumar, Krishna Kant Chintalapudi, Venkat Padmanabhan, and Ramarathnam Venkatesan. 2013. Dhwani: secure peer-to-peer acoustic NFC. ACM SIGCOMM Computer Communication Review 43, 4 (2013), 63--74.
28. Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. Fingerio: Using active sonar for fine-grained finger tracking. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 1515--1525.
29. Boaz Rafaely. 2004. Analysis and design of spherical microphone arrays. IEEE Transactions on speech and audio processing 13, 1 (2004), 135--143.
30. Hermann Rohling and M-M Meinecke. 2001. Waveform design principles for automotive radar systems. In 2001 CIE International Conference on Radar Proceedings (Cat No. 01TH8559). IEEE, 1--4.
31. Wenjie Ruan, Quan Z Sheng, Lei Yang, Tao Gu, Peipei Xu, and Longfei Shangguan. 2016. AudioGest: enabling fine-grained hand gesture detection by decoding echo signal. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 474--485.
32. Ralph Schmidt. 1986. Multiple emitter location and signal parameter estimation. IEEE transactions on antennas and propagation 34, 3 (1986), 276--280.
33. Sheng Shen, Daguan Chen, Yu-Lin Wei, Zhijian Yang, and Romit Roy Choudhury. 2020. Voice Localization Using Nearby Wall Reflections. In The 25th Annual International Conference on Mobile Computing and Networking. ACM. Preprint for MobiCom 2020.
34. Ke Sun, Wei Wang, Alex X Liu, and Haipeng Dai. 2018. Depth aware finger tapping on virtual displays. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 283--295.
35. Ke Sun, Ting Zhao, Wei Wang, and Lei Xie. 2018. Vskin: Sensing touch gestures on surfaces of mobile devices using acoustic signals. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. ACM, 591--605.
36. Bela Team. 2020. Bela Platform. https://bela.io
37. Cozmo Team. 2020. Cozmo Smart Robot. https://anki.com
38. Hercules Team. 2020. 4WD Hercules Mobile Robotic Platform. http://wiki.seeedstudio.com/Skeleton_Bot-4WD_hercules_mobile_robotic_platform
39. Elisabet Tiana-Roig, Finn Jacobsen, and Efrén Fernández Grande. 2010. Beam-forming with a circular microphone array for localization of environmental noise sources. The Journal of the Acoustical Society of America 128, 6 (2010), 3535--3542.
40. Yu-Chih Tung, Duc Bui, and Kang G Shin. 2018. Cross-platform support for rapid development of mobile acoustic sensing applications. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. 455--467.
41. Anran Wang and Shyamnath Gollakota. 2019. MilliSonic: Pushing the Limits of Acoustic Motion Tracking. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 18.
42. Anran Wang, Jacob E Sunshine, and Shyamnath Gollakota. 2019. Contactless infant monitoring using white noise. In The 25th Annual International Conference on Mobile Computing and Networking. 1--16.
43. Ju Wang, Jie Xiong, Hongbo Jiang, Xiaojiang Chen, and Dingyi Fang. 2017. D-Watch: Embracing "Bad" multipaths for device-free localization with COTS RFID devices. IEEE/ACM Transactions on Networking 25, 6 (2017), 3559--3572.
44. Tianben Wang, Daqing Zhang, Yuanqing Zheng, Tao Gu, Xingshe Zhou, and Bernadette Dorizzi. 2018. C-FMCW based contactless respiration detection using acoustic signal. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018), 170.
45. Wei Wang, Alex X Liu, and Ke Sun. 2016. Device-free gesture tracking using acoustic signals. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. ACM, 82--94.
46. Yadong Xie, Fan Li, Yue Wu, Song Yang, and Yu Wang. 2019. D3-Guard: Acoustic-based Drowsy Driving Detection Using Smartphones. In IEEE INFOCOM 2019-IEEE Conference on Computer Communications. IEEE, 1225--1233.
47. Yaxiong Xie, Jie Xiong, Mo Li, and Kyle Jamieson. 2019. mD-Track: Leveraging multi-dimensionality for passive indoor Wi-Fi tracking. In The 25th Annual International Conference on Mobile Computing and Networking. ACM, 1--16.
48. Jie Xiong and Kyle Jamieson. 2013. Arraytrack: A fine-grained indoor location system. In Presented as part of the 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI 13). 71--84.
49. Jie Xiong, Karthikeyan Sundaresan, and Kyle Jamieson. 2015. Tonetrack: Leveraging frequency-agile radios for time-based indoor wireless localization. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. 537--549.
50. Xiangyu Xu, Jiadi Yu, Yingying Chen, Yanmin Zhu, Linghe Kong, and Minglu Li. 2019. BreathListener: Fine-grained Breathing Monitoring in Driving Environments Utilizing Acoustic Signals. In Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 54--66.
51. Lei Yang, Yekui Chen, Xiang-Yang Li, Chaowei Xiao, Mo Li, and Yunhao Liu. 2014. Tagoram: Real-time tracking of mobile RFID tags to high precision using COTS devices. In Proceedings of the 20th annual international conference on Mobile computing and networking. 237--248.
52. Sangki Yun, Yi-Chao Chen, and Lili Qiu. 2015. Turning a mobile device into a mouse in the air. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services. 15--29.
53. Sangki Yun, Yi-Chao Chen, Huihuang Zheng, Lili Qiu, and Wenguang Mao. 2017. Strata: Fine-grained acoustic-based device-free tracking. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 15--28.
54. Cheng Zhang, Qiuyue Xue, Anandghan Waghmare, Sumeet Jain, Yiming Pu, Sinan Hersek, Kent Lyons, Kenneth A Cunefare, Omer T Inan, and Gregory D Abowd. 2017. Soundtrak: Continuous 3d tracking of a finger using active acoustics. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 2 (2017), 1--25.
55. Maotian Zhang, Qian Dai, Panlong Yang, Jie Xiong, Chang Tian, and Chaocan Xiang. 2018. idial: Enabling a virtual dial plate on the hand back for around-device interaction. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 1 (2018), 55.

Index Terms

• Human-centered computing

  • Ubiquitous and mobile computing

    • Ubiquitous and mobile computing systems and tools