Abstract
Gesture recognition plays a fundamental role in emerging Human-Computer Interaction (HCI) paradigms. Recent advances in wireless sensing show promise for device-free and pervasive gesture recognition. Among them, RFID has gained much attention given its low-cost, light-weight and pervasiveness, but pioneer studies on RFID sensing still suffer two major problems when it comes to gesture recognition. The first is they are only evaluated on simple whole-body activities, rather than complex and fine-grained hand gestures. The second is they can not effectively work without retraining in new domains, i.e. new users or environments. To tackle these problems, in this paper, we propose RFree-GR, a domain-independent RFID system for complex and fine-grained gesture recognition. First of all, we exploit signals from the multi-tag array to profile the sophisticated spatio-temporal changes of hand gestures. Then, we elaborate a Multimodal Convolutional Neural Network (MCNN) to aggregate information across signals and abstract complex spatio-temporal patterns. Furthermore, we introduce an adversarial model to our deep learning architecture to remove domain-specific information while retaining information relevant to gesture recognition. We extensively evaluate RFree-GR on 16 commonly used American Sign Language (ASL) words. The average accuracy for new users and environments (new setup and new position) are $89.03%$, $90.21%$ and $88.38%$, respectively, significantly outperforming existing RFID based solutions, which demonstrates the superior effectiveness and generalizability of RFree-GR.
Supplemental Material
- Han Ding, Chen Qian, Jinsong Han, Ge Wang, Wei Xi, Kun Zhao, and Jizhong Zhao. 2017. Rfipad: Enabling cost-efficient and device-free in-air handwriting using passive tags. In 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS). IEEE, 447--457.Google Scholar
Cross Ref
- Xiaoyi Fan, Wei Gong, and Jiangchuan Liu. 2018. TagFree Activity Identification with RFIDs. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies , Vol. 2, 1 (2018), 7.Google Scholar
Digital Library
- Biyi Fang, Jillian Co, and Mi Zhang. 2017. DeepASL: Enabling Ubiquitous and Non-Intrusive Word and Sentence-Level Sign Language Translation. In Proceedings of the 15th ACM Conference on Embedded Network Sensor Systems. ACM, 5.Google Scholar
Digital Library
- Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised domain adaptation by backpropagation. In International conference on machine learning . 1180--1189.Google Scholar
Digital Library
- EPC Global. 2008. EPC radio-frequency identity protocols class-1 generation-2 UHF RFID protocol for communications at 860 MHz--960 MHz. Version , Vol. 1, 0 (2008), 23.Google Scholar
- Jiahui Hou, Xiang-Yang Li, Peide Zhu, Zefan Wang, Yu Wang, Jianwei Qian, and Panlong Yang. 2019. SignSpeaker: A Real-time, High-Precision SmartWatch-based Sign Language Translator. (2019).Google Scholar
- Anna Huang, Dong Wang, Run Zhao, and Qian Zhang. 2019. Au-Id: Automatic User Identification and Authentication through the Motions Captured from Sequential Human Activities Using RFID. , Vol. 3, 2, Article 48 (2019), bibinfonumpages26 pages. https://doi.org/10.1145/3328919Google Scholar
- Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015).Google Scholar
Digital Library
- Yasha Iravantchi, Mayank Goel, and Chris Harrison. 2019. BeamBand: Hand Gesture Sensing with Ultrasonic Beamforming. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 15.Google Scholar
Digital Library
- Wenjun Jiang, Chenglin Miao, Fenglong Ma, Shuochao Yao, Yaqing Wang, Ye Yuan, Hongfei Xue, Chen Song, Xin Ma, Dimitrios Koutsonikolas, et almbox. 2018. Towards environment independent device free human activity recognition. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking . 289--304.Google Scholar
Digital Library
- Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
- Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur. 2015. Audio augmentation for speech recognition. In Sixteenth Annual Conference of the International Speech Communication Association .Google Scholar
Cross Ref
- Gierad Laput and Chris Harrison. 2019. Sensing Fine-Grained Hand Activity with Smartwatches. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 338.Google Scholar
Digital Library
- Gierad Laput, Robert Xiao, and Chris Harrison. 2016. Viband: High-fidelity bio-acoustic sensing using commodity smartwatch accelerometers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 321--333.Google Scholar
Digital Library
- Kehuang Li, Zhengyu Zhou, and Chin-Hui Lee. 2016. Sign transition modeling and a scalable solution to continuous sign language recognition for real-world applications. ACM Transactions on Accessible Computing (TACCESS) , Vol. 8, 2 (2016), 7.Google Scholar
Digital Library
- Jaime Lien, Nicholas Gillian, M Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous gesture sensing with millimeter wave radar. ACM Transactions on Graphics (TOG) , Vol. 35, 4 (2016), 142.Google Scholar
Digital Library
- Yongsen Ma, Gang Zhou, Shuangquan Wang, Hongyang Zhao, and Woosub Jung. 2018. SignFi: Sign language recognition using WiFi. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies , Vol. 2, 1 (2018), 23.Google Scholar
Digital Library
- Pavlo Molchanov, Xiaodong Yang, Shalini Gupta, Kihwan Kim, Stephen Tyree, and Jan Kautz. 2016. Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4207--4215.Google Scholar
Cross Ref
- Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In ICML .Google Scholar
- Leigh Ellen Potter, Jake Araullo, and Lewis Carter. 2013. The leap motion controller: a view on sign language. In Proceedings of the 25th Australian computer-human interaction conference: augmentation, application, innovation, collaboration. ACM, 175--178.Google Scholar
Digital Library
- Panneer Selvam Santhalingam, Al Amin Hosain, Ding Zhang, Parth Pathak, Huzefa Rangwala, and Raja Kushalnagar. 2020. mmASL: Environment-Independent ASL Gesture Recognition Using 60 GHz Millimeter-wave Signals. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies , Vol. 4, 1 (2020), 1--30.Google Scholar
Digital Library
- Stefano Savazzi, Stephan Sigg, Monica Nicoli, Vittorio Rampa, Sanaz Kianoush, and Umberto Spagnolini. 2016. Device-free radio vision for assisted living: Leveraging wireless channel quality information for human sensing. IEEE Signal Processing Magazine , Vol. 33, 2 (2016), 45--58.Google Scholar
Cross Ref
- Sheng Shen, He Wang, and Romit Roy Choudhury. 2016. I am a smartwatch and i can track my user's arm. In Proceedings of the 14th annual international conference on Mobile systems, applications, and services. ACM, 85--96.Google Scholar
Digital Library
- Hagen Soltau, Hank Liao, and Hasim Sak. 2016. Neural speech recognizer: Acoustic-to-word LSTM model for large vocabulary speech recognition. arXiv preprint arXiv:1610.09975 (2016).Google Scholar
- Sheng Tan and Jie Yang. 2016. WiFinger: leveraging commodity WiFi for fine-grained finger gesture recognition. In Proceedings of the 17th ACM international symposium on mobile ad hoc networking and computing. ACM, 201--210.Google Scholar
Digital Library
- American Sign Language University. [n.d.]. BASIC ASL. http://lifeprint.com/asl101/pages-layout/concepts.htmGoogle Scholar
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998--6008.Google Scholar
- Raghav H Venkatnarayan, Griffin Page, and Muhammad Shahzad. 2018. Multi-user gesture recognition using WiFi. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 401--413.Google Scholar
Digital Library
- Chuyu Wang, Jian Liu, Yingying Chen, Hongbo Liu, Lei Xie, Wei Wang, Bingbing He, and Sanglu Lu. 2018. Multi-touch in the air: Device-free finger tracking and gesture recognition via COTS RFID. In IEEE INFOCOM 2018-IEEE Conference on Computer Communications. IEEE, 1691--1699.Google Scholar
Digital Library
- Jue Wang, Deepak Vasisht, and Dina Katabi. 2015. RF-IDraw: virtual touch screen in the air using RF signals. ACM SIGCOMM Computer Communication Review , Vol. 44, 4 (2015), 235--246.Google Scholar
Digital Library
- Minsi Wang, Bingbing Ni, and Xiaokang Yang. 2017. Recurrent modeling of interaction context for collective activity recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 3048--3056.Google Scholar
Cross Ref
- Saiwen Wang, Jie Song, Jaime Lien, Ivan Poupyrev, and Otmar Hilliges. 2016b. Interacting with soli: Exploring fine-grained dynamic gesture recognition in the radio-frequency spectrum. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 851--860.Google Scholar
Digital Library
- Wei Wang, Alex X Liu, and Ke Sun. 2016a. Device-free gesture tracking using acoustic signals. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. ACM, 82--94.Google Scholar
- Yan Wang, Jian Liu, Yingying Chen, Marco Gruteser, Jie Yang, and Hongbo Liu. 2014. E-eyes: device-free location-oriented activity identification using fine-grained wifi signatures. In Proceedings of the 20th annual international conference on Mobile computing and networking. ACM, 617--628.Google Scholar
Digital Library
- Huatao Xu, Dong Wang, Run Zhao, and Qian Zhang. 2019. AdaRF: Adaptive RFID-Based Indoor Localization Using Deep Learning Enhanced Holography. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. , Vol. 3, 3, Article 113 (Sept. 2019), bibinfonumpages22 pages. https://doi.org/10.1145/3351271Google Scholar
Digital Library
- Jianfei Yang, Han Zou, Yuxun Zhou, and Lihua Xie. 2019. Learning gestures from wifi: A siamese recurrent convolutional architecture. IEEE Internet of Things Journal , Vol. 6, 6 (2019), 10763--10772.Google Scholar
Cross Ref
- Lei Yang, Yekui Chen, Xiang-Yang Li, Chaowei Xiao, Mo Li, and Yunhao Liu. 2014. Tagoram: Real-time tracking of mobile RFID tags to high precision using COTS devices. In Proceedings of the 20th annual international conference on Mobile computing and networking. ACM, 237--248.Google Scholar
Digital Library
- Lei Yang, Qiongzheng Lin, Xiangyang Li, Tianci Liu, and Yunhao Liu. 2015. See Through Walls with COTS RFID System!. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (MobiCom '15). Association for Computing Machinery, New York, NY, USA, 487--499. https://doi.org/10.1145/2789168.2790100Google Scholar
Digital Library
- Hui-Shyong Yeo, Juyoung Lee, Hyung-il Kim, Aakar Gupta, Andrea Bianchi, Daniel Vogel, Hideki Koike, Woontack Woo, and Aaron Quigley. 2019. WRIST: Watch-Ring Interaction and Sensing Technique for Wrist Gestures and Macro-Micro Pointing. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services . 1--15.Google Scholar
Digital Library
- Youwei Zeng, Dan Wu, Ruiyang Gao, Tao Gu, and Daqing Zhang. 2018. Fullbreathe: Full human respiration detection exploiting complementarity of csi phase and amplitude of wifi signals. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies , Vol. 2, 3 (2018), 1--19.Google Scholar
Digital Library
- Q. Zhang, D. Li , R. Zhao, D. Wang, Y. Deng, and B. Chen. 2018. RFree-ID: An Unobtrusive Human Identification System Irrespective of Walking Cofactors Using COTS RFID. In 2018 IEEE International Conference on Pervasive Computing and Communications (PerCom). 1--10.Google Scholar
- Qian Zhang, Dong Wang, Run Zhao, and Yinggang Yu. 2019 a. MyoSign: enabling end-to-end sign language recognition with wearables. In Proceedings of the 24th International Conference on Intelligent User Interfaces. ACM, 650--660.Google Scholar
Digital Library
- Shigeng Zhang, Chengwei Yang, Xiaoyan Kui, Jianxin Wang, Xuan Liu, and Song Guo. 2019 b. ReActor: Real-time and Accurate Contactless Gesture Recognition with RFID. In 2019 16th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON) . IEEE, 1--9.Google Scholar
- Zhengyou Zhang. 2012. Microsoft kinect sensor and its effect. IEEE multimedia , Vol. 19, 2 (2012), 4--10.Google Scholar
Digital Library
- Mingmin Zhao, Shichao Yue, Dina Katabi, Tommi S Jaakkola, and Matt T Bianchi. 2017. Learning sleep stages from radio signals: A conditional adversarial architecture. In International Conference on Machine Learning . 4100--4109.Google Scholar
- R. Zhao, D. Wang , Q. Zhang, H. Chen, and A. Huang. 2018. CRH: A Contactless Respiration and Heartbeat Monitoring System with COTS RFID Tags. In 2018 15th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON). 1--9.Google Scholar
- Yongpan Zou, Jiang Xiao, Jinsong Han, Kaishun Wu, Yun Li, and Lionel M Ni. 2016. Grfid: A device-free rfid-based gesture recognition system. IEEE Transactions on Mobile Computing , Vol. 16, 2 (2016), 381--393.Google Scholar
Digital Library
Index Terms
Towards Domain-independent Complex and Fine-grained Gesture Recognition with RFID
Recommendations
Multi-scenario gesture recognition using Kinect
CGAMES '12: Proceedings of the 2012 17th International Conference on Computer Games: AI, Animation, Mobile, Interactive Multimedia, Educational & Serious Games (CGAMES)Hand gesture recognition (HGR) is an important research topic because some situations require silent communication with sign languages. Computational HGR systems assist silent communication, and help people learn a sign language. In this article, a ...
Gesture recognition using RFID technology
We propose a gesture recognition technique based on RFID: cheap and unintrusive passive RFID tags can be easily attached to or interweaved into user clothes, which are then read by RFID antennas. These readings can be used to recognize hand gestures, ...
A Method to Recognize Eyeball Movement Gesture using Infrared Distance Sensor Array on Eyewear
iiWAS2021: The 23rd International Conference on Information Integration and Web IntelligenceSensing technology for eyeball movement (i.e., gaze movement) has enabled various applications, e.g., hands-free input interfaces and provided information that helps us understand human beings in various fields. However, the existing method, which ...






Comments