skip to main content
research-article

Towards Domain-independent Complex and Fine-grained Gesture Recognition with RFID

Authors Info & Claims
Published:04 November 2020Publication History
Skip Abstract Section

Abstract

Gesture recognition plays a fundamental role in emerging Human-Computer Interaction (HCI) paradigms. Recent advances in wireless sensing show promise for device-free and pervasive gesture recognition. Among them, RFID has gained much attention given its low-cost, light-weight and pervasiveness, but pioneer studies on RFID sensing still suffer two major problems when it comes to gesture recognition. The first is they are only evaluated on simple whole-body activities, rather than complex and fine-grained hand gestures. The second is they can not effectively work without retraining in new domains, i.e. new users or environments. To tackle these problems, in this paper, we propose RFree-GR, a domain-independent RFID system for complex and fine-grained gesture recognition. First of all, we exploit signals from the multi-tag array to profile the sophisticated spatio-temporal changes of hand gestures. Then, we elaborate a Multimodal Convolutional Neural Network (MCNN) to aggregate information across signals and abstract complex spatio-temporal patterns. Furthermore, we introduce an adversarial model to our deep learning architecture to remove domain-specific information while retaining information relevant to gesture recognition. We extensively evaluate RFree-GR on 16 commonly used American Sign Language (ASL) words. The average accuracy for new users and environments (new setup and new position) are $89.03%$, $90.21%$ and $88.38%$, respectively, significantly outperforming existing RFID based solutions, which demonstrates the superior effectiveness and generalizability of RFree-GR.

Skip Supplemental Material Section

Supplemental Material

V4issA187.mov

Supplemental video

References

  1. Han Ding, Chen Qian, Jinsong Han, Ge Wang, Wei Xi, Kun Zhao, and Jizhong Zhao. 2017. Rfipad: Enabling cost-efficient and device-free in-air handwriting using passive tags. In 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS). IEEE, 447--457.Google ScholarGoogle ScholarCross RefCross Ref
  2. Xiaoyi Fan, Wei Gong, and Jiangchuan Liu. 2018. TagFree Activity Identification with RFIDs. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies , Vol. 2, 1 (2018), 7.Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Biyi Fang, Jillian Co, and Mi Zhang. 2017. DeepASL: Enabling Ubiquitous and Non-Intrusive Word and Sentence-Level Sign Language Translation. In Proceedings of the 15th ACM Conference on Embedded Network Sensor Systems. ACM, 5.Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised domain adaptation by backpropagation. In International conference on machine learning . 1180--1189.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. EPC Global. 2008. EPC radio-frequency identity protocols class-1 generation-2 UHF RFID protocol for communications at 860 MHz--960 MHz. Version , Vol. 1, 0 (2008), 23.Google ScholarGoogle Scholar
  6. Jiahui Hou, Xiang-Yang Li, Peide Zhu, Zefan Wang, Yu Wang, Jianwei Qian, and Panlong Yang. 2019. SignSpeaker: A Real-time, High-Precision SmartWatch-based Sign Language Translator. (2019).Google ScholarGoogle Scholar
  7. Anna Huang, Dong Wang, Run Zhao, and Qian Zhang. 2019. Au-Id: Automatic User Identification and Authentication through the Motions Captured from Sequential Human Activities Using RFID. , Vol. 3, 2, Article 48 (2019), bibinfonumpages26 pages. https://doi.org/10.1145/3328919Google ScholarGoogle Scholar
  8. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015).Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Yasha Iravantchi, Mayank Goel, and Chris Harrison. 2019. BeamBand: Hand Gesture Sensing with Ultrasonic Beamforming. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 15.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Wenjun Jiang, Chenglin Miao, Fenglong Ma, Shuochao Yao, Yaqing Wang, Ye Yuan, Hongfei Xue, Chen Song, Xin Ma, Dimitrios Koutsonikolas, et almbox. 2018. Towards environment independent device free human activity recognition. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking . 289--304.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google ScholarGoogle Scholar
  12. Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur. 2015. Audio augmentation for speech recognition. In Sixteenth Annual Conference of the International Speech Communication Association .Google ScholarGoogle ScholarCross RefCross Ref
  13. Gierad Laput and Chris Harrison. 2019. Sensing Fine-Grained Hand Activity with Smartwatches. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 338.Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Gierad Laput, Robert Xiao, and Chris Harrison. 2016. Viband: High-fidelity bio-acoustic sensing using commodity smartwatch accelerometers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 321--333.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Kehuang Li, Zhengyu Zhou, and Chin-Hui Lee. 2016. Sign transition modeling and a scalable solution to continuous sign language recognition for real-world applications. ACM Transactions on Accessible Computing (TACCESS) , Vol. 8, 2 (2016), 7.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Jaime Lien, Nicholas Gillian, M Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous gesture sensing with millimeter wave radar. ACM Transactions on Graphics (TOG) , Vol. 35, 4 (2016), 142.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Yongsen Ma, Gang Zhou, Shuangquan Wang, Hongyang Zhao, and Woosub Jung. 2018. SignFi: Sign language recognition using WiFi. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies , Vol. 2, 1 (2018), 23.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Pavlo Molchanov, Xiaodong Yang, Shalini Gupta, Kihwan Kim, Stephen Tyree, and Jan Kautz. 2016. Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4207--4215.Google ScholarGoogle ScholarCross RefCross Ref
  19. Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In ICML .Google ScholarGoogle Scholar
  20. Leigh Ellen Potter, Jake Araullo, and Lewis Carter. 2013. The leap motion controller: a view on sign language. In Proceedings of the 25th Australian computer-human interaction conference: augmentation, application, innovation, collaboration. ACM, 175--178.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Panneer Selvam Santhalingam, Al Amin Hosain, Ding Zhang, Parth Pathak, Huzefa Rangwala, and Raja Kushalnagar. 2020. mmASL: Environment-Independent ASL Gesture Recognition Using 60 GHz Millimeter-wave Signals. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies , Vol. 4, 1 (2020), 1--30.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Stefano Savazzi, Stephan Sigg, Monica Nicoli, Vittorio Rampa, Sanaz Kianoush, and Umberto Spagnolini. 2016. Device-free radio vision for assisted living: Leveraging wireless channel quality information for human sensing. IEEE Signal Processing Magazine , Vol. 33, 2 (2016), 45--58.Google ScholarGoogle ScholarCross RefCross Ref
  23. Sheng Shen, He Wang, and Romit Roy Choudhury. 2016. I am a smartwatch and i can track my user's arm. In Proceedings of the 14th annual international conference on Mobile systems, applications, and services. ACM, 85--96.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Hagen Soltau, Hank Liao, and Hasim Sak. 2016. Neural speech recognizer: Acoustic-to-word LSTM model for large vocabulary speech recognition. arXiv preprint arXiv:1610.09975 (2016).Google ScholarGoogle Scholar
  25. Sheng Tan and Jie Yang. 2016. WiFinger: leveraging commodity WiFi for fine-grained finger gesture recognition. In Proceedings of the 17th ACM international symposium on mobile ad hoc networking and computing. ACM, 201--210.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. American Sign Language University. [n.d.]. BASIC ASL. http://lifeprint.com/asl101/pages-layout/concepts.htmGoogle ScholarGoogle Scholar
  27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998--6008.Google ScholarGoogle Scholar
  28. Raghav H Venkatnarayan, Griffin Page, and Muhammad Shahzad. 2018. Multi-user gesture recognition using WiFi. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 401--413.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Chuyu Wang, Jian Liu, Yingying Chen, Hongbo Liu, Lei Xie, Wei Wang, Bingbing He, and Sanglu Lu. 2018. Multi-touch in the air: Device-free finger tracking and gesture recognition via COTS RFID. In IEEE INFOCOM 2018-IEEE Conference on Computer Communications. IEEE, 1691--1699.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Jue Wang, Deepak Vasisht, and Dina Katabi. 2015. RF-IDraw: virtual touch screen in the air using RF signals. ACM SIGCOMM Computer Communication Review , Vol. 44, 4 (2015), 235--246.Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Minsi Wang, Bingbing Ni, and Xiaokang Yang. 2017. Recurrent modeling of interaction context for collective activity recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 3048--3056.Google ScholarGoogle ScholarCross RefCross Ref
  32. Saiwen Wang, Jie Song, Jaime Lien, Ivan Poupyrev, and Otmar Hilliges. 2016b. Interacting with soli: Exploring fine-grained dynamic gesture recognition in the radio-frequency spectrum. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 851--860.Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Wei Wang, Alex X Liu, and Ke Sun. 2016a. Device-free gesture tracking using acoustic signals. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. ACM, 82--94.Google ScholarGoogle Scholar
  34. Yan Wang, Jian Liu, Yingying Chen, Marco Gruteser, Jie Yang, and Hongbo Liu. 2014. E-eyes: device-free location-oriented activity identification using fine-grained wifi signatures. In Proceedings of the 20th annual international conference on Mobile computing and networking. ACM, 617--628.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Huatao Xu, Dong Wang, Run Zhao, and Qian Zhang. 2019. AdaRF: Adaptive RFID-Based Indoor Localization Using Deep Learning Enhanced Holography. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. , Vol. 3, 3, Article 113 (Sept. 2019), bibinfonumpages22 pages. https://doi.org/10.1145/3351271Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Jianfei Yang, Han Zou, Yuxun Zhou, and Lihua Xie. 2019. Learning gestures from wifi: A siamese recurrent convolutional architecture. IEEE Internet of Things Journal , Vol. 6, 6 (2019), 10763--10772.Google ScholarGoogle ScholarCross RefCross Ref
  37. Lei Yang, Yekui Chen, Xiang-Yang Li, Chaowei Xiao, Mo Li, and Yunhao Liu. 2014. Tagoram: Real-time tracking of mobile RFID tags to high precision using COTS devices. In Proceedings of the 20th annual international conference on Mobile computing and networking. ACM, 237--248.Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Lei Yang, Qiongzheng Lin, Xiangyang Li, Tianci Liu, and Yunhao Liu. 2015. See Through Walls with COTS RFID System!. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (MobiCom '15). Association for Computing Machinery, New York, NY, USA, 487--499. https://doi.org/10.1145/2789168.2790100Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Hui-Shyong Yeo, Juyoung Lee, Hyung-il Kim, Aakar Gupta, Andrea Bianchi, Daniel Vogel, Hideki Koike, Woontack Woo, and Aaron Quigley. 2019. WRIST: Watch-Ring Interaction and Sensing Technique for Wrist Gestures and Macro-Micro Pointing. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services . 1--15.Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Youwei Zeng, Dan Wu, Ruiyang Gao, Tao Gu, and Daqing Zhang. 2018. Fullbreathe: Full human respiration detection exploiting complementarity of csi phase and amplitude of wifi signals. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies , Vol. 2, 3 (2018), 1--19.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Q. Zhang, D. Li , R. Zhao, D. Wang, Y. Deng, and B. Chen. 2018. RFree-ID: An Unobtrusive Human Identification System Irrespective of Walking Cofactors Using COTS RFID. In 2018 IEEE International Conference on Pervasive Computing and Communications (PerCom). 1--10.Google ScholarGoogle Scholar
  42. Qian Zhang, Dong Wang, Run Zhao, and Yinggang Yu. 2019 a. MyoSign: enabling end-to-end sign language recognition with wearables. In Proceedings of the 24th International Conference on Intelligent User Interfaces. ACM, 650--660.Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Shigeng Zhang, Chengwei Yang, Xiaoyan Kui, Jianxin Wang, Xuan Liu, and Song Guo. 2019 b. ReActor: Real-time and Accurate Contactless Gesture Recognition with RFID. In 2019 16th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON) . IEEE, 1--9.Google ScholarGoogle Scholar
  44. Zhengyou Zhang. 2012. Microsoft kinect sensor and its effect. IEEE multimedia , Vol. 19, 2 (2012), 4--10.Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Mingmin Zhao, Shichao Yue, Dina Katabi, Tommi S Jaakkola, and Matt T Bianchi. 2017. Learning sleep stages from radio signals: A conditional adversarial architecture. In International Conference on Machine Learning . 4100--4109.Google ScholarGoogle Scholar
  46. R. Zhao, D. Wang , Q. Zhang, H. Chen, and A. Huang. 2018. CRH: A Contactless Respiration and Heartbeat Monitoring System with COTS RFID Tags. In 2018 15th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON). 1--9.Google ScholarGoogle Scholar
  47. Yongpan Zou, Jiang Xiao, Jinsong Han, Kaishun Wu, Yun Li, and Lionel M Ni. 2016. Grfid: A device-free rfid-based gesture recognition system. IEEE Transactions on Mobile Computing , Vol. 16, 2 (2016), 381--393.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Towards Domain-independent Complex and Fine-grained Gesture Recognition with RFID

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!