RF-net: a unified meta-learning framework for RF-enabled one-shot human activity recognition

ABSTRACT

Radio-Frequency (RF) based device-free Human Activity Recognition (HAR) rises as a promising solution for many applications. However, device-free (or contactless) sensing is often more sensitive to environment changes than device-based (or wearable) sensing. Also, RF datasets strictly require on-line labeling during collection, starkly different from image and text data collections where human interpretations can be leveraged to perform off-line labeling. Therefore, existing solutions to RF-HAR entail a laborious data collection process for adapting to new environments. To this end, we propose RF-Net as a meta-learning based approach to one-shot RF-HAR; it reduces the labeling efforts for environment adaptation to the minimum level. In particular, we first examine three representative RF sensing techniques and two major meta-learning approaches. The results motivate us to innovate in two designs: i) a dual-path base HAR network, where both time and frequency domains are dedicated to learning powerful RF features including spatial and attention-based temporal ones, and ii) a metric-based meta-learning framework to enhance the fast adaption capability of the base network, including an RF-specific metric module along with a residual classification module. We conduct extensive experiments based on all three RF sensing techniques in multiple real-world indoor environments; all results strongly demonstrate the efficacy of RF-Net compared with state-of-the-art baselines.

References

1. Abdelnasser, H., Harras, K., and Youssef, M. A Ubiquitous WiFi-Based Fine-Grained Gesture Recognition System. IEEE Transactions on Mobile Computing 18, 11 (2019), 2474--2487.
2. Adib, F., Kabelac, Z., and Katabi, D. Multi-Person Localization via RF Body Reflections. In Proc. of the 11st USENIX NSDI (2015), p. 279--292.
3. Ali, K., Liu, A. X., Wang, W., and Shahzad, M. Keystroke Recognition Using WiFi Signals. In Proc. of the 21st ACM MobiCom (2015), pp. 90--102.
4. Bi, C., Xing, G., Hao, T., Huh-Yoo, J., Peng, W., Ma, M., and Chang, X. Family-Log: Monitoring Family Mealtime Activities by Mobile Devices. IEEE Transactions on Mobile Computing 19, 8 (2020), 1818--1830.
5. Bianco, S., Cadene, R., Celona, L., and Napoletano, P. Benchmark Analysis of Representative Deep Neural Network Architectures. IEEE Access 6 (2018), 64270--64277.
6. Cai, C., Chen, Z., Pu, H., Ye, L., Hu, M., and Luo, J. AcuTe: Acoustic Thermometer Empowered by a Single Smartphone. In Proc. of the 18th ACM SenSys (2020), pp. 1--14.
7. Cai, C., Pu, H., Hu, M., Zheng, R., and Luo, J. SST: Software Sonic Thermometer on Acoustic-enabled IoT Devices. IEEE Transactions on Mobile Computing (2020), 1--14.
8. Chen, W.-Y., Liu, Y.-C., Kira, Z., Wang, Y.-C. F., and Huang, J.-B. A Closer Look at Few-shot Classification. In Proc. of the 7th ICLR (2019), pp. 1--16.
9. Chen, Z., Li, Z., Zhang, X., Zhu, G., Xu, Y., Xiong, J., and Wang, X. AWL: Turning Spatial Aliasing From Foe to Friend for Accurate WiFi Localization. In Proc. of the 13th ACM CoNEXT (2017), pp. 238--250.
10. Chen, Z., Zhang, L., Jiang, C., Cao, Z., and Cui, W. WiFi CSI Based Passive Human Activity Recognition Using Attention Based BLSTM. IEEE Transaction on Mobile Computing 18, 11 (2018), 2714--2724.
11. Chi, Z., Yao, Y., Xie, T., Liu, X., Huang, Z., Wang, W., and Zhu, T. EAR: Exploiting Uncontrollable Ambient RF Signals in Heterogeneous Networks for Gesture Recognition. In Proc. of the 16th ACM SenSys (2018), pp. 237--249.
12. Finn, C., Abbeel, P., and Levine, S. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In Proc. of the 34th ICML (2017), pp. 1126--1135.
13. Ghose, A., Chakravarty, K., Agrawal, A. K., and Ahmed, N. Unobtrusive Indoor Surveillance of Patients at Home Using Multiple Kinect Sensors. In Proc. of the 11st ACM SenSys (2013), pp. 1--2.
14. Gong, T., Kim, Y., Shin, J., and Lee, S.-J. MetaSense: Few-Shot Adaptation to Untrained Conditions in Deep Mobile Sensing. In Proc. of the 17th ACM Sensys (2019), pp. 110--123.
15. Halperin, D., Hu, W., Sheth, A., and Wetherall, D. Tool Release: Gathering 802.11n Traces with Channel State Information. ACM SIGCOMM Computer Communication Review 41, 1 (2011), 53--53.
16. Hao, T., Bi, C., Xing, G., Chan, R., and Tu, L. MindfulWatch: A Smartwatch-Based System For Real-Time Respiration Monitoring During Meditation. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3 (Sept. 2017).
17. Hao, T., Xing, G., and Zhou, G. iSleep: Unobtrusive Sleep Quality Monitoring Using Smartphones. In Proc. of the 11st ACM SenSys (2013), pp. 1--14.
18. He, K., Zhang, X., Ren, S., and Sun, J. Deep Residual Learning for Image Recognition. In Proc. of the 29th IEEE CVPR (2016), pp. 770--778.
19. He, Y., Liang, J., and Liu, Y. Pervasive Floorplan Generation Based on Only Inertial Sensing: Feasibility, Design, and Implementation. IEEE Journal on Selected Areas in Communications 35, 5 (2017), 1132--1140.
20. Hnat, T. W., Srinivasan, V., Lu, J., Sookoor, T. I., Dawson, R., Stankovic, J., and Whitehouse, K. The Hitchhiker's Guide to Successful Residential Sensing Deployments. In Proc. of the 9th ACM SenSys (2011), pp. 232--245.
21. Hu, J.-F., Zheng, W.-S., Lai, J., and Zhang, J. Jointly Learning Heterogeneous Features for RGB-D Activity Recognition. In Proc. of the 28th IEEE CVPR (2015), pp. 5344--5352.
22. Humenberger, M., Schraml, S., Sulzbachner, C., Belbachir, A. N., Srp, A., and Vajda, F. Embedded Fall Detection with A Neural Network and Bio-Inspired Stereo Vision. In Proc. of the 25th IEEE CVPR Workshops (2012), pp. 60--67.
23. Ioffe, S., and Szegedy, C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proc. of the 32th ICML Workshop (2015), pp. 448--456.
24. Jalal, A., Uddin, M. Z., and Kim, T.-S. Depth Video-Based Human Activity Recognition System Using Translation and Scaling Invariant Features for Life Logging at Smart Home. IEEE Transaction on Consumer Electronics 58, 3 (2012), 863--871.
25. Jamal, M. A., and Qi, G.-J. Task Agnostic Meta-Learning for Few-shot Learning. In Proc. of the 32nd IEEE CVPR (2019), pp. 11719--11727.
26. Jiang, W., Miao, C., Ma, F., Yao, S., Wang, Y., Yuan, Y., Xue, H., Song, C., Ma, X., Koutsonikolas, D., and et al. Towards Environment Independent Device Free Human Activity Recognition. In Proc. of the 24th ACM MobiCom (2018), pp. 289--304.
27. Keally, M., Zhou, G., Xing, G., Wu, J., and Pyles, A. PBN: Towards Practical Activity Recognition Using Smartphone-Based Body Sensor Networks. In Proc. of the 9th ACM SenSys (2011), pp. 246--259.
28. Kempe, V. Inertial MEMS: Principles and Practice. Cambridge University Press, 2011.
29. Kim, J.-H., Jun, J., and Zhang, B.-T. Bilinear Attention Networks. In Proc. of the 32nd NIPS (2018), pp. 1564--1574.
30. Kim, S. Y., Han, H. G., Kim, J. W., Lee, S., and Kim, T. W. A Hand Gesture Recognition Sensor Using Reflected Impulses. IEEE Sensors Journal 17, 10 (2017), 2975--2976.
31. Koch, G., Zemel, R., and Salakhutdinov, R. Siamese Neural Networks for One-Shot Image Recognition. In Proc. of the 32th ICML Workshop (2015), vol. 2.
32. Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet Classification with Deep Convolutional Neural Networks. In Proc. of the 26th NIPS (2012), pp. 1097--1105.
33. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-Based Learning Applied to Document Recognition. Proc. of the IEEE 86, 11 (1998), 2278--2324.
34. Li, Z., Zhou, F., Chen, F., and Li, H. Meta-SGD: Learning to Learn Quickly for Few-shot Learning. arXiv preprint arXiv:1707.09835 (2017).
35. Lin, J., Gan, C., and Han, S. TSM: Temporal Shift Module for Efficient Video Understanding. In Proc. of the 33rd IEEE ICCV (2019), pp. 7083--7093.
36. Liu, X., Ghosh, P., Ulutan, O., Manjunath, B., Chan, K., and Govindan, R. Caesar: Cross-Camera Complex Activity Recognition. In Proc. of the 17th ACM SenSys (2019), pp. 232--244.
37. Ma, M., Fan, H., and Kitani, K. M. Going Deeper into First-Person Activity Recognition. In Proc. of the 29th IEEE CVPR (2016), pp. 1894--1903.
38. Ma, Y., Zhou, G., Wang, S., Zhao, H., and Jung, W. SignFi: Sign Language Recognition Using WiFi. In Proc. of the 18th ACM UbiComp (2018), pp. 23:1--21.
39. Mahafza, B. R. Radar Systems Analysis and Design Using MATLAB. CRC press, 2002.
40. Mao, W., Zhang, Z., Qiu, L., He, J., Cui, Y., and Yun, S. Indoor Follow Me Drone. In Proc. of the 15th ACM MobiSys (2017), pp. 345--358.
41. McIntosh, J., Marzo, A., Fraser, M., and Phillips, C. EchoFlex: Hand Gesture Recognition Using Ultrasound Imaging. In Proc. of the 29th ACM CHI (2017), pp. 1923--1934.
42. Munkhdalai, T., and Yu, H. Meta Networks. In Proc. of the 34th ICML (2017), pp. 2554--2563.
43. Nichol, A., and Schulman, J. Reptile: A Scalable Metalearning Algorithm. arXiv preprint arXiv:1803.02999 2 (2018), 2.
44. Park, J., and Cho, S. H. IR-UWB Radar Sensor for Human Gesture Recognition by Using Machine Learning. In Proc. of IEEE HPCC-SmartCity-DSS (2016), pp. 1246--1249.
45. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Proc. of the 32nd NIPS (2019), pp. 8024--8035.
46. Rusu, A. A., Rao, D., Sygnowski, J., Vinyals, O., Pascanu, R., Osindero, S., and Hadsell, R. Meta-Learning with Latent Embedding Optimization. Proc. of the 7th ICLR (2019).
47. Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., and Lillicrap, T. Meta-Learning with Memory-Augmented Neural Networks. In Proc. of the 33rd ICML (2016), pp. 1842--1850.
48. Sigg, S., Scholz, M., Shi, S., Ji, Y., and Beigl, M. RF-Sensing of Activities from Non-Cooperative Subjects in Device-Free Recognition Systems Using Ambient and Local Signals. IEEE Transaction on Mobile Computing 13, 4 (2013), 907--920.
49. Simonyan, K., and Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Proc. of the 3rd ICLR (2015).
50. Snell, J., Swersky, K., and Zemel, R. Prototypical Networks for Few-shot Learning. In Proc. of the 31st NIPS (2017).
51. Sun, Q., Liu, Y., Chua, T.-S., and Schiele, B. Meta-Transfer Learning for Few-shot Learning. In Proc. of the 32nd IEEE CVPR (2019), pp. 403--412.
52. Tian, Y., Lee, G.-H., He, H., Hsu, C.-Y., and Katabi, D. RF-Based Fall Monitoring Using Convolutional Neural Networks. In Proc. of the 18th ACM UbiComp (2018), pp. 137:1--24.
53. Truong, H., Zhang, S., Muncuk, U., Nguyen, P., Bui, N., Nguyen, A., Lv, Q., Chowdhury, K., Dinh, T., and Vu, T. Capband: Battery-Free Successive Capacitance Sensing Wristband for Hand Gesture Recognition. In Proc. of the 16th ACM SenSys (2018), pp. 54--67.
54. Tse, D., and Viswanath, P. Fundamentals of Wireless Communication. Cambridge University Press, USA, 2005.
55. Venkatnarayan, R. H., Page, G., and Shahzad, M. Multi-User Gesture Recognition Using WiFi. In Proc. of the 16th ACM MobiSys (2018), pp. 401--413.
56. Vinyals, O., Blundell, C., Lillicrap, T., Kavukcuoglu, K., and Wierstra, D. Matching Networks for One Shot Learning. In Proc. of the 30th NIPS (2016), pp. 3630--3638.
57. Virmani, A., and Shahzad, M. Position and Orientation Agnostic Gesture Recognition Using WiFi. In Proc. of the 15th ACM MobiSys (2017), pp. 252--264.
58. Wang, A., and Gollakota, S. Millisonic: Pushing the Limits of Acoustic Motion Tracking. In Proc. of the 31st ACM CHI (2019), pp. 1--11.
59. Wang, W., Liu, A. X., Shahzad, M., Ling, K., and Lu, S. Understanding and Modeling of WiFi Signal Based Human Activity Recognition. In Proc. of the 21st ACM MobiCom (2015), pp. 65--76.
60. Wang, Y., Wu, K., and Ni, L. M. WiFall: Device-Free Fall Detection by Wireless Networks. In Proc. of the 33rd IEEE INFOCOM (2014), pp. 271--279.
61. Wang, Z., Chen, Z., Singh, A., Garcia, L., Luo, J., and Srivastava, M. UWHear: Through-wall Extraction and Separation of Audio Vibrations Using Wireless Signals. In Proc. of the 18th ACM SenSys (2020), pp. 1--14.
62. Wu, D., Zhang, D., Xu, C., Wang, H., and Li, X. Device-Free WiFi Human Sensing: From Pattern-Based to Model-Based Approaches. IEEE Communications Magazine 55, 10 (2017), 91--97.
63. Xu, X., Yu, J., Chen, Y., Zhu, Y., Kong, L., and Li, M. BreathListener: Fine-Grained Breathing Monitoring in Driving Environments Utilizing Acoustic Signals. In Proc. of the 17th ACM MobiSys (2019), pp. 54--66.
64. Yang, Y., Hao, J., Luo, J., and Pan, S. CeilingCast: Energy Efficient and Location-Bound Broadcast Through LED-Camera Communication. In Proc. of the 35th IEEE INFOCOM (2016), pp. 1--9.
65. Yang, Y., Hao, J., Luo, J., and Pan, S. J. CeilingSee: Device-Free Occupancy Inference through Lighting Infrastructure based LED Sensing. In Proc. of the 15th IEEE PerCom (2017), p. 247--256.
66. Yang, Y., Luo, J., Hao, J., and Pan, S.J. Counting via LED Sensing: Inferring Occupancy Using Lighting Infrastructure. Elsevier Pervasive and Mobile Computing 45 (2018), 35 -- 54.
67. Yang, Y., Nie, J., and Luo, J. ReflexCode: Coding with Superposed Reflection Light for LED-Camera Communication. In Proc. of the 23rd ACM MobiCom (2017), p. 193--205.
68. Zhang, C., Li, F., Luo, J., and He, Y. iLocScan: Harnessing Multipath for Simultaneous Indoor Source Localization and Space Scanning. In Proc. of the 12th ACM SenSys (2014), p. 91--104.
69. Zhang, J., Tang, Z., Li, M., Fang, D., Nurmi, P., and Wang, Z. CrossSense: Towards Cross-Site and Large-Scale WiFi Sensing. In Proc. of the 24th ACM MobiCom (2018), pp. 305--320.
70. Zhang, X., Yao, L., Huang, C., Wang, S., Tan, M., Long, G., and Wang, C. Multi-Modality Sensor Data Classification with Selective Attention. In Proc. of the 27th IJCAI (2018), pp. 3111--3117.
71. Zhang, X., Zhou, X., Lin, M., and Sun, J. ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices. In Proc. of the 31st IEEE CVPR (2018), pp. 6848--6856.
72. Zhao, M., Li, T., Abu Alsheikh, M., Tian, Y., Zhao, H., Torralba, A., and Katabi, D. Through-Wall Human Pose Estimation Using Radio Signals. In Proc. of the 31st IEEE CVPR (2018), pp. 7356--7365.
73. Zhao, M., Tian, Y., Zhao, H., Alsheikh, M. A., Li, T., Hristov, R., Kabelac, Z., Katabi, D., and Torralba, A. RF-Based 3D Skeletons. In Proc. of ACM SIGCOMM (2018), p. 267--281.
74. Zheng, T., Chen, Z., Cai, C., Luo, J., and Zhang, X. V²iFi: in-Vehicle Vital Sign Monitoring via Compact RF Sensing. In Proc. of the 20th ACM UbiComp (2020), pp. 70:1--27.
75. Zheng, Y., Zhang, Y., Qian, K., Zhang, G., Liu, Y., Wu, C., and Yang, Z. Zero-Effort Cross-Domain Gesture Recognition with Wi-Fi. In Proc. of the 17th ACM MobiSys (2019), pp. 313--325.
76. Zhou, X., Huang, Q., Sun, X., Xue, X., and Wei, Y. Towards 3D Human Pose Construction in the Wild: a Weakly-Supervised Approach. In Proc. of the 26th ACM MobiCom (2020).

Index Terms

• Human-centered computing

  • Ubiquitous and mobile computing

    • Ubiquitous and mobile computing design and evaluation methods