skip to main content
research-article

Sensor-based Human Activity Recognition Using Graph LSTM and Multi-task Classification Model

Authors Info & Claims
Published:31 October 2022Publication History
Skip Abstract Section

Abstract

This paper explores human activities recognition from sensor-based multi-dimensional streams. Recently, deep learning-based methods such as LSTM and CNN have achieved important progress in practical application scenarios. However, in most previous deep learning-based methods exist potential challenges such as class imbalance and multi-modal heterogeneity with time and sensor signals. To handle those problems, we propose a graph LSTM and Metric Learning model (GLML) with multiple construction graph fusion by modeling the sensor-aspect signals and the graph-aspect activities. GLML is a semi-supervised co-training architecture, which can be seen as several iteratively pseudo-labels sampling processing in the unlabeled data. Specifically, we construct three graphs to capture the different relations in each timestamp. Meanwhile, the graph attention model and attention mechanism are proposed to integrate multiple graph interactions for different sensor signals. Furthermore, to obtain a fixed representation of hidden state units and their neighboring nodes, we introduce the Graph LSTM to learn the graph-aspect relations from graph-structured constructed graphs. Notably, we propose a multi-task classification model combining loss function for classification distribution with deep metric learning to enhance the representation ability of the multi-modal sensor data. Experimental results on three public datasets demonstrate that our proposed GLML model has at least 2.44% improved in average against the state-of-the-art methods.

REFERENCES

  1. [1] Abidine Bilal M’hamed, Fergani Lamya, Fergani Belkacem, and Oussalah Mourad. 2018. The joint use of sequence features combination and modified weighted SVM for improving daily activity recognition. Pattern Anal. Appl. 21 (2018), 119138.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. [2] Aggarwal Jake K. and Ryoo Michael S.. 2011. Human activity analysis: A review. ACM Computing Surveys (CSUR) 43, 3 (2011), 143.Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. [3] Alharbi Fayez, Ouarbya Lahcen, and Ward Jamie A.. 2022. Comparing sampling strategies for tackling imbalanced data in human activity recognition. Sensors 22, 4 (2022), 1373.Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Alsheikh Mohammad Abu, Selim Ahmed, Niyato Dusit, Doyle Linda, Lin Shaowei, and Tan Hwee-Pink. 2015. Deep activity recognition models with triaxial accelerometers. arXiv preprint arXiv:1511.04664 (2015).Google ScholarGoogle Scholar
  5. [5] Anguita Davide, Ghio Alessandro, Oneto Luca, Parra Xavier, and Reyes-Ortiz Jorge Luis. 2013. A public domain dataset for human activity recognition using smartphones. In Esann, Vol. 3. 3.Google ScholarGoogle Scholar
  6. [6] Colpas Paola Ariza, Vicario Enrico, De-La-Hoz-Franco Emiro, Pineres-Melo Marlon, Oviedo-Carrascal Ana, and Patara Fulvio. 2020. Unsupervised human activity recognition using the clustering approach: A review. Sensors 20, 9 (2020), 2702.Google ScholarGoogle ScholarCross RefCross Ref
  7. [7] Bai Lei, Yao Lina, Wang Xianzhi, Kanhere Salil S., Guo Bin, and Yu Zhiwen. 2020. Adversarial multi-view networks for activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 2 (2020), 122.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. [8] Bai Lu, Yeung Chris, Efstratiou Christos, and Chikomo Moyra. 2019. Motion2Vector: Unsupervised learning in human activity recognition using wrist-sensing data. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers. Association for Computing Machinery, New York, NY, USA, 537542.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. [9] Blanke U. and Schiele B.. 2010. Remember and transfer what you have learned - recognizing composite activities based on activity spotting. In International Symposium on Wearable Computers (ISWC) 2010. 18. Google ScholarGoogle ScholarCross RefCross Ref
  10. [10] Bulling Andreas, Blanke Ulf, and Schiele Bernt. 2014. A tutorial on human activity recognition using body-worn inertial sensors. ACM Computing Surveys (CSUR) 46, 3 (2014), 133.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. [11] Chavarriaga Ricardo, Sagha Hesam, Calatroni Alberto, Digumarti Sundara Tejaswi, Tröster Gerhard, Millán José del R., and Roggen Daniel. 2013. The opportunity challenge: A benchmark database for on-body sensor-based activity recognition. Pattern Recognition Letters 34, 15 (2013), 20332042.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. [12] Chen Kaixuan, Yao Lina, Zhang Dalin, Guo Bin, and Yu Zhiwen. 2019. Multi-agent attentional activity recognition. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI 2019). 13441350.Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] Chen K., Yao L., Zhang D., Wang X., Chang X., and Nie F.. 2020. A semisupervised recurrent convolutional attention model for human activity recognition. IEEE Transactions on Neural Networks and Learning Systems 31, 5 (2020), 17471756. Google ScholarGoogle ScholarCross RefCross Ref
  14. [14] Chen Kaixuan, Zhang Dalin, Yao Lina, Guo Bin, Yu Zhiwen, and Liu Yunhao. 2020. Deep learning for sensor-based human activity recognition: Overview, challenges and opportunities. arXiv preprint arXiv:2001.07416 (2020).Google ScholarGoogle Scholar
  15. [15] Chen Liming, Hoey Jesse, Nugent Chris D., Cook Diane J., and Yu Zhiwen. 2012. Sensor-based activity recognition. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 42, 6 (2012), 790808.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. [16] Dhiman Chhavi, Vishwakarma Dinesh Kumar, and Agarwal Paras. 2021. Part-wise spatio-temporal attention driven CNN-based 3D human action recognition. ACM Transactions on Multimidia Computing Communications and Applications 17, 3 (2021), 124.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. [17] Donahue J., Hendricks L. A., Rohrbach M., Venugopalan S., Guadarrama S., Saenko K., and Darrell T.. 2016. Long-term recurrent convolutional networks for visual recognition and description.IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 4 (2016), 677691.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. [18] Gu Fuqiang, Chung Mu-Huan, Chignell Mark, Valaee Shahrokh, Zhou Baoding, and Liu Xue. 2021. A survey on deep learning for human activity recognition. ACM Computing Surveys (CSUR) 54, 8 (2021), 134.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. [19] Guan Yu and Plötz Thomas. 2017. Ensembles of deep LSTM learners for activity recognition using wearables. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 2 (2017), 128.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. [20] Hacohen Guy and Weinshall Daphna. 2019. On the power of curriculum learning in training deep networks. In International Conference on Machine Learning. PMLR, 25352544.Google ScholarGoogle Scholar
  21. [21] Haque Albert, Alahi Alexandre, and Fei-Fei Li. 2016. Recurrent attention models for depth-based person identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 12291238.Google ScholarGoogle ScholarCross RefCross Ref
  22. [22] Huang Yi, Yang Xiaoshan, Gao Junyu, Sang Jitao, and Xu Changsheng. 2020. Knowledge-driven egocentric multimodal activity recognition. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 16, 4 (2020), 1133.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. [23] Iscen Ahmet, Tolias Giorgos, Avrithis Yannis, and Chum Ondrej. 2019. Label propagation for deep semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 50705079.Google ScholarGoogle ScholarCross RefCross Ref
  24. [24] Jha Saurav, Schiemer Martin, Zambonelli Franco, and Ye Juan. 2021. Continual learning in sensor-based human activity recognition: An empirical benchmark analysis. Information Sciences 575 (2021), 121.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. [25] Kautz Thomas, Groh Benjamin H., Hannink Julius, Jensen Ulf, Strubberg Holger, and Eskofier Bjoern M.. 2017. Activity recognition in beach volleyball using a deep convolutional neural network. Data Mining and Knowledge Discovery 31, 6 (2017), 16781705.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. [26] Lara O. D. and Labrador M. A.. 2013. A survey on human activity recognition using wearable sensors. IEEE Communications Surveys Tutorials 15, 3 (2013), 11921209. Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Li Chenglin, Tong Carrie Lu, Niu Di, Jiang Bei, Zuo Xiao, Cheng Lei, Xiong Jian, and Yang Jianming. 2021. Similarity embedding networks for robust human activity recognition. ACM Transactions on Knowledge Discovery from Data (TKDD) 15, 6 (2021), 117.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. [28] Liang Xiaodan, Shen Xiaohui, Feng Jiashi, Lin Liang, and Yan Shuicheng. 2016. Semantic object parsing with graph LSTM. In European Conference on Computer Vision. Springer, 125143.Google ScholarGoogle ScholarCross RefCross Ref
  29. [29] Liu Fanzhen, Xue Shan, Wu Jia, Zhou Chuan, Hu Wenbin, Paris Cecile, Nepal Surya, Yang Jian, and Yu Philip S.. 2021. Deep learning for community detection: Progress, challenges and opportunities. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence. 49814987.Google ScholarGoogle Scholar
  30. [30] Liu Jiaying, Song Sijie, Liu Chunhui, Li Yanghao, and Hu Yueyu. 2020. A benchmark dataset and comparison study for multi-modal human action analytics. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 16, 2 (2020), 124.Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. [31] Liu Li, Wang Shu, Peng Yuxin, Huang Zigang, Liu Ming, and Hu Bin. 2016. Mining intricate temporal rules for recognizing complex activities of daily living under uncertainty. Pattern Recognition 60 (2016), 10151028. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. [32] Liu Shengzhong, Yao Shuochao, Li Jinyang, Liu Dongxin, Wang Tianshi, Shao Huajie, and Abdelzaher Tarek. 2020. GlobalFusion: A global attentional deep learning framework for multisensor information fusion. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 1 (2020), 127.Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. [33] Ma Fan, Meng Deyu, Dong Xuanyi, and Yang Yi. 2020. Self-paced multi-view co-training.Journal of Machine Learning Research 21, 57 (2020), 138.Google ScholarGoogle Scholar
  34. [34] Ma Haojie, Li Wenzhong, Zhang Xiao, Gao Songcheng, and Lu Sanglu. 2019. AttnSense: Multi-level attention mechanism for multimodal human activity recognition. In IJCAI. 31093115.Google ScholarGoogle Scholar
  35. [35] Ma Xiaoxiao, Wu Jia, Xue Shan, Yang Jian, Zhou Chuan, Sheng Quan Z., Xiong Hui, and Akoglu Leman. 2021. A comprehensive survey on graph anomaly detection with deep learning. IEEE Transactions on Knowledge and Data Engineering (2021).Google ScholarGoogle ScholarCross RefCross Ref
  36. [36] Noori Farzan Majeed, Riegler Michael, Uddin Md. Zia, and Torresen Jim. 2020. Human activity recognition from multiple sensors data using multi-fusion representations and CNNs. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 16, 2 (2020), 119.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. [37] Nweke Henry Friday, Teh Ying Wah, Al-Garadi Mohammed Ali, and Alo Uzoma Rita. 2018. Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: State of the art and research challenges. Expert Systems with Applications 105 (2018), 233261.Google ScholarGoogle ScholarCross RefCross Ref
  38. [38] Ordóñez Francisco Javier and Roggen Daniel. 2016. Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition. Sensors 16, 1 (2016), 115.Google ScholarGoogle ScholarCross RefCross Ref
  39. [39] Qian Hangwei, Pan Sinno Jialin, and Miao Chunyan. 2021. Weakly-supervised sensor-based activity segmentation and recognition via learning from distributions. Artificial Intelligence 292 (2021), 103429.Google ScholarGoogle ScholarCross RefCross Ref
  40. [40] Qian Hangwei, Pan Sinno Jialin, Miao Chunyan, Qian H., Pan S. J., and Miao C.. 2021. Latent independent excitation for generalizable sensor-based cross-person activity recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 1192111929.Google ScholarGoogle ScholarCross RefCross Ref
  41. [41] Reiss Attila and Stricker Didier. 2012. Introducing a new benchmarked dataset for activity monitoring. In 2012 16th International Symposium on Wearable Computers. IEEE, 108109.Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. [42] Shang Chao, Chen Jie, and Bi Jinbo. 2021. Discrete graph structure learning for forecasting multiple time series. In Proceedings of the International Conference on Learning Representations (ICLR 2021). 113.Google ScholarGoogle Scholar
  43. [43] Shi Weiwei, Gong Yihong, Ding Chris, Tao Zhiheng MaXiaoyu, and Zheng Nanning. 2018. Transductive semi-supervised deep learning using min-max features. In Proceedings of the European Conference on Computer Vision (ECCV). 299315.Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. [44] Shu Xiangbo, Zhang Liyan, Qi Guo-Jun, Liu Wei, and Tang Jinhui. 2019. Spatiotemporal co-attention recurrent neural networks for human-skeleton motion prediction. arXiv preprint arXiv:1909.13245 (2019).Google ScholarGoogle Scholar
  45. [45] Stikic M., Larlus D., and Schiele B.. 2009. Multi-graph based semi-supervised learning for activity recognition. In 2009 International Symposium on Wearable Computers. 8592. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. [46] Su Xing, Xue Shan, Liu Fanzhen, Wu Jia, Yang Jian, Zhou Chuan, Hu Wenbin, Paris Cecile, Nepal Surya, Jin Di, et al. 2022. A comprehensive survey on community detection with deep learning. IEEE Transactions on Neural Networks and Learning Systems (2022).Google ScholarGoogle ScholarCross RefCross Ref
  47. [47] Suh Sungho, Rey Vitor Fortes, and Lukowicz Paul. 2022. Adversarial deep feature extraction network for user independent human activity recognition. In 2022 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 217226.Google ScholarGoogle ScholarCross RefCross Ref
  48. [48] Tang Jinhui, Shu Xiangbo, Yan Rui, and Zhang Liyan. 2019. Coherence constrained graph LSTM for group activity recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (2019).Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. [49] Veličković Petar, Cucurull Guillem, Casanova Arantxa, Romero Adriana, Lio Pietro, and Bengio Yoshua. 2018. Graph attention networks. In Proceedings of The International Conference on Learning Representations (ICLR 2018). 112.Google ScholarGoogle Scholar
  50. [50] Wang Dong, Hu Di, Li Xingjian, and Dou Dejing. 2020. Temporal relational modeling with self-supervision for action segmentation. arXiv preprint arXiv:2012.07508 (2020).Google ScholarGoogle Scholar
  51. [51] Wang Hongsong and Wang Liang. 2017. Modeling temporal dynamics and spatial configurations of actions using two-stream recurrent neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 499508.Google ScholarGoogle ScholarCross RefCross Ref
  52. [52] Wang Jindong, Chen Yiqiang, Hao Shuji, Peng Xiaohui, and Hu Lisha. 2019. Deep learning for sensor-based activity recognition: A survey. Pattern Recognition Letters 119 (2019), 311.Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. [53] Wang Xiaohan, Wu Yu, Zhu Linchao, and Yang Yi. 2020. Symbiotic attention with privileged information for egocentric action recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 1224912256.Google ScholarGoogle ScholarCross RefCross Ref
  54. [54] Weber Niklas, Härmä Aki, and Heskes Eindhoven Prof. Dr. Tom. 2017. Unsupervised Learning in Human Activity Recognition: A First Foray into Clustering Data Gathered from Wearable Sensors. Ph.D. Dissertation. Ph.D. Thesis, Radboud University, Nijmegen, The Netherlands.Google ScholarGoogle Scholar
  55. [55] Xue Hongfei, Jiang Wenjun, Miao Chenglin, Ma Fenglong, Wang Shiyang, Yuan Ye, Yao Shuochao, Zhang Aidong, and Su Lu. 2020. DeepMV: Multi-view deep learning for device-free human activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 1 (2020), 126.Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. [56] Yang Jianbo, Nguyen Minh Nhut, San Phyo Phyo, Li Xiao Li, and Krishnaswamy Shonali. 2015. Deep convolutional neural networks on multichannel time series for human activity recognition. In Twenty-fourth International Joint Conference on Artificial Intelligence. 39954001.Google ScholarGoogle Scholar
  57. [57] Yao Shuochao, Hu Shaohan, Zhao Yiran, Zhang Aston, and Abdelzaher Tarek. 2017. DeepSense: A unified deep learning framework for time-series mobile sensing data processing. In Proceedings of the 26th International Conference on World Wide Web. 351360.Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. [58] Zeng M., Yu T., Wang X., Nguyen L. T., Mengshoel O. J., and Lane I.. 2017. Semi-supervised convolutional neural networks for human activity recognition. In 2017 IEEE International Conference on Big Data (Big Data). 522529. Google ScholarGoogle ScholarCross RefCross Ref
  59. [59] Zhang Mi and Sawchuk Alexander A.. 2012. Motion primitive-based human activity recognition using a bag-of-features approach. In Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium. Association for Computing Machinery, New York, NY, USA, 631640.Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. [60] Zhang Xiang, Yao Lina, Huang Chaoran, Wang Sen, Tan Mingkui, Long Guodong, and Wang Can. 2018. Multi-modality sensor data classification with selective attention. In Proceedings of the 27th International Joint Conference on Artificial Intelligence. 31113117.Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. [61] Zhou Xiuzhuang, Wei Zeqiang, Xu Min, Qu Shan, and Guo Guodong. 2020. Facial depression recognition by deep joint label distribution and metric learning. IEEE Transactions on Affective Computing (2020).Google ScholarGoogle Scholar

Index Terms

  1. Sensor-based Human Activity Recognition Using Graph LSTM and Multi-task Classification Model

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Multimedia Computing, Communications, and Applications
          ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 18, Issue 3s
          October 2022
          381 pages
          ISSN:1551-6857
          EISSN:1551-6865
          DOI:10.1145/3567476
          • Editor:
          • Abdulmotaleb El Saddik
          Issue’s Table of Contents

          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 31 October 2022
          • Online AM: 8 September 2022
          • Accepted: 24 August 2022
          • Revised: 20 July 2022
          • Received: 15 November 2021
          Published in tomm Volume 18, Issue 3s

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Refereed

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Full Text

        View this article in Full Text.

        View Full Text

        HTML Format

        View this article in HTML Format .

        View HTML Format
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!