Abstract
This paper explores human activities recognition from sensor-based multi-dimensional streams. Recently, deep learning-based methods such as LSTM and CNN have achieved important progress in practical application scenarios. However, in most previous deep learning-based methods exist potential challenges such as class imbalance and multi-modal heterogeneity with time and sensor signals. To handle those problems, we propose a graph LSTM and Metric Learning model (GLML) with multiple construction graph fusion by modeling the sensor-aspect signals and the graph-aspect activities. GLML is a semi-supervised co-training architecture, which can be seen as several iteratively pseudo-labels sampling processing in the unlabeled data. Specifically, we construct three graphs to capture the different relations in each timestamp. Meanwhile, the graph attention model and attention mechanism are proposed to integrate multiple graph interactions for different sensor signals. Furthermore, to obtain a fixed representation of hidden state units and their neighboring nodes, we introduce the Graph LSTM to learn the graph-aspect relations from graph-structured constructed graphs. Notably, we propose a multi-task classification model combining loss function for classification distribution with deep metric learning to enhance the representation ability of the multi-modal sensor data. Experimental results on three public datasets demonstrate that our proposed GLML model has at least 2.44% improved in average against the state-of-the-art methods.
- [1] . 2018. The joint use of sequence features combination and modified weighted SVM for improving daily activity recognition. Pattern Anal. Appl. 21 (2018), 119–138.Google Scholar
Digital Library
- [2] . 2011. Human activity analysis: A review. ACM Computing Surveys (CSUR) 43, 3 (2011), 1–43.Google Scholar
Digital Library
- [3] . 2022. Comparing sampling strategies for tackling imbalanced data in human activity recognition. Sensors 22, 4 (2022), 1373.Google Scholar
Cross Ref
- [4] . 2015. Deep activity recognition models with triaxial accelerometers. arXiv preprint arXiv:1511.04664 (2015).Google Scholar
- [5] . 2013. A public domain dataset for human activity recognition using smartphones. In Esann, Vol. 3. 3.Google Scholar
- [6] . 2020. Unsupervised human activity recognition using the clustering approach: A review. Sensors 20, 9 (2020), 2702.Google Scholar
Cross Ref
- [7] . 2020. Adversarial multi-view networks for activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 2 (2020), 1–22.Google Scholar
Digital Library
- [8] . 2019. Motion2Vector: Unsupervised learning in human activity recognition using wrist-sensing data. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers. Association for Computing Machinery, New York, NY, USA, 537–542.Google Scholar
Digital Library
- [9] . 2010. Remember and transfer what you have learned - recognizing composite activities based on activity spotting. In International Symposium on Wearable Computers (ISWC) 2010. 1–8. Google Scholar
Cross Ref
- [10] . 2014. A tutorial on human activity recognition using body-worn inertial sensors. ACM Computing Surveys (CSUR) 46, 3 (2014), 1–33.Google Scholar
Digital Library
- [11] . 2013. The opportunity challenge: A benchmark database for on-body sensor-based activity recognition. Pattern Recognition Letters 34, 15 (2013), 2033–2042.Google Scholar
Digital Library
- [12] . 2019. Multi-agent attentional activity recognition. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI 2019). 1344–1350.Google Scholar
Cross Ref
- [13] . 2020. A semisupervised recurrent convolutional attention model for human activity recognition. IEEE Transactions on Neural Networks and Learning Systems 31, 5 (2020), 1747–1756. Google Scholar
Cross Ref
- [14] . 2020. Deep learning for sensor-based human activity recognition: Overview, challenges and opportunities. arXiv preprint arXiv:2001.07416 (2020).Google Scholar
- [15] . 2012. Sensor-based activity recognition. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 42, 6 (2012), 790–808.Google Scholar
Digital Library
- [16] . 2021. Part-wise spatio-temporal attention driven CNN-based 3D human action recognition. ACM Transactions on Multimidia Computing Communications and Applications 17, 3 (2021), 1–24.Google Scholar
Digital Library
- [17] . 2016. Long-term recurrent convolutional networks for visual recognition and description.IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 4 (2016), 677–691.Google Scholar
Digital Library
- [18] . 2021. A survey on deep learning for human activity recognition. ACM Computing Surveys (CSUR) 54, 8 (2021), 1–34.Google Scholar
Digital Library
- [19] . 2017. Ensembles of deep LSTM learners for activity recognition using wearables. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 2 (2017), 1–28.Google Scholar
Digital Library
- [20] . 2019. On the power of curriculum learning in training deep networks. In International Conference on Machine Learning. PMLR, 2535–2544.Google Scholar
- [21] . 2016. Recurrent attention models for depth-based person identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1229–1238.Google Scholar
Cross Ref
- [22] . 2020. Knowledge-driven egocentric multimodal activity recognition. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 16, 4 (2020), 1–133.Google Scholar
Digital Library
- [23] . 2019. Label propagation for deep semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5070–5079.Google Scholar
Cross Ref
- [24] . 2021. Continual learning in sensor-based human activity recognition: An empirical benchmark analysis. Information Sciences 575 (2021), 1–21.Google Scholar
Digital Library
- [25] . 2017. Activity recognition in beach volleyball using a deep convolutional neural network. Data Mining and Knowledge Discovery 31, 6 (2017), 1678–1705.Google Scholar
Digital Library
- [26] . 2013. A survey on human activity recognition using wearable sensors. IEEE Communications Surveys Tutorials 15, 3 (2013), 1192–1209. Google Scholar
Cross Ref
- [27] . 2021. Similarity embedding networks for robust human activity recognition. ACM Transactions on Knowledge Discovery from Data (TKDD) 15, 6 (2021), 1–17.Google Scholar
Digital Library
- [28] . 2016. Semantic object parsing with graph LSTM. In European Conference on Computer Vision. Springer, 125–143.Google Scholar
Cross Ref
- [29] . 2021. Deep learning for community detection: Progress, challenges and opportunities. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence. 4981–4987.Google Scholar
- [30] . 2020. A benchmark dataset and comparison study for multi-modal human action analytics. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 16, 2 (2020), 1–24.Google Scholar
Digital Library
- [31] . 2016. Mining intricate temporal rules for recognizing complex activities of daily living under uncertainty. Pattern Recognition 60 (2016), 1015–1028. Google Scholar
Digital Library
- [32] . 2020. GlobalFusion: A global attentional deep learning framework for multisensor information fusion. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 1 (2020), 1–27.Google Scholar
Digital Library
- [33] . 2020. Self-paced multi-view co-training.Journal of Machine Learning Research 21, 57 (2020), 1–38.Google Scholar
- [34] . 2019. AttnSense: Multi-level attention mechanism for multimodal human activity recognition. In IJCAI. 3109–3115.Google Scholar
- [35] . 2021. A comprehensive survey on graph anomaly detection with deep learning. IEEE Transactions on Knowledge and Data Engineering (2021).Google Scholar
Cross Ref
- [36] . 2020. Human activity recognition from multiple sensors data using multi-fusion representations and CNNs. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 16, 2 (2020), 1–19.Google Scholar
Digital Library
- [37] . 2018. Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: State of the art and research challenges. Expert Systems with Applications 105 (2018), 233–261.Google Scholar
Cross Ref
- [38] . 2016. Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition. Sensors 16, 1 (2016), 115.Google Scholar
Cross Ref
- [39] . 2021. Weakly-supervised sensor-based activity segmentation and recognition via learning from distributions. Artificial Intelligence 292 (2021), 103429.Google Scholar
Cross Ref
- [40] . 2021. Latent independent excitation for generalizable sensor-based cross-person activity recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 11921–11929.Google Scholar
Cross Ref
- [41] . 2012. Introducing a new benchmarked dataset for activity monitoring. In 2012 16th International Symposium on Wearable Computers. IEEE, 108–109.Google Scholar
Digital Library
- [42] . 2021. Discrete graph structure learning for forecasting multiple time series. In Proceedings of the International Conference on Learning Representations (ICLR 2021). 1–13.Google Scholar
- [43] . 2018. Transductive semi-supervised deep learning using min-max features. In Proceedings of the European Conference on Computer Vision (ECCV). 299–315.Google Scholar
Digital Library
- [44] . 2019. Spatiotemporal co-attention recurrent neural networks for human-skeleton motion prediction. arXiv preprint arXiv:1909.13245 (2019).Google Scholar
- [45] . 2009. Multi-graph based semi-supervised learning for activity recognition. In 2009 International Symposium on Wearable Computers. 85–92. Google Scholar
Digital Library
- [46] . 2022. A comprehensive survey on community detection with deep learning. IEEE Transactions on Neural Networks and Learning Systems (2022).Google Scholar
Cross Ref
- [47] . 2022. Adversarial deep feature extraction network for user independent human activity recognition. In 2022 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 217–226.Google Scholar
Cross Ref
- [48] . 2019. Coherence constrained graph LSTM for group activity recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (2019).Google Scholar
Digital Library
- [49] . 2018. Graph attention networks. In Proceedings of The International Conference on Learning Representations (ICLR 2018). 1–12.Google Scholar
- [50] . 2020. Temporal relational modeling with self-supervision for action segmentation. arXiv preprint arXiv:2012.07508 (2020).Google Scholar
- [51] . 2017. Modeling temporal dynamics and spatial configurations of actions using two-stream recurrent neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 499–508.Google Scholar
Cross Ref
- [52] . 2019. Deep learning for sensor-based activity recognition: A survey. Pattern Recognition Letters 119 (2019), 3–11.Google Scholar
Digital Library
- [53] . 2020. Symbiotic attention with privileged information for egocentric action recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 12249–12256.Google Scholar
Cross Ref
- [54] . 2017. Unsupervised Learning in Human Activity Recognition: A First Foray into Clustering Data Gathered from Wearable Sensors. Ph.D. Dissertation. Ph.D. Thesis, Radboud University, Nijmegen, The Netherlands.Google Scholar
- [55] . 2020. DeepMV: Multi-view deep learning for device-free human activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 1 (2020), 1–26.Google Scholar
Digital Library
- [56] . 2015. Deep convolutional neural networks on multichannel time series for human activity recognition. In Twenty-fourth International Joint Conference on Artificial Intelligence. 3995–4001.Google Scholar
- [57] . 2017. DeepSense: A unified deep learning framework for time-series mobile sensing data processing. In Proceedings of the 26th International Conference on World Wide Web. 351–360.Google Scholar
Digital Library
- [58] . 2017. Semi-supervised convolutional neural networks for human activity recognition. In 2017 IEEE International Conference on Big Data (Big Data). 522–529. Google Scholar
Cross Ref
- [59] . 2012. Motion primitive-based human activity recognition using a bag-of-features approach. In Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium. Association for Computing Machinery, New York, NY, USA, 631–640.Google Scholar
Digital Library
- [60] . 2018. Multi-modality sensor data classification with selective attention. In Proceedings of the 27th International Joint Conference on Artificial Intelligence. 3111–3117.Google Scholar
Digital Library
- [61] . 2020. Facial depression recognition by deep joint label distribution and metric learning. IEEE Transactions on Affective Computing (2020).Google Scholar
Index Terms
Sensor-based Human Activity Recognition Using Graph LSTM and Multi-task Classification Model
Recommendations
Human activity recognition by combining external features with accelerometer sensor data using deep learning network model
AbstractVarious Human Activities are classified through time-series data generated by the sensors of wearable devices. Many real-time scenarios such as Healthcare Surveillance, Smart Cities and Intelligent surveillance etc. are based upon Human Activity ...
Employing a deep convolutional neural network for human activity recognition based on binary ambient sensor data
PETRA '20: Proceedings of the 13th ACM International Conference on PErvasive Technologies Related to Assistive EnvironmentsDue to rising cost of social care, the number of older adults who prefer to live independently in their own home has increased. The independent lifestyle cannot be achieved if the elderly user suffers from mild cognitive impairment unless a suitable ...
A survey on wearable sensor modality centred human activity recognition in health care
Highlights- Discuss limitations and strengths of different sensor modalities.
- Focus on the ...
AbstractIncreased life expectancy coupled with declining birth rates is leading to an aging population structure. Aging-caused changes, such as physical or cognitive decline, could affect people's quality of life, result in injuries, mental ...






Comments