Abstract
Falling is ranked highly among the threats in elderly healthcare, which promotes the development of automatic fall detection systems with extensive concern. With the fast development of the Internet of Things (IoT) and Artificial Intelligence (AI), camera vision-based solutions have drawn much attention for single-frame prediction and video understanding on fall detection in the elderly by using Convolutional Neural Network (CNN) and 3D-CNN, respectively. However, these methods hardly supervise the intermediate features with good accurate and efficient performance on edge devices, which makes the system difficult to be applied in practice. This work introduces a fast and lightweight video fall detection network based on a spatio-temporal joint-point model to overcome these hurdles. Instead of detecting fall motion by the traditional CNNs, we propose a Long Short-Term Memory (LSTM) model based on time-series joint-point features extracted from a pose extractor. We also introduce the increasingly mature RGB-D camera and propose 3D pose estimation network to further improve the accuracy of the system. We propose to apply tensor train decomposition on the model to reduce storage and computational consumption so the deployment on edge devices can to realized. Experiments are conducted to verify the proposed framework. For fall detection task, the proposed video fall detection framework achieves a high sensitivity of 98.46% on Multiple Cameras Fall, 100% on UR Fall, and 98.01% on NTU RGB-D 120. For pose estimation task, our 2D model attains 73.3 mAP in the COCO keypoint challenge, which outperforms the OpenPose by 8%. Our 3D model attains 78.6% mAP on NTU RGB-D dataset with 3.6× faster speed than OpenPose.
- [1] . 2020. Privacy preserving human fall detection using video data. In Proceedings of the Machine Learning for Health Workshop. 39–51.Google Scholar
- [2] . 2010. Multiple Cameras Fall Dataset. Technical report. DIRO-Université de Montréal, Tech. Rep. 1350.Google Scholar
- [3] . 2017. A novel approach for fall detection in home environment. In Proceedings of the IEEE 6th Global Conference on Consumer Electronics (GCCE). IEEE, 1–5.Google Scholar
Cross Ref
- [4] . 2014. Fall detection based on the gravity vector using a wide-angle camera. Exp. Syst. Applic. 41, 17 (2014), 7980–7986.Google Scholar
Digital Library
- [5] . 2020. Vision-based fall detection with multi-task hourglass convolutional auto-encoder. IEEE Access 8 (2020), 44493–44502.Google Scholar
Cross Ref
- [6] . 2018. OpenPose: Realtime multi-person 2D pose estimation using part affinity fields. arXiv preprint arXiv:1812.08008 (2018).Google Scholar
- [7] . 2010. A hybrid human fall detection scheme. In Proceedings of the IEEE International Conference on Image Processing. IEEE, 3485–3488.Google Scholar
Cross Ref
- [8] . 2020. An anomaly comprehension neural network for surveillance videos on terminal devices. In Proceedings of the Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 1396–1401.Google Scholar
Cross Ref
- [9] . 2020. DEEPEYE: A deeply tensor-compressed neural network for video comprehension on terminal devices. ACM Trans. Embed. Comput. Syst. 19, 3 (2020), 1–25.Google Scholar
Digital Library
- [10] . 2021. S3-Net: A fast scene understanding network by single-shot segmentation for autonomous driving. ACM Trans. Intell. Syst. Technol. 12, 5 (2021), 1–19.Google Scholar
Digital Library
- [11] . 2015. P-CNN: Pose-based CNN features for action recognition. In Proceedings of the IEEE International Conference on Computer Vision. 3218–3226.Google Scholar
Digital Library
- [12] . 2017. Radar and RGB-depth sensors for fall detection: A review. IEEE Sensors J. 17, 12 (2017), 3585–3604.
DOI: Google ScholarCross Ref
- [13] . 2020. Spatio-temporal fall event detection in complex scenes using attention guided LSTM. Pattern Recog. Lett. 130 (2020), 242–249.Google Scholar
Digital Library
- [14] . 2008. Intelligent video surveillance for monitoring fall detection of elderly in home environments. In Proceedings of the 11th International Conference on Computer and Information Technology. IEEE, 219–224.Google Scholar
Cross Ref
- [15] . 2021. Comprehensive review of vision-based fall detection systems. Sensors 21, 3 (2021), 947.Google Scholar
- [16] . 2017. Vision-based fall detection system for improving safety of elderly people. IEEE Instrum. Measur. Mag. 20, 6 (2017), 49–55.Google Scholar
Cross Ref
- [17] . 2017. Mask R-CNN. In Proceedings of the IEEE International Conference on Computer Vision. 2961–2969.Google Scholar
Cross Ref
- [18] . 1927. The expression of a tensor or a polyadic as a sum of products. J. Math. Phys. 6, 1–4 (1927), 164–189.Google Scholar
Cross Ref
- [19] . 2013. Challenges, issues and trends in fall detection systems. Biomed. Eng. Onl. 12, 1 (2013), 66.Google Scholar
Cross Ref
- [20] . 2020. Detection and multi-class classification of falling in elderly people by deep belief network algorithms. J. Amb. Intell. Human. Comput. (2020), 1–21.Google Scholar
- [21] . 2018. MultiPoseNet: Fast multi-person pose estimation using pose residual network. In Proceedings of the European Conference on Computer Vision (ECCV). 417–433.Google Scholar
Digital Library
- [22] . 2014. Human fall detection on embedded platform using depth maps and wireless accelerometer. Comput. Meth. Prog. Biomed. 117, 3 (2014), 489–501.Google Scholar
Digital Library
- [23] . 2009. Accurate, fast fall detection using gyroscopes and accelerometer-derived posture information. In Proceedings of the 6th International Workshop on Wearable and Implantable Body Sensor Networks. IEEE, 138–143.Google Scholar
Digital Library
- [24] . 2019. TSM: Temporal shift module for efficient video understanding. In Proceedings of the IEEE International Conference on Computer Vision. 7083–7093.Google Scholar
Cross Ref
- [25] . 2014. Microsoft COCO: Common objects in context. In Proceedings of the European Conference on Computer Vision. Springer, 740–755.Google Scholar
Cross Ref
- [26] . 2019. NTU RGB+D 120: A large-scale benchmark for 3D human activity understanding. IEEE Trans. Pattern Anal. Mach. Intell. (2019).
DOI: Google ScholarDigital Library
- [27] . 2018. Deep learning for fall detection: Three-dimensional CNN combined with LSTM on video kinematic data. IEEE J. Biomed. Health Inform. 23, 1 (2018), 314–323.Google Scholar
Cross Ref
- [28] . 2017. A simple yet effective baseline for 3D human pose estimation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).Google Scholar
Cross Ref
- [29] . 2013. A survey on fall detection: Principles and approaches. Neurocomputing 100 (2013), 144–152.Google Scholar
Digital Library
- [30] . 2018. Convolutional neural networks and long short-term memory for skeleton-based human activity and hand gesture recognition. Pattern Recog. 76 (2018), 80–94.Google Scholar
Digital Library
- [31] . 2017. Vision-based fall detection with convolutional neural networks. Wirel. Commun. Mob. Comput. (2017).Google Scholar
- [32] . 2019. Exploring RGB+ depth fusion for real-time object detection. Sensors 19, 4 (2019), 866.Google Scholar
Cross Ref
- [33] . 2008. WHO Global Report on Falls Prevention in Older Age. World Health Organization.Google Scholar
- [34] . 2011. Tensor-train decomposition. SIAM J. Sci. Comput. 33, 5 (2011), 2295–2317.
DOI: Google ScholarCross Ref
- [35] . 2018. PersonLab: Person pose estimation and instance segmentation with a bottom-up, part-based, geometric embedding model. In Proceedings of the European Conference on Computer Vision (ECCV). 269–286.Google Scholar
Digital Library
- [36] . 2017. An end-to-end spatio-temporal attention model for human action recognition from skeleton data. In Proceedings of the 31st AAAI Conference on Artificial Intelligence.Google Scholar
Cross Ref
- [37] . 1966. Some mathematical notes on three-mode factor analysis. Psychometrika 31 (1966), 279–311.
DOI: Google ScholarCross Ref
- [38] . 2011. Video based automatic fall detection in indoor environment. In Proceedings of the International Conference on Recent Trends in Information Technology (ICRTIT). IEEE, 1016–1020.Google Scholar
Cross Ref
- [39] . 2016. Automatic fall detection of human in video using combination of features. In Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, 1228–1233.Google Scholar
- [40] . 2016. Human fall detection in surveillance video based on PCANet. Multim. Tools Applic. 75, 19 (2016), 11603–11613.Google Scholar
Digital Library
- [41] . 2021. An edge-device based fast fall detection using spatio-temporal optical flow model. In Proceedings of the 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, 5067–5071.Google Scholar
Cross Ref
- [42] . 2016. Fall detection using supervised machine learning algorithms: A comparative study. In Proceedings of the 8th International Conference on Modelling, Identification and Control (ICMIC). IEEE, 665–670.Google Scholar
Cross Ref
- [43] . 2021. Fast video facial expression recognition by a deeply tensor-compressed LSTM neural network for mobile devices. ACM Trans. Internet Things 2, 4 (2021), 1–26.Google Scholar
Digital Library
Index Terms
A Fall Detection Network by 2D/3D Spatio-temporal Joint Models with Tensor Compression on Edge
Recommendations
Pre-Impact Fall Detection Using 3D Convolutional Neural Network
2019 IEEE 16th International Conference on Rehabilitation Robotics (ICORR)Early fall detection is an important issue during gait rehabilitation training. This paper proposes an approach for pre-impact fall detection during gait rehabilitation training based on a 3D convolutional neural network (CNN). Firstly, pre-training data ...
Spatio-temporal fall event detection in complex scenes using attention guided LSTM
Highlights- A new fall event dataset in crowded and complex scenes is created.
- A novel fall ...
AbstractFall events are one of the greatest risks for public safety, especially in some complex scenes with large number of people. Nevertheless, there are few researches on fall detection in complex scenes, and even no public datasets. A fall ...
Reliable and secure body fall detection algorithm in a wireless mesh network
BodyNets '13: Proceedings of the 8th International Conference on Body Area NetworksFalls in elderly is one of the most serious causes of severe injury and lack in immediate medical help makes these injuries life threatening. An automatic fall detection system, presented in this research, will help reduce the arrival time of medical ...






Comments