Abstract
In chronic pain rehabilitation, physiotherapists adapt physical activity to patients’ performance based on their expression of protective behavior, gradually exposing them to feared but harmless and essential everyday activities. As rehabilitation moves outside the clinic, technology should automatically detect such behavior to provide similar support. Previous works have shown the feasibility of automatic protective behavior detection (PBD) within a specific activity. In this article, we investigate the use of deep learning for PBD across activity types, using wearable motion capture and surface electromyography data collected from healthy participants and people with chronic pain. We approach the problem by continuously detecting protective behavior within an activity rather than estimating its overall presence. The best performance reaches mean F1 score of 0.82 with leave-one-subject-out cross validation. When protective behavior is modeled per activity type, performance achieves a mean F1 score of 0.77 for bend-down, 0.81 for one-leg-stand, 0.72 for sit-to-stand, 0.83 for stand-to-sit, and 0.67 for reach-forward. This performance reaches excellent level of agreement with the average experts’ rating performance suggesting potential for personalized chronic pain management at home. We analyze various parameters characterizing our approach to understand how the results could generalize to other PBD datasets and different levels of ground truth granularity.
- B. De Gelder. 2009. Why bodies? Twelve reasons for including bodily expressions in affective neuroscience. Philosophical Transactions of the Royal Society B: Biological Sciences 364, 1535 (2009), 3475–3484.Google Scholar
Cross Ref
- Andreas Bulling, Ulf Blanke, and Bernt Schiele. 2014. A tutorial on human activity recognition using body-worn inertial sensors. ACM Computing Surveys 46, 3 (2014), 1–33.Google Scholar
Digital Library
- D. K. Ahern, Michael J. Follick, James R. Council, Nancy Laser-Wolston, and Henry Litchman. 1988. Comparison of lumbar paravertebral EMG patterns in chronic low back pain patients and non-patient controls. Pain 34, 2 (1988), 153–160.Google Scholar
Cross Ref
- Fahd Albinali, Matthew S. Goodwin, and Stephen S. Intille. 2009. Recognizing stereotypical motor movements in the laboratory and classroom: A case study with children on the autism spectrum. In Proceedings of the 11th International Conference on Ubiquitous Computing (Ubicomp’09).Google Scholar
- M. S. H. Aung, Nadia Bianchi-Berthouze, Paul Watson, and AC de C. Williams. 2014. Automatic recognition of fear-avoidance behavior in chronic pain physical rehabilitation. In Proceedings of the 8th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth’14).Google Scholar
Digital Library
- M. S. H. Aung, Sebastian Kaltwang, Bernardino Romera-Paredes, Brais Martinez, Aneesha Singh, Matteo Cella, Michel Valstar, Hongying Meng, Andrew Kemp, and Moshen Shafizadeh et al., 2016. The automatic detection of chronic pain-related expression: Requirements, challenges and the multimodal EmoPain dataset. IEEE Transactions on Affective Computing 7, 4 (2016), 435–451.Google Scholar
Digital Library
- H. Breivik, Beverly Collett, Vittorio Ventafridda, Rob Cohen, and Derek Gallacher. 2006. Survey of chronic pain in Europe: Prevalence, impact on daily life, and treatment. European Journal of Pain 10, 4 (2006), 287–333.Google Scholar
Cross Ref
- Marc Bachlin, Daniel Roggen, Gerhard Troster, Meir Plotnik, Noit Inbar, Inbal Meidan, Talia Herman, Marina Brozgol, Eliya Shaviv, Nir Giladi, and others. 2009. Potentials of enhanced context awareness in wearable assistants for Parkinson’s disease patients with the freezing of gait syndrome. 13th International Symposium on Wearable Computers (ISWC) (2009).Google Scholar
- K. Cook, Francis Keefe, Mark P. Jensen, Toni S. Roddey, Leigh F. Callahan, Dennis Revicki, Alyssa M. Bamer, Jiseon Kim, Hyewon Chung, and Salem Rana et al. 2013. Development and validation of a new self-report measure of pain behaviors. Pain 154, 12 (2013), 2867–2876.Google Scholar
Cross Ref
- Ricardo Chavarriaga, Hesam Sagha, Alberto Calatroni, Sundara Tejaswi Digumarti, Gerhard Tröster, José del R. Millán, and Daniel Roggen. 2013. The opportunity challenge: A benchmark database for on-body sensor-based activity recognition. Pattern Recognition Letters 34, 15 (2013), 2033–2042.Google Scholar
Digital Library
- Chongyang Wang, Temitayo A. Olugbade, Akhil Mathur, Amanda C. De C. Williams, Nicholas D. Lane, and Nadia Bianchi-Berthouze. 2019. Recurrent network based automatic detection of chronic pain protective behavior using MoCap and sEMG data. In Proceedings of the 23rd ACM International Symposium on Wearable Computers (ISWC’19). 225–230.Google Scholar
Digital Library
- Cui Yin, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. 2019. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’19). 9268–9277.Google Scholar
- J. P. Dickey, Michael R. Pierrynowski, Drew A. Bednar, and Simon X. Yang. 2013. Relationship between pain and vertebral motion in chronic low-back pain subjects. Clinical Biomechanics 7, 4 (2013), 412–418.Google Scholar
- Daniel Roggen, Daniel Roggen, Georg Ogris, Paul Lukowicz, and Gerhard Tröster. 2008. Wearable activity tracking in car manufacturing. IEEE Pervasive Computing 7, 2 (2008), 42–50.Google Scholar
- W. E. Fordyce, David Lansky, Donald A. Calsyn, John L. Shelton, Walter C. Stolov, and Daniel L. Rock. 1984. Pain measurement and pain behavior. Pain 18, 1 (1984), 53–69.Google Scholar
- H. Grip, Fredrik Ohberg, Urban Wiklund, Ylva Sterner, J. Stefan Karlsson, Björn Gerdle. 2013. Classification of neck movement patterns related to whiplash-associated disorders using neural networks. IEEE Transactions on Information Technology in Biomedicine 7, 4 (2013), 412–418.Google Scholar
- Greff Klaus, Rupesh K. Srivastava, Jan Koutník, Bas R. Steunebrink, and Jürgen Schmidhuber. 2017. LSTM: A search space odyssey. IEEE Transactions on Neural Networks and Learning Systems 28, 10 (2017), 2222–2232.Google Scholar
- Matthew S. Goodwin et al. 2014. Moving towards a real-time system for automatically recognizing stereotypical motor movements in individuals on the autism spectrum using wireless accelerometry. In Proceedings of the 16th International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’14).Google Scholar
- Guan Yu and Thomas Plötz. 2017. Ensembles of deep LSTM learners for activity recognition using wearables. In Proceedings of the ACM on Interactive, Mobile, Wearable, and Ubiquitous Technologies (IMWUT’17). Article 11.Google Scholar
- Nils Y. Hammerla, Shane Halloran, and Thomas Plötz. 2016. Deep, convolutional, and recurrent models for human activity recognition using wearables. arXiv:1604.08880Google Scholar
- Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9, 8 (1997), 1735–1780.Google Scholar
Digital Library
- Sojeong Ha and Seungjin Choi. 2016. Convolutional neural networks for human activity recognition using multiple accelerometer and gyroscope sensors. In Proceedings of the International Joint Conference on Neural Networks (IJCNN’16). 381–388.Google Scholar
Cross Ref
- T. Huynh, Ulf Blanke and Bernt Schiele. 2007. Scalable recognition of daily activities with wearable sensors. In Location- and Context-Awareness. Lecture Notes in Computer Science, Vol. 4718. Springer, 50–67.Google Scholar
- Andrea Kleinsmith et al. 2011. Automatic recognition of non-acted affective postures. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 41, 4 (2011), 1027–1038.Google Scholar
- Andrea Kleinsmith and Nadia Bianchi-Berthouze. 2013. Affective body expression perception and recognition: A survey. IEEE Transactions on Affective Computing 4, 1 (2013), 15–33.Google Scholar
Digital Library
- The Pain Consortium. 2016. The pain consortium. UK Pain Messages: Pain News (2016), 21–22.Google Scholar
- Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980Google Scholar
- M. Karg, Nadia Bianchi-Berthouze, and Anthony Steed. 2013. Body movements for affective expression: A survey of automatic recognition and generation. IEEE Transactions on Affective Computing 4, 4 (2013), 341–359.Google Scholar
- Simon Kohl, Bernardino Romera-Paredes, Clemens Meyer, Jeffrey De Fauw, Joseph R. Ledsam, Klaus H. Maier-Hein, S. M. Eslami, Danilo Rezende, Jimenez, and Olaf Ronneberger. 2018. A probabilistic U-Net for segmentation of ambiguous images. Advances in Neural Information Processing Systems (NeurIPS’18), Vol. 31.6965–6975.Google Scholar
- Patrick Lucey, Jeffrey F. Cohn, Kenneth M. Prkachin, and Patricia E. Solomon, Iain Matthews. 2011. Painful data: The UNBC-McMaster shoulder pain expression archive database. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition and Workshops (FG’11).Google Scholar
- F. J. O. Morales and Daniel Roggen. 2008. Deep convolutional feature transfer across mobile activity recognition domains, sensor modalities and locations. In Proceedings of the 12th International Symposium on Wearable Computers (ISWC’08).Google Scholar
- K. O. McGraw and Seok P. Wong. 1996. Forming inferences about some intraclass correlation coefficients. Psychological Methods 1, 1 (1996), 30–46.Google Scholar
Cross Ref
- Riccardo Miotto, Fei Wang, Shuang Wang, Xiaoqian Jiang, and Joel T. Dudley. 2017. Deep learning for healthcare: Review, opportunities and challenges. Briefings in Bioinformatics 19, 6 (2017), 1236–1246.Google Scholar
Cross Ref
- T. A. Olugbade, Min S. H. Aung, Nadia Bianchi-Berthouze, Nicolai Marquardt, and Amanda C. De C. Williams. 2014. Bi-modal detection of painful reaching for chronic pain rehabilitation systems. In Proceedings of the 16th International Conference on Multimodal Interaction (ICMI’14). 455–458.Google Scholar
- T. A. Olugbade, Nadia Bianchi-Berthouze, Nicolai Marquardt, and Amanda C. De C. Williams. 2015. Pain level recognition using kinematics and muscle activity for physical rehabilitation in chronic pain. In Proceedings of the 6th Conference on Affective Computing and Intelligent Interaction (ACII’15). 243–249.Google Scholar
- T. A. Olugbade, Nadia Bianchi-Berthouze, Nicolai Marquardt, and Amanda C. De C. Williams. 2018. Human observer and automatic assessment of movement related self-efficacy in chronic pain: From movement to functional activity. IEEE Transactions on Affective Computing 11, 2 (2018), 214–229.Google Scholar
Cross Ref
- T. A. Olugbade, Aneesha Singh, Nadia Bianchi-Berthouze, Nicolai Marquardt, Min SH Aung, and Amanda C. De C. Williams. 2019. How can affect be detected and represented in technological support for physical rehabilitation?ACM Transactions on Computer-Human Interaction 26, 1 (2019), Article 1.Google Scholar
- Enrica Papi, Athina Belsi, and Alison H. McGregor. 2015. A knee monitoring device and the preferences of patients living with osteoarthritis: A qualitative study. BMJ Open 5, 9 (2015), 3007980.Google Scholar
- Enrica Papi, Ged M. Murtagh, and Alison H. McGregor. 2016. Wearable technologies in osteoarthritis: A qualitative study of clinicians’ preferences. BMJ Open 6 (2016), 2044–6055.Google Scholar
- Kenneth M. Prkachin et al. 2008. The structure, reliability and validity of pain expression: Evidence from patients with shoulder pain. Pain 139, 2 (2008), 267–274.Google Scholar
Cross Ref
- N. M. Rad and Cesare Furlanello. 2016. Applying deep learning to stereotypical motor movement detection in autism spectrum disorders. In Proceedings of the 16th International Conference on Data Mining Workshops (ICDMW’16).Google Scholar
- Attila Reiss and Didier Stricker. 2012. Introducing a new benchmarked dataset for activity monitoring. In Proceedings of the 16th International Symposium on Wearable Computers (ISWC’12).Google Scholar
- J. Rivas, Maria del Carmen Lara, Luis Castrejon, Jorge Hernandez-Franco, Felipe Orihuela-Espina, Lorena Palafox, Amanda Williams, Nadia Berthouze, and Enrique Sucar. 2021. Multi-label and multimodal classifier for affective states recognition in virtual rehabilitation. IEEE Transactions on Affective Computing. Early access, February 1, 2021.Google Scholar
- A. Singh, Annina Klapper, Jinni Jia, Antonio Fidalgo, Ana Tajadura-Jiménez, Natalie Kanakam, NadiaBianchi-Berthouze, and Amanda Williams. 2014. Motivating people with chronic pain to do physical activity: Opportunities for technology design. In Proceedings of the International Conference on Human Factors in Computing Systems (CHI’14). 2803–2812.Google Scholar
- A. Singh, Stefano Piana, Davide Pollarolo, Gualtiero Volpe, Giovanna Varni, Ana Tajadura-Jimenez, Amanda CdeC Williams, Antonio Camurri, and Nadia Bianchi-Berthouze. 2016. Go-with-the-flow: Tracking, analysis and sonification of movement and breathing to build confidence in activity despite chronic pain. Human–Computer Interaction 31, 3-4 (2016), 1–49.Google Scholar
Digital Library
- A. Singh, Nadia Bianchi-Berthouze, and Amanda C. de C. Williams. 2017. Supporting everyday function in chronic pain using wearable technology. In Proceedings of the International Conference on Human Factors in Computing Systems (CHI’17). 3903–3915.Google Scholar
- M. J. L. Sullivan, Pascal Thibault, André Savard, Richard Catchlove, John Kozey, and William D. Stanish. 2006. The influence of communication goals and physical demands on different dimensions of pain behavior. Pain 125, 3 (2006), 270–277.Google Scholar
Cross Ref
- T. T. Um, Franz M. J. Pfister, Daniel Pichler, Satoshi Endo, Muriel Lang, Sandra Hirche, Urban Fietzek, and Dana Kulić. 2017. Data augmentation of wearable sensor data for Parkinson’s disease monitoring using convolutional neural networks. In Proceedings of the 19th ACM International Conference on Multimodal Interaction (ICMI’17). 216–220.Google Scholar
Digital Library
- J. W. S. Vlaeyen, Stephen Morley, and Geert Crombez. 2016. The experimental analysis of the interruptive, interfering, and identity-distorting effects of chronic pain. Behaviour Research and Therapy 86 (2016), 23–34.Google Scholar
Cross Ref
- Limin Wang, Yu Qiao, and Xiaoou Tang. 2015. Action recognition with trajectory-pooled deep-convolutional descriptors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). 4305–4314.Google Scholar
- P. J. Watson, C. Kerry Booker, and Chris J. Main. 1997. Evidence for the role of psychological factors in abnormal paraspinal activity in patients with chronic low back pain. Journal of Musculoskeletal Pain 5, 4 (1997), 41–56.Google Scholar
Cross Ref
- D. V. Cicchetti. 1994. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assess 6, 4 (1994), 284–290.Google Scholar
Cross Ref
- I. Tracey and M. C. Bushnell. 2009. How neuroimaging studies have challenged us to rethink: Is chronic pain a disease? Journal of Pain 10, 11 (2009), 1113–1120.Google Scholar
Cross Ref
- F. J. Keefe and A. R. Block 1982. Development of an observation method for assessing pain behavior in chronic low back pain patients. Behavior Therapy 13, 14 (1982), 363–375.Google Scholar
Cross Ref
- K. A. Hallgren. 2012. Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology 8, 1 (2012), 23–24.Google Scholar
Cross Ref
- R. N. Jamison. 2016. Are we really ready for telehealth cognitive behavioral therapy for pain?Pain 158, 4 (2016), 538–540.Google Scholar
- J. W. S. Vlaeyen and S. J. Linton. 2000. Fear-avoidance and its consequences in chronic musculoskeletal pain: A state of the artPain 85, 3 (2000), 317–332.Google Scholar
Index Terms
Chronic Pain Protective Behavior Detection with Deep Learning
Recommendations
Recurrent network based automatic detection of chronic pain protective behavior using MoCap and sEMG data
ISWC '19: Proceedings of the 2019 ACM International Symposium on Wearable ComputersIn chronic pain physical rehabilitation, physiotherapists adapt exercise sessions according to the movement behavior of patients. As rehabilitation moves beyond clinical sessions, technology is needed to similarly assess movement behaviors and provide ...
Leveraging Activity Recognition to Enable Protective Behavior Detection in Continuous Data
Protective behavior exhibited by people with chronic pain (CP) during physical activities is very informative to understanding their physical and emotional states. Existing automatic protective behavior detection (PBD) methods rely on pre-segmentation ...
Supporting Everyday Function in Chronic Pain Using Wearable Technology
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing SystemsWhile most rehabilitation technologies target situated exercise sessions and associated performance metrics, physiotherapists recommend physical activities that are integrated with everyday functioning. We conducted a 1-2 week home study to explore how ...






Comments