ERICA: enabling real-time mistake detection & corrective feedback for free-weights exercises

ABSTRACT

We present ERICA, a digital personal trainer for users performing free weights exercises, with two key differentiators: (a) First, unlike prior approaches that either require multiple on-body wearables or specialized infrastructural sensing, ERICA uses a single in-ear "earable" device (piggybacking on a form factor routinely used by millions of gym-goers) and a simple inertial sensor mounted on each weight equipment; (b) Second, unlike prior work that focuses primarily on quantifying a workout, ERICA additionally identifies a variety of fine-grained exercising mistakes and delivers real-time, in-situ corrective instructions. To achieve this, we (a) design a robust approach for user-equipment association that can handle multiple (even 15) concurrently exercising users; (b) develop a suite of statistical models to detect several commonplace repetition-level mistakes; and (c) experimentally study the efficacy of multiple in-situ corrective feedback strategies. Via an end-to-end evaluation of ERICA with 33 participants naturally performing 3 dumbbell-based exercises, we show that (a) ERICA identifies over 94% of mistakes during the first 5 repetitions of a set, (b) the resulting feedback is viewed favorably by 78% of users, and (c) the feedback is effective, reducing mistakes by 10+% during subsequent repetitions.

References

1. Oliver Amft, Mathias Stäger, Paul Lukowicz, and Gerhard Tröster. 2005. Analysis of chewing sounds for dietary monitoring. In (UbiComp'05).
2. Joseph L Andreacci, Linda M Lemura, Steven L Cohen, Ethan A Urbansky, Sara A Chelland, and Serge P von Duvillard. 2002. The effects of frequency of encouragement on performance during maximal exercise testing. Journal of sports sciences 20, 4 (2002), 345--352.
3. Louis Atallah, Anatole Wiik, Gareth G Jones, Benny Lo, Justin P Cobb, Andrew Amis, and Guang-Zhong Yang. 2012. Validation of an ear-worn sensor for gait monitoring using a force-plate instrumented treadmill. Gait & posture 35, 4 (2012), 674--676.
4. Shengjie Bi, Tao Wang, Nicole Tobias, Josephine Nordrum, Shang Wang, George Halvorsen, Sougata Sen, Ronald Peterson, Kofi Odame, Kelly Caine, et al. 2018. Auracle: Detecting Eating Episodes with an Ear-mounted Sensor. Proc. of the ACM IMWUT 2, 3 (2018), 92.
5. Sizhen Bian, Vitor Rey, Peter Hevesi, and Paul Lukowicz. [n.d.]. Passive Capacitive based Approach for Full Body Gym Workout Recognition and Counting. In Proc. of PerCom'19.
6. Daniel G. Carey, Leslie A. Schwarz, German J. Pliego, and Robert L. Raymond. 2005. Respiratory Rate is a Valid and Reliable Marker for the Anaerobic Threshold: Implications for Measuring Change in Fitness. Journal of Sports Science & Medicine 4, 4 (Dec. 2005), 482--488.
7. Keng-Hao Chang, Mike Y Chen, and John Canny. 2007. Tracking free-weight exercises. In International Conference on Ubiquitous Computing. Springer, 19--37.
8. C Crema, A Depari, A Flammini, Emiliano Sisinni, T Haslwanter, and S Salzmann. 2017. IMU-based solution for automatic detection and classification of exercises in the fitness scenario. In 2017 IEEE Sensors Applications Symposium (SAS). IEEE, 1--6.
9. Dialog Semiconductor. 2016. SmartBond^TM DA14583 IoT Sensor Development Kit. https://www.dialog-semiconductor.com/sites/default/files/smartbond_da14583_iot_sensor_development_kit_product_brief_november_2016.pdf; Last Accessed: October 2020.
10. Han Ding, Longfei Shangguan, Zheng Yang, Jinsong Han, Zimu Zhou, Panlong Yang, Wei Xi, and Jizhong Zhao. 2015. Femo: A platform for free-weight exercise monitoring with rfids. In Proc. of ACM SenSys'15. ACM, 141--154.
11. Andrea Ferlini, Alessandro Montanari, Mascolo Cecilia, and Robert Harle. 2019. Head Motion Tracking Through in-Ear Wearables. In Proc. of EarComp'19.
12. Biying Fu, Lennart Jarms, Florian Kirchbuchner, and Arjan Kuijper. 2020. ExerTrack---Towards Smart Surfaces to Track Exercises. Technologies 8, 1 (2020), 17.
13. M Gallagher. 1996. Ten most common causes of training injury. Muscle & Fitness (1996).
14. Bruno Gil, Salzitsa Anastasova, and Guang Z Yang. 2019. A Smart Wireless Ear-Worn Device for Cardiovascular and Sweat Parameter Monitoring During Physical Exercise: Design and Performance Results. Sensors 19, 7 (2019), 1616.
15. Xiaonan Guo, Jian Liu, Cong Shi, Hongbo Liu, Yingying Chen, and Mooi Choo Chuah. 2018. Device-free Personalized Fitness Assistant Using WiFi. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 4 (2018), 165.
16. Stuart Hagler, Holly B Jimison, Ruzena Bajcsy, and Misha Pavel. 2014. Quantification of human movement for assessment in automated exercise coaching. In Engineering in Medicine and Biology Society (EMBC), 2014 36th Annual International Conference of the IEEE. IEEE, 5836--5839.
17. Timothy C Havens, Gregory L Alexander, Carmen Abbott, James M Keller, Marjorie Skubic, and Marilyn Rantz. 2009. Contour tracking of human exercises. In Computational Intelligence for Visual Intelligence, 2009. CIVI'09. IEEE Workshop on. IEEE, 22--28.
18. Jefit Inc.[US]. [n.d.]. JEFIT: Working Tracking Platform. https://www.jefit.com/; Last Accessed: June 2020.
19. Zachary Y Kerr, Christy L Collins, and R Dawn Comstock. 2010. Epidemiology of weight training-related injuries presenting to United States emergency departments, 1990 to 2007. The American Journal of Sports Medicine 38, 4 (2010), 765--771.
20. Rushil Khurana, Karan Ahuja, Zac Yu, Jennifer Mankoff, Chris Harrison, and Mayank Goel. 2018. GymCam: Detecting, Recognizing and Tracking Simultaneous Exercises in Unconstrained Scenes. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 4 (2018), 185.
21. Emily Knight, Melanie I Stuckey, Harry Prapavessis, and Robert J Petrella. 2015. Public health guidelines for physical activity: is there an app for that? A review of android and apple app stores. JMIR mHealth and uHealth 3, 2 (2015), e43.
22. Yousef Kowsar, Masud Moshtaghi, Eduardo Velloso, Lars Kulik, and Christopher Leckie. 2016. Detecting unseen anomalies in weight training exercises. In Proc. of ACM OzCHI'16. ACM, 517--526.
23. Harold W Kuhn. 1955. The Hungarian method for the assignment problem. Naval research logistics quarterly 2, 1--2 (1955), 83--97.
24. Pradeep Kumar, Rajkumar Saini, Mahendra Yadava, Partha Pratim Roy, Debi Prosad Dogra, and Raman Balasubramanian. 2017. Virtual trainer with real-time feedback using kinect sensor. In 2017 IEEE Region 10 Symposium (TENSYMP). IEEE, 1--5.
25. Chin Guan Lim, Chin Yi Tsai, and Mike Y Chen. 2020. MuscleSense: Exploring Weight Sensing using Wearable Surface Electromyography (sEMG). In Proceedings of the Fourteenth International Conference on Tangible, Embedded, and Embodied Interaction. 255--263.
26. Stefano Melzi, Luca Pietro Borsani, and Matteo Cesana. 2009. The virtual trainer: Supervising movements through a wearable wireless sensor network. In 2009 6th IEEE Annual Communications Society Conference on Sensor, Mesh and Ad Hoc Communications and Networks Workshops. IEEE, 1--3.
27. Slobodan Milanko and Shubham Jain. 2020. LiftRight: Quantifying strength training performance using a wearable sensor. Smart Health (2020), 100115.
28. Frank Mokaya, Roland Lucas, Hae Young Noh, and Pei Zhang. 2016. Burnout: a wearable system for unobtrusive skeletal muscle fatigue estimation. In Proc. of IPSN'16. IEEE Press, 8.
29. Dan Morris, T Scott Saponas, Andrew Guillory, and Ilya Kelner. 2014. RecoFit: using a wearable sensor to find, recognize, and count repetitive exercises. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 3225--3234.
30. Shahriar Nirjon, Robert F Dickerson, Qiang Li, Philip Asare, John A Stankovic, Dezhi Hong, Ben Zhang, Xiaofan Jiang, Guobin Shen, and Feng Zhao. 2012. Musicalheart: A hearty way of listening to music. In SenSys'12.
31. Mehdi Niroomand and Hamid Reza Foroughi. 2016. A rotary electromagnetic microgenerator for energy harvesting from human motions. Journal of Applied Research and Technology 14, 4 (2016), 259 -- 267.
32. Igor Pernek, Karin Anna Hummel, and Peter Kokol. 2013. Exercise repetition detection for resistance training based on smartphones. Personal and ubiquitous computing 17, 4 (2013), 771--782.
33. Jay Prakash, Zhijian Yang, Yu-Lin Wei, and Romit Roy Choudhury. 2019. STEAR: Robust Step Counting from Earables. In Proc. of EarComp'19.
34. Danish Pruthi, Ayush Jain, Krishna Murthy Jatavallabhula, Ruppesh Nalwaya, and Puneet Teja. 2015. Maxxyt: An autonomous wearable device for real-time tracking of a wide range of exercises. In 2015 17th UKSim-AMSS International Conference on Modelling and Simulation (UKSim). IEEE, 137--141.
35. Md Fazlay Rabbi, Taiwoo Park, Biyi Fang, Mi Zhang, and Youngki Lee. 2018. When Virtual Reality Meets IoT in the Gym: Enabling Immersive and Interactive Machine Exercise. ACM.
36. Meera Radhakrishnan and Archan Misra. 2019. Can Earables Support Effective User Engagement during Weight-Based Gym Exercises?. In Proceedings of the 1st International Workshop on Earable Computing. 42--47.
37. Meera Radhakrishnan, Archan Misra, and Rajesh K. Balan. 2020. W8Scope: Fine-grained, Practical Monitoring of Weight Stack-based Exercises. In Proc. of IEEE PerCom 2020. IEEE.
38. Tobias Röddiger, Daniel Wolffram, David Laubenstein, Matthias Budde, and Michael Beigl. 2019. Towards Respiration Rate Monitoring Using an In-Ear Headphone Inertial Measurement Unit. In Proc. of EarComp'19.
39. Christian Seeger, Kristof Van Laerhoven, and Alejandro Buchmann. 2015. MyHealthAssistant: An event-driven middleware for multiple medical applications on a smartphone-mediated body sensor network. IEEE journal of biomedical and health informatics 19, 2 (2015), 752--760.
40. Chenguang Shen, Bo-Jhang Ho, and Mani Srivastava. 2018. Milift: Efficient smartwatch-based workout tracking using automatic segmentation. IEEE Transactions on Mobile Computing 17, 7 (2018), 1609--1622.
41. Gabriele Spina, Guannan Huang, Anouk Vaes, Martijn Spruit, and Oliver Amft. 2013. COPDTrainer: a smartphone-based motion rehabilitation training system with real-time acoustic feedback. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing. 597--606.
42. Mathias Sundholm, Jingyuan Cheng, Bo Zhou, Akash Sethi, and Paul Lukowicz. 2014. Smart-mat: Recognizing and counting gym exercises with low-cost resistive pressure sensing matrix. In Proc. of ACM UbiComp'14. ACM, 373--382.
43. Gábor J Székely, Maria L Rizzo, Nail K Bakirov, et al. 2007. Measuring and testing dependence by correlation of distances. The annals of statistics 35, 6 (2007), 2769--2794.
44. Kazuhiro Taniguchi and Atsushi Nishikawa. 2018. Earable POCER: Development of a Point-of-Care Ear Sensor for Respiratory Rate Measurement. Sensors 18, 9 (2018), 3020.
45. Techsmith Corporation. [n.d.]. Coach's Eye Sports Video Analysis App. https://www.coachseye.com/; Last Accessed: June 2020.
46. Vu H. Tran, Archan Misra, Jie Xiong, and Rajesh Krishna Balan. 2019. WiWear: Wearable Sensing via Directional WiFi Energy Harvesting. In 2019 IEEE International Conference on Pervasive Computing and Communications, PerCom, Kyoto, Japan, March 11--15, 2019. IEEE, 1--10.
47. Eduardo Velloso, Andreas Bulling, Hans Gellersen, Wallace Ugulino, and Hugo Fuks. 2013. Qualitative activity recognition of weight lifting exercises. In Proceedings of the 4th Augmented Human International Conference. ACM, 116--123.
48. Vimo Labs Inc. [n.d.]. TrackMyFitness. http://trackmy.fit; Last Accessed: June 2020.
49. Stefan Vogel, Markus HÃ1/4lsbusch, Thomas Hennig, Vladimir Blazek, and Steffen Leonhardt. 2009. In-Ear Vital Signs Monitoring Using a Novel Microoptic Reflective Sensor. IEEE transactions on information technology in biomedicine : a publication of the IEEE Engineering in Medicine and Biology Society 13 (11 2009), 882--9.
50. Kyle M Wilson, William S Helton, Neil R de Joux, James R Head, and Jonathon JS Weakley. 2017. Real-time quantitative performance feedback during strength exercise improves motivation, competitiveness, mood, and performance. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 61. SAGE Publications Sage CA: Los Angeles, CA, 1546--1550.
51. Fu Xiao, Jing Chen, Xiao Hui Xie, Linqing Gui, Juan Li Sun, and Wang none Ruchuan. 2018. SEARE: A System for Exercise Activity Recognition and Quality Evaluation Based on Green Sensing. IEEE Transactions on Emerging Topics in Computing (2018).
52. Zhixian Yan, Vigneshwaran Subbaraju, Dipanjan Chakraborty, Archan Misra, and Karl Aberer. [n.d.]. Energy-efficient continuous activity recognition on mobile phones: An activity-adaptive approach. In Proc. of ISWC'12.
53. Bo Zhou, Mathias Sundholm, Jingyuan Cheng, Heber Cruz, and Paul Lukowicz. 2016. Never skip leg day: A novel wearable approach to monitoring gym leg exercises. In Proc. of IEEE PerCom'16. IEEE, 1--9.

Index Terms

• Computer systems organization

  • Embedded and cyber-physical systems

    • Sensors and actuators

• Information systems

  • Information systems applications

    • Mobile information processing systems