Free-hand interaction with leap motion controller for stroke rehabilitation

ABSTRACT

In recent years, the field of Human-Computer Interaction (HCI) has been advanced with many technologies, however, most are limited to healthy users. In this paper, we leveraged the technology of free-hand interaction to rehabilitate patients with stroke. We modified the game of Fruit Ninja to use Leap Motion controller's hand tracking data for stroke patients with arm and hand weakness to practice their finger individuation. In a pilot study, we recruited 14 patients with chronic stroke to play the game using natural interaction. Their Fruit Ninja (FN) scores show high correlation with the standard clinical assessment scores such as Fugl-Meyer (FMA) and Box-and-Blocks Test (BBT) scores. This finding suggests that our free-hand Fruit Ninja's score is a good indicator of the patient's hand function and therefore will be informative if used in their rehabilitation.

References

1. D. Kim, O. Hilliges, S. Izadi, A. D. Butler, J. Chen, I. Oikonomidis, and P. Olivier, "Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor," in Proceedings of the 25th annual ACM symposium on User interface software and technology, New York, NY, USA, 2012, pp. 167--176.
2. H. Kim, G. Albuquerque, S. Havemann, and D. W. Fellner, "Tangible 3D: hand gesture interaction for immersive 3D modeling," in Proceedings of the 11th Eurographics conference on Virtual Environments, Aire-la-Ville, Switzerland, Switzerland, 2005, pp. 191--199.
3. D. Datcu and S. Lukosch, "Free-hands interaction in augmented reality," in Proceedings of the 1st symposium on Spatial user interaction, New York, NY, USA, 2013, pp. 33--40.
4. J. Sutton, "Air painting with Corel Painter Freestyle and the leap motion controller: a revolutionary new way to paint!" in ACM SIGGRAPH 2013 Studio Talks, New York, NY, USA, 2013, pp. 21:1--21:1.
5. S. Vikram, L. Li, and S. Russell, "Writing and sketching in the air, recognizing and controlling on the fly," in CHI'13 Extended Abstracts on Human Factors in Computing Systems, New York, NY, USA, 2013, pp. 1179--1184.
6. S. C. Cramer et al., "Harnessing neuroplasticity for clinical applications," Brain J. Neurol., vol. 134, no. 6, pp. 1591--1609, Jun. 2011. Figure 5. Fingertip position and directions provided by Leap sensor Figure 6. Full hand skeleton made by applying inverse kinematics on hand's anatomical model and Leap's data x (mm)y (mm) z(mm)
7. S. C. Cramer, "An overview of therapies to promote repair of the brain after stroke," Head Neck, vol. 33 Suppl 1, pp. S5--7, Oct. 2011.
8. J. Hallam and V. Whiteley, "Interactive therapy gloves: reconnecting partners after a stroke," in CHI '11 Extended Abstracts on Human Factors in Computing Systems, New York, NY, USA, 2011, pp. 989--994.
9. C. Boulanger, A. Boulanger, L. de Greef, A. Kearney, K. Sobel, R. Transue, Z. Sweedyk, P. H. Dietz, and S. Bathiche, "Stroke rehabilitation with a sensing surface," in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, NY, USA, 2013, pp. 1243--1246.
10. G. Salivia and J. P. Hourcade, "PointAssist: assisting individuals with motor impairments," in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, NY, USA, 2013, pp. 1213--1222.
11. S. Trewin, S. Keates, and K. Moffatt, "Individual responses to a method of cursor assistance," Disabil. Rehabil. Assist. Technol., vol. 3, no. 1, pp. 2--21, Jan. 2008.
12. S. P. Eli Carmeli, "HandTutorTM enhanced hand rehabilitation after stroke," Physiother. Res. Int. J. Res. Clin. Phys. Ther., vol. 16, no. 4, pp. 191--200, 2011.
13. N. Friedman, V. Chan, D. Zondervan, M. Bachman, and D. J. Reinkensmeyer, "MusicGlove: motivating and quantifying hand movement rehabilitation by using functional grips to play music," Conf. Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. IEEE Eng. Med. Biol. Soc. Conf., vol. 2011, pp. 2359--2363, 2011.
14. fruit-ninja, GitHub. {Online}. Available: https://github.com/ChineseDron/fruit-ninja. {Accessed: 02-Aug-2013}.
15. H. M. Hondori, M. Khademi, A. McKenzie, L. Dodakian, C. V. Lopes, and S. C. Cramer, "Abstract T MP43: Utility of Augmented Reality in Relation to Virtual Reality in Stroke Rehabilitation," Stroke, vol. 45, no. Suppl 1, pp. ATMP43--ATMP43, Feb. 2014.
16. Fruit Ninja. {Online}. Available: http://fruitninja.com/. {Accessed: 16-Jul-2013}.
17. A. R. Fugl-Meyer, L. Jääskö, I. Leyman, S. Olsson, and S. Steglind, "The post-stroke hemiplegic patient. 1. A method for evaluation of physical performance," Scand. J. Rehabil. Med., vol. 7, no. 1, pp. 13--31, 1975.
18. V. Mathiowetz, G. Volland, N. Kashman, and K. Weber, "Adult norms for the Box and Block Test of manual dexterity," Am. J. Occup. Ther. Off. Publ. Am. Occup. Ther. Assoc., vol. 39, no. 6, pp. 386--391, Jun. 1985.
19. J. Lee and T. L. Kunii, "Model-based analysis of hand posture," IEEE Comput. Graph. Appl., vol. 15, no. 5, pp. 77--86, 1995.

Index Terms

• Applied computing

  • Computers in other domains

    • Personal computers and PC applications

  • Life and medical sciences

• Human-centered computing

  • Human computer interaction (HCI)