Hand modeling and simulation using stabilized magnetic resonance imaging

Abstract

We demonstrate how to acquire complete human hand bone anatomy (meshes) in multiple poses using magnetic resonance imaging (MRI). Such acquisition was previously difficult because MRI scans must be long for high-precision results (over 10 minutes) and because humans cannot hold the hand perfectly still in non-trivial and badly supported poses. We invent a manufacturing process whereby we use lifecasting materials commonly employed in film special effects industry to generate hand molds, personalized to the subject, and to each pose. These molds are both ergonomic and encasing, and they stabilize the hand during scanning. We also demonstrate how to efficiently segment the MRI scans into individual bone meshes in all poses, and how to correspond each bone's mesh to same mesh connectivity across all poses. Next, we interpolate and extrapolate the MRI-acquired bone meshes to the entire range of motion of the hand, producing an accurate data-driven animation-ready rig for bone meshes. We also demonstrate how to acquire not just bone geometry (using MRI) in each pose, but also a matching highly accurate surface geometry (using optical scanners) in each pose, modeling skin pores and wrinkles. We also give a soft tissue Finite Element Method simulation "rig", consisting of novel tet meshing for stability at the joints, spatially varying geometric and material detail, and quality constraints to the acquired skeleton kinematic rig. Given an animation sequence of hand joint angles, our FEM soft tissue rig produces quality hand surface shapes in arbitrary poses in the hand range of motion. Our results qualitatively reproduce important features seen in the photographs of the subject's hand, such as similar overall organic shape and fold formation.

References

1. Agisoft. 2018. Photoscan, http://www.agisoft.com.
2. AljaSafe. 2018. SmoothOn Inc. www.smooth-on.com.
3. Amira. 2018. Amira for Life Sciences. https://www.fei.com/software/amira-3d-for-life-sciences/.
4. Artec3D. 2018. Spider Scanner, http://www.artec3d.com.
5. Noelle M. Austin. 2005. Chapter 9: The Wrist and Hand Complex". In Levangie, Pamela K.; Norkin, Cynthia C. Joint Structure and Function: A Comprehensive Analysis (4th ed.). F. A. Davis Company.
6. J. Barbič and D. L. James. 2008. Six-DoF haptic rendering of contact between geometrically complex reduced deformable models. IEEE Trans. on Haptics 1, 1 (2008), 39--52.
7. Pierre-Yves Baudin, Noura Azzabou, Pierre G Carlier, and Nikos Paragios. 2012. Automatic skeletal muscle segmentation through random walks and graph-based seed placement. In IEEE Int. Symp. on Biomedical Imaging (ISBI). 1036--1039.
8. Paul J. Besl and N.D. McKay. 1992. A Method for Registration of 3-D Shapes. IEEE Trans. on Pattern Analysis and Machine Intelligence 14, 2 (1992), 239--256.
9. Jean-Daniel Boissonnat and Steve Oudot. 2005. Provably good sampling and meshing of surfaces. Graphical Models 67, 5 (2005), 405 -- 451.
10. S. Capell, M. Burkhart, B. Curless, T. Duchamp, and Z. Popović. 2005. Physically Based Rigging for Deformable Characters. In Symp. on Computer Animation (SCA). 301--310.
11. Paolo Cignoni, Marco Callieri, Massimiliano Corsini, Matteo Dellepiane, Fabio Ganovelli, and Guido Ranzuglia. 2008. MeshLab: an Open-Source Mesh Processing Tool. In Eurographics Italian Chapter Conference.
12. CyberGlove Systems. 2017. CyberGrasp. http://www.cyberglovesystems.com/cybergrasp.
13. Mary F Dempsey, Barrie Condon, and Donald M Hadley. 2002. MRI safety review. In Seminars in Ultrasound, CT and MRI, Vol. 23. Elsevier, 392--401.
14. C. M. Deniz, S. Xiang, S. Hallyburton, A. Welbeck, S. Honig, K. Cho, and G. Chang. 2017. Segmentation of the Proximal Femur from MR Images using Deep Convolutional Neural Networks. arXiv preprint arXiv:1704.06176 (2017).
15. D. E. Discher, D. J. Mooney, and P. W. Zandstra. 2009. Growth factors, matrices, and forces combine and control stem cells. Science 324, 5935 (2009), 1673--1677.
16. James S Duncan and Nicholas Ayache. 2000. Medical image analysis: Progress over two decades and the challenges ahead. IEEE Trans. on Pattern Analysis and Machine Intelligence 22, 1 (2000), 85--106.
17. M. S. Farvid, T. W. K. Ng, D. C. Chan, P. H. R. Barrett, and G. F. Watts. 2005. Association of adiponectin and resistin with adipose tissue compartments, insulin resistance and dyslipidaemia. Diabetes, Obesity and Metabolism 7, 4 (2005), 406--413.
18. A. Fenster and D. B. Downey. 1996. 3-D ultrasound imaging: a review. IEEE Engineering in Medicine and Biology Magazine 15, 6 (1996), 41--51.
19. Carlos Garre, Fernando Hernández, Antonio Gracia, and Miguel A Otaduy. 2011. Interactive simulation of a deformable hand for haptic rendering. In IEEE World Haptics Conference (WHC). IEEE, 239--244.
20. Benjamin Gilles and Nadia Magnenat-Thalmann. 2010. Musculoskeletal MRI segmentation using multi-resolution simplex meshes with medial representations. Medical image analysis 14, 3 (2010), 291--302.
21. Leo Grady. 2006. Random walks for image segmentation. IEEE Trans. on Pattern Analysis and Machine Intelligence 28, 11 (2006), 1768--1783.
22. Agneta Gustus and Patrick van der Smagt. 2016. Evaluation of joint type modelling in the human hand. Journal of Biomechanics 49, 13 (2016), 3097 -- 3100.
23. Shangchen Han, Beibei Liu, Robert Wang, Yuting Ye, Christopher D Twigg, and Kenrick Kin. 2018. Online optical marker-based hand tracking with deep labels. ACM Transactions on Graphics (SIGGRAPH 2018) 37, 4 (2018), 166.
24. Hang Si. 2011. TetGen: A Quality Tetrahedral Mesh Generator and a 3D Delaunay Triangulator.
25. Yixin Hu, Qingnan Zhou, Xifeng Gao, Alec Jacobson, Denis Zorin, and Daniele Panozzo. 2018. Tetrahedral Meshing in the Wild. ACM Trans. on Graphics (SIGGRAPH 2018) 37, 4 (2018), 60:1--60:14.
26. G. Irving, J. Teran, and R. Fedkiw. 2004. Invertible Finite Elements for Robust Simulation of Large Deformation. In Symp. on Computer Animation (SCA). 131--140.
27. ITK-SNAP. 2018. ITK-SNAP. http://www.itksnap.org/pmwiki/pmwiki.php.
28. Alec Jacobson, Zhigang Deng, Ladislav Kavan, and JP Lewis. 2014. Skinning: Real-time Shape Deformation. In ACM SIGGRAPH 2014 Courses.
29. A.I. Kapandji. 2009. The physiology of the joints, 6th Edition, Vol. 1: The Upper Limb. Elsevier Exclusive.
30. L. Kavan, S. Collins, J. Zara, and C. O'Sullivan. 2008. Geometric Skinning with Approximate Dual Quaternion Blending. ACM Trans. on Graphics 27, 4 (2008).
31. Baris Kayalibay, Grady Jensen, and Patrick van der Smagt. 2017. CNN-based segmentation of medical imaging data. arXiv preprint arXiv:1701.03056 (2017).
32. Junggon Kim and Nancy S Pollard. 2011. Fast simulation of skeleton-driven deformable body characters. ACM Trans. on Graphics (TOG) 30, 5 (2011), 121.
33. Jonathan P King, Dominik Bauer, Cornelia Schlagenhauf, Kai-Hung Chang, Daniele Moro, Nancy Pollard, and Stelian Coros. 2018. Design. Fabrication, and Evaluation of Tendon-Driven Multi-Fingered Foam Hands. In IEEE-RAS Int. Conf. on Humanoid Robots (Humanoids). 1--9.
34. Paul G. Kry, Doug L. James, and Dinesh K. Pai. 2002. EigenSkin: Real Time Large Deformation Character Skinning in Hardware. In Proc. of the Symp. on Comp. Animation 2002. 153--160.
35. Tsuneya Kurihara and Natsuki Miyata. 2004. Modeling deformable human hands from medical images. In Symp. on Computer Animation (SCA). 355--363.
36. LeapMotion. 2017. https://www.leapmotion.com.
37. S. H. Lee, E. Sifakis, and D. Terzopoulos. 2009. Comprehensive Biomechanical Modeling and Simulation of the Upper Body. ACM Trans. on Graphics 28, 4 (2009), 99:1--99:17.
38. J. P. Lewis, Matt Cordner, and Nickson Fong. 2000. Pose Space Deformations: A Unified Approach to Shape Interpolation and Skeleton-Driven Deformation. In Proc. of ACM SIGGRAPH 2000. 165--172.
39. Duo Li, Shinjiro Sueda, Debanga R Neog, and Dinesh K Pai. 2013. Thin Skin Elastodynamics. ACM Trans. Graph. (Proc. SIGGRAPH) 32, 4 (2013), 49:1--49:9.
40. Libin Liu, KangKang Yin, Bin Wang, and Baining Guo. 2013. Simulation and control of skeleton-driven soft body characters. ACM Trans. on Graphics (SIGGRAPH Asia 2013) 32, 6 (2013), 215.
41. N. Magnenat-Thalmann, R. Laperrire, and D. Thalmann. 1988. Joint-dependent local deformations for hand animation and object grasping. In Proc. of Graphics Interface.
42. Joe Mancewicz, Matt L. Derksen, Hans Rijpkema, and Cyrus A. Wilson. 2014. Delta Mush: Smoothing Deformations While Preserving Detail. In Proceedings of the Fourth Symposium on Digital Production (DigiPro '14). 7--11.
43. G. Marai, D. Laidlaw, J. Coburn, M. Upal, and J. Crisco. 2003. A 3D method for segmenting and registering carpal bones from CT volume images. In Proc. of Annual Meeting of the American Society of Biomechanics.
44. A. McAdams, Y. Zhu, A. Selle, M. Empey, R. Tamstorf, J. Teran, and E. Sifakis. 2011. Efficient elasticity for character skinning with contact and collisions. ACM Trans. on Graphics (SIGGRAPH 2011) 30, 4 (2011).
45. Tim McInerney and Demetri Terzopoulos. 2008. Deformable models in medical image analysis: a survey. Medical Image Analysis 1, 2 (2008), 91--108.
46. Fernand Meyer. 1992. Color image segmentation. In International Conf. on Image Processing and its Applications. IET, 303--306.
47. Aslan Miriyev, Kenneth Stack, and Hod Lipson. 2017. Soft material for soft actuators. Nature Communications 8, 596 (2017).
48. N. Miyata, M. Kouch, M. Mochimaru, and T. Kurihara. 2005. Finger joint kinematics from MR images. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems. 2750--2755.
49. NimbleVR. 2012. http://nimblevr.com.
50. RadiologyInfo. 2018. Radiation Dose in X-Ray and CT Exams. https://www.radiologyinfo.org/en/pdf/safety-xray.pdf.
51. Taehyun Rhee, J.P. Lewis, and Ulrich Neumann. 2006. Real-Time Weighted Pose-Space Deformation on the GPU. In Proc. of Eurographics 2006, Vol. 25.
52. Javier Romero, Dimitrios Tzionas, and Michael J. Black. 2017. Embodied Hands: Modeling and Capturing Hands and Bodies Together. ACM Trans. on Graphics (SIGGRAPH Asia 2017) 36, 6 (2017), 245:1--245:17.
53. Alexandru Rusu. 2011. Segmentation of bone structures in Magnetic Resonance Images (MRI) for human hand skeletal kinematics modelling. Master's thesis. German Aerospace Center.
54. Prashant Sachdeva, Shinjiro Sueda, Susanne Bradley, Mikhail Fain, and Dinesh K. Pai. 2015. Biomechanical Simulation and Control of Hands and Tendinous Systems. ACM Trans. Graph. 34, 4 (2015), 42:1--42:10.
55. Cornelia Schlagenhauf, Dominik Bauer, Kai-Hung Chang, Jonathan P King, Daniele Moro, Stelian Coros, and Nancy Pollard. 2018. Control of tendon-driven soft foam robot hands. In IEEE-RAS Int. Conf. on Humanoid Robots (Humanoids). 1--7.
56. Jérôme Schmid, Jinman Kim, and Nadia Magnenat-Thalmann. 2011. Robust statistical shape models for MRI bone segmentation in presence of small field of view. Medical image analysis 15, 1 (2011), 155--168.
57. Jérôme Schmid and Nadia Magnenat-Thalmann. 2008. MRI bone segmentation using deformable models and shape priors. In Int. Conf. on Medical Image Computing and Computer-Assisted Intervention. 119--126.
58. Breannan Smith, Fernando De Goes, and Theodore Kim. 2018. Stable Neo-Hookean Flesh Simulation. ACM Trans. Graph. 37, 2 (2018), 12:1--12:15.
59. Ole Vegard Solberg, Frank Lindseth, Hans Torp, Richard E. Blake, and Toril A. Nagelhus Hernes. 2007. Freehand 3D Ultrasound Reconstruction Algorithms: A Review. Ultrasound in Medicine and Biology 33, 7 (2007), 991 -- 1009.
60. Georg Stillfried. 2015. Kinematic modelling of the human hand for robotics. Ph.D. Dissertation. Technische Universität München.
61. Shinjiro Sueda, Andrew Kaufman, and Dinesh K. Pai. 2008. Musculotendon Simulation for Hand Animation. ACM Trans. Graph. (Proc. SIGGRAPH) 27, 3 (2008).
62. Tissue. 2013. Weta Digital: Tissue Muscle and Fat Simulation System.
63. Rodolphe Vaillant, Gäel Guennebaud, Loïc Barthe, Brian Wyvill, and Marie-Paule Cani. 2014. Robust Iso-surface Tracking for Interactive Character Skinning. ACM Trans. on Graphics (SIGGRAPH Asia 2014) 33, 6 (2014), 189:1--137:11.
64. Patrick van der Smagt and Georg Stillfried. 2008. Using MRI data to compute a hand kinematic model. In Conf. on Motion and Vibration Control (MOVIC).
65. VTK. 2018. VTK. https://www.vtk.org/.
66. Andreas Wächter and Lorenz T. Biegler. 2006. On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming. Mathematical Programming 106, 1 (01 Mar 2006), 25--57.
67. Andrea Walther and Andreas Griewank. 2009. Getting Started with ADOL-C. In Combinatorial scientific computing. 181--202.
68. R. Y. Wang, S. Paris, and J. Popović. 2011. 6D Hands: Markerless Hand Tracking for Computer Aided Design. In ACM User Interface Software and Technology (UIST).
69. Robert Y. Wang and Jovan Popović. 2009. Real-time hand-tracking with a color glove. ACM Trans. on Graphics (SIGGRAPH 2009) 28, 3 (2009), 63:1--63:8.
70. Robert E. Watson. 2015. Lessons Learned from MRI Safety Events. Current Radiology Reports 3, 10 (12 Aug 2015), 37.
71. C. Wex, S. Arndt, A. Stoll, C. Bruns, and Y. Kupriyanova. 2015. Isotropic incompressible hyperelastic models for modelling the mechanical behaviour of biological tissues: a review. Biomedical Engineering / Biomedizinische Technik 60, 6 (2015), 577--592.
72. Nkenge Wheatland, Yingying Wang, Huaguang Song, Michael Neff, Victor Zordan, and Sophie Jörg. 2015. State of the art in hand and finger modeling and animation. In Computer Graphics Forum, Vol. 34. 735--760.
73. Wrap3. 2018. Nonlinear Iterative Closest Point mesh registration software. https://www.russian3dscanner.com.
74. Shanxin Yuan, Qi Ye, Björn Stenger, Siddhant Jain, and Tae-Kyun Kim. 2017. Bighand2. 2M benchmark: Hand pose dataset and state of the art analysis. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2605--2613.
75. Zygote. 2016. Zygote body. http://www.zygotebody.com.

Index Terms

• Computing methodologies

  • Computer graphics

    • Animation

      • Physical simulation