Animation cartography—intrinsic reconstruction of shape and motion

Abstract

In this article, we consider the problem of animation reconstruction, that is, the reconstruction of shape and motion of a deformable object from dynamic 3D scanner data, without using user-provided template models. Unlike previous work that addressed this problem, we do not rely on locally convergent optimization but present a system that can handle fast motion, temporally disrupted input, and can correctly match objects that disappear for extended time periods in acquisition holes due to occlusion. Our approach is motivated by cartography: We first estimate a few landmark correspondences, which are extended to a dense matching and then used to reconstruct geometry and motion. We propose a number of algorithmic building blocks: a scheme for tracking landmarks in temporally coherent and incoherent data, an algorithm for robust estimation of dense correspondences under topological noise, and the integration of local matching techniques to refine the result. We describe and evaluate the individual components and propose a complete animation reconstruction pipeline based on these ideas. We evaluate our method on a number of standard benchmark datasets and show that we can obtain correct reconstructions in situations where other techniques fail completely or require additional user guidance such as a template model.

References

1. Ahmed, N., Theobalt, C., Rossl, C., Thrun, S., and Seidel, H.-P. 2008. Dense correspondence finding for parameterization-free animation reconstruction from video. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR'08). 1--8.
2. Allen, B., Curless, B., and Popović, Z. 2003. The space of human body shapes: Reconstruction and parameterization from range scans. ACM Trans. Graph. 22, 3.
3. Anguelov, D., Srinivasan, P., Pang, H.-C., Koller, D., Thrun, S., and Davis, J. 2004. The correlated correspondence algorithm for unsupervised registration of nonrigid surfaces. In Proceedings of the Conference on Neural Information Processing Systems (NIPS'04).
4. Anuar, N. and Guskov, I. 2004. Extracting animated meshes with adaptive motion estimation. In Proceedings of the Vision, Modeling, and Visualization Workshop (VMV'04). 63--71.
5. Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M., Pfister, H., and Gross, M. 2007. Multi-Scale capture of facial geometry and motion. ACM Trans. Graph. 26, 3, 33.
6. Bradley, D., Heidrich, W., Popa, T., and Sheffer, A. 2010. High resolution passive facial performance capture. ACM Trans. Graph. 29, 41:1-41:10.
7. Bradley, D., Popa, T., Sheffer, A., Heidrich, W., and Boufekeur, T. 2008. Markerless garment capture. ACM Trans. Graph. 27, 3, 99.
8. Bronstein, A. M., Bronstein, M. M., and Kimmel, R. 2006. Generalized multidimensional scaling: A framework for isometry-invariant partial surface matching. Proc. Nat. Acad. Sci. 103, 5, 1168--1172.
9. Bronstein, A. M., Bronstein, M. M., and Kimmel, R. 2009. Topology-Invariant similarity of nonrigid shapes. Int. J. Comput. Vis. 81, 3, 281--301.
10. Bronstein, A. M., Bronstein, M. M., Kimmel, R., Mahmoudi, M., and Sapiro, G. 2010. A gromov-hausdorff framework with diffusion geometry for topologically-robust nonrigid shape matching. Int. J. Comput. Vis. 89, 266--286.
11. Carranza, J., Theobalt, C., Magnor, M. A., and Seidel, H.-P. 2003. Free-Viewpoint video of human actors. ACM Trans. Graph. 22, 569--577.
12. Chang, W. and Zwicker, M. 2008. Automatic registration for articulated shapes. Comput. Graph. Forum 27, 5, 1459--1468.
13. Chang, W. and Zwicker, M. 2009. Range scan registration using reduced deformable models. Comput. Graph. Forum 28, 2, 447--456.
14. Chum, O. and Matas, J. 2005. Matching with PROSAC—Progressive sample consensus. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR'05). Vol. 1. 220--226.
15. Davis, J., Nehab, D., Ramamoorthi, R., and Rusinkiewicz, S. 2005. Spacetime stereo: A unifying framework for depth from triangularization. IEEE Trans. Pattern Anal. Mach. Intell. 27, 2, 296--302.
16. De Aguiar, E., Stoll, C., Theobalt, C., Ahmed, N., Seidel, H.-P., and Thrun, S. 2008. Performance capture from sparse multi-view video. ACM Trans. Graph. 27, 3, 98.
17. Fang, Q., Gao, J., Guibas, L. J., de Silva, V., and Zhang, L. 2005. Glider: Gradient landmark-based distributed routing for sensor networks. In Proceedings of the 24^th Conference of the IEEE Communications Society (InfoCom).
18. Gelfand, N. and Guibas, L. J. 2004. Shape segmentation using local slippage analysis. In Proceedings of the Symposium on Geometry Processing (SGP'04). 214--223.
19. Häehnel, D., Thrun, S., and Burgard, W. 2003. An extension of the icp algorithm for modeling nonrigid objects with mobile robots. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). 915--920.
20. Horn, B. K. P. 1987. Closed-Form solution of absolute orientation using unit quaternions. J. Opt. Soc. Amer. A4, 4, 629--642.
21. Huang, Q.-X., Adams, B., Wicke, M., and Guibas, L. J. 2008. Nonrigid registration under isometric deformations. Comput. Graph. Forum 27, 5, 1449--1457.
22. Huang, Q.-X., Flöry, S., Gelfand, N., Hofer, M., and Pottmann, H. 2006. Reassembling fractured objects by geometric matching. ACM Trans. Graph. 25, 3, 569--578.
23. König, S. and Gumhold, S. 2008. Image-Based motion compensation for structured light scanning of dynamic scenes. Int. J. Int. Syst. Tech. Appl. 5, 3-4, 434--441.
24. Leordenu, M. and Hebert, M. 2005. A spectral technique for correspondence problems using pairwise constraints. In Proceedings of the International Conference on Computer Vision (ICCV'05). Vol. 2, 1482--1489.
25. Li, H., Adams, B., Guibas, L. J., and Pauly, M. 2009. Robust single-view geometry and motion reconstruction. ACM Trans. Graph. 28, 5, 175.
26. Li, H., Sumner, R. W., and Pauly, M. 2008. Global correspondence optimization for nonrigid registration of depth scans. Comput. Graph. Forum 27, 5, 1421--1430.
27. Liao, M., Zhang, Q., Wang, H., Yang, R., and Gong, M. 2009. Modeling deformable objects from a single depth camera. In Proceedings of the International Conference on Computer Vision (ICCV'09).
28. Lipman, Y. and Funkhouser, T. 2009. Möbius voting for surface correspondence. In Proceedings of the SIGGRAPH'09 Conference. ACM Press, New York, 1--12.
29. Lucas, B. D. and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI'81). 674--679.
30. Mitra, N. J., Flory, S., Ovsjanikov, M., Gelfand, N., Guibas, L., and Pottsmann, H. 2007. Dynamic geometry registration. In Proceedings of the Symposium on Geometry Processing (SGP'07). 173--182.
31. Ovsjanikov, M., Mérigot, Q., Mémoli, F., and Guibas, L. 2010. One point isometric matching with the heat kernel. In Proceedings of the Symposium on Geometry Processing (SGP'10). 1555--1564.
32. Park, S. I. and Hodgins, J. K. 2006. Capturing and animating skin deformation in human motion. ACM Trans. Graph. 25, 3, 881--889.
33. Pekelny, Y. and Gotsman, C. 2008. Articulated object reconstruction and markerless motion capture from depth video. Comput. Graph. Forum 27, 2, 399--408.
34. Popa, T., South-Dickenson, I., Bradley, D., Sheffer, A., and Heidrich, W. 2010. Globally consistent space-time reconstruction. Comput. Graph. Forum 29, 5, 1633--1642.
35. Sand, P., McMillan, L., and Popović, J. 2003. Continuous capture of skin deformation. ACM Trans. Graph. 22, 3, 578--586.
36. Sharf, A., Alcantara, D. A., Lewiner, T., Greif, C., Sheffer, A., Amenta, N., and Cohen-Or, D. 2008. Space-Time surface reconstruction using incompressible flow. ACM Trans. Graph. 27, 5, 110.
37. Starck, J. and Hilton, A. 2007. Correspondence labeling for wide-timeframe free-form surface matching. In Proceedings of the International Conference on Computer Vision (ICCV'07). 1--8.
38. Süssmuth, J., Winter, M., and Greiner, G. 2008. Reconstructing animated meshes from time-varying point clouds. Comput. Graph. Forum 27, 5, 1469--1476.
39. Tevs, A., Berner, A., Wand, M., Ihrke, I., and Seidel, H.-P. 2011. Intrinsic shape matching by planned landmark sampling. Comput. Graph. Forum 30, 543--552.
40. Tevs, A., Bokeloh, M., Wand, M., Schilling, A., and Seidel, H.-P. 2009. Isometric registration of ambiguous and partial data. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR'09). 1185--1192.
41. Varanasi, K., Zaharescu, A., Boyer, E., and Horaud, R. P. 2008. Temporal surface tracking using mesh evolution. In Proceedings of the European Conference on Computer Vision (ECCV'08). 30--43.
42. Vlasic, D., Peers, P., Baran, I., Debevec, P., Popović, J., Rusinkiewicz, S., and Matusik, W. 2009. Dynamic shape capture using multi-view photometric stereo. ACM Trans. Graph. 28, 5, 174.
43. Wand, M., Adams, B., Ovsjanikov, M., Berner, A., Bokeloh, M., Jenke, P., Guibas, L., Seidel, H.-P., and Schilling, A. 2009. Efficient reconstruction of nonrigid shape and motion from real-time 3D scanner data. ACM Trans. Graph. 28, 2, 1--15.
44. Wand, M., Jenke, P., Huang, Q., Bokeloh, M., Guibas, L., and Schilling, A. 2007. Reconstruction of deforming geometry from time-varying point clouds. In Proceedings of the Symposium on Geometric Processing (SGP'07). 49--58.
45. Weise, T., Leife, B., and Gool, L. V. 2007. Fast 3D scanning with automatic motion compensation. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR'07). 1--8.
46. Würmlin, S., Lamboray, E., Staadt, O. G., and Gross, M. H. 2002. 3D video recorder. In Proceedings of the 10^th Pacific Conference on Computer Graphics and Applications (PG'02). IEEE Computer Society, Washington, DC, 325.
47. Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Space-Time faces: High resolution capture for modeling and animation. ACM Trans. Graph. 23, 3, 548--558.
48. Zheng, Q., Sharf, A., Tagliasacchi, A., Chen, B., Zhang, H., Sheffer, A., and Cohen-Or, D. 2010. Consensus skeleton for nonrigid space-time registration. Comput. Graph. Forum 29, 2, 635--644.
49. Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2004. High-Quality video view interpolation using a layered representation. In ACM SIGGRAPH'04 Papers. ACM, New York, 600--608.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

      • Computer vision tasks

        • Scene understanding

  • Computer graphics

    • Animation

    • Shape modeling

      • Parametric curve and surface models