Spacetime expression cloning for blendshapes

Abstract

The goal of a practical facial animation retargeting system is to reproduce the character of a source animation on a target face while providing room for additional creative control by the animator. This article presents a novel spacetime facial animation retargeting method for blendshape face models. Our approach starts from the basic principle that the source and target movements should be similar. By interpreting movement as the derivative of position with time, and adding suitable boundary conditions, we formulate the retargeting problem as a Poisson equation. Specified (e.g., neutral) expressions at the beginning and end of the animation as well as any user-specified constraints in the middle of the animation serve as boundary conditions. In addition, a model-specific prior is constructed to represent the plausible expression space of the target face during retargeting. A Bayesian formulation is then employed to produce target animation that is consistent with the source movements while satisfying the prior constraints. Since the preservation of temporal derivatives is the primary goal of the optimization, the retargeted motion preserves the rhythm and character of the source movement and is free of temporal jitter. More importantly, our approach provides spacetime editing for the popular blendshape representation of facial models, exhibiting smooth and controlled propagation of user edits across surrounding frames.

References

1. Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., and Gross, M. 2011. High-quality passive facial performance capture using anchor frames. ACM Trans. Graph. 30, 75:1--75:10.
2. Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M., Pfister, H., and Gross, M. 2007. Multi-Scale capture of facial geometry and motion. ACM Trans. Graph. 26, 3, 33.
3. Blanz, V. and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive techniques. ACM Press/Addison-Wesley Publishing Co., New York, 187--194.
4. Borshukov, G., Piponi, D., Larsen, O., Lewis, J., and Tempelaar-Lietz, C. 2003. Universal capture: Image-based facial animation for “The Matrix Reloaded.” In ACM SIGGRAPH 2003 Sketches & Applications. ACM, New York, 1--1.
5. Bregler, C., Loeb, L., Chuang, E., and Deshpande, H. 2002. Turning to the masters: Motion capturing cartoons. ACM Trans. Graph. 21, 3, 399--407.
6. Cao, Y., Faloutsos, P., and Pighin, F. 2003. Unsupervised learning for speech motion editing. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Eurographics Association, 225--231.
7. Choe, B., Lee, H., and seok Ko, H. 2001. Performance-driven muscle-based facial animation. J. Vis. Comput. Anim. 12, 67--79.
8. Chuang, E. and Bregler, C. 2002. Performance driven facial animation using blendshape interpolation. Tech. rep., Department of Computer Science, Stanford University.
9. Chuang, E. and Bregler, C. 2005. Mood swings: Expressive speech animation. ACM Trans. Graph. 24, 2, 331--347.
10. Deng, Z., Chiang, P.-Y., Fox, P., and Neumann, U. 2006. Animating blendshape faces by cross-mapping motion capture data. In Proceedings of the Symposium on Interactive 3D Graphics and Games. ACM, New York, 43--48.
11. Deng, Z. and Noh, J. 2007. Computer Facial Animation: A Survey. Springer, London.
12. Ekman, P. and Friesen, W. 1977. Manual for the Facial Action Coding System. Consulting Psychologists Press, Palo Alto, CA.
13. Havaldar, P. 2006. Performance driven facial animation. In ACM SIGGRAPH '06 Course #30 Notes.
14. Hertzmann, A. 2004. Introduction to Bayesian learning. In ACM SIGGRAPH'04 Course Notes. ACM, New York, 22.
15. Joshi, P., Tien, W. C., Desbrun, M., and Pighin, F. 2003. Learning controls for blend shape based realistic facial animation. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Eurographics Association, 187--192.
16. Kovar, L. and Gleicher, M. 2003. Flexible automatic motion blending with registration curves. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Eurographics Association, 214--224.
17. Kraevoy, V., Sheffer, A., and Gotsman, C. 2003. Matchmaker: constructing constrained texture maps. ACM SIGGRAPH'03 Papers. ACM, 326--333.
18. Lau, M., Chai, J., Xu, Y.-Q., and Shum, H.-Y. 2009. Face poser: Interactive modeling of 3d facial expressions using facial priors. ACM Trans. Graph. 29, 1, 1--17.
19. Lee, J., Chai, J., Reitsma, P. S. A., Hodgins, J. K., and Pollard, N. S. 2002. Interactive control of avatars animated with human motion data. ACM Trans. Graph. 21, 3, 491--500.
20. Levy, B., Petitjean, S., Ray, N., and Maillo t, J. 2002. Least squares conformal maps for automatic texture atlas generation. In ACM SIGGRAPH Conference Proceedings.
21. Lewis, J. and Anjyo, K. 2010. Direct manipulation blendshapes. IEEE Comput. Graph. Appl. 30, 4, 42--50.
22. Li, H., Sumner, R. W., and Pauly, M. 2008. Global correspondence optimization for non-rigid registration of depth scans. Comput. Graph. Forum 27, 5.
23. Li, H., Weise, T., and Pauly, M. 2010. Example-based facial rigging. ACM Trans. Graph. 29, 4, 1--6.
24. Li, Q. and Deng, Z. 2008. Orthogonal blendshape based editing system for facial motion capture data. IEEE Comput. Graph. Appl., 76--82.
25. Luamanuvae, J. 2010. Personal communication (Weta Digital).
26. Ma, W.-C., Jones, A., Chiang, J.-Y., Hawkins, T., Frederiksen, S., Peers, P., Vukovic, M., Ouhyoung, M., and Debevec, P. 2008. Facial performance synthesis using deformation-driven polynomial displacement maps. ACM Trans. Graph. 27, 5, 1--10.
27. Ma, X., Le, B. H., and Deng, Z. 2009. Style learning and transferring for facial animation editing. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. ACM, New York, 123--132.
28. Noh, J. and Neumann, U. 2001. Expression cloning. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. ACM. 277--288.
29. Orvalho, V. C., Zacur, E., and Susin, A. 2008. Transferring the rig and animations from a character to different face models. Comput. Graph. Forum 27, 8, 1997--2012.
30. Parke, F. I. 1972. Computer generated animation of faces. In Proceedings of the ACM Annual Conference. ACM, New York, 451--457.
31. Parke, F. I. and Waters, K. 1996. Computer Facial Animation. A. K. Peters.
32. Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Trans. Graph. 22, 3, 313--318.
33. Pighin, F., Szeliski, R., and Salesin, D. H. 2002. Modeling and animating realistic faces from images. Int. J. Comput. Vis. 50, 2, 143--169.
34. Pyun, H., Kim, Y., Chae, W., Kang, H. W., and Shin, S. Y. 2003. An example-based approach for facial expression cloning. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Eurographics Association, 167--176.
35. Reveret, L. and Essa, I. 2001. Visual coding and tracking of speech related facial motion. Tech. rep., IEEE CVPR International Workshop on Cues in Communication.
36. Sagar, M. and Grossman, R. 2006. Facial performance capture and expressive translation for King Kong. In SIGGRAPH Sketches.
37. Sumner, R. W. and Popovic, J. 2004. Deformation transfer for triangle meshes. ACM SIGGRAPH Papers. ACM. 399--405.
38. Sumner, R. W., Zwicker, M., Gotsman, C., and Popović, J. 2005. Mesh-based inverse kinematics. In ACM SIGGRAPH Papers. ACM, New York, 488--495.
39. Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. ACM Trans. Graph. 24, 3, 426--433.
40. Wahba, G. 1990. Spline Models for Observational Data. SIAM.
41. Weise, T., Li, H., Van Gool, L., and Pauly, M. 2009. Face/off: Live facial puppetry. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. ACM, New York, 7--16.
42. Williams, L. 1990. Performance-driven facial animation. In Proceedings of the 17th Annual Conference on Computer Graphics and Interactive Techniques. ACM, New York, 235--242.
43. Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: High resolution capture for modeling and animation. ACM SIGGRAPH Papers. ACM, New York, 548--558.

Index Terms

• Computing methodologies

  • Computer graphics

    • Animation