Optimizing UI layouts for deformable face-rig manipulation

Abstract

Complex deformable face-rigs have many independent parameters that control the shape of the object. A human face has upwards of 50 parameters (FACS Action Units), making conventional UI controls hard to find and operate. Animators address this problem by tediously hand-crafting in-situ layouts of UI controls that serve as visual deformation proxies, and facilitate rapid shape exploration. We propose the automatic creation of such in-situ UI control layouts. We distill the design choices made by animators into mathematical objectives that we optimize as the solution to an integer quadratic programming problem. Our evaluation is three-fold: we show the impact of our design principles on the resulting layouts; we show automated UI layouts for complex and diverse face rigs, comparable to animator handcrafted layouts; and we conduct a user study showing our UI layout to be an effective approach to face-rig manipulation, preferable to a baseline slider interface.

References

1. Rinat Abdrashitov, Fanny Chevalier, and Karan Singh. 2020. Interactive Exploration and Refinement of Facial Expression Using Manifold Learning. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST '20). Association for Computing Machinery, New York, NY, USA, 778--790.
2. Maneesh Agrawala, Wilmot Li, and Floraine Berthouzoz. 2011. Design Principles for Visual Communication. Commun. ACM 54, 4 (April 2011), 60--69.
3. Eric Allen and Kelly L. Murdock. 2008. Body Language: Advanced 3D Character Rigging (pap/cdr ed.). SYBEX Inc., USA.
4. Norman I Badler, Brian A Barsky, and David Zeltzer. 1990. Making Them Move: Mechanics, Control & Animation of Articulated Figures. Routledge.
5. Stephen W. Bailey, Dalton Omens, Paul Dilorenzo, and James F. O'Brien. 2020. Fast and Deep Facial Deformations. ACM Trans. Graph. 39, 4, Article 94 (July 2020), 15 pages.
6. Eric Allan Bier. 1987. Skitters and jacks: interactive 3D positioning tools. In Proceedings of the 1986 workshop on Interactive 3D graphics. ACM, 183--196.
7. Mario Botsch, Leif Kobbelt, Mark Pauly, Pierre Alliez, and Bruno Lévy. 2010. Polygon mesh processing. CRC press.
8. Eric Brochu, Tyson Brochu, and Nando de Freitas. 2010. A Bayesian interactive optimization approach to procedural animation design. In Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Eurographics Association, 103--112.
9. Ozan Cetinaslan and Verónica Orvalho. 2018. Direct Manipulation of Blendshapes Using a Sketch-Based Interface. In Proceedings of the 23rd International ACM Conference on 3D Web Technology (Poznań, Poland) (Web3D '18). Association for Computing Machinery, New York, NY, USA, Article 14, 10 pages.
10. Pif Edwards, Chris Landreth, Mateusz Popławski, Robert Malinowski, Sarah Watling, Eugene Fiume, and Karan Singh. 2020. JALI-Driven Expressive Facial Animation and Multilingual Speech in Cyberpunk 2077. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Talks (Virtual Event, USA) (SIGGRAPH '20). Association for Computing Machinery, New York, NY, USA, Article 60, 2 pages.
11. Rosenberg Ekman. 1997. What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS). Oxford University Press, USA.
12. Krzysztof Z Gajos, Mary Czerwinski, Desney S Tan, and Daniel S Weld. 2006. Exploring the design space for adaptive graphical user interfaces. In Proceedings of the working conference on Advanced visual interfaces. 201--208.
13. Sarah Gibson, Paul Beardsley, Wheeler Ruml, Thomas Kang, Brian Mirtich, Joshua Seims, William Freeman, Jessica Hodgins, Hanspeter Pfister, Joe Marks, et al. 1997. Design galleries: A general approach to setting parameters for computer graphics and animation. (1997).
14. Michael Gleicher. 1992. Integrating constraints and direct manipulation. In Symposium on Interactive 3 D Graphics: Proceedings of the 1992 symposium on Interactive 3 D graphics, Vol. 1992. 171--174.
15. Martin Guay, Marie-Paule Cani, and Rémi Ronfard. 2013. The line of action: an intuitive interface for expressive character posing. ACM Transactions on Graphics (TOG) 32, 6 (2013), 1--8.
16. Yuki Koyama, Daisuke Sakamoto, and Takeo Igarashi. 2014. Crowd-powered parameter analysis for visual design exploration. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 65--74.
17. Paul G Kry, Doug L James, and Dinesh K Pai. 2002. Eigenskin: real time large deformation character skinning in hardware. In Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation. 153--159.
18. Manfred Lau, Jinxiang Chai, Ying-Qing Xu, and Heung-Yeung Shum. 2009. Face poser: Interactive modeling of 3D facial expressions using facial priors. ACM Transactions on Graphics (TOG) 29, 1 (2009), 3.
19. Binh H. Le, Mingyang Zhu, and Zhigang Deng. 2013. Marker Optimization for Facial Motion Acquisition and Deformation. IEEE Transactions on Visualization and Computer Graphics 19, 11 (Nov. 2013), 1859--1871.
20. John P Lewis and Ken-ichi Anjyo. 2010. Direct manipulation blendshapes. IEEE Computer Graphics and Applications 30, 4 (2010), 42--50.
21. Hao Li, Jihun Yu, Yuting Ye, and Chris Bregler. 2013. Realtime Facial Animation with On-the-Fly Correctives. ACM Trans. Graph. 32, 4, Article 42 (July 2013), 10 pages.
22. José Carlos Miranda, Xenxo Alvarez, João Orvalho, Diego Gutierrez, A Augusto Sousa, and Verónica Orvalho. 2011. Sketch express: facial expressions made easy. In Proceedings of the Eighth Eurographics Symposium on Sketch-Based Interfaces and Modeling. ACM, 87--94.
23. Niloy J. Mitra, Mark Pauly, Michael Wand, and Duygu Ceylan. 2012. Symmetry in 3D Geometry: Extraction and Applications. In EUROGRAPHICS State-of-the-art Report.
24. Fionn Murtagh and Pedro Contreras. 2012. Algorithms for hierarchical clustering: an overview. WIREs Data Mining and Knowledge Discovery 2, 1 (2012), 86--97. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/widm.53
25. Victor Navone. 2020. Facial Animation for Feature Animated Films: Animating stylized facial expressions. https://www.thegnomonworkshop.com/tutorials/facial-animation-for-feature-animated-films (2020).
26. Jason Osipa. 2010. Stop Staring: Facial Modeling and Animation Done Right (3rd ed.). SYBEX Inc., USA.
27. Rick Parent. 2012. Computer animation: algorithms and techniques. Newnes.
28. Ryan Schmidt, Karan Singh, and Ravin Balakrishnan. 2008. Sketching and composing widgets for 3d manipulation. In Computer graphics forum, Vol. 27. Wiley Online Library, 301--310.
29. Thomas W Sederberg and Scott R Parry. 1986. Free-form deformation of solid geometric models. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques. 151--160.
30. Mike Seymour. 2016. Put your (digital) game face on. (2016). https://www.fxguide.com/fxfeatured/put-your-digital-game-face-on/
31. Mike Seymour. 2018. Making Thanos Face the Avengers. (2018). https://www.fxguide.com/fxfeatured/making-thanos-face-the-avengers/
32. Mike Seymour. 2019. Bebyface in Bebylon. (2019). https://www.fxguide.com/fxfeatured/bebyface-in-bebylon/
33. Karan Singh and Eugene Fiume. 1998. Wires: a geometric deformation technique. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques. 405--414.
34. Marc P Stevens, Robert C Zeleznik, and John F Hughes. 1994. An architecture for an extensible 3D interface toolkit. In Proceedings of the 7th annual ACM symposium on User interface software and technology. 59--67.
35. Tanasai Sucontphunt, Zhenyao Mo, Ulrich Neumann, and Zhigang Deng. 2008. Interactive 3D facial expression posing through 2D portrait manipulation. In Proceedings of graphics interface 2008. Canadian Information Processing Society, 177--184.
36. Jerry O Talton, Daniel Gibson, Lingfeng Yang, Pat Hanrahan, and Vladlen Koltun. 2009. Exploratory modeling with collaborative design spaces. ACM Transactions on Graphics-TOG 28, 5 (2009), 167.
37. J. Rafael Tena, Fernando De la Torre, and Iain Matthews. 2011. Interactive Region-Based Linear 3D Face Models. ACM Trans. Graph. 30, 4, Article 76 (July 2011), 10 pages.
38. Ayush Tewari, Michael Zollöfer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Theobalt Christian. 2017. MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction. In The IEEE International Conference on Computer Vision (ICCV).
39. Thibaut Weise, Sofien Bouaziz, Hao Li, and Mark Pauly. 2011. Realtime performance-based facial animation. In ACM transactions on graphics (TOG), Vol. 30. ACM.
40. Lance Williams. 1990. Performance-driven facial animation. In ACM SIGGRAPH Computer Graphics, Vol. 24. ACM, 235--242.
41. Eduard Zell, JP Lewis, Junyong Noh, Mario Botsch, et al. 2017. Facial retargeting with automatic range of motion alignment. ACM Transactions on Graphics (TOG) 36, 4 (2017), 154.

Index Terms

• Computing methodologies

  • Computer graphics

    • Animation