Jose Serra
ABSTRACT
We describe a hybrid pipeline that leverages: 1) video-driven animation transfer [Moser et al. 2021] for regressing high-quality animation under partially-controlled conditions from a single input image, and 2) a marker-based tracking approach [Moser et al. 2017] that, while more complex and slower, is capable of handling the most challenging scenarios seen in the capture set. By applying the most suited approach to each shot, we have an overall pipeline that, without loss of quality, is faster and has less user intervention. We also improve the prior work [Moser et al. 2021] with augmentations during training to make it more robust for the Head Mounted Camera (HMC) scenario. The new pipeline is currently being integrated into our offline and real-time workflows.
Supplemental Material
- DD. 2022. Digital Domain’s Masquerade Offline Capture. Retrieved Feb 20, 2022 from https://digitaldomain.com/technology/masquerade-offline-capture/Google Scholar
- DI4D. 2022. DI4D Pro. Retrieved Feb 20, 2022 from https://www.di4d.com/di4d-pro/Google Scholar
- Disney. 2022. Anyma. Retrieved Feb 20, 2022 from https://studios.disneyresearch.com/anyma/Google Scholar
- Martin Klaudiny, Steven McDonagh, Derek Bradley, Thabo Beeler, and Kenny Mitchell. 2017. Real-Time Multi-View Facial Capture with Synthetic Training. Computer Graphics Forum 36 (2017).Google Scholar
- Lucio Moser, Chinyu Chien, Mark Williams, Jose Serra, Darren Hendler, and Doug Roble. 2021. Semi-Supervised Video-Driven Facial Animation Transfer for Production. ACM Trans. Graph. 40, 6 (2021).Google ScholarDigital Library
- Lucio Moser, Darren Hendler, and Doug Roble. 2017. Masquerade: Fine-Scale Details for Head-Mounted Camera Motion Capture Data. In ACM SIGGRAPH 2017 Talks. New York, NY, USA, Article 18, 2 pages.Google Scholar
- Weta. 2022. Weta Digital - FACETS. Retrieved Feb 20, 2022 from https://www.wetafx.co.nz/research-and-tech/technology/facets/Google Scholar
- Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. In Computer Vision (ICCV), 2017 IEEE International Conference on.Google ScholarCross Ref
Recommendations
Artist friendly facial animation retargeting
This paper presents a novel facial animation retargeting system that is carefully designed to support the animator's workflow. Observation and analysis of the animators' often preferred process of key-frame animation with blendshape models informed our ...
Artist friendly facial animation retargeting
SA '11: Proceedings of the 2011 SIGGRAPH Asia ConferenceThis paper presents a novel facial animation retargeting system that is carefully designed to support the animator's workflow. Observation and analysis of the animators' often preferred process of key-frame animation with blendshape models informed our ...
Motion capture assisted animation: texturing and synthesis
SIGGRAPH '02: Proceedings of the 29th annual conference on Computer graphics and interactive techniquesWe discuss a method for creating animations that allows the animator to sketch an animation by setting a small number of keyframes on a fraction of the possible degrees of freedom. Motion capture data is then used to enhance the animation. Detail is ...
Comments