Abstract
Artificial data synthesis is currently a well-studied topic with useful applications in data science, computer vision, graphics, and many other fields. Generating realistic data is especially challenging, since human perception is highly sensitive to non-realistic appearance. In recent times, new levels of realism have been achieved by advances in GAN training procedures and architectures. These successful models, however, are tuned mostly for use with regularly sampled data such as images, audio, and video. Despite the successful application of the architecture on these types of media, applying the same tools to geometric data poses a far greater challenge. The study of geometric deep learning is still a debated issue within the academic community, as the lack of intrinsic parametrization inherent to geometric objects prohibits the direct use of convolutional filters, a main building block of today’s machine learning systems.
In this article, we propose a new method for generating realistic human facial geometries coupled with overlayed textures. We circumvent the parametrization issue by utilizing a specialized non-rigid alignment procedure, and imposing a global mapping from our data to the unit rectangle. This mapping enables the representation of our geometric data as regularly sampled 2D images. We further discuss how to design such a mapping to control the distortion and conserve area within the target image. By representing geometric textures and geometries as images, we are able to use advanced GAN methodologies to generate new plausible textures and geometries. We address the often-neglected topic of relationship between texture and geometry and propose different methods for fitting generated geometries to generated textures. In addition, we widen the scope of our discussion and offer a new method for training GAN models on partially corrupted data. Finally, we provide empirical evidence demonstrating our generative model’s ability to produce examples of new facial identities, independent from the training data, while maintaining a high level of realism—two traits that are often at odds.
- Brandon Amos, Bartosz Ludwiczuk, and Mahadev Satyanarayanan. 2016. OpenFace: A General-purpose Face Recognition Library with Mobile Applications. Technical Report. CMU-CS-16-118, CMU School of Computer Science.Google Scholar
- Anil Bas, Patrik Huber, William A. P. Smith, Muhammad Awais, and Josef Kittler. 2017. 3D morphable models as spatial transformer networks. In Proceedings of the ICCV Workshop on Geometry Meets Deep Learning. 904--912.Google Scholar
Cross Ref
- Volker Blanz and Thomas Vetter. 1999. A morphable model for the synthesis of 3D faces. In Proceedings of the 26th Conference on Computer Graphics and Interactive Techniques. ACM Press/Addison-Wesley Publishing Co., 187--194.Google Scholar
Digital Library
- Blender Online Community. 2017. Blender—A 3D Modelling and Rendering Package. Blender Foundation, Blender Institute, Amsterdam. Retrieved from: http://www.blender.org.Google Scholar
- James Booth, Epameinondas Antonakos, Stylianos Ploumpis, George Trigeorgis, Yannis Panagakis, Stefanos Zafeiriou et al. 2017. 3D face morphable models “in-the-wild.” In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition.Google Scholar
- James Booth, Anastasios Roussos, Allan Ponniah, David Dunaway, and Stefanos Zafeiriou. 2018. Large scale 3D morphable models. Int. J. Comput. Vis. 126, 2--4 (2018), 233--254.Google Scholar
Digital Library
- James Booth, Anastasios Roussos, Stefanos Zafeiriou, Allan Ponniah, and David Dunaway. 2016. A 3D morphable model learnt from 10,000 faces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5543--5552.Google Scholar
Cross Ref
- Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. 2017. Geometric deep learning: Going beyond Euclidean data. IEEE Sig. Proc. Mag. 34, 4 (2017), 18--42.Google Scholar
Cross Ref
- Baptiste Chu, Sami Romdhani, and Liming Chen. 2014. 3D-aided face recognition robust to expression and pose variations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1899--1906.Google Scholar
Digital Library
- Jiankang Deng, Shiyang Cheng, Niannan Xue, Yuxiang Zhou, and Stefanos Zafeiriou. 2018. UV-GAN: Adversarial facial UV map completion for pose-invariant face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18).Google Scholar
Cross Ref
- Michael S. Floater. 1997. Parametrization and smooth approximation of surface triangulations. Comput. Aided Geo. Des. 14, 3 (1997), 231--250.Google Scholar
Digital Library
- Baris Gecer, Binod Bhattarai, Josef Kittler, and Tae-Kyun Kim. 2018. Semi-supervised adversarial learning to generate photorealistic face images of new identities from 3D morphable model. In Proceedings of the European Conference on Computer Vision (ECCV’18).Google Scholar
Cross Ref
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proceedings of the Conference on Advances in Neural Information Processing Systems. 2672--2680.Google Scholar
Digital Library
- Harold Hotelling. 1936. Relations between two sets of variates. Biometrika 28, 3/4 (1936), 321--377.Google Scholar
Cross Ref
- Rui Huang, Shu Zhang, Tianyu Li, and Ran He. 2017. Beyond face rotation: Global and local perception GAN for photorealistic and identity preserving frontal view synthesis. In Proceedings of the IEEE International Conference on Computer Vision. 2439--2448.Google Scholar
Cross Ref
- Ian T. Jolliffe. 1986. Principal component analysis and factor analysis. In Principal Component Analysis. Springer, 115--128.Google Scholar
- Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2017. Progressive growing of GANs for improved quality, stability, and variation. In Proceedings of the International Conference on Learning Representations (ICLR’17).Google Scholar
- Davis E. King. 2009. Dlib-ml: A machine learning toolkit. J. Mach. Learn. Res. 10 (2009), 1755--1758.Google Scholar
Digital Library
- Or Litany, Alex Bronstein, Michael Bronstein, and Ameesh Makadia. 2018. Deformable shape completion with graph convolutional autoencoders. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18).Google Scholar
Cross Ref
- Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, Nov. (2008), 2579--2605.Google Scholar
- Iacopo Masi, Anh Tuážěn Tražǧn, Tal Hassner, Jatuporn Toy Leksut, and Gérard Medioni. 2016. Do we really need to collect millions of faces for effective face recognition? In Proceedings of the European Conference on Computer Vision. Springer, 579--596.Google Scholar
Cross Ref
- Julien Rabin, Gabriel Peyré, Julie Delon, and Marc Bernot. 2011. Wasserstein barycenter and its application to texture mixing. In Proceedings of the International Conference on Scale Space and Variational Methods in Computer Vision. Springer, 435--446.Google Scholar
- Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J. Black. 2018. Generating 3D faces using convolutional mesh autoencoders. In Proceedings of the European Conference on Computer Vision. Springer, 725--741.Google Scholar
- Elad Richardson, Matan Sela, and Ron Kimmel. 2016. 3D face reconstruction by learning from synthetic data. In Proceedings of the 4th International Conference on 3D Vision (3DV’16). IEEE, 460--469.Google Scholar
Cross Ref
- Elad Richardson, Matan Sela, Roy Or-El, and Ron Kimmel. 2017. Learning detailed face reconstruction from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). IEEE, 5553--5562.Google Scholar
Cross Ref
- Epameinondas Antonakos, Patrick Snape, Stefanos Zafeiriou, Iasonas Kokkinos, Riza Alp Güler, George Trigeorgis. 2016. DenseReg: Fully convolutional dense shape regression in-the-wild. Retrieved from: Arxiv:1612.01202 (2016).Google Scholar
- Shunsuke Saito, Lingyu Wei, Liwen Hu, Koki Nagano, and Hao Li. 2017. Photorealistic facial texture inference using deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17), Vol. 3.Google Scholar
Cross Ref
- Matan Sela, Elad Richardson, and Ron Kimmel. 2017. Unrestricted facial geometry reconstruction using image-to-image translation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17). IEEE, 1585--1594.Google Scholar
Cross Ref
- Gil Shamai, Michael Zibulevsky, and Ron Kimmel. 2018. Efficient inter-geodesic distance computation and fast classical scaling. IEEE Trans. Pattern Anal. Mach. Intell. (2018). https://ieeexplore.ieee.org/document/8509134/media#media.Google Scholar
- Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, and Russ Webb. 2017. Learning from simulated and unsupervised images through adversarial training. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17), Vol. 3. 6.Google Scholar
Cross Ref
- Ron Slossberg, Gil Shamai, and Ron Kimmel. 2018. High quality facial surface and texture synthesis via generative adversarial networks. Retrieved from: Arxiv Preprint Arxiv:1808.08281 (2018).Google Scholar
- Ayush Tewari, Michael Zollhöfer, Pablo Garrido, Florian Bernard, Hyeongwoo Kim, Patrick Pérez, and Christian Theobalt. 2018. Self-supervised multi-level face model learning for monocular reconstruction at over 250Hz. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18).Google Scholar
Cross Ref
- Anh Tuan Tran, Tal Hassner, Iacopo Masi, and Gérard Medioni. 2017. Regressing robust and discriminative 3D morphable models with a very deep neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). IEEE, 1493--1502.Google Scholar
Cross Ref
- Luan Tran and Xiaoming Liu. 2018. Nonlinear 3D face morphable model. Retrieved from: Arxiv Preprint Arxiv:1804.03786 (2018).Google Scholar
- Luan Tran, Xi Yin, and Xiaoming Liu. 2017. Disentangled representation learning GAN for pose-invariant face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1415--1424.Google Scholar
Cross Ref
- Thibaut Weise, Hao Li, Luc Van Gool, and Mark Pauly. 2009. Face/off: Live facial puppetry. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. ACM, 7--16.Google Scholar
Digital Library
- Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. 2016. Learning a probabilistic latent space of object shapes via 3D generative-adversarial modeling. In Proceedings of the Conference on Advances in Neural Information Processing Systems. 82--90.Google Scholar
Index Terms
Synthesizing Facial Photometries and Corresponding Geometries Using Generative Adversarial Networks
Recommendations
3D-Aided Face Recognition Robust to Expression and Pose Variations
CVPR '14: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern RecognitionExpression and pose variations are major challenges for reliable face recognition (FR) in 2D. In this paper, we aim to endow state of the art face recognition SDKs with robustness to facial expression variations and pose changes by using an extended 3D ...
Geometry Guided Adversarial Facial Expression Synthesis
MM '18: Proceedings of the 26th ACM international conference on MultimediaFacial expression synthesis has drawn much attention in the field of computer graphics and pattern recognition. It has been widely used in face animation and recognition. However, it is still challenging due to the high-level semantic presence of large ...
Learning inter-class optical flow difference using generative adversarial networks for facial expression recognition
AbstractFacial expression recognition is a fine-grained task because different emotions have subtle facial movements. This paper proposes to learn inter-class optical flow difference using generative adversarial networks (GANs) for facial expression ...






Comments