skip to main content
research-article

Synthesizing Facial Photometries and Corresponding Geometries Using Generative Adversarial Networks

Published:15 October 2019Publication History
Skip Abstract Section

Abstract

Artificial data synthesis is currently a well-studied topic with useful applications in data science, computer vision, graphics, and many other fields. Generating realistic data is especially challenging, since human perception is highly sensitive to non-realistic appearance. In recent times, new levels of realism have been achieved by advances in GAN training procedures and architectures. These successful models, however, are tuned mostly for use with regularly sampled data such as images, audio, and video. Despite the successful application of the architecture on these types of media, applying the same tools to geometric data poses a far greater challenge. The study of geometric deep learning is still a debated issue within the academic community, as the lack of intrinsic parametrization inherent to geometric objects prohibits the direct use of convolutional filters, a main building block of today’s machine learning systems.

In this article, we propose a new method for generating realistic human facial geometries coupled with overlayed textures. We circumvent the parametrization issue by utilizing a specialized non-rigid alignment procedure, and imposing a global mapping from our data to the unit rectangle. This mapping enables the representation of our geometric data as regularly sampled 2D images. We further discuss how to design such a mapping to control the distortion and conserve area within the target image. By representing geometric textures and geometries as images, we are able to use advanced GAN methodologies to generate new plausible textures and geometries. We address the often-neglected topic of relationship between texture and geometry and propose different methods for fitting generated geometries to generated textures. In addition, we widen the scope of our discussion and offer a new method for training GAN models on partially corrupted data. Finally, we provide empirical evidence demonstrating our generative model’s ability to produce examples of new facial identities, independent from the training data, while maintaining a high level of realism—two traits that are often at odds.

References

  1. Brandon Amos, Bartosz Ludwiczuk, and Mahadev Satyanarayanan. 2016. OpenFace: A General-purpose Face Recognition Library with Mobile Applications. Technical Report. CMU-CS-16-118, CMU School of Computer Science.Google ScholarGoogle Scholar
  2. Anil Bas, Patrik Huber, William A. P. Smith, Muhammad Awais, and Josef Kittler. 2017. 3D morphable models as spatial transformer networks. In Proceedings of the ICCV Workshop on Geometry Meets Deep Learning. 904--912.Google ScholarGoogle ScholarCross RefCross Ref
  3. Volker Blanz and Thomas Vetter. 1999. A morphable model for the synthesis of 3D faces. In Proceedings of the 26th Conference on Computer Graphics and Interactive Techniques. ACM Press/Addison-Wesley Publishing Co., 187--194.Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Blender Online Community. 2017. Blender—A 3D Modelling and Rendering Package. Blender Foundation, Blender Institute, Amsterdam. Retrieved from: http://www.blender.org.Google ScholarGoogle Scholar
  5. James Booth, Epameinondas Antonakos, Stylianos Ploumpis, George Trigeorgis, Yannis Panagakis, Stefanos Zafeiriou et al. 2017. 3D face morphable models “in-the-wild.” In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition.Google ScholarGoogle Scholar
  6. James Booth, Anastasios Roussos, Allan Ponniah, David Dunaway, and Stefanos Zafeiriou. 2018. Large scale 3D morphable models. Int. J. Comput. Vis. 126, 2--4 (2018), 233--254.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. James Booth, Anastasios Roussos, Stefanos Zafeiriou, Allan Ponniah, and David Dunaway. 2016. A 3D morphable model learnt from 10,000 faces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5543--5552.Google ScholarGoogle ScholarCross RefCross Ref
  8. Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. 2017. Geometric deep learning: Going beyond Euclidean data. IEEE Sig. Proc. Mag. 34, 4 (2017), 18--42.Google ScholarGoogle ScholarCross RefCross Ref
  9. Baptiste Chu, Sami Romdhani, and Liming Chen. 2014. 3D-aided face recognition robust to expression and pose variations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1899--1906.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Jiankang Deng, Shiyang Cheng, Niannan Xue, Yuxiang Zhou, and Stefanos Zafeiriou. 2018. UV-GAN: Adversarial facial UV map completion for pose-invariant face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18).Google ScholarGoogle ScholarCross RefCross Ref
  11. Michael S. Floater. 1997. Parametrization and smooth approximation of surface triangulations. Comput. Aided Geo. Des. 14, 3 (1997), 231--250.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Baris Gecer, Binod Bhattarai, Josef Kittler, and Tae-Kyun Kim. 2018. Semi-supervised adversarial learning to generate photorealistic face images of new identities from 3D morphable model. In Proceedings of the European Conference on Computer Vision (ECCV’18).Google ScholarGoogle ScholarCross RefCross Ref
  13. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proceedings of the Conference on Advances in Neural Information Processing Systems. 2672--2680.Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Harold Hotelling. 1936. Relations between two sets of variates. Biometrika 28, 3/4 (1936), 321--377.Google ScholarGoogle ScholarCross RefCross Ref
  15. Rui Huang, Shu Zhang, Tianyu Li, and Ran He. 2017. Beyond face rotation: Global and local perception GAN for photorealistic and identity preserving frontal view synthesis. In Proceedings of the IEEE International Conference on Computer Vision. 2439--2448.Google ScholarGoogle ScholarCross RefCross Ref
  16. Ian T. Jolliffe. 1986. Principal component analysis and factor analysis. In Principal Component Analysis. Springer, 115--128.Google ScholarGoogle Scholar
  17. Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2017. Progressive growing of GANs for improved quality, stability, and variation. In Proceedings of the International Conference on Learning Representations (ICLR’17).Google ScholarGoogle Scholar
  18. Davis E. King. 2009. Dlib-ml: A machine learning toolkit. J. Mach. Learn. Res. 10 (2009), 1755--1758.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Or Litany, Alex Bronstein, Michael Bronstein, and Ameesh Makadia. 2018. Deformable shape completion with graph convolutional autoencoders. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18).Google ScholarGoogle ScholarCross RefCross Ref
  20. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, Nov. (2008), 2579--2605.Google ScholarGoogle Scholar
  21. Iacopo Masi, Anh Tuážěn Tražǧn, Tal Hassner, Jatuporn Toy Leksut, and Gérard Medioni. 2016. Do we really need to collect millions of faces for effective face recognition? In Proceedings of the European Conference on Computer Vision. Springer, 579--596.Google ScholarGoogle ScholarCross RefCross Ref
  22. Julien Rabin, Gabriel Peyré, Julie Delon, and Marc Bernot. 2011. Wasserstein barycenter and its application to texture mixing. In Proceedings of the International Conference on Scale Space and Variational Methods in Computer Vision. Springer, 435--446.Google ScholarGoogle Scholar
  23. Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J. Black. 2018. Generating 3D faces using convolutional mesh autoencoders. In Proceedings of the European Conference on Computer Vision. Springer, 725--741.Google ScholarGoogle Scholar
  24. Elad Richardson, Matan Sela, and Ron Kimmel. 2016. 3D face reconstruction by learning from synthetic data. In Proceedings of the 4th International Conference on 3D Vision (3DV’16). IEEE, 460--469.Google ScholarGoogle ScholarCross RefCross Ref
  25. Elad Richardson, Matan Sela, Roy Or-El, and Ron Kimmel. 2017. Learning detailed face reconstruction from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). IEEE, 5553--5562.Google ScholarGoogle ScholarCross RefCross Ref
  26. Epameinondas Antonakos, Patrick Snape, Stefanos Zafeiriou, Iasonas Kokkinos, Riza Alp Güler, George Trigeorgis. 2016. DenseReg: Fully convolutional dense shape regression in-the-wild. Retrieved from: Arxiv:1612.01202 (2016).Google ScholarGoogle Scholar
  27. Shunsuke Saito, Lingyu Wei, Liwen Hu, Koki Nagano, and Hao Li. 2017. Photorealistic facial texture inference using deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17), Vol. 3.Google ScholarGoogle ScholarCross RefCross Ref
  28. Matan Sela, Elad Richardson, and Ron Kimmel. 2017. Unrestricted facial geometry reconstruction using image-to-image translation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17). IEEE, 1585--1594.Google ScholarGoogle ScholarCross RefCross Ref
  29. Gil Shamai, Michael Zibulevsky, and Ron Kimmel. 2018. Efficient inter-geodesic distance computation and fast classical scaling. IEEE Trans. Pattern Anal. Mach. Intell. (2018). https://ieeexplore.ieee.org/document/8509134/media#media.Google ScholarGoogle Scholar
  30. Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, and Russ Webb. 2017. Learning from simulated and unsupervised images through adversarial training. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17), Vol. 3. 6.Google ScholarGoogle ScholarCross RefCross Ref
  31. Ron Slossberg, Gil Shamai, and Ron Kimmel. 2018. High quality facial surface and texture synthesis via generative adversarial networks. Retrieved from: Arxiv Preprint Arxiv:1808.08281 (2018).Google ScholarGoogle Scholar
  32. Ayush Tewari, Michael Zollhöfer, Pablo Garrido, Florian Bernard, Hyeongwoo Kim, Patrick Pérez, and Christian Theobalt. 2018. Self-supervised multi-level face model learning for monocular reconstruction at over 250Hz. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18).Google ScholarGoogle ScholarCross RefCross Ref
  33. Anh Tuan Tran, Tal Hassner, Iacopo Masi, and Gérard Medioni. 2017. Regressing robust and discriminative 3D morphable models with a very deep neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). IEEE, 1493--1502.Google ScholarGoogle ScholarCross RefCross Ref
  34. Luan Tran and Xiaoming Liu. 2018. Nonlinear 3D face morphable model. Retrieved from: Arxiv Preprint Arxiv:1804.03786 (2018).Google ScholarGoogle Scholar
  35. Luan Tran, Xi Yin, and Xiaoming Liu. 2017. Disentangled representation learning GAN for pose-invariant face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1415--1424.Google ScholarGoogle ScholarCross RefCross Ref
  36. Thibaut Weise, Hao Li, Luc Van Gool, and Mark Pauly. 2009. Face/off: Live facial puppetry. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. ACM, 7--16.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. 2016. Learning a probabilistic latent space of object shapes via 3D generative-adversarial modeling. In Proceedings of the Conference on Advances in Neural Information Processing Systems. 82--90.Google ScholarGoogle Scholar

Index Terms

  1. Synthesizing Facial Photometries and Corresponding Geometries Using Generative Adversarial Networks

            Recommendations

            Comments

            Login options

            Check if you have access through your login credentials or your institution to get full access on this article.

            Sign in

            Full Access

            PDF Format

            View or Download as a PDF file.

            PDF

            eReader

            View online with eReader.

            eReader

            HTML Format

            View this article in HTML Format .

            View HTML Format
            About Cookies On This Site

            We use cookies to ensure that we give you the best experience on our website.

            Learn more

            Got it!