skip to main content
research-article

FaceBlit: Instant Real-time Example-based Style Transfer to Facial Videos

Authors Info & Claims
Published:28 April 2021Publication History
Skip Abstract Section

Abstract

We present FaceBlit---a system for real-time example-based face video stylization that retains textural details of the style in a semantically meaningful manner, i.e., strokes used to depict specific features in the style are present at the appropriate locations in the target image. As compared to previous techniques, our system preserves the identity of the target subject and runs in real-time without the need for large datasets nor lengthy training phase. To achieve this, we modify the existing face stylization pipeline of Fišer et al. [2017] so that it can quickly generate a set of guiding channels that handle identity preservation of the target subject while are still compatible with a faster variant of patch-based synthesis algorithm of Sýkora et al. [2019]. Thanks to these improvements we demonstrate a first face stylization pipeline that can instantly transfer artistic style from a single portrait to the target video at interactive rates even on mobile devices.

References

  1. Pierre Bénard, Forrester Cole, Michael Kass, Igor Mordatch, James Hegarty, Martin Sebastian Senn, Kurt Fleischer, Davide Pesare, and Katherine Breeden. 2013. Stylizing Animation By Example. ACM Transactions on Graphics 32, 4 (2013), 119.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Pierre Bénard, Ares Lagae, Peter Vangorp, Sylvain Lefebvre, George Drettakis, and Joëlle Thollot. 2010. A Dynamic Noise Primitive for Coherent Stylization. Computer Graphics Forum 29, 4 (2010), 1497--1506.Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. James F. Blinn and Martin E. Newell. 1976. Texture and Reflection in Computer Generated Images. Commun. ACM 19, 10 (1976), 542--547.Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Adrien Bousseau, Matthew Kaplan, Joëlle Thollot, and François X. Sillion. 2006. Interactive watercolor rendering with temporal coherence and abstraction. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering. 141--149.Google ScholarGoogle Scholar
  5. Adrien Bousseau, Fabrice Neyret, Joëlle Thollot, and David Salesin. 2007. Video watercolorization using bidirectional texture advection. ACM Transactions on Graphics 26, 3 (2007), 104.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Simon Breslav, Karol Szerszen, Lee Markosian, Pascal Barla, and Joëlle Thollot. 2007. Dynamic 2D patterns for shading 3D scenes. ACM Transactions on Graphics 26, 3 (2007), 20.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Cassidy J. Curtis, Sean E. Anderson, Joshua E. Seims, Kurt W. Fleischer, and David H. Salesin. 1997. Computer-generated watercolor. In SIGGRAPH Conference Proceedings. 421--430.Google ScholarGoogle Scholar
  8. Paul E. Debevec, Camillo J. Taylor, and Jitendra Malik. 1996. Modeling and Rendering Architecture from Photographs: A Hybrid Geometry- and Image-Based Approach. In SIGGRAPH Conference Proceedings. 11--20.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Jakub Fišer, Ondřej Jamriška, Michal Lukáč, Eli Shechtman, Paul Asente, Jingwan Lu, and Daniel Sýkora. 2016. StyLit: Illumination-Guided Example-Based Stylization of 3D Renderings. ACM Transactions on Graphics 35, 4 (2016), 92.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Jakub Fišer, Ondřej Jamriška, David Simons, Eli Shechtman, Jingwan Lu, Paul Asente, Michal Lukáč, and Daniel Sýkora. 2017. Example-Based Synthesis of Stylized Facial Animations. ACM Transactions on Graphics 36, 4 (2017), 155.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Jakub Fišer, Michal Lukáč, Ondřej Jamriška, Martin Čadík, Yotam Gingold, Paul Asente, and Daniel Sýkora. 2014. Color Me Noisy: Example-based Rendering of Hand-colored Animations with Temporal Noise Control. Computer Graphics Forum 33, 4 (2014), 1--10.Google ScholarGoogle ScholarCross RefCross Ref
  12. David Futschik, Menglei Chai, Chen Cao, Chongyang Ma, Aleksei Stoliar, Sergey Korolev, Sergey Tulyakov, Michal Kucěra, and Daniel Sýkora. 2019. Real-Time Patch-Based Stylization of Portraits Using Generative Adversarial Network. In Proceedings of the ACM/EG Expressive Symposium. 33--42.Google ScholarGoogle Scholar
  13. Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image Style Transfer Using Convolutional Neural Networks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2414--2423.Google ScholarGoogle ScholarCross RefCross Ref
  14. William Van Haevre, Tom Van Laerhoven, Fabian Di Fiore, and Frank Van Reeth. 2007. From Dust Till Drawn: A real-time bidirectional pastel simulation. The Visual Computer 23, 9--11 (2007), 925--934.Google ScholarGoogle Scholar
  15. Filip Hauptfleisch, Ondřej Texler, Aneta Texler, Jaroslav Křivánek, and Daniel Sýkora. 2020. StyleProp: Real-time Example-based Stylization of 3D Models. Computer Graphics Forum 39, 7 (2020), 575--586.Google ScholarGoogle ScholarCross RefCross Ref
  16. James Hays and Irfan A. Essa. 2004. Image and Video Based Painterly Animation. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering. 113--120.Google ScholarGoogle Scholar
  17. Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salesin. 2001. Image Analogies. In SIGGRAPH Conference Proceedings. 327--340.Google ScholarGoogle Scholar
  18. Xun Huang and Serge J. Belongie. 2017. Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization. Proceedings of IEEE International Conference on Computer Vision (2017), 1510--1519.Google ScholarGoogle Scholar
  19. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 5967--5976.Google ScholarGoogle Scholar
  20. Ondřej Jamriška, Jakub Fišer, Paul Asente, Jingwan Lu, Eli Shechtman, and Daniel Sýkora. 2015. LazyFluids: Appearance Transfer for Fluid Animations. ACM Transactions on Graphics 34, 4 (2015), 92.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Ondřej Jamriška, Šárka Sochorová, Ondřej Texler, Michal Lukáč, Jakub Fišer, Jingwan Lu, Eli Shechtman, and Daniel Sýkora. 2019. Stylizing Video by Example. ACM Transactions on Graphics 38, 4 (2019), 107.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In Proceedings of European Conference on Computer Vision. 694--711.Google ScholarGoogle ScholarCross RefCross Ref
  23. Tero Karras, Samuli Laine, and Timo Aila. 2019. A Style-Based Generator Architecture for Generative Adversarial Networks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 4401--4410.Google ScholarGoogle ScholarCross RefCross Ref
  24. Alexandre Kaspar, Boris Neubert, Dani Lischinski, Mark Pauly, and Johannes Kopf. 2015. Self Tuning Texture Optimization. Computer Graphics Forum 34, 2 (2015), 349--360.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Vahid Kazemi and Josephine Sullivan. 2014. One Millisecond Face Alignment with an Ensemble of Regression Trees. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 1867--1874.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Nicholas I. Kolkin, Jason Salavon, and Gregory Shakhnarovich. 2019. Style Transfer by Relaxed Optimal Transport and Self-Similarity. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 10051--10060.Google ScholarGoogle ScholarCross RefCross Ref
  27. Dmytro Kotovenko, Artsiom Sanakoyeu, Sabine Lang, and Björn Ommer. 2019a. Content and Style Disentanglement for Artistic Style Transfer. In Proceedings of IEEE International Conference on Computer Vision. 4421--4430.Google ScholarGoogle ScholarCross RefCross Ref
  28. Dmytro Kotovenko, Artsiom Sanakoyeu, Pingchuan Ma, Sabine Lang, and Björn Ommer. 2019b. A Content Transformation Block for Image Style Transfer. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 10032--10041.Google ScholarGoogle ScholarCross RefCross Ref
  29. Jan Eric Kyprianidis, John Collomosse, Tinghuai Wang, and Tobias Isenberg. 2013. State of the "Art": A Taxonomy of Artistic Stylization Techniques for Images and Video. IEEE Transactions on Visualization and Computer Graphics 19, 5 (2013), 866--885.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. 2020. MaskGAN: Towards Diverse and Interactive Facial Image Manipulation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 5548--5557.Google ScholarGoogle ScholarCross RefCross Ref
  31. Chuan Li and Michael Wand. 2016. Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2479--2486.Google ScholarGoogle ScholarCross RefCross Ref
  32. Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. 2017. Universal Style Transfer via Feature Transforms. In Advances in Neural Information Processing Systems. 385--395.Google ScholarGoogle Scholar
  33. Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, and Sing Bing Kang. 2017. Visual Attribute Transfer Through Deep Image Analogy. ACM Transactions on Graphics 36, 4 (2017), 120.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Peter Litwinowicz. 1997. Processing Images and Video for an Impressionist Effect. In SIGGRAPH. 407--414.Google ScholarGoogle Scholar
  35. Ming-Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. 2019. Few-Shot Unsupervised Image-to-Image Translation. In Proceedings of IEEE International Conference on Computer Vision. 10551--10560.Google ScholarGoogle ScholarCross RefCross Ref
  36. Cewu Lu, Li Xu, and Jiaya Jia. 2012. Combining sketch and tone for pencil drawing production. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering. 65--73.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Ming Lu, Hao Zhao, Anbang Yao, Feng Xu, Yurong Chen, and Xiang Lin. 2017. Decoder Network over Lightweight Reconstructed Feature for Fast Semantic Style Transfer. Proceedings of IEEE International Conference on Computer Vision (2017), 2488--2496.Google ScholarGoogle Scholar
  38. Santiago E Montesdeoca, Hock Soon Seah, Amir Semmo, Pierre Bénard, Romain Vergne, Joëlle Thollot, and Davide Benvenuti. 2018. MNPR: A Framework for Real-Time Expressive Non-Photorealistic Rendering of 3D Computer Graphics. In Proceedings of The Joint Symposium on Computational Aesthetics and Sketch Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering. 11.Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Emil Praun, Hugues Hoppe, Matthew Webb, and Adam Finkelstein. 2001. Real-Time Hatching. In SIGGRAPH. 581--586.Google ScholarGoogle Scholar
  40. Michael P. Salisbury, Michael T. Wong, John F. Hughes, and David H. Salesin. 1997. Orientable Textures for Image-based Pen-and-ink Illustration. In SIGGRAPH Conference Proceedings. 401--406.Google ScholarGoogle Scholar
  41. Scott Schaefer, Travis McPhail, and Joe Warren. 2006. Image Deformation Using Moving Least Squares. ACM Transactions on Graphics 25, 3 (2006), 533--540.Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Johannes Schmid, Martin Sebastian Senn, Markus Gross, and Robert W. Sumner. 2011. OverCoat: an implicit canvas for 3D painting. ACM Transactions on Graphics 30, 4 (2011), 28.Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Yi-Chang Shih, Sylvain Paris, Connelly Barnes, William T. Freeman, and Frédo Durand. 2014. Style Transfer for Headshot Portraits. ACM Transactions on Graphics 33, 4 (2014), 148.Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. 2019a. Animating Arbitrary Objects via Deep Motion Transfer. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2377--2386.Google ScholarGoogle ScholarCross RefCross Ref
  45. Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. 2019b. First Order Motion Model for Image Animation. In Advances in Neural Information Processing Systems. 7135--7145.Google ScholarGoogle Scholar
  46. Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR abs/1409.1556 (2014).Google ScholarGoogle Scholar
  47. Peter-Pike J. Sloan, William Martin, Amy Gooch, and Bruce Gooch. 2001. The Lit Sphere: A Model for Capturing NPR Shading from Art. In Proceedings of Graphics Interface. 143--150.Google ScholarGoogle Scholar
  48. Noah Snavely, C. Lawrence Zitnick, Sing Bing Kang, and Michael F. Cohen. 2006. Stylizing 2.5-D video. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering. 63--69.Google ScholarGoogle Scholar
  49. Daniel Sýkora, Ondřej Jamriška, Ondřej Texler, Jakub Fišer, Michal Lukáč, Jingwan Lu, and Eli Shechtman. 2019. StyleBlit: Fast Example-Based Stylization with Local Guidance. Computer Graphics Forum 38, 2 (2019), 83--91.Google ScholarGoogle ScholarCross RefCross Ref
  50. Ondřej Texler, David Futschik, Jakub Fišer, Michal Lukáč, Jingwan Lu, Eli Shechtman, and Daniel Sýkora. 2020a. Arbitrary Style Transfer Using Neurally-Guided Patch-Based Synthesis. Computers & Graphics 87 (2020), 62--71.Google ScholarGoogle Scholar
  51. Ondřej Texler, David Futschik, Michal Kučera, Ondřej Jamriška, Š. Sochorová, Menglei Chai, Sergey Tulyakov, and Daniel Sýkora. 2020b. Interactive Video Stylization Using Few-Shot Patch-Based Training. ACM Transactions on Graphics 39, 4 (2020), 73.Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor S. Lempitsky. 2016a. Texture Networks: Feed-Forward Synthesis of Textures and Stylized Images. In ICML, Vol. 48. 1349--1357.Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. 2016b. Instance Normalization: The Missing Ingredient for Fast Stylization. CoRR abs/1607.08022 (2016).Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. 2017. Improved Texture Networks: Maximizing Quality and Diversity in Feed-Forward Stylization and Texture Synthesis. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 4105--4113.Google ScholarGoogle Scholar
  55. Ting-Chun Wang, Ming-Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro. 2019. Few-shot Video-to-Video Synthesis. In Advances in Neural Information Processing Systems. 5014--5025.Google ScholarGoogle Scholar
  56. Xin Wang, Geoffrey Oxholm, Da Zhang, and Yuan-Fang Wang. 2017. Multimodal Transfer: A Hierarchical Deep Convolutional Neural Network for Fast Artistic Style Transfer. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 7178--7186.Google ScholarGoogle ScholarCross RefCross Ref
  57. Yonatan Wexler, Eli Shechtman, and Michal Irani. 2007. Space-Time Completion of Video. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 3 (2007), 463--476.Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Pierre Wilmot, Eric Risser, and Connelly Barnes. 2017. Stable and Controllable Neural Texture Synthesis and Style Transfer Using Histogram Losses. CoRR abs/1701.08893 (2017).Google ScholarGoogle Scholar
  59. Jordan Yaniv, Yael Newman, and Ariel Shamir. 2019. The Face of Art: Landmark detection and geometric style in portraits. ACM Transactions on Graphics 38, 4 (2019), 60.Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. Mingtian Zhao and Song-Chun Zhu. 2011. Portrait Painting Using Active Templates. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering. 117--124.Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. 2017a. Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. In Proceedings of IEEE International Conference on Computer Vision. 2242--2251.Google ScholarGoogle Scholar
  62. Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A. Efros, Oliver Wang, and Eli Shechtman. 2017b. Toward Multimodal Image-to-Image Translation. In Advances in Neural Information Processing Systems. 465--476.Google ScholarGoogle Scholar

Index Terms

  1. FaceBlit: Instant Real-time Example-based Style Transfer to Facial Videos

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image Proceedings of the ACM on Computer Graphics and Interactive Techniques
      Proceedings of the ACM on Computer Graphics and Interactive Techniques  Volume 4, Issue 1
      April 2021
      274 pages
      EISSN:2577-6193
      DOI:10.1145/3463840
      Issue’s Table of Contents

      Copyright © 2021 ACM

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 28 April 2021
      Published in pacmcgit Volume 4, Issue 1

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Research
      • Refereed
    • Article Metrics

      • Downloads (Last 12 months)45
      • Downloads (Last 6 weeks)7

      Other Metrics

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!