Stroke Transfer: Example-based Synthesis of Animatable Stroke Styles

ABSTRACT

We present stroke transfer, an example-based synthesis method of brushstrokes for animated scenes under changes in viewpoint, lighting conditions, and object shapes. We introduce stroke field for guiding the generation of strokes, consisting of spatially varying attributes of strokes, namely, their orientations, lengths, widths, and colors. Strokes are synthesized as the integral curves of the stroke field. In essence, we separate elements that constitute the artistic stroke into style-specific transferable elements and instance-intrinsic ones. To generate the stroke field, we first compute a set of vector fields that reflect the instance-intrinsic elements and then combine them using style-specific weight functions learned from exemplars, with the weights computed in a proxy feature space shared among a variety of objects. The rendered animation using our method captures time-varying viewpoint, lighting conditions, and object shapes, as well as the artistic style given in the form of exemplars.

References

1. Katie Bassett, Ilya Baran, Johannes Schmid, Markus Gross, and Robert W. Sumner. 2013. Authoring and Animating Painterly Characters. ACM Transactions on Graphics 32, 5, Article 156 (oct 2013), 12 pages. https://doi.org/10.1145/2484238
2. Bill Baxter, Vincent Scheib, Ming C. Lin, and Dinesh Manocha. 2001. DAB: Interactive Haptic Painting with 3D Virtual Brushes. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques(SIGGRAPH ’01). Association for Computing Machinery, 461––468. https://doi.org/10.1145/383259.383313
3. Pierre Bénard, Forrester Cole, Michael Kass, Igor Mordatch, James Hegarty, Martin Sebastian Senn, Kurt Fleischer, Davide Pesare, and Katherine Breeden. 2013. Stylizing Animation by Example. ACM Transactions on Graphics 32, 4 (Proc. of SIGGRAPH 2013), Article 119 (jul 2013), 12 pages. https://doi.org/10.1145/2461912.2461929
4. John Bowers, Rui Wang, Li-Yi Wei, and David Maletz. 2010. Parallel Poisson Disk Sampling with Spectrum Analysis on Surfaces. ACM Transactions on Graphics 29, 6 (Proc. of SIGGRAPH ASIA 2010), Article 166 (dec 2010), 10 pages. https://doi.org/10.1145/1882261.1866188
5. Luis Cardona and Suguru Saito. 2015. Hybrid-Space Localized Stylization Method for View-Dependent Lines Extracted from 3D Models. In Proceedings of the Workshop on Non-Photorealistic Animation and Rendering(NPAR ’15). Eurographics Association, 79––89.
6. Guoning Chen, Vivek Kwatra, Li-Yi Wei, Charles D Hansen, and Eugene Zhang. 2011. Design of 2D Time-Varying Vector Fields. IEEE Transactions on Visualization and Computer Graphics 18, 10(2011), 1717–1730.
7. Zhili Chen, Byungmoon Kim, Daichi Ito, and Huamin Wang. 2015. Wetbrush: GPU-Based 3D Painting Simulation at the Bristle Level. ACM Transactions on Graphics 34, 6 (Proc. of SIGGRAPH ASIA 2015), Article 200 (oct 2015), 11 pages. https://doi.org/10.1145/2816795.2818066
8. Nelson S.-H. Chu and Chiew-Lan Tai. 2005. MoXi: Real-Time Ink Dispersion in Absorbent Paper. ACM Transactions on Graphics 24, 3 (Proc. of SIGGRAPH 2005) (jul 2005), 504––511. https://doi.org/10.1145/1073204.1073221
9. Keenan Crane, Fernando de Goes, Mathieu Desbrun, and Peter Schröder. 2013. Digital Geometry Processing with Discrete Exterior Calculus. In ACM SIGGRAPH 2013 courses (Anaheim, California) (SIGGRAPH ’13). ACM, New York, NY, USA, 126 pages.
10. Cassidy J. Curtis, Sean E. Anderson, Joshua E. Seims, Kurt W. Fleischer, and David H. Salesin. 1997. Computer-Generated Watercolor. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques(SIGGRAPH ’97). ACM Press/Addison-Wesley Publishing Co., 421––430. https://doi.org/10.1145/258734.258896
11. Doug DeCarlo, Adam Finkelstein, Szymon Rusinkiewicz, and Anthony Santella. 2003. Suggestive Contours for Conveying Shape. ACM Transactions on Graphics 22, 3 (Proc. of SIGGRAPH 2003) (jul 2003), 848––855. https://doi.org/10.1145/882262.882354
12. Mathieu Desbrun, Eva Kanso, and Yiying Tong. 2006. Discrete Differential Forms for Computational Modeling. In ACM SIGGRAPH 2006 Courses (Boston, Massachusetts) (SIGGRAPH ’06). Association for Computing Machinery, New York, NY, USA, 39––54. https://doi.org/10.1145/1185657.1185665
13. Frédo Durand. 2002. An Invitation to Discuss Computer Depiction. In Proceedings of the 2nd International Symposium on Non-Photorealistic Animation and Rendering (Annecy, France) (NPAR ’02). Association for Computing Machinery, New York, NY, USA, 111––124. https://doi.org/10.1145/508530.508550
14. Matthew Fisher, Peter Schröder, Mathieu Desbrun, and Hugues Hoppe. 2007. Design of Tangent Vector Fields. ACM Transactions on Graphics 26, 3 (Proc. of SIGGRAPH 2007) (jul 2007), 56––65. https://doi.org/10.1145/1276377.1276447
15. Jakub Fišer, Ondřej Jamriška, Michal Lukáč, Eli Shechtman, Paul Asente, Jingwan Lu, and Daniel Sýkora. 2016. StyLit: Illumination-Guided Example-Based Stylization of 3D Renderings. ACM Transactions on Graphics 35, 4 (Proc. of SIGGRAPH 2016), Article 92 (jul 2016), 11 pages. https://doi.org/10.1145/2897824.2925948
16. Hongbo Fu, Yichen Wei, Chiew-Lan Tai, and Long Quan. 2007. Sketching Hairstyles. In Proceedings of the 4th Eurographics Workshop on Sketch-Based Interfaces and Modeling(SBIM ’07). Association for Computing Machinery, New York, NY, USA, 31––36. https://doi.org/10.1145/1384429.1384439
17. David Futschik, Michal Kučera, Michal Lukáč, Zhaowen Wang, Eli Shechtman, and Daniel Sýkora. 2021. STALP: Style Transfer with Auxiliary Limited Pairing. Computer Graphics Forum 40, 2 (Proc. of EUROGRAPHICS 2021)(2021), 563–573.
18. Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image Style Transfer Using Convolutional Neural Networks. (2016), 2414–2423. https://doi.org/10.1109/CVPR.2016.265
19. T. Haga, Henry Johan, and Tomoyuki Nishita. 2001. Animation Method for Pen-and-Ink Illustrations Using Stroke Coherency. In Proc. of CAD & Graphics 2001. 333–343.
20. Filip Hauptfleisch, Ondřej Texler, Aneta Texler, Jaroslav Křivánek, and Daniel Sýkora. 2020. StyleProp: Real-Time Example-Based Stylization of 3D Models. Computer Graphics Forum 39, 7 (Proc. of Pacific Graphics 20+21)(2020), 575–586.
21. Siddharth Hegde, Christos Gatzidis, and Feng Tian. 2013. Painterly Rendering Techniques: A State-of-The-Art Review of Current Approaches. Computer Animation and Virtual Worlds 24, 1 (2013), 43–64. https://doi.org/10.1002/cav.1435
22. Aaron Hertzmann. 2003. Tutorial: A Survey of Stroke-Based Rendering. IEEE Computer Graphics and Applications 23, 4 (jul 2003), 70––81. https://doi.org/10.1109/MCG.2003.1210867
23. Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salesin. 2001. Image Analogies. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques(SIGGRAPH ’01). Association for Computing Machinery, New York, NY, USA, 327––340. https://doi.org/10.1145/383259.383295
24. Anil Nirmal Hirani. 2003. Discrete Exterior Calculus. Ph. D. Dissertation. USA. Advisor(s) Marsden, Jerrold E. https://doi.org/10.7907/ZHY8-V329
25. Jin Huang, Zherong Pan, Guoning Chen, Wei Chen, and Hujun Bao. 2013. Image-Space Texture-Based Output-Coherent Surface Flow Visualization. IEEE Transactions on Visualization and Computer Graphics 19, 9(2013), 1476–1487. https://doi.org/10.1109/TVCG.2013.62
26. Emmanuel Iarussi, David Bommes, and Adrien Bousseau. 2015. BendFields: Regularized Curvature Fields from Rough Concept Sketches. ACM Transactions on Graphics 34, 3, Article 24 (apr 2015), 16 pages. https://doi.org/10.1145/2710026
27. Wenzel Jakob, Marco Tarini, Daniele Panozzo, and Olga Sorkine-Hornung. 2015. Instant Field-Aligned Meshes. ACM Transactions on Graphics 34, 6 (Proc. of SIGGRAPH ASIA 2015), Article 189 (nov 2015), 15 pages. https://doi.org/10.1145/2816795.2818078
28. Ondřej Jamriška, Šárka Sochorová, Ondřej Texler, Michal Lukáč, Jakub Fišer, Jingwan Lu, Eli Shechtman, and Daniel Sýkora. 2019. Stylizing Video by Example. ACM Transactions on Graphics 38, 4 (Proc. of SIGGRAPH 2019), Article 107 (aug 2019), 11 pages. https://doi.org/10.1145/3306346.3323006
29. Mizuki Kagaya, William Brendel, Qingqing Deng, Todd Kesterson, Sinisa Todorovic, Patrick J. Neill, and Eugene Zhang. 2011. Video Painting with Space-Time-Varying Style Parameters. IEEE Transactions on Visualization and Computer Graphics 17, 1(2011), 74–87. https://doi.org/10.1109/TVCG.2010.25
30. Robert D. Kalnins, Philip L. Davidson, Lee Markosian, and Adam Finkelstein. 2003. Coherent Stylized Silhouettes. ACM Transactions on Graphics 22, 3 (Proc. of SIGGRAPH 2003) (jul 2003), 856––861. https://doi.org/10.1145/882262.882355
31. Robert D. Kalnins, Lee Markosian, Barbara J. Meier, Michael A. Kowalski, Joseph C. Lee, Philip L. Davidson, Matthew Webb, John F. Hughes, and Adam Finkelstein. 2002. WYSIWYG NPR: Drawing Strokes Directly on 3D Models. ACM Transactions on Graphics 21, 3 (Proc. of SIGGRAPH 2002) (jul 2002), 755––762. https://doi.org/10.1145/566654.566648
32. Evangelos Kalogerakis, Derek Nowrouzezahrai, Simon Breslav, and Aaron Hertzmann. 2012. Learning Hatching for Pen-and-Ink Illustration of Surfaces. ACM Trans. Graph. 31, 1, Article 1 (jan 2012), 17 pages. https://doi.org/10.1145/2077341.2077342
33. George Katanics and Tasso Lappa. 2003. Deep Canvas: Integrating 3D Painting and Painterly Rendering. In Theory and Practice of Non-Photorealistic Graphics: Algorithms, Methods, and Production Systems(ACM SIGGRAPH 2003 Course Notes).
34. Dorota Kobiela and Hugh Welchman. 2017. Loving Vincent. BreakThru Productions, Trademark Films.
35. Michael A. Kowalski, Lee Markosian, J. D. Northrup, Lubomir Bourdev, Ronen Barzel, Loring S. Holden, and John F. Hughes. 1999. Art-Based Rendering of Fur, Grass, and Trees. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques(SIGGRAPH ’99). ACM Press/Addison-Wesley Publishing Co., USA, 433––438. https://doi.org/10.1145/311535.311607
36. Julian Kratt, Ferdinand Eisenkeil, Marc Spicker, Yunhai Wang, Daniel Weiskopf, and Oliver Deussen. 2017. Structure-Aware Stylization of Mountainous Terrains. In Proceedings of the Conference on Vision, Modeling and Visualization (Bonn, Germany) (VMV ’17). Eurographics Association, 17––27. https://doi.org/10.2312/vmv.20171255
37. Paul Kubelka and Franz Munk. 1931. Ein Beitrag zur Optik der Farbanstriche. Zeitschrift für Technishen Physik 12 (1931), 593––601.
38. Bei-Bei Liu, Yan-Lin Weng, Jian-Nan Wang, and Yi-Ying Tong. 2013. Orientation Field Guided Texture Synthesis. Journal of Computer Science and Technology 28, 5 (2013), 827–835.
39. Jingwan Lu, Connelly Barnes, Stephen DiVerdi, and Adam Finkelstein. 2013. RealBrush: Painting with Examples of Physical Media. ACM Transactions on Graphics 32, 4 (Proc. of SIGGRAPH 2013), Article 117 (jul 2013), 12 pages. https://doi.org/10.1145/2461912.2461998
40. Lukasz Mackiewicz and Francho Melendez. 2016. Loving Vincent: Guiding Painters through 64.000 Frames. In ACM SIGGRAPH 2016 Talks(SIGGRAPH ’16). Association for Computing Machinery, Article 6, 2 pages. https://doi.org/10.1145/2897839.2927394
41. Lee Markosian, Barbara J. Meier, Michael A. Kowalski, Loring S. Holden, J. D. Northrup, and John F. Hughes. 2000. Art-Based Rendering with Continuous Levels of Detail. In Proceedings of the 1st International Symposium on Non-Photorealistic Animation and Rendering(NPAR ’00). Association for Computing Machinery, 59––66. https://doi.org/10.1145/340916.340924
42. Barbara J. Meier. 1996. Painterly Rendering for Animation. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques(SIGGRAPH ’96). Association for Computing Machinery, 477––484. https://doi.org/10.1145/237170.237288
43. J. D. Northrup and Lee Markosian. 2000. Artistic Silhouettes: A Hybrid Approach. In Proceedings of the 1st International Symposium on Non-Photorealistic Animation and Rendering(NPAR ’00). Association for Computing Machinery, 31––37. https://doi.org/10.1145/340916.340920
44. Nicolas Ray and Dmitry Sokolov. 2014. Robust Polylines Tracing for N-Symmetry Direction Field on Triangulated Surfaces. ACM Transactions on Graphics 33, 3, Article 30 (may 2014), 11 pages. https://doi.org/10.1145/2602145
45. Dave Rudolf, David Mould, and Eric Neufeld. 2003. Simulating Wax Crayons. In Proceedings of the 11th Pacific Conference on Computer Graphics and Applications(PG ’03). IEEE Computer Society, 163–172. https://doi.org/10.5555/946250.946956
46. Johannes Schmid, Martin Sebastian Senn, Markus Gross, and Robert W. Sumner. 2011. OverCoat: An Implicit Canvas for 3D Painting. ACM Transactions on Graphics 30, 4 (Proc. of SIGGRAPH 2011), Article 28 (jul 2011), 10 pages. https://doi.org/10.1145/2010324.1964923
47. Mayank Singh and Scott Schaefer. 2010. Suggestive Hatching. In Proceedings of the Sixth International Conference on Computational Aesthetics in Graphics, Visualization and Imaging (London, United Kingdom) (Computational Aesthetics’10). Eurographics Association, Goslar, DEU, 25––32.
48. Peter-Pike J. Sloan, William Martin, Amy Ashurst Gooch, and Bruce Gooch. 2001. The Lit Sphere: A Model for Capturing NPR Shading from Art. In Proceedings of Graphics Interface 2001 (Ottawa, Ontario, Canada) (GI ’01). Canadian Information Processing Society, CAN, 143––150.
49. Benjamin Spencer, Robert S. Laramee, Guoning Chen, and Eugene Zhang. 2009. Evenly Spaced Streamlines for Surfaces: An Image-Based Approach. Computer Graphics Forum 28, 6 (2009), 1618–1631. https://doi.org/10.1111/j.1467-8659.2009.01352.x
50. Daniel Sýkora, Ondřej Jamriška, Ondřej Texler, Jakub Fišer, Michal Lukáč, Jingwan Lu, and Eli Shechtman. 2019. StyleBlit: Fast Example-Based Stylization with Local Guidance. Computer Graphics Forum 38, 2 (Proc. of EUGRAPHICS 2019)(2019), 83–91.
51. Isao Takahata. 2014. The Tale of The Princess Kaguya. Studio Ghibli.
52. Jianchao Tan, Marek Dvorožňák, Daniel Sýkora, and Yotam Gingold. 2015. Decomposing Time-Lapse Paintings into Layers. ACM Transactions on Graphics 34, 4 (Proc. of SIGGRAPH 2015), Article 61 (aug 2015), 10 pages. https://doi.org/10.1145/2766960
53. Ondřej Texler, David Futschik, Michal kučera, Ondřej jamriška, Šárka Sochorová, Menclei Chai, Sergey Tulyakov, and Daniel SÝkora. 2020. Interactive Video Stylization Using Few-Shot Patch-Based Training. ACM Transactions on Graphics 39, 4 (Proc. of SIGGRAPH 2020), Article 73 (jul 2020), 11 pages. https://doi.org/10.1145/3386569.3392453
54. Greg Turk. 2001. Texture Synthesis on Surfaces. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques(SIGGRAPH ’01). Association for Computing Machinery, New York, NY, USA, 347––354. https://doi.org/10.1145/383259.383297
55. Hassler Whitney. 1957. Geometric Integration Theory. Princeton University Press.
56. Kai Xu, Daniel Cohen-Or, Tao Ju, Ligang Liu, Hao Zhang, Shizhe Zhou, and Yueshan Xiong. 2009. Feature-Aligned Shape Texturing. ACM Transactions on Graphics 28, 5 (Proc. of SIGGRAPH ASIA 2009) (dec 2009), 1––7. https://doi.org/10.1145/1618452.1618454
57. Yonghao Yue, Breannan Smith, Christopher Batty, Changxi Zheng, and Eitan Grinspun. 2015. Continuum Foam: A Material Point Method for Shear-Dependent Flows. ACM Transactions on Graphics 34, 5 (2015), 160:1–20.
58. Eugene Zhang, James Hays, and Greg Turk. 2007. Interactive Tensor Field Design and Visualization on Surfaces. IEEE Transactions on Visualization and Computer Graphics 13, 1(2007), 94–107. https://doi.org/10.1109/TVCG.2007.16
59. Eugene Zhang, Konstantin Mischaikow, and Greg Turk. 2006. Vector Field Design on Surfaces. ACM Transactions on Graphics 25, 4 (oct 2006), 1294––1326. https://doi.org/10.1145/1183287.1183290
60. Ming Zheng, Antoine Milliez, Markus Gross, and Robert W. Sumner. 2017. Example-Based Brushes for Coherent Stylized Renderings. In Proceedings of NPAR’17. 3:1–3:10. https://doi.org/10.1145/3092919.3092929 Los Angeles, CA, USA, July 28-29.

Index Terms

• Computing methodologies

  • Computer graphics

    • Animation

    • Shape modeling