Toonsynth: example-based synthesis of hand-colored cartoon animations

Abstract

We present a new example-based approach for synthesizing hand-colored cartoon animations. Our method produces results that preserve the specific visual appearance and stylized motion of manually authored animations without requiring artists to draw every frame from scratch. In our framework, the artist first stylizes a limited set of known source skeletal animations from which we extract a style-aware puppet that encodes the appearance and motion characteristics of the artwork. Given a new target skeletal motion, our method automatically transfers the style from the source examples to create a hand-colored target animation. Compared to previous work, our technique is the first to preserve both the detailed visual appearance and stylized motion of the original hand-drawn content. Our approach has numerous practical applications including traditional animation production and content creation for games.

References

1. Rahul Arora, Ishan Darolia, Vinay Namboodiri, Karan Singh, and Adrien Bousseau. 2017. SketchSoup: Exploratory Ideation Using Design Sketches. Computer Graphics Forum 36, 8 (2017), 302--312.
2. Yunfei Bai, Danny M Kaufman, Karen Liu, and Jovan Popović. 2016. Artist-directed dynamics for 2D animation. ACM Transactions on Graphics 35, 4 (2016), 145.
3. William Baxter and Ken-ichi Anjyo. 2006. Latent Doodle Space. Computer Graphics Forum 25, 3 (2006), 477--485.
4. William Baxter, Pascal Barla, and Ken Anjyo. 2009. N-way morphing for 2D animation. Journal of Visualization and Computer Animation 20, 2--3 (2009), 79--87.
5. Pierre Bénard, Forrester Cole, Michael Kass, Igor Mordatch, James Hegarty, Martin Sebastian Senn, Kurt Fleischer, Davide Pesare, and Katherine Breeden. 2013. Stylizing animation by example. ACM Transactions on Graphics 32, 4 (2013), 119.
6. Mikhail Bessmeltsev, Nicholas Vining, and Alla Sheffer. 2016. Gesture3D: posing 3D characters via gesture drawings. ACM Transactions on Graphics 35, 6 (2016), 165.
7. Christoph Bregler, Lorie Loeb, Erika Chuang, and Hrishi Deshpande. 2002. Turning to the Masters: Motion Capturing Cartoons. ACM Transactions on Graphics 21, 3 (2002), 399--407.
8. Ian Buck, Adam Finkelstein, Charles Jacobs, Allison Klein, David Salesin, Joshua Seims, Richard Szeliski, and Kentaro Toyama. 2000. Performance-Driven Hand-Drawn Animation. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering. 101--108.
9. Nestor Burtnyk and Marceli Wein. 1976. Interactive Skeleton Techniques for Enhancing Motion Dynamics in Key Frame Animation. Commun. ACM 19, 10 (1976), 564--569.
10. Edwin Catmull. 1978. The Problems of Computer-Assisted Animation. 12, 3 (1978), 348--353.
11. James Davis, Maneesh Agrawala, Erika Chuang, Zoran Popovic, and David Salesin. 2003. A sketching interface for articulated figure animation. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 320--328.
12. Marek Dvorožňák, Pierre Bénard, Pascal Barla, Oliver Wang, and Daniel Sýkora. 2017. Example-Based Expressive Animation of 2D Rigid Bodies. ACM Transactions on Graphics 36, 4, Article 127 (2017).
13. Elmar Eisemann and Frédo Durand. 2004. Flash photography enhancement via intrinsic relighting. ACM Transactions on Graphics 23, 3 (2004), 673--678.
14. Jakub Fišer, Ondřej Jamriška, Michal Lukáč, Eli Shechtman, Paul Asente, Jingwan Lu, and Daniel Sýkora. 2016. Stylit: Illumination-guided Example-based Stylization of 3D Renderings. ACM Transactions on Graphics 35, 4 (2016), 92.
15. Jakub Fišer, Ondřej Jamriška, David Simons, Eli Shechtman, Jingwan Lu, Paul Asente, Michal Lukáč, and Daniel Sýkora. 2017. Example-Based Synthesis of Stylized Facial Animations. ACM Transactions on Graphics 36, 4, Article 155 (2017).
16. Jakub Fišer, Michal Lukáč, Ondřej Jamriška, Martin Čadík, Yotam Gingold, Paul Asente, and Daniel Sýkora. 2014. Color Me Noisy: Example-Based Rendering of Hand-Colored Animations with Temporal Noise Control. Computer Graphics Forum 33, 4 (2014), 1--10.
17. Ben Glocker, Nikos Komodakis, Georgios Tziritas, Nassir Navab, and Nikos Paragios. 2008. Dense Image Registration Through MRFs And Efficient Linear Programming. Medical Image Analysis 12, 6 (2008), 731--741.
18. William van Haevre, Fabian di Fiore, and Frank van Reeth. 2005. Uniting Cartoon Textures with Computer Assisted Animation. In Proceedings of International Conference on Computer Graphics and Interactive Techniques in Australasia and South East Asia. 245--253.
19. Aaron Hertzmann, Charles E Jacobs, Nuria Oliver, Brian Curless, and David H Salesin. 2001. Image analogies. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, 327--340.
20. Alexander Hornung, Ellen Dekkers, and Leif Kobbelt. 2007. Character Animation from 2D Pictures and 3D Motion Data. ACM Transactions on Graphics 26, 1 (2007).
21. Eakta Jain, Yaser Sheikh, and Jessica Hodgins. 2009. Leveraging the talent of hand animators to create three-dimensional animation. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 93--102.
22. Ben Jones Jovan Popovic, James McCann, Wilmot Li, and Adam Bargteil. 2015. Dynamic sprites: Artistic authoring of interactive animations. Journal of Visualization and Computer Animation 26, 2 (2015), 97--108.
23. Christina de Juan and Bobby Bodenheimer. 2004. Cartoon Textures. In Proceedings of Eurographics Symposium on Computer Animation. 267--276.
24. Christina de Juan and Bobby Bodenheimer. 2006. Re-using traditional animation: Methods for semi-automatic segmentation and inbetweening. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 223--232.
25. Rubaiat Habib Kazi, Tovi Grossman, Nobuyuki Umetani, and George Fitzmaurice. 2016. Motion Amplifiers: Sketching Dynamic Illustrations Using the Principles of 2D Animation. In Proceedings of ACM Conference on Human Factors in Computing Systems. 4599--4609.
26. Alexander Kort. 2002. Computer Aided Inbetweening. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering. 125--132.
27. Lucas Kovar, Michael Gleicher, and Frédéric Pighin. 2002. Motion Graphs. ACM Transactions on Graphics 21, 3 (2002), 473--482.
28. Vivek Kwatra, Irfan Essa, Aaron Bobick, and Nipun Kwatra. 2005. Texture optimization for example-based synthesis. ACM Transactions on Graphics 24, 3 (2005), 795--802.
29. John Lasseter. 1987. Principles of Traditional Animation Applied to 3D Computer Animation. In Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques. 35--44.
30. Seungyong Lee, George Wolberg, and Sung Yong Shin. 1998. Polymorph: Morphing Among Multiple Images. IEEE Computer Graphics and Applications 18, 1 (1998), 58--71.
31. Sun-Young Lee, Jong-Chul Yoon, Ji-Yong Kwon, and In-Kwon Lee. 2012. CartoonModes: Cartoon Stylization of Video Objects Through Modal Analysis. Graphical Models 74, 2 (2012), 51--60.
32. Dushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafiei, Hans-Peter Seidel, Weipeng Xu, Dan Casas, and Christian Theobalt. 2017. VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera. ACM Transactions on Graphics 36, 4 (2017), 44:1--44:14.
33. Johannes Schmid, Robert Sumner, Huw Bowles, and Markus Gross. 2010. Programmable Motion Effects. ACM Transactions on Graphics 29, 4 (2010), 57.
34. Daniel Sýkora, Mirela Ben-Chen, Martin Čadík, Brian Whited, and Maryann Simmons. 2011. TexToons: Practical Texture Mapping for Hand-drawn Cartoon Animations. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering. 75--83.
35. Daniel Sýkora, Jan Buriánek, and Jiří Žára. 2005. Sketching Cartoons by Example. In Proceedings of Eurographics Workshop on Sketch-Based Interfaces and Modeling. 27--34.
36. Daniel Sýkora, John Dingliana, and Steven Collins. 2009. As-Rigid-As-Possible Image Registration for Hand-Drawn Cartoon Animations. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering. 25--33.
37. Daniel Sýkora, Ladislav Kavan, Martin Čadík, Ondřej Jamriška, Alec Jacobson, Brian Whited, Maryann Simmons, and Olga Sorkine-Hornung. 2014. Ink-and-Ray: Bas-Relief Meshes for Adding Global Illumination Effects to Hand-Drawn Characters. ACM Transactions on Graphics 33, 2 (2014), 16.
38. Daniel Sýkora, David Sedlacek, Sun Jinchao, John Dingliana, and Steven Collins. 2010. Adding Depth to Cartoons Using Sparse Depth (In)equalities. Computer Graphics Forum 29, 2 (2010), 615--623.
39. Cedric Vanaken, Chris Hermans, Tom Mertens, Fabian Di Fiore, Philippe Bekaert, and Frank Van Reeth. 2008. Strike a Pose: Image-Based Pose Synthesis. In Proceedings of the Conference on Vision, Modeling and Visualization. 131--138.
40. Jue Wang, Steven Drucker, Maneesh Agrawala, and Michael Cohen. 2006. The Cartoon Animation Filter. ACM Transactions on Graphics 25, 3 (2006), 1169--1173.
41. Xun Wang, Wenwu Yang, Haoyu Peng, and Guozheng Wang. 2013. Shape-aware skeletal deformation for 2D characters. The Visual Computer 29, 6--8 (2013), 545--553.
42. Yonatan Wexler, Eli Shechtman, and Michal Irani. 2007. Space-Time Completion of Video. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 3 (2007), 463--476.
43. Brian Whited, Gioacchino Noris, Maryann Simmons, Robert Sumner, Markus Gross, and Jarek Rossignac. 2010. BetweenIT: An Interactive Tool for Tight Inbetweening. Computer Graphics Forum 29, 2 (2010), 605--614.
44. Nora Willett, Wilmot Li, Jovan Popovic, Floraine Berthouzoz, and Adam Finkelstein. 2017. Secondary Motion for Performed 2D Animation. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 97--108.
45. Wenwu Yang. 2017. Context-Aware Computer Aided Inbetweening. IEEE Transactions on Visualizaton and Computer Graphics (2017).
46. Chih-Kuo Yeh, Shi-Yang Huang, Pradeep Kumar Jayaraman, Chi-Wing Fu, and Tong-Yee Lee. 2017. Interactive High-Relief Reconstruction for Organic and Double-Sided Objects from a Photo. IEEE Transactions on Visualization and Computer Graphics 23, 7 (2017), 1796--1808.
47. Lei Zhang, Hua Huang, and Hongbo Fu. 2012. EXCOL: An EXtract-and-COmplete Layering Approach to Cartoon Animation Reusing. IEEE Transactions on Visualization and Computer Graphics 18, 7 (2012), 1156--1169.
48. Yufeng Zhu, Jovan Popović, Robert Bridson, and Danny Kaufman. 2017. Planar Interpolation with Extreme Deformation, Topology Change and Dynamics. ACM Transactions on Graphics 36, 6 (2017), 213.

Index Terms

• Computing methodologies

  • Computer graphics

    • Animation

      • Motion processing

    • Image manipulation

      • Image processing