Stylizing animation by example

Abstract

Skilled artists, using traditional media or modern computer painting tools, can create a variety of expressive styles that are very appealing in still images, but have been unsuitable for animation. The key difficulty is that existing techniques lack adequate temporal coherence to animate these styles effectively. Here we augment the range of practical animation styles by extending the guided texture synthesis method of Image Analogies [Hertzmann et al. 2001] to create temporally coherent animation sequences. To make the method art directable, we allow artists to paint portions of keyframes that are used as constraints. The in-betweens calculated by our method maintain stylistic continuity and yet change no more than necessary over time.

References

1. Agarwala, A., Hertzmann, A., Salesin, D. H., and Seitz, S. M. 2004. Keyframe-based tracking for rotoscoping and animation. ACM Transactions on Graphics 23, 3, 584--591.
2. Ashikhmin, M. 2001. Synthesizing natural textures. In Proceedings of the 7th symposium on Interactive 3D graphics, 217--226.
3. Bargteil, A. W., Sin, F., Michaels, J., Goktekin, T., and O'Brien, J. 2006. A texture synthesis method for liquid animations. In ACM SIGGRAPH/Eurographics Symposium on Computer Animation, 345--351.
4. Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. B. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics 28, 3, 24:1--24:11.
5. Barnes, C., Shechtman, E., Goldman, D. B., and Finkelstein, A. 2010. The generalized PatchMatch correspondence algorithm. In Proceedings of the 11th European conference on computer vision conference on Computer vision, 29--43.
6. Bénard, P., Bousseau, A., and Thollot, J. 2009. Dynamic solid textures for real-time coherent stylization. Proceedings of the 2009 symposium on Interactive 3D graphics and games, 121--127.
7. Bénard, P., Lagae, A., Vangorp, P., Lefebvre, S., Drettakis, G., and Thollot, J. 2010. A dynamic noise primitive for coherent stylization. Computer Graphics Forum 4, 29, 1497--1506.
8. Bénard, P., Bousseau, A., and Thollot, J. 2011. State-of-the-Art Report on Temporal Coherence for Stylized Animations. Computer Graphics Forum 30, 8, 2367--2386.
9. Bénard, P., Jingwan, L., Cole, F., Finkelstein, A., and Thollot, J. 2012. Active Strokes: Coherent Line Stylization for Animated 3D Models. In NPAR 2012 - 10th International Symposium on Non-photorealistic Animation and Rendering, ACM, 37--46.
10. Bonneel, N., van de Panne, M., Lefebvre, S., and Drettakis, G. 2010. Proxy-guided texture synthesis for rendering natural scenes. In 15th International Workshop on Vision, Modeling and Visualization, 87--95.
11. Bousseau, A., Neyret, F., Thollot, J., and Salesin, D. 2007. Video watercolorization using bidirectional texture advection. ACM Transactions on Graphics 26, 3, 104.
12. Cao, C., Chen, S., Zhang, W., and Tang, X. 2011. Automatic motion-guided video stylization and personalization. In Proceedings of the 19th ACM international conference on Multimedia, 1041--1044.
13. Chen, J., and Wang, B. 2009. High quality solid texture synthesis using position and index histogram matching. The Visual Computer 26, 4, 253--262.
14. Daniels, E. 1999. Deep canvas in Disney's Tarzan. In ACM SIGGRAPH 99 Conference abstracts and applications, 200.
15. Darabi, S., Shechtman, E., Barnes, C., Goldman, D. B., and Sen, P. 2012. Image melding. ACM Transactions on Graphics 31, 4, 82:1--82:10.
16. Efros, A. A., and Freeman, W. T. 2001. Image quilting for texture synthesis and transfer. In Proceedings of SIGGRAPH '01, 341--346.
17. Granlund, G. H., and Knutsson, H. 1995. Signal Processing for Computer Vision. Springer.
18. Han, J., Zhou, K., Wei, L.-Y., Gong, M., Bao, H., Zhang, X., and Guo, B. 2006. Fast example-based surface texture synthesis via discrete optimization. The Visual Computer 22, 9--11.
19. Haro, A. 2003. Example based processing for image and video synthesis. PhD thesis. AAI3117934.
20. Hashimoto, R., Johan, H., and Nishita, T. 2003. Creating various styles of animations using example-based filtering. In Computer Graphics International, 312--317.
21. Hertzmann, A., Jacobs, C. E., Oliver, N., Curless, B., and Salesin, D. H. 2001. Image analogies. In Proceedings of SIGGRAPH '01, 327--340.
22. Kabul, I., Pizer, S. M., Rosenman, J., and Niethammer, M. 2011. An Optimal Control Approach for Texture Metamorphosis. Computer Graphics Forum 30, 8, 2341--2353.
23. Kagaya, M., Brendel, W., Deng, Q., Kesterson, T., Todorovic, S., Neill, P. J., and Zhang, E. 2011. Video painting with space-time-varying style parameters. IEEE Transactions on Visualization and Computer Graphics 17, 1, 74--87.
24. Kalnins, R. D., Markosian, L., Meier, B. J., Kowalski, M. A., Lee, J. C., Davidson, P. L., Webb, M., Hughes, J. F., and Finkelstein, A. 2002. WYSIWYG NPR: drawing strokes directly on 3D models. In ACM Transactions on Graphics, vol. 21, 755--762.
25. Kass, M., and Pesare, D. 2011. Coherent noise for non-photorealistic rendering. ACM Transactions on Graphics 30, 4, 30:1--30:6.
26. Kulla, C., Tucek, J., Bailey, R., and Grimm, C. 2003. Using texture synthesis for non-photorealistic shading from paint samples. In Proceedings of the 11th Pacific Conference on Computer Graphics and Applications, 477--481.
27. Kwatra, V., Essa, I., Bobick, A., and Kwatra, N. 2005. Texture optimization for example-based synthesis. ACM Trans. Graph. 24, 3, 795--802.
28. Kwatra, V., Adalsteinsson, D., Kim, T., Kwatra, N., Carlson, M., and Lin, M. C. 2007. Texturing fluids. IEEE Transactions on Visualization and Computer Graphics 13, 5, 939--952.
29. Kyprianidis, J. E., Collomosse, J., Wang, T., and Isenberg, T. 2013. State of the "art": A taxonomy of artistic stylization techniques for images and video. Visualization and Computer Graphics, IEEE Transactions on 19, 5, 866--885.
30. Lai, C.-H., and Wu, J.-L. 2007. Temporal texture synthesis by patch-based sampling and morphing interpolation. Computer Animation and Virtual Worlds 18, 4--5, 415--428.
31. Lasram, A., and Lefebvre, S. 2012. Parallel patch-based texture synthesis. In Proceedings of the 4th ACM SIGGRAPH/Eurographics conference on High-Performance Graphics, 115--124.
32. Lee, H., Seo, S., and Yoon, K. 2011. Directional texture transfer with edge enhancement. Computers & Graphics 35, 1, 81--91.
33. Lefebvre, S., and Hoppe, H. 2006. Appearance-space texture synthesis. ACM Transactions on Graphics 25, 3, 541--548.
34. Lin, L., Zeng, K., Lv, H., Wang, Y., Xu, Y., and Zhu, S.-C. 2010. Painterly animation using video semantics and feature correspondence. In Proceedings of the 8th International Symposium on Non-Photorealistic Animation and Rendering, no. 212, 73--80.
35. Liu, Z., Liu, C., Shum, H.-Y., and Yu, Y. 2002. Pattern-based texture metamorphosis. Proceedings of the 10th Pacific Conference on Computer Graphics and Applications 1, 184--191.
36. Matusik, W., Zwicker, M., and Durand, F. 2005. Texture design using a simplicial complex of morphable textures. ACM Transactions on Graphics 24, 3, 787--794.
37. O'Donovan, P., and Hertzmann, A. 2011. AniPaint: Interactive Painterly Animation from Video. IEEE Transactions on Visualization and Computer Graphics 18, 3, 475--487.
38. Praun, E., Hoppe, H., Webb, M., and Finkelstein, A. 2001. Real-time hatching. Proceedings of SIGGRAPH '01, 581.
39. Ray, N., Lévy, B., Wang, H., Turk, G., and Vallet, B. 2009. Material Space Texturing. Computer Graphics Forum 28, 6, 1659--1669.
40. Ruiters, R., Schnabel, R., and Klein, R. 2010. Patch-based Texture Interpolation. Computer Graphics Forum 29, 4, 1421--1429.
41. Schmid, J., Senn, M. S., Gross, M., and Sumner, R. W. 2011. OverCoat: an implicit canvas for 3D painting. In ACM Transactions on Graphics, vol. 30, 28:1--28:10.
42. Shechtman, E., Rav-Acha, A., Irani, M., and Seitz, S. 2010. Regenerative morphing. In IEEE Conference on Computer VIsion and Pattern Recognition (CVPR), 615--622.
43. Sýkora, D., Ben-Chen, M., Čadík, M., Whited, B., and Simmons, M. 2011. TexToons. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Non-Photorealistic Animation and Rendering, 75--84.
44. Wei, L.-Y., and Levoy, M. 2000. Fast texture synthesis using tree-structured vector quantization. In Proceedings of SIGGRAPH 2000, 479--488.
45. Wei, L.-Y., Lefebvre, S., Kwatra, V., and Turk, G. 2009. State of the Art in Example-based Texture Synthesis. In Eurographics 2009 - State of the Art Report, 93--117.
46. Whited, B., Daniels, E., Kaschalk, M., Osborne, P., and Odermatt, K. 2012. Computer-assisted animation of line and paint in Disney's Paperman. In ACM SIGGRAPH 2012 Talks, SIGGRAPH '12, 19:1.
47. Winnemöller, H., Olsen, S. C., and Gooch, B. 2006. Realtime video abstraction. ACM Transactions on Graphics 25, 3, 1221--1226.
48. Zhang, J., Zhou, K., Velho, L., Guo, B., and Shum, H.-Y. 2003. Synthesis of progressively-variant textures on arbitrary surfaces. ACM Transactions on Graphics 22, 3, 295--302.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision tasks

  • Computer graphics