A Framework for content-adaptive photo manipulation macros: Application to face, landscape, and global manipulations

Abstract

We present a framework for generating content-adaptive macros that can transfer complex photo manipulations to new target images. We demonstrate applications of our framework to face, landscape, and global manipulations. To create a content-adaptive macro, we make use of multiple training demonstrations. Specifically, we use automated image labeling and machine learning techniques to learn the dependencies between image features and the parameters of each selection, brush stroke, and image processing operation in the macro. Although our approach is limited to learning manipulations where there is a direct dependency between image features and operation parameters, we show that our framework is able to learn a large class of the most commonly used manipulations using as few as 20 training demonstrations. Our framework also provides interactive controls to help macro authors and users generate training demonstrations and correct errors due to incorrect labeling or poor parameter estimation. We ask viewers to compare images generated using our content-adaptive macros with and without corrections to manually generated ground-truth images and find that they consistently rate both our automatic and corrected results as close in appearance to the ground truth. We also evaluate the utility of our proposed macro generation workflow via a small informal lab study with professional photographers. The study suggests that our workflow is effective and practical in the context of real-world photo editing.

References

1. Amini, A., Curwen, R., and Gore, J. 1996. Snakes and splines for tracking non-rigid heart motion. In Proceedings of ECCV. 249--261.
2. Bae, S., Paris, S., and Durand, F. 2006. Two-scale tone management for photographic look. In Proceedings of ACM Trans. Graph. 25, 3, 637--645.
3. Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., and Nayar, S. 2008. Face swapping: Automatically replacing faces in photographs. Trans. graph. 27, 3.
4. Bolin, M., Webber, M., Rha, P., Wilson, T., and Miller, R. C. 2005. Automation and customization of rendered web pages. In Proceedings of the UIST Symposium. 163--172.
5. Cypher, A. and Halbert, D. 1993. Watch What I Do: Programming by Demonstration. MIT Press.
6. Dewdney, A. 1989. A potpourri of programmed prose and prosody. Scientific Amer.
7. Drori, I., Cohen-Or, D., and Yeshurun, H. 2003. Example-based style synthesis. In In Proceedings of the Conference on Computer Vision and Pattern Recognition. 143--150.
8. Efron, B., Hastie, T., Johnstone, I., and Tibshirani, R. 2004. Least angle regression. In Annals of Statistics, 407--451.
9. Efros, A. and Freeman, W. 2001. Image quilting for texture synthesis and transfer. In Proceedings of the SIGGRAPH Conference. 341--346.
10. Felzenszwalb, P., McAllester, D., and Ramanan, D. 2008. A discriminatively trained, multiscale, deformable part model. In Proceedings of the CVPR Conference.
11. Grabler, F., Agrawala, M., Li, W., Dontcheva, M., and Igarashi, T. 2009. Generating photo manipulation tutorials by demonstration. ACM Trans. Graph. 28, 3, 66.
12. Guo, D. and Sim, T. 2009. Digital face makeup by example. In Proceedings of the Computer Vision and Pattern Recognition Conference. IEEE Computer Society, 73--79.
13. Hasinoff, S., Józwiak, M., Durand, F., and Freeman, W. 2010. Search-and-replace editing for personal photo collections. In Proceedings of the ICCP. 2. 8.
14. Hertzmann, A., Jacobs, C., Oliver, N., Curless, B., and Salesin, D. 2001. Image analogies. In Proceedings of the SIGGRAPH Conference. 327--340.
15. Hertzmann, A., Oliver, N., Curless, B., and Seitz, S. 2002. Curve analogies. In Proceedings of the Eurographics Workshop on Rendering. 233--246.
16. Hoiem, D., Efros, A., and Hebert, M. 2005. Geometric context from a single image. In Proceedings of the ICCV. 654--661.
17. Huggins, B. 2005. Photoshop: Retouching Cookbook for Digital Photographers. O'Reilly.
18. Jones, M. and Rehg, J. 2002. Statistical color models with application to skin detection. Int. J. Comput. Vision 46, 1, 81--96.
19. Kalnins, R., Markosian, L., Meier, B., Kowalski, M., Lee, J., Davidson, P., Webb, M., Hughes, J., and Finkelstein, A. 2002. WYSIWYG NPR: Drawing strokes directly on 3D models. ACM Trans. Graph. 21, 3, 755--762.
20. Kang, S., Kapoor, A., and Lischinski, D. 2010. Personalization of image enhancement. In Proceedings of the CVPR.
21. Kass, M., Witkin, A., and Terzopoulos, D. 1988. Snakes: Active contour models. Int. J. comput. Vis. 1, 4, 321--331.
22. Kelby, S. 2007. The Adobe Photoshop CS3 Book for Digital Photographers. Voices That Matter.
23. Kurlander, D. and Feiner, S. 1992. A history-based macro by example system. In Proceedings of the UIST Symposium. 99--106.
24. Lau, T., Bergman, L., Castelli, V., and Oblinger, D. 2004. Sheepdog: Learning procedures for technical support. In Proceedings of the IUI Conference. 109--116.
25. Lewis, D. 1998. Naive (Bayes) at forty: The independence assumption in information retrieval. In Proceedings of the ECML Conference. 8, 4--15.
26. Lieberman, H. 1993. Mondrian: A teachable graphical editor. In Watch What I Do: Programming by Demonstration, 341--358.
27. Lieberman, H. 2001. Your Wish is My Command: Giving Users the Power to Instruct their Software. Morgan Kaufmann.
28. Little, G., Lau, T., Cypher, A., Lin, J., Haber, E., and Kandogan, E. 2007. Koala: Capture, share, automate, personalize business processes on the web. In Proceedings of the CHI. 943--946.
29. Liu, Z., Shan, Y., and Zhang, Z. 2001. Expressive expression mapping with ratio images. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. ACM, 276.
30. Modugno, F. and Myers, B. 1994. Pursuit: Graphically representing programs in a demonstrational visual shell. In Proceedings of the CHI. 455--456.
31. Nguyen, M., Lalonde, J., Efros, A., and De la Torre, F. 2008. Image-based shaving. Comput. Graph. Forum. 27, 627--635.
32. Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Comput. Graph. Appl. 34--41.
33. Schwarz, D. 2005. Current research in concatenative sound synthesis. In Proceedings of the ICMC. 9--12.
34. Simhon, S. and Dudek, G. 2003. Curve Synthesis from Learned Refinement Models. http://www.clm.mcgill.ca/saol/pubs/eq03.pdf.
35. Zhou, Y., Gu, L., and Zhang, H. 2003. Bayesian tangent shape model: Estimating shape and pose parameters via bayesian inference. In Proceedings of the CVPR Conference. 109--116.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Image and video acquisition

        • 3D imaging

  • Computer graphics

    • Animation