Example-based video color grading

Abstract

In most professional cinema productions, the color palette of the movie is painstakingly adjusted by a team of skilled colorists -- through a process referred to as color grading -- to achieve a certain visual look. The time and expertise required to grade a video makes it difficult for amateurs to manipulate the colors of their own video clips. In this work, we present a method that allows a user to transfer the color palette of a model video clip to their own video sequence. We estimate a per-frame color transform that maps the color distributions in the input video sequence to that of the model video clip. Applying this transformation naively leads to artifacts such as bleeding and flickering. Instead, we propose a novel differential-geometry-based scheme that interpolates these transformations in a manner that minimizes their curvature, similarly to curvature flows. In addition, we automatically determine a set of keyframes that best represent this interpolated transformation curve, and can be used subsequently, to manually refine the color grade. We show how our method can successfully transfer color palettes between videos for a range of visual styles and a number of input video clips.

References

1. Amari, S., and Nagaoka, H. 2000. Methods of Information Geometry, vol. 191 of Translations of Mathematical monographs. Oxford University Press.
2. An, X., and Pellacini, F. 2010. User-controllable color transfer. Computer Graphics Forum 29, 2, 263--271.
3. Bae, S., Paris, S., and Durand, F. 2006. Two-scale tone management for photographic look. ACM Trans. on Graphics (Proc. of ACM SIGGRAPH 2006) 25, 3, 637--645.
4. Bai, X., Wang, J., Simons, D., and Sapiro, G. 2009. Video SnapCut: robust video object cutout using localized classifiers. In ACM Trans. on Graphics (Proc. of ACM SIGGRAPH 2009), 70:1--70:11.
5. Bonneel, N., van de Panne, M., Paris, S., and Heidrich, W. 2011. Displacement interpolation using lagrangian mass transport. In ACM Trans. on Graphics (Proc. of ACM SIGGRAPH Asia 2011), 158:1--158:12.
6. Chang, Y., Saito, S., and Nakajima, M. 2007. Example-based color transformation of image and video using basic color categories. Image Processing, IEEE Trans. on 16, 2, 329--336.
7. Devroye, L. 1986. Non-Uniform Random Variate Generation. Springer-Verlag. Section 2.2. Inversion by numerical solution of F(X) = U.
8. do Carmo, M. P. 1992. Riemannian Geometry. Springer, Jan.
9. Farbman, Z., and Lischinski, D. 2011. Tonal stabilization of video. ACM Trans. on Graphics (Proc. of ACM SIGGRAPH 2011) 30, 4, 89:1--89:9.
10. Ferradans, S., Xia, G.-S., Peyré, G., and Aujol, J.-F. 2012. Optimal transport mixing of gaussian texture models. Tech. rep., Preprint Hal-00662720.
11. Galassi, M., Davis, J., Gough, B., Jungman, G., Booth, M., and Rossi, F., 2011. GNU scientific library - reference manual, version 1.15, sec. 21.7 weighted samples.
12. Gastal, E. S. L., and Oliveira, M. M. 2011. Domain transform for edge-aware image and video processing. In ACM Trans. on Graphics (Proc. of ACM SIGGRAPH 2011), 69:1--69:12.
13. HaCohen, Y., Shechtman, E., Goldman, D. B., and Lischinski, D. 2011. Non-rigid dense correspondence with applications for image enhancement. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH 2011) 30, 4, 70:1--70:9.
14. Ilmanen, T. 1995. Lectures on mean curvature flow and related equations. In Lecture Notes, ICTP, Trieste.
15. Johnson, M. K., Dlale, K., Avidan, S., Pfister, H., Freeman, W. T., and Matusik, W. 2011. CG2Real: improving the realism of computer generated images using a large collection of photographs. IEEE Trans. on Visualization and Computer Graphics 17, 9 (Sept.), 1273--1285.
16. Kagarlitsky, S., Moses, Y., and Hel-Or, Y. 2009. Piecewise-consistent color mappings of images acquired under various conditions. In Computer Vision, 2009 IEEE 12th International Conference on, 2311--2318.
17. Kiser, C., Reinhard, E., Tocci, M., and Tocci, N. 2012. Real time automated tone mapping system for HDR video. In Proc. of the IEEE Int. Conference on Image Processing, IEEE.
18. Lang, M., Wang, O., Aydin, T., Smolic, A., and Gross, M. 2012. Practical temporal consistency for image-based graphics applications. ACM Trans. on Graphics (Proc. of ACM SIGGRAPH 2012) 31, 4 (July), 34:1--34:8.
19. Levin, A., Lischinski, D., and Weiss, Y. 2008. A closed-form solution to natural image matting. IEEE Trans. on Pattern Analysis and Machine Intelligence 30, 2 (Feb.), 228--242.
20. Murray, N., Skaff, S., Marchesotti, L., and Perronnin, F. 2011. Towards automatic concept transfer. In Proc. of the Eurographics Symposium on Non-Photorealistic Animation and Rendering, ACM, NPAR '11, 167--176.
21. Oldenborg, M. 2006. A comparison between techniques for color grading in games. PhD thesis, University of Skövde, School of Humanities and Informatics.
22. Oskam, T., Hornung, A., Sumner, R. W., and Gross, M. 2012. Fast and stable color balancing for images and augmented reality. In 2nd Int. Conf. on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT), 49--56.
23. Paris, S. 2008. Edge-preserving smoothing and mean-shift segmentation of video streams. In Proc. of the 10th European Conference on Computer Vision, ECCV '08, 460--473.
24. Park, H.-S., and Jun, C.-H. 2009. A simple and fast algorithm for K-medoids clustering. Expert Systems with Applications 36, 2, Part 2 (Mar.), 3336--3341.
25. Pitié, F., and Kokaram, A. C. 2007. The linear mongekantorovitch linear colour mapping for example-based colour transfer. In 4th European Conference on Visual Media Production, 2007. IETCVMP, 1--9.
26. Pitié, F., Kokaram, A. C., and Dahyot, R. 2005. N-dimensional probablility density function transfer and its application to colour transfer. In Proceedings of the Tenth IEEE Int. Conf. on Computer Vision - Vol. 2, ICCV '05, 1434--1439.
27. Pouli, T., and Reinhard, E. 2011. Progressive color transfer for images of arbitrary dynamic range. Computers & Graphics 35, 1 (Feb.), 67--80.
28. Rabin, J., Delon, J., and Gousseau, Y. 2010. Regularization of transportation maps for color and contrast transfer. In 2010 17th IEEE Int. Conf. on Image Processing (ICIP), 1933--1936.
29. Reinhard, E., and Pouli, T. 2011. Colour spaces for colour transfer. In Proceedings of the Third international conference on Computational color imaging, Springer-Verlag, Berlin, Heidelberg, CCIW'11, 1--15.
30. Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Comput. Graph. Appl. 21, 5 (Sept.), 34--41.
31. Selan, J. 2012. Cinematic color: From your monitor to the big screen. Visual Effects Soc. Tech. Committee White Paper (Oct.).
32. Tai, Y.-W., Jia, J., and Tang, C.-K. 2005. Local color transfer via probabilistic segmentation by expectation-maximization. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, vol. 1, 747--754 vol. 1.
33. Takatsu, A. 2011. Wasserstein geometry of gaussian measures. Osaka Journal of Mathematics 48, 4 (Dec.), 1005--1026. Mathematical Reviews number (MathSciNet): MR2648273.
34. Villani, C. 2003. Topics in Optimal Transportation. Graduate Studies in Mathematics Series. Amer Mathematical Society.
35. Wang, B., Yu, Y., and Xu, Y.-Q. 2011. Example-based image color and tone style enhancement. In ACM Trans. on Graphics (Proc. of ACM SIGGRAPH 2011), 64:1--64:12.
36. Xue, S., Agarwala, A., Dorsey, J., and Rushmeier, H. E. 2012. Understanding and improving the realism of image composites. ACM Trans. on Graphics (Proc. of ACM SIGGRAPH 2012) 31, 4, 84:1--84:10.
37. Yatziv, L., and Sapiro, G. 2006. Fast image and video colorization using chrominance blending. IEEE Trans. on Image Processing 15, 5 (May), 1120--1129.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

  • Computer graphics

    • Image manipulation