Progressive Color Transfer With Dense Semantic Correspondences

Abstract

We propose a new algorithm for color transfer between images that have perceptually similar semantic structures. We aim to achieve a more accurate color transfer that leverages semantically meaningful dense correspondence between images. To accomplish this, our algorithm uses neural representations for matching. Additionally, the color transfer should be spatially variant and globally coherent. Therefore, our algorithm optimizes a local linear model for color transfer satisfying both local and global constraints. Our proposed approach jointly optimizes matching and color transfer, adopting a coarse-to-fine strategy. The proposed method can be successfully extended from one-to-one to one-to-many color transfer. The latter further addresses the problem of mismatching elements of the input image. We validate our proposed method by testing it on a large variety of image content.

References

1. Xiaobo An and Fabio Pellacini. 2008. AppProp: All-pairs appearance-space edit propagation. ACM Transactions on Graphics 27, 3 (2008), Article 40.
2. Xiaobo An and Fabio Pellacini. 2010. User-controllable color transfer. Computer Graphics Forum 29 (2010), 263--271.
3. Benoit Arbelot, Romain Vergne, Thomas Hurtut, and Joëlle Thollot. 2017. Local texture-based color transfer and colorization. Computers and Graphics 62 (2017), 15--27.
4. Soonmin Bae, Sylvain Paris, and Frédo Durand. 2006. Two-scale tone management for photographic look. ACM Transactions on Graphics 25, 3 (2006), 637--645.
5. Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan Goldman. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics 28, 3 (2009), 24.
6. Connelly Barnes, Eli Shechtman, Dan Goldman, and Adam Finkelstein. 2010. The generalized PatchMatch correspondence algorithm. In Proceedings of the European Conference on Computer Vision (ECCV’10). 29--43.
7. Nicolas Bonneel, Gabriel Peyré, and Marco Cuturi. 2016. Wasserstein barycentric coordinates: Histogram regression using optimal transport. ACM Transactions on Graphics 35, 4 (2016), Article 71.
8. Dongdong Chen, Jing Liao, Lu Yuan, Nenghai Yu, and Gang Hua. 2017a. Coherent online video style transfer. In Proceedings of the International Conference on Computer Vision (ICCV’17).
9. Dongdong Chen, Lu Yuan, Jing Liao, Nenghai Yu, and Gang Hua. 2017b. StyleBank: An explicit representation for neural image style transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17), Vol. 1. 4.
10. Dongdong Chen, Lu Yuan, Jing Liao, Nenghai Yu, and Gang Hua. 2018b. Stereoscopic neural style transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18), Vol. 10.
11. Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. 2018a. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence 40, 4 (2018), 834--848.
12. Xiaowu Chen, Dongqing Zou, Steven Zhiying Zhou, Qinping Zhao, and Ping Tan. 2013. Image matting with local and nonlocal smooth priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’13). 1902--1907.
13. Kevin Dale, Micah K. Johnson, Kalyan Sunkavalli, Wojciech Matusik, and Hanspeter Pfister. 2009. Image restoration using online photo collections. In Proceedings of the 2009 IEEE International Conference on Computer Vision (ICCV’09). IEEE, Los Alamitos, CA, 2217--2224.
14. Yuki Endo, Satoshi Iizuka, Yoshihiro Kanamori, and Jun Mitani. 2016. DeepProp: Extracting deep features from a single image for edit propagation. Computer Graphics Forum 35, 2 (2016), 189--201.
15. Zeev Farbman, Raanan Fattal, Dani Lischinski, and Richard Szeliski. 2008. Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM Transactions on Graphics 27, 3 (2008), Article 67.
16. Daniel Freedman and Pavel Kisilev. 2010. Object-to-object color transfer: Optimal flows and SMSP transformations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’10). IEEE, Los Alamitos, CA, 287--294.
17. Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2015. A neural algorithm of artistic style. arXiv:1508.06576.
18. Yoav HaCohen, Eli Shechtman, Dan B. Goldman, and Dani Lischinski. 2011. Non-rigid dense correspondence with applications for image enhancement. ACM Transactions on Graphics 30, 4 (2011), 70.
19. Mingming He, Dongdong Chen, Jing Liao, Pedro V. Sander, and Lu Yuan. 2018. Deep exemplar-based colorization. ACM Transactions on Graphics 37, 4 (2018), 110.
20. Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salesin. 2001. Image analogies. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. ACM, New York, NY, 327--340.
21. Hristina Hristova, Olivier Le Meur, Rémi Cozot, and Kadi Bouatouch. 2015. Style-aware robust color transfer. In Proceedings of the Workshop on Computational Aesthetics. 67--77.
22. Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2016. Let there be color!: Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Transactions on Graphics 35, 4 (2016), 110.
23. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-to-image translation with conditional adversarial networks. arXiv:1611.07004.
24. Asad Khan, Luo Jiang, Wei Li, and Ligang Liu. 2017. Fast color transfer from multiple images. Applied Mathematics-A Journal of Chinese Universities 32, 2 (2017), 183--200.
25. Vladimir Kolmogorov and Ramin Zabin. 2004. What energy functions can be minimized via graph cuts? IEEE Transactions on Pattern Analysis and Machine Intelligence 26, 2 (2004), 147--159.
26. Pierre-Yves Laffont, Zhile Ren, Xiaofeng Tao, Chao Qian, and James Hays. 2014. Transient attributes for high-level understanding and editing of outdoor scenes. ACM Transactions on Graphics 33, 4 (2014), 149.
27. Joon-Young Lee, Kalyan Sunkavalli, Zhe Lin, Xiaohui Shen, and In So Kweon. 2016. Automatic content-aware color and tone stylization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16). 2470--2478.
28. Anat Levin, Dani Lischinski, and Yair Weiss. 2004. Colorization using optimization. ACM Transactions on Graphics 23, 3 (2004), 689--694.
29. Anat Levin, Dani Lischinski, and Yair Weiss. 2008. A closed-form solution to natural image matting. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 2 (2008), 228--242.
30. Chuan Li and Michael Wand. 2016. Combining Markov random fields and convolutional neural networks for image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16). 2479--2486.
31. Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, and Sing Bing Kang. 2017. Visual attribute transfer through deep image analogy. ACM Transactions on Graphics 36, 4 (2017), Article 120, 15 pages.
32. Dani Lischinski, Zeev Farbman, Matt Uyttendaele, and Richard Szeliski. 2006. Interactive local adjustment of tonal values. ACM Transactions on Graphics 25, 3 (2006), 646--653.
33. Yiming Liu, Michael Cohen, Matt Uyttendaele, and Szymon Rusinkiewicz. 2014. AutoStyle: Automatic style transfer from image collections to users’ images. Computer Graphics Forum 33 (2014), 21--31.
34. Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. 2017. Deep photo style transfer. arXiv:1703.07511.
35. Roey Mechrez, Eli Shechtman, and Lihi Zelnik-Manor. 2017. Photorealistic style transfer with screened Poisson equation. arXiv:1709.09828.
36. Augustus Odena, Vincent Dumoulin, and Chris Olah. 2016. Deconvolution and checkerboard artifacts. Distill. Retrieved March 7, 2019 from http://distill.pub/2016/deconv-checkerboard.
37. Francois Pitie, Anil C. Kokaram, and Rozenn Dahyot. 2005. N-dimensional probability density function transfer and its application to color transfer. In Proceedings of the 2005 IEEE International Conference on Computer Vision (ICCV’05), Vol. 2. IEEE, Los Alamitos, CA, 1434--1439.
38. Erik Reinhard, Michael Adhikhmin, Bruce Gooch, and Peter Shirley. 2001. Color transfer between images. IEEE Computer Graphics and Applications 21, 5 (2001), 34--41.
39. Sam T. Roweis and Lawrence K. Saul. 2000. Nonlinear dimensionality reduction by locally linear embedding. Science 290, 5500 (2000), 2323--2326.
40. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, et al. 2015. ImageNet large scale visual recognition challenge. International Journal of Computer Vision 115, 3 (2015), 211--252.
41. Xiaoyong Shen, Xin Tao, Chao Zhou, Hongyun Gao, and Jiaya Jia. 2016. Regional foremost matching for Internet scene images. ACM Transactions on Graphics 35, 6 (2016), 178.
42. YiChang Shih, Sylvain Paris, Connelly Barnes, William T. Freeman, and Frédo Durand. 2014. Style transfer for headshot portraits. ACM Transactions on Graphics 33, 4 (2014), Article 148, 14 pages.
43. YiChang Shih, Sylvain Paris, Frédo Durand, and William T. Freeman. 2013. Data-driven hallucination of different times of day from a single outdoor photo. ACM Transactions on Graphics 32, 6 (2013), Article 200.
44. Denis Simakov, Yaron Caspi, Eli Shechtman, and Michal Irani. 2008. Summarizing visual data using bidirectional similarity. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’08). IEEE, Los Alamitos, CA, 1--8.
45. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556.
46. Kalyan Sunkavalli, Micah K. Johnson, Wojciech Matusik, and Hanspeter Pfister. 2010. Multi-scale image harmonization. ACM Transactions on Graphics 29, 4 (2010), Article 125, 10 pages.
47. Yu-Wing Tai, Jiaya Jia, and Chi-Keung Tang. 2005. Local color transfer via probabilistic segmentation by expectation-maximization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’05), Vol. 1. IEEE, Los Alamitos, CA, 747--754.
48. Wai-Shun Tong, Chi-Keung Tang, Michael S. Brown, and Ying-Qing Xu. 2007. Example-based cosmetic transfer. In Proceedings of the 15th Pacific Conference on Computer Graphics and Applications (PG’07). IEEE, Los Alamitos, CA, 211--218.
49. Tomihisa Welsh, Michael Ashikhmin, and Klaus Mueller. 2002. Transferring color to greyscale images. ACM Transactions on Graphics 21, 3 (2002), 277--280.
50. Zhicheng Yan, Hao Zhang, Baoyuan Wang, Sylvain Paris, and Yizhou Yu. 2016. Automatic photo adjustment using deep neural networks. ACM Transactions on Graphics 35, 2 (2016), 11.
51. Jae-Doug Yoo, Min-Ki Park, Ji-Ho Cho, and Kwan H. Lee. 2013. Local color transfer between images using dominant colors. Journal of Electronic Imaging 22, 3 (2013), 033003.
52. Richard Zhang, Phillip Isola, and Alexei A. Efros. 2016. Colorful image colorization. In Proceedings of the European Conference on Computer Vision (ECCV’16). 649--666.
53. Qi Zhao, Ping Tan, Qiang Dai, Li Shen, Enhua Wu, and Stephen Lin. 2012. A closed-form solution to retinex with nonlocal texture constraints. IEEE Transactions on Pattern Analysis and Machine Intelligence 34, 7 (2012), 1437--1444.
54. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv:1703.10593.

Index Terms

• Computing methodologies

  • Computer graphics

    • Image manipulation

      • Computational photography