research-article

Public Access

Real-time user-guided image colorization with learned deep priors

Authors:

Alexei A. EfrosAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 36, Issue 4

Article No.: 119, Pages 1 - 11

https://doi.org/10.1145/3072959.3073703

Published: 20 July 2017 Publication History

Abstract

We propose a deep learning approach for user-guided image colorization. The system directly maps a grayscale image, along with sparse, local user "hints" to an output colorization with a Convolutional Neural Network (CNN). Rather than using hand-defined rules, the network propagates user edits by fusing low-level cues along with high-level semantic information, learned from large-scale data. We train on a million images, with simulated user inputs. To guide the user towards efficient input selection, the system recommends likely colors based on the input image and current user inputs. The colorization is performed in a single feed-forward pass, enabling real-time use. Even with randomly simulated user inputs, we show that the proposed system helps novice users quickly create realistic colorizations, and offers large improvements in colorization quality with just a minute of use. In addition, we demonstrate that the framework can incorporate other user "hints" to the desired colorization, showing an application to color histogram transfer.

Supplementary Material

MP4 File (papers-0527.mp4)

Download
153.32 MB

References

[1]

Xiaobo An and Fabio Pellacini. 2008. AppProp: all-pairs appearance-space edit propagation. 27, 3 (2008), 40.

[2]

Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan Goldman. 2009. Patch-Match: a randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics (TOG) 28, 3 (2009), 24.

Digital Library

[3]

Jonathan T Barron and Ben Poole. 2016. The Fast Bilateral Solver. ECCV.

[4]

Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2015. Material recognition in the wild with the materials in context database. In CVPR. 3479--3487.

[5]

Huiwen Chang, Ohad Fried, Yiming Liu, Stephen DiVerdi, and Adam Finkelstein. 2015. Palette-based photo recoloring. ACM Transactions on Graphics (TOG) 34, 4 (2015), 139.

Digital Library

[6]

Guillaume Charpiat, Matthias Hofmann, and Bernhard Schölkopf. 2008. Automatic image colorization via multimodal predictions. In ECCV. Springer, 126--139.

Digital Library

[7]

Xiaowu Chen, Dongqing Zou, Qinping Zhao, and Ping Tan. 2012. Manifold preserving edit propagation. ACM Transactions on Graphics (TOG) 31, 6 (2012), 132.

Digital Library

[8]

Li Cheng and SVN Vishwanathan. 2007. Learning to compress images and videos. In Proceedings of the 24th international conference on Machine learning. ACM, 161--168.

Digital Library

[9]

Zezhou Cheng, Qingxiong Yang, and Bin Sheng. 2015. Deep Colorization. In ICCV. 415--423.

Digital Library

[10]

Alex Yong-Sang Chia, Shaojie Zhuo, Raj Kumar Gupta, Yu-Wing Tai, Siu-Yeung Cho, Ping Tan, and Stephen Lin. 2011. Semantic colorization with internet images. In ACM Transactions on Graphics (TOG), Vol. 30. ACM, 156.

Digital Library

[11]

Aditya Deshpande, Jason Rock, and David Forsyth. 2015. Learning Large-Scale Automatic Image Colorization. In ICCV. 567--575.

Digital Library

[12]

Yuki Endo, Satoshi Iizuka, Yoshihiro Kanamori, and Jun Mitani. 2016. DeepProp: Extracting Deep Features from a Single Image for Edit Propagation. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 189--201.

[13]

Kevin Frans. 2017. Outline Colorization through Tandem Adversarial Networks. In arXiv:1704.08834.

[14]

Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2016. Image style transfer using convolutional neural networks. In CVPR. 2414--2423.

[15]

Michael Gharbi, Gaurav Chaurasia, Sylvain Paris, and Frédo Durand. 2016. Deep joint demosaicking and denoising. ACM Transactions on Graphics (TOG) 35, 6 (2016), 191.

Digital Library

[16]

Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR.

Digital Library

[17]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In NIPS. 2672--2680.

[18]

Raj Kumar Gupta, Alex Yong-Sang Chia, Deepu Rajan, Ee Sin Ng, and Huang Zhiyong. 2012. Image colorization using similar images. In Proceedings of the 20th ACM international conference on Multimedia. ACM, 369--378.

Digital Library

[19]

Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. 2015. Hyper-columns for object segmentation and fine-grained localization. In CVPR. 447--456.

[20]

Xiaofei He, Ming Ji, and Hujun Bao. 2009. A unified active and semi-supervised learning framework for image compression. In CVPR. IEEE, 65--72.

[21]

Aaron Hertzmann, Charles E Jacobs, Nuria Oliver, Brian Curless, and David H Salesin. 2001. Image analogies. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, 327--340.

Digital Library

[22]

Yi-Chin Huang, Yi-Shin Tung, Jun-Cheng Chen, Sung-Wen Wang, and Ja-Ling Wu. 2005. An adaptive edge detection based colorization algorithm and its applications. In Proceedings of the 13th annual ACM international conference on Multimedia. ACM, 351--354.

Digital Library

[23]

Peter J Huber. 1964. Robust estimation of a location parameter. The Annals of Mathematical Statistics 35, 1 (1964), 73--101.

[24]

Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2016. Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification. SIGGRAPH 35, 4 (2016).

Digital Library

[25]

Preferred Networks Inc. 2017. Paints Chainer. (2017). https://github.com/pfnet/PaintsChainer

[26]

Revital Irony, Daniel Cohen-Or, and Dani Lischinski. 2005. Colorization by example. In Eurographics Symp. on Rendering, Vol. 2. Citeseer.

[27]

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-image translation with conditional adversarial networks. CVPR (2017).

[28]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In NIPS. 1097--1105.

[29]

Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. 2016. Learning Representations for Automatic Colorization. ECCV.

[30]

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278--2324.

[31]

Anat Levin, Dani Lischinski, and Yair Weiss. 2004. Colorization using optimization. In ACM Transactions on Graphics (TOG), Vol. 23. ACM, 689--694.

Digital Library

[32]

Xujie Li, Hanli Zhao, Guizhi Nie, and Hui Huang. 2015. Image recoloring using geodesic distance based color harmonization. Computational Visual Media 1, 2 (2015), 143--155.

[33]

Yuanzhen Li, Edward Adelson, and Aseem Agarwala. 2008. ScribbleBoost: Adding Classification to Edge-Aware Interpolation of Local Image and Video Adjustments. In Computer Graphics Forum, Vol. 27. Wiley Online Library, 1255--1264.

Digital Library

[34]

Xiaopei Liu, Liang Wan, Yingge Qu, Tien-Tsin Wong, Stephen Lin, Chi-Sing Leung, and Pheng-Ann Heng. 2008. Intrinsic colorization. In ACM Transactions on Graphics (TOG), Vol. 27. ACM, 152.

Digital Library

[35]

Yiming Liu, Michael Cohen, Matt Uyttendaele, and Szymon Rusinkiewicz. 2014. AutoStyle: automatic style transfer from image collections to users' images. In Computer Graphics Forum, Vol. 33. Wiley Online Library, 21--31.

Digital Library

[36]

Qing Luan, Fang Wen, Daniel Cohen-Or, Lin Liang, Ying-Qing Xu, and Heung-Yeung Shum. 2007. Natural image colorization. In Proceedings of the 18th Eurographics conference on Rendering Techniques. Eurographics Association, 309--320.

Digital Library

[37]

Yuji Morimoto, Yuichi Taguchi, and Takeshi Naemura. 2009. Automatic colorization of grayscale images using multiple images on the web. In SIGGRAPH'09: Posters. ACM, 32.

Digital Library

[38]

Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei Efros. 2016. Context Encoders: Feature Learning by Inpainting. In CVPR.

[39]

Yingge Qu, Tien-Tsin Wong, and Pheng-Ann Heng. 2006. Manga colorization. In ACM Transactions on Graphics (TOG), Vol. 25. ACM, 1214--1220.

Digital Library

[40]

Erik Reinhard, Michael Ashikhmin, Bruce Gooch, and Peter Shirley. 2001. Color Transfer Between Images. IEEE Comput. Graph. Appl. 21, 5 (Sept. 2001), 34--41.

Digital Library

[41]

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 234--241.

[42]

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, and others. 2015. Imagenet large scale visual recognition challenge. IJCV 115, 3 (2015).

Digital Library

[43]

Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, and James Hays. 2017. Scribbler: Controlling Deep Image Synthesis with Sketch and Color. CVPR (2017).

[44]

Ahmed Selim, Mohamed Elgharib, and Linda Doyle. 2016. Painting style transfer for head portraits using convolutional neural networks. ACM Transactions on Graphics (TOG) 35, 4 (2016), 129.

Digital Library

[45]

Edgar Simo-Serra, Satoshi Iizuka, Kazuma Sasaki, and Hiroshi Ishikawa. 2016. Learning to simplify: fully convolutional networks for rough sketch cleanup. ACM Transactions on Graphics (TOG) 35, 4 (2016), 121.

Digital Library

[46]

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).

[47]

Baoyuan Wang, Yizhou Yu, Tien-Tsin Wong, Chun Chen, and Ying-Qing Xu. 2010. Data-driven image color theme enhancement. In ACM Transactions on Graphics (TOG), Vol. 29. ACM, 146.

Digital Library

[48]

Ting-Chun Wang, Jun-Yan Zhu, Ebi Hiroaki, Manmohan Chandraker, Alexei A Efros, and Ravi Ramamoorthi. 2016. A 4D light-field dataset and CNN architectures for material recognition. In ECCV. Springer, 121--138.

[49]

Tomihisa Welsh, Michael Ashikhmin, and Klaus Mueller. 2002. Transferring color to greyscale images. ACM Transactions on Graphics (TOG) 21, 3 (2002), 277--280.

Digital Library

[50]

Saining Xie and Zhuowen Tu. 2015. Holistically-nested edge detection. In ICCV.

Digital Library

[51]

Kun Xu, Yong Li, Tao Ju, Shi-Min Hu, and Tian-Qiang Liu. 2009. Efficient affinity-based edit propagation using kd tree. ACM Transactions on Graphics (TOG) 28, 5 (2009), 118.

Digital Library

[52]

Li Xu, Qiong Yan, and Jiaya Jia. 2013. A sparse control model for image and video editing. ACM Transactions on Graphics (TOG) 32, 6 (2013), 197.

Digital Library

[53]

Ning Xu, Brian Price, Scott Cohen, Jimei Yang, and Thomas S Huang. 2016. Deep interactive object selection. In CVPR.

[54]

Zhicheng Yan, Hao Zhang, Baoyuan Wang, Sylvain Paris, and Yizhou Yu. 2016. Automatic photo adjustment using deep neural networks. ACM Transactions on Graphics (TOG) 35, 2 (2016), 11.

Digital Library

[55]

Fisher Yu and Vladlen Koltun. 2016. Multi-Scale Context Aggregation by Dilated Convolutions. International Conference on Learning Representations (2016).

[56]

Richard Zhang, Phillip Isola, and Alexei A Efros. 2016. Colorful Image Colorization. ECCV (2016).

[57]

Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. 2014. Learning deep features for scene recognition using places database. In NIPS.

[58]

Jun-Yan Zhu, Philipp Krahenbuhl, Eli Shechtman, and Alexei A Efros. 2015. Learning a discriminative model for the perception of realism in composite images. In CVPR.

[59]

Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros. 2016. Generative visual manipulation on the natural image manifold. (2016), 597--613.

Cited By

Yang PZhao HXie Z(2025)Colorization of fundus images based on advanced degradation modelsJournal of Radiation Research and Applied Sciences10.1016/j.jrras.2024.10128518:1(101285)Online publication date: Mar-2025
https://doi.org/10.1016/j.jrras.2024.101285
Abbasi MVáz PSilva JMartins P(2024)Enhancing Visual Perception in Immersive VR and AR Environments: AI-Driven Color and Clarity Adjustments Under Dynamic Lighting ConditionsTechnologies10.3390/technologies1211021612:11(216)Online publication date: 3-Nov-2024
https://doi.org/10.3390/technologies12110216
Sosnovyy VLashchevska N(2024)DETECTION OF MALICIOUS ACTIVITY USING A NEURAL NETWORK FOR CONTINUOUS OPERATIONCybersecurity: Education, Science, Technique10.28925/2663-4023.2024.23.2132243:23(213-224)Online publication date: 2024
https://doi.org/10.28925/2663-4023.2024.23.213224
Show More Cited By

Index Terms

Real-time user-guided image colorization with learned deep priors
1. Computing methodologies

Recommendations

Deep exemplar-based colorization

We propose the first deep learning approach for exemplar-based local colorization. Given a reference color image, our convolutional neural network directly maps a grayscale image to an output colorized image. Rather than using hand-crafted rules as in ...
User Guided Digital Artwork Colorization
CSAI '19: Proceedings of the 2019 3rd International Conference on Computer Science and Artificial Intelligence

In this paper, we present a user-guided digital artwork colorization approach. We propose a novel network structure for this task. Our network takes grayscale artworks along with sparse user-specified color points as input and outputs color artworks ...
A Novel Approach for Colorization of a Grayscale Image using Soft Computing Techniques

Colorization of grayscale image is a process to convert a grayscale image into a color one. Few research works reported in literature on this but there is hardly any generalized method that successfully colorizes all types of grayscale image. This study ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 36, Issue 4

August 2017

2155 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3072959

Issue’s Table of Contents

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 July 2017

Published in TOG Volume 36, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

BAIR
NSF
Google

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

354
Total Citations
View Citations
2,080
Total Downloads

Downloads (Last 12 months)333
Downloads (Last 6 weeks)49

Reflects downloads up to 21 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yang PZhao HXie Z(2025)Colorization of fundus images based on advanced degradation modelsJournal of Radiation Research and Applied Sciences10.1016/j.jrras.2024.10128518:1(101285)Online publication date: Mar-2025
https://doi.org/10.1016/j.jrras.2024.101285
Abbasi MVáz PSilva JMartins P(2024)Enhancing Visual Perception in Immersive VR and AR Environments: AI-Driven Color and Clarity Adjustments Under Dynamic Lighting ConditionsTechnologies10.3390/technologies1211021612:11(216)Online publication date: 3-Nov-2024
https://doi.org/10.3390/technologies12110216
Sosnovyy VLashchevska N(2024)DETECTION OF MALICIOUS ACTIVITY USING A NEURAL NETWORK FOR CONTINUOUS OPERATIONCybersecurity: Education, Science, Technique10.28925/2663-4023.2024.23.2132243:23(213-224)Online publication date: 2024
https://doi.org/10.28925/2663-4023.2024.23.213224
何荣(2024)Research on Interactive Grayscale Image Coloring Method Based on Improved Swin TransformerSoftware Engineering and Applications10.12677/sea.2024.13506413:05(629-636)Online publication date: 2024
https://doi.org/10.12677/sea.2024.135064
Rajaram SNuman NKumaravel BMarquardt NWilson A(2024)BlendScape: Enabling End-User Customization of Video-Conferencing Environments through Generative AIProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676326(1-19)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676326
Nakashima YYang MBaba Y(2024)SwipeGANSpace: Swipe-to-Compare Image Generation via Efficient Latent Space ExplorationProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645141(675-685)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645141
Shi XLiu MZhou ZNeshati ARossi RZhao J(2024)Exploring Interactive Color Palettes for Abstraction-Driven Exploratory Image ColorizationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642223(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642223
Wang SZhang SLiu DZhong J(2024)UCGAN: a contrastive learning-based unaligned image translation model with importance reweightingJournal of Electronic Imaging10.1117/1.JEI.33.1.01305533:01Online publication date: 1-Jan-2024
https://doi.org/10.1117/1.JEI.33.1.013055
Lee SPark E(2024)AutoCaCoNet: Automatic Cartoon Colorization Network Using Self-Attention GAN, Segmentation, and Color Correction2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW)10.1109/WACVW60836.2024.00050(403-411)Online publication date: 1-Jan-2024
https://doi.org/10.1109/WACVW60836.2024.00050
Suri SMeshry MDavis LShrivastava A(2024)GRIT: GAN Residuals for Paired Image-to-Image Translation2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00489(4953-4963)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00489
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents