research-article

Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification

Authors:

Satoshi Iizuka,

Edgar Simo-Serra,

Hiroshi IshikawaAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 35, Issue 4

Article No.: 110, Pages 1 - 11

https://doi.org/10.1145/2897824.2925974

Published: 11 July 2016 Publication History

Abstract

We present a novel technique to automatically colorize grayscale images that combines both global priors and local image features. Based on Convolutional Neural Networks, our deep network features a fusion layer that allows us to elegantly merge local information dependent on small image patches with global priors computed using the entire image. The entire framework, including the global and local priors as well as the colorization model, is trained in an end-to-end fashion. Furthermore, our architecture can process images of any resolution, unlike most existing approaches based on CNN. We leverage an existing large-scale scene classification database to train our model, exploiting the class labels of the dataset to more efficiently and discriminatively learn the global priors. We validate our approach with a user study and compare against the state of the art, where we show significant improvements. Furthermore, we demonstrate our method extensively on many different types of images, including black-and-white photography from over a hundred years ago, and show realistic colorizations.

Supplementary Material

MP4 File (a110.mp4)

Download
312.83 MB

References

[1]

An, X., and Pellacini, F. 2008. AppProp: All-pairs appearance-space edit propagation. ACM Trans. Graph. 27, 3 (Aug.), 40:1--40:9.

Digital Library

[2]

Bell, S., and Bala, K. 2015. Learning visual similarity for product design with convolutional neural networks. ACM Trans. on Graphics (SIGGRAPH) 34, 4.

Digital Library

[3]

Bromley, J., Guyon, I., Lecun, Y., Säckinger, E., and Shah, R. 1994. Signature verification using a "siamese" time delay neural network. In NIPS.

[4]

Charpiat, G., Hofmann, M., and Schölkopf, B. 2008. Automatic image colorization via multimodal predictions. In ECCV.

Digital Library

[5]

Chen, X., Zou, D., Zhao, Q., and Tan, P. 2012. Manifold preserving edit propagation. ACM Trans. Graph. 31, 6 (Nov.), 132:1--132:7.

Digital Library

[6]

Cheng, Z., Yang, Q., and Sheng, B. 2015. Deep colorization. In Proceedings of ICCV 2015, 29--43.

Digital Library

[7]

Chia, A. Y.-S., Zhuo, S., Gupta, R. K., Tai, Y.-W., Cho, S.-Y., Tan, P., and Lin, S. 2011. Semantic colorization with internet images. ACM Trans. Graph. 30, 6, 156:1--156:8.

Digital Library

[8]

Dong, C., Loy, C. C., He, K., and Tang, X. 2016. Image super-resolution using deep convolutional networks. PAMI 38, 2, 295--307.

Digital Library

[9]

Eigen, D., and Fergus, R. 2015. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In ICCV.

Digital Library

[10]

Fischer, P., Dosovitskiy, A., Ilg, E., Häusser, P., Hazirbas, C., Golkov, V., van der Smagt, P., Cremers, D., and Brox, T. 2015. Flownet: Learning optical flow with convolutional networks.

[11]

Fukushima, K. 1988. Neocognitron: A hierarchical neural network capable of visual pattern recognition. Neural networks 1, 2, 119--130.

[12]

Gupta, R. K., Chia, A. Y.-S., Rajan, D., Ng, E. S., and Zhiyong, H. 2012. Image colorization using similar images. In ACM International Conference on Multimedia, 369--378.

Digital Library

[13]

Huang, Y.-C., Tung, Y.-S., Chen, J.-C., Wang, S.-W., and Wu, J.-L. 2005. An adaptive edge detection based colorization algorithm and its applications. In ACM International Conference on Multimedia, 351--354.

Digital Library

[14]

Ioffe, S., and Szegedy, C. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML.

[15]

Irony, R., Cohen-Or, D., and Lischinski, D. 2005. Colorization by example. In Eurographics Conference on Rendering Techniques, 201--210.

Digital Library

[16]

Krizhevsky, A., Sutskever, I., and Hinton, G. E. 2012. Imagenet classification with deep convolutional neural networks. In NIPS.

[17]

LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86, 11, 2278--2324.

[18]

Levin, A., Lischinski, D., and Weiss, Y. 2004. Colorization using optimization. ACM Transactions on Graphics 23, 689--694.

Digital Library

[19]

Li, Y., Ju, T., and Hu, S.-M. 2010. Instant propagation of sparse edits on images and videos. Computer Graphics Forum 29, 7, 2049--2054.

[20]

Liu, X., Wan, L., Qu, Y., Wong, T.-T., Lin, S., Leung, C.-S., and Heng, P.-A. 2008. Intrinsic colorization. ACM Transactions on Graphics (SIGGRAPH Asia 2008 issue) 27, 5 (December), 152:1--152:9.

Digital Library

[21]

Long, J., Shelhamer, E., and Darrell, T. 2015. Fully convolutional networks for semantic segmentation. In CVPR.

[22]

Luan, Q., Wen, F., Cohen-Or, D., Liang, L., Xu, Y.-Q., and Shum, H.-Y. 2007. Natural image colorization. In Eurographics Conference on Rendering Techniques, 309--320.

Digital Library

[23]

Pitié, F., Kokaram, A. C., and Dahyot, R. 2007. Automated colour grading using colour distribution transfer. Comput. Vis. Image Underst. 107, 1--2 (July), 123--137.

Digital Library

[24]

Qu, Y., Wong, T.-T., and Heng, P.-A. 2006. Manga colorization. ACM Trans. Graph. 25, 3 (July), 1214--1220.

Digital Library

[25]

Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Computer Graphics and Applications 21, 5 (sep), 34--41.

Digital Library

[26]

Rumelhart, D., Hinton, G., and Williams, R. 1986. Learning representations by back-propagating errors. In Nature.

[27]

Shen, W., Wang, X., Wang, Y., Bai, X., and Zhang, Z. 2015. Deepcontour: A deep convolutional feature learned by positive-sharing loss for contour detection. In CVPR.

[28]

Simonyan, K., and Zisserman, A. 2015. Very deep convolutional networks for large-scale image recognition. In ICLR.

[29]

Springenberg, J. T., Dosovitskiy, A., Brox, T., and Riedmiller, M. A. 2015. Striving for simplicity: The all convolutional net. In ICLR Workshop Track.

[30]

Sýkora, D., Dingliana, J., and Collins, S. 2009. Lazy-Brush: Flexible painting tool for hand-drawn cartoons. Computer Graphics Forum 28, 2, 599--608.

[31]

Tai, Y., Jia, J., and Tang, C. 2005. Local color transfer via probabilistic segmentation by expectation-maximization. In CVPR, 747--754.

Digital Library

[32]

Wang, L., Guo, S., Huang, W., and Qiao, Y. 2015. Places205-vggnet models for scene recognition. CoRR abs/1508.01667.

[33]

Wang, L., Lee, C., Tu, Z., and Lazebnik, S. 2015. Training deeper convolutional networks with deep supervision. CoRR abs/1505.02496.

[34]

Wang, X., Fouhey, D. F., and Gupta, A. 2015. Designing deep networks for surface normal estimation. In CVPR.

[35]

Welsh, T., Ashikhmin, M., and Mueller, K. 2002. Transferring color to greyscale images. ACM Trans. Graph. 21, 3 (July), 277--280.

Digital Library

[36]

Wu, F., Dong, W., Kong, Y., Mei, X., Paul, J.-C., and Zhang, X. 2013. Content-based colour transfer. Computer Graphics Forum 32, 1, 190--203.

[37]

Xu, K., Li, Y., Ju, T., Hu, S.-M., and Liu, T.-Q. 2009. Efficient affinity-based edit propagation using k-d tree. ACM Trans. Graph. 28, 5 (Dec.), 118:1--118:6.

Digital Library

[38]

Xu, L., Yan, Q., and Jia, J. 2013. A sparse control model for image and video editing. ACM Trans. Graph. 32, 6 (Nov.), 197:1--197:10.

Digital Library

[39]

Yatziv, L., Yatziv, L., Sapiro, G., and Sapiro, G. 2004. Fast image and video colorization using chrominance blending. IEEE Transaction on Image Processing 15, 2006.

Digital Library

[40]

Zeiler, M. D. 2012. ADADELTA: an adaptive learning rate method. CoRR abs/1212.5701.

[41]

Zhou, B., Lapedriza, A., Xiao, J., Torralba, A., and Oliva, A. 2014. Learning deep features for scene recognition using places database. In NIPS.

Digital Library

Cited By

Niu CTao MBao B(2025)SEMACOL: Semantic-enhanced multi-scale approach for text-guided grayscale image colorizationPattern Recognition10.1016/j.patcog.2024.111203160(111203)Online publication date: Apr-2025
https://doi.org/10.1016/j.patcog.2024.111203
Zhao WWang LDong MCai Y(2025)CSPDAE: Colorization via SuperPixel Downsampler Denoising AutoEncoder towards enhanced color structural clarityNeurocomputing10.1016/j.neucom.2024.129028618(129028)Online publication date: Feb-2025
https://doi.org/10.1016/j.neucom.2024.129028
Yang PZhao HXie Z(2025)Colorization of fundus images based on advanced degradation modelsJournal of Radiation Research and Applied Sciences10.1016/j.jrras.2024.10128518:1(101285)Online publication date: Mar-2025
https://doi.org/10.1016/j.jrras.2024.101285
Show More Cited By

Index Terms

Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification
1. Computing methodologies
  1. Computer graphics
    1. Image manipulation
      1. Image processing
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks

Recommendations

Color recovery of black-and-white halftoned images via categorized color-embedding look-up tables

A new method of recovering the original colors of black-and-white (B&W) halftoned images with homogeneous dot patterns is proposed. The conventional inverse halftoning method, which uses a look-up table (LUT), can establish the relation between the ...
Colorization in YCbCr color space and its application to JPEG images

This paper presents a colorization method in YCbCr color space, which is based on the maximum a posteriori estimation of a color image given a monochrome image as is our previous method in RGB color space. The presented method in YCbCr space is much ...
Using assorted color spaces and pixel window sizes for colorization of grayscale images
ICWET '10: Proceedings of the International Conference and Workshop on Emerging Trends in Technology

There is no exact solution for colorization of grayscale images. In the initial work done [23], color traits transfer techniques to color grayscale images are proposed. The main focus of the techniques [23] is to minimise the human efforts needed in ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 35, Issue 4

July 2016

1396 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2897824

Issue’s Table of Contents

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 July 2016

Published in TOG Volume 35, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

607
Total Citations
View Citations
4,967
Total Downloads

Downloads (Last 12 months)303
Downloads (Last 6 weeks)38

Reflects downloads up to 21 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Niu CTao MBao B(2025)SEMACOL: Semantic-enhanced multi-scale approach for text-guided grayscale image colorizationPattern Recognition10.1016/j.patcog.2024.111203160(111203)Online publication date: Apr-2025
https://doi.org/10.1016/j.patcog.2024.111203
Zhao WWang LDong MCai Y(2025)CSPDAE: Colorization via SuperPixel Downsampler Denoising AutoEncoder towards enhanced color structural clarityNeurocomputing10.1016/j.neucom.2024.129028618(129028)Online publication date: Feb-2025
https://doi.org/10.1016/j.neucom.2024.129028
Yang PZhao HXie Z(2025)Colorization of fundus images based on advanced degradation modelsJournal of Radiation Research and Applied Sciences10.1016/j.jrras.2024.10128518:1(101285)Online publication date: Mar-2025
https://doi.org/10.1016/j.jrras.2024.101285
Anwar STahir MLi CMian AKhan FMuzaffar A(2025)Image colorization: A survey and datasetInformation Fusion10.1016/j.inffus.2024.102720114(102720)Online publication date: Feb-2025
https://doi.org/10.1016/j.inffus.2024.102720
Zhang SChen Y(2025)ATM-DEN: Image Inpainting via attention transfer module and Decoder-Encoder networkSignal Processing: Image Communication10.1016/j.image.2025.117268133(117268)Online publication date: Apr-2025
https://doi.org/10.1016/j.image.2025.117268
Ohayon GMichaeli TElad MSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)The perception-robustness tradeoff in deterministic image restorationProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693636(38599-38638)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693636
Saini BVenkatesh DGanesh AParameswaran APatil SKamat PChoudhury T(2024)Colorizing Multi-Modal Medical Data: An Autoencoder-based Approach for Enhanced Anatomical Information in X-ray ImagesEAI Endorsed Transactions on Pervasive Health and Technology10.4108/eetpht.10.554010Online publication date: 25-Mar-2024
https://doi.org/10.4108/eetpht.10.5540
Watanabe YTogo RMaeda KOgawa THaseyama M(2024)Text-Guided Image Editing Based on Post Score for Gaining Attention on Social MediaSensors10.3390/s2403092124:3(921)Online publication date: 31-Jan-2024
https://doi.org/10.3390/s24030921
Li SMa DDing YXian YZhang T(2024)DBSF-Net: Infrared Image Colorization Based on the Generative Adversarial Model with Dual-Branch Feature Extraction and Spatial-Frequency-Domain DiscriminationRemote Sensing10.3390/rs1620376616:20(3766)Online publication date: 10-Oct-2024
https://doi.org/10.3390/rs16203766
Zhang GHong XLiu YQian YCai X(2024)Video Colorization Based on Variational AutoencoderElectronics10.3390/electronics1312241213:12(2412)Online publication date: 20-Jun-2024
https://doi.org/10.3390/electronics13122412
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents