research-article

Semantic colorization with internet images

Authors:

Alex Yong-Sang Chia,

Raj Kumar Gupta,

Stephen LinAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 30, Issue 6

Pages 1 - 8

https://doi.org/10.1145/2070781.2024190

Published: 12 December 2011 Publication History

Abstract

Colorization of a grayscale photograph often requires considerable effort from the user, either by placing numerous color scribbles over the image to initialize a color propagation algorithm, or by looking for a suitable reference image from which color information can be transferred. Even with this user supplied data, colorized images may appear unnatural as a result of limited user skill or inaccurate transfer of colors. To address these problems, we propose a colorization system that leverages the rich image content on the internet. As input, the user needs only to provide a semantic text label and segmentation cues for major foreground objects in the scene. With this information, images are downloaded from photo sharing websites and filtered to obtain suitable reference images that are reliable for color transfer to the given grayscale photo. Different image colorizations are generated from the various reference images, and a graphical user interface is provided to easily select the desired result. Our experiments and user study demonstrate the greater effectiveness of this system in comparison to previous techniques.

Supplementary Material

JPG File (a156-chia.jpg)

Download
11.13 KB

WMV File (a156-chia.wmv)

Download
51.93 MB

References

[1]

Belongie, S., Malik, J., and Puzhica, J. 2002. Shape matching and object recognition using shape contexts. IEEE Trans. PAMI 24, 4, 509--532.

Digital Library

[2]

Charpiat, G., Hofmann, M., and Schölkopf, B. 2008. Automatic image colorization via multimodal predictions. In Proc. ECCV, 126--139.

Digital Library

[3]

Chen, T., Cheng, M.-M., Tan, P., Shamir, A., and Hu, S.-M. 2009. Sketch2photo: internet image montage. In ACM Trans. Graph., vol. 28, 1--10.

Digital Library

[4]

Comanicu, D., and Meer, P. 2002. Mean shift: A robust approach toward feature space analysis. In IEEE Trans. PAMI.

Digital Library

[5]

Hasler, D., and Strunk, S. 2003. Measuring colourfulness in natural images. In Human Vision and Electronic Imaging.

[6]

Hays, J., and Efros, A. 2007. Scene completion using millions of photographs. ACM Trans. Graph. 26, 87--94.

Digital Library

[7]

He, K., Sun, J., and Tang, X. 2010. Guided image filtering. In Proc. ECCV, 1--14.

Digital Library

[8]

Hertzmann, A., Jacobs, C. E., Oliver, N., Curless, B., and Salesin, D. H. 2001. Image analogies. In SIGGRAPH.

Digital Library

[9]

Huang, Y.-C., Tung, Y.-S., Chen, J.-C., Wang, S.-W., and Wu, J.-L. 2005. An adaptive edge detection based colorization algorithm and its applications. In Proc. ACM Multimedia.

Digital Library

[10]

Irony, R., Cohen-Or, D., and Lischinski, D. 2005. Colorization by example. In Proc. EGSR, 201--210.

Digital Library

[11]

Komodakis, N., and Tziritas, G. 2007. Approximate labeling via graph-cuts based on linear programming. IEEE Trans. PAMI 29, 8, 1436--1453.

Digital Library

[12]

Levin, A., Lischinski, D., and Weiss, Y. 2004. Colorization using optimization. ACM Trans. Graph. 23, 689--694.

Digital Library

[13]

Li, Y., Sun, J., Tang, C.-K., and Shum, H.-Y. 2004. Lazy snapping. In ACM Trans. Graph., 303--308.

Digital Library

[14]

Liu, T., Sun, J., Zheng, N.-N., Tang, X., and Shum, H.-Y. 2007. Learning to detect a salient object. In Proc. CVPR, 1--8.

[15]

Liu, X., Wan, L., Qu, Y., Wong, T.-T., Lin, S., Leung, C.-S., and Heng, P.-A. 2008. Intrinsic colorization. ACM Trans. Graph. 27, 1--9.

Digital Library

[16]

Lowe, D. G. 1999. Object recognition from local scale-invariant features. In Proc. ICCV, 1150--1157.

Digital Library

[17]

Luan, Q., Wen, F., Cohen-Or, D., Liang, L., Xu, Y.-Q., and Shum, H.-Y. 2007. Natural image colorization. In Proc. EGSR, 309--320.

Digital Library

[18]

Oliva, A., and Torralba, A. 2006. Building the gist of a scene: The role of global image features in recognition. In Visual Perception, Progress in Brain Research.

[19]

Oliva, A., and Torralba, A. 2007. The role of context in object recognition. Trends in Cognitive Sciences 11, 12.

[20]

Qu, Y., Wong, T.-T., and Heng, P.-A. 2006. Manga colorization. ACM Trans. Graph. 25, 1214--1220.

Digital Library

[21]

Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Comput. Graph. Appl. 21, 34--41.

Digital Library

[22]

Rother, C., Kolmogorov, V., and Blake, A. 2004. Grabcut: Interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. 23, 309--314.

Digital Library

[23]

Tai, Y.-W., Jia, J., and Tang, C.-K. 2005. Local color transfer via probabilistic segmentation by expectation-maximization. In Proc. CVPR, 747--754.

Digital Library

[24]

Welsh, T., Ashikhmin, M., and Mueller, K. 2002. Transferring color to greyscale images. In Proc. ACM SIGGRAPH.

Digital Library

[25]

Yatziv, L., and Sapiro, G. 2006. Fast image and video colorization using chrominance blending. In IEEE Trans. PAMI.

[26]

Zhu, J., Hoi, S., Lyu, M., and Yan, S. 2011. Near-duplicate keyframe retrieval by semi-supervised learning and nonrigid image matching. ACM Trans. Multimedia Comput. Commun. Appl..

Digital Library

Cited By

Bai CDang SQian JTian XLi RPeng TLi XYang YDan DYao B(2025)Deep full-color optically-sectioned microscopy with multi-reference colorizationOptics & Laser Technology10.1016/j.optlastec.2024.111577180(111577)Online publication date: Jan-2025
https://doi.org/10.1016/j.optlastec.2024.111577
Zhao WWang LDong MCai Y(2025)CSPDAE: Colorization via SuperPixel Downsampler Denoising AutoEncoder towards enhanced color structural clarityNeurocomputing10.1016/j.neucom.2024.129028618(129028)Online publication date: Feb-2025
https://doi.org/10.1016/j.neucom.2024.129028
Sun QNie YZhang QLi G(2024)Building Coarse to Fine Convex Hulls With Auxiliary Vertices for Palette-Based Image RecoloringIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.329638630:8(5581-5595)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3296386
Show More Cited By

Index Terms

Semantic colorization with internet images
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks
        Scene understanding
    2. Knowledge representation and reasoning
  2. Computer graphics
    1. Image manipulation
2. Theory of computation
  1. Logic

Recommendations

Semantic colorization with internet images
SA '11: Proceedings of the 2011 SIGGRAPH Asia Conference

Colorization of a grayscale photograph often requires considerable effort from the user, either by placing numerous color scribbles over the image to initialize a color propagation algorithm, or by looking for a suitable reference image from which color ...
Colorization in YCbCr color space and its application to JPEG images

This paper presents a colorization method in YCbCr color space, which is based on the maximum a posteriori estimation of a color image given a monochrome image as is our previous method in RGB color space. The presented method in YCbCr space is much ...
Using assorted color spaces and pixel window sizes for colorization of grayscale images
ICWET '10: Proceedings of the International Conference and Workshop on Emerging Trends in Technology

There is no exact solution for colorization of grayscale images. In the initial work done [23], color traits transfer techniques to color grayscale images are proposed. The main focus of the techniques [23] is to minimise the human efforts needed in ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 30, Issue 6

December 2011

678 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2070781

Issue’s Table of Contents

Copyright © 2011 ACM.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 December 2011

Published in TOG Volume 30, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

Korea
Singapore

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

178
Total Citations
View Citations
1,736
Total Downloads

Downloads (Last 12 months)43
Downloads (Last 6 weeks)8

Reflects downloads up to 21 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bai CDang SQian JTian XLi RPeng TLi XYang YDan DYao B(2025)Deep full-color optically-sectioned microscopy with multi-reference colorizationOptics & Laser Technology10.1016/j.optlastec.2024.111577180(111577)Online publication date: Jan-2025
https://doi.org/10.1016/j.optlastec.2024.111577
Zhao WWang LDong MCai Y(2025)CSPDAE: Colorization via SuperPixel Downsampler Denoising AutoEncoder towards enhanced color structural clarityNeurocomputing10.1016/j.neucom.2024.129028618(129028)Online publication date: Feb-2025
https://doi.org/10.1016/j.neucom.2024.129028
Sun QNie YZhang QLi G(2024)Building Coarse to Fine Convex Hulls With Auxiliary Vertices for Palette-Based Image RecoloringIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.329638630:8(5581-5595)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3296386
Jin XLiu LRen XJiang QLee SZhang JYao S(2024)A Restoration Scheme for Spatial and Spectral Resolution of the Panchromatic Image Using the Convolutional Neural NetworkIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing10.1109/JSTARS.2024.335185417(3379-3393)Online publication date: 2024
https://doi.org/10.1109/JSTARS.2024.3351854
Cong XWu YChen QLei C(2024)Automatic Controllable Colorization via Imagination2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00252(2609-2619)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00252
Yang WCai FShu YZhang ZLiu QDing Y(2024)Colorize at Will: Harnessing Diffusion Prior for Image ColorizationIEEE Access10.1109/ACCESS.2024.343548512(107287-107296)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3435485
Yin WLu PPeng XZhao ZYu J(2024)Hierarchical Color Fusion Network (HCFN): Enhancing exemplar-based video colorizationNeurocomputing10.1016/j.neucom.2024.128121598(128121)Online publication date: Sep-2024
https://doi.org/10.1016/j.neucom.2024.128121
Alhalabi WSharma GUpadhyay UKumar AGupta BArya VChui K(2023)Cyberbullying in the MetaverseJournal of Global Information Management10.4018/JGIM.32579331:1(1-25)Online publication date: 10-Jul-2023
https://dl.acm.org/doi/10.4018/JGIM.325793
Wamba SKamdjoug JTchana PTeutio A(2023)Task-Technology Fit and ICT Use in Remote Work Practice During the COVID-19 PandemicJournal of Global Information Management10.4018/JGIM.32409731:1(1-24)Online publication date: 8-Jun-2023
https://dl.acm.org/doi/10.4018/JGIM.324097
Batta AKar ASatpathy S(2023)Cross-Platform Analysis of Seller Performance and Churn for Ecommerce Using Artificial IntelligenceJournal of Global Information Management10.4018/JGIM.32243931:1(1-21)Online publication date: 2-May-2023
https://dl.acm.org/doi/10.4018/JGIM.322439
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents