research-article

UniColor: A Unified Framework for Multi-Modal Colorization with Transformer

Authors:

Jing LiaoAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 41, Issue 6

Article No.: 205, Pages 1 - 16

https://doi.org/10.1145/3550454.3555471

Published: 30 November 2022 Publication History

Abstract

We propose the first unified framework UniColor to support colorization in multiple modalities, including both unconditional and conditional ones, such as stroke, exemplar, text, and even a mix of them. Rather than learning a separate model for each type of condition, we introduce a two-stage colorization framework for incorporating various conditions into a single model. In the first stage, multi-modal conditions are converted into a common representation of hint points. Particularly, we propose a novel CLIP-based method to convert the text to hint points. In the second stage, we propose a Transformer-based network composed of Chroma-VQGAN and Hybrid-Transformer to generate diverse and high-quality colorization results conditioned on hint points. Both qualitative and quantitative comparisons demonstrate that our method outperforms state-of-the-art methods in every control modality and further enables multi-modal colorization that was not feasible before. Moreover, we design an interactive interface showing the effectiveness of our unified framework in practical usage, including automatic colorization, hybrid-control colorization, local recolorization, and iterative color editing. Our code and models are available at https://luckyhzt.github.io/unicolor.

Supplemental Material

MP4 File

presentation

Download
412.49 MB

References

[1]

Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. 2012. SLIC Superpixels Compared to State-of-the-Art Superpixel Methods. IEEE Transactions on Pattern Analysis and Machine Intelligence 34, 11 (2012), 2274--2282.

Digital Library

[2]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015.

[3]

Hyojin Bahng, Seungjoo Yoo, Wonwoong Cho, David Keetae Park, Ziming Wu, Xiaojuan Ma, and Jaegul Choo. 2018. Coloring with Words: Guiding Image Colorization Through Text-Based Palette Generation. In ECCV 2018. 443--459.

[4]

Yun Cao, Zhiming Zhou, Weinan Zhang, and Yong Yu. 2017. Unsupervised Diverse Colorization via Generative Adversarial Networks. In Machine Learning and Knowledge Discovery in Databases. 151--166.

[5]

Jianbo Chen, Yelong Shen, Jianfeng Gao, Jingjing Liu, and Xiaodong Liu. 2018. Language-Based Image Editing with Recurrent Attentive Models. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018), 8721--8729.

[6]

Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. 2020. Generative Pretraining From Pixels. In Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 119). PMLR, 1691--1703.

[7]

Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating Long Sequences with Sparse Transformers. URL https://openai.com/blog/sparse-transformers (2019).

[8]

Deoldify. 2018. Deoldify: A Deep Learning based Project for Colorizing and Restoring Old Images and Video. https://github.com/jantic/DeOldify.

[9]

Aditya Deshpande, Jiajun Lu, Mao Chuang Yeh, Min Jin Chong, and David Forsyth. 2017. Learning diverse image colorization. In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 (Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017). 2877--2885.

[10]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT 2019, Minneapolis, MN, USA, June, 2019, Volume 1. 4171--4186.

[11]

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3--7, 2021.

[12]

Yuki Endo, Satoshi Iizuka, Yoshihiro Kanamori, and Jun Mitani. 2016. DeepProp: Extracting Deep Features from a Single Image for Edit Propagation. Computer Graphics Forum 35 (05 2016), 189--201.

[13]

Patrick Esser, Robin Rombach, and Bjorn Ommer. 2021. Taming Transformers for High-Resolution Image Synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 12873--12883.

[14]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems, Vol. 27.

Digital Library

[15]

David Hasler and Sabine E. Suesstrunk. 2003. Measuring colorfulness in natural images. In Human Vision and Electronic Imaging VIII, Vol. 5007. International Society for Optics and Photonics, SPIE, 87--95.

[16]

Mingming He, Dongdong Chen, Jing Liao, Pedro V Sander, and Lu Yuan. 2018. Deep exemplar-based colorization. ACM Transactions on Graphics (TOG) 37, 4 (2018), 47.

Digital Library

[17]

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances in Neural Information Processing Systems, Vol. 30.

[18]

Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020a. Denoising Diffusion Probabilistic Models. In Advances in Neural Information ProcessingSystems, Vol. 33. 6840--6851.

[19]

Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. 2020b. Axial Attention in Multidimensional Transformers. https://openreview.net/forum?id=H1e5GJBtDr

[20]

Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2016. Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification. ACM Transactions on Graphics (Proc. of SIGGRAPH 2016) 35, 4 (2016).

Digital Library

[21]

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5967--5976.

[22]

Xin Jin, Zhonglan Li, Ke Liu, Dongqing Zou, Xiaodong Li, Xingfan Zhu, Ziyin Zhou, Qi song Sun, and Qingyu Liu. 2021. Focusing on Persons: Colorizing Old Images Learning from Modern Historical Movies. Proceedings of the 29th ACM International Conference on Multimedia (2021).

Digital Library

[23]

Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. ReferItGame: Referring to Objects in Photographs of Natural Scenes. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). 787--798.

[24]

Leila Kiani, Masoudnia Saeed, and Hossein Nezamabadi-pour. 2020. Image Colorization Using Generative Adversarial Networks and Transfer Learning. 2020 International Conference on Machine Vision and Image Processing (MVIP) (2020), 1--6.

[25]

Yoon Kim, Carl Denton, Luong Hoang, and Alexander M. Rush. 2017. Structured Attention Networks. CoRR abs/1702.00887 (2017). arXiv:1702.00887

[26]

Manoj Kumar, Dirk Weissenborn, and Nal Kalchbrenner. 2021. Colorization Transformer. In International Conference on Learning Representations.

[27]

Anat Levin, Dani Lischinski, and Yair Weiss. 2004. Colorization Using Optimization (SIGGRAPH '04). ACM, 689--694.

Digital Library

[28]

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In Computer Vision - ECCV 2014. 740--755.

[29]

Qing Luan, Fang Wen, Daniel Cohen-Or, Lin Liang, Ying-Qing Xu, and Heung-Yeung Shum. 2007. Natural Image Colorization. In Rendering Techniques.

[30]

Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji. 2020. Multi-Task Collaborative Network for Joint Referring Expression Comprehension and Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[31]

Varun Manjunatha, Mohit Iyyer, Jordan Boyd-Graber, and Larry Davis. 2018. Learning to Color from Language. In North American Chapter of the Association for Computational Linguistics.

[32]

Roey Mechrez, Itamar Talmi, and Lihi Zelnik-Manor. 2018. The Contextual Loss for Image Transformation with Non-Aligned Data. In Proceedings of the European Conference on Computer Vision (ECCV).

Digital Library

[33]

Safa Messaoud, David A. Forsyth, and Alexander G. Schwing. 2018. Structural Consistency and Controllability for Diverse Colorization. In ECCV (6). 603--619.

[34]

Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. 2018. Image Transformer. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 80). PMLR, 4055--4064.

[35]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In ICML.

[36]

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-Shot Text-to-Image Generation. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139). 8821--8831.

[37]

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) 115, 3 (2015), 211--252.

Digital Library

[38]

Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi. 2021. Palette: Image-to-Image Diffusion Models.

[39]

Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7--9, 2015, Conference Track Proceedings.

[40]

Jheng-Wei Su, Hung-Kuo Chu, and Jia-Bin Huang. 2020. Instance-aware Image Colorization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, Vol. 30.

[42]

Patricia Vitoria, Lara Raad, and Coloma Ballester. 2020. ChromaGAN: Adversarial Picture Colorization with Semantic Class Distribution. In The IEEE Winter Conference on Applications of Computer Vision. 2445--2454.

[43]

Zhou Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. 2004. Image Quality Assessment: From Error Visibility to Structural Similarity. Trans. Img. Proc. 13, 4 (apr 2004), 600--612.

Digital Library

[44]

Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. 2022. CRIS: CLIP-Driven Referring Image Segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.

[45]

Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. 2020. Scaling Autoregressive Video Models. In International Conference on Learning Representations.

[46]

Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. 2021a. NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion.

[47]

Yanze Wu, Xintao Wang, Yu Li, Honglun Zhang, Xun Zhao, and Ying Shan. 2021b. Towards Vivid and Diverse Image Colorization with Generative Color Prior. In International Conference on Computer Vision (ICCV).

[48]

Yi Xiao, Peiyao Zhou, Yan Zheng, and Chi-Sing Leung. 2019. Interactive Deep Colorization Using Simultaneous Global and Local Inputs. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 1887--1891.

[49]

Zhongyou Xu, Tingting Wang, Faming Fang, Yun Sheng, and Guixu Zhang. 2020. Stylization-Based Architecture for Fast Deep Exemplar Colorization. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 9360--9369.

[50]

Bo Zhang, Mingming He, Jing Liao, Pedro V Sander, Lu Yuan, Amine Bermak, and Dong Chen. 2019. Deep exemplar-based video colorization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8052--8061.

[51]

Richard Zhang, Phillip Isola, and Alexei A Efros. 2016. Colorful Image Colorization. In ECCV.

[52]

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR.

[53]

Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S Lin, Tianhe Yu, and Alexei A Efros. 2017. Real-Time User-Guided Image Colorization with Learned Deep Priors. ACM Transactions on Graphics (TOG) 9, 4 (2017).

[54]

Zhu Zhang, Jianxin Ma, Chang Zhou, Rui Men, Zhikang Li, Ming Ding, Jie Tang, Jingren Zhou, and Hongxia Yang. 2021. UFC-BERT: Unifying Multi-Modal Controls for Conditional Image Synthesis. In Advances in Neural Information Processing Systems.

[55]

Nanxuan Zhao, Quanlong Zheng, Jing Liao, Ying Cao, Hanspeter Pfister, and Rynson W.H. Lau. 2021. Selective Region-based Photo Color Adjustment for Graphic Designs. ACM Transactions on Graphics 40 (2021). Issue 2.

Cited By

Yang GYu CWang XFang XZhang J(2025)LVAST: a lightweight vision transformer for effective arbitrary style transferThe Journal of Supercomputing10.1007/s11227-024-06787-281:1Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1007/s11227-024-06787-2
Cheng YWan RWeng SZhu CChang YShi BWooldridge MDy JNatarajan S(2024)Colorizing monochromatic radiance fieldsProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i2.27895(1317-1325)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i2.27895
Yuan CYang ZXiaojuan ZXiaoping L(2024)Sketch colorization with finite color space priorJournal of Image and Graphics10.11834/jig.23018929:4(978-988)Online publication date: 2024
https://doi.org/10.11834/jig.230189
Show More Cited By

Index Terms

UniColor: A Unified Framework for Multi-Modal Colorization with Transformer
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
    2. Image manipulation
      1. Computational photography
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks

Recommendations

Color recovery of black-and-white halftoned images via categorized color-embedding look-up tables

A new method of recovering the original colors of black-and-white (B&W) halftoned images with homogeneous dot patterns is proposed. The conventional inverse halftoning method, which uses a look-up table (LUT), can establish the relation between the ...
Colorization in YCbCr color space and its application to JPEG images

This paper presents a colorization method in YCbCr color space, which is based on the maximum a posteriori estimation of a color image given a monochrome image as is our previous method in RGB color space. The presented method in YCbCr space is much ...
Using assorted color spaces and pixel window sizes for colorization of grayscale images
ICWET '10: Proceedings of the International Conference and Workshop on Emerging Trends in Technology

There is no exact solution for colorization of grayscale images. In the initial work done [23], color traits transfer techniques to color grayscale images are proposed. The main focus of the techniques [23] is to minimise the human efforts needed in ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 41, Issue 6

December 2022

1428 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3550454

Issue’s Table of Contents

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 November 2022

Published in TOG Volume 41, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Hong Kong Research Grants Council (RGC) GRF Scheme

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

28
Total Citations
View Citations
383
Total Downloads

Downloads (Last 12 months)131
Downloads (Last 6 weeks)20

Reflects downloads up to 21 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yang GYu CWang XFang XZhang J(2025)LVAST: a lightweight vision transformer for effective arbitrary style transferThe Journal of Supercomputing10.1007/s11227-024-06787-281:1Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1007/s11227-024-06787-2
Cheng YWan RWeng SZhu CChang YShi BWooldridge MDy JNatarajan S(2024)Colorizing monochromatic radiance fieldsProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i2.27895(1317-1325)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i2.27895
Yuan CYang ZXiaojuan ZXiaoping L(2024)Sketch colorization with finite color space priorJournal of Image and Graphics10.11834/jig.23018929:4(978-988)Online publication date: 2024
https://doi.org/10.11834/jig.230189
Li YBai YYang SLiu JCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)COCO-LC: Colorfulness Controllable Language-based ColorizationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680662(10939-10947)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680662
Bozic VDjelouah AZhang YTimofte RGross MSchroers C(2024)Versatile Vision Foundation Model for Image and Video ColorizationACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657509(1-11)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657509
Lee GShin SNa TWoo S(2024)Real-Time User-guided Adaptive Colorization with Vision Transformer2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00054(473-482)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00054
Wang HChai XWang YZhang YXie RSong L(2024)Multimodal Semantic-Aware Automatic Colorization with Diffusion Prior2024 IEEE International Conference on Multimedia and Expo Workshops (ICMEW)10.1109/ICMEW63481.2024.10645422(1-6)Online publication date: 15-Jul-2024
https://doi.org/10.1109/ICMEW63481.2024.10645422
Lee HKang KOk JCho S(2024)CLIPtone: Unsupervised Learning for Text-Based Image Tone Adjustment2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00284(2942-2951)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00284
Cong XWu YChen QLei C(2024)Automatic Controllable Colorization via Imagination2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00252(2609-2619)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00252
Yang WCai FShu YZhang ZLiu QDing Y(2024)Colorize at Will: Harnessing Diffusion Prior for Image ColorizationIEEE Access10.1109/ACCESS.2024.343548512(107287-107296)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3435485
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents