research-article

Adversarial Colorization of Icons Based on Contour and Color Conditions

Authors:

Chien-Hsun Lai,

Sai-Keung Wong,

Yu-Shuen WangAuthors Info & Claims

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

Pages 683 - 691

https://doi.org/10.1145/3343031.3351041

Published: 15 October 2019 Publication History

Abstract

We present a system to help designers create icons that are widely used in banners, signboards, billboards, homepages, and mobile apps. Designers are tasked with drawing contours, whereas our system colorizes contours in different styles. This goal is achieved by training a dual conditional generative adversarial network (GAN) on our collected icon dataset. One condition requires the generated image and the drawn contour to possess a similar contour, while the other anticipates the image and the referenced icon to be similar in color style. Accordingly, the generator takes a contour image and a man-made icon image to colorize the contour, and then the discriminators determine whether the result fulfills the two conditions. The trained network is able to colorize icons demanded by designers and greatly reduces their workload. For the evaluation, we compared our dual conditional GAN to several state-of-the-art techniques. Experiment results demonstrate that our network is over the previous networks. Finally, we will provide the source code, icon dataset, and trained network for public use.

Supplementary Material

ZIP File (fp814aux.zip)

The auxiliary material contains a demo video of our system.

Download
9.01 MB

References

[1]

Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein Generative Adversarial Networks. Proceedings of the 34th International Conference on Machine Learning, Vol. 70 (2017), 214--223.

[2]

David Berthelot, Thomas Schumm, and Luke Metz. 2017. BEGAN: boundary equilibrium generative adversarial networks. arXiv preprint arXiv:1703.10717 (2017).

[3]

Navaneeth Bodla, Gang Hua, and Rama Chellappa. 2018. Semi-supervised FusedGAN for Conditional Image Generation. In ECCV . https://doi.org/10.1145/2897824.2925974

[4]

Yun Cao, Zhiming Zhou, Weinan Zhang, and Yong Yu. 2017. Unsupervised diverse colorization via generative adversarial networks. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 151--166.

[5]

Yuanzheng Ci, Xinzhu Ma, Zhihui Wang, Haojie Li, and Zhongxuan Luo. 2018. User-Guided Deep Anime Line Art Colorization with Conditional Adversarial Networks. In Proceedings of ACM Multimedia. 1--6.

Digital Library

[6]

Jeff Donahue, Philipp Kr"ahenbühl, and Trevor Darrell. 2017. Adversarial feature learning. ICLR .

[7]

Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. 2017. Adversarially learned inference. In International Conference on Learning Representations .

[8]

Chie Furusawa, Kazuyuki Hiroshiba, Keisuke Ogaki, and Yuri Odagiri. 2017. Comicolorization: semi-automatic manga colorization. In SIGGRAPH Asia 2017 Technical Briefs. ACM, 12.

[9]

Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics. 249--256.

[10]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672--2680.

[11]

Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. 2017. Improved training of wasserstein gans. In Advances in Neural Information Processing Systems. 5767--5777.

[12]

D Hawthorn. 2000. Possible implications of aging for interface designers. Interacting with Computers, Vol. 12, 5 (2000), 507 -- 528. https://doi.org/10.1016/S0953--5438(99)00021--1

[13]

Steven Heim. 2007. The resonant interface: HCI foundations for interaction design .Addison-Wesley Longman Publishing Co., Inc.

[14]

Paulina Hensman and Kiyoharu Aizawa. 2017. cGAN-based Manga Colorization Using a Single Training Image. In Document Analysis and Recognition (ICDAR), 2017 14th IAPR International Conference on, Vol. 3. IEEE, 72--77.

[15]

William K Horton. 1994. The icon book: Visual symbols for computer systems and documentation .John Wiley & Sons, Inc.

[16]

Shih-Miao Huang, Kong-King Shieh, and Chai-Fen Chi. 2002. Factors affecting the design of computer icons. International Journal of Industrial Ergonomics, Vol. 29, 4 (2002), 211 -- 218. https://doi.org/10.1016/S0169--8141(01)00064--6

[17]

Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. 2018. Multimodal unsupervised image-to-image translation. In Proceedings of the European Conference on Computer Vision (ECCV). 172--189.

Digital Library

[18]

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017), 5967--5976.

[19]

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

[20]

Shigenobu Kobayashi. 1992. Color Image Scale .Kodansha Amer Inc.

[21]

J. Krause. 2002. Color Index: Over 1,100 Color Combinations, CMYK and RGB Formulas, for Print and Web Media .F&W Publications, Inc.

[22]

Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic Denoyer, et almbox. 2017. Fader networks: Manipulating images by sliding attributes. In Advances in Neural Information Processing Systems. 5967--5976.

[23]

Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. 2016. Autoencoding Beyond Pixels Using a Learned Similarity Metric. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48. 1558--1566.

[24]

Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et almbox. 2017. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In CVPR, Vol. 2. 4.

[25]

Chuan Li and Michael Wand. 2016. Precomputed real-time texture synthesis with markovian generative adversarial networks. In European Conference on Computer Vision. Springer, 702--716.

[26]

Yan-Peng Lim and Peter Charles Woods. 2010. Visual Information Communication .Springer, Chapter: Experimental Color in Computer Icons, 149--158.

[27]

Jianxin Lin, Yingce Xia, Tao Qin, Zhibo Chen, and Tie-Yan Liu. 2018. Conditional Image-to-Image Translation. In CVPR. 5524--5532.

[28]

Tomas Lindberg and Risto N"as"anen. 2003. The effect of icon spacing and size on the speed of icon processing in the human visual system. Displays, Vol. 24, 3 (2003), 111--120.

[29]

Yifan Liu, Zengchang Qin, Tao Wan, and Zhenbo Luo. 2018. Auto-painter: Cartoon image generation from sketch by using conditional Wasserstein generative adversarial networks. Neurocomputing, Vol. 311 (2018), 78--87.

Digital Library

[30]

Michael Mathieu, Camille Couprie, and Yann LeCun. 2016. Deep multi-scale video prediction beyond mean square error. In ICLR .

[31]

Mehdi Mirza and Simon Osindero. 2014. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014).

[32]

Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. 2018a. Spectral Normalization for Generative Adversarial Networks. In ICLR .

[33]

Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. 2018b. Spectral normalization for generative adversarial networks. In ICLR .

[34]

Yingge Qu, Tien-Tsin Wong, and Pheng-Ann Heng. 2006. Manga Colorization. ACM Trans. Graph., Vol. 25, 3 (July 2006), 1214--1220. https://doi.org/10.1145/1141911.1142017

Digital Library

[35]

Alec Radford, Luke Metz, and Soumith Chintala. 2016. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR .

[36]

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved techniques for training gans. In Advances in Neural Information Processing Systems. 2234--2242.

[37]

Yaniv Taigman, Adam Polyak, and Lior Wolf. 2017. Unsupervised cross-domain image generation. In ICLR .

[38]

Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. 2018. Mocogan: Decomposing motion and content for video generation. In CVPR .

[39]

Xiaolong Wang and Abhinav Gupta. 2016. Generative image modeling using style and structure adversarial networks. In European Conference on Computer Vision. Springer, 318--335.

[40]

Wei Xiong, Wenhan Luo, Lin Ma, Wei Liu, and Jiebo Luo. 2018. Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2364--2373.

[41]

Xinchen Yan, Jimei Yang, Kihyuk Sohn, and Honglak Lee. 2016. Attribute2image: Conditional image generation from visual attributes. In European Conference on Computer Vision. Springer, 776--791.

[42]

Zili Yi, Hao (Richard) Zhang, Ping Tan, and Minglun Gong. 2017. DualGAN: Unsupervised Dual Learning for Image-to-Image Translation. In ICCV . 2868--2876.

[43]

Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. 2017b. StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks. In ICCV.

[44]

Lvmin Zhang, Yi Ji, and Xin Lin. 2017a. Style transfer for anime sketches with enhanced residual U-net and auxiliary classifier gan. arXiv preprint arXiv:1706.03319 (2017).

[45]

Junbo Zhao, Michael Mathieu, and Yann LeCun. 2016. Energy-based generative adversarial network. arXiv preprint arXiv:1609.03126 (2016).

[46]

Jun-Yan Zhu, Philipp Kr"ahenbühl, Eli Shechtman, and Alexei A Efros. 2016. Generative visual manipulation on the natural image manifold. In European Conference on Computer Vision. Springer, 597--613.

[47]

Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. 2017a. Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. 2017 IEEE International Conference on Computer Vision (ICCV) (2017), 2242--2251.

[48]

Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. 2017b. Toward multimodal image-to-image translation. In Advances in Neural Information Processing Systems. 465--476.

Cited By

Miyauchi KOrihara RSei YTahara YOhsuga A(2024)Hollowed-Out Icon Colorization with Pre-trained Large-Scale Image Generation Model事前学習済み大規模画像生成モデルを用いた中空構造アイコン線画彩色Transactions of the Japanese Society for Artificial Intelligence10.1527/tjsai.39-6_AG24-E39:6(AG24-E_1-11)Online publication date: 1-Nov-2024
https://doi.org/10.1527/tjsai.39-6_AG24-E
Miyauchi KOrihara RSei YTahara YOhsuga A(2024)Hollowed-Out Icon Colorization with Controllable Diffusion Model2024 7th International Conference on Information and Computer Technologies (ICICT)10.1109/ICICT62343.2024.00038(204-210)Online publication date: 15-Mar-2024
https://doi.org/10.1109/ICICT62343.2024.00038
Li HXue TZhang ALuo XKong LHuang G(2024)The application and impact of artificial intelligence technology in graphic design: A critical interpretive synthesisHeliyon10.1016/j.heliyon.2024.e4003710:21(e40037)Online publication date: Nov-2024
https://doi.org/10.1016/j.heliyon.2024.e40037
Show More Cited By

Index Terms

Adversarial Colorization of Icons Based on Contour and Color Conditions
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interactive systems and tools
2. Information systems
  1. Information systems applications
    1. Multimedia information systems
      1. Multimedia content creation

Recommendations

Research on Colorization of Qinghai Farmer Painting Image Based on Generative Adversarial Networks
CACML '23: Proceedings of the 2023 2nd Asia Conference on Algorithms, Computing and Machine Learning

At present, deep learning method is widely used in the field of gray image colorization. Qinghai farmer painting has distinct national characteristics. The farmer painting has bright colors, high saturation, chaotic color distribution and low color ...
Color recovery of black-and-white halftoned images via categorized color-embedding look-up tables

A new method of recovering the original colors of black-and-white (B&W) halftoned images with homogeneous dot patterns is proposed. The conventional inverse halftoning method, which uses a look-up table (LUT), can establish the relation between the ...
Colorization in YCbCr color space and its application to JPEG images

This paper presents a colorization method in YCbCr color space, which is based on the maximum a posteriori estimation of a color image given a monochrome image as is our previous method in RGB color space. The presented method in YCbCr space is much ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

October 2019

2794 pages

ISBN:9781450368896

DOI:10.1145/3343031

General Chairs:
Laurent Amsaleg
CNRS-IRISA, France
,
Benoit Huet
EURECOM, France
,
Martha Larson
Radboud University and TU Delft (Netherlands)
,
Program Chairs:
Guillaume Gravier
CNRS-IRISA, France
,
Hayley Hung
Delft University of Technology Netherlands
,
Chong-Wah Ngo
City University of Hong Kong Hong Kong
,
Wei Tsang Ooi
National University of Singapore Singapore

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Ministry of Science and Technology, Taiwan

Conference

MM '19

Sponsor:

SIGMM

MM '19: The 27th ACM International Conference on Multimedia

October 21 - 25, 2019

Nice, France

Acceptance Rates

MM '19 Paper Acceptance Rate 252 of 936 submissions, 27%;

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

36
Total Citations
View Citations
472
Total Downloads

Downloads (Last 12 months)32
Downloads (Last 6 weeks)3

Reflects downloads up to 21 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Miyauchi KOrihara RSei YTahara YOhsuga A(2024)Hollowed-Out Icon Colorization with Pre-trained Large-Scale Image Generation Model事前学習済み大規模画像生成モデルを用いた中空構造アイコン線画彩色Transactions of the Japanese Society for Artificial Intelligence10.1527/tjsai.39-6_AG24-E39:6(AG24-E_1-11)Online publication date: 1-Nov-2024
https://doi.org/10.1527/tjsai.39-6_AG24-E
Miyauchi KOrihara RSei YTahara YOhsuga A(2024)Hollowed-Out Icon Colorization with Controllable Diffusion Model2024 7th International Conference on Information and Computer Technologies (ICICT)10.1109/ICICT62343.2024.00038(204-210)Online publication date: 15-Mar-2024
https://doi.org/10.1109/ICICT62343.2024.00038
Li HXue TZhang ALuo XKong LHuang G(2024)The application and impact of artificial intelligence technology in graphic design: A critical interpretive synthesisHeliyon10.1016/j.heliyon.2024.e4003710:21(e40037)Online publication date: Nov-2024
https://doi.org/10.1016/j.heliyon.2024.e40037
Wang NNing SShe YLiu BLi HWang Z(2024)Vectorized Colorization of Icon Line Art Based on Closed Contour ExtractionQuality, Reliability, Security and Robustness in Heterogeneous Systems10.1007/978-3-031-65123-6_4(38-53)Online publication date: 20-Aug-2024
https://doi.org/10.1007/978-3-031-65123-6_4
Amorim MDimuro GBorges EDalmazo BMarco-Detchart CLucca GBustince H(2023)Systematic Review of Aggregation Functions Applied to Image Edge DetectionAxioms10.3390/axioms1204033012:4(330)Online publication date: 28-Mar-2023
https://doi.org/10.3390/axioms12040330
Shi YGao TJiao XCao N(2023)Understanding Design Collaboration Between Designers and Artificial Intelligence: A Systematic Literature ReviewProceedings of the ACM on Human-Computer Interaction10.1145/36102177:CSCW2(1-35)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610217
Long WZhao WXu XZhao XLi Z(2023)Image Colorization with Fast Fourier ConvolutionProceedings of the 2023 7th International Conference on Innovation in Artificial Intelligence10.1145/3594409.3594431(60-65)Online publication date: 3-Mar-2023
https://dl.acm.org/doi/10.1145/3594409.3594431
Wu SYang YXu SLiu WYan XZhang SEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)FlexIcon: Flexible Icon Colorization via Guided Images and PalettesProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612182(8662-8673)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612182
Lin MShen IChin HChen RChen B(2023)Palette‐Based and Harmony‐Guided Colorization for Vector IconsComputer Graphics Forum10.1111/cgf.1495042:7Online publication date: 30-Oct-2023
https://doi.org/10.1111/cgf.14950
Li ZZhao NWu ZDai YWang JJing YHe S(2023)Reference‐based Screentone Transfer via Pattern Correspondence and RegularizationComputer Graphics Forum10.1111/cgf.1480042:6Online publication date: 17-Apr-2023
https://doi.org/10.1111/cgf.14800
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents