skip to main content
research-article

Invertible Grayscale with Sparsity Enforcing Priors

Authors Info & Claims
Published:22 July 2021Publication History
Skip Abstract Section

Abstract

Color dimensionality reduction is believed as a non-invertible process, as re-colorization results in perceptually noticeable and unrecoverable distortion. In this article, we propose to convert a color image into a grayscale image that can fully recover its original colors, and more importantly, the encoded information is discriminative and sparse, which saves storage capacity. Particularly, we design an invertible deep neural network for color encoding and decoding purposes. This network learns to generate a residual image that encodes color information, and it is then combined with a base grayscale image for color recovering. In this way, the non-differentiable compression process (e.g., JPEG) of the base grayscale image can be integrated into the network in an end-to-end manner. To further reduce the size of the residual image, we present a specific layer to enhance Sparsity Enforcing Priors (SEP), thus leading to negligible storage space. The proposed method allows color embedding on a sparse residual image while keeping a high, 35dB PSNR on average. Extensive experiments demonstrate that the proposed method outperforms state-of-the-arts in terms of image quality and tolerability to compression.

References

  1. Mohammad Haris Baig and Lorenzo Torresani. 2017. Multiple hypothesis colorization and its application to image compression. Comput. Vis. Image Underst. 164 (2017), 111–123.Google ScholarGoogle ScholarCross RefCross Ref
  2. Johannes Ballé, Valero Laparra, and Eero P. Simoncelli. 2017. End-to-end optimized image compression. In ICLR.Google ScholarGoogle Scholar
  3. Zezhou Cheng, Qingxiong Yang, and Bin Sheng. 2015. Deep colorization. In ICCV. 415–423. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Hao Du, Shengfeng He, Bin Sheng, Lizhuang Ma, and Rynson W. H. Lau. 2015. Saliency-guided color-to-gray conversion using region-based optimization. IEEE Trans. Image Proc. 24, 1 (2015), 434–443.Google ScholarGoogle ScholarCross RefCross Ref
  5. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. 2010. The Pascal Visual Object Classes (VOC) challenge. Int. J. Comput. Vis. 88, 2 (June 2010), 303–338. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Amy A. Gooch, Sven C. Olsen, Jack Tumblin, and Bruce Gooch. 2005. Color2gray: Salience-preserving color removal. ACM Trans Graph. 24, 3 (2005), 634–639. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Will Grathwohl, Dami Choi, Yuhuai Wu, Geoffrey Roeder, and David Duvenaud. 2018. Backpropagation through the void: Optimizing control variates for black-box gradient estimation. In ICLR.Google ScholarGoogle Scholar
  8. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. 770–778.Google ScholarGoogle Scholar
  9. Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In ECCV. Springer, 694–711.Google ScholarGoogle Scholar
  10. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.Google ScholarGoogle Scholar
  11. Diederik P. Kingma and Max Welling. 2014. Auto-encoding variational bayes. In ICLR.Google ScholarGoogle Scholar
  12. Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. 2016. Learning representations for automatic colorization. In ECCV. 577–593.Google ScholarGoogle Scholar
  13. Anat Levin, Dani Lischinski, and Yair Weiss. 2004. Colorization using optimization. ACM Trans Graph., Vol. 23. 689–694. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Mu Li, Wangmeng Zuo, Shuhang Gu, Debin Zhao, and David Zhang. 2018. Learning convolutional networks for content-weighted image compression. In CVPR. 3214–3223.Google ScholarGoogle Scholar
  15. Christos Louizos, Max Welling, and Diederik P. Kingma. 2018. Learning sparse neural networks through regularization. In ICLR.Google ScholarGoogle Scholar
  16. Cewu Lu, Li Xu, and Jiaya Jia. 2012. Real-time contrast preserving decolorization. In SIGGRAPH Asia. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Cewu Lu, Li Xu, and Jiaya Jia. 2014. Contrast preserving decolorization with perception-based quality metrics. Int. J. Comput. Vis. 110, 2 (2014), 222–239. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Qing Luan, Fang Wen, Daniel Cohen-Or, Lin Liang, Ying-Qing Xu, and Heung-Yeung Shum. 2007. Natural image colorization. In EGSR. 309–320. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Yu Luo, Yong Xu, and Hui Ji. 2015. Removing rain from a single image via discriminative sparse coding. In ICCV. 3397–3405. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2016. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712 (2016).Google ScholarGoogle Scholar
  21. Ichiro Matsuda, Hirofumi Mori, and Susumu Itoh. 2000. Lossless coding of still images using minimum-rate predictors. In ICIP, Vol. 1. IEEE, 132–135.Google ScholarGoogle ScholarCross RefCross Ref
  22. Laszlo Neumann, Martin Čadík, and Antal Nemcsics. 2007. An efficient perception-based adaptive color to gray transformation. In Computational Aesthetics. 73–80. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Yingge Qu, Tien-Tsin Wong, and Pheng-Ann Heng. 2006. Manga colorization. ACM Trans Graph., Vol. 25. 1214–1220. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In ICLR.Google ScholarGoogle Scholar
  25. Athanassios Skodras, Charilaos Christopoulos, and Touradj Ebrahimi. 2001. The JPEG 2000 still image compression standard. IEEE Sig. Proc. Mag. 18, 5 (2001), 36–58.Google ScholarGoogle ScholarCross RefCross Ref
  26. Kaleigh Smith, Pierre-Edouard Landes, Joëlle Thollot, and Karol Myszkowski. 2008. Apparent greyscale: A simple and fast conversion to perceptually accurate images and video. Comput. Graph. Forum, Vol. 27. 193–200.Google ScholarGoogle ScholarCross RefCross Ref
  27. George Tucker, Andriy Mnih, Chris J. Maddison, John Lawson, and Jascha Sohl-Dickstein. 2017. REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models. In NeurIPS. 2627–2636. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Gregory K. Wallace. 1992. The JPEG still picture compression standard. IEEE Trans. Consum. Electron. 38, 1 (1992), xviii–xxxiv. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8, 3-4 (1992), 229–256. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Menghan Xia, Xueting Liu, and Tien-Tsin Wong. 2018. Invertible grayscale. ACM Trans Graph. 37, 6 (Dec. 2018), 246:1–246:10.Google ScholarGoogle Scholar
  31. Li Xu, Qiong Yan, and Jiaya Jia. 2013. A sparse control model for image and video editing. ACM Trans Graph. 32, 6 (2013), 197. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. T. Ye, Y. Du, J. Deng, and S. He. 2020. Invertible grayscale via dual features ensemble. IEEE Access 8 (2020), 89670–89679.Google ScholarGoogle ScholarCross RefCross Ref
  33. Richard Zhang, Phillip Isola, and Alexei A. Efros. 2016. Colorful image colorization. In ECCV. 649–666.Google ScholarGoogle Scholar
  34. Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S. Lin, Tianhe Yu, and Alexei A. Efros. 2017. Real-time user-guided image colorization with learned deep priors. ACM Trans Graph. 36, 4 (2017), 119:1–119:11. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networkss. In ICCV.Google ScholarGoogle Scholar

Index Terms

  1. Invertible Grayscale with Sparsity Enforcing Priors

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Multimedia Computing, Communications, and Applications
        ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 17, Issue 3
        August 2021
        443 pages
        ISSN:1551-6857
        EISSN:1551-6865
        DOI:10.1145/3476118
        Issue’s Table of Contents

        Copyright © 2021 Association for Computing Machinery.

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 22 July 2021
        • Accepted: 1 February 2021
        • Revised: 1 January 2021
        • Received: 1 September 2020
        Published in tomm Volume 17, Issue 3

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Refereed
      • Article Metrics

        • Downloads (Last 12 months)42
        • Downloads (Last 6 weeks)0

        Other Metrics

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!