skip to main content
research-article

Deep Illumination-Enhanced Face Super-Resolution Network for Low-Light Images

Authors Info & Claims
Published:04 March 2022Publication History
Skip Abstract Section

Abstract

Face images are typically a key component in the fields of security and criminal investigation. However, due to lighting and shooting angles, faces taken under low-light conditions are often difficult to recognize. Face super-resolution (FSR) technology can restore high-resolution faces based on low-resolution inputs. However, existing face super-resolution methods typically rely on prior knowledge of inaccurate faces estimated from low-resolution images. Faces restored by low-light inputs may suffer from problems such as low brightness and many missing details. In this article, we proposed an Illumination-Enhanced Face Super-Resolution (IEFSR) model that can progressively super-resolve low-light faces of 32 × 32 pixels by an upscaling factor of 8. While reconstructing the low-light low-resolution face into a clear and high-quality face, we introduce a coarse low-resolution (LR) restoration network to recover the LR face details hidden in the dark. In the generator, we use a series of style blocks with noise to make the generated faces appear to have a more realistic visual aesthetic. Additionally, we introduce spectrum normalization in the discriminator to improve training stability. Extensive experimental evaluations show that the proposed IEFSR yields visually and metrically more attractive results than existing state-of-the-art FSR methods.

REFERENCES

  1. [1] Gao Honghao and Zhang Yudong. 2020. Introduction to the special issue on smart communications and networking for future video surveillance. ACM Transactions on Multimedia Computing, Communications, and Applications 16, 58 (Jul. 2020), 12. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. [2] Kuo Wen-Kuang and Wu Kuo-Wei. 2011. Traffic prediction and QoS transmission of real-time live VBR videos in WLANs. ACM Transactions on Multimedia Computing, Communications, and Applications 7, 36 (Nov. 2011), 121. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. [3] Zhu Nengjun, Cao Jian, Shen Kunwei, Chen Xiaosong, and Zhu Siji. 2020. A decision support system with intelligent recommendation for multi-disciplinary medical treatment. ACM Transactions on Multimedia Computing, Communications, and Applications 16, 33 (Apr. 2020), 123. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. [4] Bruce Vicki and Young Andy. 1986. Understanding face recognition. British Journal of Psychology. 77, 3 (Aug. 1986), 305327. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  5. [5] Bach Jeffrey R., Fuller Charles, and Gupta Amarnath. 1996. Virage image search engine: an open framework for image management. In Proceedings of the SPIE. Storage and Retrieval for Still Image and Video Databases, Vol. IV, International Society for Optics and Photonics, 7687. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  6. [6] Irani Michal and Pele Shmuel. 1991. Improving resolution by image registration. Graphical Models and Image Processing. 53, 3 (May. 1991), 231239. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. [7] Jia Kui. and Gong Shaogang. 2008. Generalized face super-resolution. IEEE Transactions on Image Processing 17, 6 (Jun. 2008), 873886. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. [8] Hua Huang, Huiting He, and Xin Fan. 2010. Super-resolution of human face image using canonical correlation analysis. Pattern Recognition 43, 7 (Feb. 2010), 2532--2543. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. [9] Zhu Shizhan, Liu Sifei, Loy Chen Change, and Tang Xiaoou. 2016. Deep cascaded bi-network for face hallucination. In Proceedings of the European Conference on Computer Vision. 614630. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  10. [10] Yu Xin, Fernando Basura, and Ghanem Bernard. 2018. Face super-resolution guided by facial component heatmaps. In Proceedings of the European Conference on Computer Vision. 217233. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  11. [11] Chen Yu, Tai Ying, and Liu Xiaoming. 2018. FSRNet: End-to-end learning face super-resolution with facial priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 24922501. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  12. [12] Lv Feifan, Lu Feng, Wu Jianhua, and Lim Chongsoon. 2018. MBLLEN: Low-light image/video enhancement using CNNs. In Proceedings of the British Machine Vision Conference. 220233.Google ScholarGoogle Scholar
  13. [13] Miyato Takeru, Kataoka Toshiki, Koyama Masanori, and Yoshida Yuichi. 2018. Spectral normalization for generative adversarial networks. In Proceedings of the International Conference on Learning Representation. 126.Google ScholarGoogle Scholar
  14. [14] Yu Xin, Fernando Basura, Hartley Richard, and Porikli Fatih. 2018. Super-resolving very low-resolution face images with supplementary attributes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 908917. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  15. [15] Lee Cheng-Han, Zhang Kaipeng, and Lee Hu-Cheng. 2018. Attribute augmented convolutional neural network for face hallucination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 721729. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  16. [16] Goodfellow Ian, Pouget-Abadie Jean, and Mirza Mehdi. 2014. Generative adversarial networks. Advances in Neural Information Processing Systems 27 (2014), 26722680.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. [17] Ledig Christian, Theis Lucas, and Huszár Ferenc. 2017. Photo-Realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 46814690. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  18. [18] Wang Xintao, Yu Ke, and Wu Shixiang. 2018. ESRGAN: Enhanced super-resolution generative adversarial networks. In Proceedings of the European Conference on Computer Vision. 116. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. [19] Bulat Adrian and Tzimiropoulos Georgios. 2018. Super-FAN: Integrated facial landmark localization and super-resolution of real-world low-resolution faces in arbitrary poses with GANs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 109117. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Ma Cheng, Jiang Zhenyu, and Rao Yongming. 2020. Deep face super-resolution with iterative collaboration between attentive recovery and landmark estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 55695578. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  21. [21] Wang Ting-Chun, Liu Ming-Yu, Zhu Jun-Yan, Tao Andrew, Kautz Jan, and Catanzaro Bryan. 2017. High-resolution image synthesis and semantic manipulation with conditional GANs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 87988807.Google ScholarGoogle Scholar
  22. [22] Karras Tero, Timo Aila, Laine Samuli, and Lehtinen Jaakko. 2018. Progressive growing of GANs for improved quality, stability, and variation. In Proceedings of the International Conference on Learning Representations. 126.Google ScholarGoogle Scholar
  23. [23] Karras Tero, Laine Samuli, and Aila Timo. 2019. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 44014410. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  24. [24] Karras Tero, Laine Samuli, Aittala Miika, Hellsten Janne, Lehtinen Jaakko, and Aila Timo. 2020. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 81108119.Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Chen Chaofeng, Gong Dihong, Wang Hao, Li Zhifeng, and Wong Kwan-Yee K.. 2021. Learning spatial attention for face super-resolution. IEEE Transactions on Image Processing 30 (2021), 12191231. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. [26] Pisano Etta D. and Zong Shuquan. 1998. Contrast limited adaptive histogram equalization image processing to improve the detection of simulated spiculations in dense mammograms. Journal of Digital Imaging 11, 4 (Nov. 1998), 193200.Google ScholarGoogle Scholar
  27. [27] Dong Xuan, Wang Guan, and Pang Yi. 2011. Fast efficient algorithm for enhancement of low lighting video. In Proceedings of the IEEE International Conference on Multimedia and Expo. 16. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. [28] Land Edwin H.. 1977. The retinex theory of color vision. Scientific American. 237, 6 (Dec. 1977), 108128.Google ScholarGoogle ScholarCross RefCross Ref
  29. [29] Jobson Daniel J., Rahman Zia-ur, and Woodell Glenn A.. 1997. Properties and performance of a center/surround retinex. IEEE Transactions on Image Processing 6, 3 (Mar. 1997), 451462. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. [30] Jobson Daniel J., Rahman Zia-ur, and Woodell Glenn A.. 1997. A multiscale retinex for bridging the gap between color images and the human observation of scenes. IEEE Transactions on Image Processing. 6, 7 (Jul. 1997), 965976. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. [31] Guo Xiaojie, Li Yu, and Ling Haibin. 2016. LIME: Low-light image enhancement via illumination map estimation. IEEE Transactions on Image Processing 26, 2 (Feb. 2016), 982993. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. [32] Lore Kin Gwn, Akintayo Adedotun, and Sarkar Soumik. 2017. LLNet: A deep autoencoder approach to natural low-light image enhancement. Pattern Recognition 61 (2017), 650662. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. [33] Chen Chen, Chen Qifeng, and Xu Jia. 2018. Learning to see in the dark. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 32913300. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  34. [34] Tao Li and Zhu Chuang. 2017. LLCNN: A convolutional neural network for low-light image enhancement. In Proceedings of theIEEE Visual Communications and Image Processing. 14. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  35. [35] Wei Chen, Wang Wenjing, and Yang Wenhan. 2018. Deep retinex decomposition for low-light enhancement. In Proceedings of the British Machine Vision Conference. 127136.Google ScholarGoogle Scholar
  36. [36] Wang Ruixing, Zhang Qing, and Fu Chi-Wing. 2019. Underexposed photo enhancement using deep illumination estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 68496857. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  37. [37] Heusel Martin, Ramsauer Hubert, and Unterthiner Thomas. 2017. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems. 66266637.Google ScholarGoogle Scholar
  38. [38] Simonyan Tero Karen and Zisserman Andrew. 2014. Very deep convolutional networks for large-scale image recognition. In Proceedings of the 3rd International Conference on Learning Representations. 114.Google ScholarGoogle Scholar
  39. [39] Jongyeon Lee. 2020. Face super-resolution with styled feature channel attention. Hongik University Graduation Project. Retrieved Sept. 27, 2020, from https://github.com/tinnunculus/Styled-Attention-Face-Super-Resolution.Google ScholarGoogle Scholar
  40. [40] Zhang Yulun, Li Kunpeng, and Li Kai. 2018. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision. 286301. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  41. [41] Zhang Han, Goodfellow Ian, and Metaxas Dimitris. 2019. Self-attention generative adversarial networks. In Proceedings of the International Conference on Machine Learning. 73547363.Google ScholarGoogle Scholar
  42. [42] Bulat Adrian and Tzimiropoulos Georgios. 2017. How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks). In Proceedings of the IEEE International Conference on Computer Vision. 10211030. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  43. [43] Huynh-Thu Q and Ghanbari Mohammed. 2008. Scope of validity of PSNR in image/video quality assessment. Electronics Letters 44, 13 (Feb. 2008), 800801. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  44. [44] Wang Zhou, Bovik Alan Conrad, and Sheikh Hamid Rahim. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (Apr. 2004), 600612. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. [45] Haris Muhammad, Shakhnarovich Greg, and Ukita Norimichi. 2018. Deep back-projection networks for super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 16641673. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  46. [46] Wang Xintao, Li Yu, Zhang Honglun, and Shan Ying. 2021. Towards real-world blind face restoration with generative facial prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9164--9174. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  47. [47] Guan Weili, Chen Zhaozheng, Feng Fuli, Liu Weifeng, and Nie Liqiang. 2021. Urban perception: Sensing cities via a deep interactive multi-task learning framework. ACM Transactions on Multimedia Computing, Communications, and Applications 17, 1 (Jan. 2021), Article 13, 20 pages. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. [48] Nagrani Arsha, Chung Joon Son, and Zisserman Andrew. 2017. VoxCeleb: A large-scale speaker identification dataset. In Proceedings of the Conference of the International Speech Communication Association. 16.Google ScholarGoogle ScholarCross RefCross Ref
  49. [49] Kalantari Nima Khademi and Ramamoorthi Ravi. 2017. Deep high dynamic range imaging of dynamic scenes. ACM Transactions on Graphics 36, 4 (Jul. 2017), Article 144, 12 pages. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Deep Illumination-Enhanced Face Super-Resolution Network for Low-Light Images

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Multimedia Computing, Communications, and Applications
      ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 18, Issue 3
      August 2022
      478 pages
      ISSN:1551-6857
      EISSN:1551-6865
      DOI:10.1145/3505208
      Issue’s Table of Contents

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 4 March 2022
      • Accepted: 1 November 2021
      • Revised: 1 October 2021
      • Received: 1 May 2021
      Published in tomm Volume 18, Issue 3

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Refereed

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    View Full Text

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!