skip to main content
research-article

A Deep Learning Approach for Face Hallucination Guided by Facial Boundary Responses

Authors Info & Claims
Published:04 March 2020Publication History
Skip Abstract Section

Abstract

Face hallucination is a domain-specific super-resolution (SR) problem of learning a mapping between a low-resolution (LR) face image and its corresponding high-resolution (HR) image. Tremendous progress on deep learning has shown exciting potential for a variety of face hallucination tasks. However, most deep-learning–based methods are limited to handle facial appearance information without paying attention to facial structure priors. In this article, we propose an open source1 Boundary-aware Dual-branch Network (BDN) for face hallucination, which simultaneously extracts face features and estimates facial boundary responses from LR inputs, ultimately fusing them to reconstruct HR results. Specifically, we first upsample LR face images to HR feature maps, and then feed the upsampled HR features into a memory unit and an attention unit synchronously to obtain the refined features and predict facial boundary responses. Next, they are fed into a feature map fusion unit to combine facial appearance and structure information by a spatial attention mechanism. Moreover, we employ a series of stacked units to boost performance before recovering HR face images. Finally, a discriminative network is developed to improve visual quality by introducing adversarial learning strategy. Extensive experiments show that the proposed approach achieves superior face hallucination results against the state-of-the-art ones.

References

  1. Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein generative adversarial networks. In Proceedings of the International Conference on Machine Learning. 214--223.Google ScholarGoogle Scholar
  2. Simon Baker and Takeo Kanade. 2000. Hallucinating faces. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition (FG’00). 83--88.Google ScholarGoogle ScholarCross RefCross Ref
  3. Simon Baker and Takeo Kanade. 2002. Limits on super-resolution and how to break them. IEEE Trans. Pattern Anal. Mach. Intell. 9 (2002), 1167--1183.Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Adrian Bulat and Georgios Tzimiropoulos. 2017. How far are we from solving the 2D 8 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks). In Proceedings of the International Conference on Computer Vision.Google ScholarGoogle ScholarCross RefCross Ref
  5. Adrian Bulat and Georgios Tzimiropoulos. 2018. Super-FAN: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with GANs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 109--117.Google ScholarGoogle ScholarCross RefCross Ref
  6. Qingxing Cao, Liang Lin, Yukai Shi, Xiaodan Liang, and Guanbin Li. 2017. Attention-aware face hallucination via deep reinforcement learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 690--698.Google ScholarGoogle ScholarCross RefCross Ref
  7. Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, and Jian Yang. 2018. FSRNet: End-to-end learning face super-resolution with facial priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2492--2501.Google ScholarGoogle ScholarCross RefCross Ref
  8. Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. 2018. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8789--8797.Google ScholarGoogle ScholarCross RefCross Ref
  9. Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A. Bharath. 2018. Generative adversarial networks: An overview. IEEE Sig. Proc. Mag. 35, 1 (2018), 53--65.Google ScholarGoogle ScholarCross RefCross Ref
  10. Peter Dayan, Laurence F. Abbott, and L. Abbott. 2001. Theoretical neuroscience: Computational and mathematical modeling of neural systems. Journal of Cognitive Neuroscience 15, 1 (2003), 154--155.Google ScholarGoogle Scholar
  11. Berk Dogan, Shuhang Gu, and Radu Timofte. 2019. Exemplar guided face image super-resolution without facial landmarks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.Google ScholarGoogle Scholar
  12. Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2016. Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. 8 Mach. Intell. 38, 2 (2016), 295--307.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 2672--2680.Google ScholarGoogle Scholar
  14. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770--778.Google ScholarGoogle ScholarCross RefCross Ref
  15. Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4700--4708.Google ScholarGoogle Scholar
  16. Gary B. Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. 2008. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. In Proceedings of the Workshop on Faces in “Real-Life” Images: Detection, Alignment, and Recognition.Google ScholarGoogle Scholar
  17. Huaibo Huang, Ran He, Zhenan Sun, and Tieniu Tan. 2017. Wavelet-SRNet: A wavelet-based CNN for multi-scale face super resolution. In Proceedings of the IEEE International Conference on Computer Vision. 1689--1697.Google ScholarGoogle ScholarCross RefCross Ref
  18. Andrey Ignatov, Nikolay Kobyshev, Radu Timofte, Kenneth Vanhoey, and Luc Van Gool. 2018. WESPE: Weakly supervised photo enhancer for digital cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 691--700.Google ScholarGoogle ScholarCross RefCross Ref
  19. Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2017. Globally and locally consistent image completion. ACM Trans. Graph. 36, 4 (2017), 107.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1125--1134.Google ScholarGoogle Scholar
  21. Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In Proceedings of the European Conference on Computer Vision. Springer, 694--711.Google ScholarGoogle ScholarCross RefCross Ref
  22. Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1646--1654.Google ScholarGoogle ScholarCross RefCross Ref
  23. Soheil Kolouri and Gustavo K. Rohde. 2015. Transport-based single frame super resolution of very low resolution face images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4876--4884.Google ScholarGoogle Scholar
  24. Lingchao Kong and Rui Dai. 2018. Efficient video encoding for automatic video analysis in distributed wireless surveillance systems. ACM Trans. Multim. Comput. Commun. Applic. 14, 3 (2018), 72.Google ScholarGoogle Scholar
  25. Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4681--4690.Google ScholarGoogle ScholarCross RefCross Ref
  26. Mengyan Li, Yuechuan Sun, Zhaoyu Zhang, and Jun Yu. 2018. A coarse-to-fine face hallucination method by exploiting facial prior knowledge. In Proceedings of the 25th IEEE International Conference on Image Processing (ICIP’18). IEEE, 61--65.Google ScholarGoogle ScholarCross RefCross Ref
  27. Xianguo Li, Yemei Sun, Yanli Yang, and Changyun Miao. 2019. Symmetrical residual connections for single image super-resolution. ACM Trans. Multim. Comput. Commun. Applic. 15, 1 (2019), 19.Google ScholarGoogle Scholar
  28. Yongchao Li, Cheng Cai, Guoping Qiu, and Kin-Man Lam. 2014. Face hallucination based on sparse local-pixel structure. Pattern Recog. 47, 3 (2014), 1261--1270.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Ming Liang and Xiaolin Hu. 2015. Recurrent convolutional neural network for object recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3367--3375.Google ScholarGoogle ScholarCross RefCross Ref
  30. Ce Liu, Heung-Yeung Shum, and Chang-Shui Zhang. 2001. A two-step approach to hallucinating faces: Global parametric model and local nonparametric model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’01). Citeseer, 192--198.Google ScholarGoogle Scholar
  31. Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2015. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision. 3730--3738.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Yongyi Lu, Yu-Wing Tai, and Chi-Keung Tang. 2018. Attribute-guided face generation using conditional CycleGAN. In Proceedings of the European Conference on Computer Vision (ECCV’18). 282--297.Google ScholarGoogle ScholarCross RefCross Ref
  33. Xiang Ma, Junping Zhang, and Chun Qi. 2010. Hallucinating face by position-patch. Pattern Recog. 43, 6 (2010), 2224--2236.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Iacopo Masi, Feng-Ju Chang, Jongmoo Choi, Shai Harel, Jungyeon Kim, Kang Geon Kim, Jatuporn Leksut, Stephen Rawls, Yue Wu, Tal Hassner, et al. 2019. Learning pose-aware models for pose-invariant face recognition in the wild. IEEE Trans. Pattern Anal. 8 Mach. Intell. 41, 2 (2019), 379--393.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, et al. 2015. Deep face recognition. In Proceedings of the British Machine Vision Conference, Vol. 1. 6.Google ScholarGoogle ScholarCross RefCross Ref
  36. Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 815--823.Google ScholarGoogle ScholarCross RefCross Ref
  37. Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1874--1883.Google ScholarGoogle ScholarCross RefCross Ref
  38. Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. 2017. Memnet: A persistent memory network for image restoration. In Proceedings of the IEEE International Conference on Computer Vision. 4539--4547.Google ScholarGoogle ScholarCross RefCross Ref
  39. Zhiqiang Tang, Xi Peng, Shijie Geng, Lingfei Wu, Shaoting Zhang, and Dimitris Metaxas. 2018. Quantized densely connected u-nets for efficient landmark localization. In Proceedings of the European Conference on Computer Vision (ECCV’18). 339--354.Google ScholarGoogle ScholarCross RefCross Ref
  40. Nannan Wang, Dacheng Tao, Xinbo Gao, Xuelong Li, and Jie Li. 2014. A comprehensive survey to face hallucination. Int. J. Comput. Vis. 106, 1 (2014), 9--30.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Xiaogang Wang and Xiaoou Tang. 2005. Hallucinating face by eigentransformation. IEEE Trans. Syst. Man. Cyber. Part C (Applic. Rev.) 35, 3 (2005), 425--434.Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Wayne Wu, Chen Qian, Shuo Yang, Quan Wang, Yici Cai, and Qiang Zhou. 2018. Look at boundary: A boundary-aware face alignment algorithm. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2129--2138.Google ScholarGoogle ScholarCross RefCross Ref
  43. Taihong Xiao, Jiapeng Hong, and Jinwen Ma. 2018. ELEGANT: Exchanging latent encodings with GAN for transferring multiple face attributes. In Proceedings of the European Conference on Computer Vision (ECCV’18). 168--184.Google ScholarGoogle ScholarCross RefCross Ref
  44. Chih-Yuan Yang, Sifei Liu, and Ming-Hsuan Yang. 2013. Structured face hallucination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1099--1106.Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Raymond A. Yeh, Chen Chen, Teck Yian Lim, Alexander G. Schwing, Mark Hasegawa-Johnson, and Minh N. Do. 2017. Semantic image inpainting with deep generative models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5485--5493.Google ScholarGoogle Scholar
  46. Xin Yu, Basura Fernando, Bernard Ghanem, Fatih Porikli, and Richard Hartley. 2018. Face super-resolution guided by facial component heatmaps. In Proceedings of the European Conference on Computer Vision (ECCV’18). 217--233.Google ScholarGoogle ScholarCross RefCross Ref
  47. Xin Yu, Basura Fernando, Richard Hartley, and Fatih Porikli. 2018. Super-resolving very low-resolution face images with supplementary attributes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 908--917.Google ScholarGoogle ScholarCross RefCross Ref
  48. Xin Yu and Fatih Porikli. 2016. Ultra-resolving face images by discriminative generative networks. In Proceedings of the European Conference on Computer Vision. Springer, 318--333.Google ScholarGoogle ScholarCross RefCross Ref
  49. Xin Yu and Fatih Porikli. 2017. Face hallucination with tiny unaligned images by transformative discriminative neural networks. In Proceedings of the 31st AAAI Conference on Artificial Intelligence.Google ScholarGoogle Scholar
  50. Xin Yu and Fatih Porikli. 2017. Hallucinating very low-resolution unaligned and noisy face images by transformative discriminative autoencoders. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3760--3768.Google ScholarGoogle ScholarCross RefCross Ref
  51. Xin Yu and Fatih Porikli. 2018. Imagining the unimaginable faces by deconvolutional networks. IEEE Trans. Image Proc. 27, 6 (2018), 2747--2761.Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Xin Yu, Fatih Porikli, Basura Fernando, and Richard Hartley. 2020. Hallucinating unaligned face images by multiscale transformative discriminative networks. Int. J. Comput. Vis. 128, 2 (2020), 500--526.Google ScholarGoogle ScholarCross RefCross Ref
  53. Kaipeng Zhang, Zhanpeng Zhang, Chia-Wen Cheng, Winston H. Hsu, Yu Qiao, Wei Liu, and Tong Zhang. 2018. Super-identity convolutional neural network for face hallucination. In Proceedings of the European Conference on Computer Vision (ECCV’18). 183--198.Google ScholarGoogle ScholarCross RefCross Ref
  54. Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. 2016. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Sig. Proc. Lett. 23, 10 (2016), 1499--1503.Google ScholarGoogle ScholarCross RefCross Ref
  55. Erjin Zhou, Haoqiang Fan, Zhimin Cao, Yuning Jiang, and Qi Yin. 2015. Learning face hallucination in the wild. In Proceedings of the 29th AAAI Conference on Artificial Intelligence.Google ScholarGoogle Scholar
  56. Joey Tianyi Zhou, Jiawei Du, Hongyuan Zhu, Xi Peng, Yong Liu, and Rick Siow Mong Goh. 2019. AnomalyNet: An anomaly detection network for video surveillance. IEEE Trans. Inf. Forens. Secur. 14, 10 (2019), 2537--2550.Google ScholarGoogle ScholarCross RefCross Ref
  57. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision. 2223--2232.Google ScholarGoogle Scholar
  58. Shizhan Zhu, Sifei Liu, Chen Change Loy, and Xiaoou Tang. 2016. Deep cascaded bi-network for face hallucination. In Proceedings of the European Conference on Computer Vision. Springer, 614--630.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. A Deep Learning Approach for Face Hallucination Guided by Facial Boundary Responses

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!