Abstract
Face hallucination is a domain-specific super-resolution (SR) problem of learning a mapping between a low-resolution (LR) face image and its corresponding high-resolution (HR) image. Tremendous progress on deep learning has shown exciting potential for a variety of face hallucination tasks. However, most deep-learning–based methods are limited to handle facial appearance information without paying attention to facial structure priors. In this article, we propose an open source1 Boundary-aware Dual-branch Network (BDN) for face hallucination, which simultaneously extracts face features and estimates facial boundary responses from LR inputs, ultimately fusing them to reconstruct HR results. Specifically, we first upsample LR face images to HR feature maps, and then feed the upsampled HR features into a memory unit and an attention unit synchronously to obtain the refined features and predict facial boundary responses. Next, they are fed into a feature map fusion unit to combine facial appearance and structure information by a spatial attention mechanism. Moreover, we employ a series of stacked units to boost performance before recovering HR face images. Finally, a discriminative network is developed to improve visual quality by introducing adversarial learning strategy. Extensive experiments show that the proposed approach achieves superior face hallucination results against the state-of-the-art ones.
- Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein generative adversarial networks. In Proceedings of the International Conference on Machine Learning. 214--223.Google Scholar
- Simon Baker and Takeo Kanade. 2000. Hallucinating faces. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition (FG’00). 83--88.Google Scholar
Cross Ref
- Simon Baker and Takeo Kanade. 2002. Limits on super-resolution and how to break them. IEEE Trans. Pattern Anal. Mach. Intell. 9 (2002), 1167--1183.Google Scholar
Digital Library
- Adrian Bulat and Georgios Tzimiropoulos. 2017. How far are we from solving the 2D 8 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks). In Proceedings of the International Conference on Computer Vision.Google Scholar
Cross Ref
- Adrian Bulat and Georgios Tzimiropoulos. 2018. Super-FAN: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with GANs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 109--117.Google Scholar
Cross Ref
- Qingxing Cao, Liang Lin, Yukai Shi, Xiaodan Liang, and Guanbin Li. 2017. Attention-aware face hallucination via deep reinforcement learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 690--698.Google Scholar
Cross Ref
- Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, and Jian Yang. 2018. FSRNet: End-to-end learning face super-resolution with facial priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2492--2501.Google Scholar
Cross Ref
- Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. 2018. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8789--8797.Google Scholar
Cross Ref
- Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A. Bharath. 2018. Generative adversarial networks: An overview. IEEE Sig. Proc. Mag. 35, 1 (2018), 53--65.Google Scholar
Cross Ref
- Peter Dayan, Laurence F. Abbott, and L. Abbott. 2001. Theoretical neuroscience: Computational and mathematical modeling of neural systems. Journal of Cognitive Neuroscience 15, 1 (2003), 154--155.Google Scholar
- Berk Dogan, Shuhang Gu, and Radu Timofte. 2019. Exemplar guided face image super-resolution without facial landmarks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.Google Scholar
- Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2016. Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. 8 Mach. Intell. 38, 2 (2016), 295--307.Google Scholar
Digital Library
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 2672--2680.Google Scholar
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770--778.Google Scholar
Cross Ref
- Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4700--4708.Google Scholar
- Gary B. Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. 2008. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. In Proceedings of the Workshop on Faces in “Real-Life” Images: Detection, Alignment, and Recognition.Google Scholar
- Huaibo Huang, Ran He, Zhenan Sun, and Tieniu Tan. 2017. Wavelet-SRNet: A wavelet-based CNN for multi-scale face super resolution. In Proceedings of the IEEE International Conference on Computer Vision. 1689--1697.Google Scholar
Cross Ref
- Andrey Ignatov, Nikolay Kobyshev, Radu Timofte, Kenneth Vanhoey, and Luc Van Gool. 2018. WESPE: Weakly supervised photo enhancer for digital cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 691--700.Google Scholar
Cross Ref
- Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2017. Globally and locally consistent image completion. ACM Trans. Graph. 36, 4 (2017), 107.Google Scholar
Digital Library
- Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1125--1134.Google Scholar
- Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In Proceedings of the European Conference on Computer Vision. Springer, 694--711.Google Scholar
Cross Ref
- Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1646--1654.Google Scholar
Cross Ref
- Soheil Kolouri and Gustavo K. Rohde. 2015. Transport-based single frame super resolution of very low resolution face images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4876--4884.Google Scholar
- Lingchao Kong and Rui Dai. 2018. Efficient video encoding for automatic video analysis in distributed wireless surveillance systems. ACM Trans. Multim. Comput. Commun. Applic. 14, 3 (2018), 72.Google Scholar
- Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4681--4690.Google Scholar
Cross Ref
- Mengyan Li, Yuechuan Sun, Zhaoyu Zhang, and Jun Yu. 2018. A coarse-to-fine face hallucination method by exploiting facial prior knowledge. In Proceedings of the 25th IEEE International Conference on Image Processing (ICIP’18). IEEE, 61--65.Google Scholar
Cross Ref
- Xianguo Li, Yemei Sun, Yanli Yang, and Changyun Miao. 2019. Symmetrical residual connections for single image super-resolution. ACM Trans. Multim. Comput. Commun. Applic. 15, 1 (2019), 19.Google Scholar
- Yongchao Li, Cheng Cai, Guoping Qiu, and Kin-Man Lam. 2014. Face hallucination based on sparse local-pixel structure. Pattern Recog. 47, 3 (2014), 1261--1270.Google Scholar
Digital Library
- Ming Liang and Xiaolin Hu. 2015. Recurrent convolutional neural network for object recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3367--3375.Google Scholar
Cross Ref
- Ce Liu, Heung-Yeung Shum, and Chang-Shui Zhang. 2001. A two-step approach to hallucinating faces: Global parametric model and local nonparametric model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’01). Citeseer, 192--198.Google Scholar
- Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2015. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision. 3730--3738.Google Scholar
Digital Library
- Yongyi Lu, Yu-Wing Tai, and Chi-Keung Tang. 2018. Attribute-guided face generation using conditional CycleGAN. In Proceedings of the European Conference on Computer Vision (ECCV’18). 282--297.Google Scholar
Cross Ref
- Xiang Ma, Junping Zhang, and Chun Qi. 2010. Hallucinating face by position-patch. Pattern Recog. 43, 6 (2010), 2224--2236.Google Scholar
Digital Library
- Iacopo Masi, Feng-Ju Chang, Jongmoo Choi, Shai Harel, Jungyeon Kim, Kang Geon Kim, Jatuporn Leksut, Stephen Rawls, Yue Wu, Tal Hassner, et al. 2019. Learning pose-aware models for pose-invariant face recognition in the wild. IEEE Trans. Pattern Anal. 8 Mach. Intell. 41, 2 (2019), 379--393.Google Scholar
Digital Library
- Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, et al. 2015. Deep face recognition. In Proceedings of the British Machine Vision Conference, Vol. 1. 6.Google Scholar
Cross Ref
- Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 815--823.Google Scholar
Cross Ref
- Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1874--1883.Google Scholar
Cross Ref
- Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. 2017. Memnet: A persistent memory network for image restoration. In Proceedings of the IEEE International Conference on Computer Vision. 4539--4547.Google Scholar
Cross Ref
- Zhiqiang Tang, Xi Peng, Shijie Geng, Lingfei Wu, Shaoting Zhang, and Dimitris Metaxas. 2018. Quantized densely connected u-nets for efficient landmark localization. In Proceedings of the European Conference on Computer Vision (ECCV’18). 339--354.Google Scholar
Cross Ref
- Nannan Wang, Dacheng Tao, Xinbo Gao, Xuelong Li, and Jie Li. 2014. A comprehensive survey to face hallucination. Int. J. Comput. Vis. 106, 1 (2014), 9--30.Google Scholar
Digital Library
- Xiaogang Wang and Xiaoou Tang. 2005. Hallucinating face by eigentransformation. IEEE Trans. Syst. Man. Cyber. Part C (Applic. Rev.) 35, 3 (2005), 425--434.Google Scholar
Digital Library
- Wayne Wu, Chen Qian, Shuo Yang, Quan Wang, Yici Cai, and Qiang Zhou. 2018. Look at boundary: A boundary-aware face alignment algorithm. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2129--2138.Google Scholar
Cross Ref
- Taihong Xiao, Jiapeng Hong, and Jinwen Ma. 2018. ELEGANT: Exchanging latent encodings with GAN for transferring multiple face attributes. In Proceedings of the European Conference on Computer Vision (ECCV’18). 168--184.Google Scholar
Cross Ref
- Chih-Yuan Yang, Sifei Liu, and Ming-Hsuan Yang. 2013. Structured face hallucination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1099--1106.Google Scholar
Digital Library
- Raymond A. Yeh, Chen Chen, Teck Yian Lim, Alexander G. Schwing, Mark Hasegawa-Johnson, and Minh N. Do. 2017. Semantic image inpainting with deep generative models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5485--5493.Google Scholar
- Xin Yu, Basura Fernando, Bernard Ghanem, Fatih Porikli, and Richard Hartley. 2018. Face super-resolution guided by facial component heatmaps. In Proceedings of the European Conference on Computer Vision (ECCV’18). 217--233.Google Scholar
Cross Ref
- Xin Yu, Basura Fernando, Richard Hartley, and Fatih Porikli. 2018. Super-resolving very low-resolution face images with supplementary attributes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 908--917.Google Scholar
Cross Ref
- Xin Yu and Fatih Porikli. 2016. Ultra-resolving face images by discriminative generative networks. In Proceedings of the European Conference on Computer Vision. Springer, 318--333.Google Scholar
Cross Ref
- Xin Yu and Fatih Porikli. 2017. Face hallucination with tiny unaligned images by transformative discriminative neural networks. In Proceedings of the 31st AAAI Conference on Artificial Intelligence.Google Scholar
- Xin Yu and Fatih Porikli. 2017. Hallucinating very low-resolution unaligned and noisy face images by transformative discriminative autoencoders. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3760--3768.Google Scholar
Cross Ref
- Xin Yu and Fatih Porikli. 2018. Imagining the unimaginable faces by deconvolutional networks. IEEE Trans. Image Proc. 27, 6 (2018), 2747--2761.Google Scholar
Digital Library
- Xin Yu, Fatih Porikli, Basura Fernando, and Richard Hartley. 2020. Hallucinating unaligned face images by multiscale transformative discriminative networks. Int. J. Comput. Vis. 128, 2 (2020), 500--526.Google Scholar
Cross Ref
- Kaipeng Zhang, Zhanpeng Zhang, Chia-Wen Cheng, Winston H. Hsu, Yu Qiao, Wei Liu, and Tong Zhang. 2018. Super-identity convolutional neural network for face hallucination. In Proceedings of the European Conference on Computer Vision (ECCV’18). 183--198.Google Scholar
Cross Ref
- Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. 2016. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Sig. Proc. Lett. 23, 10 (2016), 1499--1503.Google Scholar
Cross Ref
- Erjin Zhou, Haoqiang Fan, Zhimin Cao, Yuning Jiang, and Qi Yin. 2015. Learning face hallucination in the wild. In Proceedings of the 29th AAAI Conference on Artificial Intelligence.Google Scholar
- Joey Tianyi Zhou, Jiawei Du, Hongyuan Zhu, Xi Peng, Yong Liu, and Rick Siow Mong Goh. 2019. AnomalyNet: An anomaly detection network for video surveillance. IEEE Trans. Inf. Forens. Secur. 14, 10 (2019), 2537--2550.Google Scholar
Cross Ref
- Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision. 2223--2232.Google Scholar
- Shizhan Zhu, Sifei Liu, Chen Change Loy, and Xiaoou Tang. 2016. Deep cascaded bi-network for face hallucination. In Proceedings of the European Conference on Computer Vision. Springer, 614--630.Google Scholar
Cross Ref
Index Terms
A Deep Learning Approach for Face Hallucination Guided by Facial Boundary Responses
Recommendations
A novel correspondence-based face-hallucination method
This paper addresses the problem of estimating high-resolution (HR) facial images from a single low-resolution (LR) input. We assume that the input LR and estimated HR images are under the same view-point and illumination condition, i.e. the setting of ...
An Example-Based Face Hallucination Method for Single-Frame, Low-Resolution Facial Images
This paper proposes a face hallucination method for the reconstruction of high-resolution facial images from single-frame, low-resolution facial images. The proposed method has been derived from example-based hallucination methods and morphable face ...
Face Hallucination: Theory and Practice
In this paper, we study face hallucination, or synthesizing a high-resolution face image from an input low-resolution image, with the help of a large collection of other high-resolution face images. Our theoretical contribution is a two-step statistical ...






Comments