Abstract
Face-mask occluded restoration aims at restoring the masked region of a human face, which has attracted increasing attention in the context of the COVID-19 pandemic. One major challenge of this task is the large visual variance of masks in the real world. To solve it we first construct a large-scale Face-mask Occluded Restoration (FMOR) dataset, which contains 5,500 unmasked images and 5,500 face-mask occluded images with various illuminations, and involves 1,100 subjects of different races, face orientations, and mask types. Moreover, we propose a Face-Mask Occluded Detection and Restoration (FMODR) framework, which can detect face-mask regions with large visual variations and restore them to realistic human faces. In particular, our FMODR contains a self-adaptive contextual attention module specifically designed for this task, which is able to exploit the contextual information and correlations of adjacent pixels for achieving high realism of the restored faces, which are however often neglected in existing contextual attention models. Our framework achieves state-of-the-art results of face restoration on three datasets, including CelebA, AR, and our FMOR datasets. Moreover, experimental results on AR and FMOR datasets demonstrate that our framework can significantly improve masked face recognition and verification performance.
- [1] . 1998. ANN: Library for approximate nearest neighbor searching. In Proceedings of the IEEE CGC Workshop on Computational Geometry, Providence, RI.Google Scholar
- [2] . 2001. Synthesizing natural textures. In Proceedings of the 2001 Symposium on Interactive 3D Graphics. 217–226.Google Scholar
Digital Library
- [3] . 2020. SD-GAN: Structural and denoising GAN reveals facial parts under occlusion[J]. arXiv preprint arXiv:2002.08448(2020).Google Scholar
- [4] . 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics 28, 3 (2009), 24.Google Scholar
Digital Library
- [5] . 2002. Missing data correction in still images and image sequences. In Proceedings of the 10th ACM International Conference on Multimedia. 355–361.Google Scholar
Digital Library
- [6] . 2011. Robust principal component analysis? The Journal of the ACM 58, 3 (2011), 1–37.Google Scholar
Digital Library
- [7] . 2021. Progressive semantic-aware style transformation for blind face restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11896–11905.Google Scholar
Cross Ref
- [8] . 2018. From eyes to face synthesis: A new approach for human-centered smart surveillance. IEEE Access 6 (2018), 14567–14575.Google Scholar
Cross Ref
- [9] . 2004. Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on Image Processing 13, 9 (2004), 1200–1212.Google Scholar
Digital Library
- [10] . 2020. Sub-center arcface: Boosting face recognition by large-scale noisy web faces. In Proceedings of the European Conference on Computer Vision. Springer, 741–757.Google Scholar
Digital Library
- [11] . 2001. Image quilting for texture synthesis and transfer. In Proceedings of the CVPR. ACM, 341–346.Google Scholar
Digital Library
- [12] . 1999. Texture synthesis by non-parametric sampling. In Proceedings of the ICCV. IEEE, 1033–1038.Google Scholar
Digital Library
- [13] . 2007. The CAS-PEAL large-scale chinese face database and baseline evaluations. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans 38, 1 (2007), 149–161.Google Scholar
- [14] . 2017. Detecting masked faces in the wild with lle-cnns. In Proceedings of the CVPR. 2682–2690.Google Scholar
Cross Ref
- [15] . 2014. Generative adversarial nets. In Proceedings of the NeurIPS. 2672–2680.Google Scholar
- [16] . 2005. Face databases. In Proceedings of the Handbook of Face Recognition. Springer, 301–327.Google Scholar
Cross Ref
- [17] . 2010. Multi-pie. Image and Vision Computing 28, 5 (2010), 807–813.Google Scholar
Digital Library
- [18] . 2021. Image inpainting via conditional texture and structure dual generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 14134–14143.Google Scholar
Cross Ref
- [19] . 2012. Statistics of patch offsets for image completion. In Proceedings of the ECCV. Springer, 16–29.Google Scholar
Cross Ref
- [20] . 2016. Multiscale representation for partial face recognition under near infrared illumination. In Proceedings of the BTAS. IEEE, 1–7.Google Scholar
Digital Library
- [21] . 2004. Locality preserving projections. In Proceedings of the Advances in Neural Information Processing Systems.153–160.Google Scholar
- [22] . 2017. In defense of the triplet loss for person re-identification[J]. arXiv preprint arXiv:1703.07737(2017).Google Scholar
- [23] . 2017. Globally and locally consistent image completion. ACM Transactions on Graphics 36, 4 (2017), 1–14.Google Scholar
Digital Library
- [24] . 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the CVPR. 1125–1134.Google Scholar
Cross Ref
- [25] . 2013. Ways to retouch photos. Laboratory Ence 16, 4 (2013), 14–17.Google Scholar
- [26] . 2012. Robust exemplar-based inpainting algorithm using region segmentation. IEEE Transactions on Consumer Electronics 58, 2 (2012), 553–561.Google Scholar
Cross Ref
- [27] . 2008. Gabor volume based local binary pattern for face representation and recognition. In Proceedings of the International Conference on Automatic Face Gesture Recognition. IEEE, 1–6.Google Scholar
Cross Ref
- [28] . 2019. Boosted gan with semantically interpretable information for image inpainting. In Proceedings of the IJCNN. IEEE, 1–8.Google Scholar
Cross Ref
- [29] . 2020. Look through masks: Towards masked face recognition with de-occlusion distillation. In Proceedings of the 28th ACM International Conference on Multimedia. 3016–3024.Google Scholar
Digital Library
- [30] . 2017. Generative face completion. In Proceedings of the CVPR. 3911–3919.Google Scholar
Cross Ref
- [31] . 2001. A two-step approach to hallucinating faces: Global parametric model and local nonparametric model. In Proceedings of the CVPR. IEEE, I–I.Google Scholar
- [32] . 2015. Deep learning face attributes in the wild. In Proceedings of the ICCV. 3730–3738.Google Scholar
Digital Library
- [33] . 2015. Deep Face Recognition. British Machine Vision Association.Google Scholar
- [34] . 2016. Context encoders: Feature learning by inpainting. In Proceedings of the CVPR. 2536–2544.Google Scholar
Cross Ref
- [35] . 2008. LabelMe. International Journal of Computer Vision 77, 1 (2008), 157–173.Google Scholar
Digital Library
- [36] . 2008. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 11 (2008), 1958–1970.Google Scholar
Digital Library
- [37] . 2017. Disentangled representation learning gan for pose-invariant face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1415–1424.Google Scholar
Cross Ref
- [38] . 1991. Face recognition using eigenfaces. In Proceedings of the CVPR. IEEE Computer Society, 586–587.Google Scholar
Cross Ref
- [39] . 2006. Eye detection in facial images with unconstrained background. JPRR 1, 1 (2006), 55–62.Google Scholar
Cross Ref
- [40] . 2021. Towards real-world blind face restoration with generative facial prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9168–9178.Google Scholar
Cross Ref
- [41] . 2018. Image inpainting via generative multi-column convolutional neural networks. In Proceedings of the NeurIPS. 331–340.Google Scholar
- [42] . 2020. Masked face recognition dataset and application. arXiv preprint arXiv:2003.09093 (2020).Google Scholar
- [43] . 2008. Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 2 (2008), 210–227.Google Scholar
Digital Library
- [44] . 2021. Towards open-world text-guided face image generation and manipulation[J]. arXiv preprint arXiv:2104.08910 (2021).Google Scholar
- [45] . 2017. High-resolution image inpainting using multi-scale neural patch synthesis. In Proceedings of the CVPR. 6721–6729.Google Scholar
Cross Ref
- [46] . 2019. Towards interpretable face recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 9348–9357.Google Scholar
Cross Ref
- [47] . 2018. Generative image inpainting with contextual attention. In Proceedings of the CVPR. 5505–5514.Google Scholar
Cross Ref
- [48] . 2019. Free-form image inpainting with gated convolution. In Proceedings of the ICCV. 4471–4480.Google Scholar
Cross Ref
- [49] . 2021. Multimodal learning for temporally coherent talking face generation with articulator synergy. IEEE Transactions on Multimedia 24 (2021), 2950–2962.Google Scholar
- [50] . 2020. Region normalization for image inpainting. In Proceedings of the AAAI Conference on Artificial Intelligence. 12733–12740.Google Scholar
Cross Ref
- [51] . 2021. Face inpainting based on GAN by facial prediction and fusion as guidance information. Applied Soft Computing 111 (2021), 107626.Google Scholar
Digital Library
- [52] . 2017. Robust lstm-autoencoders for face de-occlusion in the wild. IEEE Transactions on Image Processing 27, 2 (2017), 778–790.Google Scholar
Cross Ref
- [53] . 2018. Towards pose invariant face recognition in the wild. In Proceedings of the CVPR. 2207–2216.Google Scholar
Cross Ref
- [54] . 2019. Multi-prototype networks for unconstrained set-based face recognition. In Proceedings of the 28th International Joint Conference on Artificial Intelligence. 4397–4403.Google Scholar
- [55] . 2017. Dual-agent gans for photorealistic and identity preserving profile face synthesis. In Proceedings of the NeurIPS. 66–76.Google Scholar
- [56] . 2020. Learning oracle attention for high-fidelity face completion. In Proceedings of the CVPR. 7680–7689.Google Scholar
Cross Ref
- [57] . 2018. Visually interpretable representation learning for depression recognition from facial images. IEEE Transactions on Affective Computing 11, 3 (2018), 542–552.Google Scholar
Cross Ref
- [58] . 2021. WebFace260M: A benchmark unveiling the power of million-scale deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10492–10502.Google Scholar
Cross Ref
Index Terms
Toward High-quality Face-Mask Occluded Restoration
Recommendations
Does face restoration improve face verification?
AbstractMethods for face verification works reasonably well on face images with standardized (frontal) face positions and good spatial resolution. However such methods have significant challenges on poor resolution images, poor lighting conditions and not ...
Restoration of a Frontal Illuminated Face Image Based on KPCA
ICPR '10: Proceedings of the 2010 20th International Conference on Pattern RecognitionIn this paper, we propose a novel illumination-normalization method. By using the combination of the Kernel Principal Component Analysis (KPCA) and Pre-image technology, this method can restore the frontal-illuminated face image from a single non-...
Three-Dimensional Occlusion Detection and Restoration of Partially Occluded Faces
This paper presents an innovative three dimensional occlusion detection and restoration strategy for the recognition of three dimensional faces partially occluded by unforeseen, extraneous objects. The detection method considers occlusions as local ...






Comments