Abstract
During the COVID-19 coronavirus epidemic, wearing masks has become increasingly popular. Traditional occlusion face recognition algorithms are almost ineffective for such heavy mask occlusion. Therefore, it is urgent to improve the recognition performance of the existing face recognition technology on masked faces. Due to the limited visible feature points of the masked face image relative to the normal face image, we have to exploit the identification potential of eyebrow (referring to eyes and brows) features. This article proposes a local eyebrow feature attention network for masked face recognition, which consists of feature extraction, eyebrow region pooling, and feature fusion. To highlight the eyebrow region, we first use the eyebrow region pooling to separate the local features of eyebrows from the learned overall facial features. We then make full use of the symmetry of left and right eyebrows to emphasize their discriminant ability, due to the inadequate fine information of the low-resolution eyebrows. In particular, in view of the symmetrical similarity between eyebrow pairs and the subordinate relationship between facial components and the whole, we propose a feature fusion model based on graph convolutional network (GCN) to learn the feature association structure of eye features, brow features, and global facial features. We construct the benchmark datasets for masked face recognition to validate our approach, including real-world masked face recognition dataset (RMFRD) and synthetic masked face recognition dataset (SMFRD). Extensive experimental results on both public datasets and our built masked face datasets show that our approach significantly outperforms the state-of-the-arts.
- [1] Ngan. 2020. NIST. https://www.domesticpreparedness.com/updates/nist-launches-investigation-of-face-masks-effect-on-face-recognition-software/.Google Scholar
- [2] Davis E. King. 2009. Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research 10 (2009), 1755–1758.Google Scholar
- [3] Kentaro Wada. 2021. Labelme: Image Polygonal Annotation with Python. Google Scholar
Cross Ref
- [4] . 2017. How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks). In IEEE International Conference on Computer Vision. 1021–1030.Google Scholar
Cross Ref
- [5] . 2018. Local gradient hexa pattern: A descriptor for face recognition and retrieval. IEEE Trans. Circ. Syst. Vid. Technol. 28, 1 (2018), 171–180.Google Scholar
Digital Library
- [6] . 2018. ArcFace: Additive angular margin loss for deep face recognition. In IEEE Conference on Computer Vision and Pattern Recognition. 4685–4694.Google Scholar
- [7] Jiankang Deng, Jia Guo, Zhou Yuxiang, Jinke Yu, Irene Kotsia, and Stefanos Zafeiriou. 2019. RetinaFace: Single-stage dense face localisation in the wild. In IEEE Conference on Computer Vision and Pattern Recognition. 5203–5212.Google Scholar
- [8] . 2011. Graph Laplace for occluded face completion and recognition. IEEE Trans. Image Process. 20, 8 (2011), 2329–2338.Google Scholar
Digital Library
- [9] . 2020. Masked face recognition with latent part detection. In 28th ACM International Conference on Multimedia. 2281–2289.Google Scholar
Digital Library
- [10] . 2020. Masked face recognition with generative data augmentation and domain constrained ranking. In 28th ACM International Conference on Multimedia. 2246–2254.Google Scholar
Digital Library
- [11] . 2016. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition. 770–778.Google Scholar
Cross Ref
- [12] . 2018. Local large-margin multi-metric learning for face and kinship verification. IEEE Trans. Circ. Syst. Vid. Technol. 28, 8 (2018), 1875–1891.Google Scholar
Digital Library
- [13] . 2018. Squeeze-and-Excitation networks. In IEEE Conference on Computer Vision and Pattern Recognition. 7132–7141.Google Scholar
Cross Ref
- [14] . 2022. Joint segmentation and identification feature learning for occlusion face recognition. IEEE Trans. Neural Netw. Learn. Syst. (2022).
DOI: Google ScholarCross Ref
- [15] . 2008. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. In Workshop on Faces in “Real-Life” Images: Detection, Alignment, and Recognition.Google Scholar
- [16] . 2017. Generative face completion. In IEEE Conference on Computer Vision and Pattern Recognition. 5892–5900.Google Scholar
Cross Ref
- [17] . 2021. xCos: An explainable cosine metric for face verification task. ACM Trans. Multim. Comput., Commun. Applic. 17, 3s (2021), 1–16.Google Scholar
Digital Library
- [18] . 2019. Fair loss: Margin-aware reinforcement learning for deep face recognition. In IEEE International Conference on Computer Vision. 10052–10061.Google Scholar
Cross Ref
- [19] . 2017. SphereFace: Deep hypersphere embedding for face recognition. In IEEE Conference on Computer Vision and Pattern Recognition. 6738–6746.Google Scholar
Cross Ref
- [20] . 2019. Multi-Fold Gabor, PCA, and ICA filter convolution descriptor for face recognition. IEEE Trans. Circ. Syst. Vid. Technol. 29, 1 (2019), 115–129.Google Scholar
Digital Library
- [21] . 2018. IARPA Janus Benchmark - C: Face dataset and protocol. In International Conference on Biometrics (ICB).Google Scholar
Cross Ref
- [22] . 2017. AgeDB: The first manually collected, in-the-wild age database. In IEEE Conference on Computer Vision and Pattern Recognition. 1997–2005.Google Scholar
Cross Ref
- [23] Aleix M. Martinez. 2002. Recognizing imprecisely localized, partially occluded, and expression variant faces from a single sample per class. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 6 (2002), 748–763.Google Scholar
- [24] . 2014. Robust low-rank regularized regression for face recognition with occlusion. In IEEE Conference on Computer Vision and Pattern Recognition. 21–26.Google Scholar
Digital Library
- [25] . 2015. FaceNet: A unified embedding for face recognition and clustering. In IEEE Conference on Computer Vision and Pattern Recognition. 815–823.Google Scholar
Cross Ref
- [26] . 2016. Frontal to profile face verification in the wild. In Workshop on Applications of Computer Vision. 1–9.Google Scholar
- [27] . 2021. Biased feature learning for occlusion invariant face recognition. In 29th International Conference on International Joint Conferences on Artificial Intelligence. 666–672.Google Scholar
- [28] Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.Google Scholar
- [29] . 2017. Disguised face identification (DFI) with facial keypoints using spatial fusion convolutional network. In IEEE International Conference on Computer Vision. 1648–1655.Google Scholar
Cross Ref
- [30] . 2019. Occlusion robust face recognition based on mask learning with pairwise differential siamese network. In IEEE International Conference on Computer Vision. 773–782.Google Scholar
Cross Ref
- [31] . 2015. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition. 1–9.Google Scholar
Cross Ref
- [32] . 2014. DeepFace: Closing the gap to human-level performance in face verification. In IEEE Conference on Computer Vision and Pattern Recognition. 1701–1708.Google Scholar
Digital Library
- [33] . 2017. Occlusion robust face recognition based on mask learning. In IEEE International Conference on Image Processing. 3795–3799.Google Scholar
Digital Library
- [34] . 2018. CosFace: Large margin cosine loss for deep face recognition. In IEEE Conference on Computer Vision and Pattern Recognition. 5265–5274.Google Scholar
Cross Ref
- [35] . 2018. Orthogonal deep features decomposition for age-invariant face recognition. In European Conference on Computer Vision. 764–779.Google Scholar
Digital Library
- [36] . 2016. A discriminative feature learning approach for deep face recognition. In European Conference on Computer Vision. 499–515.Google Scholar
Cross Ref
- [37] . 2011. Robust sparse coding for face recognition. In IEEE Conference on Computer Vision and Pattern Recognition. 625–632.Google Scholar
Digital Library
- [38] Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z. Li. 2014. Learning face representation from scratch. arXiv preprint arXiv:1411.7923.Google Scholar
- [39] . 2018. DeMeshNet: Blind face inpainting for deep MeshFace verification. IEEE Trans. Inf. Forens. Secur. 13, 3 (2018), 637–647.Google Scholar
Cross Ref
- [40] . 2019. 3D-aided dual-agent GANs for unconstrained face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 41, 10 (2019), 2380–2394.Google Scholar
Digital Library
- [41] . 2018. Improving periocular recognition by explicit attention to critical regions in deep neural network. IEEE Trans. Inf. Forens. Secur. 13, 12 (2018), 2937–2952.Google Scholar
Digital Library
Index Terms
Local Eyebrow Feature Attention Network for Masked Face Recognition
Recommendations
Masked Face Recognition with 3D Facial Geometric Attributes
IPMV '22: Proceedings of the 4th International Conference on Image Processing and Machine VisionDuring the coronavirus pandemic, the demand for contactless biometrics technology has promoted the development of masked face recognition. Training a masked face recognition model needs to address two crucial issues: a lack of large-scale realistic ...
Dual-Proxy Modeling for Masked Face Recognition
AbstractWith the recent worldwide COVID-19 pandemic, almost everyone wears a mask daily, leading to severe degradation in the accuracy of conventional face recognition systems. Several works improve the performance of masked faces by adopting synthetic ...
VGG16-random fourier hybrid model for masked face recognition
AbstractWith the recent COVID-19 pandemic, wearing masks has become a necessity in our daily lives. People are encouraged to wear masks to protect themselves from the outside world and thus from infection with COVID-19. The presence of masks raised ...






Comments