skip to main content
research-article

Local Eyebrow Feature Attention Network for Masked Face Recognition

Published:25 February 2023Publication History
Skip Abstract Section

Abstract

During the COVID-19 coronavirus epidemic, wearing masks has become increasingly popular. Traditional occlusion face recognition algorithms are almost ineffective for such heavy mask occlusion. Therefore, it is urgent to improve the recognition performance of the existing face recognition technology on masked faces. Due to the limited visible feature points of the masked face image relative to the normal face image, we have to exploit the identification potential of eyebrow (referring to eyes and brows) features. This article proposes a local eyebrow feature attention network for masked face recognition, which consists of feature extraction, eyebrow region pooling, and feature fusion. To highlight the eyebrow region, we first use the eyebrow region pooling to separate the local features of eyebrows from the learned overall facial features. We then make full use of the symmetry of left and right eyebrows to emphasize their discriminant ability, due to the inadequate fine information of the low-resolution eyebrows. In particular, in view of the symmetrical similarity between eyebrow pairs and the subordinate relationship between facial components and the whole, we propose a feature fusion model based on graph convolutional network (GCN) to learn the feature association structure of eye features, brow features, and global facial features. We construct the benchmark datasets for masked face recognition to validate our approach, including real-world masked face recognition dataset (RMFRD) and synthetic masked face recognition dataset (SMFRD). Extensive experimental results on both public datasets and our built masked face datasets show that our approach significantly outperforms the state-of-the-arts.

REFERENCES

  1. [1] Ngan. 2020. NIST. https://www.domesticpreparedness.com/updates/nist-launches-investigation-of-face-masks-effect-on-face-recognition-software/.Google ScholarGoogle Scholar
  2. [2] Davis E. King. 2009. Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research 10 (2009), 1755–1758.Google ScholarGoogle Scholar
  3. [3] Kentaro Wada. 2021. Labelme: Image Polygonal Annotation with Python. Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Bulat Adrian and Tzimiropoulos Georgios. 2017. How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks). In IEEE International Conference on Computer Vision. 10211030.Google ScholarGoogle ScholarCross RefCross Ref
  5. [5] Chakraborty Soumendu, Singh Satish K., and Chakraborty Pavan. 2018. Local gradient hexa pattern: A descriptor for face recognition and retrieval. IEEE Trans. Circ. Syst. Vid. Technol. 28, 1 (2018), 171180.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. [6] Deng Jiankang, Guo Jia, Xue Niannan, and Zafeiriou Stefanos. 2018. ArcFace: Additive angular margin loss for deep face recognition. In IEEE Conference on Computer Vision and Pattern Recognition. 46854694.Google ScholarGoogle Scholar
  7. [7] Jiankang Deng, Jia Guo, Zhou Yuxiang, Jinke Yu, Irene Kotsia, and Stefanos Zafeiriou. 2019. RetinaFace: Single-stage dense face localisation in the wild. In IEEE Conference on Computer Vision and Pattern Recognition. 5203–5212.Google ScholarGoogle Scholar
  8. [8] Deng Yue, Dai Qionghai, and Zhang Zengke. 2011. Graph Laplace for occluded face completion and recognition. IEEE Trans. Image Process. 20, 8 (2011), 23292338.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. [9] Ding Feifei, Peng Peixi, Huang Yangru, Geng Mengyue, and Tian Yonghong. 2020. Masked face recognition with latent part detection. In 28th ACM International Conference on Multimedia. 22812289.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. [10] Geng Mengyue, Peng Peixi, Huang Yangru, and Tian Yonghong. 2020. Masked face recognition with generative data augmentation and domain constrained ranking. In 28th ACM International Conference on Multimedia. 22462254.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. [11] He Kaiming, Zhang Xiangyu, Ren Shaoqing, and Sun Jian. 2016. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition. 770778.Google ScholarGoogle ScholarCross RefCross Ref
  12. [12] Hu Junlin, Lu Jiwen, Tan Yappeng, Yuan Junsong, and Zhou Jie. 2018. Local large-margin multi-metric learning for face and kinship verification. IEEE Trans. Circ. Syst. Vid. Technol. 28, 8 (2018), 18751891.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. [13] Hu Jie, Shen Li, and Sun Gang. 2018. Squeeze-and-Excitation networks. In IEEE Conference on Computer Vision and Pattern Recognition. 71327141.Google ScholarGoogle ScholarCross RefCross Ref
  14. [14] Huang Baojin, Wang Zhongyuan, Jiang Kui, Zou Qin, Tian Xin, Lu Tao, and Han Zhen. 2022. Joint segmentation and identification feature learning for occlusion face recognition. IEEE Trans. Neural Netw. Learn. Syst. (2022). DOI:Google ScholarGoogle ScholarCross RefCross Ref
  15. [15] Huang Gary B., Mattar Marwan, Berg Tamara, and Learned-Miller Eric. 2008. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. In Workshop on Faces in “Real-Life” Images: Detection, Alignment, and Recognition.Google ScholarGoogle Scholar
  16. [16] Li Yijun, Liu Sifei, Yang Jimei, and Yang Minghsuan. 2017. Generative face completion. In IEEE Conference on Computer Vision and Pattern Recognition. 58925900.Google ScholarGoogle ScholarCross RefCross Ref
  17. [17] Lin Yu-Sheng, Liu Zhe-Yu, Chen Yu-An, Wang Yu-Siang, Chang Ya-Liang, and Hsu Winston H.. 2021. xCos: An explainable cosine metric for face verification task. ACM Trans. Multim. Comput., Commun. Applic. 17, 3s (2021), 116.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. [18] Liu Bingyu, Deng Weihong, Zhong Yaoyao, Wang Mei, Hu Jiani, Tao Xunqiang, and Huang Yaohai. 2019. Fair loss: Margin-aware reinforcement learning for deep face recognition. In IEEE International Conference on Computer Vision. 1005210061.Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Liu Weiyang, Wen Yandong, Yu Zhiding, Li Ming, Raj Bhiksha, and Song Le. 2017. SphereFace: Deep hypersphere embedding for face recognition. In IEEE Conference on Computer Vision and Pattern Recognition. 67386746.Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Low Chengyaw, Teoh Andrew Beng Jin, and Ng Congjie. 2019. Multi-Fold Gabor, PCA, and ICA filter convolution descriptor for face recognition. IEEE Trans. Circ. Syst. Vid. Technol. 29, 1 (2019), 115129.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. [21] Maze B., Adams J., Duncan J. A., Kalka N., and Grother P.. 2018. IARPA Janus Benchmark - C: Face dataset and protocol. In International Conference on Biometrics (ICB).Google ScholarGoogle ScholarCross RefCross Ref
  22. [22] Moschoglou Stylianos, Papaioannou Athanasios, Sagonas Christos, Deng Jiankang, Kotsia Irene, and Zafeiriou Stefanos. 2017. AgeDB: The first manually collected, in-the-wild age database. In IEEE Conference on Computer Vision and Pattern Recognition. 19972005.Google ScholarGoogle ScholarCross RefCross Ref
  23. [23] Aleix M. Martinez. 2002. Recognizing imprecisely localized, partially occluded, and expression variant faces from a single sample per class. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 6 (2002), 748–763.Google ScholarGoogle Scholar
  24. [24] Qian Jianjun, Yang Jian, Zhang Fanglong, and Lin Zhouchen. 2014. Robust low-rank regularized regression for face recognition with occlusion. In IEEE Conference on Computer Vision and Pattern Recognition. 2126.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. [25] Schroff Florian, Kalenichenko Dmitry, and Philbin James. 2015. FaceNet: A unified embedding for face recognition and clustering. In IEEE Conference on Computer Vision and Pattern Recognition. 815823.Google ScholarGoogle ScholarCross RefCross Ref
  26. [26] Sengupta Soumyadip, Chen Juncheng, Castillo Carlos D., Patel Vishal M., Chellappa Rama, and Jacobs David W.. 2016. Frontal to profile face verification in the wild. In Workshop on Applications of Computer Vision. 19.Google ScholarGoogle Scholar
  27. [27] Shao Changbin, Huo Jing, Qi Lei, Feng Zhen-Hua, Li Wenbin, Dong Chuanqi, and Gao Yang. 2021. Biased feature learning for occlusion invariant face recognition. In 29th International Conference on International Joint Conferences on Artificial Intelligence. 666672.Google ScholarGoogle Scholar
  28. [28] Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.Google ScholarGoogle Scholar
  29. [29] Singh Amarjot, Patii Devendra, Reddy G. Meghana, and Omkar S. N.. 2017. Disguised face identification (DFI) with facial keypoints using spatial fusion convolutional network. In IEEE International Conference on Computer Vision. 16481655.Google ScholarGoogle ScholarCross RefCross Ref
  30. [30] Song Lingxue, Gong Dihong, Li Zhifeng, Liu Changsong, and Liu Wei. 2019. Occlusion robust face recognition based on mask learning with pairwise differential siamese network. In IEEE International Conference on Computer Vision. 773782.Google ScholarGoogle ScholarCross RefCross Ref
  31. [31] Szegedy Christian, Liu Wei, Jia Yangqing, Sermanet Pierre, Reed Scott, Anguelov Dragomir, Erhan Dumitru, Vanhoucke Vincent, and Rabinovich Andrew. 2015. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition. 19.Google ScholarGoogle ScholarCross RefCross Ref
  32. [32] Taigman Yaniv, Yang Ming, Ranzato Marcaurelio, and Wolf Lior. 2014. DeepFace: Closing the gap to human-level performance in face verification. In IEEE Conference on Computer Vision and Pattern Recognition. 17011708.Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. [33] Wan Weitao and Chen Jiansheng. 2017. Occlusion robust face recognition based on mask learning. In IEEE International Conference on Image Processing. 37953799.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. [34] Wang Hao, Wang Yitong, Zhou Zheng, Ji Xing, Gong Dihong, Zhou Jingchao, Li Zhifeng, and Liu Wei. 2018. CosFace: Large margin cosine loss for deep face recognition. In IEEE Conference on Computer Vision and Pattern Recognition. 52655274.Google ScholarGoogle ScholarCross RefCross Ref
  35. [35] Wang Yitong, Gong Dihong, Zhou Zheng, Ji Xing, Wang Hao, Li Zhifeng, Liu Wei, and Zhang Tong. 2018. Orthogonal deep features decomposition for age-invariant face recognition. In European Conference on Computer Vision. 764779.Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. [36] Wen Yandong, Zhang Kaipeng, Li Zhifeng, and Qiao Yu. 2016. A discriminative feature learning approach for deep face recognition. In European Conference on Computer Vision. 499515.Google ScholarGoogle ScholarCross RefCross Ref
  37. [37] Yang Meng, Zhang Lei, Yang Jian, and Zhang David. 2011. Robust sparse coding for face recognition. In IEEE Conference on Computer Vision and Pattern Recognition. 625632.Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. [38] Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z. Li. 2014. Learning face representation from scratch. arXiv preprint arXiv:1411.7923.Google ScholarGoogle Scholar
  39. [39] Zhang Shu, He Ran, Sun Zhenan, and Tan Tieniu. 2018. DeMeshNet: Blind face inpainting for deep MeshFace verification. IEEE Trans. Inf. Forens. Secur. 13, 3 (2018), 637647.Google ScholarGoogle ScholarCross RefCross Ref
  40. [40] Zhao Jian, Xiong Lin, Li Jianshu, Xing Junliang, Yan Shuicheng, and Feng Jiashi. 2019. 3D-aided dual-agent GANs for unconstrained face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 41, 10 (2019), 23802394.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. [41] Zhao Zijing and Kumar Ajay. 2018. Improving periocular recognition by explicit attention to critical regions in deep neural network. IEEE Trans. Inf. Forens. Secur. 13, 12 (2018), 29372952.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Local Eyebrow Feature Attention Network for Masked Face Recognition

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Multimedia Computing, Communications, and Applications
        ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 19, Issue 3
        May 2023
        514 pages
        ISSN:1551-6857
        EISSN:1551-6865
        DOI:10.1145/3582886
        • Editor:
        • Abdulmotaleb El Saddik
        Issue’s Table of Contents

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 25 February 2023
        • Online AM: 27 October 2022
        • Accepted: 24 October 2022
        • Revised: 9 September 2022
        • Received: 20 January 2022
        Published in tomm Volume 19, Issue 3

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
      • Article Metrics

        • Downloads (Last 12 months)173
        • Downloads (Last 6 weeks)19

        Other Metrics

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      View Full Text

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!