skip to main content
research-article

Toward High-quality Face-Mask Occluded Restoration

Published:06 January 2023Publication History
Skip Abstract Section

Abstract

Face-mask occluded restoration aims at restoring the masked region of a human face, which has attracted increasing attention in the context of the COVID-19 pandemic. One major challenge of this task is the large visual variance of masks in the real world. To solve it we first construct a large-scale Face-mask Occluded Restoration (FMOR) dataset, which contains 5,500 unmasked images and 5,500 face-mask occluded images with various illuminations, and involves 1,100 subjects of different races, face orientations, and mask types. Moreover, we propose a Face-Mask Occluded Detection and Restoration (FMODR) framework, which can detect face-mask regions with large visual variations and restore them to realistic human faces. In particular, our FMODR contains a self-adaptive contextual attention module specifically designed for this task, which is able to exploit the contextual information and correlations of adjacent pixels for achieving high realism of the restored faces, which are however often neglected in existing contextual attention models. Our framework achieves state-of-the-art results of face restoration on three datasets, including CelebA, AR, and our FMOR datasets. Moreover, experimental results on AR and FMOR datasets demonstrate that our framework can significantly improve masked face recognition and verification performance.

REFERENCES

  1. [1] Arya Sunil and Mount D. M.. 1998. ANN: Library for approximate nearest neighbor searching. In Proceedings of the IEEE CGC Workshop on Computational Geometry, Providence, RI.Google ScholarGoogle Scholar
  2. [2] Ashikhmin Michael. 2001. Synthesizing natural textures. In Proceedings of the 2001 Symposium on Interactive 3D Graphics. 217226.Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. [3] Banerjee Samik and Das Sukhendu. 2020. SD-GAN: Structural and denoising GAN reveals facial parts under occlusion[J]. arXiv preprint arXiv:2002.08448(2020).Google ScholarGoogle Scholar
  4. [4] Barnes Connelly, Shechtman Eli, Finkelstein Adam, and Goldman Dan B.. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics 28, 3 (2009), 24.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. [5] Bornard Raphaël, Lecan Emmanuelle, Laborelli Louis, and Chenot Jean-Hugues. 2002. Missing data correction in still images and image sequences. In Proceedings of the 10th ACM International Conference on Multimedia. 355361.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. [6] Candès Emmanuel J., Li Xiaodong, Ma Yi, and Wright John. 2011. Robust principal component analysis? The Journal of the ACM 58, 3 (2011), 137.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. [7] Chen Chaofeng, Li Xiaoming, Yang Lingbo, Lin Xianhui, Zhang Lei, and Wong Kwan-Yee K.. 2021. Progressive semantic-aware style transformation for blind face restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1189611905.Google ScholarGoogle ScholarCross RefCross Ref
  8. [8] Chen Xiang, Qing Linbo, He Xiaohai, Su Jie, and Peng Yonghong. 2018. From eyes to face synthesis: A new approach for human-centered smart surveillance. IEEE Access 6 (2018), 1456714575.Google ScholarGoogle ScholarCross RefCross Ref
  9. [9] Criminisi Antonio, Pérez Patrick, and Toyama Kentaro. 2004. Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on Image Processing 13, 9 (2004), 12001212.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. [10] Deng Jiankang, Guo Jia, Liu Tongliang, Gong Mingming, and Zafeiriou Stefanos. 2020. Sub-center arcface: Boosting face recognition by large-scale noisy web faces. In Proceedings of the European Conference on Computer Vision. Springer, 741757.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. [11] Efros Alexei A. and Freeman William T.. 2001. Image quilting for texture synthesis and transfer. In Proceedings of the CVPR. ACM, 341346.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. [12] Efros Alexei A. and Leung Thomas K.. 1999. Texture synthesis by non-parametric sampling. In Proceedings of the ICCV. IEEE, 10331038.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. [13] Gao Wen, Cao Bo, Shan Shiguang, Chen Xilin, Zhou Delong, Zhang Xiaohua, and Zhao Debin. 2007. The CAS-PEAL large-scale chinese face database and baseline evaluations. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans 38, 1 (2007), 149161.Google ScholarGoogle Scholar
  14. [14] Ge Shiming, Li Jia, Ye Qiting, and Luo Zhao. 2017. Detecting masked faces in the wild with lle-cnns. In Proceedings of the CVPR. 26822690.Google ScholarGoogle ScholarCross RefCross Ref
  15. [15] Goodfellow Ian, Pouget-Abadie Jean, Mirza Mehdi, Xu Bing, Warde-Farley David, Ozair Sherjil, Courville Aaron, and Bengio Yoshua. 2014. Generative adversarial nets. In Proceedings of the NeurIPS. 26722680.Google ScholarGoogle Scholar
  16. [16] Gross Ralph. 2005. Face databases. In Proceedings of the Handbook of Face Recognition. Springer, 301327.Google ScholarGoogle ScholarCross RefCross Ref
  17. [17] Gross Ralph, Matthews Iain, Cohn Jeffrey, Kanade Takeo, and Baker Simon. 2010. Multi-pie. Image and Vision Computing 28, 5 (2010), 807813.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. [18] Guo Xiefan, Yang Hongyu, and Huang Di. 2021. Image inpainting via conditional texture and structure dual generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1413414143.Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] He Kaiming and Sun Jian. 2012. Statistics of patch offsets for image completion. In Proceedings of the ECCV. Springer, 1629.Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] He Lingxiao, Li Haiqing, Zhang Qi, Sun Zhenan, and He Zhaofeng. 2016. Multiscale representation for partial face recognition under near infrared illumination. In Proceedings of the BTAS. IEEE, 17.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. [21] He Xiaofei and Niyogi Partha. 2004. Locality preserving projections. In Proceedings of the Advances in Neural Information Processing Systems.153160.Google ScholarGoogle Scholar
  22. [22] Hermans Alexander, Beyer Lucas, and Leibe Bastian. 2017. In defense of the triplet loss for person re-identification[J]. arXiv preprint arXiv:1703.07737(2017).Google ScholarGoogle Scholar
  23. [23] Iizuka Satoshi, Simo-Serra Edgar, and Ishikawa Hiroshi. 2017. Globally and locally consistent image completion. ACM Transactions on Graphics 36, 4 (2017), 114.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. [24] Isola Phillip, Zhu Jun-Yan, Zhou Tinghui, and Efros Alexei A.. 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the CVPR. 11251134.Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Jian-Ke Wen. 2013. Ways to retouch photos. Laboratory Ence 16, 4 (2013), 14–17.Google ScholarGoogle Scholar
  26. [26] Lee Jino, Lee Dong-Kyu, and Park Rae-Hong. 2012. Robust exemplar-based inpainting algorithm using region segmentation. IEEE Transactions on Consumer Electronics 58, 2 (2012), 553561.Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Lei Zhen, Liao Shengcai, He Ran, Pietikainen Matti, and Li Stan Z.. 2008. Gabor volume based local binary pattern for face representation and recognition. In Proceedings of the International Conference on Automatic Face Gesture Recognition. IEEE, 16.Google ScholarGoogle ScholarCross RefCross Ref
  28. [28] Li Ang, Qi Jianzhong, Zhang Rui, and Kotagiri Ramamohanarao. 2019. Boosted gan with semantically interpretable information for image inpainting. In Proceedings of the IJCNN. IEEE, 18.Google ScholarGoogle ScholarCross RefCross Ref
  29. [29] Li Chenyu, Ge Shiming, Zhang Daichi, and Li Jia. 2020. Look through masks: Towards masked face recognition with de-occlusion distillation. In Proceedings of the 28th ACM International Conference on Multimedia. 30163024.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. [30] Li Yijun, Liu Sifei, Yang Jimei, and Yang Ming-Hsuan. 2017. Generative face completion. In Proceedings of the CVPR. 39113919.Google ScholarGoogle ScholarCross RefCross Ref
  31. [31] Liu Ce, Shum Heung-Yeung, and Zhang Chang-Shui. 2001. A two-step approach to hallucinating faces: Global parametric model and local nonparametric model. In Proceedings of the CVPR. IEEE, I–I.Google ScholarGoogle Scholar
  32. [32] Liu Ziwei, Luo Ping, Wang Xiaogang, and Tang Xiaoou. 2015. Deep learning face attributes in the wild. In Proceedings of the ICCV. 37303738.Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. [33] Parkhi Omkar M., Vedaldi Andrea, and Zisserman Andrew. 2015. Deep Face Recognition. British Machine Vision Association.Google ScholarGoogle Scholar
  34. [34] Pathak Deepak, Krahenbuhl Philipp, Donahue Jeff, Darrell Trevor, and Efros Alexei A.. 2016. Context encoders: Feature learning by inpainting. In Proceedings of the CVPR. 25362544.Google ScholarGoogle ScholarCross RefCross Ref
  35. [35] Russell Bryan Christopher, Torralba Antonio J., Murphy Kevin Patrick, and Freeman William T.. 2008. LabelMe. International Journal of Computer Vision 77, 1 (2008), 157–173.Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. [36] Torralba Antonio, Fergus Rob, and Freeman William T.. 2008. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 11 (2008), 19581970.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. [37] Tran Luan, Yin Xi, and Liu Xiaoming. 2017. Disentangled representation learning gan for pose-invariant face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 14151424.Google ScholarGoogle ScholarCross RefCross Ref
  38. [38] Turk Matthew A. and Pentland Alex P.. 1991. Face recognition using eigenfaces. In Proceedings of the CVPR. IEEE Computer Society, 586587.Google ScholarGoogle ScholarCross RefCross Ref
  39. [39] Wang Qiong and Yang Jingyu. 2006. Eye detection in facial images with unconstrained background. JPRR 1, 1 (2006), 5562.Google ScholarGoogle ScholarCross RefCross Ref
  40. [40] Wang Xintao, Li Yu, Zhang Honglun, and Shan Ying. 2021. Towards real-world blind face restoration with generative facial prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 91689178.Google ScholarGoogle ScholarCross RefCross Ref
  41. [41] Wang Yi, Tao Xin, Qi Xiaojuan, Shen Xiaoyong, and Jia Jiaya. 2018. Image inpainting via generative multi-column convolutional neural networks. In Proceedings of the NeurIPS. 331340.Google ScholarGoogle Scholar
  42. [42] Wang Zhongyuan, Wang Guangcheng, Huang Baojin, Zhangyang Xiong, Qi Hong, Hao Wu, Peng Yi, Kui Jiang, Nanxi Wang, Yingjiao Pei, Heling Chen, Yu Miao, Zhibing Huang, and Jinbi Liang. 2020. Masked face recognition dataset and application. arXiv preprint arXiv:2003.09093 (2020).Google ScholarGoogle Scholar
  43. [43] Wright John, Yang Allen Y., Ganesh Arvind, Sastry S. Shankar, and Ma Yi. 2008. Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 2 (2008), 210227.Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. [44] Xia Weihao, Yang Yujiu, Xue Jing-Hao, and Wu Baoyuan. 2021. Towards open-world text-guided face image generation and manipulation[J]. arXiv preprint arXiv:2104.08910 (2021).Google ScholarGoogle Scholar
  45. [45] Yang Chao, Lu Xin, Lin Zhe, Shechtman Eli, Wang Oliver, and Li Hao. 2017. High-resolution image inpainting using multi-scale neural patch synthesis. In Proceedings of the CVPR. 67216729.Google ScholarGoogle ScholarCross RefCross Ref
  46. [46] Yin Bangjie, Tran Luan, Li Haoxiang, Shen Xiaohui, and Liu Xiaoming. 2019. Towards interpretable face recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 93489357.Google ScholarGoogle ScholarCross RefCross Ref
  47. [47] Yu Jiahui, Lin Zhe, Yang Jimei, Shen Xiaohui, Lu Xin, and Huang Thomas S.. 2018. Generative image inpainting with contextual attention. In Proceedings of the CVPR. 55055514.Google ScholarGoogle ScholarCross RefCross Ref
  48. [48] Yu Jiahui, Lin Zhe, Yang Jimei, Shen Xiaohui, Lu Xin, and Huang Thomas S.. 2019. Free-form image inpainting with gated convolution. In Proceedings of the ICCV. 44714480.Google ScholarGoogle ScholarCross RefCross Ref
  49. [49] Yu Lingyun, Xie Hongtao, and Zhang Yongdong. 2021. Multimodal learning for temporally coherent talking face generation with articulator synergy. IEEE Transactions on Multimedia 24 (2021), 2950–2962.Google ScholarGoogle Scholar
  50. [50] Yu Tao, Guo Zongyu, Jin Xin, Wu Shilin, Chen Zhibo, Li Weiping, Zhang Zhizheng, and Liu Sen. 2020. Region normalization for image inpainting. In Proceedings of the AAAI Conference on Artificial Intelligence. 1273312740.Google ScholarGoogle ScholarCross RefCross Ref
  51. [51] Zhang Xian, Shi Canghong, Wang Xin, Wu Xi, Li Xiaojie, Lv Jiancheng, and Mumtaz Imran. 2021. Face inpainting based on GAN by facial prediction and fusion as guidance information. Applied Soft Computing 111 (2021), 107626.Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. [52] Zhao Fang, Feng Jiashi, Zhao Jian, Yang Wenhan, and Yan Shuicheng. 2017. Robust lstm-autoencoders for face de-occlusion in the wild. IEEE Transactions on Image Processing 27, 2 (2017), 778790.Google ScholarGoogle ScholarCross RefCross Ref
  53. [53] Zhao Jian, Cheng Yu, Xu Yan, Xiong Lin, Li Jianshu, Zhao Fang, Jayashree Karlekar, Pranata Sugiri, Shen Shengmei, Xing Junliang, et al. 2018. Towards pose invariant face recognition in the wild. In Proceedings of the CVPR. 22072216.Google ScholarGoogle ScholarCross RefCross Ref
  54. [54] Zhao Jian, Li Jianshu, Tu Xiaoguang, Zhao Fang, Xin Yuan, Xing Junliang, Liu Hengzhu, Yan Shuicheng, and Feng Jiashi. 2019. Multi-prototype networks for unconstrained set-based face recognition. In Proceedings of the 28th International Joint Conference on Artificial Intelligence. 4397–4403.Google ScholarGoogle Scholar
  55. [55] Zhao Jian, Xiong Lin, Jayashree Panasonic Karlekar, Li Jianshu, Zhao Fang, Wang Zhecan, Pranata Panasonic Sugiri, Shen Panasonic Shengmei, Yan Shuicheng, and Feng Jiashi. 2017. Dual-agent gans for photorealistic and identity preserving profile face synthesis. In Proceedings of the NeurIPS. 6676.Google ScholarGoogle Scholar
  56. [56] Zhou Tong, Ding Changxing, Lin Shaowen, Wang Xinchao, and Tao Dacheng. 2020. Learning oracle attention for high-fidelity face completion. In Proceedings of the CVPR. 76807689.Google ScholarGoogle ScholarCross RefCross Ref
  57. [57] Zhou Xiuzhuang, Jin Kai, Shang Yuanyuan, and Guo Guodong. 2018. Visually interpretable representation learning for depression recognition from facial images. IEEE Transactions on Affective Computing 11, 3 (2018), 542552.Google ScholarGoogle ScholarCross RefCross Ref
  58. [58] Zhu Zheng, Huang Guan, Deng Jiankang, Yun Ye, Junjie Huang, Xinze Chen, Jiagang Zhu, Tian Yang, Jiwen Lu, Dalong Du, and Jie Zhou. 2021. WebFace260M: A benchmark unveiling the power of million-scale deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10492–10502.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Toward High-quality Face-Mask Occluded Restoration

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Multimedia Computing, Communications, and Applications
      ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 19, Issue 1
      January 2023
      505 pages
      ISSN:1551-6857
      EISSN:1551-6865
      DOI:10.1145/3572858
      • Editor:
      • Abdulmotaleb El Saddik
      Issue’s Table of Contents

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 6 January 2023
      • Online AM: 25 March 2022
      • Accepted: 6 March 2022
      • Revised: 19 November 2021
      • Received: 5 July 2021
      Published in tomm Volume 19, Issue 1

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Refereed

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    View Full Text

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!