skip to main content
research-article

Expression Robust 3D Facial Landmarking via Progressive Coarse-to-Fine Tuning

Authors Info & Claims
Published:13 February 2019Publication History
Skip Abstract Section

Abstract

Facial landmarking is a fundamental task in automatic machine-based face analysis. The majority of existing techniques for such a problem are based on 2D images; however, they suffer from illumination and pose variations that may largely degrade landmarking performance. The emergence of 3D data theoretically provides an alternative to overcome these weaknesses in the 2D domain. This article proposes a novel approach to 3D facial landmarking, which combines both the advantages of feature-based methods as well as model-based ones in a progressive three-stage coarse-to-fine manner (initial, intermediate, and fine stages). For the initial stage, a few fiducial landmarks (i.e., the nose tip and two inner eye corners) are robustly detected through curvature analysis, and these points are further exploited to initialize the subsequent stage. For the intermediate stage, a statistical model is learned in the feature space of three normal components of the facial point-cloud rather than the smooth original coordinates, namely Active Normal Model (ANM). For the fine stage, cascaded regression is employed to locally refine the landmarks according to their geometry attributes. The proposed approach can accurately localize dozens of fiducial points on each 3D face scan, greatly surpassing the feature-based ones, and it also improves the state of the art of the model-based ones in two aspects: sensitivity to initialization and deficiency in discrimination. The proposed method is evaluated on the BU-3DFE, Bosphorus, and BU-4DFE databases, and competitive results are achieved in comparison with counterparts in the literature, clearly demonstrating its effectiveness.

References

  1. Stefano Berretti, Boulbaba Ben Amor, Mohamed Daoudi, and Alberto Del Bimbo. 2011. 3D facial expression recognition using SIFT descriptors of automatically detected keypoints. The Visual Computer 27, 11 (2011), 1021--1036. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Paul J. Besl and Ramesh C. Jain. 1986. Invariant surface characteristics for 3D object recognition in range images. Computer Vision, Graphics, and Image Processing 33, 1 (1986), 33--80. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Volker Blanz and Thomas Vetter. 2003. Face recognition based on fitting a 3D morphable model. IEEE Transactions on Pattern Analysis and Machine Intelligence 25, 9 (2003), 1063--1074. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Leo Breiman. 2001. Random forests. Machine Learning 45, 1 (2001), 5--32. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Adrian Bulat and Georgios Tzimiropoulos. 2017. How far are we from solving the 2d 8 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks). In Proceedings on the IEEE International Conference on Computer Vision. IEEE, Venice, Italy, 1021--1030.Google ScholarGoogle ScholarCross RefCross Ref
  6. Xavier P. Burgos-Artizzu, Pietro Perona, and Piotr Dollár. 2013. Robust face landmark estimation under occlusion. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, Sydney, NSW, Australia, 1513--1520. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Dirk Colbry, George Stockman, and Anil Jain. 2005. Detection of anchor points for 3d face verification. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. IEEE, San Diego, CA, USA, 118--118. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Alessandro Colombo, Claudio Cusano, and Raimondo Schettini. 2006. 3D face detection using curvature analysis. Pattern Recognition 39, 3 (2006), 444--455. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Timothy F. Cootes, Gareth J. Edwards, and Christopher J. Taylor. 2001. Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 6 (2001), 681--685. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Timothy F. Cootes, Christopher J. Taylor, David H. Cooper, and Jim Graham. 1995. Active shape models-their training and application. Computer Vision and Image Understanding 61, 1 (1995), 38--59. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Clement Creusot, Nick Pears, and Jim Austin. 2013. A machine-learning approach to keypoint detection and landmarking on 3D meshes. International Journal of Computer Vision 102, 1--3 (2013), 146--179. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Hamdi Dibeklioglu, Albert A. Salah, and Lale Akarun. 2008. 3D facial landmarking under expression, pose, and occlusion variations. In Proceedings of the IEEE International Conference on Biometrics: Theory, Applications and Systems. IEEE, Arlington, VA, 1--6.Google ScholarGoogle ScholarCross RefCross Ref
  13. Gabriele Fanelli, Matthias Dantone, Juergen Gall, Andrea Fossati, and Luc Van Gool. 2013. Random forests for real time 3d face analysis. International Journal of Computer Vision 101, 3 (2013), 437--458. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Gabriele Fanelli, Matthias Dantone, and Luc Van Gool. 2013. Real time 3D face alignment with random forests-based active appearance models. In Proceedings of the IEEE International Conference on Face and Gesture Recognition. IEEE, Shanghai, China, 1--8.Google ScholarGoogle ScholarCross RefCross Ref
  15. Tianhong Fang, Xi Zhao, Omar Ocegueda, Shishir K. Shah, and Ioannis A. Kakadiaris. 2012. 3D/4D facial expression analysis: An advanced annotated face model approach. Image and Vision Computing 30, 10 (2012), 738--749. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Syed Zulqarnain Gilani, Ajmal Mian, and Peter Eastwood. 2017. Deep, dense and accurate 3D face correspondence for generating population specific deformable models. Pattern Recognition 69 (2017), 238--250. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Shalini Gupta, Kenneth R. Castleman, Mia K. Markey, and Alan C. Bovik. 2010. Texas 3D face recognition database. In Proceedings of the IEEE Southwest Symposium on Image Analysis and Interpretation. IEEE, Austin, TX, 97--100.Google ScholarGoogle Scholar
  18. Shalini Gupta, Mia K. Markey, and Alan C. Bovik. 2010. Anthropometric 3D face recognition. International Journal of Computer Vision 90, 3 (2010), 331--349. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Di Huang, Mohsen Ardabilian, Yunhong Wang, and Liming Chen. 2012. 3-D face recognition using eLBP-based facial description and local feature hybrid matching. IEEE Transactions on Information Forensics and Security 7, 5 (2012), 1551--1565. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Di Huang, Karima Ouji, Mohsen Ardabilian, Yunhong Wang, and Liming Chen. 2011. 3D face recognition based on local shape patterns and sparse representation classifier. In Proceedings of the International Conference on Multimedia Modeling. Springer, Taipei, Taiwan, 206--216. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Vuong Le, Jonathan Brandt, Zhe Lin, Lubomir Bourdev, and Thomas S. Huang. 2012. Interactive facial feature localization. In Proceedings of the European Conference on Computer Vision. Springer, Florence, Italy, 679--692. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Huibin Li, Huaxiong Ding, Di Huang, Yunhong Wang, Xi Zhao, Jean-Marie Morvan, and Liming Chen. 2015. An efficient multimodal 2D+ 3D feature-based approach to automatic facial expression recognition. Computer Vision and Image Understanding 140 (2015), 83--92. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Huibin Li, Di Huang, Jean-Marie Morvan, and Liming Chen. 2011. Learning weighted sparse representation of encoded facial normal information for expression-robust 3D face recognition. In Proceedings of the IEEE International Conference on Biometrics. IEEE, Washington, DC, 1--7. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Huibin Li, Di Huang, Jean-Marie Morvan, Yunhong Wang, and Liming Chen. 2015. Towards 3D face recognition in the real: A registration-free approach using fine-grained matching of 3D keypoint descriptors. International Journal of Computer Vision 113, 2 (2015), 128--142. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Xiaoguang Lu and Anil K. Jain. 2006. Automatic feature extraction for multiview 3D face recognition. In Proceedings of the International Conference on Automatic Face and Gesture Recognition. IEEE, Southampton, 585--590. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Ajmal Mian, Mohammed Bennamoun, and Robyn Owens. 2007. An efficient multimodal 2D-3D hybrid approach to automatic face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 11 (2007), 1927--1943. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Prathap Nair and Andrea Cavallaro. 2009. 3-D face detection, landmark localization, and registration using a point distribution model. IEEE Transactions on Multimedia 11, 4 (2009), 611--623. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Alex Pentland, Baback Moghaddam, and Thad Starner. 1994. View-based and modular eigenspaces for face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Seattle, WA, 84--91.Google ScholarGoogle ScholarCross RefCross Ref
  29. Panagiotis Perakis, Georgios Passalis, Theoharis Theoharis, and Ioannis A. Kakadiaris. 2013. 3D facial landmark detection under large yaw and expression variations. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 7 (2013), 1552--1564. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Panagiotis Perakis, Theoharis Theoharis, Georgios Passalis, and Ioannis A. Kakadiaris. 2009. Automatic 3D facial region retrieval from multi-pose facial datasets. In Proceedings of the Eurographics Conference on 3D Object Retrieval. Eurographics Association, Munich, Germany, 37--44. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Shaoqing Ren, Xudong Cao, Yichen Wei, and Jian Sun. 2014. Face alignment at 3000 fps via regressing local binary features. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Columbus, OH, USA, 1685--1692. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Marcelo Romero-Huertas and Nick Pears. 2008. 3D facial landmark localisation by matching simple descriptors. In Proceedings of the IEEE International Conference on Biometrics: Theory, Applications and Systems. IEEE, Arlington, VA, 1--6.Google ScholarGoogle ScholarCross RefCross Ref
  33. Patrick Sauer, Tim Cootes, and Chris Taylor. 2011. Accurate regression procedures for active appearance models. In Proceedings of the British Machine Vision Conference. BMVA Press, 1--11.Google ScholarGoogle ScholarCross RefCross Ref
  34. Arman Savran, Neşe Alyüz, Hamdi Dibeklioğlu, Oya Çeliktutan, Berk Gökberk, Bülent Sankur, and Lale Akarun. 2008. Bosphorus database for 3D face analysis. In Proceedings of the European Workshop on Biometrics and Identity. Springer, Roskilde, Denmark, 47--56. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Maurício Pamplona Segundo, Luciano Silva, Olga Regina Pereira Bellon, and Chauã C. Queirolo. 2010. Automatic face segmentation and facial landmark detection in range images. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 40, 5 (2010), 1319--1330. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Mingli Song, Dacheng Tao, Shengpeng Sun, Chun Chen, and Stephen J. Maybank. 2014. Robust 3D face landmark localization based on local coordinate coding. IEEE Transactions on Image Processing 23, 12 (2014), 5108--5122.Google ScholarGoogle ScholarCross RefCross Ref
  37. Federico M. Sukno, John L. Waddington, and Paul F. Whelan. 2015. 3-D facial landmark localization with asymmetry patterns and shape regression from incomplete local features. IEEE Transactions on Cybernetics 45, 9 (2015), 1717--1730.Google ScholarGoogle ScholarCross RefCross Ref
  38. Jia Sun, Di Huang, Yunhong Wang, and Liming Chen. 2014. A coarse-to-fine approach to robust 3D facial landmarking via curvature analysis and active normal model. In Proceedings of the IEEE International Joint Conference on Biometrics. IEEE, Clearwater, FL, 1--7.Google ScholarGoogle ScholarCross RefCross Ref
  39. Przemyslaw Szeptycki, Mohsen Ardabilian, and Liming Chen. 2009. A coarse-to-fine curvature analysis-based rotation invariant 3D face landmarking. In Proceedings of the IEEE International Conference on Biometrics: Theory, Applications, and Systems. IEEE, Washington, DC, 1--6. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. John M. Vincent, Jon B. Waite, and David J. Myers. 1992. Automatic location of visual features by a system of multilayered perceptrons. IEEE Proceedings F-Radar and Signal Processing 139, 6 (1992), 405--412.Google ScholarGoogle ScholarCross RefCross Ref
  41. Renliang Weng, Jiwen Lu, Yap-Peng Tan, and Jie Zhou. 2016. Learning cascaded deep auto-encoder networks for face alignment. IEEE Transactions on Multimedia 18, 10 (2016), 2066--2078.Google ScholarGoogle ScholarCross RefCross Ref
  42. Xuehan Xiong and Fernando De la Torre. 2013. Supervised descent method and its applications to face alignment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Portland, OR, 532--539. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Yongqiang Yao, Di Huang, Xudong Yang, Yunhong Wang, and Liming Chen. 2018. Texture and geometry scattering representation-based facial expression recognition in 2D+ 3D videos. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 14, 1s (2018), 18. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Lijun Yin, Xiaochen Chen, Yi Sun, Tony Worm, and Michael Reale. 2008. A high-resolution 3D dynamic facial expression database. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition. IEEE, Amsterdam, Netherlands, 1--6.Google ScholarGoogle ScholarCross RefCross Ref
  45. Lijun Yin, Xiaozhou Wei, Yi Sun, Jun Wang, and Matthew J. Rosato. 2006. A 3D facial expression database for facial behavior research. In Proceedings of the IEEE International Conference on Face and Gesture Recognition. IEEE, Southampton, UK, 211--216. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Jie Zhang, Meina Kan, Shiguang Shan, and Xilin Chen. 2015. Leveraging datasets with varying annotations for face alignment via deep regression network. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, Santiago, Chile, 3801--3809. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Xi Zhao, Emmanuel Dellandrea, Liming Chen, and Ioannis A. Kakadiaris. 2011. Accurate landmarking of three-dimensional facial data in the presence of facial expressions and occlusions using a three-dimensional statistical facial feature model. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 41, 5 (2011), 1417--1428. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Qingkai Zhen, Di Huang, Yunhong Wang, and Liming Chen. 2016. Muscular movement model-based automatic 3D/4D facial expression recognition. IEEE Transactions on Multimedia 18, 7 (2016), 1438--1450. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Xiangyu Zhu, Zhen Lei, Xiaoming Liu, Hailin Shi, and Stan Z. Li. 2016. Face alignment across large poses: A 3d solution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Las Vegas, NV, 146--155.Google ScholarGoogle Scholar

Index Terms

  1. Expression Robust 3D Facial Landmarking via Progressive Coarse-to-Fine Tuning

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!