Abstract
Facial landmarking is a fundamental task in automatic machine-based face analysis. The majority of existing techniques for such a problem are based on 2D images; however, they suffer from illumination and pose variations that may largely degrade landmarking performance. The emergence of 3D data theoretically provides an alternative to overcome these weaknesses in the 2D domain. This article proposes a novel approach to 3D facial landmarking, which combines both the advantages of feature-based methods as well as model-based ones in a progressive three-stage coarse-to-fine manner (initial, intermediate, and fine stages). For the initial stage, a few fiducial landmarks (i.e., the nose tip and two inner eye corners) are robustly detected through curvature analysis, and these points are further exploited to initialize the subsequent stage. For the intermediate stage, a statistical model is learned in the feature space of three normal components of the facial point-cloud rather than the smooth original coordinates, namely Active Normal Model (ANM). For the fine stage, cascaded regression is employed to locally refine the landmarks according to their geometry attributes. The proposed approach can accurately localize dozens of fiducial points on each 3D face scan, greatly surpassing the feature-based ones, and it also improves the state of the art of the model-based ones in two aspects: sensitivity to initialization and deficiency in discrimination. The proposed method is evaluated on the BU-3DFE, Bosphorus, and BU-4DFE databases, and competitive results are achieved in comparison with counterparts in the literature, clearly demonstrating its effectiveness.
- Stefano Berretti, Boulbaba Ben Amor, Mohamed Daoudi, and Alberto Del Bimbo. 2011. 3D facial expression recognition using SIFT descriptors of automatically detected keypoints. The Visual Computer 27, 11 (2011), 1021--1036. Google Scholar
Digital Library
- Paul J. Besl and Ramesh C. Jain. 1986. Invariant surface characteristics for 3D object recognition in range images. Computer Vision, Graphics, and Image Processing 33, 1 (1986), 33--80. Google Scholar
Digital Library
- Volker Blanz and Thomas Vetter. 2003. Face recognition based on fitting a 3D morphable model. IEEE Transactions on Pattern Analysis and Machine Intelligence 25, 9 (2003), 1063--1074. Google Scholar
Digital Library
- Leo Breiman. 2001. Random forests. Machine Learning 45, 1 (2001), 5--32. Google Scholar
Digital Library
- Adrian Bulat and Georgios Tzimiropoulos. 2017. How far are we from solving the 2d 8 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks). In Proceedings on the IEEE International Conference on Computer Vision. IEEE, Venice, Italy, 1021--1030.Google Scholar
Cross Ref
- Xavier P. Burgos-Artizzu, Pietro Perona, and Piotr Dollár. 2013. Robust face landmark estimation under occlusion. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, Sydney, NSW, Australia, 1513--1520. Google Scholar
Digital Library
- Dirk Colbry, George Stockman, and Anil Jain. 2005. Detection of anchor points for 3d face verification. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. IEEE, San Diego, CA, USA, 118--118. Google Scholar
Digital Library
- Alessandro Colombo, Claudio Cusano, and Raimondo Schettini. 2006. 3D face detection using curvature analysis. Pattern Recognition 39, 3 (2006), 444--455. Google Scholar
Digital Library
- Timothy F. Cootes, Gareth J. Edwards, and Christopher J. Taylor. 2001. Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 6 (2001), 681--685. Google Scholar
Digital Library
- Timothy F. Cootes, Christopher J. Taylor, David H. Cooper, and Jim Graham. 1995. Active shape models-their training and application. Computer Vision and Image Understanding 61, 1 (1995), 38--59. Google Scholar
Digital Library
- Clement Creusot, Nick Pears, and Jim Austin. 2013. A machine-learning approach to keypoint detection and landmarking on 3D meshes. International Journal of Computer Vision 102, 1--3 (2013), 146--179. Google Scholar
Digital Library
- Hamdi Dibeklioglu, Albert A. Salah, and Lale Akarun. 2008. 3D facial landmarking under expression, pose, and occlusion variations. In Proceedings of the IEEE International Conference on Biometrics: Theory, Applications and Systems. IEEE, Arlington, VA, 1--6.Google Scholar
Cross Ref
- Gabriele Fanelli, Matthias Dantone, Juergen Gall, Andrea Fossati, and Luc Van Gool. 2013. Random forests for real time 3d face analysis. International Journal of Computer Vision 101, 3 (2013), 437--458. Google Scholar
Digital Library
- Gabriele Fanelli, Matthias Dantone, and Luc Van Gool. 2013. Real time 3D face alignment with random forests-based active appearance models. In Proceedings of the IEEE International Conference on Face and Gesture Recognition. IEEE, Shanghai, China, 1--8.Google Scholar
Cross Ref
- Tianhong Fang, Xi Zhao, Omar Ocegueda, Shishir K. Shah, and Ioannis A. Kakadiaris. 2012. 3D/4D facial expression analysis: An advanced annotated face model approach. Image and Vision Computing 30, 10 (2012), 738--749. Google Scholar
Digital Library
- Syed Zulqarnain Gilani, Ajmal Mian, and Peter Eastwood. 2017. Deep, dense and accurate 3D face correspondence for generating population specific deformable models. Pattern Recognition 69 (2017), 238--250. Google Scholar
Digital Library
- Shalini Gupta, Kenneth R. Castleman, Mia K. Markey, and Alan C. Bovik. 2010. Texas 3D face recognition database. In Proceedings of the IEEE Southwest Symposium on Image Analysis and Interpretation. IEEE, Austin, TX, 97--100.Google Scholar
- Shalini Gupta, Mia K. Markey, and Alan C. Bovik. 2010. Anthropometric 3D face recognition. International Journal of Computer Vision 90, 3 (2010), 331--349. Google Scholar
Digital Library
- Di Huang, Mohsen Ardabilian, Yunhong Wang, and Liming Chen. 2012. 3-D face recognition using eLBP-based facial description and local feature hybrid matching. IEEE Transactions on Information Forensics and Security 7, 5 (2012), 1551--1565. Google Scholar
Digital Library
- Di Huang, Karima Ouji, Mohsen Ardabilian, Yunhong Wang, and Liming Chen. 2011. 3D face recognition based on local shape patterns and sparse representation classifier. In Proceedings of the International Conference on Multimedia Modeling. Springer, Taipei, Taiwan, 206--216. Google Scholar
Digital Library
- Vuong Le, Jonathan Brandt, Zhe Lin, Lubomir Bourdev, and Thomas S. Huang. 2012. Interactive facial feature localization. In Proceedings of the European Conference on Computer Vision. Springer, Florence, Italy, 679--692. Google Scholar
Digital Library
- Huibin Li, Huaxiong Ding, Di Huang, Yunhong Wang, Xi Zhao, Jean-Marie Morvan, and Liming Chen. 2015. An efficient multimodal 2D+ 3D feature-based approach to automatic facial expression recognition. Computer Vision and Image Understanding 140 (2015), 83--92. Google Scholar
Digital Library
- Huibin Li, Di Huang, Jean-Marie Morvan, and Liming Chen. 2011. Learning weighted sparse representation of encoded facial normal information for expression-robust 3D face recognition. In Proceedings of the IEEE International Conference on Biometrics. IEEE, Washington, DC, 1--7. Google Scholar
Digital Library
- Huibin Li, Di Huang, Jean-Marie Morvan, Yunhong Wang, and Liming Chen. 2015. Towards 3D face recognition in the real: A registration-free approach using fine-grained matching of 3D keypoint descriptors. International Journal of Computer Vision 113, 2 (2015), 128--142. Google Scholar
Digital Library
- Xiaoguang Lu and Anil K. Jain. 2006. Automatic feature extraction for multiview 3D face recognition. In Proceedings of the International Conference on Automatic Face and Gesture Recognition. IEEE, Southampton, 585--590. Google Scholar
Digital Library
- Ajmal Mian, Mohammed Bennamoun, and Robyn Owens. 2007. An efficient multimodal 2D-3D hybrid approach to automatic face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 11 (2007), 1927--1943. Google Scholar
Digital Library
- Prathap Nair and Andrea Cavallaro. 2009. 3-D face detection, landmark localization, and registration using a point distribution model. IEEE Transactions on Multimedia 11, 4 (2009), 611--623. Google Scholar
Digital Library
- Alex Pentland, Baback Moghaddam, and Thad Starner. 1994. View-based and modular eigenspaces for face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Seattle, WA, 84--91.Google Scholar
Cross Ref
- Panagiotis Perakis, Georgios Passalis, Theoharis Theoharis, and Ioannis A. Kakadiaris. 2013. 3D facial landmark detection under large yaw and expression variations. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 7 (2013), 1552--1564. Google Scholar
Digital Library
- Panagiotis Perakis, Theoharis Theoharis, Georgios Passalis, and Ioannis A. Kakadiaris. 2009. Automatic 3D facial region retrieval from multi-pose facial datasets. In Proceedings of the Eurographics Conference on 3D Object Retrieval. Eurographics Association, Munich, Germany, 37--44. Google Scholar
Digital Library
- Shaoqing Ren, Xudong Cao, Yichen Wei, and Jian Sun. 2014. Face alignment at 3000 fps via regressing local binary features. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Columbus, OH, USA, 1685--1692. Google Scholar
Digital Library
- Marcelo Romero-Huertas and Nick Pears. 2008. 3D facial landmark localisation by matching simple descriptors. In Proceedings of the IEEE International Conference on Biometrics: Theory, Applications and Systems. IEEE, Arlington, VA, 1--6.Google Scholar
Cross Ref
- Patrick Sauer, Tim Cootes, and Chris Taylor. 2011. Accurate regression procedures for active appearance models. In Proceedings of the British Machine Vision Conference. BMVA Press, 1--11.Google Scholar
Cross Ref
- Arman Savran, Neşe Alyüz, Hamdi Dibeklioğlu, Oya Çeliktutan, Berk Gökberk, Bülent Sankur, and Lale Akarun. 2008. Bosphorus database for 3D face analysis. In Proceedings of the European Workshop on Biometrics and Identity. Springer, Roskilde, Denmark, 47--56. Google Scholar
Digital Library
- Maurício Pamplona Segundo, Luciano Silva, Olga Regina Pereira Bellon, and Chauã C. Queirolo. 2010. Automatic face segmentation and facial landmark detection in range images. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 40, 5 (2010), 1319--1330. Google Scholar
Digital Library
- Mingli Song, Dacheng Tao, Shengpeng Sun, Chun Chen, and Stephen J. Maybank. 2014. Robust 3D face landmark localization based on local coordinate coding. IEEE Transactions on Image Processing 23, 12 (2014), 5108--5122.Google Scholar
Cross Ref
- Federico M. Sukno, John L. Waddington, and Paul F. Whelan. 2015. 3-D facial landmark localization with asymmetry patterns and shape regression from incomplete local features. IEEE Transactions on Cybernetics 45, 9 (2015), 1717--1730.Google Scholar
Cross Ref
- Jia Sun, Di Huang, Yunhong Wang, and Liming Chen. 2014. A coarse-to-fine approach to robust 3D facial landmarking via curvature analysis and active normal model. In Proceedings of the IEEE International Joint Conference on Biometrics. IEEE, Clearwater, FL, 1--7.Google Scholar
Cross Ref
- Przemyslaw Szeptycki, Mohsen Ardabilian, and Liming Chen. 2009. A coarse-to-fine curvature analysis-based rotation invariant 3D face landmarking. In Proceedings of the IEEE International Conference on Biometrics: Theory, Applications, and Systems. IEEE, Washington, DC, 1--6. Google Scholar
Digital Library
- John M. Vincent, Jon B. Waite, and David J. Myers. 1992. Automatic location of visual features by a system of multilayered perceptrons. IEEE Proceedings F-Radar and Signal Processing 139, 6 (1992), 405--412.Google Scholar
Cross Ref
- Renliang Weng, Jiwen Lu, Yap-Peng Tan, and Jie Zhou. 2016. Learning cascaded deep auto-encoder networks for face alignment. IEEE Transactions on Multimedia 18, 10 (2016), 2066--2078.Google Scholar
Cross Ref
- Xuehan Xiong and Fernando De la Torre. 2013. Supervised descent method and its applications to face alignment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Portland, OR, 532--539. Google Scholar
Digital Library
- Yongqiang Yao, Di Huang, Xudong Yang, Yunhong Wang, and Liming Chen. 2018. Texture and geometry scattering representation-based facial expression recognition in 2D+ 3D videos. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 14, 1s (2018), 18. Google Scholar
Digital Library
- Lijun Yin, Xiaochen Chen, Yi Sun, Tony Worm, and Michael Reale. 2008. A high-resolution 3D dynamic facial expression database. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition. IEEE, Amsterdam, Netherlands, 1--6.Google Scholar
Cross Ref
- Lijun Yin, Xiaozhou Wei, Yi Sun, Jun Wang, and Matthew J. Rosato. 2006. A 3D facial expression database for facial behavior research. In Proceedings of the IEEE International Conference on Face and Gesture Recognition. IEEE, Southampton, UK, 211--216. Google Scholar
Digital Library
- Jie Zhang, Meina Kan, Shiguang Shan, and Xilin Chen. 2015. Leveraging datasets with varying annotations for face alignment via deep regression network. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, Santiago, Chile, 3801--3809. Google Scholar
Digital Library
- Xi Zhao, Emmanuel Dellandrea, Liming Chen, and Ioannis A. Kakadiaris. 2011. Accurate landmarking of three-dimensional facial data in the presence of facial expressions and occlusions using a three-dimensional statistical facial feature model. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 41, 5 (2011), 1417--1428. Google Scholar
Digital Library
- Qingkai Zhen, Di Huang, Yunhong Wang, and Liming Chen. 2016. Muscular movement model-based automatic 3D/4D facial expression recognition. IEEE Transactions on Multimedia 18, 7 (2016), 1438--1450. Google Scholar
Digital Library
- Xiangyu Zhu, Zhen Lei, Xiaoming Liu, Hailin Shi, and Stan Z. Li. 2016. Face alignment across large poses: A 3d solution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Las Vegas, NV, 146--155.Google Scholar
Index Terms
Expression Robust 3D Facial Landmarking via Progressive Coarse-to-Fine Tuning
Recommendations
Expression robust 3D face landmarking using thresholded surface normals
A novel 3D facial landmarking algorithm based on thresholded surface normals maps is proposed.The potential of using surface normals maps for feature detection is explored.Using thresholded surface normals can help facial roll and yaw rotation ...
Expression-invariant face recognition by facial expression transformations
In this paper, we present a method of expression-invariant face recognition that transforms input face image with an arbitrary expression into its corresponding neutral facial expression image. When a new face image with an arbitrary expression is ...
Pose-Robust Facial Expression Recognition Using View-Based 2D + 3D AAM
This paper proposes a pose-robust face tracking and facial expression recognition method using a view-based 2D 3D active appearance model (AAM) that extends the 2D 3D AAM to the view-based approach, where one independent face model is used for a ...






Comments