skip to main content
research-article

Predicting Occupation from Images by Combining Face and Body Context Information

Published:02 December 2016Publication History
Skip Abstract Section

Abstract

Facial images embed age, gender, and other rich information that is implicitly related to occupation. In this work, we advocate that occupation prediction from a single facial image is a doable computer vision problem. We extract multilevel hand-crafted features associated with locality-constrained linear coding and convolutional neural network features as image occupation descriptors. To avoid the curse of dimensionality and overfitting, a boost strategy called multichannel SVM is used to integrate features from face and body. Intra- and interclass visual variations are jointly considered in the boosting framework to further improve performance. In the evaluation, we verify the effectiveness of predicting occupation from face and demonstrate promising performance obtained by combining face and body information. More importantly, our work further integrates deep features into the multichannel SVM framework and shows significantly better performance over the state of the art.

References

  1. John Antonakis and Olaf Dalgas. 2009. Predicting elections: Child’s play! Science 323, 5918 (2009), 1183.Google ScholarGoogle Scholar
  2. Kingsley R. Browne. 2006. Evolved sex differences and occupational segregation. Journal of Organizational Behavior 27 (2006), 143--162.Google ScholarGoogle ScholarCross RefCross Ref
  3. Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Return of the devil in the details: Delving deep into convolutional nets. In Proceedings of the British Machine Vision Conference.Google ScholarGoogle ScholarCross RefCross Ref
  4. Bor-Chun Chen, Yan-Ying Chen, Yin-Hsi Kuo, and Winston H. Hsu. 2013a. Scalable face image retrieval using attribute-enhanced sparse codewords. IEEE Transactions on Multimedia 15, 5 (2013), 1163--1173. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Huizhong Chen, Andrew C. Gallagher, and Bernd Girod. 2013b. What’s in a name? First names as facial attributes. In Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition. 3366--3373. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Yan-Ying Chen, Winston H. Hsu, and Hong-Yuan Mark Liao. 2013c. Automatic training image acquisition and effective feature selection from community-contributed photos for facial attribute detection. IEEE Transactions on Multimedia 15, 6 (2013), 1388--1399. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Wei-Ta Chu and Chih-Hao Chiu. 2014. Predicting occupation from single facial images. In Proceedings of IEEE International Symposium on Multimedia. 9--12. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Navneet Dalal and Bill Triggs. 2005. Histograms of oriented gradients for human detection. In Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition. 886--893. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Hamdi Dibeklioglu, Albert A. Salah, and Theo Gevers. 2015. Recognition of genuine smiles. IEEE Transactions on Multimedia 17, 3 (2015), 279--294.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Haoqiang Fan, Mu Yang, Zhimin Cao, Yuning Jiang, and Qi Yin. 2014. Learning compact face representation: Packing a face into an Int32. In Proceedings of ACM Multimedia. 933--936. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Paul E. Gabriel and Susanne Schmitz. 2007. Gender differences in occupational distributions among workers. Monthly Labor Review 130 (2007), 19--24.Google ScholarGoogle Scholar
  12. Robert L. Kaufman and Seymour Spilerman. 1982. The age structures of occupations and jobs. American Journal of Sociology 87, 4 (1982), 827--851.Google ScholarGoogle ScholarCross RefCross Ref
  13. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of Advances in Neural Information Processing System. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Iljung S. Kwak, Ana C. Murillo, Peter N. Belhumeur, David Kriegman, and Serge Belongie. 2013. From bikers to surfers: Visual recognition of urban tribes. In Proceedings of the British Machine Vision Conference.Google ScholarGoogle ScholarCross RefCross Ref
  15. Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce. 2006. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition. 2169--2178. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient based learning applied to document recognition. Proceedings of the IEEE 86, 11 (1998), 2278--2324.Google ScholarGoogle ScholarCross RefCross Ref
  17. Ce Liu, Jenny Yuen, Antonio Torralba, Josef Sivic, and William T. Freeman. 2008. SIFT flow: Dense correspondence across different scenes. In Proceedings of the European Conference on Computer Vision. 28--42. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Si Liu, Zheng Song, Guangcan Liu, Changsheng Xu, Hanqing Lu, and Shuicheng Yan. 2012. Street-to-shop: Cross-scenario clothing retrieval via parts alignment and auxiliary set. In Proceedings of the European Conference on Computer Vision. 3330--3337.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. David R. Martin, Charless C. Fowlkes, and Jitendra Malik. 2004. Learning to detect natural image boundaries using local brightness, color, and texture cues. IEEE Transactions on Pattern Analysis and Machine Intelligence 26, 5 (2004), 530--549. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Laurent S. Nguyen, Denise Frauendorfer, Marianne S. Mast, and Daniel Gatica-Perez. 2014. Hire me: Computational inference of hirability in employment interviews based on nonverbal behavior. IEEE Transactions on Multimedia 16, 4 (2014), 1018--1031. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Neil O’Hare and Alan F. Smeaton. 2009. Context-aware person identification in personal photo collections. IEEE Transactions on Multimedia 11, 2 (2009), 220--228. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Timo Ojala, Matti Pietikainen, and Topi Maenpaa. 2002. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 7 (2002), 971--987. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Lei Pang and Chong-Wah Ngo. 2015. Unsupervised celebrity face naming in web videos. IEEE Transactions on Multimedia 17, 6 (2015), 854--866.Google ScholarGoogle ScholarCross RefCross Ref
  24. Vignesh Ramanathan, Bangpeng Yao, and Li Fei-Fei. 2013. Social role discovery in human events. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition. 2475--2482. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Ali S. Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. 2014. CNN features off-the-shelf: An astounding baseline for recognition. In Proceedings of the CVPR Workshop on DeepVision. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Nicholas O. Rule and Nalini Ambady. 2008. The face of success: Inferences from chief executive officers’ appearance predict company profits. Psychological Science 19, 2 (2008), 109--111.Google ScholarGoogle ScholarCross RefCross Ref
  27. Ming Shao, Liangyue Li, and Yun Fu. 2013. What do you do? Occupation recognition in a photo via social context. In Proceedings of the International Conference on Computer Vision. 3631--3638. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. John M. Smith. 1973. Age and occupation: The determinants of male occupational age structures -- hypothesis H and hypothesis A. Journal of Gerontology 28, 4 (1973), 484--490.Google ScholarGoogle ScholarCross RefCross Ref
  29. Zheng Song, Meng Wang, Xian-Sheng Hua, and Shuicheng Yan. 2011. Predicting occupation via human clothing and contexts. In Proceedings of the International Conference on Computer Vision. 1084--1091. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Alexander Todorov, Anesu N. Mandisodza, Amir Goren, and Crystal C. Hall. 2005. Inferences of competence from faces predict election outcomes. Science 308, 5728 (2005), 1623--1626.Google ScholarGoogle Scholar
  31. Andrea Vedaldi and Karel Lenc. 2015. MatConvNet -- convolutional neural networks for MATLAB. In Proceedings of the ACM International Conference on Multimedia. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Paul Viola and Michael J. Jones. 2004. Robust real-time face detection. International Journal of Computer Vision 57 (2004), 137--154. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Jinjun Wang, Jianchao Yang, Kai Yu, Fengjun Lv, Thomas Huang, and Yihong Gong. 2010. Locality constrained linear coding for image classification. In Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition. 3360--3367.Google ScholarGoogle ScholarCross RefCross Ref
  34. Siyu Xia, Ming Shao, Jiebo Luo, and Yun Fu. 2012. Understanding kin relationships in a photo. IEEE Transactions on Multimedia 14, 4 (2012), 1046--1056. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Chao Xiong, Guangyu Gao, Zhengjun Zha, Shuicheng Yan, Huadong Ma, and Tae-Kyun Kim. 2014. Adaptive learning for celebrity identification with video context. IEEE Transactions on Multimedia 16, 5 (2014), 1473--1485.Google ScholarGoogle ScholarCross RefCross Ref
  36. Xiao Zhang, Lei Zhang, Xin-Jing Wang, and Heung-Yeung Shum. 2012. Finding celebrities in billions of web images. IEEE Transactions on Multimedia 14, 4 (2012), 995--1007. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Predicting Occupation from Images by Combining Face and Body Context Information

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!