skip to main content
research-article

A New Foreground-Background based Method for Behavior-Oriented Social Media Image Classification

Authors Info & Claims
Published:12 November 2021Publication History
Skip Abstract Section

Abstract

Due to various applications, research on personal traits using information on social media has become an important area. In this paper, a new method for the classification of behavior-oriented social images uploaded on various social media platforms is presented. The proposed method introduces a multimodality concept using skin of different parts of human body and background information, such as indoor and outdoor environments. For each image, the proposed method detects skin candidate components based on R, G, B color spaces and entropy features. The iterative mutual nearest neighbor approach is proposed to detect accurate skin candidate components, which result in foreground components. Next, the proposed method detects the remaining part (other than skin components) as background components based on structure tensor of R, G, B color spaces, and Maximally Stable Extremal Regions (MSER) concept in the wavelet domain. We then explore Hanman Transform for extracting context features from foreground and background components through clustering and fusion operation. These features are then fed to an SVM classifier for the classification of behavior-oriented images. Comprehensive experiments on 10-class datasets of Normal Behavior-Oriented Social media Image (NBSI) and Abnormal Behavior-Oriented Social media Image (ABSI) show that the proposed method is effective and outperforms the existing methods in terms of average classification rate. Also, the results on the benchmark dataset of five classes of personality traits and two classes of emotions of different facial expressions (FERPlus dataset) demonstrated the robustness of the proposed method over the existing methods.

REFERENCES

  1. [1] Cheung M., She J., and Jie Z.. 2015. Connection discovery using big data of user shared images in social media. IEEE Trans, MM 17, (2015), 14171428.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. [2] Hu Y., Manikonda L., and Kambhampati S.. 2014. What we Instagram: A first analysis of Instagram photo content and user types. In Proc. AAAI, 595598.Google ScholarGoogle Scholar
  3. [3] Liu L., Pietro D. P., Samani Z. R., Moghadadam M. E., and Ungar L.. 2016. Analyzing personality through social media profile picture choice. In Proc. ICWSM, 2016.Google ScholarGoogle Scholar
  4. [4] Tiwari A., Weth C. V. D., and Kankanhalli M.. 2018. Multimodal multiplatform social media event summarization. ACM. TOMM 14, 38 (2018).Google ScholarGoogle Scholar
  5. [5] Wang T. and Li B.. 2017. Sentiment analysis for social media images. In Proc. ICDAMW, 15841591.Google ScholarGoogle Scholar
  6. [6] Wang D., Otto C., and Jain A. K.. 2017. Face search at scale. IEEE Trans. PAMI 39, 11221136, 2017.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. [7] Farzindar A. and Inkpen D.. 2015. Natural language processing for social media. Synthesis Lectures on Human Language Techniques, Morgan and Claypool Publishers, 2015.Google ScholarGoogle ScholarCross RefCross Ref
  8. [8] Shen C., Jin Z., Chu W., Jiang R., Chen Y., Qi G. J., and Hua X. S.. 2019. Multi-level similarity perception network for person re-identification. ACM. TOMM 15, 32 (2019).Google ScholarGoogle Scholar
  9. [9] Han H., Otto C., Liu X., and Jain A. K.. 2015. Demographic estimation from face images: Human vs. machine performance. IEEE Trans. PAMI 37, (2015), 11481161.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. [10] Mabrouk A. B. and Zagrouba E.. 2018. Abnormal behavior recognition for intelligent video surveillance systems: A review. Expert Systems with Applications 91, (2018), 480491.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. [11] Dwivedi R. and Dey S.. 2018. Score level fusion for cancelable multi-biometric verification. Pattern Recognition Letters, 2018.Google ScholarGoogle Scholar
  12. [12] Kumano S., Otsuka K., Ishii R., and Yamato J.. 2017. Collective first person for automatic gaze analysis in multiparty conversations. IEEE. Trans. MM 1, 107122.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. [13] Huang X., Dhall A., Goecke R., Pietikainen M., and Zhao G.. 2017. Multi-modal framework for analyzing the affect of a group of people. IEEE. Trans. MM 8, 116, 2017.Google ScholarGoogle Scholar
  14. [14] Jiang S., Chen G., Song X., and Liu L.. 2019. Deep patch representation with shared codebook for scene classification. ACM. TOMM 15, 5 (2019).Google ScholarGoogle Scholar
  15. [15] Chen Z., Al S., and Jia C.. 2019. Structure-aware deep learning for product image classification. ACM. TOMM 15, 4, (2019).Google ScholarGoogle Scholar
  16. [16] Wang A., Cai J., Lu J., and Cham T. J.. 2018. Structure-aware multimodal features fusion for RGB-D scene classification and beyond. ACM. TOMM 14, 39 (2018).Google ScholarGoogle Scholar
  17. [17] Howard A. G., Zhu M., Chen B., Kalenichenko D., Wang W., Weyand T., Andreetto M., and Adam H.. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017).Google ScholarGoogle Scholar
  18. [18] Benezeth Y., Jodoin P. M., Saligrama V., and Rosenberger C.. 2009. Abnormal event detection based on spatio-temporal co-occurences. In Proc. CVPR, 24582465, 2009.Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Beeck K. V., Engeland K. V., Vennekens J., and Goedeme T.. 2017. Abnormal behavior detection in LWIR surveillance of railway platforms. In Proc. AVSS 2017.Google ScholarGoogle Scholar
  20. [20] Rougier C., Meunier J., Arnaud A. S., and Rousseau J.. 2011. Robust video surveillance for fall detection based on human shape deformation. IEEE Trans. CSVT 21, (2011), 611622.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. [21] Cosar S., Donatiello G., Bogorny V., Garate C., Otavio L., and Bremond F.. 2017. Toward abnormal trajectory and event detection in video surveillance. IEEE Trans. CSVT 27, (2017), 683695.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. [22] Liu C. W., Chen H. T., Lo K. H., Wang C. J., and Chuang J. H.. 2017. Accelerating vanishing point based line sampling scheme for real time people localization. IEEE Trans. CSVT 27, (2017), 409420.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. [23] Zhang Y., Qin L., Ji R., Yao H., and Huang Q.. 2015. Social attribute aware force model: Exploiting richness of interaction for abnormal crowd detection. IEEE Trans. CSVT 25, (2015), 12311245.Google ScholarGoogle Scholar
  24. [24] Wang T. and Snoussi H.. 2014. Detection of abnormal visual events via global optical flow orientation histogram. IEEE Trans. IFS 9, (2014), 988998.Google ScholarGoogle Scholar
  25. [25] Tiwari C., Hanmandlu M., and Vasikarla S.. 2015. Suspicious face detection based on eye and other facial features movement monitoring. In Proc. AIPR, 18.Google ScholarGoogle ScholarCross RefCross Ref
  26. [26] Hsu S. C., Chuang C. H., Huang C. L., Teng P. R., and Lin M. J.. 2018. A video based abnormal human behavior detection for psychiatric patient monitoring. In Proc. IWAIT, 14, 2018.Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Barsoum E., Zhang C., Ferrer C. C., and Zhang Z.. 2016. Training deep networks for facial expression recognition with crowd-sourced label distribution. In Proc. ACMMM, 279283, 2016.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. [28] Sharama M., Jalal A. S., and Khan A.. 2019. Emotion recognition using facial expression by using key points descriptor and texture features. Multimedia Tools and Applications 78, (2019), 1619516219.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. [29] Mungra D., Agrawal A., Sharma P., Tanwar S., and Obaidat M. S.. 2020. PRATIT: A CNN-based emotion recognition system using histogram equalization and data augmentation. Multimedia Tools and Applications, 22852307.Google ScholarGoogle ScholarCross RefCross Ref
  30. [30] Krishnani D., Shivakumara P., Lu T., Pal U., and Ramachandra R.. 2019. Structure function based transform features for behavior-oriented social media image classification. In Proc. ACPR, 594608.Google ScholarGoogle Scholar
  31. [31] Grover J. and Hanmandlu M.. 2018. The fusion of multispectral palmprints using the information set based features and classifier. Engineering Applications of Artificial Intelligence 67, (2018), 111125.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. [32] Sarkar A., Abbott A. L., and Doerzaph Z.. 2017. Universal skin detection without color information. In Proc. WCACV, 2028.Google ScholarGoogle ScholarCross RefCross Ref
  33. [33] Raghunandan K. S., Shivakumara P., Roy S., Kumar G. H., Pal U., and Lu T.. 2018. Multi-script-oriented text detection and recognition in video/scene/born digital images. IEEE Trans. CSVT, 116.Google ScholarGoogle Scholar
  34. [34] Platt J. C.. 1999. Fast training of support vector machines using sequential minimal optimization. Advances in Kernel Methods, 185208.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. [35] Zhang K., Zhang Z., Li Z., and Qiao Y.. 2016. Joint face detection and alignment using multi-task cascaded convolutional networks. IEEE Signal Processing Letters 23, (2016), 14991503.Google ScholarGoogle Scholar
  36. [36] Lee H. and Kim C.. 2014. Blurred image region detection and segmentation. In Proc. ICIP 4427–4431, 2014.Google ScholarGoogle ScholarCross RefCross Ref
  37. [37] Xu X., Wang Y., and Chen S.. 2016. Medical image fusion using discrete fractional wavelet transform. Biomedical Signal Processing and Control 27, 103111, 2016.Google ScholarGoogle ScholarCross RefCross Ref
  38. [38] Krizhevsky A., Sutskever I., and Hinton G. E.. 2012. ImageNet classification with deep convolutional neural networks. In Proc. NIPS, 19, 2012.Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. [39] Smonyan K. and Zisserman A.. 2015. Very deep convolutional networks for large scale image recognition. In Proc. ICLR, 114.Google ScholarGoogle Scholar
  40. [40] Szegedy C., Liu W., Jia Y., Sermanet P., Reed S., Anguelov D., Erthan D., Vanhoucke V., and Rabinovich A.. 2015. Going deeper with convolutions. In Proc. CVPR, 19.Google ScholarGoogle Scholar
  41. [41] He K., Zhang X., Ren S., and Sun J.. 2016. Deep residual learning for image recognition. In Proc. CVPR, 770778, 2016.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. A New Foreground-Background based Method for Behavior-Oriented Social Media Image Classification

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Multimedia Computing, Communications, and Applications
          ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 17, Issue 4
          November 2021
          529 pages
          ISSN:1551-6857
          EISSN:1551-6865
          DOI:10.1145/3492437
          Issue’s Table of Contents

          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 12 November 2021
          • Accepted: 1 March 2021
          • Revised: 1 August 2020
          • Received: 1 July 2019
          Published in tomm Volume 17, Issue 4

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Refereed

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Full Text

        View this article in Full Text.

        View Full Text

        HTML Format

        View this article in HTML Format .

        View HTML Format
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!