Abstract
Before the arrival of the web as a corpus, people detected positive and negative news based on the understanding of the textual content from physical newspaper rather than an automatic identification approach from readily available e-newspapers. Thus, the earlier sentiment analysis approach is based on unimodal data, and less effort is paid to the multimodal data. However, the presence of multimodal information helps us to get a clearer understanding of the sentiment. To the best of our knowledge, less work has been introduced on the image–text multimodal sentiment analysis framework of Assamese, a low-resource Indian language mostly spoken in the northeast part of India. We built an Assamese news articles dataset consisting of news text and associated images and one image caption to conduct an experimental study. Focusing on important words and discriminative regions of the images mostly related to sentiment, two individual unimodal such as textual and visual models are proposed. The visual model is developed using an encoder-decoder–based image caption generation system. An image–text multimodal approach is proposed to explore the internal correlation between textual and visual features for joint sentiment classification. Finally, we propose the multimodal sentiment analysis framework, i.e., Textual Visual Multimodal Fusion, by employing a late fusion scheme to merge the three different modalities for the final sentiment prediction. Experimental results conducted on the Assamese dataset built in-house demonstrate that the contextual integration of multimodal features delivers better performance than unimodal features.
- [1] . 2013. Arabic/English sentiment analysis: An empirical study. In Proceedings of the 4th International Conference on Information and Communication Systems (ICICS’13). 23–25.Google Scholar
- [2] . 2013. Large-scale visual sentiment ontology and detectors using adjective noun pairs. In Proceedings of the 21st ACM International Conference on Multimedia. 223–232.Google Scholar
Digital Library
- [3] . 2017. From pixels to sentiment: Fine-tuning CNNs for visual sentiment prediction. Image Vis. Comput. 65 (2017), 15–22.Google Scholar
Digital Library
- [4] . 2016. Visual sentiment topic model based microblog image sentiment analysis. Multimedia Tools Appl. 75, 15 (2016), 8955–8968.Google Scholar
Digital Library
- [5] . 2017. Visual and textual sentiment analysis using deep fusion convolutional neural networks. In Proceedings of the IEEE International Conference on Image Processing (ICIP’17). IEEE, 1557–1561.Google Scholar
Digital Library
- [6] . 2010. Opinion-polarity identification in bengali. In Proceedings of the International Conference on Computer Processing of Oriental Languages. 169–182.Google Scholar
- [7] . 2021. Image caption generation framework for assamese news using attention mechanism. In Proceedings of the 18th International Conference on Natural Language Processing (ICON’21). 231–239.Google Scholar
- [8] . 2021. A step towards sentiment analysis of assamese news articles using lexical features. In Proceedings of the International Conference on Computing and Communication Systems (I3CS’20), Vol. 170. Springer, 15.Google Scholar
Cross Ref
- [9] . 2022. Assamese news image caption generation using attention mechanism. Multimedia Tools Appl. 81, 7 (2022), 10051–10069.Google Scholar
Digital Library
- [10] . 2022. A multi-stage multimodal framework for sentiment analysis of Assamese in low resource setting. Expert Syst. Appl. (2022), 117575.Google Scholar
Digital Library
- [11] . 2017. Social media sentiment analysis: Lexicon versus machine learning. J. Cons. Market. (2017).Google Scholar
Cross Ref
- [12] . 2004. Mining opinion features in customer reviews. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’04), Vol. 4. 755–760.Google Scholar
- [13] . 2019. Image–text sentiment analysis via deep multimodal attentive fusion. Knowl.-Bas. Syst. 167 (2019), 26–37.Google Scholar
Digital Library
- [14] . 2014. A study and analysis of opinion mining research in Indo-Aryan, Dravidian and Tibeto-Burman language families. Int. J. Data Min. Emerg. Technol. 4, 2 (2014), 53–60.Google Scholar
Cross Ref
- [15] . 2004. Determining the sentiment of opinions. In Proceedings of the 20th International Conference on Computational Linguistics. Association for Computational Linguistics, 1367.Google Scholar
Digital Library
- [16] . 2016. Sentiment analysis for low resource languages: A study on informal Indonesian tweets. In Proceedings of the 12th Workshop on Asian Language Resources (ALR12’16). 123–131.Google Scholar
- [17] . 2021. Low resource language specific pre-processing and features for sentiment analysis task. Lang. Resourc. Eval. (2021), 1–23.Google Scholar
- [18] . 2019. Sentiment analysis for a resource poor language–Roman Urdu. ACM Trans. Asian Low-Resour. Lang. Inf. Proc. 19, 1 (2019), 1–15.Google Scholar
- [19] . 2013. Sentiment analysis in twitter using machine learning techniques. In Proceedings of the 4th International Conference on Computing, Communications and Networking Technologies (ICCCNT’13). IEEE, 1–5.Google Scholar
Cross Ref
- [20] . 2020. Exploiting objective text description of images for visual sentiment analysis. Multimedia Tools Appl. (2020), 1–24.Google Scholar
- [21] . 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, 271.Google Scholar
Digital Library
- [22] . 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, 115–124.Google Scholar
Digital Library
- [23] . 2002. Thumbs up?: Sentiment classification using machine learning techniques. In Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing-Volume 10. Association for Computational Linguistics, 79–86.Google Scholar
Digital Library
- [24] . Text-image sentiment analysis.Google Scholar
- [25] . 2019. A journey of Indian languages over sentiment analysis: A systematic review. Artif. Intell. Rev. 52, 2 (2019), 1415–1462.Google Scholar
Digital Library
- [26] . 2009. Part of speech tagger for Assamese text. In Proceedings of the ACL-IJCNLP Conference Short Papers. 33–36.Google Scholar
Cross Ref
- [27] . 2017. Sentiment polarity detection in bengali tweets using multinomial Naïve Bayes and support vector machines. In Proceedings of the IEEE Calcutta Conference (CALCON’17). IEEE, 31–36.Google Scholar
Cross Ref
- [28] . 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556. Retrieved from https://arxiv.org/abs/1409.1556.Google Scholar
- [29] . 2021. An efficient keyframes selection based framework for video captioning. In Proceedings of the 18th International Conference on Natural Language Processing (ICON’21). 240–250.Google Scholar
- [30] . 2021. Review comments of manipuri online video: Good, bad or ugly. In Proceedings of the International Conference on Computing and Communication Systems (I3CS’20), Vol. 170. Springer, 45.Google Scholar
Cross Ref
- [31] . 2018. Boosting image sentiment analysis with visual attention. Neurocomputing 312 (2018), 218–228.Google Scholar
Digital Library
- [32] . 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3156–3164.Google Scholar
Cross Ref
- [33] . 2016. Beyond object recognition: Visual sentiment analysis with deep coupled adjective and noun neural networks. In Proceedings of the Internationa Joint Conference on Artificial Intelligence (IJCAI’16). 3484–3490.Google Scholar
- [34] . 2022. A survey on sentiment analysis methods, applications, and challenges. Artif. Intell. Rev. (2022), 1–50.Google Scholar
- [35] . [n.d.]. Visual sentiment prediction with deep convolutional neural networks. arXiv:1411.5731. Retrieved from https://arixv.org/abs/1411.5731.Google Scholar
- [36] . 2020. Adaptive deep metric learning for affective image retrieval and classification. IEEE Trans. Multimedia (2020).Google Scholar
- [37] . 2016. Robust visual-textual sentiment analysis: When attention meets tree-structured recursive neural networks. In Proceedings of the 24th ACM International Conference on Multimedia. 1008–1017.Google Scholar
Digital Library
- [38] . 2017. Visual sentiment analysis by attending on local image regions. In Proceedings of the 31st AAAI Conference on Artificial Intelligence.Google Scholar
Cross Ref
- [39] . 2015. Joint visual-textual sentiment analysis with deep neural networks. In Proceedings of the 23rd ACM International Conference on Multimedia. 1071–1074.Google Scholar
Digital Library
- [40] . 2013. Sentribute: Image sentiment analysis from a mid-level perspective. In Proceedings of the 2nd International Workshop on Issues of Sentiment Discovery and Opinion Mining. 1–8.Google Scholar
Digital Library
- [41] . 2015. Sentiment analysis on microblogging by integrating text and image features. In Pacific-Asia Conference on Knowledge Discovery and Data Mining. Springer, 52–63.Google Scholar
Cross Ref
- [42] . 2019. An image-text consistency driven multimodal sentiment analysis approach for social media. Inf. Process. Manage. 56, 6 (2019), 102097.Google Scholar
Digital Library
- [43] . 2008. Mention detection crossing the language barrier. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 600–609.Google Scholar
Digital Library
Index Terms
Image–Text Multimodal Sentiment Analysis Framework of Assamese News Articles Using Late Fusion
Recommendations
A multi-stage multimodal framework for sentiment analysis of Assamese in low resource setting
AbstractMultimodality has shown to be helpful in several natural language processing tasks. Thus, adding multiple modalities to the traditional sentiment analysis has also proven to be useful. However, multimodality in a low resource setting ...
Highlights- A multimodal and multi-purpose Assamese news dataset is presented.
- Two models ...
Multimodal Social Media Sentiment Analysis Based on Cross-Modal Hierarchical Attention Fusion
Artificial Intelligence and Mobile Services – AIMS 2021AbstractWith the diversification of data forms on social media, more and more multimodal information mixed with image and text replaces the traditional single text description. Compared with single-modal data, multimodal data can more fully express people’...
Multimodal sentiment analysis with word-level fusion and reinforcement learning
ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal InteractionWith the increasing popularity of video sharing websites such as YouTube and Facebook, multimodal sentiment analysis has received increasing attention from the scientific community. Contrary to previous works in multimodal sentiment analysis which ...






Comments