skip to main content
research-article

Emotion Recognition with Conversational Generation Transfer

Published:19 January 2022Publication History
Skip Abstract Section

Abstract

Emotion recognition in conversation is one of the essential tasks of natural language processing. However, this task’s annotation data is insufficient since such data is hard to collect and annotate. Meanwhile, there is large-scale data for conversational generation, and this data does not need annotation manually. But, whether the vector space between different datasets is similar will be a problem. Therefore, we utilize a same dataset to train the conversational generator and the classifier, and transfer knowledge between them. In particular, we propose an Emotion Recognition with Conversational Generation Transfer (ERCGT) framework to model the interaction among utterances by transfer learning. First, we train a conversational generator. In the second step, a transfer learning model is used to transfer the knowledge of generator to the emotion recognition model. Empirical studies illustrate the effectiveness of the proposed framework over several strong baselines on three benchmark emotion classification datasets.

REFERENCES

  1. [1] Aguilar Gustavo, Rozgic Viktor, Wang Weiran, and Wang Chao. 2019. Multimodal and multi-view models for emotion recognition. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 9911002. https://doi.org/10.18653/v1/P19-1095Google ScholarGoogle ScholarCross RefCross Ref
  2. [2] Busso Carlos, Bulut Murtaza, Lee Chi Chun, Kazemzadeh Abe, Mower Emily, Kim Samuel, Chang Jeannette N., Lee Sungbok, and Narayanan Shrikanth S.. 2008. IEMOCAP: Interactive emotional dyadic motion capture database. Language Resources & Evaluation 42, 4 (2008), 335359.Google ScholarGoogle ScholarCross RefCross Ref
  3. [3] Dahou Abdelghani, Xiong Shengwu, Zhou Junwei, Haddoud Mohamed Houcine, and Duan Pengfei. 2016. Word embeddings and convolutional neural network for Arabic sentiment classification. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. The COLING 2016 Organizing Committee, Osaka, Japan, 24182427.Google ScholarGoogle Scholar
  4. [4] Danescu-Niculescu-Mizil Cristian and Lee Lillian. 2011. Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. In Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguistics. Association for Computational Linguistics, Portland, Oregon, USA, 7687. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. [5] Devlin Jacob, Chang Ming-Wei, Lee Kenton, and Toutanova Kristina. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 41714186. https://doi.org/10.18653/v1/N19-1423Google ScholarGoogle Scholar
  6. [6] Dinan Emily, Roller Stephen, Shuster Kurt, Fan Angela, Auli Michael, and Weston Jason. 2018. Wizard of Wikipedia: Knowledge-powered conversational agents. arXiv preprint arXiv:1811.01241 (2018).Google ScholarGoogle Scholar
  7. [7] Duchi John, Hazan Elad, and Singer Yoram. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research 12, Jul (2011), 21212159. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. [8] Felbo Bjarke, Mislove Alan, Søgaard Anders, Rahwan Iyad, and Lehmann Sune. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 16151625. https://doi.org/10.18653/v1/D17-1169Google ScholarGoogle ScholarCross RefCross Ref
  9. [9] Ghosal Deepanway, Majumder Navonil, Poria Soujanya, Chhaya Niyati, and Gelbukh Alexander. 2019. DialogueGCN: A graph convolutional neural network for emotion recognition in conversation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, 154164. https://doi.org/10.18653/v1/D19-1015Google ScholarGoogle ScholarCross RefCross Ref
  10. [10] Gideon John, Khorram Soheil, Aldeneh Zakaria, Dimitriadis Dimitrios, and Provost Emily Mower. 2017. Progressive neural networks for transfer learning in emotion recognition. 10981102. https://doi.org/10.21437/Interspeech.2017-1637Google ScholarGoogle Scholar
  11. [11] Gu Yue, Yang Kangning, Fu Shiyu, Chen Shuhong, Li Xinyu, and Marsic Ivan. 2018. Hybrid attention based multimodal network for spoken language classification. In Proceedings of the 27th International Conference on Computational Linguistics. Association for Computational Linguistics, Santa Fe, New Mexico, USA, 23792390.Google ScholarGoogle Scholar
  12. [12] Hazarika Devamanyu, Poria Soujanya, Mihalcea Rada, Cambria Erik, and Zimmermann Roger. 2018. ICON: Interactive conversational memory network for multimodal emotion detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Brussels, Belgium, 25942604. https://doi.org/10.18653/v1/D18-1280Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] Hazarika Devamanyu, Poria Soujanya, Zadeh Amir, Cambria Erik, Morency Louis-Philippe, and Zimmermann Roger. 2018. Conversational memory network for emotion recognition in dyadic dialogue videos. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics, New Orleans, Louisiana, 21222132. https://doi.org/10.18653/v1/N18-1193Google ScholarGoogle ScholarCross RefCross Ref
  14. [14] Hazarika Devamanyu, Poria Soujanya, Zimmermann Roger, and Mihalcea Rada. 2021. Conversational transfer learning for emotion recognition. Information Fusion 65 (2021), 112.Google ScholarGoogle ScholarCross RefCross Ref
  15. [15] Hsu Chao-Chun, Chen Sheng-Yeh, Kuo Chuan-Chun, Huang Ting-Hao, and Ku Lun-Wei. 2018. EmotionLines: An emotion corpus of multi-party conversations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA), Miyazaki, Japan, 15971601.Google ScholarGoogle Scholar
  16. [16] Huang Minlie, Cao Yujie, and Dong Chao. 2016. Modeling rich contexts for sentiment classification with LSTM. arXiv preprint arXiv:1605.01478 (2016).Google ScholarGoogle Scholar
  17. [17] Huang Yen-Hao, Lee Ssu-Rui, Ma Mau-Yun, Chen Yi-Hsin, Yu Ya-Wen, and Chen Yi-Shin. 2019. EmotionX-IDEA: Emotion BERT – An affectional model for conversation. arXiv preprint arXiv:1908.06264.Google ScholarGoogle Scholar
  18. [18] Jiao Wenxiang, Lyu Michael R., and King Irwin. 2019. PT-CoDE: Pre-trained Context-Dependent Encoder for Utterancelevel Emotion Recognition. arXiv preprint arXiv: 1910.08916 (2019).Google ScholarGoogle Scholar
  19. [19] Li Qintong, Chen Hongshen, Ren Zhaochun, Ren Pengjie, Tu Zhaopeng, and Chen Zhumin. 2020. EmpDG: Multi-resolution interactive empathetic dialogue generation. In Proceedings of the 28th International Conference on Computational Linguistics. International Committee on Computational Linguistics, Barcelona, Spain (Online), 44544466. https://doi.org/10.18653/v1/2020.coling-main.394Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Li Yanran, Su Hui, Shen Xiaoyu, Li Wenjie, Cao Ziqiang, and Niu Shuzi. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Asian Federation of Natural Language Processing, Taipei, Taiwan, 986995.Google ScholarGoogle Scholar
  21. [21] Lowe Ryan, Pow Nissan, Serban Iulian, and Pineau Joelle. 2015. The Ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue. Association for Computational Linguistics, Prague, Czech Republic, 285294. https://doi.org/10.18653/v1/W15-4640Google ScholarGoogle ScholarCross RefCross Ref
  22. [22] Majumder Navonil, Poria Soujanya, Hazarika Devamanyu, Mihalcea Rada, Gelbukh Alexander, and Cambria Erik. 2019. DialogueRNN: An attentive RNN for emotion detection in conversations. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 68186825. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. [23] Matsumoto Shotaro, Takamura Hiroya, and Okumura Manabu. 2005. Sentiment classification using word sub-sequences and dependency sub-trees. In Pacific-Asia Conference on Knowledge Discovery and Data Mining. Springer, 301311. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. [24] Morris Michael W. and Keltner Dacher. 2000. How emotions work: The social functions of emotional expression in negotiations. Research in Organizational Behavior 22 (2000), 150.Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] O’Keefe Tim and Koprinska Irena. 2009. Feature selection and weighting methods in sentiment analysis. In Proceedings of the 14th Australasian Document Computing Symposium, Sydney. Citeseer, 6774.Google ScholarGoogle Scholar
  26. [26] Pang Bo, Lee Lillian, and Vaithyanathan Shivakumar. 2002. Thumbs up? Sentiment classification using machine learning techniques. EMNLP 10 (06 2002), 7986. https://doi.org/10.3115/1118693.1118704 Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. [27] Park Yookoon, Cho Jaemin, and Kim Gunhee. 2018. A hierarchical latent structure for variational conversation modeling. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics, New Orleans, Louisiana, 17921801. https://doi.org/10.18653/v1/N18-1162Google ScholarGoogle ScholarCross RefCross Ref
  28. [28] Poria Soujanya, Cambria Erik, Hazarika Devamanyu, Majumder Navonil, Zadeh Amir, and Morency Louis-Philippe. 2017. Context-Dependent sentiment analysis in user-generated videos. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada, 873883. https://doi.org/10.18653/v1/P17-1081Google ScholarGoogle ScholarCross RefCross Ref
  29. [29] Poria Soujanya, Hazarika Devamanyu, Majumder Navonil, Naik Gautam, Cambria Erik, and Mihalcea Rada. 2019. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 527536. https://doi.org/10.18653/v1/P19-1050Google ScholarGoogle ScholarCross RefCross Ref
  30. [30] Poria Soujanya, Majumder Navonil, Mihalcea Rada, and Hovy Eduard. 2019. Emotion recognition in conversation: Research challenges, datasets, and recent advances. IEEE Access 7 (May 2019), 100943100953. https://doi.org/10.1109/ACCESS.2019.2929050Google ScholarGoogle ScholarCross RefCross Ref
  31. [31] Serban Iulian, Sordoni Alessandro, Lowe Ryan, Charlin Laurent, Pineau Joelle, Courville Aaron, and Bengio Yoshua. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. Proceedings of the AAAI Conference on Artificial Intelligence 31, 1, 32953301. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. [32] Serban Iulian V., Sordoni Alessandro, Bengio Yoshua, Courville Aaron, and Pineau Joelle. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Vol. 33. 37763783. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. [33] Wei Jia, Feng Shi, Wang Daling, Zhang Yifei, and Li Xiangju. 2019. Attentional neural network for emotion detection in conversations with speaker influence awareness. In Natural Language Processing and Chinese Computing. Springer, Springer International Publishing, Cham, 287297.Google ScholarGoogle Scholar
  34. [34] Zhang Changli, Zeng Daniel, Li Jiexun, Wang Fei-Yue, and Zuo Wanli. 2009. Sentiment analysis of Chinese documents: From sentence to document level. Journal of the American Society for Information Science and Technology 60, 12 (2009), 24742487. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. [35] Zhang Dong, Wu Liangqing, Sun Changlong, Li Shoushan, Zhu Qiaoming, and Zhou Guodong. 2019. Modeling both context- and speaker-sensitive dependence for emotion detection in multi-speaker conversations. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19. International Joint Conferences on Artificial Intelligence Organization, 54155421. https://doi.org/10.24963/ijcai.2019/752 Google ScholarGoogle ScholarCross RefCross Ref
  36. [36] Zhang Weinan, Li Lingzhi, Cao Dongyan, and Liu Ting. 2018. Exploring implicit feedback for open domain conversation generation. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th Innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2–7, 2018, McIlraith Sheila A. and Weinberger Kilian Q. (Eds.). AAAI Press, 547554. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. [37] Zhou Guangyou, Fang Yizhen, Peng Yehong, and Lu Jiaheng. 2019. Neural conversation generation with auxiliary emotional supervised models. ACM Trans. Asian Low-Resour. Lang. Inf. Process. 19, 2, Article 19 (Sept. 2019), 17 pages. https://doi.org/10.1145/3344788 Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. [38] Zhou Hao, Huang Minlie, Zhang Tianyang, Zhu Xiaoyan, and Liu Bing. 2018. Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory. Proceedings of the AAAI Conference on Artificial Intelligence 32, 1 (2018), 730738. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Emotion Recognition with Conversational Generation Transfer

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Asian and Low-Resource Language Information Processing
      ACM Transactions on Asian and Low-Resource Language Information Processing  Volume 21, Issue 4
      July 2022
      464 pages
      ISSN:2375-4699
      EISSN:2375-4702
      DOI:10.1145/3511099
      Issue’s Table of Contents

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 19 January 2022
      • Accepted: 1 October 2021
      • Revised: 1 August 2021
      • Received: 1 December 2020
      Published in tallip Volume 21, Issue 4

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Refereed
    • Article Metrics

      • Downloads (Last 12 months)222
      • Downloads (Last 6 weeks)15

      Other Metrics

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    View Full Text

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!