skip to main content
research-article

Chinese Emotional Dialogue Response Generation via Reinforcement Learning

Authors Info & Claims
Published:22 July 2021Publication History
Skip Abstract Section

Abstract

In an open-domain dialogue system, recognition and expression of emotions are the key factors for success. Most of the existing research related to Chinese dialogue systems aims at improving the quality of content but ignores the expression of human emotions. In this article, we propose a Chinese emotional dialogue response generation algorithm based on reinforcement learning that can generate responses not only according to content but also according to emotion. In the proposed method, a multi-emotion classification model is first used to add emotion labels to the corpus of post-response pairs. Then, with the help of reinforcement learning, the reward function is constructed based on two aspects, namely, emotion and content. Among the generated candidates, the system selects the one with long-term success as the best reply. At the same time, to avoid safe responses and diversify dialogue, a diversity beam search algorithm is applied in the decoding process. The comparative experiments demonstrate that the proposed model achieves satisfactory results according to both automatic and human evaluations.

References

  1. Layla El Asri, Jing He, and Kaheer Suleman. 2016. A sequence-to-sequence model for user simulation in spoken dialogue systems. In Interspeech 2016. 1151–1155. DOI:https://doi.org/10.21437/Interspeech.2016-1175Google ScholarGoogle ScholarCross RefCross Ref
  2. Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations (ICLR’15).Google ScholarGoogle Scholar
  3. Z. Cai, X. Zheng, and J. Yu. 2019. A differential-private framework for urban traffic flows estimation via taxi companies. IEEE Transactions on Industrial Informatics 15, 12 (2019), 6492–6499.Google ScholarGoogle ScholarCross RefCross Ref
  4. Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology 2, 3 (May 2011), Article 27, 27 pages. DOI:https://doi.org/10.1145/1961189.1961199 Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Songsak Channarukul, Susan W. McRoy, and Syed S. Ali. 2001. YAG: A template-based text realization system for dialog. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 9, 06 (2001), 649–659.Google ScholarGoogle ScholarCross RefCross Ref
  6. Yuling Chen and Zhi Zhang. 2018. Research on text sentiment analysis based on CNNs and SVM. In 2018 13th IEEE Conference on Industrial Electronics and Applications (ICIEA’18). IEEE, 2731–2734.Google ScholarGoogle ScholarCross RefCross Ref
  7. Zhenrong Deng, Fuxin Ma, Rushi Lan, Wenming Huang, and Xiaonan Luo. 2020. A two-stage Chinese text summarization algorithm using keyword information and adversarial learning. Neurocomputing 425 (2020), 117–126. DOI:https://doi.org/10.1016/j.neucom.2020.02.102Google ScholarGoogle ScholarCross RefCross Ref
  8. Markus Freitag and Yaser Al-Onaizan. 2017. Beam search strategies for neural machine translation. In Proceedings of the 1st Workshop on Neural Machine Translation. 56–60.Google ScholarGoogle ScholarCross RefCross Ref
  9. Daniel Guo, Gokhan Tur, Wen-tau Yih, and Geoffrey Zweig. 2014. Joint semantic utterance classification and slot filling with recursive neural networks. In 2014 IEEE Spoken Language Technology Workshop (SLT’14). IEEE, 554–559.Google ScholarGoogle ScholarCross RefCross Ref
  10. Hongyu Guo. 2015. Generating text with deep reinforcement learning. CoRR abs/1510.09202 (2015). arxiv:1510.09202.http://arxiv.org/abs/1510.09202Google ScholarGoogle Scholar
  11. Matthew Henderson, Blaise Thomson, and Steve Young. 2013. Deep neural network approach for the dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference. 467–471.Google ScholarGoogle Scholar
  12. R. Lan, L. Sun, Z. Liu, H. Lu, C. Pang, and X. Luo. 2020a. MADNet: A fast and lightweight network for single-image super resolution. IEEE Transactions on Cybernetics 51 (2021), 1443–1453.Google ScholarGoogle Scholar
  13. R. Lan, L. Sun, Z. Liu, H. Lu, Z. Su, C. Pang, and X. Luo. 2020b. Cascading and enhanced residual networks for accurate single-image super-resolution. IEEE Transactions on Cybernetics 51, 1 (2021), 115–121.Google ScholarGoogle ScholarCross RefCross Ref
  14. R. Lan, Y. Zhou, Z. Liu, and X. Luo. 2020c. Prior knowledge-based probabilistic collaborative representation for visual recognition. IEEE Transactions on Cybernetics 50, 4 (2020), 1498–1508.Google ScholarGoogle ScholarCross RefCross Ref
  15. Cheongjae Lee, Sangkeun Jung, Seokhwan Kim, and Gary Geunbae Lee. 2009. Example-based dialog modeling for practical multi-domain dialog system. Speech Communication 51, 5 (2009), 466–484. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. H. Li, C. Pun, F. Xu, L. Pan, R. Zong, H. Gao, and H. Lu. 2020. A hybrid feature selection algorithm based on a discrete artificial bee colony for Parkinson’s diagnosis. ACM Transactions on Internet Technology (2020), doi:10.1145/3397161 Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Jiwei Li, Will Monroe, and Dan Jurafsky. 2016a. A simple, fast diverse decoding algorithm for neural generation. arXiv:1611.08562.Google ScholarGoogle Scholar
  18. Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016b. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 1192–1202.Google ScholarGoogle ScholarCross RefCross Ref
  19. Jing Li, Yan Song, Haisong Zhang, and Shuming Shi. 2018. A manually annotated Chinese corpus for non-task-oriented dialogue systems. arXiv:1805.05542.Google ScholarGoogle Scholar
  20. Shudong Li, Zhou Yan, Xiaobo Wu, Aiping Li, and Bin Zhou. 2017. A method of emotional analysis of movie based on convolution neural network and bi-directional LSTM RNN. In 2017 IEEE S2nd International Conference on Data Science in Cyberspace (DSC’17). IEEE, 156–161.Google ScholarGoogle ScholarCross RefCross Ref
  21. Xiang Li, Lili Mou, Rui Yan, and Ming Zhang. 2016c. Stalematebreaker: A proactive content-introducing approach to automatic human-computer conversation. In Proceedings of the 25th International Joint Conference on Artificial Intelligence. AAAI Press, 2845–2851. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Zhongyang Li, Xiao Ding, and Ting Liu. 2018. Generating reasonable and diversified story ending using sequence to sequence model with adversarial training. In Proceedings of the 27th International Conference on Computational Linguistics. 1033–1043.Google ScholarGoogle Scholar
  23. Fuhua Lin and Douglas H. Norrie. 2001. Schema-based conversation modeling for agent-oriented manufacturing systems. Computers in Industry 46, 3 (2001), 259–274.Google ScholarGoogle ScholarCross RefCross Ref
  24. Chia-Wei Liu, Ryan Lowe, Iulian V. Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2122–2132.Google ScholarGoogle ScholarCross RefCross Ref
  25. Weibo Liu, Zidong Wang, Xiaohui Liu, Nianyin Zeng, Yurong Liu, and Fuad E. Alsaadi. 2017. A survey of deep neural network architectures and their applications. Neurocomputing 234 (2017), 11–26.Google ScholarGoogle ScholarCross RefCross Ref
  26. Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, 1064–1074. DOI:https://doi.org/10.18653/v1/P16-1101Google ScholarGoogle ScholarCross RefCross Ref
  27. Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2017. Coherent dialogue with attention-based language models. In Proceedings of the 31st AAAI Conference on Artificial Intelligence. AAAI Press, 3252–3258. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv:1312.5602.Google ScholarGoogle Scholar
  29. Manel Mrabet, Yosra Ben Saied, and Leila Azouz Saidane. 2019. CAN-TM: Chain augmented Naïve Bayes-based trust model for reliable cloud service selection. ACM Transactions on Internet Technology 19, 4 (Sept. 2019), Article 47, 20 pages. DOI:https://doi.org/10.1145/3341732 Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Tong Niu and Mohit Bansal. 2018. Polite dialogue generation without parallel data. Transactions of the Association of Computational Linguistics 6 (2018), 373–389.Google ScholarGoogle ScholarCross RefCross Ref
  31. H. Peng, E. Cambria, and A. Hussain. 2017. A review of sentiment analysis research in Chinese language. Cognitive Computation 9, 8 (2017), 423–435.Google ScholarGoogle ScholarCross RefCross Ref
  32. Alan Ritter, Colin Cherry, and William B. Dolan. 2011. Data-driven response generation in social media. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 583–593. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Iulian Vlad Serban, Tim Klinger, Gerald Tesauro, Kartik Talamadupula, Bowen Zhou, Yoshua Bengio, and Aaron Courville. 2017. Multiresolution recurrent neural networks: An application to dialogue response generation. In Proceedings of the 31st AAAI Conference on Artificial Intelligence. AAAI Press, 3288–3294. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Iulian V. Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the 30h AAAI Conference on Artificial Intelligence. AAAI Press, 3776–3783. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Amanda Stent, Rashmi Prasad, and Marilyn Walker. 2004. Trainable sentence planning for complex information presentation in spoken dialog systems. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, 79. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. David Suendermann, Keelan Evanini, Jackson Liscombe, Phillip Hunter, Krishna Dayanidhi, and Roberto Pieraccini. 2009. From rule-based to statistical grammars: Continuous improvement of large-scale spoken dialog systems. In 2009 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 4713–4716. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Xiao Sun, Chengcheng Li, and Fuji Ren. 2016. Sentiment analysis for Chinese microblog based on deep neural networks with convolutional extension features. Neurocomputing 210 (2016), 227–236. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems. 3104–3112. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2018. Ruber: An unsupervised method for automatic evaluation of open-domain dialog systems. In 32nd AAAI Conference on Artificial Intelligence.Google ScholarGoogle Scholar
  40. Zhiyang Teng, Duy Tin Vo, and Yue Zhang. 2016. Context-sensitive lexicon features for neural sentiment analysis. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 1629–1638.Google ScholarGoogle ScholarCross RefCross Ref
  41. Ashwin K. Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse beam search for improved description of complex scenes. In 32nd AAAI Conference on Artificial Intelligence. 7371–7379.Google ScholarGoogle Scholar
  42. Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv:1506.05869.Google ScholarGoogle Scholar
  43. Y. Wang, Y. Gao, S. Li, and X. Tong. 2020. A worker-selection incentive mechanism for optimizing platform-centric mobile crowdsourcing systems. Computer Networks 107 (2020), 107144.Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Zhuoran Wang, Hongliang Chen, Guanchun Wang, Hao Tian, Hua Wu, and Haifeng Wang. 2014. Policy learning for domain selection in an extensible multi-domain spoken dialogue system. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP’14). 57–67.Google ScholarGoogle ScholarCross RefCross Ref
  45. Jason Williams, Antoine Raux, Deepak Ramachandran, and Alan Black. 2013. The dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference. 404–413.Google ScholarGoogle Scholar
  46. Sam Wiseman and Alexander M. Rush. 2016. Sequence-to-sequence learning as beam-search optimization. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 1296–1306.Google ScholarGoogle Scholar
  47. Hong Xie, Yongkun Li, and John C. S. Lui. 2020. A reinforcement learning approach to optimize discount and reputation tradeoffs in e-commerce systems. ACM Transactions on Internet Technology 20, 4 (Oct. 2020), Article 37, 26 pages. DOI:https://doi.org/10.1145/3400024 Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei-Ying Ma. 2017. Topic aware neural response generation. In Proceedings of the 31st AAAI Conference on Artificial Intelligence. AAAI Press, 3351–3357. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. Emotional chatting machine: Emotional conversation generation with internal and external memory. In 32nd AAAI Conference on Artificial Intelligence. 730–738.Google ScholarGoogle Scholar
  50. Yucan Zhou, Qinghua Hu, Jie Liu, and Yuan Jia. 2015. Combining heterogeneous deep neural networks with conditional random fields for Chinese dialogue act recognition. Neurocomputing 168 (2015), 408–417. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Chinese Emotional Dialogue Response Generation via Reinforcement Learning

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!