Abstract
In an open-domain dialogue system, recognition and expression of emotions are the key factors for success. Most of the existing research related to Chinese dialogue systems aims at improving the quality of content but ignores the expression of human emotions. In this article, we propose a Chinese emotional dialogue response generation algorithm based on reinforcement learning that can generate responses not only according to content but also according to emotion. In the proposed method, a multi-emotion classification model is first used to add emotion labels to the corpus of post-response pairs. Then, with the help of reinforcement learning, the reward function is constructed based on two aspects, namely, emotion and content. Among the generated candidates, the system selects the one with long-term success as the best reply. At the same time, to avoid safe responses and diversify dialogue, a diversity beam search algorithm is applied in the decoding process. The comparative experiments demonstrate that the proposed model achieves satisfactory results according to both automatic and human evaluations.
- Layla El Asri, Jing He, and Kaheer Suleman. 2016. A sequence-to-sequence model for user simulation in spoken dialogue systems. In Interspeech 2016. 1151–1155. DOI:https://doi.org/10.21437/Interspeech.2016-1175Google Scholar
Cross Ref
- Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations (ICLR’15).Google Scholar
- Z. Cai, X. Zheng, and J. Yu. 2019. A differential-private framework for urban traffic flows estimation via taxi companies. IEEE Transactions on Industrial Informatics 15, 12 (2019), 6492–6499.Google Scholar
Cross Ref
- Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology 2, 3 (May 2011), Article 27, 27 pages. DOI:https://doi.org/10.1145/1961189.1961199 Google Scholar
Digital Library
- Songsak Channarukul, Susan W. McRoy, and Syed S. Ali. 2001. YAG: A template-based text realization system for dialog. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 9, 06 (2001), 649–659.Google Scholar
Cross Ref
- Yuling Chen and Zhi Zhang. 2018. Research on text sentiment analysis based on CNNs and SVM. In 2018 13th IEEE Conference on Industrial Electronics and Applications (ICIEA’18). IEEE, 2731–2734.Google Scholar
Cross Ref
- Zhenrong Deng, Fuxin Ma, Rushi Lan, Wenming Huang, and Xiaonan Luo. 2020. A two-stage Chinese text summarization algorithm using keyword information and adversarial learning. Neurocomputing 425 (2020), 117–126. DOI:https://doi.org/10.1016/j.neucom.2020.02.102Google Scholar
Cross Ref
- Markus Freitag and Yaser Al-Onaizan. 2017. Beam search strategies for neural machine translation. In Proceedings of the 1st Workshop on Neural Machine Translation. 56–60.Google Scholar
Cross Ref
- Daniel Guo, Gokhan Tur, Wen-tau Yih, and Geoffrey Zweig. 2014. Joint semantic utterance classification and slot filling with recursive neural networks. In 2014 IEEE Spoken Language Technology Workshop (SLT’14). IEEE, 554–559.Google Scholar
Cross Ref
- Hongyu Guo. 2015. Generating text with deep reinforcement learning. CoRR abs/1510.09202 (2015). arxiv:1510.09202.http://arxiv.org/abs/1510.09202Google Scholar
- Matthew Henderson, Blaise Thomson, and Steve Young. 2013. Deep neural network approach for the dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference. 467–471.Google Scholar
- R. Lan, L. Sun, Z. Liu, H. Lu, C. Pang, and X. Luo. 2020a. MADNet: A fast and lightweight network for single-image super resolution. IEEE Transactions on Cybernetics 51 (2021), 1443–1453.Google Scholar
- R. Lan, L. Sun, Z. Liu, H. Lu, Z. Su, C. Pang, and X. Luo. 2020b. Cascading and enhanced residual networks for accurate single-image super-resolution. IEEE Transactions on Cybernetics 51, 1 (2021), 115–121.Google Scholar
Cross Ref
- R. Lan, Y. Zhou, Z. Liu, and X. Luo. 2020c. Prior knowledge-based probabilistic collaborative representation for visual recognition. IEEE Transactions on Cybernetics 50, 4 (2020), 1498–1508.Google Scholar
Cross Ref
- Cheongjae Lee, Sangkeun Jung, Seokhwan Kim, and Gary Geunbae Lee. 2009. Example-based dialog modeling for practical multi-domain dialog system. Speech Communication 51, 5 (2009), 466–484. Google Scholar
Digital Library
- H. Li, C. Pun, F. Xu, L. Pan, R. Zong, H. Gao, and H. Lu. 2020. A hybrid feature selection algorithm based on a discrete artificial bee colony for Parkinson’s diagnosis. ACM Transactions on Internet Technology (2020), doi:10.1145/3397161 Google Scholar
Digital Library
- Jiwei Li, Will Monroe, and Dan Jurafsky. 2016a. A simple, fast diverse decoding algorithm for neural generation. arXiv:1611.08562.Google Scholar
- Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016b. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 1192–1202.Google Scholar
Cross Ref
- Jing Li, Yan Song, Haisong Zhang, and Shuming Shi. 2018. A manually annotated Chinese corpus for non-task-oriented dialogue systems. arXiv:1805.05542.Google Scholar
- Shudong Li, Zhou Yan, Xiaobo Wu, Aiping Li, and Bin Zhou. 2017. A method of emotional analysis of movie based on convolution neural network and bi-directional LSTM RNN. In 2017 IEEE S2nd International Conference on Data Science in Cyberspace (DSC’17). IEEE, 156–161.Google Scholar
Cross Ref
- Xiang Li, Lili Mou, Rui Yan, and Ming Zhang. 2016c. Stalematebreaker: A proactive content-introducing approach to automatic human-computer conversation. In Proceedings of the 25th International Joint Conference on Artificial Intelligence. AAAI Press, 2845–2851. Google Scholar
Digital Library
- Zhongyang Li, Xiao Ding, and Ting Liu. 2018. Generating reasonable and diversified story ending using sequence to sequence model with adversarial training. In Proceedings of the 27th International Conference on Computational Linguistics. 1033–1043.Google Scholar
- Fuhua Lin and Douglas H. Norrie. 2001. Schema-based conversation modeling for agent-oriented manufacturing systems. Computers in Industry 46, 3 (2001), 259–274.Google Scholar
Cross Ref
- Chia-Wei Liu, Ryan Lowe, Iulian V. Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2122–2132.Google Scholar
Cross Ref
- Weibo Liu, Zidong Wang, Xiaohui Liu, Nianyin Zeng, Yurong Liu, and Fuad E. Alsaadi. 2017. A survey of deep neural network architectures and their applications. Neurocomputing 234 (2017), 11–26.Google Scholar
Cross Ref
- Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, 1064–1074. DOI:https://doi.org/10.18653/v1/P16-1101Google Scholar
Cross Ref
- Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2017. Coherent dialogue with attention-based language models. In Proceedings of the 31st AAAI Conference on Artificial Intelligence. AAAI Press, 3252–3258. Google Scholar
Digital Library
- Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv:1312.5602.Google Scholar
- Manel Mrabet, Yosra Ben Saied, and Leila Azouz Saidane. 2019. CAN-TM: Chain augmented Naïve Bayes-based trust model for reliable cloud service selection. ACM Transactions on Internet Technology 19, 4 (Sept. 2019), Article 47, 20 pages. DOI:https://doi.org/10.1145/3341732 Google Scholar
Digital Library
- Tong Niu and Mohit Bansal. 2018. Polite dialogue generation without parallel data. Transactions of the Association of Computational Linguistics 6 (2018), 373–389.Google Scholar
Cross Ref
- H. Peng, E. Cambria, and A. Hussain. 2017. A review of sentiment analysis research in Chinese language. Cognitive Computation 9, 8 (2017), 423–435.Google Scholar
Cross Ref
- Alan Ritter, Colin Cherry, and William B. Dolan. 2011. Data-driven response generation in social media. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 583–593. Google Scholar
Digital Library
- Iulian Vlad Serban, Tim Klinger, Gerald Tesauro, Kartik Talamadupula, Bowen Zhou, Yoshua Bengio, and Aaron Courville. 2017. Multiresolution recurrent neural networks: An application to dialogue response generation. In Proceedings of the 31st AAAI Conference on Artificial Intelligence. AAAI Press, 3288–3294. Google Scholar
Digital Library
- Iulian V. Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the 30h AAAI Conference on Artificial Intelligence. AAAI Press, 3776–3783. Google Scholar
Digital Library
- Amanda Stent, Rashmi Prasad, and Marilyn Walker. 2004. Trainable sentence planning for complex information presentation in spoken dialog systems. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, 79. Google Scholar
Digital Library
- David Suendermann, Keelan Evanini, Jackson Liscombe, Phillip Hunter, Krishna Dayanidhi, and Roberto Pieraccini. 2009. From rule-based to statistical grammars: Continuous improvement of large-scale spoken dialog systems. In 2009 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 4713–4716. Google Scholar
Digital Library
- Xiao Sun, Chengcheng Li, and Fuji Ren. 2016. Sentiment analysis for Chinese microblog based on deep neural networks with convolutional extension features. Neurocomputing 210 (2016), 227–236. Google Scholar
Digital Library
- Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems. 3104–3112. Google Scholar
Digital Library
- Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2018. Ruber: An unsupervised method for automatic evaluation of open-domain dialog systems. In 32nd AAAI Conference on Artificial Intelligence.Google Scholar
- Zhiyang Teng, Duy Tin Vo, and Yue Zhang. 2016. Context-sensitive lexicon features for neural sentiment analysis. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 1629–1638.Google Scholar
Cross Ref
- Ashwin K. Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse beam search for improved description of complex scenes. In 32nd AAAI Conference on Artificial Intelligence. 7371–7379.Google Scholar
- Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv:1506.05869.Google Scholar
- Y. Wang, Y. Gao, S. Li, and X. Tong. 2020. A worker-selection incentive mechanism for optimizing platform-centric mobile crowdsourcing systems. Computer Networks 107 (2020), 107144.Google Scholar
Digital Library
- Zhuoran Wang, Hongliang Chen, Guanchun Wang, Hao Tian, Hua Wu, and Haifeng Wang. 2014. Policy learning for domain selection in an extensible multi-domain spoken dialogue system. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP’14). 57–67.Google Scholar
Cross Ref
- Jason Williams, Antoine Raux, Deepak Ramachandran, and Alan Black. 2013. The dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference. 404–413.Google Scholar
- Sam Wiseman and Alexander M. Rush. 2016. Sequence-to-sequence learning as beam-search optimization. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 1296–1306.Google Scholar
- Hong Xie, Yongkun Li, and John C. S. Lui. 2020. A reinforcement learning approach to optimize discount and reputation tradeoffs in e-commerce systems. ACM Transactions on Internet Technology 20, 4 (Oct. 2020), Article 37, 26 pages. DOI:https://doi.org/10.1145/3400024 Google Scholar
Digital Library
- Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei-Ying Ma. 2017. Topic aware neural response generation. In Proceedings of the 31st AAAI Conference on Artificial Intelligence. AAAI Press, 3351–3357. Google Scholar
Digital Library
- Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. Emotional chatting machine: Emotional conversation generation with internal and external memory. In 32nd AAAI Conference on Artificial Intelligence. 730–738.Google Scholar
- Yucan Zhou, Qinghua Hu, Jie Liu, and Yuan Jia. 2015. Combining heterogeneous deep neural networks with conditional random fields for Chinese dialogue act recognition. Neurocomputing 168 (2015), 408–417. Google Scholar
Digital Library
Index Terms
Chinese Emotional Dialogue Response Generation via Reinforcement Learning
Recommendations
CAB: Empathetic Dialogue Generation with Cognition, Affection and Behavior
Database Systems for Advanced ApplicationsAbstractEmpathy is an important characteristic to be considered when building a more intelligent and humanized dialogue agent. However, existing methods did not fully comprehend empathy as a complex process involving three aspects: cognition, affection ...
Neural Emotional Response Generation via Adversarial Transfer Learning
ICIAI '19: Proceedings of the 2019 3rd International Conference on Innovation in Artificial IntelligenceEmotional response generation is a key step to build an empathetic chatbot. However, previous emotional chatting models mainly focus on single-turn conversation, and multi-turn context emotional response generation has not been explored. In this paper, ...
Emotion in reinforcement learning agents and robots: a survey
This article provides the first survey of computational models of emotion in reinforcement learning (RL) agents. The survey focuses on agent/robot emotions, and mostly ignores human user emotions. Emotions are recognized as functional in decision-making ...






Comments