Abstract
Learning to generate coherent and informative dialogs is an enduring challenge for open-domain conversation generation. Previous work leverage knowledge graph or documents to facilitate informative dialog generation, with little attention on dialog coherence. In this article, to enhance multi-turn open-domain dialog coherence, we propose to leverage a new knowledge source, web search session data, to facilitate hierarchical knowledge sequence planning, which determines a sketch of a multi-turn dialog. Specifically, we formulate knowledge sequence planning or dialog policy learning as a graph grounded Reinforcement Learning (RL) problem. To this end, we first build a two-level query graph with queries as utterance-level vertices and their topics (entities in queries) as topic-level vertices. We then present a two-level dialog policy model that plans a high-level topic sequence and a low-level query sequence over the query graph to guide a knowledge aware response generator. In particular, to foster forward-looking knowledge planning decisions for better dialog coherence, we devise a heterogeneous graph neural network to incorporate neighbouring vertex information, or possible future RL action information, into each vertex (as an RL action) representation. Experiment results on two benchmark dialog datasets demonstrate that our framework can outperform strong baselines in terms of dialog coherence, informativeness, and engagingness.
- Nabiha Asghar, Pascal Poupart, Jiang Xin, and Hang Li. 2016. Online sequence-to-sequence reinforcement learning for open-domain conversational agents. arXiv:1612.039293. Retrieved from https://arxiv.org/abs/1612.039293.Google Scholar
- Robert S. Belvin, Ron Burns, and Cheryl Hein. 2001. Development of the hrl route navigation dialogue system. In Proceedings of the 1st International Conference on Human Language Technology Research. Google Scholar
Digital Library
- Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems. Google Scholar
Digital Library
- Chaotao Chen, Jinhua Peng, Fan Wang, Jun Xu, and Hua Wu. 2019. Generating multiple diverse responses with multi-mapping and posterior mapping selection. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI'19). Google Scholar
Cross Ref
- Lu Chen, Cheng Chang, Zhi Chen, Bowen Tan, Milica Gašić, and Kai Yu. 2018. Policy adaptation for deep reinforcement learning-based dialogue management. In Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'18). IEEE, 6074–6078.Google Scholar
Cross Ref
- Lu Chen, Zhi Chen, Bowen Tan, Sishan Long, Milica Gašić, and Kai Yu. 2019. Agentgraph: Toward universal dialogue management with structured deep reinforcement learning. IEEE/ACM Trans. Aud. Speech Lang. Process. 27, 9 (2019), 1378–1391. Google Scholar
Digital Library
- Lu Chen, Boer Lv, Chi Wang, Su Zhu, Bowen Tan, and Kai Yu. 2020. Schema-guided multi-domain dialogue state tracking with graph attention neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 7521–7528.Google Scholar
Cross Ref
- Lu Chen, Bowen Tan, Sishan Long, and Kai Yu. 2018. Structured dialogue policy with graph neural networks. In Proceedings of the 27th International Conference on Computational Linguistics. 1257–1268.Google Scholar
- Qibin Chen, Junyang Lin, Yichang Zhang, Ming Ding, Yukuo Cen, Hongxia Yang, and Jie Tang. 2019. Towards knowledge-based recommender dialog system. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'19).Google Scholar
Cross Ref
- Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard Of Wikipedia: Knowledge-powered conversational agents. In Proceedings of the International Conference on Learning Representations (ICLR'19). Association for Computational Linguistics.Google Scholar
- Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen tau Yih, and Michel Galley. 2018. A knowledge-grounded neural conversation model. In Proceedings of the AAAI Conference on Artificial Intelligence. Association for the Advancement of Artificial Intelligence, 5110–5117.Google Scholar
- Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. 2019. DialogueGCN: A graph convolutional neural network for emotion recognition in conversation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'19).Google Scholar
Cross Ref
- Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 1631–1640. https://doi.org/10.18653/v1/P16-1154Google Scholar
- Sangdo Han, Jeesoo Bang, Seonghan Ryu, and Gary Geunbae Lee. 2015. Exploiting knowledge base to generate responses for natural language dialog listening agents. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL'15).Google Scholar
Cross Ref
- Ahmed Hassan. 2013. Identifying web search query reformulation using concept based matching. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP'13).Google Scholar
- Brent Hecht, Jaime Teevan, Meredith Ringel Morris, and Dan Liebling. 2012. Searchbuddies: Bringing search engines into the conversation. In Proceedings of the 6th International AAAI Conference on Weblogs and Social Media.Google Scholar
- Toru Hirano, Ryuichiro Higashinaka, and Yoshihiro Matsuo. 2016. Analyzing post-dialogue comments by speakers–how do humans personalize their utterances in dialogue? In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue. 157–165.Google Scholar
Cross Ref
- Jizhou Huang, Ming Zhou, and Dan Yang. 2007. Extracting chatbot knowledge from online discussion forums. In Proceedings of the 20th International Joint Conference on Artifical intelligence. 423–428. Google Scholar
Digital Library
- Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with gumbel-softmax. In Proceedings of the International Conference on Learning Representations (ICLR'17).Google Scholar
- Zongcheng Ji, Zhengdong Lu, and Hang Li. 2014. An information retrieval approach to short text conversation. arXiv preprint arXiv:1408.6988 (2014).Google Scholar
- Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Proceedings of the International Conference on Learning Representations (ICLR'17).Google Scholar
- Giovanni Yoko Kristianto, Huiwen Zhang, Bin Tong, Makoto Iwayama, and Yoshiyuki Kobayashi. 2018. Autonomous sub-domain modeling for dialogue policy with hierarchical deep reinforcement learning. In Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI. 9–16.Google Scholar
Cross Ref
- Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2443–2453.Google Scholar
Cross Ref
- Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT'16).Google Scholar
Cross Ref
- Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016. Deep reinforcement learning for dialogue generation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'16).Google Scholar
Cross Ref
- Shuman Liu, Hongshen Chen, Zhaochun Ren, Yang Feng, Qun Liu, and Dawei Yin. 2018. Knowledge diffusion for neural dialogue generation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL'18).Google Scholar
Cross Ref
- Zhibin Liu, Zheng-Yu Niu, Hua Wu, and Haifeng Wang. 2019. Knowledge aware conversation generation with explainable reasoning over augmented graphs. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing (EMNLP-IJCNLP'19).Google Scholar
Cross Ref
- Yinong Long, Jianan Wang, Zhen Xu, Zongsheng Wang, Baoxun Wang, and Zhuoran Wang. 2017. A knowledge enhanced generative conversational service agent. In Proceedings of the Dialog System Technology Challenge 6 (DSTC6'17) Workshop.Google Scholar
- Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'15).Google Scholar
- Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. 2013. Rectifier nonlinearities improve neural network acoustic models. In Proceedings of the International Conference on Machine Learning (ICML'13).Google Scholar
- Sahisnu Mazumder, Nianzu Ma, and Bing Liu. 2018. Towards a continuous knowledge learning engine for chatbots. arXiv:1802.06024. Retrieved from https://arxiv.org/abs/1802.06024.Google Scholar
- Nikita Moghe, Siddhartha Arora, Suman Banerjee, and Mitesh M. Khapra. 2018. Towards exploiting background knowledge for building conversation systems. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'18). Association for Computational Linguistics, 2322–2332.Google Scholar
- Seungwhan Moon, Pararth Shah, Anuj Kumar, and Rajen Subba. 2019. Opendialkg: Explainable conversational reasoning with attention-based walks over knowledge graphs. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL'19).Google Scholar
Cross Ref
- Gaurav Pandey, Danish Contractor, Vineet Kumar, and Sachindra Joshi. 2018. Exemplar encoder-decoder for neural conversation generation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL'18).Google Scholar
Cross Ref
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, 311–318. Google Scholar
Digital Library
- Baolin Peng, Xiujun Li, Lihong Li, Jianfeng Gao, Asli Celikyilmaz, Sungjin Lee, and Kam-Fai Wong. 2017. Composite task-completion dialogue policy learning via hierarchical deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2231–2240.Google Scholar
Cross Ref
- Lianhui Qin, Michel Galley, Chris Brockett, Xiaodong Liu, Xiang Gao, Bill Dolan, Yejin Choi, and Jianfeng Gao. 2019. Conversing by reading: Contentful neural conversation with on-demand machine reading. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 5427–5436.Google Scholar
Cross Ref
- Filip Radlinski and Nick Craswell. 2017. A theoretical framework for conversational search. In Proceedings of the 2017 Conference on Human Information Interaction and Retrieval. 117–126. Google Scholar
Digital Library
- Justus J. Randolph. 2005. Free-marginal multirater kappa (multirater free): An alternative to fleiss' fixed-marginal multirater kappa. In Presented at the Joensuu Learning and Instruction Symposium, Vol. 2005.Google Scholar
- Pengjie Ren, Zhumin Chen, Zhaochun Ren, Evangelos Kanoulas, Christof Monz, and Maarten de Rijke. 2020. Conversations with search engines. arxiv:2004.14162. Retrieved fromhttps://arxiv.org/abs/2004.14162.Google Scholar
- Alan Ritter, Colin Cherry, and William B. Dolan. 2011. Data-driven response generation in social media. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'11). Google Scholar
Digital Library
- Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Now Publishers Inc. Google Scholar
Digital Library
- Alexander I. Rudnicky. 2019. C ha D: Chat-oriented dialog systems. In Advanced Social Interaction with Agents. Springer, 57–60.Google Scholar
- Abdelrhman Saleh, Natasha Jaques, Asma Ghandeharioun, Judy Shen, and Rosalind Picard. 2020. Hierarchical reinforcement learning for open-domain dialog. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 8741–8748.Google Scholar
Cross Ref
- Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL'15).Google Scholar
Cross Ref
- Yuanlong Shao, Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, and Ray Kurzweil. 2017. Generating high-quality and informative conversation responses with sequence-to-sequence models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'17).Google Scholar
Cross Ref
- Yiping Song, Cheng-Te Li, Jian-Yun Nie, Ming Zhang, Dongyan Zhao, and Rui Yan. [n.d.]. An ensemble of retrieval-based and generation-based human-computer conversation systems. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI'18). International Joint Conferences on Artificial Intelligence Organization, 4382–4388. https://doi.org/10.24963/ijcai.2018/609 Google Scholar
Digital Library
- Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. MIT Press. Google Scholar
Digital Library
- Da Tang, Xiujun Li, Jianfeng Gao, Chong Wang, Lihong Li, and Tony Jebara. 2018. Subgoal discovery for hierarchical dialogue policy learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 2298–2309.Google Scholar
Cross Ref
- Jianheng Tang, Tiancheng Zhao, Chengyan Xiong, Xiaodan Liang, Eric P. Xing, and Zhiting Hu. 2019. Target-guided open-domain conversation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL'19).Google Scholar
Cross Ref
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. Google Scholar
Digital Library
- Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks. In Proceedings of the International Conference on Learning Representations (ICLR'18). https://openreview.net/forum?id=rJXMpikCZ.Google Scholar
- Alexandra Vtyurina, Denis Savenkov, Eugene Agichtein, and Charles L. A. Clarke. 2017. Exploring conversational search with humans, assistants, and wizards. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. 2187–2193. Google Scholar
Digital Library
- Richard S. Wallace. 2009. The anatomy of ALICE. In Parsing the Turing Test. Springer, 181–210.Google Scholar
- Jian Wang, Junhao Liu, Wei Bi, Xiaojiang Liu, Kejing He, Ruifeng Xu, and Min Yang. 2020. Improving knowledge-aware dialogue generation via knowledge base question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 9169–9176.Google Scholar
Cross Ref
- Joseph Weizenbaum. 1966. ELIZA—A computer program for the study of natural language communication between man and machine. Commun. ACM (1966). Google Scholar
Digital Library
- Jason D. Williams, Pascal Poupart, and Steve Young. 2008. Partially observable Markov decision processes with continuous observations for dialogue management. In Recent Trends in Discourse and Dialogue. Springer, 191–217.Google Scholar
- Xianchao Wu, Jie Zhou, Yu Sun, Zhanyi Liu, Dianhai Yu, Hua Wu, and Haifeng Wang. 2013. Generalization of words for chinese dependency parsing. In Proceedings of the International Conference on Parsing Technologies (IWPT'13), 73–81.Google Scholar
- Yu Wu, Wei Wu, Chen Xing, Can Xu, Zhoujun Li, and Ming Zhou. 2019. A sequential matching framework for multi-turn response selection in retrieval-based chatbots. Comput. Ling. 45, 1 (2019), 163–197. Google Scholar
Digital Library
- Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li. 2017. Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots. In Proceedings of the AAAI Conference on Artificial Intelligence.Google Scholar
Cross Ref
- Hongcai Xu, Junpeng Bao, and Junqing Wang. 2020. Knowledge-graph based proactive dialogue generation with improved meta-learning. In 2020 2nd International Conference on Image Processing and Machine Vision. 40–46. Google Scholar
Digital Library
- Jun Xu, Zeyang Lei, Haifeng Wang, Zheng-Yu Niu, Hua Wu, and Wanxiang Che. 2020. Enhancing dialog coherence with event graph grounded content planning. In Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI'20), Christian Bessiere (Ed.). International Joint Conferences on Artificial Intelligence Organization, 3941–3947.Google Scholar
- Jun Xu, Haifeng Wang, Zhengyu Niu, Hua Wu, and Wanxiang Che. 2020. Knowledge graph grounded goal planning for open-domain conversation generation. In Proceedings of the 34th AAAI Conference on Artificial Intelligence.Google Scholar
Cross Ref
- Zhen Xu, Bingquan Liu, Baoxun Wang, Chengjie Sun, and Xiaolong Wang. 2017. Incorporating loose-structured knowledge into conversation modeling via recall-gate LSTM. In Proceedings of the International Joint Conference on Neural Networks (IJCNN'17).Google Scholar
Cross Ref
- Rui Yan, Yiping Song, and Hua Wu. 2016. Learning to respond with deep neural networks for retrieval-based human-computer conversation system. In Proceedings of the SIGIR Conference on Research and Development in Information Retrieval (SIGIR'16). Google Scholar
Digital Library
- Lili Yao, Ruijian Xu, Chao Li, Dongyan Zhao, and Rui Yan. 2018. Chat more if you like: Dynamic cue words planning to flow longer conversations. arXiv:1811.07631. Retrieved from https://arxiv.org/abs/1811.07631.Google Scholar
- Tom Young, Erik Cambria, Iti Chaturvedi, Hao Zhou, Subham Biswas, and Minlie Huang. 2018. Augmenting end-to-end dialogue systems with commonsense knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence.Google Scholar
- Zhou Yu, Ziyu Xu, Alan W. Black, and Alexander Rudnicky. 2016. Strategy and policy learning for non-task-oriented conversational systems. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue. 404–412.Google Scholar
Cross Ref
- Jiaping Zhang, Tiancheng Zhao, and Zhou Yu. 2018. Multimodal hierarchical reinforcement learning policy for task-oriented visual dialog. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue. 140–150.Google Scholar
Cross Ref
- Wei-Nan Zhang, Lingzhi Li, Dongyan Cao, and Ting Liu. 2018. Exploring implicit feedback for open domain conversation generation. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence.Google Scholar
- Yongfeng Zhang, Xu Chen, Qingyao Ai, Liu Yang, and W Bruce Croft. 2018. Towards conversational search and recommendation: System ask, user respond. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management. 177–186. Google Scholar
Digital Library
- Tiancheng Zhao, Kaige Xie, and Maxine Eskenazi. 2019. Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 1208–1218.Google Scholar
Cross Ref
- Zilong Zheng, Wenguan Wang, Siyuan Qi, and Song-Chun Zhu. 2019. Reasoning visual dialogs with structural and partial observations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6669–6678.Google Scholar
Cross Ref
- Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. 2017. Emotional chatting machine: Emotional conversation generation with internal and external memory. In Proceedings of the AAAI Conference on Artificial Intelligence.Google Scholar
- Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018. Commonsense knowledge aware conversation generation with graph attention. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI'18). Google Scholar
Digital Library
- Hao Zhou, Chujie Zheng, Kaili Huang, Minlie Huang, and Xiaoyan Zhu. 2020. KdConv: A chinese multi-domain dialogue dataset towards multi-turn knowledge-driven conversation. In Proceedings of the 58th Conference of the Association for Computational Linguistics.Google Scholar
Cross Ref
- Xiangyang Zhou, Lu Li, Daxiang Dong, Yi Liu, Ying Chen, Wayne Xin Zhao, Dianhai Yu, and Hua Wu. 2018. Multi-turn response selection for chatbots with deep attention matching network. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).Google Scholar
Cross Ref
- Wenya Zhu, Kaixiang Mo, Yu Zhang, Zhangbin Zhu, Xuezheng Peng, and Qiang Yang. 2017. Flexible end-to-end dialogue system for knowledge grounded conversation. arXiv preprint arXiv:1709.04264 (2017).Google Scholar
- Yimeng Zhuang, Xianliang Wang, Han Zhang, Jinghui Xie, and Xuan Zhu. 2017. An ensemble approach to conversation generation. In Proceedings of the National CCF Conference on Natural Language Processing and Chinese Computing.Google Scholar
Index Terms
Coherent Dialog Generation with Query Graph
Recommendations
Dialog-to-Actions: Building Task-Oriented Dialogue System via Action-Level Generation
SIGIR '23: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information RetrievalEnd-to-end generation-based approaches have been investigated and applied in task-oriented dialogue systems. However, in industrial scenarios, existing methods face the bottlenecks of reliability (e.g., domain-inconsistent responses, repetition problem, ...
Exploring Mixed-Initiative Dialogue Using Computer Dialogue Simulation
This paper experimentally shows that mixed-initiative dialogue is not always more efficient than non-mixed initiative dialogue in route finding tasks. Based on the dialogue model proposed in Conversation Analysis and Discourse Analysis a lá the ...
Dialog acts in greeting and leavetaking in social talk
ISIAA 2017: Proceedings of the 1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial AgentsConversation proceeds through dialogue moves or acts, and dialog act annotation can aid the design of artificial dialog. While many dialogs are task-based or instrumental, with clear goals, as in the case of a service encounter or business meeting, ...






Comments