skip to main content
research-article

Response Generation via Structure-Aware Constraints

Published:16 December 2022Publication History
Skip Abstract Section

Abstract

End-to-end neural modeling with the encoder-decoder architecture has shown great promise in response generation. However, it often generates dull and generic responses due to its failure to effectively perceive various kinds of act, sentiment, and topic information. To address these challenges, we propose a response-generation model with structure-aware constraints to capture the structure of dialog and generate a better response with various constraints of the act, sentiment, and topic. In particular, given an utterance sequence, we first learn the representation of each utterance in the encoding stage. We then learn the turn, speaker, and dialog representation from the utterance representations and construct the structure of dialog. Third, we employ an attention mechanism to extract the constraints of act, sentiment, and topic based on the structure of the dialog. Finally, we utilize these structure-aware constraints to control the response-generation process in decoding stage. Extensive experimental results validate the superiority of our proposed model against the state-of-the-art baselines. In addition, the results also show that the proposed model can generate responses with more appropriate content based on the structure-aware constraints.

REFERENCES

  1. [1] Bahdanau Dzmitry, Cho Kyunghyun, and Bengio Yoshua. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations. 1–15.Google ScholarGoogle Scholar
  2. [2] Bao Siqi, He Huang, Wang Fan, Wu Hua, and Wang Haifeng. 2020. PLATO: Pre-trained dialogue generation model with discrete latent variable. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Jurafsky Dan, Chai Joyce, Schluter Natalie, and Tetreault Joel R. (Eds.). Association for Computational Linguistics, 8596.Google ScholarGoogle ScholarCross RefCross Ref
  3. [3] Budzianowski Pawel, Wen Tsung-Hsien, Tseng Bo-Hsiang, Casanueva Iñigo, Ultes Stefan, Ramadan Osman, and Gasic Milica. 2018. MultiWOZ—A large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Riloff Ellen, Chiang David, Hockenmaier Julia, and Tsujii Jun’ichi (Eds.). Association for Computational Linguistics, 50165026.Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Chen Hongshen, Ren Zhaochun, Tang Jiliang, Zhao Yihong Eric, and Yin Dawei. 2018. Hierarchical variational memory network for dialogue generation. In Proceedings of the 2018 World Wide Web Conference on World Wide Web. 16531662.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. [5] Chen Wenhu, Chen Jianshu, Qin Pengda, Yan Xifeng, and Wang William Yang. 2019. Semantically conditioned dialog response generation via hierarchical disentangled self-attention. In Proceedings of the 57th Conference of the Association for Computational Linguistics, Korhonen Anna, Traum David R., and Màrquez Lluís (Eds.). Association for Computational Linguistics, 36963709.Google ScholarGoogle ScholarCross RefCross Ref
  6. [6] Chen Wenhu, Chen Jianshu, Su Yu, Wang Xin, Yu Dong, Yan Xifeng, and Wang William Yang. 2018. XL-NBT: A cross-lingual neural belief tracking framework. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Riloff Ellen, Chiang David, Hockenmaier Julia, and Tsujii Jun’ichi (Eds.). Association for Computational Linguistics, 414424.Google ScholarGoogle ScholarCross RefCross Ref
  7. [7] Chen Xiuyi, Xu Jiaming, and Xu Bo. 2019. A working memory model for task-oriented dialog response generation. In Proceedings of the 57th Conference of the Association for Computational Linguistics. 26872693.Google ScholarGoogle ScholarCross RefCross Ref
  8. [8] Choi Byung-Ju, Hong Jimin, Park David Keetae, and Lee Sang Wan. 2020. F-softmax: Diversifying neural text generation via frequency factorized softmax. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 91679182.Google ScholarGoogle ScholarCross RefCross Ref
  9. [9] Galetzka Fabian, Eneh Chukwuemeka Uchenna, and Schlangen David. 2020. A corpus of controlled opinionated and knowledgeable movie discussions for training neural conversation models. In Proceedings of the 12th Language Resources and Evaluation Conference. 565573.Google ScholarGoogle Scholar
  10. [10] Galetzka Fabian, Rose Jewgeni, Schlangen David, and Lehmann Jens. 2021. Space efficient context encoding for non-task-oriented dialogue generation with graph attention transformer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 70287041.Google ScholarGoogle ScholarCross RefCross Ref
  11. [11] Gao Silin, Zhang Yichi, Ou Zhijian, and Yu Zhou. 2020. Paraphrase augmented task-oriented dialog generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Jurafsky Dan, Chai Joyce, Schluter Natalie, and Tetreault Joel R. (Eds.). Association for Computational Linguistics, 639649.Google ScholarGoogle ScholarCross RefCross Ref
  12. [12] Goo Chih-Wen, Gao Guang, Hsu Yun-Kai, Huo Chih-Li, Chen Tsung-Chieh, Hsu Keng-Wei, and Chen Yun-Nung. 2018. Slot-gated modeling for joint slot filling and intent prediction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Walker Marilyn A., Ji Heng, and Stent Amanda (Eds.). Association for Computational Linguistics, 753757.Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] Gupta Prakhar, Bigham Jeffrey P., Tsvetkov Yulia, and Pavel Amy. 2020. Controlling dialogue generation with semantic exemplars. arXiv preprint arXiv:2008.09075 (2020).Google ScholarGoogle Scholar
  14. [14] He He, Balakrishnan Anusha, Eric Mihail, and Liang Percy. 2017. Learning symmetric collaborative dialogue agents with dynamic knowledge graph embeddings. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. 17661776.Google ScholarGoogle ScholarCross RefCross Ref
  15. [15] Holtzman Ari, Buys Jan, Du Li, Forbes Maxwell, and Choi Yejin. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations.Google ScholarGoogle Scholar
  16. [16] Huang Minlie, Zhu Xiaoyan, and Gao Jianfeng. 2020. Challenges in building intelligent open-domain dialog systems. ACM Transactions on Information Systems 38, 3 (2020), 21:1–21:32.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. [17] Kingma Diederik P. and Ba Jimmy. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations.Google ScholarGoogle Scholar
  18. [18] Lei Wenqiang, Jin Xisen, Kan Min-Yen, Ren Zhaochun, He Xiangnan, and Yin Dawei. 2018. Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Gurevych Iryna and Miyao Yusuke (Eds.). Association for Computational Linguistics, 14371447.Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Li Jiwei, Galley Michel, Brockett Chris, Gao Jianfeng, and Dolan Bill. 2016. A diversity-promoting objective function for neural conversation models. In the 2016 Conference of the North American Chapter of the Association for Computational Linguistics, Knight Kevin, Nenkova Ani, and Rambow Owen (Eds.). Association for Computational Linguistics, 110119.Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Li Jiwei, Monroe Will, Shi Tianlin, Jean Sébastien, Ritter Alan, and Jurafsky Dan. 2017. Adversarial learning for neural dialogue generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 21572169.Google ScholarGoogle ScholarCross RefCross Ref
  21. [21] Li Lu, Li Chenliang, and Ji Donghong. 2021. Deep context modeling for multi-turn response selection in dialogue systems. Information Processing and Management 58, 1 (2021), 102415.Google ScholarGoogle ScholarCross RefCross Ref
  22. [22] Li Yanran, Su Hui, Shen Xiaoyu, Li Wenjie, Cao Ziqiang, and Niu Shuzi. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the 8th International Joint Conference on Natural Language Processing. 986995.Google ScholarGoogle Scholar
  23. [23] Lin F., Zhang C., and Liu S.. 2020. A hierarchical structured multi-head attention network for multi-turn response generation. IEEE Access 8 (2020), 4680246810.Google ScholarGoogle ScholarCross RefCross Ref
  24. [24] Lin Zibo, Cai Deng, Wang Yan, Liu Xiaojiang, Zheng Haitao, and Shi Shuming. 2020. The world is not binary: Learning to rank with grayscale data for dialogue response selection. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 92209229.Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Ling Yanxiang, Cai Fei, Hu Xuejun, Liu Jun, Chen Wanyu, and Chen Honghui. 2021. Context-controlled topic-aware neural response generation for open-domain dialog systems. Information Processing and Management 58, 1 (2021), 102392.Google ScholarGoogle ScholarCross RefCross Ref
  26. [26] Liu Haochen, Wang Wentao, Wang Yiqi, Liu Hui, Liu Zitao, and Tang Jiliang. 2020. Mitigating gender bias for neural dialogue generation with adversarial learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Webber Bonnie, Cohn Trevor, He Yulan, and Liu Yang (Eds.). Association for Computational Linguistics, 893903.Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Liu Qian, Chen Yihong, Chen Bei, Lou Jian-Guang, Chen Zixuan, Zhou Bin, and Zhang Dongmei. 2020. You impress me: Dialogue generation via mutual persona perception. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Jurafsky Dan, Chai Joyce, Schluter Natalie, and Tetreault Joel R. (Eds.). Association for Computational Linguistics, 14171427.Google ScholarGoogle ScholarCross RefCross Ref
  28. [28] Meng Chuan, Ren Pengjie, Chen Zhumin, Monz Christof, Ma Jun, and Rijke Maarten de. 2019. RefNet: A reference-aware network for background based conversation. arXiv preprint arXiv:1908.06449 (2019).Google ScholarGoogle Scholar
  29. [29] Mrksic Nikola, Séaghdha Diarmuid Ó, Wen Tsung-Hsien, Thomson Blaise, and Young Steve J.. 2017. Neural belief tracker: Data-driven dialogue state tracking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Barzilay Regina and Kan Min-Yen (Eds.). Association for Computational Linguistics, 17771788.Google ScholarGoogle Scholar
  30. [30] Papineni Kishore, Roukos Salim, Ward Todd, and Zhu Wei-Jing. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. 311318.Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. [31] Ritter Alan, Cherry Colin, and Dolan William B.. 2011. Data-driven response generation in social media. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 583593.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. [32] Roller Stephen, Dinan Emily, Goyal Naman, Ju Da, Williamson Mary, Liu Yinhan, Xu Jing, Ott Myle, Smith Eric Michael, Boureau Y-Lan, and Weston Jason. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics. 300325.Google ScholarGoogle ScholarCross RefCross Ref
  33. [33] Serban Iulian Vlad, Sordoni Alessandro, Lowe Ryan, Charlin Laurent, Pineau Joelle, Courville Aaron C., and Bengio Yoshua. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In Proceedings of the 31st AAAI Conference on Artificial Intelligence. 32953301.Google ScholarGoogle ScholarCross RefCross Ref
  34. [34] Shang Lifeng, Lu Zhengdong, and Li Hang. 2015. Neural responding machine for short-text conversation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 15771586.Google ScholarGoogle ScholarCross RefCross Ref
  35. [35] Shen Siqi, Welch Charles, Mihalcea Rada, and Pérez-Rosas Verónica. 2020. Counseling-style reflection generation using generative pretrained transformers with augmented context. In Proceedings of the 21st Annual Meeting of the Special Interest Group on Discourse and Dialogue. 1020.Google ScholarGoogle Scholar
  36. [36] Shi Yangyang, Yao Kaisheng, Tian Le, and Jiang Daxin. 2016. Deep LSTM based feature mapping for query classification. In 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Knight Kevin, Nenkova Ani, and Rambow Owen (Eds.). Association for Computational Linguistics, 15011511.Google ScholarGoogle ScholarCross RefCross Ref
  37. [37] Song Haoyu, Wang Yan, Zhang Kaiyan, Zhang Wei-Nan, and Liu Ting. 2021. BoB: BERT over BERT for training persona-based dialogue models from limited personalized data. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 167177.Google ScholarGoogle ScholarCross RefCross Ref
  38. [38] Song Haoyu, Wang Yan, Zhang Weinan, Liu Xiaojiang, and Liu Ting. 2020. Generate, delete and rewrite: A three-stage framework for improving persona consistency of dialogue generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 58215831.Google ScholarGoogle ScholarCross RefCross Ref
  39. [39] Su Yixuan, Cai Deng, Zhou Qingyu, Lin Zibo, Baker Simon, Cao Yunbo, Shi Shuming, Collier Nigel, and Wang Yan. 2021. Dialogue response selection with hierarchical curriculum learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 17401751.Google ScholarGoogle ScholarCross RefCross Ref
  40. [40] Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, Gomez Aidan N., Kaiser Lukasz, and Polosukhin Illia. 2017. Attention is all you need. In Proceedings of NIPS. 59986008.Google ScholarGoogle Scholar
  41. [41] Wen Tsung-Hsien, Miao Yishu, Blunsom Phil, and Young Steve J.. 2017. Latent intention dialogue models. In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research), Precup Doina and Teh Yee Whye (Eds.), Vol. 70. PMLR, 37323741.Google ScholarGoogle Scholar
  42. [42] Wen Tsung-Hsien, Vandyke David, Mrksic Nikola, Gasic Milica, Rojas-Barahona Lina Maria, Su Pei-Hao, Ultes Stefan, and Young Steve J.. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, Lapata Mirella, Blunsom Phil, and Koller Alexander (Eds.). Association for Computational Linguistics, 438449.Google ScholarGoogle ScholarCross RefCross Ref
  43. [43] Williams Jason D., Raux Antoine, and Henderson Matthew. 2016. The dialog state tracking challenge series: A review. D&D 7, 3 (2016), 433.Google ScholarGoogle ScholarCross RefCross Ref
  44. [44] Williams Jason D., Raux Antoine, Ramachandran Deepak, and Black Alan W.. 2013. The dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference. Association for Computer Linguistics, 404413.Google ScholarGoogle Scholar
  45. [45] Wu Chien-Sheng, Hoi Steven C. H., Socher Richard, and Xiong Caiming. 2020. TOD-BERT: Pre-trained natural language understanding for task-oriented dialogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Webber Bonnie, Cohn Trevor, He Yulan, and Liu Yang (Eds.). Association for Computational Linguistics, 917929.Google ScholarGoogle ScholarCross RefCross Ref
  46. [46] Xing Chen, Wu Yu, Wu Wei, Huang Yalou, and Zhou Ming. 2018. Hierarchical recurrent attention network for response generation. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, McIlraith Sheila A. and Weinberger Kilian Q. (Eds.). AAAI Press, 56105617.Google ScholarGoogle ScholarCross RefCross Ref
  47. [47] Xu Zhen, Liu Bingquan, Wang Baoxun, Sun Chengjie, Wang Xiaolong, Wang Zhuoran, and Qi Chao. 2017. Neural response generation via GAN with an approximate embedding layer. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 617626.Google ScholarGoogle ScholarCross RefCross Ref
  48. [48] Yang Ze, Wu Wei, Xu Can, Liang Xinnian, Bai Jiaqi, Wang Liran, Wang Wei, and Li Zhoujun. 2020. StyleDGPT: Stylized response generation with pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 15481559.Google ScholarGoogle ScholarCross RefCross Ref
  49. [49] You Yongjian, Jia Weijia, Liu Tianyi, and Yang Wenmian. 2019. Improving abstractive document summarization with salient information modeling. In Proceedings of the 57th Conference of the Association for Computational Linguistics. 21322141.Google ScholarGoogle ScholarCross RefCross Ref
  50. [50] Zeng Min, Wang Yisen, and Luo Yuan. 2019. Dirichlet latent variable hierarchical recurrent encoder-decoder in dialogue generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. 12671272.Google ScholarGoogle ScholarCross RefCross Ref
  51. [51] Zhang Bo and Zhang Xiaoming. 2019. Hierarchy response learning for neural conversation generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. 17721781.Google ScholarGoogle ScholarCross RefCross Ref
  52. [52] Zhang Hainan, Lan Yanyan, Pang Liang, Guo Jiafeng, and Cheng Xueqi. 2019. ReCoSa: Detecting the relevant contexts with self-attention for multi-turn dialogue generation. In Proceedings of the 57th Conference of the Association for Computational Linguistics. 37213730.Google ScholarGoogle ScholarCross RefCross Ref
  53. [53] Zhang Weisheng, Song Kaisong, Kang Yangyang, Wang Zhongqing, Sun Changlong, Liu Xiaozhong, Li Shoushan, Zhang Min, and Si Luo. 2020. Multi-turn dialogue generation in e-commerce platform with the context of historical dialogue. In EMNLP 2020, Cohn Trevor, He Yulan, and Liu Yang (Eds.). 19811990.Google ScholarGoogle Scholar
  54. [54] Zhang Yizhe, Sun Siqi, Galley Michel, Chen Yen-Chun, Brockett Chris, Gao Xiang, Gao Jianfeng, Liu Jingjing, and Dolan Bill. 2020. DIALOGPT: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. 270278.Google ScholarGoogle ScholarCross RefCross Ref
  55. [55] Zhao Xueliang, Wu Wei, Tao Chongyang, Xu Can, Zhao Dongyan, and Yan Rui. 2020. Low-resource knowledge-grounded dialogue generation. arXiv preprint arXiv:2002.10348 (2020).Google ScholarGoogle Scholar
  56. [56] Zhu Qingfu, Zhang Wei-Nan, Liu Ting, and Wang William Yang. 2021. Neural stylistic response generation with disentangled latent variables. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 43914401.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Response Generation via Structure-Aware Constraints

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Asian and Low-Resource Language Information Processing
      ACM Transactions on Asian and Low-Resource Language Information Processing  Volume 21, Issue 6
      November 2022
      372 pages
      ISSN:2375-4699
      EISSN:2375-4702
      DOI:10.1145/3568970
      Issue’s Table of Contents

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 16 December 2022
      • Online AM: 26 March 2022
      • Accepted: 12 March 2022
      • Received: 26 September 2021
      Published in tallip Volume 21, Issue 6

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Refereed
    • Article Metrics

      • Downloads (Last 12 months)90
      • Downloads (Last 6 weeks)9

      Other Metrics

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    View Full Text

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!