skip to main content
research-article

Multi-domain Spoken Language Understanding Using Domain- and Task-aware Parameterization

Authors Info & Claims
Published:20 January 2022Publication History
Skip Abstract Section

Abstract

Spoken language understanding (SLU) has been addressed as a supervised learning problem, where a set of training data is available for each domain. However, annotating data for a new domain can be both financially costly and non-scalable. One existing approach solves the problem by conducting multi-domain learning where parameters are shared for joint training across domains, which is domain-agnostic and task-agnostic. In the article, we propose to improve the parameterization of this method by using domain-specific and task-specific model parameters for fine-grained knowledge representation and transfer. Experiments on five domains show that our model is more effective for multi-domain SLU and obtain the best results. In addition, we show its transferability when adapting to a new domain with little data, outperforming the prior best model by 12.4%. Finally, we explore the strong pre-trained model in our framework and find that the contributions from our framework do not fully overlap with contextualized word representations (RoBERTa).

REFERENCES

  1. [1] Coucke Alice, Saade Alaa, Ball Adrien, Bluche Théodore, Caulier Alexandre, Leroy David, Doumouro Clément, Gisselbrecht Thibault, Caltagirone Francesco, Lavril Thibaut et al. 2018. Snips voice platform: An embedded spoken language understanding system for private-by-design voice interfaces. arXiv preprint arXiv:1805.10190 (2018).Google ScholarGoogle Scholar
  2. [2] Cao Nicola De, Aziz Wilker, and Titov Ivan. 2019. Question answering by reasoning across documents with graph convolutional networks. In Proceedings of the NAACL. Association for Computational Linguistics, 23062317. DOI: https://doi.org/10.18653/v1/N19-1240Google ScholarGoogle Scholar
  3. [3] E Haihong. Niu Peiqing, Chen Zhongfu, and Song Meina. 2019. A novel bi-directional interrelated model for joint intent detection and slot filling. In Proceedings of the ACL. Association for Computational Linguistics, 54675471. DOI: https://doi.org/10.18653/v1/P19-1544Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Goo Chih-Wen, Gao Guang, Hsu Yun-Kai, Huo Chih-Li, Chen Tsung-Chieh, Hsu Keng-Wei, and Chen Yun-Nung. 2018. Slot-gated modeling for joint slot filling and intent prediction. In Proceedings of the NAACL. Association for Computational Linguistics, 753757. DOI: https://doi.org/10.18653/v1/N18-2118Google ScholarGoogle ScholarCross RefCross Ref
  5. [5] Guo Zhijiang, Zhang Yan, and Lu Wei. 2019. Attention guided graph convolutional networks for relation extraction. In Proceedings of the ACL. Association for Computational Linguistics, 241251. DOI: https://doi.org/10.18653/v1/P19-1024Google ScholarGoogle ScholarCross RefCross Ref
  6. [6] Guo Zhijiang, Zhang Yan, Teng Zhiyang, and Lu Wei. 2019. Densely connected graph convolutional networks for graph-to-sequence learning. Trans. Assoc. Computat. Ling. 7 (Mar. 2019), 297312. DOI: https://doi.org/10.1162/tacl_a_00269Google ScholarGoogle ScholarCross RefCross Ref
  7. [7] Gupta Raghav, Rastogi Abhinav, and Hakkani-Tur Dilek. 2018. An efficient approach to encoding context for spoken language understanding. arXiv preprint arXiv:1807.00267 (2018).Google ScholarGoogle Scholar
  8. [8] Haffner Patrick, Tur Gokhan, and Wright Jerry H.. 2003. Optimizing SVMs for complex call classification. In Proceedings of the ICASSP.Google ScholarGoogle ScholarCross RefCross Ref
  9. [9] Hakkani-Tür Dilek, Tür Gökhan, Celikyilmaz Asli, Chen Yun-Nung, Gao Jianfeng, Deng Li, and Wang Ye-Yi. 2016. Multi-domain joint semantic frame parsing using bi-directional RNN-LSTM. In Proceedings of the Interspeech.Google ScholarGoogle ScholarCross RefCross Ref
  10. [10] Hemphill Charles T., Godfrey John J., and Doddington George R.. 1990. The ATIS spoken language systems pilot corpus. In Proceedings of the Workshop on Speech and Natural Language. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. [11] Hochreiter Sepp and Schmidhuber Jürgen. 1997. Long short-term memory. Neural Computat. 9, 8 (1997). Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. [12] Huang Binxuan and Carley Kathleen. 2019. Syntax-aware aspect level sentiment classification with graph attention networks. In Proceedings of the EMNLP.Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] Kim Young-Bum, Lee Sungjin, and Stratos Karl. 2017. OneNet: Joint domain, intent, slot prediction for spoken language understanding. In Proceedings of the ASRU. IEEE, 547553.Google ScholarGoogle ScholarCross RefCross Ref
  14. [14] Kingma Diederik P. and Ba Jimmy. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google ScholarGoogle Scholar
  15. [15] Kipf Thomas N. and Welling Max. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016).Google ScholarGoogle Scholar
  16. [16] Lee Jihwan, Sarikaya Ruhi, and Kim Young-Bum. 2019. Locale-agnostic universal domain classification model in spoken language understanding. In Proceedings of the NAACL. Association for Computational Linguistics.Google ScholarGoogle ScholarCross RefCross Ref
  17. [17] Li Changliang, Li Liang, and Qi Ji. 2018. A self-attentive model with gate mechanism for spoken language understanding. In Proceedings of the EMNLP. Association for Computational Linguistics, 38243833. DOI: https://doi.org/10.18653/v1/D18-1417Google ScholarGoogle ScholarCross RefCross Ref
  18. [18] Lin Bill Yuchen, Chen Xinyue, Chen Jamin, and Ren Xiang. 2019. KagNet: Knowledge-aware graph networks for commonsense reasoning. In Proceedings of the EMNLP. Association for Computational Linguistics, 28292839. DOI: https://doi.org/10.18653/v1/D19-1282Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Liu Bing and Lane Ian. 2017. Multi-domain adversarial learning for slot filling in spoken language understanding. arXiv preprint arXiv:1711.11310 (2017).Google ScholarGoogle Scholar
  20. [20] Liu Pengfei, Qiu Xipeng, and Huang Xuanjing. 2017. Adversarial multi-task learning for text classification. In Proceedings of the ACL.Google ScholarGoogle ScholarCross RefCross Ref
  21. [21] Liu Yijin, Meng Fandong, Zhang Jinchao, Zhou Jie, Chen Yufeng, and Xu Jinan. 2019. CM-Net: A novel collaborative memory network for spoken language understanding. In Proceedings of the EMNLP. Association for Computational Linguistics, 10511060. DOI: https://doi.org/10.18653/v1/D19-1097Google ScholarGoogle ScholarCross RefCross Ref
  22. [22] Liu Yinhan, Ott Myle, Goyal Naman, Du Jingfei, Joshi Mandar, Chen Danqi, Levy Omer, Lewis Mike, Zettlemoyer Luke, and Stoyanov Veselin. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019).Google ScholarGoogle Scholar
  23. [23] Liu Zihan, Winata Genta Indra, Xu Peng, and Fung Pascale. 2020. Coach: A coarse-to-fine approach for cross-domain slot filling. arxiv:cs.CL/2004.11727 (2020).Google ScholarGoogle Scholar
  24. [24] Manning Christopher, Surdeanu Mihai, Bauer John, Finkel Jenny, Bethard Steven, and McClosky David. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of the ACL.Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Marcheggiani Diego and Titov Ivan. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the EMNLP. Association for Computational Linguistics, 15061515. DOI: https://doi.org/10.18653/v1/D17-1159Google ScholarGoogle ScholarCross RefCross Ref
  26. [26] Qin Libo, Che Wanxiang, Li Yangming, Ni Minheng, and Liu Ting. 2020. DCR-Net: A deep co-interactive relation network for joint dialog act recognition and sentiment classification. In AAAI. 86658672.Google ScholarGoogle Scholar
  27. [27] Qin Libo, Che Wanxiang, Li Yangming, Wen Haoyang, and Liu Ting. 2019. A stack-propagation framework with token-level intent detection for spoken language understanding. In Proceedings of the EMNLP. Association for Computational Linguistics, 20782087. DOI: https://doi.org/10.18653/v1/D19-1214Google ScholarGoogle ScholarCross RefCross Ref
  28. [28] Qin Libo, Liu Tailu, Che Wanxiang, Kang Bingbing, Zhao Sendong, and Liu Ting. 2020. A co-interactive transformer for joint slot filling and intent detection. arXiv preprint arXiv:2010.03880 (2020).Google ScholarGoogle Scholar
  29. [29] Qin Libo, Ni Minheng, Zhang Yue, and Che Wanxiang. 2020. CoSDA-ML: Multi-lingual code-switching data augmentation for zero-shot cross-lingual NLP. arxiv:cs.CL/2006.06402 (2020). Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. [30] Qin Libo, Wei Fuxuan, Xie Tianbao, Xu Xiao, Che Wanxiang, and Liu Ting. 2021. GL-GIN: Fast and accurate non-autoregressive model for joint multiple intent detection and slot filling. In Proceedings of the ACL.Google ScholarGoogle ScholarCross RefCross Ref
  31. [31] Qin Libo, Xie Tianbao, Che Wanxiang, and Liu Ting. 2021. A survey on spoken language understanding: Recent advances and new frontiers. arxiv:cs.CL/2103.03095 (2021).Google ScholarGoogle Scholar
  32. [32] Qin Libo, Xu Xiao, Che Wanxiang, and Liu Ting. 2020. AGIF: An adaptive graph-interactive framework for joint multiple intent detection and slot filling. In Proceedings of the EMNLP. Association for Computational Linguistics, 18071816. DOI: https://doi.org/10.18653/v1/2020.findings-emnlp.163Google ScholarGoogle ScholarCross RefCross Ref
  33. [33] Raymond Christian and Riccardi Giuseppe. 2007. Generative and discriminative algorithms for spoken language understanding. In Proceedings of the Interspeech.Google ScholarGoogle ScholarCross RefCross Ref
  34. [34] Sarikaya Ruhi, Hinton Geoffrey E., and Ramabhadran Bhuvana. 2011. Deep belief nets for natural language call-routing. In Proceedings of the ICASSP.Google ScholarGoogle ScholarCross RefCross Ref
  35. [35] Schuster Sebastian, Gupta Sonal, Shah Rushin, and Lewis Mike. 2019. Cross-lingual transfer learning for multilingual task oriented dialog. In Proceedings of the NAACL. Association for Computational Linguistics, 37953805. DOI: https://doi.org/10.18653/v1/N19-1380Google ScholarGoogle ScholarCross RefCross Ref
  36. [36] Shen Tao, Zhou Tianyi, Long Guodong, Jiang Jing, Pan Shirui, and Zhang Chengqi. 2018. DiSAN: Directional self-attention network for RNN/CNN-free language understanding. In Proceedings of the AAAI. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. [37] Strubell Emma, Verga Patrick, Andor Daniel, Weiss David, and McCallum Andrew. 2018. Linguistically informed self-attention for semantic role labeling. arXiv preprint arXiv:1804.08199 (2018).Google ScholarGoogle Scholar
  38. [38] Tan Zhixing, Wang Mingxuan, Xie Jun, Chen Yidong, and Shi Xiaodong. 2018. Deep semantic role labeling with self-attention. In Proceedings of the AAAI. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. [39] Teng Dechuang, Qin Libo, Che Wanxiang, Zhao Sendong, and Liu Ting. 2021. Injecting word information with multi-level word adapter for Chinese spoken language understanding. arxiv:cs.CL/2010.03903 (2021).Google ScholarGoogle Scholar
  40. [40] Tur Gokhan and Mori Renato De. 2011. Spoken Language Understanding: Systems for Extracting Semantic Information from Speech. John Wiley & Sons.Google ScholarGoogle ScholarCross RefCross Ref
  41. [41] Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, Gomez Aidan N., Łukasz Kaiser, and Polosukhin Illia. 2017. Attention is all you need. In Proceedings of the NIPS. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. [42] Wang Yu, Shen Yilin, and Jin Hongxia. 2018. A bi-model based RNN semantic frame parsing model for intent detection and slot filling. In Proceedings of the ACL.Google ScholarGoogle ScholarCross RefCross Ref
  43. [43] Xia Congying, Zhang Chenwei, Yan Xiaohui, Chang Yi, and Yu Philip. 2018. Zero-shot user intent detection via capsule neural networks. In Proceedings of the EMNLP. Association for Computational Linguistics, 30903099. DOI: https://doi.org/10.18653/v1/D18-1348Google ScholarGoogle ScholarCross RefCross Ref
  44. [44] Xu Puyang and Sarikaya Ruhi. 2013. Convolutional neural network based triangular CRF for joint intent detection and slot filling. In Proceedings of the ASRU.Google ScholarGoogle ScholarCross RefCross Ref
  45. [45] Xu Puyang and Sarikaya Ruhi. 2013. Exploiting shared information for multi-intent natural language sentence classification. In Proceedings of the Interspeech.Google ScholarGoogle ScholarCross RefCross Ref
  46. [46] Yang Zhilin, Salakhutdinov Ruslan, and Cohen William. 2016. Multi-task cross-lingual sequence tagging from scratch. arXiv preprint arXiv:1603.06270 (2016).Google ScholarGoogle Scholar
  47. [47] Yao Kaisheng, Peng Baolin, Zhang Yu, Yu Dong, Zweig Geoffrey, and Shi Yangyang. 2014. Spoken language understanding using long short-term memory neural networks. In Proceedings of the SLT.Google ScholarGoogle ScholarCross RefCross Ref
  48. [48] Young Steve, Gašić Milica, Thomson Blaise, and Williams Jason D.. 2013. POMDP-based statistical spoken dialog systems: A review. In Proceedings of the IEEE. DOI: 10.1109/JPROC.2012.2225812Google ScholarGoogle Scholar
  49. [49] Zhang Chen, Li Qiuchi, and Song Dawei. 2019. Aspect-based sentiment classification with aspect-specific graph convolutional networks. In Proceedings of the EMNLP.Google ScholarGoogle ScholarCross RefCross Ref
  50. [50] Zhang Chenwei, Li Yaliang, Du Nan, Fan Wei, and Yu Philip. 2019. Joint slot filling and intent detection via capsule neural networks. In Proceedings of the ACL.Google ScholarGoogle ScholarCross RefCross Ref
  51. [51] Zhang Xiaodong and Wang Houfeng. 2016. A joint model of intent determination and slot filling for spoken language understanding. In Proceedings of the IJCAI. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. [52] Zhang Yuhao, Qi Peng, and Manning Christopher D.. 2018. Graph convolution over pruned dependency trees improves relation extraction. In Proceedings of the EMNLP.Google ScholarGoogle ScholarCross RefCross Ref
  53. [53] Zhong Victor, Xiong Caiming, and Socher Richard. 2018. Global-locally self-attentive encoder for dialogue state tracking. In Proceedings of the ACL. Association for Computational Linguistics, 14581467. DOI: https://doi.org/10.18653/v1/P18-1135Google ScholarGoogle ScholarCross RefCross Ref
  54. [54] Zhong Victor, Xiong Caiming, and Socher Richard. 2018. Global-locally self-attentive encoder for dialogue state tracking. In Proceedings of the ACL.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Multi-domain Spoken Language Understanding Using Domain- and Task-aware Parameterization

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Asian and Low-Resource Language Information Processing
      ACM Transactions on Asian and Low-Resource Language Information Processing  Volume 21, Issue 4
      July 2022
      464 pages
      ISSN:2375-4699
      EISSN:2375-4702
      DOI:10.1145/3511099
      Issue’s Table of Contents

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 20 January 2022
      • Accepted: 1 November 2021
      • Revised: 1 September 2021
      • Received: 1 April 2020
      Published in tallip Volume 21, Issue 4

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Refereed
    • Article Metrics

      • Downloads (Last 12 months)172
      • Downloads (Last 6 weeks)3

      Other Metrics

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    View Full Text

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!