skip to main content
research-article

Building Dialogue Understanding Models for Low-resource Language Indonesian from Scratch

Published:06 April 2023Publication History
Skip Abstract Section

Abstract

Using off-the-shelf resources from resource-rich languages to transfer knowledge to low-resource languages has received a lot of attention. The requirements of enabling the model to achieve the reliable performance, including the scale of required annotated data and the effective framework, are not well guided. To address the first question, we empirically investigate the cost-effectiveness of several methods for training intent classification and slot-filling models from scratch in Indonesia (ID) using English data. Confronting the second challenge, we propose a Bi-Confidence-Frequency Cross-Lingual transfer framework (BiCF), which consists of “BiCF Mixing”, “Latent Space Refinement” and “Joint Decoder”, respectively, to overcome the lack of low-resource language dialogue data. BiCF Mixing based on the word-level alignment strategy generates code-mixed data by utilizing the importance-frequency and translating-confidence. Moreover, Latent Space Refinement trains a new dialogue understanding model using code-mixed data and word embedding models. Joint Decoder based on Bidirectional LSTM (BiLSTM) and Conditional Random Field (CRF) is used to obtain experimental results of intent classification and slot-filling. We also release a large-scale fine-labeled Indonesia dialogue dataset (ID-WOZ1) and ID-BERT for experiments. BiCF achieves 93.56% and 85.17% (F1 score) on intent classification and slot filling, respectively. Extensive experiments demonstrate that our framework performs reliably and cost-efficiently on different scales of manually annotated Indonesian data.

REFERENCES

  1. [1] Liu Hui, Yin Qingyu, and Wang William Yang. 2018. Towards explainable NLP: A generative explanation framework for text classification. In Annual Meeting of the Association for Computational Linguistics, 2018.Google ScholarGoogle Scholar
  2. [2] Wu Yu, Wu Wei, Xing Chen, Zhou Ming, and Li Zhoujun. 2016. Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots. In Annual Meeting of the Association for Computational Linguistics, 2016.Google ScholarGoogle Scholar
  3. [3] Grave Edouard, Bojanowski Piotr, Gupta Prakhar, Joulin Armand, and Mikolov Tomas. 2018. Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation.Google ScholarGoogle Scholar
  4. [4] Schuster Sebastian, Gupta Sonal, Shah Rushin, and Lewis Mike. 2018. Cross-lingual transfer learning for multilingual task oriented dialog. In North American Chapter of the Association for Computational Linguistics, 2018.Google ScholarGoogle Scholar
  5. [5] Schuster Tal, Ram Ori, Barzilay Regina, and Globerson Amir. 2019. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In North American Chapter of the Association for Computational Linguistics, 2019.Google ScholarGoogle Scholar
  6. [6] Budzianowski Paweł, Wen Tsung-Hsien, Tseng Bo-Hsiang, Casanueva Inigo, Ultes Stefan, Ramadan Osman, and Gašić Milica. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Conference on Empirical Methods in Natural Language Processing, 2018.Google ScholarGoogle Scholar
  7. [7] Vaswani Ashish, Bengio Samy, Brevdo Eugene, Chollet Francois, Gomez Aidan N., Gouws Stephan, Jones Llion, Kaiser Łukasz, Kalchbrenner Nal, Parmar Niki, Ryan Sepassi, Noam M. Shazeer, and Jakob Uszkoreit. 2018. Tensor2tensor for neural machine translation. In Conference of the Association for Machine Translation in the Americas, 2018.Google ScholarGoogle Scholar
  8. [8] Cheng Yong. 2019. Semi-supervised learning for neural machine translation. In Proceedings of the Joint Training for Neural Machine Translation. Springer, 2540.Google ScholarGoogle ScholarCross RefCross Ref
  9. [9] Devlin Jacob, Chang Ming-Wei, Lee Kenton, and Toutanova Kristina. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805 (2019).Google ScholarGoogle Scholar
  10. [10] Pires Telmo, Schlinger Eva, and Garrette Dan. 2019. How multilingual is multilingual BERT? In Annual Meeting of the Association for Computational Linguistics, 2019.Google ScholarGoogle Scholar
  11. [11] Zhang Meishan, Zhang Yue, and Fu Guohong. 2019. Cross-lingual dependency parsing using code-mixed treebank. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. 9961005.Google ScholarGoogle ScholarCross RefCross Ref
  12. [12] Tiedemann Jörg. 2015. Improving the cross-lingual projection of syntactic dependencies. In Proceedings of the 20th Nordic Conference of Computational Linguistics. Linköping University Electronic, 191199.Google ScholarGoogle Scholar
  13. [13] Tiedemann Jörg and Agić Zeljko. 2016. Synthetic treebanking for cross-lingual dependency parsing. Journal of Artificial Intelligence Research 55 (2016), 209248.Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. [14] Guo Jiang, Che Wanxiang, Yarowsky David, Wang Haifeng, and Liu Ting. 2015. Cross-lingual dependency parsing based on distributed representations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. 12341244.Google ScholarGoogle ScholarCross RefCross Ref
  15. [15] Duong Long, Cohn Trevor, Bird Steven, and Cook Paul. 2015. A neural network model for low-resource universal dependency parsing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. 339348.Google ScholarGoogle ScholarCross RefCross Ref
  16. [16] Ammar Waleed, Mulcaire George, Ballesteros Miguel, Dyer Chris, and Smith Noah A.. 2016. Many languages, one parser. Transactions of the Association for Computational Linguistics 4 (2016), 431444.Google ScholarGoogle ScholarCross RefCross Ref
  17. [17] Wang Hongmin, Zhang Yue, Chan GuangYong Leonard, Yang Jie, and Chieu Hai Leong. 2017. Universal dependencies parsing for colloquial singaporean english. ArXiv, abs/1705.06463 (2017).Google ScholarGoogle Scholar
  18. [18] Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, Gomez Aidan N., Kaiser Łukasz, and Polosukhin Illia. 2017. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems. 59986008.Google ScholarGoogle Scholar
  19. [19] Radford Alec, Narasimhan Karthik, Salimans Tim, and Sutskever Ilya. 2018. Improving language understanding by generative pre-training. (2018).Google ScholarGoogle Scholar
  20. [20] Yang Zhilin, Dai Zihang, Yang Yiming, Carbonell Jaime G., Salakhutdinov Ruslan, and Le Quoc V.. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Neural Information Processing Systems, 2019.Google ScholarGoogle Scholar
  21. [21] Liu Yinhan, Ott Myle, Goyal Naman, Du Jingfei, Joshi Mandar, Chen Danqi, Levy Omer, Lewis Mike, Zettlemoyer Luke, and Stoyanov Veselin. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692 (2019).Google ScholarGoogle Scholar
  22. [22] Raffel Colin, Shazeer Noam, Roberts Adam, Lee Katherine, Narang Sharan, Matena Michael, Zhou Yanqi, Li Wei, and Liu Peter J.. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21, 140 (2020), 167.Google ScholarGoogle Scholar
  23. [23] Liu Yinhan, Gu Jiatao, Goyal Naman, Li Xian, Edunov Sergey, Ghazvininejad Marjan, Lewis Mike, and Zettlemoyer Luke. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics 8 (2020), 726742.Google ScholarGoogle ScholarCross RefCross Ref
  24. [24] Artetxe Mikel, Labaka Gorka, and Agirre Eneko. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. 451462.Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] McCann Bryan, Bradbury James, Xiong Caiming, and Socher Richard. 2017. Learned in translation: Contextualized word vectors. In Proceedings of the Advances in Neural Information Processing Systems. 62946305.Google ScholarGoogle Scholar
  26. [26] Liu Zihan, Shin Jamin, Xu Yan, Winata Genta Indra, Xu Peng, Madotto Andrea, and Fung Pascale. 2019. Zero-shot cross-lingual dialogue systems with transferable latent variables. In Conference on Empirical Methods in Natural Language Processing (2019).Google ScholarGoogle Scholar
  27. [27] Liu Zihan, Winata Genta Indra, Lin Zhaojiang, Xu Peng, and Fung Pascale. 2020. Attention-informed mixed-language training for zero-shot cross-lingual task-oriented dialogue systems. In Proceedings of the AAAI Conference on Artificial Intelligence. 84338440.Google ScholarGoogle ScholarCross RefCross Ref
  28. [28] Xiang Lu, Zhao Yang, Zhu Junnan, Zhou Yu, and Zong Chengqing. 2021. Zero-shot deployment for cross-lingual dialogue system. In Proceedings of the CCF International Conference on Natural Language Processing and Chinese Computing. Springer, 193205.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. [29] Sun Weiwei, Meng Chuan, Meng Qi, Ren Zhaochun, Ren Pengjie, Chen Zhumin, and Rijke Maarten de. 2021. Conversations powered by cross-lingual knowledge. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 14421451.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. [30] Kim San, Jang Jin Yea, Jung Minyoung, and Shin Saim. 2021. A model of cross-lingual knowledge-grounded response generation for open-domain dialogue systems. In Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2021. 352365.Google ScholarGoogle ScholarCross RefCross Ref
  31. [31] Salton Gerard, Fox Edward A., and Wu Harry. 1982. Extended Boolean Information Retrieval. Technical Report. Cornell University.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. [32] Ramos Juan. 2003. Using TF-IDF to determine word relevance in document queries. In Proceedings of the 1st Instructional Conference on Machine Learning. Piscataway, NJ, 133142.Google ScholarGoogle Scholar
  33. [33] Dyer Chris, Chahuneau Victor, and Smith Noah A.. 2013. A simple, fast, and effective reparameterization of ibm model 2. In North American Chapter of the Association for Computational Linguistics, 2013.Google ScholarGoogle Scholar
  34. [34] Collins Michael. 2011. Statistical machine translation: IBM models 1 and 2. Columbia Columbia Univ (2011).Google ScholarGoogle Scholar
  35. [35] Chen Tao, Xu Ruifeng, He Yulan, and Wang Xuan. 2017. Improving sentiment analysis via sentence type classification using BiLSTM-CRF and CNN. Expert Systems with Applications 72 (2017), 221230.Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. [36] Chen Hongshen, Zhang Yue, and Liu Qun. 2016. Neural network for heterogeneous annotations. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 731741.Google ScholarGoogle ScholarCross RefCross Ref
  37. [37] Dozat Timothy and Manning Christopher D.. 2016. Deep biaffine attention for neural dependency parsing. ArXiv, abs/1611.01734 (2016).Google ScholarGoogle Scholar
  38. [38] Chowanda Andry and Chowanda Alan Darmasaputra. 2017. Recurrent neural network to deep learn conversation in indonesian. In International Conference on Computer Science and Computational Intelligence, 2017.Google ScholarGoogle Scholar
  39. [39] Koto Fajri. 2016. A publicly available indonesian corpora for automatic abstractive and extractive chat summarization. In Proceedings of the 10th International Conference on Language Resources and Evaluation. 801805.Google ScholarGoogle Scholar
  40. [40] Tho Cuk, Setiawan Arden S., and Chowanda Andry. 2018. Forming of dyadic conversation dataset for bahasa indonesia. Procedia Computer Science 135 (2018), 315322.Google ScholarGoogle ScholarCross RefCross Ref
  41. [41] Kelley John F.. 1984. An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Information Systems 2, 1 (1984), 2641.Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. [42] Fleiss Joseph L., Cohen Jacob, and Everitt Brian S.. 1969. Large sample standard errors of kappa and weighted kappa. Psychological Bulletin 72, 5 (1969), 323.Google ScholarGoogle ScholarCross RefCross Ref
  43. [43] Wu Chien-Sheng, Madotto Andrea, Hosseini-Asl Ehsan, Xiong Caiming, Socher Richard, and Fung Pascale. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In Annual Meeting of the Association for Computational Linguistics, 2019.Google ScholarGoogle Scholar
  44. [44] Joulin Armand, Grave Edouard, Bojanowski Piotr, Douze Matthijs, Jégou Hérve, and Mikolov Tomas. 2016. FastText.zip: Compressing text classification models. ArXiv, abs/1612.03651 (2016).Google ScholarGoogle Scholar
  45. [45] Melo Gerard de. 2017. Multilingual vector representations of words, sentences, and documents. In Proceedings of the IJCNLP 2017, Tutorial Abstracts. 35.Google ScholarGoogle Scholar
  46. [46] Moghe Nikita, Steedman Mark, and Birch Alexandra. 2021. Cross-lingual intermediate fine-tuning improves dialogue state tracking. In Conference on Empirical Methods in Natural Language Processing, 2021.Google ScholarGoogle Scholar
  47. [47] Lin Yen-Ting and Chen Yun-Nung. 2021. An empirical study of cross-lingual transferability in generative dialogue state tracker. ArXiv, abs/2101.11360 (2021).Google ScholarGoogle Scholar
  48. [48] Gunasekara Chulaka, Kim Seokhwan, D’Haro Luis Fernando, Rastogi Abhinav, Chen Yun-Nung, Eric Mihail, Hedayatnia Behnam, Gopalakrishnan Karthik, Liu Yang, Huang Chao-Wei, Dilek Hakkani-Tür, Jinchao Li, Qi Zhu, Lingxiao Luo, Lars Liden, Kaili Huang, Shahin Shayandeh, Runze Liang, Baolin Peng, Zheng Zhang, Swadheen Shukla, Minlie Huang, Jianfeng Gao, Shikib Mehri, Yulan Feng, Carla Gordon, Seyed Hossein Alavi, David R. Traum, Maxine Eskénazi, Ahmad Beirami, Eunjoon Cho, Paul A. Crook, Ankita De, Alborz Geramifard, Satwik Kottur, Seungwhan Moon, Shivani Poddar, and Rajen Subba. 2020. Overview of the ninth dialog system technology challenge: Dstc9. ArXiv, abs/2011.06486 (2020).Google ScholarGoogle Scholar
  49. [49] Feng Yue, Wang Yang, and Li Hang. 2020. A sequence-to-sequence approach to dialogue state tracking. In Annual Meeting of the Association for Computational Linguistics, 2020.Google ScholarGoogle Scholar
  50. [50] Guo Jinyu, Shuang Kai, Li Jijie, and Wang Zihan. 2021. Dual slot selector via local reliability verification for dialogue state tracking. ArXiv, abs/2107.12578 (2021).Google ScholarGoogle Scholar
  51. [51] Guo Jinyu, Shuang Kai, Li Jijie, Wang Zihan, and Liu Yixuan. 2022. Beyond the granularity: Multi-perspective dialogue collaborative selection for dialogue state tracking. ArXiv, abs/2205.10059 (2022).Google ScholarGoogle Scholar
  52. [52] Papineni Kishore, Roukos Salim, Ward Todd, and Zhu Wei-Jing. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, 311318.Google ScholarGoogle Scholar

Index Terms

  1. Building Dialogue Understanding Models for Low-resource Language Indonesian from Scratch

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Asian and Low-Resource Language Information Processing
        ACM Transactions on Asian and Low-Resource Language Information Processing  Volume 22, Issue 4
        April 2023
        682 pages
        ISSN:2375-4699
        EISSN:2375-4702
        DOI:10.1145/3588902
        Issue’s Table of Contents

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 6 April 2023
        • Online AM: 15 December 2022
        • Accepted: 27 November 2022
        • Revised: 18 July 2022
        • Received: 9 February 2022
        Published in tallip Volume 22, Issue 4

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
      • Article Metrics

        • Downloads (Last 12 months)135
        • Downloads (Last 6 weeks)11

        Other Metrics

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      View Full Text

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!