Abstract
Dialogue generation is the automatic generation of a text response, given a user’s input. Dialogue generation for low-resource languages has been a challenging tasks for researchers. However, the advancements in deep learning models have made developing conversational agents that perform the tasks of dialogue generation not only possible, but also effective and helpful in many applications spanning a variety of domains. Nevertheless, work on conversational bots for low-resource languages such as the Arabic language is still limited due to various challenges, including the language structure, vocabulary, and the scarcity of its data resources. Meta-learning has been introduced before in the natural language processing (NLP) realm and showed significant improvements in many tasks; however, it has rarely been used in natural language generation (NLG) tasks and never in Arabic NLG. In this work, we propose a meta-learning approach for Arabic dialogue generation for fast adaptation on low-resource domains, namely, Arabic. We start by using existing pre-trained models; we then meta-learn the initial parameters on high-resource dataset before finetuning the parameters on the target tasks. We prove that the proposed model that employs meta-learning techniques improves generalization and enables fast adaptation of the transformer model on low-resource NLG tasks. We report gains in the BLEU-4 and improvements in Semantic textual Similarity (STS) metrics when compared to the existing state-of-the-art approach. We also do a further study on the effectiveness of the meta-learning algorithms on the response generation of the models.
- [1] . 2016. Botta: An Arabic dialect chatbot. In Proceedings of the 26th International Conference on Computational Linguistics: System Demonstrations. The COLING 2016 Organizing Committee, 208–212. Retrieved from https://aclanthology.org/C16-2044.Google Scholar
- [2] . 2020. AraBERT: Transformer-based model for Arabic language understanding. arXiv preprint arXiv:2003.00104 (2020).Google Scholar
- [3] . 2019. Few-shot NLG with pre-trained language model. arXiv preprint arXiv:1904.09521 (2019).Google Scholar
- [4] . 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014).Google Scholar
- [5] . 1948. A formula for predicting readability: Instructions. Educ. Res. Bull. 27, 1–20 (1948), 37–54.Google Scholar
- [6] . 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).Google Scholar
- [7] . 2019. Investigating meta-learning algorithms for low-resource natural language understanding tasks. arXiv preprint arXiv:1908.10423 (2019).Google Scholar
- [8] . 2016. OSMAN: A novel Arabic readability metric. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16). European Language Resources Association (ELRA), Portorož, 250–255. https://aclanthology.org/L16-1038.Google Scholar
- [9] . 2019. Ollobot—Towards a text-based Arabic health conversational agent: Evaluation and results. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP’19). INCOMA Ltd., 295–303.
DOI: Google ScholarCross Ref
- [10] . 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning. PMLR, 1126–1135.Google Scholar
- [11] . 2019. Topical-chat: Towards knowledge-grounded open-domain conversations. In Proceedings of the INTERSPEECH Conference. 1891–1895.Google Scholar
Cross Ref
- [12] . 2020. A semi-supervised BERT approach for Arabic named entity recognition. In Proceedings of the 5th Arabic Natural Language Processing Workshop. Association for Computational Linguistics, 49–57. Retrieved from https://aclanthology.org/2020.wanlp-1.5.Google Scholar
- [13] . 2014. ArabChat: An Arabic conversational agent. In Proceedings of the 6th International Conference on Computer Science and Information Technology (CSIT). 227–237.
DOI: Google ScholarCross Ref
- [14] . 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 (2019).Google Scholar
- [15] . 1975. Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease forMula) for Navy Enlisted Personnel.
Technical Report . Naval Technical Training Command Millington, TN, Research Branch.Google ScholarCross Ref
- [16] . 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 388–395.Google Scholar
- [17] . 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 110–119.
DOI: Google ScholarCross Ref
- [18] . 2016. A persona-based neural conversation model. arXiv preprint arXiv:1603.06155 (2016).Google Scholar
- [19] . 2017. Adversarial learning for neural dialogue generation. arXiv preprint arXiv:1701.06547 (2017).Google Scholar
- [20] . 2020. Caire: An end-to-end empathetic chatbot. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 13622–13623.Google Scholar
Cross Ref
- [21] . 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. arXiv preprint arXiv:1603.08023 (2016).Google Scholar
- [22] . 2022. Meta-X\(_{NLG}\): A meta-learning approach based on language clustering for zero-shot cross-lingual transfer and generation. In Findings of the Association for Computational Linguistics. Association for Computational Linguistics, 269–284.
DOI: Google ScholarCross Ref
- [23] . 2019. Meta-learning for low-resource natural language generation in task-oriented dialogue systems. arXiv preprint arXiv:1905.05644 (2019).Google Scholar
- [24] . 2021. Empathetic BERT2BERT conversational model: Learning Arabic language generation with little data. In Proceedings of the 6th Arabic Natural Language Processing Workshop. Association for Computational Linguistics, 164–172. Retrieved from https://aclanthology.org/2021.wanlp-1.17.Google Scholar
- [25] . 2020. Empathy-driven Arabic conversational chatbot. In Proceedings of the 5th Arabic Natural Language Processing Workshop. Association for Computational Linguistics, 58–68. Retrieved from https://aclanthology.org/2020.wanlp-1.6.Google Scholar
- [26] . 2018. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999 (2018).Google Scholar
- [27] . 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. 311–318.Google Scholar
Digital Library
- [28] . 2007. An intelligent conversational agent approach to extracting queries from natural language. In Proceedings of the World Congress on Engineering.Google Scholar
- [29] . 2019. Domain adaptive dialog generation via meta learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2639–2649.
DOI: Google ScholarCross Ref
- [30] . 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 5370–5381.
DOI: Google ScholarCross Ref
- [31] . 2016. Optimization as a model for few-shot learning. International Conference on Learning Representations.Google Scholar
- [32] . 2011. Data-driven response generation in social media. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 583–593. Retrieved from https://aclanthology.org/D11-1054.Google Scholar
Digital Library
- [33] . 2020. Leveraging pre-trained checkpoints for sequence generation tasks. Trans. Assoc. Computat. Ling. 8 (2020), 264–280.Google Scholar
Cross Ref
- [34] . 2016. Meta-learning with memory-augmented neural networks. In Proceedings of the International Conference on Machine Learning. PMLR, 1842–1850.Google Scholar
- [35] . 2017. A deep reinforcement learning chatbot. arXiv preprint arXiv:1709.02349 (2017).Google Scholar
- [36] . 2020. Generating empathetic responses by looking ahead the user’s sentiment. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 7989–7993.Google Scholar
Cross Ref
- [37] . 2017. Prototypical networks for few-shot learning. Adv. Neural Inf. Process. Syst. 30 (2017).Google Scholar
- [38] . 2016. Two are better than one: An ensemble of retrieval-and generation-based dialog systems. arXiv preprint arXiv:1610.07149 (2016).Google Scholar
- [39] . 2022. LaMDA: Language models for dialog applications. arXiv preprint arXiv:2201.08239 (2022).Google Scholar
- [40] . 2020. Improving end-to-end speech-to-intent classification with Reptile. arXiv preprint arXiv:2008.01994 (2020).Google Scholar
- [41] . 2017. Attention is all you need. Adv. Neural Inf. Process. Syst. 30 (2017).Google Scholar
- [42] . 2003. The Elements of AIML Style. Vol. 139, New York, NY.Google Scholar
- [43] . 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 (2018).Google Scholar
- [44] . 1966. ELIZA—A computer program for the study of natural language communication between man and machine. Commun. ACM 9, 1 (1966), 36–45.Google Scholar
Digital Library
- [45] . 2017. Hybrid code networks: Practical and efficient end-to-end dialog control with supervised and reinforcement learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 665–677.
DOI: Google ScholarCross Ref
- [46] . 2019. HuggingFace’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 (2019).Google Scholar
- [47] . 2016. Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots. arXiv preprint arXiv:1612.01627 (2016).Google Scholar
- [48] . 2013. POMDP-based statistical spoken dialog systems: A review. Proc. IEEE 101, 5 (2013), 1160–1179.Google Scholar
Cross Ref
- [49] . 2018. Personalizing dialogue agents: I have a dog, do you have pets too?arXiv preprint arXiv:1801.07243 (2018).Google Scholar
- [50] . 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE International Conference on Computer Vision. 19–27.Google Scholar
Digital Library
Index Terms
Metadial: A Meta-learning Approach for Arabic Dialogue Generation
Recommendations
Fine-tuning GPT-2 on annotated RPG quests for NPC dialogue generation
FDG '21: Proceedings of the 16th International Conference on the Foundations of Digital GamesGPT-2, a neural language model trained on a large dataset of English web text, has been used in a variety of natural language generation tasks because of the language quality and coherence of its outputs. In order to investigate the usability of GPT-2 ...
Unsupervised generation of Arabic words
Automated word generation might be seen as the reverse process of morphology learning. The aim is to automatically coin valid words in the targeted language. As many other challenges in the field of natural language processing (NLP), the building of the ...
Memory-based morphological analysis generation and part-of-speech tagging of Arabic
Semitic '05: Proceedings of the ACL Workshop on Computational Approaches to Semitic LanguagesWe explore the application of memory-based learning to morphological analysis and part-of-speech tagging of written Arabic, based on data from the Arabic Treebank. Morphological analysis -- the construction of all possible analyses of isolated unvoweled ...






Comments