Abstract
Despite the impressive success of sequence to sequence models for generative question answering, they need a vast amount of question-answer pairs during training, which is hard and expensive to obtain, especially for low-resource languages. In this article, we present a framework that exploits the semantic clusters among the question-answer pairs to compensate for the lack of enough training data. In the training phase, the question-answer pairs are clustered, and a cluster predictor is trained to identify the cluster each question belongs to. Then, a sequence to sequence model is trained, where there is a different generator for each cluster in the decoder component. During the test phase, the cluster of the input question is first identified using the trained cluster predictor, and the appropriate decoder is exploited. Our experiments on a Persian religious dataset show that the proposed method outperforms the standard sequence to sequence model by a large margin in terms of ROUGE and BLEU scores. This is traced back to the lower number of words in each cluster, leading to a reduction in the number of effective parameters each generator needs to learn, which help the model learn from fewer training data with less overfitting.
- [1] . 2020. A question answering system in Hadith using linguistic knowledge. Comput. Speech Lang. 60 (2020), 101023.Google Scholar
Digital Library
- [2] . 2020. FarsTail: A Persian natural language inference dataset. arXiv preprint arXiv:2009.08820 (2020).Google Scholar
- [3] . 2020. Multi-document answer generation for non-factoid questions. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2477–2477.Google Scholar
Digital Library
- [4] . 2019. Incorporating external knowledge into machine reading for generative question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP’19). 2521–2530.Google Scholar
Cross Ref
- [5] . 2003. Latent Dirichlet allocation. J. Mach. Learn. Res. 3, Jan. (2003), 993–1022.Google Scholar
Digital Library
- [6] . 2018. Using Wikipedia edits in low resource grammatical error correction. In Proceedings of the EMNLP Workshop W-NUT: The 4th Workshop on Noisy User-generated Text. 79–84.Google Scholar
Cross Ref
- [7] . 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014).Google Scholar
- [8] . 2019. Exploiting multilingualism through multistage fine-tuning for low-resource neural machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP’19). 1410–1416.Google Scholar
Cross Ref
- [9] . 2019. Towards automating healthcare question answering in a noisy multilingual low-resource setting. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 948–953.Google Scholar
Cross Ref
- [10] . 2017. Question answering on knowledge bases and text using universal schema and memory networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. 358–365.Google Scholar
Cross Ref
- [11] . 2016. MixKMeans: Clustering question-answer archives. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 1576–1585.Google Scholar
- [12] . 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 4171–4186.Google Scholar
- [13] . 2018. Simple and effective semi-supervised question answering. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 582–587.Google Scholar
Cross Ref
- [14] . 2019. A robust self-learning framework for cross-lingual text classification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP’19). 6307–6311.Google Scholar
Cross Ref
- [15] , Josef Jon, and Pavel Smrz. 2021. Rethinking the objectives of extractive question answering. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering.Google Scholar
- [16] . 2018. Natural answer generation with heterogeneous memory. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 185–195.Google Scholar
Cross Ref
- [17] . 2019. A deep neural network framework for English Hindi question answering. ACM Trans. Asian Low-resour. Lang. Inf. Process. 19, 2 (2019), 1–22.Google Scholar
Digital Library
- [18] . 2017. Generating natural answers by incorporating copying and retrieving mechanisms in sequence-to-sequence learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. 199–208.Google Scholar
Cross Ref
- [19] . 2018. Multitask learning for neural generative question answering. Mach. Vis. Applic. 29, 6 (2018), 1009–1017.Google Scholar
Digital Library
- [20] . 2019. A framework for intelligent question answering system using semantic context-specific document clustering and WordNet. Sādhanā 44, 3 (2019), 1–10.Google Scholar
Cross Ref
- [21] . 2019. Cross-lingual intent classification in a low resource industrial setting. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP’19). 6420–6425.Google Scholar
Cross Ref
- [22] . 2021. Multi-level retrieval with semantic Axiomatic Fuzzy Set clustering for question answering. Appl. Soft Comput. 111 (2021), 107858.Google Scholar
Digital Library
- [23] . 2019. Learning with limited data for multilingual reading comprehension. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP’19). 2833–2843.Google Scholar
Cross Ref
- [24] . 2018. Generative question answering: Learning to answer the whole question. In Proceedings of the International Conference on Learning Representations.Google Scholar
- [25] . 2019. Unsupervised question answering by cloze translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 4896–4910.Google Scholar
Cross Ref
- [26] . 2004. Rouge: a package for automatic evaluation of summaries. In ACL Workshop on Text Summarization Branches Out, Association for Computational Linguistics.Google Scholar
- [27] . 2018. Curriculum learning for natural answer generation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence Organization, 4223–4229.
DOI: Google ScholarCross Ref
- [28] . 2018. Exploring named entity recognition as an auxiliary task for slot filling in conversational language understanding. In Proceedings of the EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI. 74–80.Google Scholar
Cross Ref
- [29] . 2008. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, Nov. (2008), 2579–2605.Google Scholar
- [30] . 2016. A survey on question answering systems with classification. J. King Saud Univ.-Comput. Inf. Sci. 28, 3 (2016), 345–361.Google Scholar
Digital Library
- [31] . 2009. A word clustering approach for language model-based sentence retrieval in question answering systems. In Proceedings of the 18th ACM Conference on Information and Knowledge Management. 1911–1914.Google Scholar
Digital Library
- [32] . 2020. Conclusion-supplement answer generation for non-factoid questions. In Proceedings of the AAAI Conference on Artificial Intelligence. 8520–8527.Google Scholar
Cross Ref
- [33] . 2020. Conversational question answering in low resource scenarios: A dataset and case study for Basque. In Proceedings of the 12th Language Resources and Evaluation Conference. 436–442.Google Scholar
- [34] . 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. 311–318.Google Scholar
Digital Library
- [35] . 2019. Abstract text summarization: A low resource challenge. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP’19). 5996–6000.Google Scholar
Cross Ref
- [36] . 2012. IPedagogy: Question answering system based on web information clustering. In Proceedings of the IEEE 4th International Conference on Technology for Education. IEEE, 245–246.Google Scholar
Digital Library
- [37] . 2018. Know what you don’t know: Unanswerable questions for SQuAD. (2018).
arxiv:cs.CL/1806.03822. Google Scholar - [38] . 2016. SQuAD: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 (2016).Google Scholar
- [39] . 2007. Question processing and clustering in INDOC: A biomedical question answering system. EURASIP J. Bioinf. Syst. Biol. (2007), 1–7.Google Scholar
Digital Library
- [40] . 2014. Sequence to sequence learning with neural networks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 3104–3112.Google Scholar
- [41] . 2018. S-Net: From answer extraction to answer synthesis for machine reading comprehension. In Proceedings of the AAAI Conference on Artificial Intelligence.Google Scholar
Cross Ref
- [42] . 2017. Attention is all you need. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 5998–6008.Google Scholar
- [43] . 2020. To pretrain or not to pretrain: Examining the benefits of pretraining on resource rich tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2209–2213.Google Scholar
Cross Ref
- [44] . 2021. Cluster-former: Clustering-based sparse transformer for question answering. In Findings of the Association for Computational Linguistics (ACL-IJCNLP’21). Online Event, 3958–3968.Google Scholar
- [45] . 2019. Natural answer generation with attention over instances. IEEE Access 7 (2019), 61008–61017.Google Scholar
Cross Ref
- [46] . 2019. Machine translation with weakly paired documents. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP’19). 4366–4375.Google Scholar
Cross Ref
- [47] . 2003. Document clustering based on non-negative matrix factorization. In Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval. 267–273.Google Scholar
Digital Library
- [48] . 2020. Lexicon-enhanced transformer with pointing for domains specific generative question answering. In Proceedings of the International Conference on Algorithms and Architectures for Parallel Processing. Springer, 340–354.Google Scholar
Digital Library
- [49] . 2016. Neural generative question answering. In Proceedings of the International Joint Conference on Artificial Intelligence.Google Scholar
Cross Ref
- [50] . 2019. A compare-aggregate model with latent clustering for answer selection. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 2093–2096.Google Scholar
Digital Library
- [51] . 2017. Learning to rank question-answer pairs using hierarchical recurrent encoder with latent topic clustering. arXiv preprint arXiv:1710.03430 (2017).Google Scholar
- [52] . 2019. Cross-lingual dependency parsing using code-mixed TreeBank. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP’19). 996–1005.Google Scholar
Cross Ref
- [53] . 2018. Learning transferable features for open-domain question answering. In Proceedings of the International Joint Conference on Neural Networks (IJCNN’18). IEEE, 1–8.Google Scholar
Cross Ref
Index Terms
Clustering-based Sequence to Sequence Model for Generative Question Answering in a Low-resource Language
Recommendations
A Modified K-means Algorithm for Sequence Clustering
HIS '09: Proceedings of the 2009 Ninth International Conference on Hybrid Intelligent Systems - Volume 01In this paper, we extend our research to construct a system which provides clustering services, more than user-active search. We use DCT mapping to extract features from sequences, and discuss sequence similarities of whole similarity and partial ...
Inter cluster distance management model with optimal centroid estimation for K-means clustering algorithm
Clustering techniques are used to group up the transactions based on the relevancy. Cluster analysis is one of the primary data analysis method. The clustering process can be done in two ways such that Hierarchical clusters and partition clustering. ...
Enhancing Low-Resource Languages Question Answering with Syntactic Graph
Database Systems for Advanced Applications. DASFAA 2022 International WorkshopsAbstractMultilingual pre-trained language models (PLMs) facilitate zero-shot cross-lingual transfer from rich-resource languages to low-resource languages in extractive question answering (QA) tasks. However, during fine-tuning on the QA task, the ...






Comments