Abstract
In recent years, along with the dramatic developments of deep learning in the natural language processing (NLP) domain, notable multilingual pre-trained language techniques have been proposed. These recent multilingual text analysis and mining models have demonstrated state-of-the-art performance in several primitive NLP tasks, including cross-lingual text classification (CLC). However, these recent multilingual pre-trained language models still suffer limitations regarding their adaptation for specific task-driven fine-tuning in the context of low-resource languages. Moreover, they also encounter problems related to the capability of preserving the global semantic (e.g., topic, etc.) and long-range relationships between words to better fine-tune and effectively handle the cross-lingual text classification task. To meet these challenges, in this article, we propose a novel topic-driven multi-typed text graph attention–based representation learning method for dealing with the cross-lingual text classification problem called TG-CTC. In the proposed TG-CTC model, we utilize a novel fused topic-driven multi-typed text graph representation to jointly learn the rich-schematic structural and global semantic information of texts to effectively handle the CLC task. More specifically, we integrate the heterogeneous text graph attention network with the neural topic modelling approach to enrich the semantic information of learned textual representations in the context of multiple languages. Extensive experiments in benchmark multilingual datasets showed the effectiveness of the proposed TG-CTC model compared with the contemporary state-of-the-art baselines.
- [1] . 2018. Wikipedia-based relatedness measurements for multilingual short text clustering. ACM Trans. Asian Low-Resour. Lang. Inf. Process. 18, 2 (2018), 1–25.Google Scholar
Digital Library
- [2] . 2021. Event graph neural network for opinion target classification of microblog comments. Trans. Asian Low-Resour. Lang. Inf. Process. 21, 1 (2021), 1–13.Google Scholar
- [3] . 2021. A transformer-based approach to multilingual fake news detection in low-resource languages. Trans. Asian Low-Resour. Lang. Inf. Process. 21, 1 (2021), 1–20.Google Scholar
- [4] . Co-training for cross-lingual sentiment classification. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP.Google Scholar
- [5] . 2011. Is machine translation ripe for cross-lingual sentiment classification? In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies.Google Scholar
Digital Library
- [6] . 2010. Cross language text classification by model translation and semi-supervised learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.Google Scholar
- [7] . 2015. Cross-lingual text classification using topic-dependent word probabilities. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.Google Scholar
Cross Ref
- [8] . 2016. Cross-lingual text classification via model translation with limited dictionaries. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management.Google Scholar
Digital Library
- [9] . 2018. Deep pivot-based modeling for cross-language cross-domain transfer with minimal guidance. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.Google Scholar
Cross Ref
- [10] . 2018. Adversarial deep averaging networks for cross-lingual sentiment classification. Trans. Assoc. Comput. Ling. 6 (2018), 557–570.Google Scholar
Cross Ref
- [11] . 2018. Deep contextualized word representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.Google Scholar
Cross Ref
- [12] . 2018. Improving language understanding by generative pre-training. OpenAI.Google Scholar
- [13] . 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.Google Scholar
- [14] . 2017. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems.Google Scholar
Digital Library
- [15] . 2016. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations (ICLR'17). Google Scholar
- [16] . 2018. Graph attention networks. In Proceedings of the International Conference on Learning Representations (ICLR’18).Google Scholar
- [17] . 2019. Graph convolutional networks for text classification. In Proceedings of the AAAI Conference on Artificial Intelligence.Google Scholar
Digital Library
- [18] . 2020. Tensor graph convolutional networks for text classification. In Proceedings of the AAAI Conference on Artificial Intelligence.Google Scholar
Cross Ref
- [19] . 2020. Text graph transformer for document classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP’20).Google Scholar
Cross Ref
- [20] . 2020. Topic modeling: a comprehensive review. EAI Endors. Trans. Scal. Inf. Syst. 7, 24, (2020).Google Scholar
- [21] . 2020. Copula guided neural topic modelling for short texts. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.Google Scholar
Digital Library
- [22] . 2020. Neural topic modeling with bidirectional adversarial training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.Google Scholar
Cross Ref
- [23] . 2020. tBERT: Topic models and BERT joining forces for semantic similarity detection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.Google Scholar
Cross Ref
- [24] . 2021. Classification aware neural topic model for COVID-19 disinformation categorisation. PloS One 16, 2 (2021), e0247086.Google Scholar
Cross Ref
- [25] . 2017. Discovering discrete latent topics with neural variational inference. In Proceedings of the International Conference on Machine Learning (PMLR’17).Google Scholar
- [26] . 2017. Autoencoding variational inference for topic models. In Proceedings of the 5th International Conference on Learning Representations (ICLR).Google Scholar
- [27] . 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.Google Scholar
Cross Ref
- [28] . 2021. Cross-lingual text classification with heterogeneous graph neural network. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). 612--620.Google Scholar
Cross Ref
- [29] . 2020. Cross-lingual unsupervised sentiment classification with multi-view transfer learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.Google Scholar
Cross Ref
- [30] . 2020. Cosda-ml: Multi-lingual code-switching data augmentation for zero-shot cross-lingual nlp. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence. 3853--3860.Google Scholar
- [31] . 2019. A survey of cross-lingual word embedding models. J. Artif. Intell. Res. 65 (2019) 569–631.Google Scholar
Digital Library
- [32] . 2020. Cross-lingual text classification with minimal resources by transferring a sparse teacher. In Findings of the Association for Computational Linguistics: EMNLP 2020. 3604--3622.Google Scholar
Cross Ref
- [33] . 2020. Exploiting cross-lingual subword similarities in low-resource document classification. In Proceedings of the AAAI Conference on Artificial Intelligence.Google Scholar
Cross Ref
- [34] . 2020. Interactive refinement of cross-lingual word embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP’20).Google Scholar
Cross Ref
- [35] . 2020. Why overfitting isn't always bad: Retrofitting cross-lingual word embeddings to dictionaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2214--2220.Google Scholar
Cross Ref
- [36] . 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems.Google Scholar
- [37] . 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In Proceedings of the International Conference on Machine Learning (PMLR).Google Scholar
- [38] . 2009. Polylingual topic models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.Google Scholar
Cross Ref
- [39] . 2018. Multilingual anchoring: interactive topic modeling and alignment across languages. In Proceedings of the Conference and Workshop on Neural Information Processing Systems (NeurIPS’18), 8667–8677.Google Scholar
- [40] . 2019. A multilingual topic model for learning weighted topic links across corpora with low comparability. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP’19).Google Scholar
Cross Ref
- [41] . 2020. Learn to cross-lingual transfer with meta graph learning across heterogeneous languages. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’20).Google Scholar
Cross Ref
- [42] . 1997. Long short-term memory. Neur. Comput. 9, 8 (1997), 1735–1780.Google Scholar
Digital Library
- [43] . 2005. Bidirectional LSTM networks for improved phoneme classification and recognition. In Proceedings of the International Conference on Artificial Neural Networks.Google Scholar
Cross Ref
- [44] . 2014. The stanford CORENLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations.Google Scholar
Cross Ref
- [45] . 2019. Heterogeneous graph attention networks for semi-supervised short text classification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP’19).Google Scholar
Cross Ref
Index Terms
An Integrated Topic Modelling and Graph Neural Network for Improving Cross-lingual Text Classification
Recommendations
Cross-lingual Text Classification via Model Translation with Limited Dictionaries
CIKM '16: Proceedings of the 25th ACM International on Conference on Information and Knowledge ManagementCross-lingual text classification (CLTC) refers to the task of classifying documents in different languages into the same taxonomy of categories. An open challenge in CLTC is to classify documents for the languages where labeled training data are not ...
An unsupervised cross-lingual topic model framework for sentiment classification
Sentiment classification aims to determine the sentiment polarity expressed in a text. In online customer reviews, the sentiment polarities of words are usually dependent on the corresponding aspects. For instance, in mobile phone reviews, we may expect ...
Low-Resource Text Classification via Cross-Lingual Language Model Fine-Tuning
Chinese Computational LinguisticsAbstractText classification tends to be difficult when data are inadequate considering the amount of manually labeled text corpora. For low-resource agglutinative languages including Uyghur, Kazakh, and Kyrgyz (UKK languages), in which words are ...






Comments