Abstract
Neural variational inference-based topic modeling has gained great success in mining abstract topics from documents. However, these topic models usually mainly focus on optimizing the topic proportions for documents, while the quality and the internal construction of topics are usually neglected. Specifically, these models lack the guarantee that semantically related words are supposed to be assigned to the same topic and are difficult to ensure the interpretability of topics. Moreover, many topical words recur frequently in the top words of different topics, which makes the learned topics semantically redundant and similar, and of little significance for further study. To solve the above problems, we propose a novel neural topic model called Neural Variational Gaussian Mixture Topic Model (NVGMTM). We use Gaussian distribution to depict the semantic relevance between words in the topics. Each topic in NVGMTM is considered as a multivariate Gaussian distribution over words in the word-embedding space. Thus, semantically related words share similar probabilities in each topic, which makes the topics more coherent and interpretable. Experimental results on two public corpora show the proposed model outperforms the state-of-the-art baselines.
- [1] . 2003. Latent dirichlet allocation. J. Mach. Learn. Res. 3 (2003), 993–1022.Google Scholar
Digital Library
- [2] . 2021. Tree-structured topic modeling with nonparametric neural variational inference. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 2343–2353.Google Scholar
Cross Ref
- [3] . 2015. Gaussian lda for topic models with word embeddings. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. 795–804.Google Scholar
Cross Ref
- [4] . 2008. Visualizing data using t-SNE. J. Mach. Learn. Res. 9 (2008), 2579–2605.Google Scholar
- [5] . 2018. Coherence-aware neural topic modeling. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 830–836.Google Scholar
Cross Ref
- [6] . 2019. Neural topic model with reinforcement learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP’19). 3469–3474.Google Scholar
Cross Ref
- [7] . 2019. Document informed neural autoregressive topic models with distributional prior. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 6505–6512.Google Scholar
Digital Library
- [8] . 2014. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations (ICLR’14). Retrieved from http://arxiv.org/abs/1312.6114.Google Scholar
- [9] . 2014. Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Gothenburg, Sweden, 530–539.
DOI: Google ScholarCross Ref
- [10] . 2004. RCV1: A new benchmark collection for text categorization research. J. Mach. Learn. Res. 5 (2004), 361–397.Google Scholar
Digital Library
- [11] . 2019. Neural variational correlated topic modeling. In Proceedings of the World Wide Web Conference. 1142–1152.Google Scholar
Digital Library
- [12] . 2002. MALLET: A Machine Learning for Language Toolkit. Retrieved from http://mallet.cs.umass.edu.Google Scholar
- [13] . 2022. Topic discovery via latent space clustering of pretrained language model representations. In Proceedings of the ACM Web Conference 2022. 3143–3152.Google Scholar
Digital Library
- [14] . 2017. Discovering discrete latent topics with neural variational inference. In Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2410–2419.Google Scholar
Digital Library
- [15] . 2016. Neural variational inference for text processing. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19–24, 2016 (JMLR Workshop and Conference Proceedings), and (Eds.), Vol. 48. JMLR.org, 1727–1736. Retrieved from http://proceedings.mlr.press/v48/miao16.html.Google Scholar
- [16] . 2014. Neural variational inference and learning in belief networks. In Proceedings of the 31st International Conference on International Conference on Machine Learning-Volume 32. II–1791.Google Scholar
Digital Library
- [17] . 2019. Topic modeling with Wasserstein autoencoders. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 6345–6381.Google Scholar
Cross Ref
- [18] . 2011. Improving topic coherence with regularized topic models. In Advances in Neural Information Processing Systems. MIT Press, 496–504.Google Scholar
Digital Library
- [19] . 2018. A Bayesian nonparametric topic model with variational auto-encoders. https://openreview.net/pdf?id=SkxqZngC.Google Scholar
- [20] . 2021. TAN-NTM: Topic attention networks for neural topic modeling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 3865–3880.Google Scholar
Cross Ref
- [21] . 2014. GloVe: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’14). 1532–1543. Retrieved from http://www.aclweb.org/anthology/D14-1162.Google Scholar
Cross Ref
- [22] . 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Berlin, Germany, 1715–1725.
DOI: Google ScholarCross Ref
- [23] . 2017. Autoencoding variational inference for topic models. In Proceedings of the 5th International Conference on Learning Representations (ICLR’17). OpenReview.net. Retrieved from https://openreview.net/forum?id=BybtVK9lg.Google Scholar
- [24] . 2018. Labeled phrase latent dirichlet allocation and its online learning algorithm. Data Min. Knowl. Discov. 32, 4 (2018), 885–912.Google Scholar
Digital Library
- [25] . 2020. Neural topic modeling with bidirectional adversarial training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 340–350.Google Scholar
Cross Ref
- [26] . 2021. Layer-assisted neural topic modeling over document networks. In Proceedings of the International Joint Conference on Artificial Intelligence. 3148–3154.Google Scholar
Cross Ref
- [27] . 2009. Distance metric learning for large margin nearest neighbor classification. J. Mach. Learn. Res. 10 (2009), 207–244. Google Scholar
Digital Library
- [28] . 2020. Short text topic modeling with topic distribution quantization and negative sampling decoder. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’20). 1772–1782.Google Scholar
Cross Ref
- [29] . 2015. Incorporating word correlation knowledge into topic modeling. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 725–734.Google Scholar
Cross Ref
- [30] . 2010. Understanding bag-of-words model: A statistical framework. Int. J. Mach. Learn. Cybernet. 1, 1 (2010), 43–52.Google Scholar
Cross Ref
- [31] . 2021. Topic modelling meets deep neural networks: A survey. In Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI’21), (Ed.). International Joint Conferences on Artificial Intelligence Organization, 4713–4720.
DOI: Survey Track. Google ScholarCross Ref
Index Terms
Neural Variational Gaussian Mixture Topic Model
Recommendations
Topic modelling for qualitative studies
Qualitative studies, such as sociological research, opinion analysis and media studies, can benefit greatly from automated topic mining provided by topic models such as latent Dirichlet allocation LDA. However, examples of qualitative studies that ...
Topic sentiment mixture: modeling facets and opinions in weblogs
WWW '07: Proceedings of the 16th international conference on World Wide WebIn this paper, we define the problem of topic-sentiment analysis on Weblogs and propose a novel probabilistic model to capture the mixture of topics and sentiments simultaneously. The proposed Topic-Sentiment Mixture (TSM) model can reveal the latent ...
The Ideal Topic: Interdependence of Topic Interpretability and Other Quality Features in Topic Modelling for Short Texts
Social Computing and Social Media. Design, Ethics, User Behavior, and Social Network AnalysisAbstractBackground. Topic modelling is a method of automated probabilistic detection of topics in a text collection. Use of topic modelling for short texts, e.g. tweets or search engine queries, is complicated due to their short length and grammatical ...






Comments