skip to main content
research-article

Neural Variational Gaussian Mixture Topic Model

Authors Info & Claims
Published:25 March 2023Publication History
Skip Abstract Section

Abstract

Neural variational inference-based topic modeling has gained great success in mining abstract topics from documents. However, these topic models usually mainly focus on optimizing the topic proportions for documents, while the quality and the internal construction of topics are usually neglected. Specifically, these models lack the guarantee that semantically related words are supposed to be assigned to the same topic and are difficult to ensure the interpretability of topics. Moreover, many topical words recur frequently in the top words of different topics, which makes the learned topics semantically redundant and similar, and of little significance for further study. To solve the above problems, we propose a novel neural topic model called Neural Variational Gaussian Mixture Topic Model (NVGMTM). We use Gaussian distribution to depict the semantic relevance between words in the topics. Each topic in NVGMTM is considered as a multivariate Gaussian distribution over words in the word-embedding space. Thus, semantically related words share similar probabilities in each topic, which makes the topics more coherent and interpretable. Experimental results on two public corpora show the proposed model outperforms the state-of-the-art baselines.

REFERENCES

  1. [1] Blei David M., Ng Andrew Y., and Jordan Michael I.. 2003. Latent dirichlet allocation. J. Mach. Learn. Res. 3 (2003), 9931022.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. [2] Chen Ziye, Ding Cheng, Zhang Zusheng, Rao Yanghui, and Xie Haoran. 2021. Tree-structured topic modeling with nonparametric neural variational inference. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 23432353.Google ScholarGoogle ScholarCross RefCross Ref
  3. [3] Das Rajarshi, Zaheer Manzil, and Dyer Chris. 2015. Gaussian lda for topic models with word embeddings. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. 795804.Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Maaten Laurens Van Der and Hinton Geoffrey E.. 2008. Visualizing data using t-SNE. J. Mach. Learn. Res. 9 (2008), 25792605.Google ScholarGoogle Scholar
  5. [5] Ding Ran, Nallapati Ramesh, and Xiang Bing. 2018. Coherence-aware neural topic modeling. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 830836.Google ScholarGoogle ScholarCross RefCross Ref
  6. [6] Gui Lin, Leng Jia, Pergola Gabriele, Xu Ruifeng, He Yulan, et al. 2019. Neural topic model with reinforcement learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP’19). 34693474.Google ScholarGoogle ScholarCross RefCross Ref
  7. [7] Gupta Pankaj, Chaudhary Yatin, Buettner Florian, and Schütze Hinrich. 2019. Document informed neural autoregressive topic models with distributional prior. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 65056512.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. [8] Kingma Diederik P. and Welling Max. 2014. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations (ICLR’14). Retrieved from http://arxiv.org/abs/1312.6114.Google ScholarGoogle Scholar
  9. [9] Lau Jey Han, Newman David, and Baldwin Timothy. 2014. Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Gothenburg, Sweden, 530539. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  10. [10] Lewis David D., Yang Yiming, Rose Tony, and Li Fan. 2004. RCV1: A new benchmark collection for text categorization research. J. Mach. Learn. Res. 5 (2004), 361397.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. [11] Liu Luyang, Huang Heyan, Gao Yang, Zhang Yongfeng, and Wei Xiaochi. 2019. Neural variational correlated topic modeling. In Proceedings of the World Wide Web Conference. 11421152.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. [12] McCallum Andrew Kachites. 2002. MALLET: A Machine Learning for Language Toolkit. Retrieved from http://mallet.cs.umass.edu.Google ScholarGoogle Scholar
  13. [13] Meng Yu, Zhang Yunyi, Huang Jiaxin, Zhang Yu, and Han Jiawei. 2022. Topic discovery via latent space clustering of pretrained language model representations. In Proceedings of the ACM Web Conference 2022. 31433152.Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. [14] Miao Yishu, Grefenstette Edward, and Blunsom Phil. 2017. Discovering discrete latent topics with neural variational inference. In Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 24102419.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. [15] Miao Yishu, Yu Lei, and Blunsom Phil. 2016. Neural variational inference for text processing. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19–24, 2016 (JMLR Workshop and Conference Proceedings), Balcan Maria-Florina and Weinberger Kilian Q. (Eds.), Vol. 48. JMLR.org, 17271736. Retrieved from http://proceedings.mlr.press/v48/miao16.html.Google ScholarGoogle Scholar
  16. [16] Mnih Andriy and Gregor Karol. 2014. Neural variational inference and learning in belief networks. In Proceedings of the 31st International Conference on International Conference on Machine Learning-Volume 32. II–1791.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. [17] Nan Feng, Ding Ran, Nallapati Ramesh, and Xiang Bing. 2019. Topic modeling with Wasserstein autoencoders. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 63456381.Google ScholarGoogle ScholarCross RefCross Ref
  18. [18] Newman David, Bonilla Edwin V., and Buntine Wray. 2011. Improving topic coherence with regularized topic models. In Advances in Neural Information Processing Systems. MIT Press, 496504.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. [19] Ning Xuefei, Zheng Yin, Jiang Zhuxi, Wang Yu, Yang Huazhong, and Huang Junzhou. 2018. A Bayesian nonparametric topic model with variational auto-encoders. https://openreview.net/pdf?id=SkxqZngC.Google ScholarGoogle Scholar
  20. [20] Panwar Madhur, Shailabh Shashank, Aggarwal Milan, and Krishnamurthy Balaji. 2021. TAN-NTM: Topic attention networks for neural topic modeling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 38653880.Google ScholarGoogle ScholarCross RefCross Ref
  21. [21] Pennington Jeffrey, Socher Richard, and Manning Christopher D.. 2014. GloVe: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’14). 15321543. Retrieved from http://www.aclweb.org/anthology/D14-1162.Google ScholarGoogle ScholarCross RefCross Ref
  22. [22] Sennrich Rico, Haddow Barry, and Birch Alexandra. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Berlin, Germany, 17151725. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  23. [23] Srivastava Akash and Sutton Charles A.. 2017. Autoencoding variational inference for topic models. In Proceedings of the 5th International Conference on Learning Representations (ICLR’17). OpenReview.net. Retrieved from https://openreview.net/forum?id=BybtVK9lg.Google ScholarGoogle Scholar
  24. [24] Tang Yikun, Mao Xianling, and Huang Heyan. 2018. Labeled phrase latent dirichlet allocation and its online learning algorithm. Data Min. Knowl. Discov. 32, 4 (2018), 885912.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. [25] Wang Rui, Hu Xuemeng, Zhou Deyu, He Yulan, Xiong Yuxuan, Ye Chenchen, and Xu Haiyang. 2020. Neural topic modeling with bidirectional adversarial training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 340350.Google ScholarGoogle ScholarCross RefCross Ref
  26. [26] Wang Yiming, Li Ximing, and Ouyang Jihong. 2021. Layer-assisted neural topic modeling over document networks. In Proceedings of the International Joint Conference on Artificial Intelligence. 31483154.Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Weinberger Kilian Q. and Saul Lawrence K.. 2009. Distance metric learning for large margin nearest neighbor classification. J. Mach. Learn. Res. 10 (2009), 207244. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. [28] Wu Xiaobao, Li Chunping, Zhu Yan, and Miao Yishu. 2020. Short text topic modeling with topic distribution quantization and negative sampling decoder. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’20). 17721782.Google ScholarGoogle ScholarCross RefCross Ref
  29. [29] Xie Pengtao, Yang Diyi, and Xing Eric. 2015. Incorporating word correlation knowledge into topic modeling. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 725734.Google ScholarGoogle ScholarCross RefCross Ref
  30. [30] Zhang Yin, Jin Rong, and Zhou Zhi-Hua. 2010. Understanding bag-of-words model: A statistical framework. Int. J. Mach. Learn. Cybernet. 1, 1 (2010), 4352.Google ScholarGoogle ScholarCross RefCross Ref
  31. [31] Zhao He, Phung Dinh, Huynh Viet, Jin Yuan, Du Lan, and Buntine Wray. 2021. Topic modelling meets deep neural networks: A survey. In Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI’21), Zhou Zhi-Hua (Ed.). International Joint Conferences on Artificial Intelligence Organization, 47134720. DOI: Survey Track.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Neural Variational Gaussian Mixture Topic Model

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Asian and Low-Resource Language Information Processing
      ACM Transactions on Asian and Low-Resource Language Information Processing  Volume 22, Issue 4
      April 2023
      682 pages
      ISSN:2375-4699
      EISSN:2375-4702
      DOI:10.1145/3588902
      Issue’s Table of Contents

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 25 March 2023
      • Online AM: 29 December 2022
      • Accepted: 19 December 2022
      • Revised: 17 October 2022
      • Received: 19 March 2022
      Published in tallip Volume 22, Issue 4

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
    • Article Metrics

      • Downloads (Last 12 months)136
      • Downloads (Last 6 weeks)13

      Other Metrics

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    View Full Text

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!