skip to main content
research-article

Q-Learning for Shift-Reduce Parsing in Indonesian Tree-LSTM-Based Text Generation

Authors Info & Claims
Published:17 May 2022Publication History
Skip Abstract Section

Abstract

Tree-LSTM algorithm accommodates tree structure processing to extract information outside the linear sequence pattern. The use of Tree-LSTM in text generation problems requires the help of an external parser at each generation iteration. Developing a good parser demands the representation of complex features and relies heavily on the grammar of the corpus. The limited corpus results in an insufficient number of vocabs for a grammar-based parser, making it less natural to link the text generation process. This research aims to solve the problem of limited corpus by proposing the use of a Reinforcement Learning algorithm in the formation of constituency trees, which link the sentence generation process given a seed phrase as the input in the Tree-LSTM model. The tree production process is modeled as a Markov’s decision process, where a set of states consists of word embedding vectors, and a set of actions of {Shift, Reduce}. The Deep Q-Network model as an approximator of the Q-Learning algorithm is trained to obtain optimal weights in representing the Q-value function.

The test results on perplexity-based evaluation show that the proposed Tree-LSTM and Q-Learning combination model achieves values 9.60 and 4.60 for two kinds of corpus with 205 and 1,000 sentences, respectively, better than the Shift-All model. Human evaluation of Friedman test and posthoc analysis showed that all five respondents tended to give the same assessment for the combination model of Tree-LSTM and Q-Learning, which on average outperforms two other nongrammar models, i.e., Shift-All and Reduce-All.

REFERENCES

  1. [1] Asghar Nabiha, Poupart Pascal, Hoey Jesse, Jiang Xin, and Mou Lili. 2018. Affective neural response generation. In Advances in Information Retrieval. Springer International Publishing, Cham, 154166.Google ScholarGoogle Scholar
  2. [2] Bojanowski Piotr, Grave Edouard, Joulin Armand, and Mikolov Tomas. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics 5 (2017), 135146.Google ScholarGoogle ScholarCross RefCross Ref
  3. [3] Bowman Samuel, Gauthier Jon, Rastogi Abhinav, Gupta Raghav, Manning Christopher D., and Potts Christopher. 2016. A fast unified model for parsing and sentence understanding. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, 14661477. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Dandekar Ashish, Zen Remmy A. M., and Bressan Stéphane. 2017. Generating fake but realistic headlines using deep neural networks. In International Conference on Database and Expert Systems Applications. Springer International Publishing, Cham, 427440.Google ScholarGoogle ScholarCross RefCross Ref
  5. [5] Dinakaramani Arawinda, Rashel Fam, Luthfi Andry, Distiawan Bayu, and Manurung Ruli. 2016. Developing (and utilizing) an Indonesian treebank. In The Second Wordnet Bahasa Workshop. Nanyang Technological University, 1516.Google ScholarGoogle Scholar
  6. [6] Goodfellow Ian, Pouget-Abadie Jean, Mirza Mehdi, Xu Bing, Warde-Farley David, Ozair Sherjil, Courville Aaron, and Bengio Yoshua. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems, Ghahramani Z., Welling M., Cortes C., Lawrence N., and Weinberger K. Q. (Eds.), Vol. 27. Curran Associates, Inc.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. [7] Grave Édouard, Bojanowski Piotr, Gupta Prakhar, Joulin Armand, and Mikolov Tomáš. 2018. Learning word vectors for 157 languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA), Miyazaki, Japan. https://aclanthology.org/L18-1550.Google ScholarGoogle Scholar
  8. [8] Hochreiter Sepp and Schmidhuber Jürgen. 1997. Long short-term memory. Neural Computation 9, 8 (1997), 17351780.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. [9] Hull David. 1993. Using statistical testing in the evaluation of retrieval experiments. In Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, New York, NY, USA, 329338. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. [10] Kingma Diederik P. and Welling Max. 2019. An introduction to variational autoencoders. Found. Trends Mach. Learn. 12, 4 (2019), 307392. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. [11] Larasati Septina Dian. 2012. IDENTIC corpus: Morphologically enriched Indonesian-English parallel corpus. In LREC. European Language Resources Association (ELRA), 902906.Google ScholarGoogle Scholar
  12. [12] Li Jiwei, Luong Minh-Thang, Jurafsky Dan, and Hovy Eduard. 2015. When are tree structures necessary for deep learning of representations?. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, 23042314. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] Liu Chia-Wei, Lowe Ryan, Serban Iulian Vlad, Noseworthy Mike, Charlin Laurent, and Pineau Joelle. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, 21222132. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  14. [14] Liu Jiangming and Zhang Yue. 2017. Shift-reduce constituent parsing with neural lookahead features. Transactions of the Association for Computational Linguistics 5 (2017), 4558.Google ScholarGoogle ScholarCross RefCross Ref
  15. [15] Maillard Jean and Clark Stephen. 2018. Latent tree learning with differentiable parsers: Shift-reduce parsing and chart parsing. In Proceedings of the Workshop on the Relevance of Linguistic Structure in Neural Architectures for NLP. Association for Computational Linguistics, Melbourne, Australia, 1318. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  16. [16] Miwa Makoto and Bansal Mohit. 2016. End-to-end relation extraction using LSTMs on sequences and tree structures. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, 11051116. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  17. [17] Mnih Volodymyr, Kavukcuoglu Koray, Silver David, Rusu Andrei A., Veness Joel, Bellemare Marc G., Graves Alex, Riedmiller Martin, Fidjeland Andreas K., Ostrovski Georg, et al. 2015. Human-level control through deep reinforcement learning. Nature 518, 7540 (2015), 529533.Google ScholarGoogle Scholar
  18. [18] Prakash Aaditya, Hasan Sadid A., Lee Kathy, Datla Vivek, Qadir Ashequl, Liu Joey, and Farri Oladimeji. 2016. Neural paraphrase generation with stacked residual LSTM networks. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. The COLING 2016 Organizing Committee, Osaka, Japan, 29232934. https://aclanthology.org/C16-1275Google ScholarGoogle Scholar
  19. [19] Rao Jinfeng, Upasani Kartikeya, Balakrishnan Anusha, White Michael, Kumar Anuj, and Subba Rajen. 2019. A tree-to-sequence model for neural NLG in task-oriented dialog. In Proceedings of the 12th International Conference on Natural Language Generation. Association for Computational Linguistics, Tokyo, Japan, 95100. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Smeaton Alan F.. 1999. Using NLP or NLP resources for information retrieval tasks. In Natural Language Information Retrieval. Springer Netherlands, Dordrecht, 99111.Google ScholarGoogle ScholarCross RefCross Ref
  21. [21] Stern Mitchell, Andreas Jacob, and Klein Dan. 2017. A minimal span-based neural constituency parser. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada, 818827.Google ScholarGoogle ScholarCross RefCross Ref
  22. [22] Sutton Richard S. and Barto Andrew G.. 2018. Reinforcement Learning: An Introduction. MIT press.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. [23] Tai Kai Sheng, Socher Richard, and Manning Christopher D.. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Beijing, China, 15561566.Google ScholarGoogle ScholarCross RefCross Ref
  24. [24] Wen Wei-dong, Ni Li, Zhen Li, Dong Ao-Xue, Qiang Xu, and Chen Chen. 2019. A shift-reduce dependency parser based on reinforcement learning. DEStech Transactions on Computer Science and Engineering (2019).Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Yogatama Dani, Blunsom Phil, Dyer Chris, Grefenstette Edward, and Ling Wang. 2017. Learning to compose words into sentences with reinforcement learning. In 5th International Conference on Learning Representations (ICLR 2017). International Conference on Learning Representations, OpenReview.net, Toulon, France.Google ScholarGoogle Scholar
  26. [26] Zhang Kaizhong and Shasha Dennis. 1989. Simple fast algorithms for the editing distance between trees and related problems. SIAM Journal on Computing 18, 6 (1989), 12451262.Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. [27] Zhang Lidan and Chan Kwok-Ping. 2009. Dependency parsing with energy-based reinforcement learning. In Proceedings of the 11th International Conference on Parsing Technologies (IWPT’09). Association for Computational Linguistics, Paris, France, 234237.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. [28] Zhang Tianyang, Huang Minlie, and Zhao Li. 2018. Learning structured representation for text classification via reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.Google ScholarGoogle ScholarCross RefCross Ref
  29. [29] Zhou Guangyou, Fang Yizhen, Peng Yehong, and Lu Jiaheng. 2019. Neural conversation generation with auxiliary emotional supervised models. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP) 19, 2 (2019), 117.Google ScholarGoogle Scholar

Index Terms

  1. Q-Learning for Shift-Reduce Parsing in Indonesian Tree-LSTM-Based Text Generation

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Asian and Low-Resource Language Information Processing
          ACM Transactions on Asian and Low-Resource Language Information Processing  Volume 21, Issue 4
          July 2022
          464 pages
          ISSN:2375-4699
          EISSN:2375-4702
          DOI:10.1145/3511099
          Issue’s Table of Contents

          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 17 May 2022
          • Accepted: 1 September 2021
          • Revised: 1 June 2021
          • Received: 1 June 2020
          Published in tallip Volume 21, Issue 4

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Refereed
        • Article Metrics

          • Downloads (Last 12 months)114
          • Downloads (Last 6 weeks)1

          Other Metrics

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Full Text

        View this article in Full Text.

        View Full Text

        HTML Format

        View this article in HTML Format .

        View HTML Format
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!