skip to main content
research-article

Punctuation Prediction in Bangla Text

Authors Info & Claims
Published:10 March 2023Publication History
Skip Abstract Section

Abstract

Punctuation prediction is critical as it can enhance the readability of machine-transcribed speeches or texts significantly by adding appropriate punctuation. Furthermore, systems like Automatic Speech Recognizer (ASR) produce texts that are unpunctuated, making the readability difficult for humans and also hampers the performance of various natural language processing (NLP) tasks. Such NLP related tasks have been investigated thoroughly for English; however, very limited work is done for punctuation prediction in the Bangla language. In this study, we train a bidirectional recurrent neural network (BRNN) along with Attention model with a plausibly large Bangla dataset. Afterwards, we apply extensive postprocessing techniques for predicting punctuation more accurately with the employed model. Initially, we perform experimentation with a relatively imbalanced dataset, and our model shows promising results F1=56.9 for Period) in punctuation prediction. Later, we also investigate the model’s performance using a balanced Bangla dataset to achieve higher performance scores (F1=62.2 for Question). Thus, the goal of this study is to propose an efficient approach that can predict punctuation in Bangla texts effectively. Our study also includes investigation on how our postprocessing techniques affect the prediction performance. Being an early attempt for the punctuation prediction in Bangla text, our work is expected to significantly contribute in the NLP field for the Bangla language, and will pave the way for future work with the Bangla language in this direction.

REFERENCES

  1. [1] Abadi Martin, Barham Paul, Chen Jianmin, Chen Zhifeng, Davis Andy, Dean Jeffrey, Devin Matthieu, Ghemawat Sanjay, Irving Geoffrey, Isard Michael, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. Tensorflow: A system for large-scale machine learning. In Proceedings of the 12th \(\lbrace\)USENIX\(\rbrace\) Conference on Operating Systems Design and Implementation (OSDI’16). USENIX Association, 265283.Google ScholarGoogle Scholar
  2. [2] Adiba Farzana Islam, Islam Tahmina, Kaiser M. Shamim, Mahmud Mufti, and Rahman Muhammad Arifur. 2020. Effect of corpora on classification of fake news using naive Bayes classifier. International Journal of Automation, Artificial Intelligence and Machine Learning 1, 1 (2020), 8092.Google ScholarGoogle Scholar
  3. [3] Alam Tanvirul, Khan Akib, and Alam Firoj. 2020. Punctuation restoration using transformer models for resource-rich and-poor languages. In Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT’20). 132142.Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Bahdanau Dzmitry, Cho Kyunghyun, and Bengio Yoshua. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014).Google ScholarGoogle Scholar
  5. [5] Ballesteros Miguel and Wanner Leo. 2016. A neural network architecture for multilingual punctuation generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing; 2016 Nov. 1–5; Austin (TX, USA).[place unknown]: ACL; 2016. p. 1048–53. ACL (Association for Computational Linguistics).Google ScholarGoogle ScholarCross RefCross Ref
  6. [6] Beeferman Doug, Berger Adam, and Lafferty John. 1998. Cyberpunc: A lightweight punctuation annotation system for speech. In Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP’98 (Cat. No. 98CH36181), Vol. 2. IEEE, 689692.Google ScholarGoogle ScholarCross RefCross Ref
  7. [7] Cho Kyunghyun, Merriënboer Bart Van, Bahdanau Dzmitry, and Bengio Yoshua. 2014. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259 (2014).Google ScholarGoogle Scholar
  8. [8] Cho Kyunghyun, Merriënboer Bart Van, Gulcehre Caglar, Bahdanau Dzmitry, Bougares Fethi, Schwenk Holger, and Bengio Yoshua. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014).Google ScholarGoogle Scholar
  9. [9] Chung Junyoung, Gulcehre Caglar, Cho KyungHyun, and Bengio Yoshua. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 (2014).Google ScholarGoogle Scholar
  10. [10] Fang Mingfeng, Zhao Haifeng, Song Xiao, Wang Xin, and Huang Shilei. 2019. Using bidirectional LSTM with BERT for Chinese punctuation prediction. In 2019 IEEE International Conference on Signal, Information and Data Processing (ICSIDP’19). IEEE, 15.Google ScholarGoogle ScholarCross RefCross Ref
  11. [11] Hasan H. M. M. and Islam M. A.. 2020. Emotion recognition from Bengali speech using RNN modulation-based categorization. In 2020 Third International Conference on Smart Systems and Inventive Technology (ICSSIT’20). IEEE, 11311136.Google ScholarGoogle ScholarCross RefCross Ref
  12. [12] Hochreiter Sepp and Schmidhuber Jürgen. 1997. Long short-term memory. Neural Computation 9, 8 (1997), 17351780.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. [13] Islam Md. Adnanul, Anik Md. Saidul Hoque, and Islam A. B. M. Alim Al. 2021. Towards achieving a delicate blending between rule-based translator and neural machine translator. Neural Computing and Applications 33, 18 (2021), 1214112167.Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. [14] Islam Md. Adnanul and Islam A. B. M. Alim Al. 2016. Polygot: Going beyond database driven and syntax-based translation. In Proceedings of the 7th Annual Symposium on Computing for Development. ACM, Article 28, 4 pages.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. [15] Islam Md. Adnanul, Islam A. B. M. Alim Al, and Anik Md. Saidul Hoque. 2017. Polygot: An approach towards reliable translation by name identification and memory optimization using semantic analysis. In 4th International Conference on Networking, Systems and Security (NSysS’17). IEEE, 18.Google ScholarGoogle ScholarCross RefCross Ref
  16. [16] Islam Md. Adnanul, Mukta Md. Saddam Hossain, Olivier Patrick, and Rahman Md. Mahbubur. 2022. Comprehensive guidelines for emotion annotation. In Proceedings of the 22nd ACM International Conference on Intelligent Virtual Agents (Faro, Portugal) (IVA’22). Association for Computing Machinery, Article 5, 8 pages. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. [17] Juin Chin Char, Wei Richard Xiong Jun, D’Haro Luis Fernando, and Banchs Rafael E.. 2017. Punctuation prediction using a bidirectional recurrent neural network with part-of-speech tagging. In TENCON 2017-2017 IEEE Region 10 Conference. IEEE, 18061811.Google ScholarGoogle ScholarCross RefCross Ref
  18. [18] Kim Seokhwan. 2019. Deep recurrent neural networks with layer-wise multi-head attentions for punctuation restoration. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’19). IEEE, 72807284.Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Li Xinxing and Lin Edward. 2020. A 43 language multilingual punctuation prediction neural network model. In INTERSPEECH. 10671071.Google ScholarGoogle Scholar
  20. [20] Liu Xin, Liu Yi, and Song Xiao. 2018. Investigating for punctuation prediction in Chinese speech transcriptions. In 2018 International Conference on Asian Language Processing (IALP’18). IEEE, 7478.Google ScholarGoogle ScholarCross RefCross Ref
  21. [21] Makhija Karan, Ho Thi-Nga, and Chng Eng-Siong. 2019. Transfer learning for punctuation prediction. In 2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC’19). IEEE, 268273.Google ScholarGoogle ScholarCross RefCross Ref
  22. [22] Makhoul John, Kubala Francis, Schwartz Richard, and Weischedel Ralph. 1999. Performance measures for information extraction. In Proceedings of DARPA Broadcast News Workshop. Herndon, VA, 249252.Google ScholarGoogle Scholar
  23. [23] Milkowski Marcin. 2010. Developing an open-source, rule-based proofreading tool. Software: Practice and Experience 40, 7 (2010), 543566.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. [24] Moro Anna and Szaszak Gyorgy. 2017. A phonological phrase sequence modelling approach for resource efficient and robust real-time punctuation recovery. In INTERSPEECH. 558562.Google ScholarGoogle Scholar
  25. [25] Mukta Md. Saddam Hossain, Islam Md. Adnanul, Khan Faisal Ahamed, Hossain Afjal, Razik Shuvanon, Hossain Shazzad, and Mahmud Jalal. 2021. A comprehensive guideline for Bengali sentiment annotation. ACM Trans. Asian Low-Resour. Lang. Inf. Process. 21, 2, Article 30 (Oct.2021), 19 pages. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. [26] Nanchen Alexandre and Garner Philip N.. 2019. Empirical evaluation and combination of punctuation prediction models applied to broadcast news. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’19). IEEE, 72757279.Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Oktem Alp, Farrus Mireia, and Wanner Leo. 2017. Attentional parallel RNNs for generating punctuation in transcribed speech. In International Conference on Statistical Language and Speech Processing. Springer, 131142.Google ScholarGoogle ScholarCross RefCross Ref
  28. [28] Parlar Tuba, Ozel Selma, and Song Fei. 2019. Analysis of data pre-processing methods for sentiment analysis of reviews. Computer Science 20 (2019).Google ScholarGoogle ScholarCross RefCross Ref
  29. [29] Peitz Stephan, Freitag Markus, Mauser Arne, and Ney Hermann. 2011. Modeling punctuation prediction as machine translation. In Proceedings of the 8th International Workshop on Spoken Language Translation: Papers.Google ScholarGoogle Scholar
  30. [30] Pham Quang H., Nguyen Binh T., and Cuong Nguyen Viet. 2019. Punctuation prediction for Vietnamese texts using conditional random fields. In Proceedings of the Tenth International Symposium on Information and Communication Technology. 322327.Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. [31] Rodriguez Pau, Bautista Miguel Angel, Gonzalez Jordi, and Escalera Sergio. 2018. Beyond one-hot encoding: Lower dimensional target embedding. CoRR abs/1806.10805 (2018). arxiv:1806.10805. http://arxiv.org/abs/1806.10805.Google ScholarGoogle Scholar
  32. [32] Salloum Wael, Finley Gregory, Edwards Erik, Miller Mark, and Suendermann-Oeft David. 2017. Deep learning for punctuation restoration in medical reports. In BioNLP 2017. 159164.Google ScholarGoogle ScholarCross RefCross Ref
  33. [33] Schuster Mike and Paliwal Kuldip K.. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing 45, 11 (1997), 26732681.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. [34] Su Yuanhang and Kuo C.-C. Jay. 2019. On extended long short-term memory and dependent bidirectional recurrent neural network. Neurocomputing 356 (2019), 151161.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. [35] Szaszak Gyorgy and Tundik Mata Akos. 2019. Leveraging a character, word and prosody triplet for an ASR error robust and agglutination friendly punctuation approach. Proc. Interspeech 2019 (2019), 29882992.Google ScholarGoogle ScholarCross RefCross Ref
  36. [36] Tiedemann Jorg. 2012. Parallel data, tools and interfaces in OPUS. In Lrec, Vol. 2012. 22142218.Google ScholarGoogle Scholar
  37. [37] Tilk Ottokar and Alumäe Tanel. 2016. Bidirectional recurrent neural network with attention mechanism for punctuation restoration. In Interspeech 2016. 30473051. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  38. [38] Tundik Mate Akos and Szaszak Gyorgy. 2018. Joint word-and character-level embedding CNN-RNN models for punctuation restoration. In 2018 9th IEEE International Conference on Cognitive Infocommunications (CogInfoCom’18). IEEE, 000135000140.Google ScholarGoogle ScholarCross RefCross Ref
  39. [39] Varavs Andris and Salimbajevs Askars. 2018. Restoring punctuation and capitalization using transformer models. In International Conference on Statistical Language and Speech Processing. Springer, 91102.Google ScholarGoogle ScholarCross RefCross Ref
  40. [40] Wang Feng, Chen Wei, Yang Zhen, and Xu Bo. 2018. Self-attention based network for punctuation restoration. In 2018 24th International Conference on Pattern Recognition (ICPR’18). IEEE, 28032808.Google ScholarGoogle ScholarCross RefCross Ref
  41. [41] Wang Tian and Cho Kyunghyun. 2015. Larger-context language modelling. arXiv preprint arXiv:1511.03729 (2015).Google ScholarGoogle Scholar
  42. [42] Xu K., Xie L., and Yao K.. 2016. Investigating LSTM for punctuation prediction. In 2016 10th International Symposium on Chinese Spoken Language Processing (ISCSLP’16). 15. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  43. [43] Yi Jiangyan and Tao Jianhua. 2019. Self-attention based model for punctuation prediction using word and speech embeddings. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’19). IEEE, 72707274.Google ScholarGoogle ScholarCross RefCross Ref
  44. [44] Nazi Zabir Al. 2020. Bangla Newspaper Dataset. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  45. [45] Zelasko Piotr, Szymanski Piotr, Mizgajski Jan, Szymczak Adrian, Carmiel Yishay, and Dehak Najim. 2018. Punctuation prediction model for conversational speech. arXiv preprint arXiv:1807.00543 (2018).Google ScholarGoogle Scholar

Index Terms

  1. Punctuation Prediction in Bangla Text

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Asian and Low-Resource Language Information Processing
        ACM Transactions on Asian and Low-Resource Language Information Processing  Volume 22, Issue 3
        March 2023
        570 pages
        ISSN:2375-4699
        EISSN:2375-4702
        DOI:10.1145/3579816
        Issue’s Table of Contents

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 10 March 2023
        • Online AM: 14 December 2022
        • Accepted: 3 October 2022
        • Revised: 27 May 2022
        • Received: 7 August 2021
        Published in tallip Volume 22, Issue 3

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
      • Article Metrics

        • Downloads (Last 12 months)94
        • Downloads (Last 6 weeks)9

        Other Metrics

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      View Full Text

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!