skip to main content
research-article

Fusion of Spatio-temporal Information for Indic Word Recognition Combining Online and Offline Text Data

Published:21 November 2019Publication History
Skip Abstract Section

Abstract

We present a novel Indic handwritten word recognition scheme by fusion of spatio-temporal information extracted from handwritten images. The main challenge in Indic word recognition lies in its complexity because of modifiers, touching characters, and compound characters. Hidden Markov Models (HMMs) are being used to model such data due to their ability to learn sequential data, however, the recognition performance is not satisfactory. We propose here a Long Short-Term Memory (LSTM)-based architecture for offline Indic word recognition. Offline recognition methods usually involve spatial data, whereas it has been observed that online recognition schemes show better performance than the offline methodologies. Online information usually refers to the temporal information obtained from the strokes of the pen tip while writing, which is missing in offline word images. In this article, an effort has been made to extract the online temporal information from offline images using stroke recovery and later it is combined with spatial information in LSTM architecture. During recognition, the character models are trained using both offline and extracted pseudo-online handwritten data separately. Finally, a novel fusion scheme has been used to combine them together. From the experiment, it is noted that recognition performance of handwritten Indic words improves considerably due to the fusion scheme of spatial and temporal data.

References

  1. Mohamed Nidhal Abdi and Maher Khemakhem. 2015. A model-based approach to offline text-independent Arabic writer identification and verification. Pattern Recog. 48, 5 (2015), 1890--1903.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Chandranath Adak, Bidyut B. Chaudhuri, and Michael Blumenstein. 2016. Offline cursive Bengali word recognition using CNNs with a recurrent model. In Proceedings of the 15th International Conference on Frontiers in Handwriting Recognition (ICFHR’16). IEEE, 429--434.Google ScholarGoogle ScholarCross RefCross Ref
  3. Irfan Ahmad and Sabri A. Mahmoud. 2012. Arabic bank check analysis and zone extraction. In Proceedings of the International Conference on Image Analysis and Recognition. Springer, 141--148.Google ScholarGoogle Scholar
  4. Jawad H. AlKhateeb, Jinchang Ren, Jianmin Jiang, and Husni Al-Muhtaseb. 2011. Offline handwritten Arabic cursive text recognition using hidden Markov models and re-ranking. Pattern Recog. Lett. 32, 8 (2011), 1081--1088.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Sandhya Arora, Debotosh Bhattacharjee, Mita Nasipuri, Dipak Kumar Basu, and Mahantapas Kundu. 2010. Multiple classifier combination for off-line handwritten Devnagari character recognition. Retrieved from Arxiv Preprint Arxiv:1006.5913 (2010).Google ScholarGoogle Scholar
  6. Ahmad-Montaser Awal, Harold Mouchère, and Christian Viard-Gaudin. 2014. A global learning approach for an online handwritten mathematical expression recognition system. Pattern Recog. Lett. 35 (2014), 68--77.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. A. Bharath and Sriganesh Madhvanath. 2012. HMM-based lexicon-driven and lexicon-free word recognition for online handwritten Indic scripts. IEEE Trans. Pattern Anal. Mach. Intell. 34, 4 (2012), 670--682.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Nilanjana Bhattacharya and Umapada Pal. 2012. Stroke segmentation and recognition from bangla online handwritten text. In Proceedings of the International Conference on Frontiers in Handwriting Recognition. 740--745.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Fadi Biadsy, Jihad El-Sana, and Nizar Habash. 2006. Online Arabic handwriting recognition using hidden Markov models. In Proceedings of the 10th International Workshop on Frontiers in Handwriting Recognition. Suvisoft.Google ScholarGoogle Scholar
  10. Anne-Laure Bianne-Bernard, Farès Menasri, Ramy Al-Hajj Mohamad, Chafic Mokbel, Christopher Kermorvant, and Laurence Likforman-Sulem. 2011. Dynamic and contextual information in HMM modeling for handwritten word recognition. IEEE Trans. Pattern Anal. Mach. Intell. 33, 10 (2011), 2066--2080.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Horst Bunke and Tamás Varga. 2007. Off-line Roman cursive handwriting recognition. In Digital Document Processing. Springer, 165--183.Google ScholarGoogle Scholar
  12. Youssouf Chherawala, Partha Pratim Roy, and Mohamed Cheriet. 2016. Feature set evaluation for offline handwriting recognition systems: Application to the recurrent neural network model. IEEE Trans. Cyber. 46, 12 (2016), 2825--2836.Google ScholarGoogle ScholarCross RefCross Ref
  13. Youssouf Chherawala, Partha Pratim Roy, and Mohamed Cheriet. 2017. Combination of context-dependent bidirectional long short-term memory classifiers for robust offline handwriting recognition. Pattern Recog. Lett. 90 (2017), 58--64.Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. P. Doetsch, M. Kozielski, and H. Ney. 2014. Fast and robust training of recurrent neural networks for offline handwriting recognition. In Proceedings of the 14th International Conference on Frontiers in Handwriting Recognition. 279--284.Google ScholarGoogle Scholar
  15. Kartik Dutta, Praveen Krishnan, Minesh Mathew, and C. V. Jawahar. 2017. Towards accurate handwritten word recognition for Hindi and Bangla. In Proceedings of the National Conference on Computer Vision, Pattern Recognition, Image Processing, and Graphics. Springer, 470--480.Google ScholarGoogle Scholar
  16. S. Espana-Boquera, M. J. Castro-Bleda, J. Gorbe-Moya, and F. Zamora-Martinez. 2011. Improving offline handwritten text recognition with hybrid HMM/ANN models. IEEE Trans. Pattern Anal. Mach. Intell. 33, 4 (2011), 767--779.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Vahid Ghods, Ehsanollah Kabir, and Farbod Razzazi. 2013. Decision fusion of horizontal and vertical trajectories for recognition of online Farsi subwords. Eng. Appl. Artific. Intell. 26, 1 (2013), 544--550.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Rajib Ghosh, Pradeep Kumar, and Partha Pratim Roy. 2019. A Dempster-Shafer theory based classifier combination for online signature recognition and verification systems. Int. J. Mach. Learn. Cyber. 10, 9 (2019), 2467--2482.Google ScholarGoogle ScholarCross RefCross Ref
  19. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine Learning. ACM, 369--376.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber, et al. 2001. Gradient flow in recurrent nets: The difficulty of learning long-term dependencies.Google ScholarGoogle Scholar
  21. Yoshiharu Kato and Makoto Yasuhara. 2000. Recovery of drawing order from single-stroke handwriting images. IEEE Trans. Pattern Anal. Mach. Intell. 22, 9 (2000), 938--949.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. JinHyung Kim and Bong-Kee Sin. 2014. Online handwriting recognition. In Handbook of Document Image Processing and Recognition. Springer, 887--915.Google ScholarGoogle Scholar
  23. Pradeep Kumar, Himaanshu Gauba, Partha Pratim Roy, and Debi Prosad Dogra. 2017a. Coupled HMM-based multi-sensor data fusion for sign language recognition. Pattern Recog. Lett. 86 (2017), 1--8.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Pradeep Kumar, Himaanshu Gauba, Partha Pratim Roy, and Debi Prosad Dogra. 2017b. A multimodal framework for sensor based sign language recognition. Neurocomputing 259 (2017), 21--38.Google ScholarGoogle ScholarCross RefCross Ref
  25. Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, and Debi Prosad Dogra. 2016. Study of text segmentation and recognition using leap motion sensor. IEEE Sens. J. 17, 5 (2016), 1293--1301.Google ScholarGoogle ScholarCross RefCross Ref
  26. Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, and Debi Prosad Dogra. 2017c. 3D text segmentation and recognition using leap motion. Multimedia Tools Appl. 76, 15 (2017), 16491--16510.Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, and Debi Prosad Dogra. 2018. A position and rotation invariant framework for sign language recognition (SLR) using Kinect. Multimedia Tools Appl. 77, 7 (2018), 8823--8846.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Longin Jan Latecki, Quan-nan Li, Xiang Bai, and Wen-yu Liu. 2007. Skeletonization using SSM of the distance transform. In Proceedings of the International Conference on Image Processing, Vol. 5. V--349.Google ScholarGoogle ScholarCross RefCross Ref
  29. Umapada Pal, Ramachandran Jayadevan, and Nabin Sharma. 2012a. Handwriting recognition in Indian regional scripts: A survey of offline techniques. ACM Trans. Asian Lang. Inform. Proc. 11, 1 (2012), 1.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Umapada Pal, Ramachandran Jayadevan, and Nabin Sharma. 2012b. Handwriting recognition in Indian regional scripts: A survey of offline techniques. ACM Trans. Asian Lang. Inform. Proc. 11, 1 (2012), 1.Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Mohammad Tanvir Parvez and Sabri A. Mahmoud. 2013a. Arabic handwriting recognition using structural and syntactic pattern attributes. Pattern Recog. 46, 1 (2013), 141--154.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Mohammad Tanvir Parvez and Sabri A. Mahmoud. 2013b. Offline Arabic handwritten text recognition: A survey. Comput. Surv. 45, 2 (2013), 23.Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Réjean Plamondon and Claudio M. Privitera. 1999. The segmentation of cursive handwriting: An approach based on off-line recovery of the motor-temporal information. IEEE Trans. Image Proc. 8, 1 (1999), 80--91.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Steve Procter and John Illingworth. 1998. Combining HMM classifiers in a handwritten text recognition system. In Proceedings of the International Conference on Image Processing, Vol. 2. 934--938.Google ScholarGoogle ScholarCross RefCross Ref
  35. Yu Qiao, Jianzhuang Liu, and Xiaoou Tang. 2007. Offline signature verification using online handwriting registration. In Proceedings of the Conference on Computer Vision and Pattern Recognition. 1--8.Google ScholarGoogle ScholarCross RefCross Ref
  36. Kaushik Roy, Szilárd Vajda, Umapada Pal, Bidyut Baran Chaudhuri, and Abdel Belaïd. 2005. A system for Indian postal automation. In Proceedings of the 8th International Conference on Document Analysis and Recognition. 1060--1064.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Partha Pratim Roy, Ayan Kumar Bhunia, Ayan Das, Prasenjit Dey, and Umapada Pal. 2016. HMM-based Indic handwritten word recognition using zone segmentation. Pattern Recog. 60 (Dec. 2016), 1057--1075.Google ScholarGoogle Scholar
  38. Partha Pratim Roy, Ayan Kumar Bhunia, Ayan Das, Prithviraj Dhar, and Umapada Pal. 2017b. Keyword spotting in doctor’s handwriting on medical prescriptions. Expert Syst. Appl. 76 (2017), 113--128.Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Partha Pratim Roy, Ayan Kumar Bhunia, and Umapada Pal. 2017a. HMM-based writer identification in music score documents without staff-line removal. Expert Syst. Appl. 89 (2017), 222--240.Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Partha Pratim Roy, Umapada Pal, and Josep Lladós. 2011. Document seal detection using GHT and character proximity graphs. Pattern Recog. 44, 6 (2011), 1282--1295.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Partha Pratim Roy, Guoqiang Zhong, and Mohamed Cheriet. 2017c. Tandem hidden Markov models using deep belief networks for offline handwriting recognition. Frontiers of IT 8 EE 18, 7 (2017), 978--988.Google ScholarGoogle Scholar
  42. Jaakko Sauvola and Matti Pietikäinen. 2000. Adaptive document image binarization. Pattern Recog. 33, 2 (2000), 225--236.Google ScholarGoogle ScholarCross RefCross Ref
  43. Ching Y. Suen and Louisa Lam. 2000. Multiple classifier combination methodologies for different output levels. In Proceedings of the International Workshop on Multiple Classifier Systems. Springer, 52--66.Google ScholarGoogle Scholar
  44. Stanford University. 1999a. Neural Networks and Regularization. Retrieved from http://cs231n.github.io/neural-networks-2/#reg.Google ScholarGoogle Scholar
  45. Stanford University. 1999b. Unsupervised Learning. Retrieved from http://ufldl.stanford.edu/tutorial/unsupervised/Autoencoders.Google ScholarGoogle Scholar
  46. Lei Xu, Adam Krzyzak, and Ching Y. Suen. 1992. Methods of combining multiple classifiers and their applications to handwriting recognition. IEEE Trans. Syst., Man, Cyber. 22, 3 (1992), 418--435.Google ScholarGoogle ScholarCross RefCross Ref
  47. Xu-Yao Zhang, Yoshua Bengio, and Cheng-Lin Liu. 2017. Online and offline handwritten Chinese character recognition: A comprehensive study and new benchmark. Pattern Recog. 61 (2017), 348--360.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Fusion of Spatio-temporal Information for Indic Word Recognition Combining Online and Offline Text Data

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Asian and Low-Resource Language Information Processing
      ACM Transactions on Asian and Low-Resource Language Information Processing  Volume 19, Issue 2
      March 2020
      301 pages
      ISSN:2375-4699
      EISSN:2375-4702
      DOI:10.1145/3358605
      Issue’s Table of Contents

      Copyright © 2019 ACM

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 21 November 2019
      • Accepted: 1 September 2019
      • Revised: 1 July 2019
      • Received: 1 November 2017
      Published in tallip Volume 19, Issue 2

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Research
      • Refereed

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!