skip to main content
research-article

DEAF-BSL: Deep lEArning Framework for British Sign Language recognition

Authors Info & Claims
Published:04 August 2022Publication History
Skip Abstract Section

Abstract

The recent development of disability studies in academic bodies has expedited the promotion of investigation on disability. With computer-aided tools, communication between the impaired person and someone who does not understand sign language could be accessible. A large number of people across the world are using sign language (e.g., British Sign Language (BSL), Asian Sign Language (ASL), Indian Sign Language (ISL), etc.) with hand gestures for communication. In BSL recognition, the involvement of both hands overlapping each other becomes the main challenge. Moreover, BSL comprises ambiguous signs concerning viewpoint. However, existing traditional techniques seem in-stable, less accurate, and inefficient. In this work, the BSL fingerspelling alphabet recognition problem explores using a Deep learning framework to address the above-mentioned concerns. Convolutional Neural Network (CNN) is employed to detect and recognize for classification of 26 alphabets after being trained on the BSL corpus dataset. The proposed work outperforms the existing works with better precision (6%), recall (4%), and F-measure (5̃%). It reported better results on the BSL corpus dataset and webcam videos. The model achieved better accuracy (98.0%) for a large lexicon of words than previous models (Goh & Holden [6]: 69.5%, Rambhau [9]: 79.2%, and Liwicki et al. [8]: 92.5%). The 3D CNN-based proposal performs robust hand detection, much more accurate sign recognition, more scalability, and less ambiguity in BSL finger-spelling recognition.

REFERENCES

  1. [1] Bowden Richard, Windridge David, Kadir Timor, Zisserman Andrew, and Brady Michael. 2004. A linguistic feature vector for the visual interpretation of sign language. In European Conference on Computer Vision. Springer, 390401.Google ScholarGoogle ScholarCross RefCross Ref
  2. [2] Buehler Patrick, Everingham Mark, Huttenlocher Daniel P., and Zisserman Andrew. 2008. Long term arm and hand tracking for continuous sign language TV broadcasts. In Proceedings of the 19th British Machine Vision Conference. BMVA Press, 11051114.Google ScholarGoogle ScholarCross RefCross Ref
  3. [3] Feng Bin, He Fangzi, Wang Xinggang, Wu Yongjiang, Wang Hao, Yi Sihua, and Liu Wenyu. 2016. Depth-projection-map-based bag of contour fragments for robust hand gesture recognition. IEEE Transactions on Human-Machine Systems 47, 4 (2016), 511523.Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Feris Rogerio, Turk Matthew, Raskar Ramesh, Tan Karhan, and Ohashi Gosuke. 2004. Exploiting depth discontinuities for vision-based fingerspelling recognition. In 2004 Conference on Computer Vision and Pattern Recognition Workshop. IEEE, 155155.Google ScholarGoogle ScholarCross RefCross Ref
  5. [5] Goh Paul. 2005. Automatic recognition of Auslan finger-spelling using hidden Markov models. Undergraduate Thesis, University of Western Australia (2005).Google ScholarGoogle Scholar
  6. [6] Goh Paul and Holden Eun-Jung. 2006. Dynamic fingerspelling recognition using geometric and motion features. In 2006 International Conference on Image Processing. IEEE, 27412744.Google ScholarGoogle ScholarCross RefCross Ref
  7. [7] Hinton Geoffrey E., Osindero Simon, and Teh Yee-Whye. 2006. A fast learning algorithm for deep belief nets. Neural Computation 18, 7 (2006), 15271554.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. [8] Isaacs Jason and Foo Simon. 2004. Hand pose estimation for American sign language recognition. In Thirty-Sixth Southeastern Symposium on System Theory, 2004. Proceedings of the. IEEE, 132136.Google ScholarGoogle ScholarCross RefCross Ref
  9. [9] Jang Youngkyoon, Jeon Ikbeom, Kim Tae-Kyun, and Woo Woontack. 2016. Metaphoric hand gestures for orientation-aware VR object manipulation with an egocentric viewpoint. IEEE Transactions on Human-Machine Systems 47, 1 (2016), 113127.Google ScholarGoogle Scholar
  10. [10] Kumar Krishan and Shrimankar Deepti D.. 2017. F-DES: Fast and deep event summarization. IEEE Transactions on Multimedia 20, 2 (2017), 323334.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. [11] Kyle Jim and Woll Bencie. 1981. British sign language.Special Education: Forward Trends 8, 1 (1981), 1923.Google ScholarGoogle Scholar
  12. [12] Cun Yann Le, Galland Conrad C., and Hinton Geoffrey E.. 1989. GEMINI: Gradient estimation through matrix inversion after noise injection. In Advances in Neural Information Processing Systems. 141148.Google ScholarGoogle Scholar
  13. [13] LeCun Yann A., Bottou Léon, Orr Genevieve B., and Müller Klaus-Robert. 2012. Efficient backprop. In Neural Networks: Tricks of the Trade. Springer, 948.Google ScholarGoogle Scholar
  14. [14] Liang Hui, Yuan Junsong, and Thalmann Daniel. 2014. Parsing the hand in depth images. IEEE Transactions on Multimedia 16, 5 (2014), 12411253.Google ScholarGoogle ScholarCross RefCross Ref
  15. [15] Liwicki Stephan and Everingham Mark. 2009. Automatic recognition of fingerspelled words in British sign language. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. IEEE, 5057.Google ScholarGoogle ScholarCross RefCross Ref
  16. [16] Lu Zhiyuan, Chen Xiang, Li Qiang, Zhang Xu, and Zhou Ping. 2014. A hand gesture recognition framework and wearable gesture-based interaction prototype for mobile devices. IEEE Transactions on Human-Machine Systems 44, 2 (2014), 293299.Google ScholarGoogle ScholarCross RefCross Ref
  17. [17] Maturana Daniel and Scherer Sebastian. 2015. VoxNet: A 3D convolutional neural network for real-time object recognition. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 922928.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. [18] Mohandes Mohamed, Deriche Mohamed, and Liu Junzhao. 2014. Image-based and sensor-based approaches to Arabic sign language recognition. IEEE Transactions on Human-Machine Systems 44, 4 (2014), 551557.Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Rambhau Pingale Prerna. 2013. Recognition of two hand gestures of word in British sign language (BSL). International Journal of Scientific and Research Publications 3, 10 (2013).Google ScholarGoogle Scholar
  20. [20] Ren Zhou, Yuan Junsong, Meng Jingjing, and Zhang Zhengyou. 2013. Robust part-based hand gesture recognition using Kinect sensor. IEEE Transactions on Multimedia 15, 5 (2013), 11101120.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. [21] Shanableh Tamer, Assaleh Khaled, and Al-Rousan Mohammad. 2007. Spatio-temporal feature-extraction techniques for isolated gesture recognition in Arabic sign language. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 37, 3 (2007), 641650.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. [22] Shi Yemin, Tian Yonghong, Wang Yaowei, and Huang Tiejun. 2017. Sequential deep trajectory descriptor for action recognition with three-stream CNN. IEEE Transactions on Multimedia 19, 7 (2017), 15101520.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. [23] Tubaiz Noor, Shanableh Tamer, and Assaleh Khaled. 2015. Glove-based continuous Arabic sign language recognition in user-dependent mode. IEEE Transactions on Human-Machine Systems 45, 4 (2015), 526533.Google ScholarGoogle ScholarCross RefCross Ref
  24. [24] Wang Chong, Liu Zhong, and Chan Shing-Chow. 2014. Superpixel-based hand gesture recognition with Kinect depth camera. IEEE Transactions on Multimedia 17, 1 (2014), 2939.Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Wang Li-Chun, Wang Ru, Kong De-Hui, and Yin Bao-Cai. 2014. Similarity assessment model for Chinese sign language videos. IEEE Transactions on Multimedia 16, 3 (2014), 751761.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. [26] Woll Bencie and Lawson Lilian. 1987. British sign language. Language, Communication and Education (1987), 1123.Google ScholarGoogle Scholar
  27. [27] Zhang Tong, Zheng Wenming, Cui Zhen, Zong Yuan, Yan Jingwei, and Yan Keyu. 2016. A deep neural network-driven feature learning method for multi-view facial expression recognition. IEEE Transactions on Multimedia 18, 12 (2016), 25282536.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. DEAF-BSL: Deep lEArning Framework for British Sign Language recognition

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Asian and Low-Resource Language Information Processing
        ACM Transactions on Asian and Low-Resource Language Information Processing  Volume 21, Issue 5
        September 2022
        486 pages
        ISSN:2375-4699
        EISSN:2375-4702
        DOI:10.1145/3533669
        Issue’s Table of Contents

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 4 August 2022
        • Online AM: 23 March 2022
        • Accepted: 1 January 2022
        • Received: 1 August 2021
        Published in tallip Volume 21, Issue 5

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Refereed

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      View Full Text

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!