Abstract
The Sign Language Recognition system intends to recognize the Sign language used by the hearing and vocally impaired populace. The interpretation of isolated sign language from static and dynamic gestures is a difficult study field in machine vision. Managing quick hand movement, facial expression, illumination variations, signer variation, and background complexity are amongst the most serious challenges in this arena. While deep learning-based models have been used to accomplish the entirety of the field's state-of-the-art outcomes, the previous issues have not been fully addressed. To overcome these issues, we propose a Hybrid Neural Network Architecture for the recognition of Isolated Indian and Russian Sign Language. In the case of static gesture recognition, the proposed framework deals with the 3D Convolution Net with an atrous convolution mechanism for spatial feature extraction. For dynamic gesture recognition, the proposed framework is an integration of semantic spatial multi-cue feature detection, extraction, and Temporal-Sequential feature extraction. The semantic spatial multi-cue feature detection and extraction module help in the generation of feature maps for Full-frame, pose, face, and hand. For face and hand detection, GradCam and Camshift algorithm have been used. The temporal and sequential module consists of a modified auto-encoder with a GELU activation function for abstract high-level feature extraction and a hybrid attention layer. The hybrid attention layer is an integration of segmentation and spatial attention mechanism. The proposed work also involves creating a novel multi-signer, single, and double-handed Isolated Sign representation dataset for Indian and Russian Sign Language. The experimentation was done on the novel dataset created. The accuracy obtained for Static Isolated Sign Recognition was 99.76%, and the accuracy obtained for Dynamic Isolated Sign Recognition was 99.85%. We have also compared the performance of our proposed work with other baseline models with benchmark datasets, and our proposed work proved to have better performance in terms of Accuracy metrics.
- [1] . 2020. Arabic sign language recognition through deep neural networks fine-tuning. https://www.learntechlib.org/p/217934/.Google Scholar
- [2] . 2020. Bhutanese sign language hand-shaped alphabets and digits detection and recognition (doctoral dissertation, naresuan university). http://nuir.lib.nu.ac.th/dspace/handle/123456789/2491.Google Scholar
- [3] . 2020. An eight-layer convolutional neural network with stochastic pooling, batch normalization and dropout for fingerspelling recognition of chinese sign language. Multimedia Tools Appl. 79, 21 (2020), 15697–15715.Google Scholar
Digital Library
- [4] . 2020. Turkish sign language digits classification with CNN using different optimizers. Int. Adv. Res. Eng. J. 4, 3 (2020), 200–207.Google Scholar
Cross Ref
- [5] . Mendeley Data, Vol. 1. https://data.mendeley.com/datasets/rc349j45m5/1.Google Scholar
- [6] . 2005. Automatic sign language analysis: A survey and the future beyond lexical meaning. IEEE Trans. Pattern Anal. Mach. Intell. 27, 6 (2005), 873–891.Google Scholar
Digital Library
- [7] . 2019. Sign language recognition, generation, and translation: An interdisciplinary perspective. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility 16–31.Google Scholar
Digital Library
- [8] . 2019. A deep neural framework for continuous sign language recognition by iterative training. IEEE Trans. Multimedia 21, 7 (2019), 1880–1891.Google Scholar
Cross Ref
- [9] . 2018. Video-based sign language recognition without temporal segmentation. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence.Google Scholar
Cross Ref
- [10] . 2019. Weakly supervised learning with multi-stream CNN-LSTM-HMMs to discover sequential parallelism in sign language videos. IEEE Trans. Pattern Anal. Mach. Intell. 42, 9 (2019), 2306–2320.Google Scholar
Digital Library
- [11] . 2019. A novel sign language recognition framework using hierarchical grassmann covariance matrix. IEEE Trans. Multimedia 21, 11 (2019), 2806–2814.Google Scholar
Cross Ref
- [12] . 2019. Iterative alignment network for continuous sign language recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4165–4174.Google Scholar
Cross Ref
- [13] . 2020. Sign language transformers: Joint end-to-end sign language recognition and translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10023–10033.Google Scholar
Cross Ref
- [14] . 2015. Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signers. Comput. Vision Image Understand. 141, (2015) 108–125.Google Scholar
Digital Library
- [15] . 2016. Gaussian error linear units (gelus). Retrieved from
DOI:
https://arXiv:1606.08415.Google Scholar
- [16] . 2012. The phonological organization of sign languages. Lang. Ling. Compass 6, 3 (2012), 162–182.Google Scholar
Cross Ref
- [17] . 2021. Machine learning-based sign language recognition: A review and its research frontier. J. Ambient Intell. Human. Comput. 12, 7 (2021), 7205–7224.Google Scholar
Cross Ref
- [18] . 2008. A multilingual multimedia Indian sign language dictionary tool. In Proceedings of the International Joint Conference on Natural language Processing (IJCNLP’08). 57.Google Scholar
- [19] . 1989. American sign language: The phonological base. Sign Lang. Studies 64, 1 (1989), 195–277.Google Scholar
Cross Ref
- [20] . 2007. Symmetry and dominance: A cross-linguistic study of signs and classifier constructions. Lingua 117, 7 (2007), 1169–1201.Google Scholar
Cross Ref
- [21] . 2013. Robust part-based hand gesture recognition using kinect sensor. IEEE Trans. Multimedia 15, 5 (2013), 1110–1120.Google Scholar
Digital Library
- [22] . 2014. Superpixel-based hand gesture recognition with kinect depth camera. IEEE Trans. Multimedia 17, 1 (2014), 29–39.Google Scholar
Cross Ref
- [23] . 2017. Continuous gesture recognition with hand-oriented spatiotemporal feature. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 3056–3064.Google Scholar
Cross Ref
- [24] . 2016. Online detection and classification of dynamic hand gestures with recurrent 3D convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4207–4215.Google Scholar
Cross Ref
- [25] . 2016. Sign language recognition. In Proceedings of the 3rd International Conference on Recent Advances in Information Technology (RAIT’16). IEEE, 422–428.Google Scholar
- [26] . 2009. Signtutor: An interactive system for sign language tutoring. IEEE MultiMedia 16, 1 (2009), 81–93.Google Scholar
Digital Library
- [27] . 2009. Sign language number recognition. In Proceedings of the 5th International Joint Conference on INC, IMS, and IDC. IEEE, 1503–1508.Google Scholar
Digital Library
- [28] . 2010. Chinese sign language recognition based on video sequence appearance modeling. In Proceedings of the 5th IEEE Conference on Industrial Electronics and Applications. IEEE, 1537–1542.Google Scholar
- [29] . 2017. Indian sign language recognition using optimized neural networks. In Information Technology and Intelligent Transportation Systems. Springer, Cham, 553–563.Google Scholar
- [30] . 2021. Bayesian k-nearest neighbour based redundancy removal and hand gesture recognition in isolated indian sign language without materials support. In Proceedings of the IOP Conference Series: Materials Science and Engineering. IOP Publishing, 012126.Google Scholar
Cross Ref
- [31] . 2017. Spotting and recognition of hand gesture for Indian sign language recognition system with skin segmentation and SVM. In Proceedings of the International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET’17). IEEE, 386–390.Google Scholar
Cross Ref
- [32] . 2018. Light invariant real-time robust hand gesture recognition. Optik 159 (2018), 283–294.Google Scholar
Cross Ref
- [33] . 2018. Real-time indian sign language (ISL) recognition. In Proceedings of the 9th International Conference on Computing, Communication and Networking Technologies (ICCCNT’18). IEEE, 1–9.Google Scholar
- [34] . 2021. Feature extraction technique for vision-based indian sign language recognition system: A review. Comput. Methods Data Eng. 39–53.Google Scholar
Cross Ref
- [35] . 2021. Bayesian K-nearest neighbour based redundancy removal and hand gesture recognition in isolated indian sign language without materials support. In Proceedings of the IOP Conference Series: Materials Science and Engineering. IOP Publishing, 1116, 1 (2021), 012126.Google Scholar
Cross Ref
- [36] . 2020. Evaluation of manual and non-manual components for sign language recognition. In Proceedings of the 12th Language Resources and Evaluation Conference. European Language Resources Association (ELRA’20).Google Scholar
- [37] . 2021. Pose-based sign language recognition using GCN and BERT. In Proceedings of the IEEE Workshop on Applications of Computer Vision (WACV’21). 31–40.Google Scholar
Cross Ref
- [38] . 2018. Neural sign language translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7784–7793.Google Scholar
Cross Ref
- [39] . 2019. Isolated sign language recognition using convolutional neural network hand modelling and hand energy image. Multimedia Tools Appl. 78, 14 (2019), 19917–19944.Google Scholar
Digital Library
- [40] . 2020. DeepArSLR: A novel signer-independent deep learning framework for isolated arabic sign language gestures recognition. IEEE Access 8, 83199–83212.Google Scholar
Cross Ref
- [41] . 2020. Transferring cross-domain knowledge for video sign language recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6205–6214.Google Scholar
Cross Ref
- [42] . 2020. Video-based isolated hand sign language recognition using a deep cascaded model. Multimedia Tools Appl. 79, 22965–22987.Google Scholar
Cross Ref
- [43] . 2021. Skeleton aware multi-modal sign language recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3413–3423.Google Scholar
Cross Ref
- [44] . 2021. Recognition of Indian sign language (ISL) using deep learning model. Wireless Personal Commun. (2021), 1–22.Google Scholar
- [45] . 2021. Gesture recognition method based on attention mechanism for complex background. J. Phys.: Conf. Ser. 1873, 1 (2021), 012009.Google Scholar
Cross Ref
- [46] . 2021. A deep learning framework for recognizing both static and dynamic gestures. Sensors 21, 6 (2021), 2227.Google Scholar
Cross Ref
- [47] . 2021, Raspoznavanie recognition of Russian and Indian sign languages based on machine learning. Analysis and Data Processing Systems 3, 83 (2021), 53–74.Google Scholar
Cross Ref
- [48] . 2019. ArASL: Arabic alphabets sign language dataset. Data Brief 23 (2019), 103777.Google Scholar
Cross Ref
- [49] . 2020. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 1459–1469.Google Scholar
Cross Ref
- [50] . 2021. Pose-based sign language recognition using GCN and BERT. In Proceedings of the IEE Workshop on Applications of Computer Vision (WACV’21). 31–40.Google Scholar
Cross Ref
- [51] . 2021. Hand pose guided 3d pooling for word-level sign language recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 3429–3439.Google Scholar
Cross Ref
- [52] . 2021. Word-level sign language recognition with multi-stream neural networks focusing on local regions. Retrieved from
DOI:
https://arXiv:2106.15989.Google Scholar
- [53] . 2017. Animation of fingerspelled words and number signs of the sinhala sign language. ACM Trans. Asian Low-Res. Lang. Info. Process. 16, 4 (2017), 1–26.Google Scholar
Digital Library
- [54] . 2019. Word reordering for translation into korean sign language using syntactically-guided classification. ACM Trans. Asian Low-Res. Lang. Info. Process. 19, 2 (2019), 1–20.Google Scholar
- [55] . 2020. Sign language generation system based on indian sign language grammar. ACM Trans. Asian Low-Res. Lang. Info. Process. 19, 4 (2020), 1–26.Google Scholar
Digital Library
- [56] . 2013. Recognition of indian sign language in live video. Retrieved from
DOI:
https://arXiv:1306.1301.Google Scholar
- [57] . 2021. KArSL: Arabic sign language database. ACM Trans. Asian Low-Res. Lang. Info. Process. 20, 1 (2021), 1–19.Google Scholar
Digital Library
- [58] . 2021. Deep multi-model fusion for human activity recognition using evolutionary algorithms. Int. J. Interact. Multimedia Artific. Intell. 7 (2021), 44–58.Google Scholar
- [59] . 2020. Two-stage human activity recognition using 2D-ConvNet. Int. J. Interact. Multimedia Artific. Intell. 6 (2020), 11.Google Scholar
- [60] . 2022. Sign pose-based transformer for word-level sign language recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 182–191.Google Scholar
Cross Ref
- [61] . 2022. Sign language recognition system using tensorflow object detection API. Retrieved from
DOI:
https://arXiv:2201.01486.Google Scholar
Index Terms
Static and Dynamic Isolated Indian and Russian Sign Language Recognition with Spatial and Temporal Feature Detection Using Hybrid Neural Network
Recommendations
Isolated Arabic Sign Language Recognition Using A Transformer-based Model and Landmark Keypoints
Pose-based approaches for sign language recognition provide light-weight and fast models that can be adopted in real-time applications. This paper presented a framework for isolated Arabic sign language (ArSL) recognition using hand and face keypoints. We ...
Sign language recognition using 3-D Hopfield neural network
ICIP '95: Proceedings of the 1995 International Conference on Image Processing (Vol.2)-Volume 2 - Volume 2This paper presents a sign language recognition system which consists of three modules: model-based hand tracking, feature extraction, and gesture recognition using a 3-D Hopfield neural network. In the experiments, we illustrate that this system can ...
Recognition of Indian Sign Language (ISL) Using Deep Learning Model
AbstractAn efficient sign language recognition system (SLRS) can recognize the gestures of sign language to ease the communication between the signer and non-signer community. In this paper, a computer-vision based SLRS using a deep learning technique has ...






Comments