Abstract
Sign language is the major means of communication for the deaf community. It uses body language and gestures such as hand shapes, lib patterns, and facial expressions to convey a message. Sign language is geography-specific, as it differs from one country to another. Arabic Sign language is used in all Arab countries. The availability of a comprehensive benchmarking database for ArSL is one of the challenges of the automatic recognition of Arabic Sign language. This article introduces KArSL database for ArSL, consisting of 502 signs that cover 11 chapters of ArSL dictionary. Signs in KArSL database are performed by three professional signers, and each sign is repeated 50 times by each signer. The database is recorded using state-of-art multi-modal Microsoft Kinect V2. We also propose three approaches for sign language recognition using this database. The proposed systems are Hidden Markov Models, deep learning images’ classification model applied on an image composed of shots of the video of the sign, and attention-based deep learning captioning system. Recognition accuracies of these systems indicate their suitability for such a large number of Arabic signs. The techniques are also tested on a publicly available database. KArSL database will be made freely available for interested researchers.
- Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. 2016. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 (2016).Google Scholar
- Magdy Abull-ela, Mohamed F. Tolba, and Ahmed Samir Elons. 2013. Pulse-coupled neural network feature generation model for Arabic sign language recognition. IET Image Proc. 7, 9 (Dec. 2013), 829--836.Google Scholar
- A. Abdelbaky Ahmed and Saleh Aly. 2014. Appearance-based Arabic sign language recognition using hidden Markov models. In Proceedings of the International Conference on Engineering and Technology (ICET’14). IEEE, 1--6.Google Scholar
- Kinda Al-Fityani and Carol Padden. 2010. Sign language geography in the Arab world. Sign Lang.: Cambridge Surv. (2010), 433--450.Google Scholar
- Omar Al-Jarrah and Alaa Halawani. 2001. Recognition of gestures in Arabic sign language using neuro-fuzzy systems. Artif. Intell. 133, 1--2 (2001), 117--138.Google Scholar
Digital Library
- M. AL-Rousan, K. Assaleh, and A. Tala’a. 2009. Video-based signer-independent Arabic sign language recognition using hidden Markov models. Appl. Soft Comput. 9, 3 (June 2009), 990--999.Google Scholar
- Nadia R. Albelwi and Yasser M. Alginahi. 2012. Real-time Arabic sign language (ARSL) recognition. In Proceedings of the International Conference on Communications and Information Technology. 497--501.Google Scholar
- Marco Alfonse, A. Ali, A. Samir Elons, Nagwa L. Badr, and Magdy Aboul-Ela. 2015. Arabic sign language benchmark database for different heterogeneous sensors. In Proceedings of the 5th International Conference on Information & Communication Technology and Accessibility (ICTA’15). IEEE, 1--9.Google Scholar
Cross Ref
- Miada A. Almasre and Hana Al-Nuaim. 2016. Recognizing Arabic sign language gestures using depth sensors and a KSVM classifier. In Proceedings of the 8th Computer Science and Electronic Engineering Conference (CEEC’16). IEEE, 146--151.Google Scholar
- Abdulaziz Almohimeed, Mike Wald, and R. I. Damper. 2011. Arabic text to Arabic sign language translation system for the deaf and hearing-impaired community. In Proceedings of the 2nd Workshop on Speech and Language Processing for Assistive Technologies. Association for Computational Linguistics, 101--109.Google Scholar
- Saleh Aly and Safaa Mohammed. 2014. Arabic sign language recognition using spatio-temporal local binary patterns and support vector machine. In Advanced Machine Learning Technologies and Applications, Aboul Ella Hassanien, Mohamed F. Tolba, and Ahmad Taher Azar (Eds.). Communications in Computer and Information Science, Vol. 488. Springer International Publishing, 36--45.Google Scholar
- Saleh Aly, Basma Osman, Walaa Aly, and Mahmoud Saber. 2016. Arabic sign language fingerspelling recognition from depth and intensity images. In Proceedings of the 12th International Computer Engineering Conference (ICENCO’16). IEEE, 99--104.Google Scholar
Cross Ref
- Omar Amin, Hazem Said, Ahmed Samy, and Hoda K. Mohammed. 2015. HMM based automatic Arabic sign language translator using Kinect. In Proceedings of the 10th International Conference on Computer Engineering & Systems (ICCES’15). IEEE, 389--392.Google Scholar
- Arab League Educational Cultural and Scientific Organization. 2006. Second Part of the Unified Arabic Sign Language Dictionary. The League of Arab States & the Supreme Council for Family Affairs, Qatar.Google Scholar
- Khaled Assaleh and M. Al-Rousan. 2005. Recognition of Arabic sign language alphabet using polynomial classifiers. EURASIP J. Adv. Sig. Proc. 2005, 13 (2005), 507614.Google Scholar
Digital Library
- Khaled Assaleh, Tamer Shanableh, Mustafa Fanaswala, Harish Bajaj, and Farnaz Amin. 2008. Vision-based system for continuous Arabic Sign Language recognition in user dependent mode. In Proceedings of the 5th International Symposium on Mechatronics and Its Applications. IEEE, 1--5.Google Scholar
Cross Ref
- Tamás Aujeszky and Mohamad Eid. 2016. A gesture recognition architecture for Arabic sign language communication system. Multimedia Tools Applic. 75, 14 (2016), 8493--8511.Google Scholar
Digital Library
- A. L. C. Barczak, N. H. Reyes, M. Abastillas, A. Piccio, and T. Susnjak. 2011. A new 2D static hand gesture colour image dataset for ASL gestures. Res. Lett. Inf. Math. Sci. 15 (2011), 12--20.Google Scholar
- K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman. 2014. Return of the devil in the details: Delving deep into convolutional nets. In Proceedings of the British Machine Vision Conference.Google Scholar
- Onno A. Crasborn, Johanna Mesch, Dafydd Waters, Annika Nonhebel, Els Van der Kooij, Bencie Woll, and Brita Bergman. 2007. Sharing sign language data online: Experiences from the ECHO project. Int. J. Corp. Ling. 12, 4 (2007), 535--562.Google Scholar
Cross Ref
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 248--255.Google Scholar
Cross Ref
- Philippe Dreuw, David Rybach, Thomas Deselaers, Morteza Zahedi, and Hermann Ney. 2007. Speech recognition techniques for a sign language recognition system. Hand 60 (2007), 80.Google Scholar
- Menna El Badawy, A. Samir Elons, Hwaida Sheded, and Mohamed F. Tolba. 2015. A proposed hybrid sensor architecture for Arabic sign language recognition. In Proceedings of the Conference on Intelligent Systems. Springer, 721--730.Google Scholar
- Sergio Escalera, Jordi Gonzàlez, Xavier Baró, Miguel Reyes, Oscar Lopes, Isabelle Guyon, Vassilis Athitsos, and Hugo Escalante. 2013. Multi-modal gesture recognition challenge 2013: Dataset and results. In Proceedings of the 15th ACM International Conference on Multimodal Interaction. ACM, 445--452.Google Scholar
Digital Library
- Michael Filhol, Mohamed N. Hadjadj, and Benoît Testu. 2016. A rule triggering system for automatic text-to-sign translation. Univ. Access Inf. Soc. 15, 4 (2016), 487--498.Google Scholar
Digital Library
- Nikolaos Gkigkelos and Christos Goumopoulos. 2017. Greek sign language vocabulary recognition using Kinect. In Proceedings of the 21st Pan-Hellenic Conference on Informatics. 1--6.Google Scholar
Digital Library
- Fatma Guesmi, Tahani Bouchrika, Olfa Jemai, Mourad Zaied, and Chokri Ben Amar. 2016. Arabic sign language recognition system based on wavelet networks. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics. IEEE, 003561--003566.Google Scholar
Cross Ref
- Ayman Hamed, Nahla A. Belal, and Khaled M. Mahar. 2016. Arabic sign language alphabet recognition based on HOG-PCA using Microsoft Kinect in complex backgrounds. In Proceedings of the IEEE 6th International Conference on Advanced Computing (IACC’16). IEEE, 451--458.Google Scholar
- Mohamed Hassan, Khaled Assaleh, and Tamer Shanableh. 2016. User-dependent sign language recognition using motion detection. In Proceedings of the International Conference on Computational Science and Computational Intelligence (CSCI’16). IEEE, 852--856.Google Scholar
Cross Ref
- Nada B. Ibrahim, Mazen M. Selim, and Hala H. Zayed. 2017. An automatic Arabic sign language recognition system (ArSLRS). J. King Saud Univ.-comput. Inf. Sci. 30, 4 (2017).Google Scholar
- Mohammed Waleed Kadous. 2002. Temporal Classification: Extending the Classification Paradigm to Multivariate Time Series. Ph.D. Dissertation. University of New South Wales.Google Scholar
- Pradeep Kumar, Himaanshu Gauba, Partha Pratim Roy, and Debi Prosad Dogra. 2017. Coupled HMM-based multi-sensor data fusion for sign language recognition. Pattern Recog. Lett. 86 (2017), 1--8.Google Scholar
Digital Library
- Hamzah Luqman and Sabri A. Mahmoud. 2018. Automatic translation of Arabic text-to-Arabic sign language. Univ. Access Inf. Soc. (2018), 1--13.Google Scholar
- Hamzah Luqman and Sabri A. Mahmoud. 2019. A machine translation system from Arabic sign language to Arabic. Univ. Access Inf. Soc. 19 (2019), 1--14.Google Scholar
- Mohamed Mohandes. 2001. Arabic sign language recognition. In Proceedings of the International Conference on Imaging Science, Systems, and Technology. 753--759.Google Scholar
- M. Mohandes, S. A.-Buraiky, T. Halawani, and S. Al-Baiyat. 2004. Automation of the Arabic sign language recognition. In Proceedings of the International Conference on Information and Communication Technologies: From Theory to Applications. IEEE, 479--480.Google Scholar
- Mohamed Mohandes, S . Aliyu, and M. Deriche. 2014a. Arabic sign language recognition using the leap motion controller. In Proceedings of the IEEE 23rd International Symposium on Industrial Electronics (ISIE’14). IEEE, 960--965.Google Scholar
- Mohamed Mohandes and Mohamed Deriche. 2005. Image based Arabic sign language recognition. In Proceedings of the 8th International Symposium on Signal Processing and Its Applications, Vol. 1. IEEE, 86--89.Google Scholar
Cross Ref
- Mohamed Mohandes and Mohamed Deriche. 2013. Arabic sign language recognition by decisions fusion using Dempster-Shafer theory of evidence. In Proceedings of the Computing, Communications and IT Applications Conference (ComComAp’13). 90--94.Google Scholar
Cross Ref
- Mohamed Mohandes, Mohamed Deriche, U. Johar, and S. Ilyas. 2012. A signer-independent Arabic sign language recognition system using face detection, geometric features, and a hidden Markov model. Comput. Electric. Eng. 38, 2 (2012), 422--433.Google Scholar
Digital Library
- Mohamed Mohandes, Mohamed Deriche, and Junzhao Liu. 2014b. Image-based and sensor-based approaches to Arabic sign language recognition. IEEE Trans. Hum.-mach. Syst. 44, 4 (2014), 551--557.Google Scholar
Cross Ref
- M. Mohandes, S. I. Quadri, and M. Deriche. 2007. Arabic sign language recognition an image-based approach. In Proceedings of the 21st International Conference on Advanced Information Networking and Applications Workshops (AINAW’07). 272--276.Google Scholar
- Sara Morrissey. 2008. Data-driven Machine Translation for Sign Languages. Ph.D. Dissertation. Dublin City University.Google Scholar
- Davi Hirafuji Neiva and Cleber Zanchettin. 2018. Gesture recognition: A review focusing on sign language in a mobile context. Exp. Syst. Applic. 103, Aug. (2018) 159--183.Google Scholar
Cross Ref
- Natalia Neverova, Christian Wolf, Graham W. Taylor, and Florian Nebout. 2014. Multi-scale deep learning for gesture detection and localization. In Proceedings of the Workshop at the European Conference on Computer Vision. Springer, 474--490.Google Scholar
- Arab League Educational, Cultural and Scientific Organization. 2000. LAS: First Part of the Unified Arabic Sign Dictionary.Google Scholar
- Nicolas Pugeault and Richard Bowden. 2011. Spelling it out: Real-time ASL fingerspelling recognition. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCV Workshops’11). IEEE, 1114--1119.Google Scholar
Cross Ref
- T. Shanableh and K. Assaleh. 2007. Arabic sign language recognition in user-independent mode. In Proceedings of the International Conference on Intelligent and Advanced Systems. IEEE, 597--600.Google Scholar
- Tamer Shanableh, Khaled Assaleh, and M. Al-Rousan. 2007. Spatio-temporal feature-extraction techniques for isolated gesture recognition in Arabic sign language. IEEE Trans. Syst., Man, Cyber. Part B, Cyber. 37, 3 (June 2007), 641--50.Google Scholar
Digital Library
- Samaa M. Shohieb, Hamdy K. Elminir, and A. M. Riad. 2015. SignsWorld atlas: A benchmark Arabic sign language database. J. King Saud Univ.-Comput. Inf. Sci. 27, 1 (2015), 68--76.Google Scholar
Digital Library
- Ala addin I. Sidig, Hamzah Luqman, and Sabri A. Mahmoud. 2018. Arabic Sign Language Recognition Using Optical Flow-based Features and HMM. Springer International Publishing, Cham, 297--305.Google Scholar
- Ala addin I. Sidig and Sabri A. Mahmoud. 2018. Trajectory based Arabic sign language recognition. Int. J. Adv. Comput. Sci. Applic. 9, 4 (2018), 283--291.Google Scholar
- Nantinee Soodtoetong and Eakbodin Gedkhaw. 2018. The efficiency of sign language recognition using 3D convolutional neural networks. In Proceedings of the 15th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON’18). IEEE, 70--73.Google Scholar
Cross Ref
- Chao Sun, Tianzhu Zhang, Bing-Kun Bao, Changsheng Xu, and Tao Mei. 2013. Discriminative exemplar coding for sign language recognition with Kinect. IEEE Trans. Cyber. 43, 5 (Oct. 2013), 1418--1428.Google Scholar
Cross Ref
- M. F. Tolba, Ahmed Samir, and Magdy Abul-Ela. 2012b. A proposed graph matching technique for Arabic sign language continuous sentences recognition. In Proceedings of the 8th International Conference on Informatics and Systems (INFOS’12). IEEE, MM--14.Google Scholar
- M. F. Tolba, Ahmed Samir, and Magdy Aboul-Ela. 2012a. Arabic sign language continuous sentences recognition using PCNN and graph matching. Neur. Comput. Applic. 23, 3--4 (Aug. 2012), 999--1010.Google Scholar
- Noor Tubaiz, Tamer Shanableh, and Khaled Assaleh. 2015. Glove-based continuous Arabic sign language recognition in user-dependent mode. IEEE Trans. Hum.-mach. Syst. 45, 4 (2015), 526--533.Google Scholar
Cross Ref
- Ulrich Von Agris and Karl-Friedrich Kraiss. 2007. Towards a video corpus for signer-independent continuous sign language recognition. In Proceedings of Gesture in Human-Computer Interaction and Simulation, International Gesture Workshop, May 23-25, 2007.Google Scholar
- Khairunizam Wan, Nazrul Hamizi Bin Adnan, A. B. Shahriman, Siti Khadijah Za’ba, Mohd Azri Abd Aziz, and Zulkifli Md Yusof. 2012. Gesture recognition b on hand postures and trajectories by using dataglove: A fuzzy probability approach--A review. In Proceedings of International Conference on Man-Machine Systems (ICoMMS'12).Google Scholar
- Frank Weichert, Daniel Bachmann, Bartholomäus Rudak, and Denis Fisseler. 2013. Analysis of the accuracy and robustness of the leap motion controller. Sensors (Switz.) 13, 5 (2013), 6380--6393.Google Scholar
Cross Ref
- Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the International Conference on Machine Learning. 2048--2057.Google Scholar
Digital Library
- Aliaa A. A. Youssif, Amal Elsayed Aboutabl, and Heba Hamdy Ali. 2011. Arabic sign language (ARSL) recognition system using HMM. Int. J. Adv. Comput. Sci. Applic. 2, 11 (2011).Google Scholar
- Morteza Zahedi, Philippe Dreuw, David Rybach, Thomas Deselaers, and Hermann Ney. 2006. Continuous sign language recognition-approaches from speech recognition and available data resources. In Proceedings of the 2nd Workshop on the Representation and Processing of Sign Languages: Lexicographic Matters and Didactic Scenarios. 21--24.Google Scholar
- Morteza Zahedi, Daniel Keysers, and Hermann Ney. 2005. Pronunciation clustering and modeling of variability for appearance-based sign language recognition. In Proceedings of the International Gesture Workshop. Springer, 68--79.Google Scholar
- Mahmoud M. Zaki and Samir I. Shaheen. 2011. Sign language recognition using a combination of new vision based features. Pattern Recog. Lett. 32, 4 (2011), 572--577.Google Scholar
Digital Library
Index Terms
KArSL: Arabic Sign Language Database
Recommendations
Isolated Arabic Sign Language Recognition Using A Transformer-based Model and Landmark Keypoints
Pose-based approaches for sign language recognition provide light-weight and fast models that can be adopted in real-time applications. This paper presented a framework for isolated Arabic sign language (ArSL) recognition using hand and face keypoints. We ...
A machine translation system from Arabic sign language to Arabic
AbstractArabic sign language (ArSL) is one of the sign languages that is used in Arab countries. This language has structure and grammar that differ from spoken Arabic. Available ArSL recognition systems perform direct mapping between the recognized sign ...
Translating sign language videos to talking faces
ICVGIP '21: Proceedings of the Twelfth Indian Conference on Computer Vision, Graphics and Image ProcessingCommunication with the deaf community relies profoundly on the interpretation of sign languages performed by the signers. In light of the recent breakthroughs in sign language translations, we propose a pipeline that we term "Translating Sign Language ...






Comments