CFP last date
20 January 2025
Call for Paper
February Edition
IJCA solicits high quality original research papers for the upcoming February edition of the journal. The last date of research paper submission is 20 January 2025

Submit your paper
Know more
The week's pick
Responsive image
SFLA-Based Line Balancing and Task Optimization in Master-Slave Controlled Hanger Transportation Systems for Garment Production

Duc Hoang Nguyen
Random Articles
Reseach Article

MotionScript: Sign Language to Voice Converter

by Nidhi Kadam, Chaitanya Kakade, Vishal Kaira
journal cover thumbnail

International Journal of Computer Applications
Foundation of Computer Science (FCS), NY, USA
Volume 186 - Number 6
Year of Publication: 2024
Authors: Nidhi Kadam, Chaitanya Kakade, Vishal Kaira
10.5120/ijca2024923399

Nidhi Kadam, Chaitanya Kakade, Vishal Kaira . MotionScript: Sign Language to Voice Converter. International Journal of Computer Applications. 186, 6 ( Jan 2024), 20-26. DOI=10.5120/ijca2024923399

@article{ 10.5120/ijca2024923399,
author = { Nidhi Kadam, Chaitanya Kakade, Vishal Kaira },
title = { MotionScript: Sign Language to Voice Converter },
journal = { International Journal of Computer Applications },
issue_date = { Jan 2024 },
volume = { 186 },
number = { 6 },
month = { Jan },
year = { 2024 },
issn = { 0975-8887 },
pages = { 20-26 },
numpages = {9},
url = { https://ijcaonline.org/archives/volume186/number6/33075-2024923399/ },
doi = { 10.5120/ijca2024923399 },
publisher = {Foundation of Computer Science (FCS), NY, USA},
address = {New York, USA}
}
%0 Journal Article
%1 2024-02-07T01:29:54.062471+05:30
%A Nidhi Kadam
%A Chaitanya Kakade
%A Vishal Kaira
%T MotionScript: Sign Language to Voice Converter
%J International Journal of Computer Applications
%@ 0975-8887
%V 186
%N 6
%P 20-26
%D 2024
%I Foundation of Computer Science (FCS), NY, USA
Abstract

Sign language serves as a vital mode of communication for the deaf and mute community, yet it presents a significant barrier in their interaction with the larger society, which often lacks proficiency in sign language. This paper presents MotionScript, an innovative sign language to voice conversion system that leverages computer vision, deep learning, Convolutional Neural Networks (CNN), Natural Language Processing (NLP) and Large Language Model (LLM) to facilitate interaction between individuals from the deaf and mute community and the rest of the world. This paper outlines a thorough comparison of four distinct neural network models, utilizing metrics to identify the most accurate model for transforming American Sign Language (ASL) into coherent and meaningful sentences voiced in natural language. This conversion process incorporates essential components such as autocorrection and the integration of a large language model.

References
  1. Hsien-I Lin, Ming-Hsiang Hsu, Wei-Kai Chen, “Human Hand gesture recognition using a convolution neural network”, 10.1109/CoASE.2014.6899454, August 2014
  2. Bikash K. Yadav, Dheeraj Jadhav, Hasan Bohra, Rahul Jain, “Sign Language to Text and Speech Conversion”, International Journal of Advance Research, Ideas, and Innovations in Technology, www.IJARIIT.com.
  3. Garcia, B., & Viesca, S. A. (2016), “Real-time American sign language recognition with convolutional neural networks, Convolutional Neural Networks for Visual Recognition, 2, 225-232
  4. S. S Kumar, T. Wangyal, V. Saboo and R. Srinath, “Time Series Neural Networks for Real Time Sign Language Translation,” 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), Orlando, FL, 2018, pp. 243-248, doi: 10.1109/ICMLA.2018.00043.
  5. AlKhuraym, Batool Yahya et al. “Arabic Sign Language Recognition using Lightweight CNN-based Architecture.” International Journal of Advanced Computer Science and Applications (2022)
  6. X. Han, F. Lu and G. Tian, “Sign Language Recognition Based on Lightweight 3D MobileNet-v2 and Knowledge Distillation,” ICETIS 2022; 7th International Conference on Electronic Technology and Information Science, 2022, pp. 1-6.
  7. M. Mahesh, A. Jayaprakash and M. Geetha, “Sign language translator for mobile platforms,” 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI), Udupi, 2017, pp. 1176-1181, doi: 10.1109/ICACCI.2017.8126001.
  8. V. N. T. Truong, C. Yang and Q. Tran, “A translator for American sign language to text and speech,” 2016 IEEE 5th Global Conference on Consumer Electronics, 2016, pp. 1-2, doi: 10.1109/GCCE.2016.7800427.
  9. Yang, Jie and Y. Xu. “Hidden Markov Model for Gesture Recognition.” (1994), doi: 10.21236/ada282845.
  10. Lee, C. K. M. et al. “American sign language recognition and training method with recurrent neural network.” Expert Syst. Appl. 167 (2021).
  11. Sari, Winda & Rini, dian Palupi & Malik, Reza. (2020), “Text Classification Using Long Short-Term Memory With GloVe Features”, Jurnal Ilmiah Teknik Elektro Komputer dan Informatika. 5. 85. 10.26555/jiteki.v5i2.15021
  12. Saleh, Yaser and Ghassan F. Issa. “Arabic Sign Language Recognition through Deep Neural Networks Fine-Tuning.” Int. J. Online Biomed. Eng. 16 (2020): 71-83.
  13. M. S. Nair, A. P. Nimitha and S. M. Idicula, “Conversion of Malayalam text to Indian sign language using synthetic animation,” 2016 International Conference on Next Generation Intelligent Systems (ICNGIS), Kottayam, 2016, pp. 1-4, doi: 10.1109/ICNGIS.2016.7854002.
  14. V. Lopez-Ludena, R. San-Segundo, R. Martin, D. Sanchez and A. Garcia, “Evaluating a Speech Communication System for Deaf People,” in IEEE Latin America Transactions, vol. 9, no. 4, pp. 565-570, July 2011, doi: 10.1109/TLA.2011.5993744.
  15. D. Kelly, J. Mc Donald and C. Markham, “Weakly Supervised Training of a Sign Language Recognition System Using Multiple Instance Learning Density Matrices,” in IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), vol. 41, no. 2, pp. 526- 541, April 2011, doi: 10.1109/TSMCB.2010.2065802.
  16. R. San Segundo, B. Gallo, J. M. Lucas, R. Barra-Chicote, L. D’Haro and F. Fernandez, “Speech into Sign Language Statistical Translation System for Deaf People,” in IEEE Latin America Transactions, vol. 7, no. 3, pp. 400- 404, July 2009, doi: 10.1109/TLA.2009.5336641.
  17. Ss, Shivashankara & S, Dr.Srinath. (2018). American Sign Language Recognition System: An Optimal Approach. International Journal of Image, Graphics and Signal Processing. 10. 10.5815/ijigsp.2018.08.03.
  18. S. Liu and W. Deng, “Very deep convolutional neural network based image classification using small training sample size,” 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), 2015, pp. 730-734, doi: 10.1109/ACPR.2015.7486599.
  19. Akash Nagaraj. (2018).ASL Alphabet [Data set]. Kaggle. https://doi.org/10.34740/KAGGLE/DSV/29550
Index Terms

Computer Science
Information Sciences

Keywords

American Sign Language (ASL) Convolutional Neural Network (CNN) Google Text-To-Speech (gTTS) Large Language Model (LLM) Long Short-Term Memory Machine Learning Natural Language Processing (NLP) Real-time Conversion Recurrent Neural Networks (RNN) Residual Networks (ResNet) Stochastic Gradient Descent (SGD) Visual Geometry Group (VGG16).

Powered by PhDFocusTM