American Sign Language Machine Learning based Translation Model

Abstract

Reducing communication barriers between the hearing-impaired community and the broader society remains a critical challenge, as recognizing sign language gestures requires expertise that most people do not possess. This paper addresses this problem by evaluating and designing an ASL recognition system using deep learning approaches. The main objectives are to identify the most accurate models for both static alphabet-level and dynamic word-level gesture recognition, and to incorporate them into a real-time recognition system. For image-based recognition, CNN-based and hybrid models are trained and evaluated on the ASL Alphabet datasets. For video-based recognition, pretrained SlowFast, Slow-only, and ViViT models are evaluated on WLASL100 and WLASL300 subsets, and further tested on a custom-collected set of 300 sign language videos. The main results are as follows: (1) image-based experiments identify ResNet as best-performing model with accuracy >93 % for different configurations; (2) video-based experiments show ViViT achieving 51.9% and 27.5%, SlowFast achieving 65.38% and 61.38%, and Slow-only achieving 64.34% and 54.24% on WLASL100 and WLASL300 respectively; (3) inference on the custom test set yields the highest accuracy of (61%) with SlowFast Networks. These results confirm that CNN-based approaches remain state-of-the-art for static recognition, while spatiotemporal models are essential for dynamic word-level gesture recognition.