Indian Sign Language Recognition using MobileNetV2 Fine-Tuned by Transfer Learning

Abstract

Sign Language is the language used for communication involving hearing impaired and hearing disabled people that involves the movement of hands to exchange information. But even with the existence of such language, people find it difficult to communicate using the same due to its vast diversity across different regions and geographical areas of the world. For instance, ISL (Indian Sign Language) and ASL are the respective sign languages used in USA and India but they are completely different from one another from the perspective of hand signs as well as understanding. This arises the requirement for a model which provides people a basis to translate and understand ISL.The model that has been used in this work  involves a pretrained model, MobileNetV2, which is further aided by fine-tuning and Transfer Learning techniques so that the model's components are reapplied to the new model thereby reducing time and computational resources. The Indian Sign Language (ISLRTC referred) dataset is employed using signs demonstrated on the ISLRTC website taken as images under different lighting conditions and backgrounds and is preprocessed and augmented thereby undergoing operations like Rescaling, Normalization, Standardization of pixels. It consists of 36 labeled classes(26 Alphabets + 10 digits) each containing a set of 1000 sample images that represent a certain gesture. The preprocessed dataset is then splitted into training and evaluation sets and the model is evaluated based on evaluation metrices that include metrices like accuracy, precision,  recall and f1-scores. For better visualization purposes, confusion matrix along with graphs between accuracy and loss with epochs were plotted. An  accuracy of 95.06\%, precision, recall, f1-scores of 0.9438, 0.9411, 0.9410 respectively and training time of 40 minutes concluded that transfer learning balances the performance and computational cost of the model unlike other deep learning models.