Bhutanese Sign Language Hand-shaped Alphabets and Digits Detection and Recognition

Abstract

The communication problem between the deaf and the public is an emerging concern for both parents and the government of Bhutan. The parents are not able to understand their children. The deaf students are not able to communicate with the general public. Therefore, deaf school and government is urging people to learn Bhutanese Sign Language (BSL) but learning Sign Language (SL) is not easy. However, Computer Vision and machine learning applications have been solving communication gaps. It has been easy to learn and understand SL with the help of signs’ translation apps. The basics of all sign languages are alphabets and numbers. The purpose of this study is to develop a suitable machine learning model to detect and recognize the BSL alphabets and digits using BSL hand-shaped alphanumeric datasets.

In this study, the first BSL hand-shaped alphanumeric dataset was created with different augmentation techniques. Different SL models were evaluated with the dataset. However, the Convolutional Neural Network (CNN) based architecture outperformed them. Using six layers of CNN with the batch normalization and different dropout ratios, 20000 digits dataset, and 30000 alphabets dataset obtained better results compared to LeNet-5, SVM, KNN, and logistic regression. Furthermore, ResNet with 43 convolutional layers obtained the best training and validation accuracy of 100% and 98.38% respectively on 60,000 alphanumeric datasets. This research is the first of its kind to study the possibility of machine learning integration with the BSL to detect and recognize hand-shaped alphabets and digits. It was found that machine learning models can be deployed to develop Computer Vision applications to make BSL learning easier and accessible to the general public. Further studies are needed to create a video-based dataset and study BSL dynamic gesture recognition for word translation.

-