AI-Powered Smart ISL Translator with Voice, Text & Gesture Recognition

Abstract: Communication barriers remain one of the major challenges faced by the deaf and hard-of-hearing community in India. Indian Sign Language (ISL) is their primary medium of communication; however, most of the population does not understand ISL. To bridge this gap, this paper presents a web-based AI-powered ISL to multilingual translator with offline support and reverse translation features. The system converts ISL gestures into multiple Indian regional languages in both text and voice formats, while also providing reverse translation of text or voice back to ISL using a 3D avatar simulation.

The application accepts multiple input formats—typed text, uploaded audio files, microphone recordings, and gesture images. It integrates gesture recognition, speech-to-text via AssemblyAI, text-to-speech using gTTS, and multilingual NLP translation using a hybrid translation framework. A unique feature of the system is its custom vocabulary expansion module, allowing users to add domain-specific ISL signs with translations. Furthermore, a chatbot interface based on Google Gemini AI enables real-time language learning and practice.

Initial results show that the system achieves reliable translation accuracy and significantly improves communication accessibility. By combining AI, NLP, gesture recognition, and digital avatars, this project contributes to inclusive communication, digital accessibility, and language learning support for the deaf and hearing-impaired community.

Keywords: Indian Sign Language (ISL), Gesture Recognition, NLP Translation, 3D Avatar, Chatbot, Accessibility, Speech-to-Text, Text-to-Speech, Multilingual Translation, Offline Support.