A Hybrid CNN–MLP Model Using Confidence-Gated Fusion Framework for Real-Time Indian Sign Language Recognition

Communication is a fundamental human need, yet for millions of people who are deaf or hard of hearing, the bridge to the wider world often remains broken. Sign language serves as a complete and rich language for these communities, but without a universal translation system, the gap between signers and non-signers persists. In recent years, scientists have turned to computers to build this bridge, teaching machines to "see" hand gestures and translate them into spoken or written words. This field relies on two main ways of seeing. One approach treats the hand like a photograph, analyzing the shape, color, and texture of the skin and fingers to guess the meaning. The other approach ignores the picture entirely, focusing instead on the precise geometry of the hand, mapping the exact position of joints and fingertips as a set of coordinates. While each method has its strengths, both struggle when the real world gets messy. A camera-based system might fail if the lighting changes or the background is cluttered, while a geometry-based system might miss subtle details that distinguish one sign from another.

In a new study, researchers at Galgotias College of Engineering & Technology in India have proposed a solution that combines these two distinct ways of seeing to create a more reliable translator for Indian Sign Language. Rather than forcing a computer to choose between looking at the hand's appearance or its structure, the team built a hybrid system that uses both simultaneously. They created two separate digital "brains": one trained to recognize gestures by looking at images of hands, and another trained to recognize them by reading the mathematical coordinates of hand landmarks. The innovation lies not just in having two models, but in how they work together. The researchers designed a smart switching mechanism that acts as a gatekeeper. In real-time operation, the system constantly checks how confident the image-based model is in its guess. If the image-based model is unsure because of poor lighting, the system automatically defers to the geometry-based model, which is prioritized for its robustness to environmental variations. If the geometry is unclear, it relies on the image. This dynamic decision-making is then smoothed out over time to ensure the final output does not flicker or jump erratically from one letter to another.

The team tested their system using a large collection of static hand images representing the alphabet and a few specific commands, totaling 87,000 pictures. They trained their image-reading model using a lightweight architecture known as MobileNetV2, which is designed to be fast enough for real-time use, and trained their geometry-reading model using a standard neural network that processes the 21 key points of the hand detected by a tool called MediaPipe. When they ran the individual models through their paces, the results were telling. The image-based model was incredibly accurate, reaching nearly 100% accuracy in controlled tests, but it showed signs of uncertainty when faced with the variable conditions of a live camera feed. The geometry-based model was slightly less accurate on its own, scoring around 97%, but it remained remarkably steady regardless of the background or lighting.

When the researchers fused these two approaches, the result was a system that prioritized stability over raw peak performance. The hybrid model achieved an accuracy of 95% in testing, a figure that, while slightly lower than the image-only model's theoretical maximum, proved far more consistent in practice. In real-time trials, the hybrid system maintained a confidence level of nearly 100%, effectively eliminating the hesitation and errors that plagued the single-model systems. The study found that by letting the two models select the answer based on confidence thresholds and then smoothing out the results over a short window of time, the system could handle the variable conditions of the environment. This approach successfully reduced misclassifications observed in the geometry-only model while maintaining stable predictions, demonstrating the effectiveness of combining visual and landmark-based features for gesture recognition.

The researchers conclude that this hybrid framework offers a practical path forward for assistive communication tools. By acknowledging that no single method is perfect in every situation, and by creating a system that can dynamically lean on whichever method is most reliable at any given moment, they have created a gesture recognizer that is robust enough for real-world use. The work demonstrates that combining visual appearance with geometric structure, managed by a confidence-based decision process, yields a tool that is not only accurate but also steady and dependable, bringing the promise of seamless communication between the hearing and the deaf one step closer to reality.