Bidirectional Deaf-Hearing Communication: a Modular Service- Oriented System Combining Multi-Purpose Artificial Intelligence and 3D Avatar Rendering

For centuries, the fundamental barrier between deaf and hearing people has been a simple mismatch of senses: one group communicates through the visual language of hands and faces, while the other relies on the sound of voices. While technology has long allowed us to translate spoken words into text for the deaf, and text into speech for the hearing, bridging the gap in real-time conversation has remained stubbornly difficult. The challenge is not just about converting one language to another, but about managing two entirely different ways of experiencing the world. Deaf individuals use sign language, a complex system where meaning is carried by the shape of a hand, the movement of an arm, the position of the body, and the expression on a face. Hearing individuals, conversely, speak in sounds that travel through the air. When a deaf person signs a sentence, a computer must watch the video, understand the specific movements, and turn them into words. When a hearing person speaks, a computer must take those words, break them down into the correct sequence of signs, and then show them to the deaf person. Doing both of these things instantly, without a lag that breaks the flow of conversation, requires a system that is both incredibly fast and incredibly smart.

A team of researchers in Portugal has built a new system designed to solve this exact problem, creating a digital bridge that allows deaf and hearing people to talk to each other as naturally as possible. They call their creation SLaDIS, a modular platform that acts like a universal translator for two-way conversation. Instead of trying to force everything into a single, massive program, the researchers broke the system into three distinct parts that work together like a well-organized relay race. The first part watches the deaf person's hands and face, the second part listens to the hearing person's words and turns them into signs, and the third part is the screen where the conversation actually happens. By keeping these parts separate but connected, the system can handle the heavy lifting of computer vision and artificial intelligence without slowing down the conversation.

The journey begins when a deaf person wants to speak. They sign their message into a camera on a mobile phone or computer. The system does not try to memorize every single pixel of the video, which would be too slow and require too much data. Instead, it uses a specialized tool to find the key points on the person's body—the joints of the fingers, the elbows, the shoulders, and the corners of the mouth. It tracks these points as they move, creating a simplified map of the gesture. This map is then fed into a deep learning model, a type of artificial intelligence trained on thousands of examples of Portuguese Sign Language. The model looks at the sequence of movements and identifies the specific signs being made, turning the visual motion into a list of words. Because sign language often leaves out small connecting words or uses a different sentence structure than spoken language, the system then passes this list of words to a large language model. This second layer of intelligence acts like a helpful editor, rearranging the words, adding necessary grammar, and smoothing out the sentence so it makes perfect sense to a hearing person. The result is a clear, natural sentence that appears on the screen for the hearing participant to read or hear.

The process works in reverse when the hearing person wants to reply. They type or speak their message, and the system immediately translates it into the specific glosses, or building blocks, of Portuguese Sign Language. This is not a simple word-for-word swap; the system understands that sign language has its own rules for word order and grammar. Once the sentence is converted into the correct sequence of signs, the system prepares to show it to the deaf user. Instead of sending a pre-recorded video of a human signer, which would take time to download and might not match the exact words, the system uses a 3D digital avatar. This avatar is a virtual human that lives inside the application. The system pulls up the specific animation for each sign from a library and strings them together, creating a fluid, continuous performance. The avatar moves its hands, arms, and face to mimic a real signer, displaying the message directly on the deaf user's screen.

The researchers tested this system extensively to see if it could keep up with the speed of a real conversation. They found that the part of the system that recognizes the signs was highly accurate, correctly identifying 95.6 percent of the signs it was tested on. When the system turned those signs into full sentences, the meaning matched the intended message with a high degree of similarity, scoring 0.81 on a scale where 1.0 is a perfect match. The speed of the system was equally impressive. The time it took to recognize a sign and turn it into a sentence was less than a tenth of a second, and the time it took for the avatar to start moving after a hearing person sent a message was just over one second. These speeds are fast enough to feel like a natural, flowing conversation rather than a stilted exchange of messages.

To ensure the system was actually useful, the researchers brought in experts from the Portuguese Association of the Deaf to evaluate the avatar. Two independent evaluators watched the avatar perform the signs and rated how well they could understand the message. One expert understood 90 percent of the signs, while the other understood 78 percent. The study noted that when the animation quality was high, the understanding was perfect, suggesting that the clarity of the movement is just as important as the accuracy of the words. The system also proved to be flexible, able to handle conversations in different settings like shopping, tourism, and healthcare, showing that it can adapt to the varied vocabulary needed in daily life.

The true innovation of this work lies in how the pieces fit together. By designing the system as a collection of independent services that talk to each other, the researchers created a platform that can grow. If better AI models are developed in the future, they can be swapped in without rebuilding the whole system. If more signs need to be added to the library, they can be recorded and added without disrupting the translation process. This approach means the system is not just a one-time experiment but a foundation for a tool that can evolve. The researchers acknowledge that the vocabulary is still limited compared to the full richness of the language, and that the translation from text to sign could be even more fluid with more training data. However, the results demonstrate that a fully functional, two-way bridge between the deaf and hearing worlds is not just a theoretical possibility, but a working reality that operates with the speed and reliability needed for genuine human connection.