🤖 AI

ViPo-MLLM: Visual-Pose Multimodal LLM for Gloss-Free Sign Language Translation

The paper introduces ViPo-MLLM, a novel framework that integrates spatio-temporal RGB and human pose features with a Large Language Model to achieve state-of-the-art gloss-free Sign Language Translation on PHOENIX14T and CSL-Daily datasets, effectively rivaling gloss-based approaches.

Original authors: Ahmed Abul Hasanaath, Bicheng Xu, Mir Rayat Imtiaz Hossain, Leonid Sigal, Hamzah Luqman

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ahmed Abul Hasanaath, Bicheng Xu, Mir Rayat Imtiaz Hossain, Leonid Sigal, Hamzah Luqman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to translate a silent movie where the actors are speaking entirely with their hands, faces, and body movements. This is Sign Language Translation (SLT). The goal is to turn those videos into normal spoken sentences (like English or German) without needing a human to write down every single "word" (called a gloss) the signer is making first.

The paper introduces a new AI system called ViPo-MLLM. Think of it as a super-smart translator that learns to "read" sign language by watching two different things at once: the video (what the person looks like) and the skeleton (where their joints are moving).

Here is a breakdown of how it works, using simple analogies:

1. The Problem: The "Blurry" vs. The "Skeleton"

Previous AI translators often tried to do this with just the video (like watching a movie) or just the skeleton (like watching a stick figure dance).

The Video (RGB): This is like watching a high-definition movie. You see the colors, the clothes, and the background. It gives the "vibe" and context, but sometimes the AI gets confused by the background noise or can't see the tiny finger movements clearly.
The Skeleton (Pose): This is like a stick-figure animation overlaid on the video. It strips away the clothes and background and focuses purely on where the hands, elbows, and face are. It's very precise about movement but lacks the "flavor" of the scene.

The authors realized that relying on just one is like trying to solve a puzzle with half the pieces missing.

2. The Solution: The "Double-Decker Bus"

The ViPo-MLLM framework is like a double-decker bus that carries two types of passengers (Video and Skeleton) and forces them to talk to each other.

Step 1: The Separate Drivers (Encoders)
The system has two specialized drivers. One driver watches the video and extracts the "look." The other driver watches the skeleton and extracts the "movement." They don't mix yet; they just gather their specific notes.
Step 2: The Conversation (Cross-Modal Attention)
This is the magic part. Usually, AI just glues these two notes together (like taping two papers side-by-side). But ViPo-MLLM uses a mechanism called Cross-Modal Attention.
- Analogy: Imagine the Video Driver says, "I see a hand moving fast," and the Skeleton Driver says, "I see the elbow bending." The system forces them to have a conversation: "Okay, the fast hand plus the bending elbow means the signer is saying 'running'."
- This allows the AI to understand when and how the body movement changes the meaning of the visual scene.
Step 3: The Translator (The LLM)
Once the two streams of information are fused into a single, rich understanding, they are handed to a Large Language Model (LLM). Think of the LLM as a very smart writer who has read millions of books.
- The system gives the writer a "prompt" (a set of instructions and examples) saying, "Here is a video of a person signing. Please write a sentence in English that matches what they are doing."
- The writer then generates the final spoken sentence.

3. How It Learned (The Training Gym)

The AI didn't just guess; it trained using a specific gym routine:

Contrastive Learning: Imagine showing the AI a video and a sentence, and then showing it a different sentence. The AI learns to say, "These two match!" and "These two don't!" This helps it understand the connection between the visual signs and the words.
Language Modeling: It also practices writing the sentences directly, trying to predict the next word in the sentence based on the video.

4. The Results: Beating the Competition

The authors tested this system on two major datasets (one in German, one in Chinese).

The Claim: ViPo-MLLM achieved the best results (State-of-the-Art) ever recorded for "gloss-free" translation.
The Surprise: It performed almost as well as systems that do use those expensive, human-written "gloss" annotations. This proves that you don't need a human to label every single sign if you have a good system that combines video and body movement correctly.

Summary

In short, ViPo-MLLM is a translator that doesn't just look at the video or the skeleton alone. It acts like a detective that combines the clues from the video (context, appearance) with the clues from the skeleton (precise movement) to figure out exactly what the signer is saying, without needing a human dictionary to help it along the way. It is currently the best at doing this without pre-written labels.

Technical Summary: ViPo-MLLM for Gloss-Free Sign Language Translation

Problem Statement
Sign Language Translation (SLT) aims to bridge communication gaps by converting sign language videos into spoken sentences. While early systems relied on "gloss-based" approaches using word-like annotations as intermediates, these methods suffer from high annotation costs, limited scalability, and rigid sentence structures. "Gloss-free" SLT, which translates videos directly to fluent text without intermediate glosses, is a promising alternative but remains challenging. Existing gloss-free methods often rely on single-modality features or weakly fused representations, failing to capture the fine-grained articulatory details (hands, body, facial cues) and long-range temporal dependencies necessary for accurate translation. Furthermore, naive fusion of visual and pose data often fails to align these modalities effectively.

Methodology: ViPo-MLLM
The authors propose ViPo-MLLM (Visual-Pose Multimodal LLM), a framework designed to integrate spatio-temporal RGB features and human pose cues through a cross-modal attention mechanism. The architecture operates as follows:

Feature Extraction:
- Spatio-Temporal RGB Features ( $Z_s$ ): Extracted using a pre-trained USTM encoder (based on Swin Transformer stages with temporal adapters). The encoder is frozen, and task-specific heads (MS-TCN, BiLSTM) are discarded to avoid temporal bias, yielding frame-level representations.
- Pose Features ( $Z_p$ ): Extracted using OpenPose to estimate full-body and fine-grained hand poses. Instead of heatmaps, intermediate features are spatially pooled per frame to create fixed-dimensional vectors.
Multimodal Fusion:
- Intra-Modal Temporal Modeling (IMTM): Both $Z_s$ and $Z_p$ are projected into a shared space and processed by a shared 2-layer 1D Temporal Convolutional Network (TCN) to capture local temporal dynamics within each modality.
- Cross-Modal Temporal Modeling (CMTM): The outputs of the IMTM blocks are concatenated and passed through a 4-layer Cross-Modal Attention (CMA) module. This block enables global temporal modeling and captures dependencies between the visual and pose modalities.
- Projection: The fused representation is projected via an MLP to match the embedding dimension of the Large Language Model (LLM).
Language Modeling and Training:
- Prompt Construction: The fused visual-pose features are concatenated with a structured prompt containing task instructions and few-shot examples (e.g., Spanish/French/English translations) to condition the LLM.
- Learning Objectives: The model is trained end-to-end using a combination of two losses:
  - Contrastive Loss: Aligns a global video representation (pooled from fused features) with the global representation of the ground-truth text embedding.
  - Cross-Entropy Loss: Standard language modeling loss where the LLM generates the target translation based on the joint input of the prompt and fused features.
- Fine-Tuning: The LLM backbone (mT0-XL) is fine-tuned using LoRA (Low-Rank Adaptation).

Key Contributions
The paper outlines four primary contributions:

ViPo-MLLM Framework: A novel architecture integrating spatio-temporal RGB and pose cues via cross-modal attention for gloss-free SLT.
Explicit Fusion Strategy: A specific video-pose fusion mechanism (IMTM + CMTM) that models when and how pose information influences translation, capturing both local and global dependencies.
Comprehensive Analysis: Extensive experiments demonstrating the complementary roles of pose information and spatio-temporal visual cues in gloss-free SLT.
State-of-the-Art Performance: Achieving new best results on benchmark datasets while maintaining competitiveness with gloss-based methods.

Experimental Results
The model was evaluated on two datasets: PHOENIX14T (German Sign Language) and CSL-Daily (Chinese Sign Language).

PHOENIX14T: ViPo-MLLM achieved a BLEU-4 score of 27.10 and ROUGE-L of 51.50, outperforming existing weakly supervised and fully gloss-free methods (e.g., VAP, ConSLT, MMSLT). It remained competitive with gloss-based approaches despite not using gloss annotations.
CSL-Daily: The model achieved a BLEU-4 of 25.85 and ROUGE-L of 54.53, setting a new state-of-the-art for gloss-free methods and matching the performance of gloss-based models.

Ablation Insights

Modalities: Combining RGB and pose features significantly outperformed using either modality alone. Contrastive learning further enhanced semantic alignment for both.
CMA Depth: Performance peaked at a CMA depth of 4; deeper layers (e.g., 8) led to performance degradation, likely due to over-smoothing or overfitting.
LLM Choice: Larger models within the mT0 family (mT0-XL) yielded better results than smaller variants.
Prompting: Instructive prompts (direct task guidance) yielded the best performance, whereas interrogative or role-playing prompts reduced accuracy.

Significance
The paper claims that ViPo-MLLM demonstrates that gloss-free training can match or exceed the performance of gloss-supervised approaches. By effectively leveraging pose cues and cross-modal attention, the proposed method successfully captures the fine-grained motion and long-range temporal patterns critical for translation without the prohibitive cost of gloss annotation. The work validates the effectiveness of integrating structured prompting and contrastive alignment within a multimodal LLM framework for sign language translation. Future work is suggested to explore additional modalities and lightweight pose representations for real-time scalability.