Proper Body Landmark Subset Enables More Accurate and 5X Faster Recognition of Isolated Signs in LIBRAS

This paper demonstrates that selecting an optimal subset of body landmarks combined with spline-based imputation enables isolated Brazilian Sign Language (LIBRAS) recognition that is both 5 times faster and as accurate as state-of-the-art methods, overcoming the speed-accuracy trade-off of previous OpenPose-based approaches.

The Gist

Imagine you are trying to teach a computer to understand Brazilian Sign Language (LIBRAS). The computer needs to "see" a person signing and figure out what word they are saying.

For a long time, the best way to do this was like hiring a super-precise, but incredibly slow, robot architect. This robot (called OpenPose) would scan a video of a signer and draw a detailed skeleton of every single joint, finger, and facial feature. It was so detailed it had over 500 points of data per frame. The problem? It took so long to draw these skeletons that the system couldn't keep up with real-time conversation. It was like trying to win a race while carrying a heavy backpack full of bricks.

Then, someone suggested using a lightweight, fast drone instead (called MediaPipe). This drone could draw the skeleton in a flash—5 times faster! But there was a catch: when the researchers just swapped the slow robot for the fast drone, the computer got confused. The accuracy dropped because the drone was too "noisy" and included too much irrelevant detail (like every tiny wrinkle on a face) that distracted the computer.

This paper is the story of how the researchers fixed the drone to make it both fast AND smart.

Here is the breakdown of their solution using simple analogies:

1. The "Needle in a Haystack" Problem

The fast drone (MediaPipe) gives you 543 points of data. Imagine you are looking for a specific word in a book, but the book is filled with 500 pages of random noise and only 50 pages of actual story. The computer gets overwhelmed trying to read the noise.

The researchers realized they didn't need all the data. They needed to pick the right pages.

2. The "Curated Playlist" Strategy

The team tested five different ways to select which body parts to focus on. Think of it like making a playlist for a party:

• The "Everything" Playlist: Includes every song ever made (too long, boring).
• The "Face-Only" Playlist: Only songs about emotions (misses the dance moves).
• The "ASL-2nd" Playlist (The Winner): They found a specific mix of body parts that worked best. It focused on the hands (where the signs happen), the mouth (for facial expressions), and the shoulders/arms (for posture). They threw away the rest of the "noise."

The Result: By using this "curated playlist" of body landmarks, the computer understood the signs just as well as the slow, heavy robot, but it did it with much less data.

3. The "Auto-Correct" Feature

Sometimes, the fast drone gets distracted by bad lighting or a hand moving too fast, and it "loses" a landmark (a point disappears from the data). It's like a text message where a few letters get dropped.

The researchers added a Spline Imputation step. Think of this as a smart "Auto-Correct" for the computer. If a point disappears for a split second, the computer looks at where the point was before and where it will be after, then draws a smooth, logical line to fill in the gap. This made the system much more reliable, boosting accuracy significantly.

4. The Final Scorecard

By combining the curated body parts (ASL-2nd strategy) with the smart auto-correct, they achieved a massive win:

• Speed: The system is now 5 times faster than the previous best method. It can process signs almost in real-time.
• Accuracy: It is just as accurate (or even better) than the slow, heavy methods.
• Efficiency: It uses a lightweight tool (MediaPipe) that can run on regular computers, not just supercomputers.

The Big Picture

Imagine you want to recognize a friend waving at you from across a busy street.

• The Old Way: You stop, pull out a microscope, measure the exact angle of every hair on their head, and calculate the wind speed. It's accurate, but you miss the wave because you took too long.
• The New Way: You quickly spot their hand, their face, and their shoulders. You ignore the background noise. You fill in the gaps if they blink. You recognize the wave instantly.

This paper proves that for teaching computers sign language, less is often more. You don't need to see everything; you just need to see the right things, and you need to be fast enough to keep up with the conversation.

Technical Summary

Here is a detailed technical summary of the paper "Proper Body Landmark Subset Enables More Accurate and 5X Faster Recognition of Isolated Signs in LIBRAS."

1. Problem Statement

The paper addresses the challenge of Isolated Sign Language Recognition (ISLR) for Brazilian Sign Language (LIBRAS). While deep learning approaches using skeleton representations have achieved high accuracy, they often rely on heavy landmark extraction frameworks like OpenPose, which incur significant computational costs and hinder real-time application.

Conversely, lightweight frameworks like MediaPipe offer superior speed but, when used with the full set of available landmarks, result in a dramatic drop in recognition accuracy. The core problem is how to leverage the speed of MediaPipe without sacrificing the accuracy typically associated with more complex pipelines or full landmark sets.

2. Methodology

The authors propose a streamlined ISLR framework that optimizes the trade-off between speed and accuracy through three main components:

A. Landmark Extraction & Subset Selection

Instead of using all 543 landmarks provided by the MediaPipe Holistic model (468 face, 33 pose, 42 hands), the authors evaluated five distinct landmark subset strategies to reduce dimensionality and noise:

1. Baseline (All): Uses all 543 landmarks.
2. Laines: 68 landmarks focusing on lips, shoulders, and hands.
3. Arcanjo: 75 landmarks (Pose + Hands), excluding dense facial points.
4. ASL-1st: 118 landmarks (Face + Hands), excluding body pose.
5. ASL-2nd: 80 landmarks (Lips, Hands, and Pose), derived from the runner-up solution of the Google ASL Signs Challenge.

B. Spline-Based Imputation

To address the instability of MediaPipe (which can cause missing or noisy landmark detections), the authors implemented a spline-based imputation technique.

• Mechanism: Missing landmark coordinates are treated as time-series data. The system uses cubic spline interpolation (or linear interpolation if fewer than four points are available) within a short temporal window (5 frames) to reconstruct missing trajectories.
• Goal: To ensure continuous, kinematically plausible trajectories before encoding.

C. Image Encoding & Classification

• Encoding: Processed landmarks are converted into 2-D skeleton images using the Skeleton-DML method. This encodes spatial structure and temporal dynamics into a 3-channel image format.
• Classification: The encoded images are resized to $224 \times 224$ and fed into a ResNet-18 classifier (initialized with ImageNet weights).
• Training Protocol: The model uses a Nested Leave-One-Person-Out (LOPO) cross-validation strategy to ensure robustness against signer variability. Data augmentation (rotation, zoom, flip) is applied on-the-fly.

3. Key Contributions

1. Optimized Landmark Subset: Demonstrated that a specific subset of landmarks (ASL-2nd) significantly outperforms using the full landmark set when using lightweight detectors.
2. Imputation Efficacy: Proved that spline-based imputation is critical for mitigating MediaPipe's detection instability, yielding substantial accuracy gains (up to 18 percentage points on certain datasets).
3. Speed-Accuracy Trade-off: Achieved state-of-the-art (SOTA) recognition performance while drastically reducing computational overhead compared to OpenPose-based pipelines.
4. Open Benchmarking: Provided a rigorous comparison against existing SOTA methods using standard LIBRAS datasets.

4. Experimental Results

The study was evaluated on two datasets: MINDS-Libras (1,155 videos, 20 signs) and LIBRAS-UFOP (3,040 videos, 56 signs).

Accuracy Performance

• Best Strategy: The ASL-2nd subset (80 landmarks) achieved the best results.
  • MINDS-Libras: 94% Accuracy, 94% F1-score.
  • LIBRAS-UFOP: 91% Accuracy, 91% F1-score.
• Comparison: The proposed method outperformed the previous SOTA work by Alves et al. [3] (which used OpenPose) on the LIBRAS-UFOP dataset (+11% F1-score improvement) and matched or exceeded performance on MINDS-Libras.
• Impact of Imputation: Spline imputation improved F1-scores by 4 to 18 percentage points across all subsets, confirming that post-processing is essential for lightweight detectors.

Speed Performance

• Landmark Extraction: The MediaPipe-based pipeline was ~6.7x faster than the OpenPose counterpart (4.4s vs. 28.7s per video).
• End-to-End Pipeline: Including inference time, the proposed solution achieved a ~5.6x speed-up overall.

5. Significance and Conclusion

This paper demonstrates that high accuracy in Sign Language Recognition does not require heavy computational resources if the input data is properly curated. By selecting a proper subset of landmarks and applying spline-based imputation, the authors successfully replaced the computationally expensive OpenPose with the lightweight MediaPipe.

Key Takeaways:

• Efficiency: The solution enables near real-time ISLR (5x faster), making it viable for deployment on edge devices or mobile platforms.
• Robustness: The combination of subset selection and imputation makes the system robust to the noise inherent in lightweight detectors.
• Generalizability: The findings suggest that for isolated sign recognition, global body pose is less critical than specific articulators (hands, lips, and key pose joints), allowing for significant dimensionality reduction without performance loss.

The authors conclude that this approach bridges the gap between high-accuracy research models and practical, scalable applications for the deaf and hard-of-hearing community. Future work aims to extend this to continuous sign language recognition and other sign languages.