🤖 AI

Dynamic LIBRAS Gesture Recognition via CNN over Spatiotemporal Matrix Representation

This paper presents a real-time LIBRAS gesture recognition system for home automation that combines MediaPipe hand landmark extraction with a CNN classifier operating on a spatiotemporal matrix, achieving up to 95% accuracy across 11 gesture classes without relying on recurrent neural networks.

Original authors: Jasmine Moreira

Published 2026-03-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jasmine Moreira

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to control your home's lights, curtains, or air conditioner just by waving your hands, like a wizard casting spells. But instead of magic, you're using Brazilian Sign Language (LIBRAS). The problem is, computers aren't naturally good at understanding the fluid, moving dance of human hands. They often get confused by how fast you move or how your hand looks in different lights.

This paper describes a clever "magic trick" (a computer program) that solves this problem. Here is how it works, broken down into simple steps:

1. The Skeleton Key (MediaPipe)

First, the computer needs to see your hand, but not the "flesh and skin" part. Imagine looking at a hand and seeing only the stick-figure skeleton underneath.

The Tool: The researchers use a tool called MediaPipe. Think of it as a super-accurate X-ray vision that instantly finds 21 specific "joints" on your hand (fingertips, knuckles, wrist).
The Result: Instead of a blurry photo, the computer gets a clean list of coordinates: "Thumb is here, index finger is there." This makes the system ignore bad lighting or skin color and focus only on the shape of the hand.

2. The "Movie Strip" (Spatiotemporal Matrix)

Static gestures (holding a hand still) are easy. But dynamic gestures (moving your hand to draw a letter in the air) are hard because they happen over time.

The Analogy: Imagine taking a movie of your hand moving. Instead of storing 30 separate video frames, the computer stacks them on top of each other to create a single, tall, skinny picture.
How it works: They take 30 frames of your hand moving. They line up the X, Y, and Z coordinates of all 21 joints from those 30 frames.
- If your hand stays still, the picture looks like straight, solid lines.
- If your hand moves (like drawing the letter "J"), the picture looks like a wavy, colorful scribble.
The Magic: The computer treats this "movie strip" just like a regular photo. It doesn't need complex time-traveling math (recurrent networks); it just looks at the pattern of the lines.

3. The "Sliding Window" (The Continuous Stream)

In real life, you don't stop and wait for the computer to say "Okay, I saw that." You keep moving.

The Problem: If the computer only checks a chunk of video every 30 frames, it might catch a gesture halfway through and miss it.
The Solution: The researchers use a Sliding Window. Imagine a conveyor belt carrying your hand movements.
- Every time a new frame comes in, the computer doesn't just look at it once. It triples it (puts three copies in the buffer).
- This creates a "slow-motion" effect for the computer, giving it more time to analyze the movement without needing a super-fast processor.
- As new frames arrive, the oldest ones slide off the back, keeping the view continuous.

4. The Brain (The CNN)

Once the "movie strip" is ready, it gets fed into a Convolutional Neural Network (CNN).

The Analogy: Think of this as a very smart, tiny detective. It has been trained on thousands of these "movie strips."
The Training: The detective learned that a straight vertical line means "Letter A" (turn on AC), while a wavy loop means "Letter J" (open window).
The Size: This detective is very small and efficient (only about 25,000 "thoughts" or parameters), meaning it can run on a regular laptop without needing a massive supercomputer.

The Results: How well does it work?

The team tested this system in a home setting with 11 different commands (like turning lights on/off or changing colors).

In the dark: It got it right 95% of the time.
In normal light: It got it right 92% of the time.
The Catch: The system was tested mostly by one person. While it works great for them, the authors admit it needs to be tested on many different people (different hand sizes, different speeds) to make sure it works for everyone.

Summary

This paper presents a smart, lightweight way to turn hand gestures into home control commands. It turns a moving hand into a static picture of movement, uses a small, efficient brain to read that picture, and uses a sliding window to keep the conversation flowing in real-time. It's a step toward making our homes respond to the simple wave of a hand, just like in the movies.

1. Problem Statement

The paper addresses the challenge of dynamic hand gesture recognition for Human-Machine Interfaces (HMI), specifically for controlling home automation devices using LIBRAS (Brazilian Sign Language).

Core Difficulty: Dynamic gestures involve movement over time, making them harder to recognize than static poses. Variations in user anatomy, expression rhythm, and finger positioning create significant generalization challenges.
Limitations of Existing Approaches:
- Raw Pixel Approaches: High dimensionality and sensitivity to lighting and skin color.
- Recurrent Neural Networks (RNNs/LSTMs): While effective for temporal data, they often increase computational complexity and training difficulty.
- Generalization: Many existing models perform well on controlled datasets but struggle with diverse users and real-world lighting conditions.

2. Methodology

The proposed system is a two-stage pipeline that converts video input into a classification task using a lightweight 2D Convolutional Neural Network (CNN).

A. Data Acquisition & Preprocessing

Keypoint Extraction: The system uses MediaPipe Hand Landmarker to extract 21 skeletal keypoints (fingertips, joints, palm center) from video frames. This abstracts the hand into geometric coordinates $(x, y, z)$ , reducing dimensionality and increasing robustness to lighting changes.
Spatiotemporal Matrix Representation:
- Instead of processing raw video or using RNNs, the authors organize the data into a $90 \times 21$ matrix.
- Structure:
  - 21 Columns: Represent the 21 hand keypoints.
  - 90 Rows: Represent the temporal evolution. For a gesture captured over 30 frames, the coordinates for the $x$ , $y$ , and $z$ axes are vertically concatenated (Rows 1–30: $x$ values; Rows 31–60: $y$ values; Rows 61–90: $z$ values).
- Normalization: Matrices are normalized via min-max scaling to the range $[0, 255]$ and converted to grayscale images. This allows the CNN to treat the temporal sequence as a spatial image pattern, making the model invariant to hand scale and distance.

B. Model Architecture

Classifier: A custom 2D CNN designed to classify the $90 \times 21$ "motion images."
Configuration:
- Layers: 3 Convolutional layers (33, 64, and 64 filters respectively) followed by 2 Dense layers.
- Parameters: Approximately 25,000 parameters, optimized for low computational cost.
- Regularization: L1 regularization ( $\lambda=0.0001$ ) applied to prevent overfitting.
- Optimizer: Adam with a learning rate of $0.000001$.
Training Data: 1,254 gestures (30 frames each) across 11 classes (e.g., Air Conditioning, Curtains, Toggle On/Off, Light Color Changes, and a "Neutral" class to prevent false positives).

C. Inference Strategy: Sliding Window with Frame Triplication

To enable continuous real-time recognition without RNNs, the authors implemented a specific buffering strategy:

Buffering: A sliding window of 30 entries is maintained.
Temporal Triplication: Every new real frame captured is inserted 3 times ( $n=3$ ) into the buffer. This effectively slows down the temporal resolution, allowing the system to capture dynamic gestures even if the user moves quickly.
Overlap: As new frames enter, the oldest 3 are removed, ensuring continuous overlap.
Confidence Threshold: Predictions are only accepted if the model confidence exceeds 98%.
State Management: The buffer is cleared upon successful gesture execution or when hands leave the scene to prevent repeated triggers.

3. Key Contributions

Spatiotemporal Matrix Representation: A novel method of encoding skeletal keypoints into a $90 \times 21$ matrix, allowing 2D CNNs to classify dynamic gestures by treating time as a spatial dimension.
RNN-Free Real-Time Inference: The use of a sliding window with temporal frame triplication enables robust continuous recognition on general-purpose hardware without the complexity of Recurrent Neural Networks (LSTMs/GRUs).
Lightweight Architecture: A highly efficient model (~25k parameters) capable of running on standard CPUs (Intel Core i7) with low latency, suitable for embedded or home automation scenarios.
Validation in Home Automation: Successful application to a real-world scenario controlling 11 distinct device actions via LIBRAS gestures.

4. Experimental Results

Accuracy:
- Low-light conditions: 95% accuracy (95/100 correct).
- Normal lighting: 92% accuracy (92/100 correct).
Performance: The system achieved these results on a standard PC without a dedicated GPU.
Error Analysis:
- The most common error was the letter "C" being confused with relaxed/neutral hand positions.
- Dynamic gestures like "J" (which involves rotation) were recognized effectively, though there is a noted limitation where subsets of movements (e.g., "I" inside "J") can trigger premature recognition.
Comparison: Results are comparable to state-of-the-art methods (e.g., 93% on MINDS-Libras using skeletal images, 92.57% in home automation control), despite using a simpler architecture.

5. Significance and Future Work

Significance: The paper demonstrates that complex temporal gesture recognition can be achieved using simple 2D CNNs and clever data structuring, removing the dependency on heavy recurrent networks. This makes the technology accessible for low-cost, real-time home automation.
Limitations:
- Generalization: The current model was trained and tested on a single user. While it achieved high accuracy, the paper acknowledges that anatomical diversity (different hand sizes, skin tones, expression styles) is a critical factor for real-world deployment.
- Dataset Size: The dataset is relatively small compared to large-scale benchmarks.
Future Directions:
- Expanding the dataset to include multiple users with diverse anatomical characteristics.
- Incorporating variable lighting conditions explicitly into the training process.
- Porting the system to low-cost embedded hardware (e.g., Raspberry Pi, mobile devices).
- Expanding the gesture vocabulary beyond the current LIBRAS subset.

In conclusion, this work presents a highly efficient, lightweight solution for dynamic sign language recognition that bridges the gap between academic deep learning models and practical, real-time home automation applications.