Development of ML model for triboelectric nanogenerator based sign language detection system

This paper presents a TENG-based sensor glove for sign language recognition that achieves 93.33% accuracy using a novel MFCC CNN-LSTM architecture, significantly outperforming traditional machine learning and time-domain deep learning models by leveraging frequency-domain features and multi-sensor fusion.

The Gist

Imagine trying to have a conversation with someone who speaks a language you don't know, but instead of using words, they use their hands. For the deaf and hard-of-hearing community, sign language is their voice. But for the hearing world, understanding those hand movements is like trying to read a book written in invisible ink—it's hard to see, easy to miss, and often blocked by obstacles like trees, walls, or bad lighting.

This paper introduces a new "translator" that doesn't rely on cameras at all. Instead, it uses a high-tech glove that "feels" the signs.

Here is the story of how they built it, explained simply:

1. The Glove: A "Magic" Fabric

The researchers didn't just put sensors on a glove; they grew their own sensors.

• The Analogy: Think of the glove fingers as tiny, invisible springs made of a special material called Zinc Oxide (ZnO). They grew these crystals on cotton fabric, kind of like how moss grows on a rock.
• How it works: When you bend your finger, the fabric stretches and squishes these tiny springs. This physical pressure creates a tiny electrical spark (voltage). The more you bend, the stronger the spark.
• The Result: A glove that turns finger movements into electrical music notes. They tested it with numbers (1–5) and letters (A–F).

2. The Problem: Too Much Noise, Not Enough Clarity

The glove sends a constant stream of electrical data. But raw data is messy. It's like listening to a radio station with a lot of static.

• The Challenge: If you just look at the raw "squiggly lines" of electricity, it's hard for a computer to tell the difference between a "5" and an "S" because people move at different speeds. One person might sign "5" quickly, another slowly.
• The Old Way: Traditional computer programs (like Random Forest) tried to guess the sign by looking at the squiggles directly. It was like trying to identify a song by looking at the sound waves on a piece of paper without hearing the melody. They got about 70% right.

3. The Solution: The "Super-Translator" (AI)

The team built a smarter brain for the glove using Deep Learning. They didn't just look at the squiggles; they changed the way they listened to them.

Step A: The "Mel-Frequency" Magic (MFCC)

Instead of looking at the raw electricity, they used a trick borrowed from speech recognition (how Siri or Alexa understands your voice).

• The Analogy: Imagine you are trying to recognize a friend's voice in a noisy room. You don't listen to the exact pitch; you listen to the timbre or the "color" of the voice.
• What they did: They converted the electrical signals into a "spectral map" (like a fingerprint of the sound). This map ignores how fast you moved your hand and focuses on what the movement pattern looked like. This made the data much clearer.

Step B: The "Parallel Brain" (CNN-LSTM)

They built a two-part brain to read these maps:

1. The Detective (CNN): This part looks at each finger individually. It says, "Okay, the thumb is doing this specific pattern, and the index finger is doing that." It processes each finger separately, like having five different detectives looking at five different clues.
2. The Storyteller (LSTM): This part takes all the clues from the detectives and puts them together in order. It understands the story of the movement. "First the thumb bent, then the index finger, then the middle..."

4. The Results: A Giant Leap Forward

When they tested this new system:

• Old Way (Traditional Math): Got it right 70% of the time.
• New Way (The AI Glove): Got it right 93.33% of the time.

That is a 23% improvement, which is massive in the world of technology. It means the glove is now reliable enough to be used in real life.

5. The Secret Sauce: "Data Augmentation"

To teach the AI, they didn't just use the data they collected once. They used a technique called Data Augmentation.

• The Analogy: Imagine you are teaching a child to recognize a dog. If you only show them one photo of a Golden Retriever, they might think all dogs look like that. But if you show them photos of dogs running, sleeping, wearing hats, or in the rain, they learn what a dog really is.
• What they did: They took their recorded signs and digitally "warped" them. They made the signs look faster, slower, louder, or noisier. This taught the AI to recognize the sign no matter how the person moved, making the system very tough and reliable.

Why Does This Matter?

This isn't just about better math; it's about connection.

• No Cameras Needed: You can wear this glove in a dark room, in a crowd, or with your back turned. The camera doesn't care about your view, but this glove does.
• Real-World Use: Because it's so accurate, it could be the key to building a device that instantly translates sign language into speech for the hearing world, or speech into sign for the deaf world, breaking down the biggest barrier between these two communities.

In short: They grew a smart fabric, taught a computer to "listen" to finger movements like music, and created a translator that is far more accurate than anything we've had before. It's a small glove with a giant impact.

Technical Summary

1. Problem Statement

Sign language recognition (SLR) is critical for bridging the communication gap between deaf and hearing communities. However, existing solutions face significant limitations:

• Vision-based approaches: Highly sensitive to occlusion, lighting conditions, camera angles, background clutter, and computational complexity. They often require controlled environments, limiting real-world applicability.
• Traditional Sensor-based approaches: While data gloves offer an alternative, classical Machine Learning (ML) algorithms often struggle with complex temporal relationships in time-series data and require extensive manual feature engineering.
• Gap: There is a lack of comprehensive, systematic comparisons between traditional ML, deep learning, and specialized architectures (like CNN-LSTM) specifically for triboelectric nanogenerator (TENG) based wearable sensors.

2. Methodology

A. Hardware: STENG Sensor Fabrication

The study utilizes a custom-fabricated Single Electrode Triboelectric Nanogenerator (STENG) glove.

• Material Synthesis: Zinc Oxide (ZnO) nanorods were grown on a cotton substrate using a chemical method involving sodium hydroxide, zinc acetate, zinc nitrate, and hexamethylenetetramine.
• Device Structure: The sensor consists of a ZnO-coated cotton layer and an aluminum foil counter-layer separated by a spacer. When fingers flex, contact and separation generate voltage signals proportional to the movement.
• Setup: Five STENG sensors were attached to the thumb, index, middle, ring, and little fingers of a nitrile glove, interfaced with an Arduino UNO for data acquisition.

B. Dataset

• Classes: 11 sign language classes (Digits 1–5 and Letters A–F).
• Data Collection: Multivariate time-series data from a single participant performing gestures multiple times.
• Preprocessing:
  • Segmentation: Continuous signals were split into fixed-length, non-overlapping windows.
  • Normalization: StandardScaler for traditional ML; per-sequence normalization for deep learning to remove inter-subject baseline variability.
  • Cleaning: Removal of sequences with excessive zero values (sensor malfunction artifacts).

C. Feature Engineering: MFCC

Instead of using raw time-domain data, the authors converted sensor signals into Mel-Frequency Cepstral Coefficients (MFCCs) to create execution-speed-invariant spectral representations.

• Process: Frame-based processing (32 samples/frame, 75% overlap) $\rightarrow$ FFT $\rightarrow$ Mel-filter bank (40 bands, 0–50 Hz) $\rightarrow$ Log compression $\rightarrow$ Discrete Cosine Transform (DCT).
• Output: A 3D tensor of shape $(5 \text{ sensors}, 22 \text{ frames}, 12 \text{ coefficients})$, resulting in 1,320 features per sequence.

D. Data Augmentation

To enhance generalization, four techniques were applied to raw time-domain data before MFCC extraction:

1. Gaussian Noise Injection: Simulates sensor uncertainty.
2. Time Warping: Non-linear temporal deformation to simulate speed variations.
3. Magnitude Scaling: Adjusts for movement intensity differences.
4. Temporal Shifting: Addresses misalignment in gesture start times.

• Result: Approximate 3x expansion of the training dataset.

E. Model Architectures Evaluated

The study benchmarked four categories of models:

1. Traditional ML: Random Forest, Gradient Boosting, SVM, KNN, Decision Tree, Logistic Regression, Naive Bayes, LDA.
2. Feedforward Neural Networks: SimpleNN, DeepNN, WideNN, UltraNN (various layer depths).
3. Recurrent Models: Standard LSTM and Attention-LSTM (Bidirectional).
4. Proposed Architecture (MFCC CNN-LSTM):
   • Parallel Processing: Five independent Conv1D branches process each sensor's MFCC sequence separately.
   • Fusion: Branch outputs are concatenated and passed through a dense fusion network with LayerNorm and Dropout.
   • Optimization: AdamW optimizer, Focal Loss, and CosineAnnealingWarmRestarts scheduler.

3. Key Contributions

1. Hardware Innovation: Development of a self-powered, ZnO-based TENG glove for gesture recognition, eliminating the need for external power sources.
2. Architectural Novelty: Introduction of a Multi-Sensor MFCC CNN-LSTM architecture that processes frequency-domain features from each sensor in parallel before fusion, outperforming both time-domain deep learning and classical ML.
3. Systematic Ablation Studies: Comprehensive evaluation of:
   • Window Size: Identifying the optimal trade-off between temporal context and data volume.
   • Feature Representation: Demonstrating the superiority of frequency-domain (MFCC) over time-domain features.
   • Augmentation: Proving the necessity of specific augmentation strategies for small datasets.
4. Performance Benchmarking: Providing a rigorous comparison between traditional ML, standard deep learning, and the proposed hybrid model.

4. Experimental Results

• Overall Performance:

  • Proposed Model (MFCC CNN-LSTM): Achieved 93.33% Accuracy and 95.56% Precision.
  • Improvement: This represents a 23 percentage point improvement over the best traditional ML algorithm (Random Forest: 70.38%).
  • Comparison: Outperformed Standard LSTM (84.13%) and Attention-LSTM (82.54%).

• Window Size Analysis:

  • 50-timestep windows: Optimal performance (84.13% for LSTM).
  • 100-timestep windows: Performance dropped significantly to 58.06% due to reduced training chunks (51% fewer samples) and vanishing gradients in longer sequences.

• Traditional ML vs. Deep Learning:

  • Random Forest was the top traditional performer but failed to capture complex temporal dependencies.
  • Simple Feedforward Neural Networks performed poorly (15.86%), highlighting the necessity of recurrent or temporal modeling for sequence data.

• Confusion Matrix: The proposed model showed high precision with minimal false positives, indicating conservative and reliable classification behavior suitable for assistive technology.

5. Significance and Conclusion

• Assistive Technology: The system offers a robust, low-cost, and self-powered solution for real-time sign language translation, overcoming the occlusion and lighting limitations of camera-based systems.
• Methodological Insight: The study proves that frequency-domain feature extraction (MFCC) combined with parallel multi-sensor processing is superior to time-domain approaches for wearable gesture recognition.
• Generalization: The use of data augmentation and specific architectural designs (parallel branches) allows the model to generalize well despite the limited dataset size.
• Future Work: The authors acknowledge the limitation of single-subject data collection. Future research will focus on multi-subject validation (leave-one-subject-out), expanding the vocabulary to full alphabets, and deploying the model on embedded hardware for real-time applications.