💻 computer science

Novel Human-Machine Interface via a Robust Hand Gesture Recognition System Using a Channel-Pruned You Only Look Once Version 5 Model

This paper presents a robust, real-time hand gesture recognition system for human-machine interfaces that utilizes a channel-pruned and fine-tuned YOLOv5s model to achieve high detection speeds exceeding 60 FPS and superior accuracy, enabling seamless control of multimedia applications like VLC and Spotify.

Original authors: Abir Sen, Rabi Shaw, Tapas Kumar Mishra, Ratnakar Dash

Published 2026-10-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Abir Sen, Rabi Shaw, Tapas Kumar Mishra, Ratnakar Dash

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Human beings have always sought ways to communicate with machines that feel natural, moving beyond the rigid constraints of keyboards and mice. For decades, scientists have explored the idea of letting people control computers with their hands, a field known as hand gesture recognition. This technology relies on cameras to capture images of a person's hand, followed by software that identifies the shape and position of the fingers to understand what the user is trying to say. While this sounds simple, the real world is messy. Lighting changes, backgrounds are cluttered, and hands can be partially hidden, all of which confuse the computer's vision. Furthermore, for these systems to feel truly responsive, they must process information almost instantly, a task that often requires powerful, energy-hungry computers that are difficult to install in everyday devices. The challenge lies in creating a system that is both smart enough to see clearly in difficult conditions and light enough to run quickly on standard hardware.

A team of researchers from institutions in India has addressed this challenge by developing a new way to make gesture control faster and more reliable. They focused on a specific type of artificial intelligence model called YOLO, which stands for "You Only Look Once." This model is designed to spot objects in images very quickly. The researchers started with a small, efficient version of this model and then applied a technique called channel pruning. Imagine a large network of roads where some lanes are rarely used; channel pruning is like closing those empty lanes to reduce traffic congestion and speed up travel, without changing the destination. By systematically removing unnecessary parts of the computer model's internal structure, the team created a streamlined version that is much smaller and faster than the original. They then carefully retrained this simplified model to ensure it did not lose its ability to recognize gestures accurately, even after the reduction in size.

The results of this work show that the simplified model performs exceptionally well. When tested on a collection of hand gesture images, the new system achieved a detection speed of over 60 frames per second. This means the computer can analyze more than 60 different images of a hand every single second, a speed fast enough to make the interaction feel instantaneous to a human user. The system was able to identify thirteen different hand shapes, such as an open palm, a fist, or specific finger counts, with a high degree of accuracy. While the process of removing parts of the model did cause a very slight drop in raw accuracy, the retraining process recovered most of this performance, resulting in a system that offers the best balance between speed and precision. The researchers found that this approach outperformed many other existing methods, including larger and more complex models that take longer to process the same images.

To prove that this technology works in a real-world setting, the team built a working interface that allows users to control media players using only their hands. They connected their gesture recognition system to VLC, a popular video player, and Spotify, a music streaming service. In their demonstration, a user could hold up a specific hand shape to tell the computer to play a song, pause a video, skip to the next track, or adjust the volume. The system responded in mere milliseconds, making the control feel seamless. However, the researchers also encountered a practical hurdle: because the computer was so fast, it sometimes registered a single gesture multiple times in rapid succession, causing the music to skip several songs or the volume to jump up too quickly. To solve this, they introduced a tiny, deliberate pause in the software's reaction time. This small delay ensured that each hand movement was treated as a single, distinct command, preventing the system from overreacting to the user's natural movements.

The study confirms that it is possible to build robust, real-time gesture control systems without needing massive computing power. By refining the internal structure of the artificial intelligence model, the researchers created a tool that is both lightweight and highly effective. This work suggests that in the near future, users might be able to control their entertainment systems, smart home devices, or other applications with simple hand movements, even in rooms with varying light or busy backgrounds. The researchers plan to expand this work in the future to include dynamic gestures, where the movement of the hand matters as much as its shape, and to combine hand signals with other inputs like eye tracking. For now, they have demonstrated a clear path toward making human-machine interaction more intuitive and responsive, turning the complex task of computer vision into a simple, reliable experience for everyday users.

Technical Summary: Novel Human-Machine Interface via a Robust Hand Gesture Recognition System

Problem Statement
Hand gesture recognition (HGR) is a critical modality for enhancing Human-Computer Interaction (HCI), particularly in multimedia applications like virtual reality, gaming, and smart home automation. However, existing systems face significant challenges in real-world scenarios, including complex backgrounds, varying illumination conditions, and occlusion. Furthermore, there is a need for lightweight deep learning models that can perform gesture detection and classification with rapid inference speeds (milliseconds) and high frame rates (frames per second) to enable responsive, real-time control of applications. Current solutions often struggle to balance model accuracy with computational efficiency, leading to latency or reduced robustness in dynamic environments.

Methodology
The authors propose a robust HGR system centered on a channel-pruned You Only Look Once version 5 small (YOLOv5s) model. The methodology follows a five-stage framework:

Baseline Selection: The YOLOv5s model was selected as the baseline due to its compact size and fast inference speed compared to newer, more complex versions (e.g., YOLOv7–v11). The model was initialized with pre-trained weights from the COCO dataset and fine-tuned on gesture data (scaled to 480×480 pixels) using an SGD optimizer.
Channel Pruning: To reduce model complexity and size, a channel-pruning algorithm was applied. This involved:
- Sparse Training: Introducing scaling factors ( $\gamma$ ) to Batch Normalization (BN) layers and applying an L1 norm regularization term to the loss function. This encourages scaling factors to approach zero, identifying less important channels.
- Pruning Execution: Channels with scaling factors below a calculated threshold (determined by a specific pruning rate) were eliminated. The study tested pruning rates of 10%, 15%, and 20%.
- Fine-Tuning: Recognizing that pruning causes a slight drop in accuracy due to the size-accuracy trade-off, the pruned models were retrained (fine-tuned) with specific hyperparameters (batch size 32, 60 epochs) to recover detection performance.
HMI Implementation: The optimized, lightweight model was deployed to control multimedia applications (VLC media player and Spotify). Gesture classes were mapped to keyboard events (e.g., 'Ok' for play, 'Two' for next track) using the playerctl command-line interface.
Robustness Enhancement: To address issues where high frame rates caused continuous functions (like volume adjustment or track skipping) to trigger multiple times in rapid succession, a time.sleep() function was incorporated. This introduced a deliberate delay to synchronize gesture mapping with application actions, preventing unintended rapid-fire commands.

Key Contributions
The paper outlines five primary contributions:

Fine-tuning: Optimization of the YOLOv5s model using specific hyperparameter settings for real-time hand gesture detection and classification.
Model Compression: Application of a channel-pruning algorithm to the YOLOv5s model to create a more compact architecture with accelerated inference speeds.
Accuracy Recovery: A retraining strategy to mitigate the accuracy loss typically associated with model pruning, ensuring high detection rates despite reduced parameter counts.
Real-Time HMI Development: Construction of a gesture-controlled interface for VLC and Spotify players that leverages the pruned model for low-latency, high-speed operation.
Robustness Improvement: Enhancement of the HMI's reliability in real-time scenarios through synchronization mechanisms that manage continuous function triggers.

Experimental Results
The system was evaluated on two datasets: the private NITR-HGR dataset (13 gesture classes) and the public ASL dataset (29 classes).

Performance vs. Baseline: On the NITR-HGR dataset, the 10% pruned YOLOv5s model achieved a 99.65% F-score and 92.50% mAP (50-95), matching the baseline YOLOv5s performance while reducing the model size from 14.5 MB to 12 MB and inference time from 7.1 ms to 6.2 ms. The study noted that pruning rates of 10%, 15%, and 20% resulted in GFLOPS reductions of 13%, 18%, and 25%, respectively, compared to the baseline.
Comparison with State-of-the-Art: The proposed model outperformed 15 other object detection models (including Faster R-CNN, SSD, YOLOv3, and various YOLOv5–v11 variants) in terms of precision, recall, and mAP. Notably, it achieved a higher F-score (99.65%) and mAP (92.50%) than the larger YOLOv7 model (99.00% F-score, 92.45% mAP) while being significantly smaller (12 MB vs. 74.9 MB).
Speed: The system achieved an average detection speed exceeding 60 frames per second (fps), meeting the requirements for real-time application control.
HMI Performance: In real-time tests controlling VLC and Spotify, the system demonstrated high detection rates (up to 100% for discrete actions like Play/Pause) and average response times ranging from 5.0 ms to 12.9 ms. The introduction of the sleep function successfully mitigated the issue of continuous commands triggering multiple times.

Significance and Claims
The authors claim that their work provides a lightweight, robust, and efficient solution for real-time gesture-controlled interfaces. By successfully balancing the trade-off between model size and accuracy through channel pruning and fine-tuning, the system enables the deployment of high-performance gesture recognition on resource-constrained environments. The paper asserts that the proposed method improves the robustness of gesture-controlled applications against complex backgrounds and lighting variations while maintaining the rapid response times necessary for seamless user interaction. The authors conclude that this approach offers a viable alternative to heavier deep learning models for real-time multimedia control, with future work planned to expand into dynamic gesture recognition and multimodal fusion (e.g., eye-gaze and facial expressions).

Technical Summary: Novel Human-Machine Interface via a Robust Hand Gesture Recognition System

More like this