💻 computer science

Lightweight Real-Time RGB-IR Pedestrian Detection Based on DoG Detail Enhancement and Region-Guided Attention

This paper proposes DRGF-YOLOv10n, a lightweight real-time RGB-IR pedestrian detection method that integrates a DualBranchStem with DoG detail enhancement and Region-Guided Spatial Attention to improve accuracy and robustness in complex environments while maintaining high inference speed.

Original authors: LinXi Zhong, Zhaohui Xiao

Published 2026-10-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: LinXi Zhong, Zhaohui Xiao

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of computer vision, machines are constantly learning to see, but they often struggle with the same limitations that plague human eyes in difficult conditions. A camera relying solely on visible light, the kind we use every day, can be easily confused by darkness, heavy rain, or blinding glare, causing it to miss people standing in the shadows. Conversely, cameras that detect heat, known as infrared sensors, can spot a warm body in the pitch black, but they often fail to show what that body looks like, rendering it as a blurry, textureless blob. To solve this, researchers have long tried to combine these two types of vision, hoping to merge the sharp details of a standard photo with the heat-sensing power of a thermal camera. The challenge lies in teaching a computer to blend these two very different streams of information without losing the specific strengths of either, all while keeping the system fast enough to run in real time on a laptop or a security camera.

A team of researchers at the Chongqing University of Technology has developed a new approach to this problem, creating a system designed to spot pedestrians with high reliability even in messy, complex environments. Their method, built upon a lightweight detection framework, treats the visible-light image and the heat image as two separate streams of information right from the start, rather than smushing them together immediately. They found that by keeping these two views distinct for the initial processing, the computer can better understand the texture and edges of a person from the visible light while simultaneously tracking their heat signature. This dual-stream strategy prevents the system from getting confused by the differences between the two types of images, allowing it to build a much clearer picture of who is in the scene.

To make the visible-light image even more useful, the researchers added a specific step before the computer even begins its main analysis. They applied a technique that acts like a sharpening filter, specifically designed to highlight the outlines and edges of objects. This process makes the fine details of a person's clothing or the curve of a shoulder stand out more clearly against a cluttered background, compensating for the fact that the heat camera often lacks these fine details. By feeding this enhanced, sharper image into the system alongside the raw heat data, the computer gains a richer set of clues to work with, making it easier to distinguish a real person from a confusing patch of background noise.

The system also includes a smart focusing mechanism that kicks in after the computer has processed the images to a certain depth. Instead of trying to pay attention to every single part of the image equally, which would waste computing power, this mechanism learns to ignore irrelevant background details like trees, fences, or shadows. It concentrates its energy specifically on the areas where a person is likely to be, effectively turning up the volume on the pedestrian and turning down the volume on the rest of the world. This selective attention helps the system avoid false alarms, ensuring that when it signals a detection, it is highly confident that a person is actually there.

When tested on a standard dataset containing paired visible and infrared images of pedestrians, the new system showed a significant improvement in precision compared to previous methods. It achieved a precision score of 0.9204, meaning that when the system said it saw a person, it was correct more than 92 percent of the time. While the system did miss some distant or very small targets, resulting in a lower rate of catching every single person, its ability to avoid false alarms was its strongest suit. The researchers noted that the system runs at a speed of roughly 31 frames per second on a standard laptop graphics card, which is fast enough for real-time video monitoring.

The authors suggest that this tool is not necessarily meant to be the first line of defense in a security system, but rather a highly reliable second opinion. In a typical surveillance setup, a first detector might scan a video feed and flag many potential targets to ensure nothing is missed, even if it flags some harmless objects by mistake. This new system would then act as a verification step, re-examining those flagged areas with its sharp edge-enhancement and focused attention to confirm which ones are truly people. This two-stage approach allows security systems to maintain high safety standards without being overwhelmed by constant, incorrect alerts, making it particularly valuable for nighttime monitoring or areas where false alarms are costly and disruptive.

Technical Summary: Lightweight Real-Time RGB-IR Pedestrian Detection Based on DoG Detail Enhancement and Region-Guided Attention

Problem Statement
Pedestrian detection is critical for intelligent surveillance and autonomous driving but faces significant challenges in complex environments. Single-modal RGB detection struggles with low illumination, backlighting, and cluttered backgrounds, while single-modal Infrared (IR) detection lacks texture and semantic boundary details despite its thermal-saliency advantages. Existing multimodal fusion approaches often rely on heavy architectures or simple channel concatenation, which may fail to preserve modality-specific features or meet real-time inference requirements. Furthermore, standard attention mechanisms can be computationally expensive or may inadvertently suppress useful local details when applied too early in the network.

Methodology
The paper proposes DRGF-YOLOv10n, a lightweight, real-time pedestrian detection framework built upon the YOLOv10n architecture. The method integrates three core components to address the limitations of single-modal detection and complex fusion strategies:

DualBranchStem Input Structure:
Instead of concatenating RGB and IR channels into a single six-channel input for standard convolution, the authors introduce a DualBranchStem. This module processes RGB and IR images through separate parallel branches at the initial stage. The RGB branch extracts texture, edge, and contour features, while the IR branch extracts thermal-saliency information. These features are fused only after initial extraction, preserving the distinct characteristics of each modality before combining them for the backbone network.
DoG Detail Enhancement (Preprocessing):
To compensate for the limited texture and boundary details in IR images and to strengthen the RGB branch's perception of weak targets, a Difference of Gaussian (DoG) operation is applied during the RGB data preprocessing stage.
- The operation is formulated as a residual enhancement: $I_{rgb\_dog} = I_{rgb} + \alpha \cdot (G_{\sigma1}(I_{rgb}) - G_{\sigma2}(I_{rgb}))$ , where $\alpha = 0.025$ .
- This step highlights high-frequency edges and local structural variations without altering the thermal-saliency characteristics of the IR input. It is applied as a fixed preprocessing step, not a learnable layer within the network.
Region-Guided Spatial Attention (RG-SA):
A lightweight Region-Guided Spatial Attention (RG-SA) module is embedded into the eighth layer of the YOLOv10n backbone (a deep semantic stage).
- Mechanism: The module utilizes a Region Guidance Module (RGM) to compress input features into low-resolution regional features ( $X_r$ ). It then generates Query ( $Q$ ) from the original high-resolution features and Key/Value ( $K, V$ ) from the regional features.
- Rationale: By using low-resolution regional features for $K$ and $V$ , the module reduces computational complexity compared to global self-attention. Placing it at the deep layer ensures the attention mechanism operates on rich semantic information rather than shallow texture noise, effectively strengthening pedestrian-related regions while suppressing background interference.

Key Contributions
The authors summarize their contributions as follows:

DualBranchStem: A dual-branch input structure that separately extracts RGB texture-edge and IR thermal-saliency features, mitigating the limitations of single-modal detection and avoiding the pitfalls of early shallow-feature mixing.
RG-SA Integration: The embedding of a lightweight region-guided attention mechanism in the deep RGB branch to enhance pedestrian-region responses and suppress background interference.
DoG Detail Enhancement: The introduction of a DoG-based preprocessing strategy for the RGB branch to reinforce local edges and contour details, addressing detail degradation caused by complex backgrounds.

Experimental Results
Experiments were conducted on the CVC-14 RGB-IR pedestrian dataset using an NVIDIA RTX 5060 Laptop GPU.

Performance: The proposed DRGF-YOLOv10n achieved a Precision of 0.9204, Recall of 0.1912, mAP50 of 0.5231, and mAP50-95 of 0.2361.
Comparisons: The model outperformed single-modal baselines (RGB-only and IR-only) and a simple six-channel concatenation baseline. It also demonstrated superior Precision and mAP compared to other fusion strategies like CBAM-6ch and SE-6ch Fusion, despite having a smaller model size (32.01 MB).
Efficiency: The model maintains real-time performance with 2.894M parameters, 1.908 GFLOPs, and an inference latency of 31.47 ms (approx. 31.78 FPS), satisfying the 25–30 FPS requirement for video surveillance.
Ablation: Removing strong data augmentations (e.g., Mosaic, MixUp) was found to be crucial, as they could amplify local background textures and counteract the benefits of DoG enhancement.

Significance and Positioning
The paper explicitly positions DRGF-YOLOv10n not as a standalone high-recall detector for the entire detection pipeline, but as a post-stage high-confidence verification module for false-alarm-sensitive surveillance systems.

High Precision, Lower Recall: The model exhibits high precision but lower recall, indicating a tendency to output only highly confident targets. This characteristic is leveraged to filter false positives generated by a front-stage detector (which uses a lower threshold to ensure high recall).
Application Scenario: The authors suggest a cascaded deployment where a front-stage detector identifies candidate regions, and DRGF-YOLOv10n performs secondary verification to filter out false alarms caused by shadows, heat sources, or non-human objects. This approach is particularly suited for scenarios where false alarms are costly, such as industrial park intrusion detection, nighttime warehouse monitoring, and perimeter security.
Future Outlook: The authors acknowledge that the model still struggles with distant, small-scale, or heavily occluded targets. Future work may focus on improving recall through better cross-modal alignment and dynamic fusion strategies, but the current contribution is framed as a robust solution for refining detection reliability in complex, real-time environments.

Technical Summary: Lightweight Real-Time RGB-IR Pedestrian Detection Based on DoG Detail Enhancement and Region-Guided Attention

More like this