💻 computer science

FDMA-Net: Four-directional Mamba alignment for tiny object detection in visible and thermal imagery

To address the challenges of spatial misalignment and limited detail in tiny object detection within paired visible and thermal aerial images, the authors propose FDMA-Net, a novel framework featuring a Four-Directional Mamba Alignment module for cross-modal feature alignment, a Thermal Detail Enhancement mechanism for preserving fine spatial details, and a Center-Size Distribution Head for precise localization, which achieves state-of-the-art performance on RGBTDronePerson and VTUAV-det benchmarks.

Original authors: Sungju An, Siwoo Kim, Jhonghyun An

Published 2026-10-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sungju An, Siwoo Kim, Jhonghyun An

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the high-altitude world of aerial surveillance, seeing clearly is often a matter of having the right kind of eyes. Cameras that capture visible light, similar to what human eyes see, excel at showing texture and color but struggle when the sun sets or clouds obscure the view. Thermal cameras, which detect heat, work well in the dark or through smoke, but they often blur fine details, making it hard to distinguish a small person from the warm ground beneath them. For years, engineers have tried to combine these two views into a single, clearer picture, hoping to get the best of both worlds. However, a persistent problem has kept this fusion from working perfectly: the two cameras rarely see the world from the exact same angle. When a tiny object, like a person walking on a roof, occupies only a few pixels, even a tiny shift between the visible and thermal images can cause the computer to mix up the person with the background. It is like trying to stitch two maps together where the borders do not quite line up; the result is a blurry, confusing mess where the most important details get lost.

A team of researchers at Gachon University in South Korea has developed a new system designed to solve this specific misalignment problem, particularly for finding very small objects in the sky. They call their creation FDMA-Net, a tool that acts like a highly skilled editor, constantly checking and adjusting the position of the visible image to match the thermal one before they are combined. The core of their innovation is a method that looks at the image in four directions—up, down, left, and right—to figure out exactly how much the visible image needs to be shifted. Instead of guessing, the system scans the rows and columns of the image, gathering clues from the surrounding area to decide where a tiny target really is. It then uses a confidence score to weigh these different directional guesses, trusting the most reliable ones to create a precise alignment. This ensures that when the computer fuses the two images, the heat signature of a person and their visible outline land on top of each other perfectly, rather than drifting apart.

To make sure these tiny targets do not disappear during the process, the researchers added a second layer of protection. They realized that the finest details of a thermal image, which are often found in the early stages of processing, were being discarded too early. Their system now keeps these sharp, shallow thermal details and feeds them directly into the final detection stage. This is crucial because tiny objects have very little spatial information to begin with; by preserving these fine thermal textures, the system retains the subtle clues needed to spot a person or a rider that might otherwise blend into the background. Furthermore, the system changes how it predicts where an object is located. Instead of trying to guess a single exact point, it calculates the probability of an object's center being in various nearby spots and its size being within a certain range. This statistical approach allows the system to handle the uncertainty inherent in spotting small, distant targets more effectively than traditional methods that demand a single, rigid answer.

When the researchers tested their new system on two different datasets containing thousands of aerial images, the results were significant. On a dataset focused on detecting people and crowds, the system achieved a detection accuracy of 49.08 percent, which was notably higher than the best previous methods. On a second dataset designed for tracking vehicles and people, it reached an accuracy of 75.7 percent. These improvements were not just marginal; they represented a clear leap forward, surpassing the leading competitors by several percentage points. Perhaps most importantly, the researchers achieved these results with a system that is surprisingly efficient, using fewer computational resources than many of the other top-performing models. By focusing on the precise alignment of images and the preservation of fine details, the team has shown that even the smallest targets can be found reliably, provided the computer knows exactly how to look at them.

The success of this work suggests that the key to seeing the invisible lies not just in having better sensors, but in smarter ways of combining what those sensors see. The researchers found that simply stacking images together is not enough; the system must actively understand the relationship between the two views and correct for their differences in real time. By using a method that looks in all directions to find the truth and by holding onto the faintest thermal details, the system overcomes the confusion that usually plagues tiny object detection. This approach offers a practical path forward for aerial surveillance, search and rescue operations, and any situation where spotting a small, distant object in the dark or through haze is a matter of safety and success. The work demonstrates that with the right alignment and a careful eye for detail, the limitations of current technology can be pushed back, revealing a clearer picture of the world from above.

Technical Summary: FDMA-Net

Problem Statement

Detecting tiny objects in paired visible and thermal aerial imagery presents two primary challenges. First, spatial misalignment between modalities weakens cross-modal correspondence; when a target occupies only a few feature locations, a displacement between the visible and thermal images can cause target information in one modality to be incorrectly fused with background information from the other. Second, tiny targets provide limited spatial detail, making it difficult for standard detectors to distinguish them from complex backgrounds. Existing multispectral detectors often struggle to estimate reliable sampling displacements when cross-modal cues are weak or ambiguous, particularly for small aerial objects.

Methodology

The authors propose FDMA-Net (Four-Directional Mamba Alignment Network), a detection framework built upon a GFL-based dense detection architecture with two independent YOLOv8m backbones. The network integrates three core components to address alignment, detail preservation, and localization:

1. Four-Directional Mamba Alignment (FDMA)

This module addresses spatial misalignment by estimating sampling displacements for visible features relative to a thermal reference.

Mechanism: It utilizes Selective State-Space Models (Mamba) to process image features. Specifically, it treats each row and column of the cross-modal relation feature map as an independent sequence.
Bidirectional Scanning: The module performs bidirectional scans (forward and reverse) along both horizontal and vertical axes. This allows the model to accumulate spatial context from surrounding areas to estimate displacement, even when local correspondence cues are weak.
Confidence Aggregation: For each direction ( $\rightarrow, \leftarrow, \downarrow, \uparrow$ ), the network predicts a bounded displacement and a confidence score. Opposing directional estimates are aggregated using learned confidence weights to produce a final displacement vector.
Regularization: A Directional Consistency loss is applied during training to encourage agreement between opposing predictions, ensuring the model learns robust displacement estimates.
Fusion: The aggregated displacements guide the bilinear sampling of visible features, which are then concatenated with thermal reference features for fusion.

2. Thermal Detail Enhancement (TDE)

To counteract the loss of fine spatial detail often associated with deep feature extraction, TDE transfers shallow thermal information to the detection pipeline.

Pathway: Shallow thermal features (stride-4 C2) are projected and upsampled via PixelUnshuffle to create a high-resolution tensor.
Application: This tensor is used to enhance the finest detection level (P3) features and provides a "subcell" shortcut directly to the center-prediction branch. This ensures that high-resolution thermal details are available for detecting tiny objects.

3. Center-Size Distribution Head (CSDH)

Instead of regressing continuous bounding box coordinates directly, the CSDH models localization as a discrete probability distribution problem.

Representation: It separately models signed center offsets and full object dimensions using discrete probability distributions (bins).
Decoding: The final bounding box coordinates are derived by computing the probability-weighted expectation of these distributions. This approach separates center displacement from object extent, aiming to improve localization precision for small targets.

Key Contributions

FDMA Module: The introduction of a Four-Directional Mamba Alignment mechanism that uses bidirectional scans and confidence-weighted aggregation to estimate cross-modal sampling displacements, leveraging surrounding spatial context for alignment.
TDE Module: A mechanism to transfer shallow thermal details to the finest detection level and the center-prediction branch, preserving spatial fidelity for tiny objects.
CSDH: A localization head that models center offsets and object dimensions as discrete probability distributions, separating center displacement from object extent.
Performance: The integration of these modules achieves state-of-the-art results on aerial visible-thermal datasets with a relatively low parameter count.

Experimental Results

The authors evaluated FDMA-Net on two datasets: RGBTDronePerson and VTUAV-det.

RGBTDronePerson:
- FDMA-Net achieved a mean AP50 of 49.08%, surpassing the previous best method (COXNet) by 3.63 percentage points.
- It achieved the highest AP25 (62.07%) and Tiny AP50 (50.58%) among all compared methods.
- The model is parameter-efficient, using only 36.74 million parameters, the lowest among the listed detectors.
- Ablation studies confirmed that adding FDMA increased AP50 by 2.86 points, while the full model (including TDE and CSDH) improved AP50 by 5.64 points over a baseline paired-fusion architecture. Directional consistency regularization provided an additional 0.49 point gain in AP50.
VTUAV-det:
- The model achieved an AP50 of 75.7%, exceeding COXNet by 3.9 percentage points.
- Improvements were also noted in small ( $AP_s$ : 19.8%) and medium ( $AP_m$ : 32.7%) object detection categories.

Significance and Claims

The paper claims that FDMA-Net effectively addresses the specific challenges of aerial tiny object detection by combining directional contextual modeling, thermal detail enhancement, and distributional localization.

Robustness to Misalignment: The study demonstrates that using surrounding row and column context via Mamba scans allows for reliable displacement estimation even when tiny targets provide weak correspondence cues.
Parameter Efficiency: The architecture achieves superior performance with the lowest parameter count among comparable detectors, highlighting the efficiency of the integrated design.
Component Synergy: The results support the hypothesis that combining explicit spatial correction (FDMA), high-resolution feature retention (TDE), and probabilistic localization (CSDH) is critical for detecting tiny objects in visible-thermal imagery.

The authors note that while the method improves strict-IoU localization and small-object detection, opportunities remain for refining large-object detection and inference throughput. The work is presented as a step toward more robust cross-modal fusion for aerial surveillance applications.