💻 computer science

TriCLE: Tri-Modal Vision-Language Reasoning for Edge-Deployed Fine-Grained Clustering

TriCLE is an edge-deployable tri-modal vision-language system that synthesizes thermal and depth data from single RGB aircraft images to achieve high-accuracy, engineering-relevant taxonomic clustering under strict memory and compute constraints.

Original authors: Kishor Datta Gupta, Md. Mahfuzur Rahman, Fahad Rahman, Ahmed Rafi Hasan, Faysal Mehrab Chowdhury, Mohd Ariful Haque, Roy George

Published 2026-08-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kishor Datta Gupta, Md. Mahfuzur Rahman, Fahad Rahman, Ahmed Rafi Hasan, Faysal Mehrab Chowdhury, Mohd Ariful Haque, Roy George

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to identify a bird in the sky, but you can only see it for a split second, and your eyesight is a bit foggy. You might guess it's a hawk because of its shape, or a falcon because of its speed, but without seeing its feathers clearly, you could easily be wrong. Now, imagine doing this not with a bird, but with a high-tech airplane, and you have to do it instantly on a tiny computer strapped to a drone that has no internet and very little battery power. This is the kind of puzzle scientists in the field of "edge computing" and "artificial intelligence" are trying to solve. They want machines to be smart enough to understand complex scenes right where the action happens, without needing to send data back to a giant supercomputer in the cloud. To do this, they use "Vision-Language Models," which are like super-smart robots that can look at a picture and talk about what they see, combining visual clues with human-like reasoning. But here's the catch: most of these robots are trained only on regular color photos. In the real world, especially for aircraft, a color photo isn't always enough. Sometimes you need to see the heat coming off the engine (thermal vision) or the 3D shape of the wings (depth vision) to tell two similar-looking planes apart. The big question is: how do you teach a robot to use all these different "senses" when you don't have a camera that can see all of them at once, and you have to run the whole system on a tiny device?

This is where a new project called TriCLE steps in. Think of TriCLE as a clever magician that can conjure up missing senses. The researchers realized that while they didn't have real thermal cameras or 3D laser scanners (LiDAR) for every airplane photo, they could use a single regular color photo to create fake but realistic versions of those other views. It's like looking at a black-and-white sketch and using a smart program to color it in with heat signatures and then sculpting it into a 3D model, all while keeping the original drawing perfectly aligned. They fed this "tri-modal" trio—the real color photo, the fake heat map, and the fake 3D depth map—into a compact AI brain (a 4-billion-parameter model based on Qwen3-VL). But simply showing the AI these three views wasn't enough; they needed to teach it to think like an airplane engineer, grouping planes by their engine type, wing shape, and design era, rather than just by how shiny they look.

To train this AI, the team tried several different teaching methods. They found that the best way to teach the robot was a technique called Group Sequence Policy Optimization (GSPO). Imagine you are teaching a student to write an essay. If you only correct them word-by-word as they type, they might get stuck in a loop, repeating the same phrase forever. But if you let them write the whole paragraph, then grade the entire flow of logic at once, they learn to keep their thoughts coherent and reach a clear conclusion. That is what GSPO does: it looks at the whole reasoning process the AI generates and rewards it for staying on track, avoiding repetitive loops, and arriving at the correct engineering classification.

The results of this experiment were quite promising. When tested on a set of airplane images the AI had never seen before, the GSPO-trained model got the classification right 78.00% of the time. It also managed to keep its output format correct 94.00% of the time, meaning it didn't get confused and spill gibberish. Perhaps most importantly for the "edge" part of the story, the researchers squeezed this smart model down using a technique called 4-bit quantization (basically compressing the AI's memory without losing its smarts). After this compression, the entire system fit into 8GB of memory and could process a complex airplane image in just 1.48 seconds. This suggests that TriCLE is a practical prototype for making drones and small devices smart enough to identify aircraft on the fly. However, the authors are careful to note that because the "heat" and "3D" views were generated by a computer program rather than measured by real sensors, this is a proof-of-concept. The system works well in these controlled simulations, but the team suggests that future work needs to test it with real, synchronized sensor data to see if it holds up in the messy, unpredictable real world.

Technical Summary: TriCLE – Tri-Modal Vision-Language Reasoning for Edge-Deployed Fine-Grained Clustering

1. Problem Statement

The paper addresses the challenge of performing fine-grained aircraft taxonomic grouping on edge platforms (e.g., drones) where resources are severely constrained by limited memory, compute, and intermittent connectivity. Standard RGB-only recognition models and general-purpose Vision-Language Models (VLMs) struggle in this setting for two primary reasons:

Modality Scarcity: High-performance aircraft recognition often benefits from thermal (FLIR) and geometric (LiDAR) data, but aligned RGB–thermal–LiDAR datasets are rarely available.
Edge Constraints: Existing multi-modal perception methods typically assume calibrated sensors and paired annotations, which are difficult to obtain, and they are often too computationally heavy for edge deployment.

The goal is to create a system that can interpret aircraft imagery under these constraints, distinguishing aircraft based on engineering-relevant properties (propulsion, airframe family, size, design era) rather than just surface appearance.

2. Methodology

The proposed system, TriCLE, is an application-oriented, tri-modal VLM pipeline designed to operate from a single RGB input. The methodology proceeds through four distinct phases:

A. Synthetic Tri-Modal Data Generation

Since aligned multi-sensor data is scarce, TriCLE synthesizes the missing modalities from a single RGB image ( $I_{rgb}$ ):

Synthetic Thermal ( $I_{th}$ ): A structure-preserving FLIR-style thermal view is generated using a modified FLUX.1-dev transformer. The generation is guided by ControlNet (conditioned on Canny edges) to maintain airframe geometry and LoRA layers trained on an empirical FLIR dataset to simulate thermal gradients around engines and nacelles.
Pseudo-LiDAR ( $I_{lidar}$ ): A dense monocular depth map is estimated from the synthetic thermal image using a Dense Prediction Transformer (DPT). This depth map is back-projected into a 3D point cloud, which is then converted into an orthographic depth projection image compatible with standard VLM vision encoders.
Result: A synchronized tri-modal triplet $T = (I_{rgb}, I_{th}, I_{lidar})$ is created for training and inference.

B. Model Architecture and Fine-Tuning

The system utilizes a compact Qwen3-VL-4B backbone. To adapt this model for edge deployment and specific reasoning tasks without catastrophic forgetting, the authors employ:

RPSFT (Rotation-Preserving Supervised Fine-Tuning): Instead of standard SFT, which can rotate pre-trained weight matrices and degrade out-of-distribution (OOD) knowledge, RPSFT freezes the base model and updates only the attention projection weights ( $W_q, W_v$ ) in the last 12 layers. These updates are regularized using SVD-anchored constraints to keep parameter drift within the original pre-trained manifold.

C. Policy Alignment Strategies

To align the model's reasoning with an expert aircraft taxonomy (grouping by propulsion, airframe, etc.), three reinforcement learning strategies are evaluated:

GRPO (Group Relative Policy Optimization): Samples multiple completions and scores them based on format and accuracy, using relative advantage within the group.
GSPO (Group Sequence Policy Optimization): Computes probability ratios over the entire generated sequence rather than token-by-token. This aims to prevent local token collapse and encourage globally coherent reasoning paths.
DAPO (Direct Advantage Policy Optimization): Optimizes a pairwise preference loss between chosen and rejected responses, using truncation masks to filter overlong generations.

D. Deployment Optimization

The aligned model undergoes 4-bit quantization and attention-memory optimization to fit within an 8GB VRAM target, enabling real-time processing on edge devices.

3. Key Contributions

The paper outlines three primary contributions:

Edge-Oriented Tri-Modal Pipeline: A complete system for aircraft taxonomic grouping using RGB, synthetic thermal, and pseudo-LiDAR inputs, specifically designed for resource-constrained environments.
Synthetic Sensing Procedure: A method to generate aligned FLIR-style and depth-projection views from monocular imagery, providing controlled auxiliary cues when real multi-sensor data is unavailable.
Deployment-Focused Evaluation: A comprehensive analysis of policy alignment strategies (GRPO, GSPO, DAPO), structured-output validity, zero-shot generalization, quantization effects, and latency.

4. Experimental Results

The system was evaluated on a curated dataset of 800 single-class aircraft images and a held-out test partition of 100 samples.

Policy Optimization Performance:
- GSPO emerged as the superior strategy. On the validation set, it achieved 88.33% accuracy and a 0.91 weighted F1 score.
- On the held-out test partition, GSPO achieved 78.00% accuracy and 0.793 weighted F1, while maintaining 94.00% parseable output formatting.
- Token-level methods (GRPO and DAPO) showed lower accuracy (67.00%) and lower format compliance (85.00%) on the held-out set, indicating vulnerability to representational loops on unfamiliar inputs.
RPSFT vs. Standard SFT: While standard SFT achieved higher in-distribution accuracy (63.75% vs. 55.00%), it suffered from catastrophic forgetting. RPSFT preserved general reasoning capabilities, achieving 90.16% OOD accuracy on the ScienceQA benchmark.
Edge Performance: After 4-bit quantization, the model fits the 8GB deployment target and processes each tri-modal triplet in 1.48 seconds.
Cross-Domain Generalization: When tested on the DSERT-RoLL road-scene dataset (zero-shot), the GSPO-aligned model maintained 100% format success and demonstrated robustness in counting tasks (e.g., MAE of ~0.54 for pedestrians), though exact class matching remained challenging due to the domain shift.

5. Significance and Claims

The paper positions TriCLE as a practical prototype for interpretable, edge-feasible aircraft grouping. Its significance lies in demonstrating that:

Synthetic Multi-Modal Fusion: It is possible to train effective tri-modal reasoning systems using synthetic thermal and pseudo-LiDAR data derived from RGB images when real aligned sensors are unavailable.
Sequence-Level Optimization: Sequence-level policy optimization (GSPO) is more effective than token-level methods for maintaining coherent reasoning and structural validity in long-form generation tasks on edge devices.
Feasibility: Complex tri-modal reasoning can be compressed into a 4B parameter model that runs within 8GB VRAM with sub-2-second latency, making it viable for real-world edge deployment.

Limitations Acknowledged:
The authors explicitly state that TriCLE is a prototype. The thermal and LiDAR inputs are synthetic and may contain generator artifacts, meaning they cannot replace calibrated hardware. The evaluation is limited to single-object images and does not fully capture multi-aircraft scenes. The cross-domain results on road scenes serve as a stress test for syntax robustness rather than a validation of aircraft grouping capabilities. Future work requires validation on real, co-registered sensor streams.