A Dragonfly-Inspired Heterogeneous Sensing System for Task-Driven Multi-Modal Perception

Technical Summary: A Dragonfly-Inspired Heterogeneous Sensing System for Task-Driven Multi-Modal Perception

Problem Statement
As artificial intelligence transitions from digital interpretation to embodied action, robotic perception faces a critical limitation: current systems primarily recover visible scene appearance and line-of-sight (LoS) geometry. These representations fail to capture occluded structures, non-line-of-sight (NLoS) content, and material-dependent physical attributes essential for interaction in real-world environments. While multi-modal sensing (e.g., combining cameras and LiDAR) has improved robustness, existing fusion pipelines are often optimized for predefined outputs like object detection or depth completion, rather than constructing a unified physical scene representation that integrates semantic identity, metric geometry, hidden structures, and material properties. Furthermore, Terahertz (THz) sensing, which offers propagation-aware structural insights and material-sensitive responses, remains largely disconnected from semantic and geometric scene representations due to a lack of explicit sensing workflows.

Methodology
The authors propose Dragonfly-TeraVision, a heterogeneous sensing architecture inspired by the dragonfly's compound eyes and nervous system. The system integrates three complementary sensing modalities:

1. Panoramic Vision: Provides dense semantic context and texture over a wide field of view.
2. LiDAR: Delivers accurate metric LoS geometry, forming a spatial scaffold of visible surfaces.
3. Monostatic THz Sensing: Captures propagation-resolved electromagnetic responses, enabling the detection of hidden structures (via penetration/scattering) and material-sensitive reflection characteristics.

System Architecture
The core innovation is a task-driven agent that functions as a "sensing brain." Unlike traditional fusion methods that treat sensor data as interchangeable streams, this agent:

• Decomposes Objectives: Breaks down a sensing task (e.g., "reconstruct hidden structures" or "associate materials") into required scene attributes.
• Selects Skills: Dynamically selects specific processing modules from a modular library (e.g., semantic segmentation, ICP registration, SAGE-based multipath component extraction) based on the task and available data manifest.
• Associates Evidence: Routes outputs from heterogeneous modalities into a common physical scene representation using physics-guided multi-modal constraints.

Experimental Setup and Processing
The system was validated in an indoor L-shaped hallway with 16 sensing positions.

• Vision Processing: Panoramic images were processed using Mask2Former (ADE20K-pretrained) to generate dense semantic masks, which were then mapped to angular coordinates.
• LiDAR Processing: Point clouds from multiple orientations were globally registered using Iterative Closest Point (ICP). Dominant planar structures were extracted using orthogonality-constrained RANSAC to create a metric geometric scaffold.
• THz Processing: Angular frequency responses (290–310 GHz) were processed using Space-Alternating Generalized Expectation-Maximization (SAGE) to extract Multipath Components (MPCs). These were spatially mapped as sparse scattering points and used to estimate reflection losses for material identification.
• Association: The agent anchored semantic labels to LiDAR geometry, integrated THz-derived scattering points to supplement geometry in occluded regions (behind a curtain), and linked THz reflection losses to specific semantic regions (e.g., walls vs. doors).

Key Results

1. Semantic-Metric-Geometric Reconstruction: The system successfully constructed a unified 3D map combining semantic labels, metric geometry, and THz-derived structural evidence.
2. NLoS Perception: In the presence of a curtain occluding a metallic door, THz sensing revealed scattering points corresponding to the hidden door and wall, supplementing the visible LiDAR scaffold where optical and LiDAR sensors were blocked.
3. Material Awareness: By associating near-normal-incidence THz paths with semantic masks, the system linked reflection-loss estimates to specific materials (e.g., distinguishing cement walls from metal doors).
4. Hidden-Object Sensing: In a controlled "opaque-cup" benchmark, the system detected hidden balls (steel, glass, wood) inside visually opaque paper cups. While vision and LiDAR only observed the cup exterior, THz measurements identified delayed reflection peaks corresponding to the hidden objects. Compensated reflection-loss estimates were consistent with the known material properties of steel, glass, and wood.

Significance and Claims
The paper claims to establish a route toward embodied sensing systems that move beyond visible-scene description to physically interpretable environment reconstruction. Key contributions include:

• Architectural Principle: Demonstrating that heterogeneous modalities should not be viewed as interchangeable sensors but as complementary perceptual modalities contributing different parts of a physical scene representation, analogous to biological compound eyes.
• Task-Driven Integration: Showing that an agentic framework can coordinate vision, LiDAR, and THz sensing to produce task-specific representations (semantic geometry, hidden evidence, material association) rather than static, pre-defined outputs.
• Unified Representation: Successfully linking visual semantics, metric geometry, NLoS content, and material-sensitive THz responses within a single physical scene model.

The authors emphasize that this work does not propose a universal ranking of sensor performance but rather a capability map where Dragonfly-TeraVision improves scene representation completeness by preserving the distinct physical meanings of each modality and organizing them according to the perceptual function required for physical world interaction.