💻 computer science

Selection, Not Fusion: Radar-Modulated State Space Models for Radar-Camera Depth Estimation

The paper proposes SemoDepth, a novel radar-camera depth estimation framework that introduces Radar-Modulated Selection (RMS) to inject sparse radar signals directly into the Mamba model's internal selection mechanism rather than fusing features externally, thereby achieving state-of-the-art accuracy and low latency on the nuScenes dataset.

Original authors: Zhangcheng Hou, Tomoaki Ohtsuki

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhangcheng Hou, Tomoaki Ohtsuki

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a 3D map of the world around a car using two different tools: a camera and a radar.

The Camera is like a high-definition painter. It sees everything in beautiful detail, capturing every leaf, sign, and car. However, it has two big flaws: it doesn't know exactly how far away things are (it's "up-to-scale"), and it gets blind in the rain, fog, or at night.
The Radar is like a blind sonar. It doesn't see details; it only sees a few scattered dots (returns) representing objects. But it knows the exact distance to those dots, and it works perfectly in rain, fog, and darkness.

The goal of this paper is to combine these two to create a perfect, all-weather 3D map.

The Old Way: "Gluing" the Tools Together

Previous methods tried to solve this by taking the "painting" from the camera and the "dots" from the radar, and then gluing them together at the end. Imagine taking a detailed sketch and a few sticky notes with distances, and trying to paste the notes onto the sketch. The paper argues this is inefficient. The "glue" (the fusion) happens after the brain has already tried to understand the image on its own.

The New Idea: "The Conductor"

The authors propose a new method called SemoDepth. Instead of gluing the tools together at the end, they let the radar act as a conductor for the camera's brain while it is thinking.

They use a new type of AI brain called a Mamba (a Selective State Space Model). Think of the Mamba as a person reading a long story (the image) word by word. As they read, they decide:

How much to remember from the previous word (the "step size").
What to say out loud based on what they are remembering (the "readout").

In the old "gluing" methods, the radar just whispered a fact to the person after they finished reading a sentence.

In this new method, Radar-Modulated Selection (RMS), the radar acts as a conductor who taps the person on the shoulder while they are reading.

If the radar sees a car 50 meters away, it taps the conductor to say, "Hey, remember this part of the story longer!" (adjusting the step size).
It also says, "When you speak, make sure to mention this distance!" (adjusting the readout).

Crucially, the radar doesn't rewrite the story (the image features); it just changes how the story is processed and remembered.

The "Smart Pyramid" Strategy

The authors realized that using this "conductor" technique everywhere is too expensive (like having a conductor for every single word in a book). So, they built a Multi-View Scan Pyramid (MVSP):

At the top (coarse view): The radar is very sparse, but its "reach" is huge. The conductor guides the whole scene at once.
In the middle: The radar points are more specific. The conductor only guides the specific areas where the radar dots are present.
At the bottom (fine detail): The radar is too sparse to guide every tiny pixel. Here, they use a simpler, cheaper method to just nudge the final details.

This ensures the "conductor" is only used where it matters most, saving time and energy.

The Results: Faster and Smarter

The paper claims their method, SemoDepth, is the new champion:

Accuracy: It creates much more accurate 3D maps than previous methods, especially in the tricky 0–80 meter range. It reduced errors by about 30% compared to the best previous method.
Speed: It is the fastest method available, processing a frame in just 26.8 milliseconds (faster than a human eye blink).
The "Aha!" Moment: They proved that once you let the radar conduct the brain during the reading (in-scan selection), you don't need to glue the data together after (out-of-scan fusion). Adding the old "glue" method on top of their new conductor method actually added zero improvement.

Summary

Instead of trying to merge two different types of data at the end of the process, this paper teaches the AI to let the radar steer the camera's attention while it is processing the image. It's like having a blind spotter guiding a painter's hand in real-time, rather than trying to fix the painting after it's done. This makes the system faster, more accurate, and better at handling bad weather.

Technical Summary: Selection, Not Fusion: Radar-Modulated State Space Models for Radar-Camera Depth Estimation

Problem Statement
Dense per-pixel metric depth estimation is critical for autonomous driving and robotics, yet monocular cameras suffer from scale ambiguity and degradation in adverse weather (rain, fog, low light). While LiDAR provides accurate metric depth, its cost and power requirements limit it to premium systems. Millimeter-wave automotive radar offers an inexpensive, all-weather, metric alternative but is severely limited by sparsity (40–100 returns per frame in the front-camera field of view) and angular/height ambiguities.

Existing radar-camera depth estimation methods typically address this by fusing radar and image features at the feature level (e.g., concatenation, gating, graph-based extraction) outside the backbone's sequence operator. Even recent cross-modal variants of Selective State Space Models (SSMs) like Mamba often stack modality-specific streams before a shared scan, leaving the core selection mechanism itself unimodal. The authors argue that this "out-of-scan fusion" approach fails to leverage the unique capabilities of SSMs, where the selection mechanism governs long-range information propagation.

Methodology: SemoDepth
The paper proposes SemoDepth, a single-stage radar-camera depth estimator that integrates radar data directly into the selective scan mechanism of the Mamba backbone, rather than fusing it as a separate feature stream.

Radar-Modulated Selection (RMS):
The core innovation is injecting radar signals into the Mamba block's selective scan parameters. In a standard Mamba block, the recurrence is governed by an input-dependent step size $\Delta_t$ (memory horizon) and a readout vector $C_t$ .
- Mechanism: Radar contributes additive, zero-initialized perturbations to $\Delta_t$ and $C_t$ . The input projection $B_t$ and state dynamics matrix $A$ remain image-only.
- Horizon Modulation: Radar modulates $\Delta_t$ via a learned gate, allowing radar to locally adjust how far information propagates along the sequence.
- Readout Modulation: Radar adds a term to $C_t$ , reshaping what is extracted from the hidden state without altering the stored recurrence.
- Initialization: All radar projections are zero-initialized. At initialization ( $t=0$ ), the block is mathematically equivalent to a vanilla, image-only Mamba. This ensures the model preserves pretrained image-only performance and allows radar gradients to flow only where they improve accuracy.
Multi-View Scan Pyramid (MVSP):
To address computational efficiency and the varying spatial reach of radar at different scales, the decoder employs a three-tiered fusion strategy:
- Coarsest Scales (Tier 3): Full-image, four-direction RMS. This approximates a 2D receptive field at linear cost, aggregating scene-wide context.
- Mid Scale (Tier 2): Windowed RMS centered on projected radar returns. Scan capacity is concentrated only where radar supports the image.
- Finest Scales (Tier 1): Constant-cost Feature-wise Linear Modulation (FiLM). This maintains an active radar signal at full resolution without the quadratic cost of scanning.
Training Objective:
The model is trained with a composite loss balancing robust dense supervision (accumulated LiDAR) and accurate sparse supervision (single-sweep LiDAR), including log-space, linear, and gradient losses.

Key Contributions

In-Scan Selection: The paper introduces a principled mechanism (RMS) where radar modulates the recurrence dynamics ( $\Delta$ and $C$ ) of an SSM. This fundamentally changes how information is stored and propagated, rather than just how features are combined.
Three Theoretical Advantages:
1. Linear-cost coupling: Radar shapes the recurrence at every token with $O(L)$ cost, avoiding the $O(L^2)$ cost of cross-attention.
2. Loss-preserving initialization: Zero-initialization ensures the model starts as a pure image backbone, preserving pretrained weights.
3. Image-only fallback: In the absence of radar, the model naturally degrades to image-only behavior without architectural reconfiguration.
MVSP Architecture: A decoder that matches the fusion operator to the effective spatial reach of radar at each resolution, concentrating compute where it is most effective.

Results
Evaluated on the nuScenes and ZJU-4DRadarCam datasets, SemoDepth achieves state-of-the-art performance:

nuScenes: SemoDepth reduces Mean Absolute Error (MAE) by 34.0% (0–50m), 29.9% (0–70m), and 29.9% (0–80m) compared to the previous best (TacoDepth). It also achieves the lowest single-frame inference latency (26.8 ms) among methods with public releases.
ZJU-4DRadarCam: SemoDepth attains the best MAE at all evaluation ranges.
Ablation Findings:
- Replacing the MVSP's Mamba-based scans with uniform FiLM modulation degrades MAE@80 by 18%, confirming the architectural necessity of the scan tiers.
- Adding "out-of-scan" feature blending (pre-scan fusion) on top of RMS provides no additional accuracy, empirically validating that in-scan selection can replace traditional out-of-scan fusion.
- The performance gap is most pronounced in daytime conditions where image evidence is strong; in nighttime (camera-adversarial) conditions, the gap narrows, and attention-based fusion remains competitive at short ranges.

Significance and Claims
The paper claims that for sequence models with input-dependent recurrence, the selection mechanism itself serves as a superior substrate for cross-modal interaction compared to feature-level fusion. By modulating the recurrence dynamics directly, the model leverages the "metric anchor" of sparse radar returns to propagate information through the entire sequence, rather than just the local neighborhood.

The authors position this work as a conceptual shift: cross-modal learning should be reconsidered from the perspective of selection rather than representation. They suggest this recipe extends to other sparse-plus-dense modality pairs (e.g., LiDAR-camera, radar-thermal) and tasks beyond depth estimation.

Limitations
The authors acknowledge several limitations:

Modality Scope: The method is validated only on front-camera + automotive radar; transfer to other modalities is hypothesized but not empirically validated.
Radar Sparsity: The performance margin shrinks on the denser ZJU-4DRadarCam dataset compared to nuScenes, suggesting the method's advantage is most pronounced when radar is extremely sparse.
Noise Sensitivity: While the zero-initialization provides protection, the paper has not stress-tested the model against systematically incorrect radar anchors (noise) which could propagate through the scan.
Data Bias: The datasets used are geographically narrow (Boston, Singapore, ZJU campus), limiting claims about robustness in under-represented environments.