🤖 machine learning

Mechanistic Interpretability-Guided Selective Fine-Tuning of Vision-Language Models for Centimeter-Level Flood Depth Estimation

This paper introduces mechanistic interpretability-guided selective fine-tuning of vision-language models, which identifies specific cross-attention layers critical for flood depth estimation to achieve centimeter-level accuracy with an 86–88% reduction in trainable parameters compared to dense fine-tuning.

Original authors: Nafis Fuad, Xiaodong Qian, Dongxiao Zhu

Published 2026-08-11

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Nafis Fuad, Xiaodong Qian, Dongxiao Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to read a story written in pictures. This robot, known as a Vision-Language Model (VLM), is like a brilliant student who has read millions of books and seen billions of photos, but it's never actually been to a flooded street. It knows what a car looks like and what water looks like separately, but it hasn't learned how to measure exactly how deep the water is just by looking at a car sitting in it. This is a big problem for our cities. When it rains hard, water can rise up and trap cars or knock out power, but right now, our navigation apps can only tell us if a road is "open" or "closed." They can't tell us if the water is 2 centimeters deep (a little splash) or 20 centimeters deep (dangerous for a car's engine). To fix this, scientists need to teach these robots to be precise, down to the centimeter, without needing a human to stand there with a ruler. The challenge is that teaching a giant robot to do this usually requires a massive amount of computing power and a mountain of real-world photos, which are hard to get because flooding is dangerous and unpredictable.

This paper tells the story of how a team of researchers taught a robot named "FloodLlama" to become a master flood detective. First, they built a giant, virtual video game world using Unreal Engine 5, filling it with 2.81 million pictures of cars in the rain, under the sun, and at night, with water levels ranging from dry to 40 centimeters deep. They used about 610,000 of these pictures to train the robot. The robot learned to look at a car and guess the water depth with incredible accuracy, getting the answer within 0.40 centimeters on average. But here is the clever part: the researchers didn't just stop at making the robot smart; they wanted to know how it was thinking. They used a special set of tools called "mechanistic interpretability" to peek inside the robot's brain. They discovered that the robot doesn't learn the whole task at once. Instead, it goes through two stages: first, it rearranges how it sees the picture (like organizing a messy desk), and then, in one specific layer of its brain (layer 23), it suddenly figures out how to turn that picture into a number.

Using this secret map of the robot's brain, the researchers built two new, super-efficient versions of the model called FloodLlama-MI5 and FloodLlama-MI6. Instead of training the robot's entire brain, they only taught the specific layers they knew were doing the important work. This was like teaching a student to solve a math problem by only focusing on the specific steps where the answer is actually calculated, ignoring the rest. The result? These new models are 86% to 88% smaller and faster to train than the original, yet they are still incredibly accurate. When tested on real-world photos of floods, the most efficient version (MI6) got the right answer 98.62% of the time, beating the previous best system by a wide margin. The paper suggests that by understanding exactly how these giant models learn, we can make them much more efficient without losing their smarts, paving the way for safer, smarter navigation systems that can tell us exactly how deep the water is before we drive into it.

Technical Summary: Mechanistic Interpretability-Guided Selective Fine-Tuning for Centimeter-Level Flood Depth Estimation

1. Problem Statement

Urban flooding poses a critical threat to transportation infrastructure, yet operational systems (e.g., Google Maps, Waze) currently rely on binary closure reports rather than quantitative depth measurements. This binary approach fails to support emerging safety-critical applications:

Electric Vehicles (EVs): Risk thermal runaway at depths of 15–25 cm.
Autonomous Vehicles (AVs): Require precise depth estimation to define operational design domains, as current perception stacks cannot distinguish between passable and impassable water.
Safety Physics: Even a 0.56–1.5 mm water film can trigger hydroplaning, and a 1 mm film can reduce tire-pavement friction by up to 61%.

Existing vision-based solutions suffer from significant limitations:

Reference-object methods: Limited to ~12 cm MAE and fail when objects are occluded.
Object-detection methods: Treat depth as discrete categories rather than continuous variables.
Zero-shot Large Multimodal Models (LMMs): While they offer open-ended reasoning, they produce high errors (MAE > 8 cm) and function as proprietary black boxes, precluding the interpretability required for safety-critical deployment.

There is a gap in the literature regarding the fine-tuning of open-source Vision-Language Models (VLMs) for continuous, centimeter-resolution depth estimation and the mechanistic understanding of how these models acquire such capabilities.

2. Methodology

A. Synthetic Data Generation

To overcome the scarcity of real-world labeled flood data, the authors generated a 2.81M-image synthetic corpus using Unreal Engine 5. The dataset includes:

Single-vehicle subset: 1.73M images with 7 vehicle types across 9 discrete depth levels (0–40 cm in 5 cm steps) and 4 weather conditions.
Mixed-vehicle subset: 1.08M images with multiple vehicles per scene, varying depth at 1 cm increments (1–40 cm) to simulate occlusion and visual complexity.
Fidelity Assessment: Sim-to-real transfer was validated using Kolmogorov-Smirnov (KS) statistics, with the mixed-vehicle subset showing better alignment with real imagery than the single-vehicle subset.

B. Baseline Model: FloodLlama-Dense

The authors fine-tuned LLaMA 3.2-11B Vision Instruct using QLoRA (4-bit NF4 quantization with LoRA adapters).

Architecture: The model couples a frozen ViT-H/14 vision encoder with a 40-layer decoder. LoRA adapters ( $r=16, \alpha=16$ ) were inserted into the Q/K/V/O projections of all 40 attention modules (32 self-attention + 8 cross-attention layers).
Training Strategy: A two-phase progressive curriculum was employed:
1. Phase 1: Fine-tuning on a balanced 340k-image subset of the single-vehicle data to establish basic vision-language mapping.
2. Phase 2: Cumulative fine-tuning on 270k images from the mixed-vehicle subset (processed in 9 sequential chunks) to handle occlusion and refine continuous regression.
Parameters: 54.4M trainable parameters (0.49% of the base model).

C. Mechanistic Interpretability Analysis

To understand how the model learns depth and to identify efficient fine-tuning targets, the authors applied four interpretability techniques to the trained FloodLlama-Dense:

Linear Probing: A ridge regressor was trained on frozen hidden states to predict depth. Results showed a sharp phase transition at Layer 23 (L23), where depth became linearly decodable ( $R^2$ jumped from -0.355 to +0.513).
Logit Lens: Revealed that numeric depth tokens only emerged in layers L33 and L38, after the encoding transition at L23.
Centered Kernel Alignment (CKA): Measured representational drift. Layers L13–L22 showed significant drift (restructuring visual features) but did not encode depth. Layers L23–L38 maintained high similarity to the base model but carried the depth information.
Cross-Attention Entropy: Measured attention focus. L13 showed the sharpest entropy reduction (spatial selectivity) but zero encoding capability.
LoRA Weight Analysis: Identified that L18, L23, L28, L33, and L38 absorbed the majority of weight updates.

Key Finding: The model exhibits a two-stage adaptation pattern: early-to-mid layers (L13–L22) restructure visual representations, while late layers (starting at L23) encode the depth information.

D. Selective Fine-Tuning: FloodLlama-MI5 and MI6

Leveraging the interpretability findings, the authors developed parameter-efficient variants that fine-tune only specific cross-attention layers:

FloodLlama-MI5: Fine-tunes 5 layers (L8, L18, L23, L33, L38).
FloodLlama-MI6: Fine-tunes 6 layers (MI5 set + L28).
Efficiency: These models reduce trainable parameters by 86–88% (6.55M and 7.86M parameters, respectively) compared to the dense baseline, while maintaining the same two-phase training curriculum.

3. Key Results

A. Synthetic Test Performance

On a held-out 16,000-image mixed-vehicle test set:

FloodLlama-Dense: Achieved MAE = 0.40 cm, RMSE = 1.97 cm, and Acc@5cm = 97.59%.
FloodLlama-MI5: MAE = 0.80 cm, Acc@5cm = 95.1%.
FloodLlama-MI6: MAE = 0.78 cm, Acc@5cm = 95.8%.
Observation: While the dense model achieved the lowest absolute error, the sparse variants maintained operational accuracy within the critical ±5 cm tolerance band with a fraction of the training cost.

B. Real-World Benchmark (vs. STURM-FloodDepth)

Evaluated on a 300-image real-world dataset against the STURM-FloodDepth baseline (which outputs discrete categories):

FloodLlama-MI6 (Chain-of-Thought prompt): Achieved 98.62% overall accuracy and 100.00% accuracy on deep flood levels (>30 cm).
STURM-FloodDepth: Achieved 86.61% overall accuracy and 88.27% on deep flood levels.
Statistical Significance: The improvement of ~12 percentage points was confirmed with non-overlapping 95% confidence intervals.
Prompt Sensitivity: Chain-of-Thought (CoT) prompts yielded the highest accuracy and best calibration for deep floods, whereas "Simple" prompts tended to under-predict deep flood levels.

4. Contributions

Dataset: Creation of a 2.81M-image synthetic flood corpus with centimeter-resolution depth labels and systematic variation in vehicle types and weather.
FloodLlama-Dense: The first open-source VLM fine-tuned for continuous centimeter-resolution flood depth regression, achieving sub-centimeter MAE on synthetic data.
Mechanistic Insight: The first application of mechanistic interpretability (linear probing, logit lens, CKA, entropy) to a flood depth model, identifying Layer 23 as the critical depth-encoding transition layer and revealing a "restructure-then-encode" adaptation pattern.
Efficient Models: Development of FloodLlama-MI5 and MI6, which achieve 86–88% parameter reduction while outperforming the state-of-the-art STURM-FloodDepth baseline on real-world data.

5. Significance and Claims

The paper claims that mechanistic interpretability can serve as a practical design driver for Parameter-Efficient Fine-Tuning (PEFT) in transportation perception tasks. By identifying specific layers responsible for task encoding (L23) versus those responsible for feature restructuring (L13–L22), the authors demonstrate that:

Full fine-tuning of all layers is not necessary for high-precision regression.
Selective fine-tuning guided by interpretability analysis can reduce computational costs by nearly 90% without sacrificing real-world generalization.
Open-source VLMs, when properly fine-tuned and interpreted, can provide the sub-decimeter precision required for next-generation vehicle safety systems, surpassing both traditional computer vision methods and proprietary zero-shot models.

The authors note limitations, including the reliance on a single VLM architecture (LLaMA 3.2) and the under-representation of extreme low-light and deep-flood scenarios in the real-world test set, suggesting future work should extend these interpretability protocols to other architectures and broader environmental conditions.