💻 computer science

Recov-Vision: Linking Street View Imagery and Vision-Language Models for Post-Disaster Recovery

This paper introduces FacadeTrack, a language-guided framework that leverages street-view imagery to assess post-disaster building occupancy with high precision and interpretability, outperforming baseline methods by separating perception from conservative reasoning to support equitable emergency resource allocation.

Original authors: Yiming Xiao, Archit Gupta, Miguel Esparza, Yu-Hsuan Ho, Antonia Sebastian, Hannah Weas, Rose Houck, Ali Mostafavi

Published 2026-02-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yiming Xiao, Archit Gupta, Miguel Esparza, Yu-Hsuan Ho, Antonia Sebastian, Hannah Weas, Rose Houck, Ali Mostafavi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive storm has just passed through a neighborhood. The city needs to know: Who is still home, and who has left?

Usually, officials fly over in planes or use satellites to take pictures from the sky. It's fast and covers a lot of ground, but it's like looking at a house from the roof up. You can see if the roof is gone, but you can't see if the front door is boarded up, if there's a pile of mud on the porch, or if a car is parked in the driveway. Those are the clues that tell you if a family is actually living there.

On the other hand, walking door-to-door is accurate but incredibly slow. You can't walk every single street in a huge city in time to help people.

Recov-Vision is a new "smart detective" system that tries to get the best of both worlds. Here is how it works, broken down into simple steps:

1. The "Drive-By" Camera

Instead of a plane, the team drove cars equipped with special 360-degree cameras (like a GoPro Max) through the neighborhoods. They drove down every street, capturing video of every house from the street level.

2. The "Magic Lens" (Turning Video into Photos)

The raw video is just a spinning, blurry mess of the whole world. The system acts like a smart photo editor. It takes the video, finds the specific house the car passed, and "unwraps" the 360-degree view to create a straight, clear, flat photo of just that house's front door and porch. It's like taking a curved fisheye photo and flattening it out so you can see the details clearly.

3. The "Two-Brain" Detective System

This is the most important part. The system uses a type of AI called a Vision-Language Model (think of it as a robot that can "see" a picture and "read" a description). The paper compares two ways this robot thinks:

The "Quick Scorer" (One-Stage): The robot looks at the photo and immediately counts up "bad signs." If it sees broken windows, debris, or mud, it adds points. If the points get too high, it says, "This house is empty." It's fast, but sometimes it gets confused by things that look bad but aren't (like a house under repair).
The "Careful Detective" (Two-Stage): This is the paper's main innovation.
- Step 1 (The Eyes): The robot looks at the photo and just lists what it sees, like a witness giving a statement: "I see a tarp on the roof," "I see a car in the driveway," "I see mud." It doesn't make a final decision yet.
- Step 2 (The Brain): A second part of the AI (a text-only reasoning engine) reads that list of clues. It acts like a cautious human inspector. It thinks, "Okay, there is mud, but there is also a car and a tarp. Maybe they are just fixing the roof. Let's be safe and assume they are still there unless we are sure they aren't."

4. The Results: Why Being "Careful" Matters

The team tested this system twice, a few months apart, after Hurricane Helene.

The "Quick Scorer" tended to be too pessimistic. It saw a messy yard and assumed the house was empty, leading it to think more people had left than actually had.
The "Careful Detective" was better at spotting the truth. It correctly identified that many houses were actually occupied even if they looked messy. It matched the real-world data (ground truth) much better, especially in figuring out how many people had returned (recovered) over time.

5. The "Map of Mistakes"

One of the coolest features is that the system doesn't just give a list of "Yes/No." It creates a heat map. If the system is unsure about a specific group of houses, it marks that area on a map. This tells human officials: "Don't check every single house randomly. Go check this specific neighborhood where the AI is confused." It saves time by focusing human effort exactly where it's needed.

Summary

Recov-Vision is like having a fleet of robot cars that drive around after a disaster, take clear photos of every front porch, and then use a "two-step" thinking process to guess who is home.

Step 1: Look and list the clues.
Step 2: Think carefully about the clues before making a decision.

The paper found that this "slow and careful" thinking approach was more accurate than the "fast and direct" approach, helping officials understand exactly how many people were safe at home and where to send help, without needing to knock on every single door.

Technical Summary: Recov-Vision

Problem Statement
Post-disaster recovery requires rapid, building-level occupancy assessments to guide triage, utility restoration, and resource allocation. While overhead (nadir) imagery offers broad coverage, it often fails to capture facade-level cues critical for determining habitability, such as blocked entries, temporary coverings, or localized debris. Conversely, street-view imagery (SVI) provides these necessary details but is inherently sparse, viewpoint-dependent, and difficult to align with parcel inventories at scale. Existing methods often rely on "black box" classifiers that lack the structured, transparent reasoning required to audit decisions or handle ambiguous visual evidence.

Methodology
The paper presents Recov-Vision, a street-level, language-guided framework designed to link panoramic video to specific land parcels and infer occupancy status. The pipeline consists of four main components:

Data Collection and Pre-processing:
- Acquisition: 360° panoramic video (5.6K resolution) was captured via a vehicle-mounted GoPro Max during field surveys in Swannanoa and Black Mountain, North Carolina, following Hurricane Helene. Data was collected across two campaigns (March and June 2025) to track recovery.
- Geospatial Alignment: GPS logs were synchronized with video frames. An R-tree spatial index matched vehicle positions to reference parcels (buffered by 25m).
- View Rectification: To standardize inputs, the system estimated vehicle heading using a temporal window of GPS points. It then calculated the yaw angle required to center the building facade in the panoramic frame. Using FFmpeg's v360 filter, the system dewarped the equirectangular input into a rectilinear, facade-centered view (16:9 aspect ratio).
Vision-Language Model (VLM) Prompting Strategies:
The study compares two decision architectures using a shared vision extractor (GPT-4o) and a standardized attribute schema (Table 2):
- One-Stage Baseline: The VLM extracts nine visual attributes (e.g., house destruction, exterior debris, emergency markings). A deterministic rule-based scoring system then classifies occupancy. The rule is: Predict "Not Occupied" if (Risk Indicators - Vehicle Presence) ≥ 2. This threshold is tuned to be conservative, requiring corroboration for negative labels.
- Two-Stage Strategy: The VLM first extracts the same visual attributes without inferring occupancy. In a second stage, a text-only LLM applies explicit reasoning rules and few-shot examples to map these attributes to a final "Occupied" or "Not Occupied" label. This stage is designed to be conservative, favoring "Not Occupied" when evidence is ambiguous.
Temporal Analysis:
The framework aggregates frame-level predictions into parcel-level labels via majority vote. It defines four change classes between the two visits: Recovered (Not Occupied → Occupied), Deteriorated (Occupied → Not Occupied), and two Stable states.
Evaluation:
Performance was evaluated against human ground truth using precision, recall, F1, and Cohen's $\kappa$ . Statistical significance was assessed via paired bootstrap resampling and McNemar's tests. Spatial autocorrelation of errors was analyzed using Global Moran's $I$ .

Key Contributions

Reproducible Pipeline: An end-to-end workflow linking 5.6K panoramic videos to parcels, including specific geometric formulations for view rectification and GPS-based heading estimation.
Interpretable Inference: A comparison of a transparent rule-based approach versus a two-stage perception-to-reasoning design. The separation of attribute extraction from decision logic allows for policy tuning without retraining vision components.
Spatio-Temporal Evaluation: A framework for quantifying net recovery and diagnosing error clustering using Global Moran's $I$ , moving beyond aggregate accuracy to spatial diagnostics.
Statistical Validation: Rigorous reporting of confidence intervals and error taxonomies, demonstrating that the two-stage approach reduces ambiguity and guides reviewers to a specific subset of difficult cases.

Results

Performance: The two-stage strategy achieved higher recall (0.781 vs. 0.728) and slightly higher F1 and Cohen's $\kappa$ scores compared to the one-stage baseline. However, paired bootstrap tests and McNemar's tests indicated these differences were not statistically significant at the current operating point.
Recovery Metrics: Both models overestimated occupancy relative to ground truth. However, the two-stage strategy more accurately reproduced the ground-truth net recovery trend (+14), whereas the one-stage baseline overstated recovery (+18).
Error Analysis: Residual errors were not randomly distributed. Global Moran's $I$ analysis revealed significant spatial clustering of both occupancy states and prediction errors. This suggests that errors concentrate in specific "pockets," making targeted quality assurance feasible.
Case Studies: The two-stage approach demonstrated a more conservative decision boundary, correctly identifying "Not Occupied" cases where the one-stage baseline failed (e.g., homes with major repairs), though both models struggled with confounding visual cues (e.g., misinterpreting logos as emergency markings).

Significance and Claims
The paper claims that the primary value of the two-stage design lies not in raw performance superiority (which was not statistically significant), but in controllability and auditability. By exposing intermediate attributes and separating perception from reasoning, the system allows operators to:

Diagnose failure modes and tune decision policies (e.g., risk thresholds) without retraining.
Prioritize human review for spatially clustered error pockets rather than conducting diffuse, random audits.
Produce decision-ready, auditable outputs suitable for integration into emergency management workflows.

The authors conclude that while the pipeline offers a scalable alternative to door-to-door inspections, it remains bounded by the quality of underlying imagery, particularly in post-repair scenarios where visual cues become subtler. Future work is suggested for multi-region validation and temporal transition modeling.