DamageArbiter: A Multimodal Arbitration Framework for Disaster Damage Assessment from Street-View Imagery
This study introduces DamageArbiter, a multimodal arbitration framework that enhances the accuracy and reliability of street-view-based disaster damage assessment by leveraging a meta-classifier to resolve model disagreements, thereby significantly improving classification performance and reducing overconfidence errors compared to existing unimodal and multimodal baselines.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
When a hurricane tears through a coastal town, the most urgent question is not just how much damage occurred, but exactly where it is worst. Emergency responders need to know which streets are blocked by fallen trees, which neighborhoods are underwater, and which buildings are unsafe, often before they can physically reach the area. For decades, scientists have tried to answer this by analyzing photos taken from the ground, known as street-view imagery. These pictures offer a human-scale perspective that satellites often miss, revealing details like cracked foundations or debris piled against a storefront. However, teaching computers to "see" these disasters has been a tricky business. Traditional computer programs, trained to recognize patterns in images, often act like overconfident students: they might guess the severity of a disaster with absolute certainty even when they are completely wrong. In a crisis, a confident mistake is far more dangerous than a hesitant one, because it can send rescue teams to the wrong place or leave the most vulnerable areas unattended.
A team of researchers at Texas A&M University and several other institutions has developed a new way to solve this problem, moving beyond the idea of a single, all-knowing computer model. Instead of relying on one algorithm to do all the work, they created a system called DamageArbiter that acts like a mediator between two different types of artificial intelligence. One type of model is a visual expert, trained to look at photos and spot damage, while the other is a multimodal model that understands both images and the text descriptions of what is happening in those images. The researchers found that the visual expert is very good at spotting damage but tends to be overly sure of its answers, even when the evidence is blurry. The multimodal model, which can "read" a description of the scene alongside the photo, is more cautious and less likely to make confident mistakes, but it is slightly less accurate at identifying the damage in the first place.
To test their idea, the researchers gathered 2,556 photos taken after Hurricane Milton struck Horseshoe Beach, Florida, in 2024. They labeled each photo by hand into three categories: mild, moderate, or severe damage. They also used a large language model to write short, factual descriptions of what was visible in each photo, creating a text version of the visual data. They then trained their computer models to classify the damage using these images and descriptions. The results confirmed their suspicion: the visual-only model achieved the highest accuracy, correctly identifying the damage level in about 74 percent of cases, but it made a dangerous error. When it was wrong, it was wrong with high confidence nearly 71 percent of the time. The multimodal model, which combined the picture with the text description, made fewer confident mistakes but was slightly less accurate overall.
The researchers then put their mediation system, DamageArbiter, to work. When the two models agreed on the damage level, the system accepted their shared answer. But when the models disagreed, the system did not simply guess or average their opinions. Instead, it used a lightweight decision-making tool to look at the confidence levels of both models and decide which one was more likely to be right in that specific instance. This arbitration process allowed the system to keep the high accuracy of the visual model while borrowing the caution of the multimodal model. The final result was a system that correctly identified damage in 75.85 percent of cases, a slight improvement over the best single model. More importantly, the system drastically reduced the number of times it made a confident mistake. The rate of overconfident errors dropped from 70.58 percent down to just 16.45 percent.
The study demonstrates that in high-stakes situations like disaster response, accuracy alone is not enough. A model that is right most of the time but confidently wrong when it matters can be a liability. By combining the strengths of different types of artificial intelligence and using a simple arbitration strategy to resolve their disagreements, the researchers created a tool that is both more accurate and more reliable. They tested this system on the real-world data from Florida and found that it could map the damage across the town with a level of trustworthiness that single models could not match. The researchers note that while their system is a significant step forward, it was tested on a single hurricane in one specific area, and future work will need to see if it holds up across different types of disasters and landscapes. Nevertheless, the approach offers a promising path toward AI systems that can assist emergency teams not just by seeing the damage, but by knowing when they are unsure, ensuring that human judgment remains the final authority in the most critical moments.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.