Detection and Measurement of Hailstones with Multimodal Large Language Models

This study demonstrates that pre-trained multimodal large language models, particularly when enhanced with two-stage prompting strategies that leverage reference objects, can effectively detect and measure hailstone diameters from crowdsourced social media images with an average error of 1.12cm, offering a promising complement to traditional hail sensors for rapid severe weather assessment.

The Gist

Imagine you are a meteorologist trying to figure out how big a hailstone is. In the old days, you'd have to wait for a storm to pass, then send a team of people out with rulers and "hail pads" (special mats) to catch the ice. It's like trying to measure the size of a tsunami by sticking a single ruler in the ocean at one spot—you miss most of the action, and you only know what happened right where you stood.

This new paper is about a smarter, faster way to do this: asking Artificial Intelligence to look at photos people post on social media and guess the size of the hail.

Here is the story of how they did it, broken down into simple parts:

1. The Problem: The "Blind Spot"

Hailstorms are expensive and dangerous. They smash cars and destroy crops. But our current weather sensors are sparse; they are like a few scattered streetlights in a huge, dark city. They miss a lot of the hail. Meanwhile, millions of people are snapping photos of the destruction and posting them on Instagram, X (Twitter), and Facebook. These photos are a goldmine of data, but they are messy. One photo might show a giant hailstone next to a hand; another might show tiny hail from far away with no reference.

2. The Solution: The "Super-Intelligent Detective"

The researchers didn't build a new robot from scratch. Instead, they used Multimodal Large Language Models (MLLMs). Think of these models as super-smart detectives who have read the entire internet and seen billions of pictures. They can "see" an image and "read" a question at the same time.

The team gathered 474 photos of hailstones from Austria (taken between 2022 and 2024). They knew the real size of the hail in these photos (the "ground truth") because they came from official storm databases. They wanted to see if the AI could guess the size just by looking at the pictures.

3. The Trick: The "Two-Step Dance"

They tested the AI with two different ways of asking the question:

• The "Direct Shot" (Strategy 1): They just asked, "How big is this hail?"
  • Result: The AI often got confused or guessed wildly, especially if the photo was taken from far away. It was like asking someone to guess the size of a car in a photo without any other objects for comparison.
• The "Two-Step Dance" (Strategy 2): This was the winner.
  • Step 1: The AI first looks for a reference object. It asks, "Is there a hand, a coin, a ruler, or a cigarette in this picture?"
  • Step 2: Once it finds the reference (e.g., "Aha! That's a human hand!"), it uses its knowledge that "a hand is usually about X inches wide" to calculate the size of the hail next to it.
  • Result: This was like giving the detective a ruler to hold up against the photo. The accuracy jumped significantly.

4. The Results: "Pretty Good, But Not Perfect"

The best AI model (GPT-4o) got the size right within about 1.12 centimeters on average.

• The Good News: If a hailstone is 5 cm, the AI guessed 4.5 or 5.5 cm. That is close enough to tell a farmer, "Hey, this is a dangerous storm!"
• The Bad News: The AI had a habit of being a bit shy. It tended to underestimate the size (guessing the hail was smaller than it really was). This is like a nervous witness saying, "I think the car was small," when it was actually huge.
• The Hand Factor: The photos with human hands in them were the easiest for the AI. The error dropped to just 0.75 cm. The AI knows what a hand looks like, so it's a perfect ruler. Photos with no reference objects were much harder.

5. Why This Matters

Imagine a future where, during a storm, an automated system instantly scans social media, finds photos of hail, and tells the weather service: "People in this neighborhood are posting photos with hail the size of golf balls!"

This happens in real-time, covering areas where official sensors don't exist. It turns the "crowd" into a massive, distributed sensor network.

The Bottom Line

This paper proves that we don't need to build special, expensive cameras to measure hail anymore. We can just use the AI tools we already have, combined with the photos people are already taking.

• The Analogy: It's like realizing you don't need a professional surveyor to measure a room; you just need a photo of the room with a person standing in it, and a smart computer to do the math.
• The Catch: We still need to build the system that automatically finds these photos on social media in real-time. Once that's done, we'll have a super-powerful tool to track severe weather faster and more accurately than ever before.

Technical Summary

1. Problem Statement

Hailstorms cause significant economic damage (exceeding $10 billion annually in North America), yet accurate measurement of hailstone size remains a challenge. Traditional methods rely on sparse ground-based sensors (hail pads, disdrometers) and weather radar, which suffer from limited spatial coverage, beam attenuation, and ground clutter. While crowd-sourced social media imagery offers a potential solution for dense spatial data, extracting quantitative measurements from these unstructured images is difficult due to varying image quality, viewing angles, and the lack of standardized reference objects.

2. Methodology

The study investigates the capability of pre-trained Multimodal Large Language Models (MLLMs) to estimate hailstone diameters from crowd-sourced images without fine-tuning (zero-shot learning).

• Dataset:
  • Source: Derived from the European Severe Weather Database (ESWD).
  • Scope: 474 crowd-sourced images from documented hail events in Austria (Jan 2022 – Sep 2024).
  • Ground Truth: Maximum hailstone diameters ranging from 2 cm to 11 cm (mean: 4.17 cm), with 0.5 cm accuracy.
  • Annotations: Images were manually annotated for the presence of reference objects (e.g., hands, coins, rulers) and viewing distance (close-up vs. distant).
• Models Evaluated: Four state-of-the-art MLLMs were tested:
  • OpenAI: GPT-4o (G4) and GPT-4o-mini (G4m).
  • Anthropic: Claude-Sonnet 4 (CS4).
  • Google: Gemini 2.5 Flash Lite (GFL).
• Prompting Strategies: Two distinct strategies were compared:
  • P1 (Single-Stage): Directly asking the model to estimate the maximum diameter in cm.
  • P2 (Two-Stage): A sequential approach where the model first identifies a reference object (or classifies the image as "unspecified") and then uses the known dimensions of that object (or contextual cues) to estimate the hailstone size in a second step.

3. Key Contributions

• Zero-Shot Application of MLLMs: Demonstrates that off-the-shelf MLLMs can perform complex visual reasoning and metric estimation on severe weather imagery without domain-specific training.
• Prompt Engineering for Scale: Introduces a two-stage prompting strategy (P2) that leverages reference object detection to significantly improve measurement reliability compared to direct estimation.
• Quantitative Analysis of Social Media Data: Provides a rigorous evaluation of the potential to extract quantitative meteorological data from unstructured social media sources, bridging the gap between citizen science and operational meteorology.

4. Results

• Overall Performance: The best-performing configuration was GPT-4o with the two-stage prompt (P2), achieving a Mean Absolute Error (MAE) of 1.12 cm and a Pearson correlation of 0.71 with ground truth.
• Impact of Prompting: The two-stage strategy (P2) reduced the average MAE by 18.6% compared to the single-stage strategy (P1) across most models. It also drastically reduced "misses" (failures to provide a numerical output) from 289 (P1) to 20 (P2).
• Reference Object Influence:
  • Human Hands: Provided the most effective scale cue, resulting in the lowest MAE (0.75 cm).
  • Unspecified/No Reference: Images without clear scale cues nearly doubled the error to 1.73 cm.
  • Rulers: Surprisingly less effective (MAE 1.32 cm) than hands, likely due to perspective distortion when tilted.
• Systematic Bias: All models exhibited a negative bias (systematic underestimation), with an average bias of -0.72 cm for the best model. This suggests models are conservative when interpreting 3D scales from 2D images.
• Model Comparison: While GPT-4o (P2) was the most accurate, GPT-4o-mini showed the smallest bias (-0.49 cm), though with slightly higher overall error.

5. Significance and Future Work

• Operational Potential: The study proves that MLLMs can complement traditional sensors by providing spatially dense, real-time hail size estimates from social media, enabling faster and more detailed severe weather assessments.
• Limitations: The current dataset is limited to Austria (2022–2024), and the study relies on manual annotation for reference objects and distance. The models currently lack automated real-time image harvesting pipelines.
• Future Directions:
  • Expanding the dataset to include diverse geographic and environmental conditions.
  • Developing automated pipelines for real-time social media image harvesting and filtering.
  • Investigating geometric priors or perspective correction techniques to mitigate the systematic underestimation bias.

In conclusion, the paper establishes that pre-trained MLLMs, when guided by effective prompt engineering, are a viable and powerful tool for extracting quantitative hail data from crowd-sourced imagery, offering a scalable solution for modernizing severe weather monitoring.