A Multi-Agent Feedback System for Detecting and Describing News Events in Satellite Imagery

This paper introduces SkyScraper, an iterative multi-agent system that geocodes news articles to synthesize captions for satellite image sequences, successfully curating a new multi-temporal event dataset with 5,000 sequences and demonstrating a fivefold improvement in event detection over traditional methods.

The Gist

Imagine you are a detective trying to solve a mystery, but instead of looking at a crime scene, you are looking at the Earth from space. Your goal is to find specific events—like a tornado, a new building going up, or a protest—and describe exactly what happened in a series of photos taken over time.

This paper introduces a new tool called SkyScraper that acts like a team of super-smart detectives working together to solve this puzzle. Here is how it works, broken down simply:

The Problem: Finding a Needle in a Haystack

Usually, when researchers want to study changes in satellite photos, they have to manually look through thousands of images or rely on old, pre-labeled maps. It's like trying to find a specific book in a library where the books aren't on shelves, and you don't know the title.

Furthermore, news articles often mention a location in a vague way (e.g., "The storm hit the region"). Traditional computer methods try to guess the exact spot by averaging all the names mentioned. This is like trying to find a specific house in a city by taking the average location of every street name mentioned in a newspaper; you often end up in the middle of a park or a river, missing the actual event entirely.

The Solution: The SkyScraper Team

The authors built SkyScraper, which uses a "multi-agent" system. Think of this not as one robot, but as a small team of specialists passing a case file back and forth until they get it right.

Here is their workflow, step-by-step:

1. The Reader (Article Agent): This team member reads the news story and pulls out a specific place name and a timeline.
2. The Mapmaker (Geocoding API): This member takes that place name and tries to pin it to exact coordinates on a map.
3. The Photo Hunter (Data API): This member goes to the satellite database and grabs the photos for that exact spot and time.
4. The Detective (Verifier Agent): This is the most important new step. This team member looks at the news story and the satellite photos together. They ask: "Does the photo actually show the event described in the article?"
   • If the answer is NO: The team doesn't give up. They mark that location as a "failed attempt," figure out why it failed (e.g., "The storm was actually 50 miles south"), and ask the Reader to try a different location from the article.
   • If the answer is YES: The team moves to the final step.
5. The Reporter (Captioning Agent): Once the event is confirmed, this member writes a clear description of what is happening in the photos, using the news article as context.

The "Try Again" Loop

The magic of SkyScraper is that it doesn't just guess once. If the first guess is wrong, the system learns from the mistake and tries again, up to a limit. It's like a game of "Hot or Cold" where the system gets feedback after every wrong guess and adjusts its search accordingly.

What They Found

The team tested this against two older, traditional methods:

• The "Average" Method: Just guessing the middle of all mentioned places.
• The "Stacking" Method: A more complex way of layering location guesses.

The Results:

• The old methods found about 1.7% to 4.7% of the events correctly.
• SkyScraper found 8.4%, which is nearly 5 times better than the simplest method.
• Using this system, they created a brand new library of 5,000 sequences of satellite images (mostly from PlanetScope and Sentinel-2 satellites) that show real-world events with descriptions.

Why It Matters

This system proves that using a team of AI agents that can "talk" to each other and check their own work is a powerful way to find and describe events in space. It turns a massive, messy database of news articles into a clean, usable collection of satellite stories, helping researchers and journalists see what is happening on our planet without needing to manually hunt for every single photo.

Technical Summary

Technical Summary: SkyScraper – A Multi-Agent Feedback System for News-Driven Satellite Event Captioning

Problem Statement

The field of remote sensing change captioning faces a significant bottleneck in the availability of multi-temporal event captioning datasets. While bi-temporal datasets (pairs of images) exist for land-use/land-cover (LULC) changes, datasets containing sequences of more than two images are scarce. This scarcity stems from two primary challenges:

1. Discovery Difficulty: Identifying visible events within vast archives of satellite imagery is labor-intensive.
2. Annotation Burden: Manually labeling multi-temporal sequences requires substantial time and effort.

Existing approaches, even those leveraging Large Language Models (LLMs), typically rely on pre-labeled temporal datasets or rules-based methods applied to bi-temporal pairs. Furthermore, traditional geocoding methods used to link news articles to satellite imagery are prone to noise, including incorrect location names and inaccuracies where the frequency of a location's mention does not correlate with the actual visibility of an event.

Methodology: The SkyScraper Framework

To address these limitations, the authors introduce SkyScraper, an iterative multi-agent workflow designed to geocode news articles and synthesize captions for corresponding satellite image sequences. Unlike traditional methods that extract all location names at once, SkyScraper employs an agentic feedback loop to refine searches iteratively.

The Iterative Pipeline

The system operates through a five-step cycle involving distinct agents:

1. Extract: An LLM "Article Agent" identifies a named geographic entity and an event timeline from the text.
2. Geocode: A geocoding API converts the named entity into latitude-longitude coordinates.
3. Fetch: A data API retrieves satellite imagery for the specific coordinates and timeline.
4. Verify: A multimodal LLM "Verifier Agent" cross-references the article text with the fetched imagery to determine if the event is visible, providing reasoning for its decision.
5. Caption: If the event is verified, a multimodal LLM "Captioning Agent" generates a change caption using the article as context.

Iterative Refinement (Algorithm 1)

The core innovation lies in the system's ability to handle failure. Rather than accepting the first geocoded location, the system requests one candidate location at a time. If the geocoding fails or the Verifier Agent determines the event is not visible, the system:

• Records the failure and the specific reasoning.
• Requests a new candidate location from the Article Agent, incorporating the history of failures.
• Repeats the cycle until a valid event is found or the maximum number of attempts is reached.

This approach mitigates the noise inherent in traditional geocoding by dynamically adjusting the search based on visual verification feedback.

Experimental Setup and Results

The authors evaluated SkyScraper against two traditional geocoding baselines:

• Weighted Centroid: Converts coordinates to Earth-Centered, Earth-Fixed (ECEF) format and computes a weighted average.
• GIPSY (Geo-referenced Information Processing SYstem): A weighted polygon-based approach that stacks 3D polyhedra based on overlap and containment.

Using a test set of 1,000 diverse global news articles (covering natural disasters, construction, and civil unrest) and PlanetScope imagery (3m resolution), the system was tested with Mapbox for geocoding and Gemini-2.5-flash for LLM agents.

Key Performance Metrics:

• Weighted Centroid: Detected 17 events (1.7% yield).
• GIPSY: Detected 47 events (4.7% yield), representing a 2.8× improvement over the centroid.
• SkyScraper: Detected 84 events (8.4% yield), demonstrating a 4.9× improvement over the centroid baseline.

The authors note that because they only count true positives, the relative improvement is a more informative metric than absolute yield, as articles without visible events naturally reduce the raw yield for all methods.

Dataset Curation

Applying the SkyScraper framework to a large database of global news articles from the Global Database of Events, Language, and Tone (GDELT) (2022–2024), the authors curated two new multi-temporal captioning datasets:

• PlanetScope Dataset: 5,000 sequences with 3,288 confirmed events (average 21 images per sequence).
• Sentinel-2 Dataset: 4,463 sequences with 2,931 confirmed events (average 8 images per sequence).

These datasets were manually verified by annotators for caption accuracy and event dates.

Significance and Claims

The paper claims that SkyScraper demonstrates the effectiveness of agentic feedback as a strategy for surfacing new multi-temporal events in satellite imagery. By automating the identification of imagery related to news events, the system:

1. Overcomes Data Scarcity: Successfully generates multi-temporal captioning datasets where manual curation is prohibitive.
2. Improves Geocoding Accuracy: Significantly outperforms traditional rules-based geocoding methods by filtering out false positives through visual verification.
3. Supports Journalism: Provides a tool to automatically link news narratives to visual evidence, supporting reporting efforts.

The authors conclude that this work facilitates event geocoding, multi-image captioning, and benchmark dataset curation in the remote sensing domain, moving beyond the limitations of bi-temporal, pre-labeled datasets.