Technical Summary: GeoR-Bench

Problem Statement

Current geoscience intelligence research has demonstrated progress in narrow tasks such as remote sensing interpretation and geographic question-answering. However, existing benchmarks remain largely task-specific, focusing on isolated recognition or captioning abilities. They fail to capture the open-ended, visually grounded nature of real-world Earth-system problems. Consequently, it remains unclear how far current AI systems are from achieving genuine geoscience intelligence, which requires not just surface-level visual realism but the ability to understand, reason about, and predict Earth-system changes to produce scientifically valid outputs.

Methodology

Benchmark Design: GeoR-Bench

To address the gap in evaluating geoscience reasoning, the authors introduce GeoR-Bench, a benchmark designed to evaluate geoscience visual reasoning through reasoning-informed visual editing. The core principle is that a model's geoscience intelligence should be measured by its ability to convert Earth-system understanding into a correct, visually grounded generation.

• Dataset Composition: The benchmark contains 440 curated samples spanning 6 geoscience categories (Geomorphology, Hydrology, Atmosphere & Ocean Dynamics, Cryosphere, GIS & Spatial Geometry, and Crustal Science) and 24 task types.
• Data Sources: The dataset integrates heterogeneous visual forms, including real-world Earth observation imagery (satellite, maps, DEMs) and formal scientific representations (cross-sections, diagrams, geological labels).
• Task Structure: Each sample follows a standardized format comprising an input image, a ground-truth output image, a natural-language instruction, and a task-specific judging rubric. Tasks range from temporal evolution (e.g., delta growth, glacier retreat) to spatial reconstruction (e.g., street-to-satellite views) and scientific diagram completion.

Evaluation Protocol

The authors propose a three-dimensional evaluation framework to judge model outputs, supplemented by a strict accuracy metric:

1. Reasoning: Evaluates whether the generated output reflects the intended physical, spatial, or process-based generation. This is assessed by comparing the input, ground truth, model prediction, and a task-specific rubric (e.g., checking if a delta front progrades correctly or if a cyclone rotates counter-clockwise).
2. Consistency: Assesses whether the generated output preserves the required visual relationships with the input image and ground truth, including style, structure, and content preservation outside the edit.
3. Quality: Measures the visual usability of the generated artifact (e.g., absence of blur, distortion, or artifacts) independent of scientific correctness.
4. Strict Accuracy: A sample is counted as correct only if it simultaneously satisfies Reasoning, Consistency, and Quality.

Experimental Setup

The authors evaluated 21 multimodal models, comprising both closed-source (e.g., GPT-Image-2, FLUX.2 variants, Nano Banana series) and open-source systems (e.g., Qwen-Edit, Step-1x, Bagel).

• Evaluation Tools: Reasoning and Consistency were scored using Gemini 3 Flash with normalized 0–100 scales based on the rubrics. Image Quality was evaluated using Q-Align.
• Validation: The automated evaluation pipeline was validated against expert ratings, showing that Gemini 3 Flash achieved the lowest Mean Absolute Error (MAE) for Reasoning and Consistency alignment.

Key Results

Overall Performance

• Geoscience Reasoning is a Critical Bottleneck: No model achieved a strict accuracy above 50%. The highest-performing model, GPT-Image-2, achieved 42.7% strict accuracy.
• Open-Source Gap: The best open-source model, Qwen-Edit-2511, achieved only 10.3% strict accuracy, roughly four times lower than the top closed-source model. Five open-source models scored below 2%, and OmniGen scored 0%.
• Discrepancy between Visual and Scientific Performance: A consistent pattern emerged where visual consistency and image quality scores were significantly higher than reasoning scores. For instance, FLUX.2 Pro achieved a quality score of 79.1 but a reasoning score of only 42.6, resulting in a strict accuracy of 5.6%. This indicates models often generate visually plausible but scientifically incorrect results.

Category-Level Analysis

• Easier Tasks: Hydrology and Cryosphere tasks were the most tractable, with top models reaching 50–57% strict accuracy. These tasks often rely on recurring temporal patterns (e.g., seasonal wet/dry transitions, glacier retreat) where appearance priors align with scientific behavior.
• Harder Tasks: Atmosphere and Ocean Dynamics proved the most difficult, with top models scoring only 35.3%. These tasks require reasoning about complex physical forces (e.g., Coriolis effect, circulation structure) that cannot be spoofed by surface visual priors. Geomorphology also posed significant challenges due to its reliance on process-based sediment transport logic.

Ablation and Analysis

• Explicit Instructions: Providing explicit procedural instructions (e.g., "add distributary channels") significantly boosted open-source models (e.g., Qwen-Edit-2511 improved from 10.3% to 24.7%), suggesting they rely on overt decomposition rather than implicit geoscience priors. Closed-source models showed more modest gains, implying stronger implicit reasoning capabilities.
• Reasoning vs. Generation: The analysis confirms that current models are substantially weaker at geoscience reasoning than at preserving visual appearance. The "Reasoning" dimension is the primary discriminator of geoscience competence, as Quality and Consistency scores tend to saturate across the leaderboard.

Significance and Claims

The paper claims that GeoR-Bench fills a critical gap by evaluating whether multimodal models can convert domain understanding into scientifically correct visual outputs, rather than just producing plausible natural images.

• Measurable Intelligence: The benchmark establishes that geoscience intelligence is measurable through reasoning-informed visual editing, where the edited image serves as direct evidence of the model's ability to externalize Earth-system understanding.
• Current Limitations: The results provide concrete evidence that current AI systems, even frontier closed-source models, fail to capture underlying Earth science processes. They generate "superficially plausible" results but lack the rigorous scientific validity required for critical domains like disaster response and climate adaptation.
• Future Direction: The authors posit that future research must move beyond aesthetic plausibility toward physically grounded and scientifically faithful image generation. The benchmark serves as a probe to drive progress in multimodal geoscience visual reasoning.