💻 computer science

Object-Level Explanations for Image Geolocation Models: a GeoGuessr use-case

This paper proposes an object-centric analysis pipeline that decomposes attribution maps into interpretable, object-like elements to demonstrate that geolocation models rely on specific visual cues rather than diffuse regions for their predictions.

Original authors: Emilie Durrieu, Christophe Hurter, Philippe Muller, Victor Boutin

Published 2026-05-05

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Emilie Durrieu, Christophe Hurter, Philippe Muller, Victor Boutin

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a game like GeoGuessr, where you look at a photo of a street and have to guess which country it's in. As a human, you don't just stare at the whole picture; you spot specific clues. You might see a specific type of road sign, the color of the vegetation, or the style of a building. You use these concrete "objects" to make your guess.

But how do computer programs (AI) do it? Do they look at the same clues, or are they just guessing based on blurry, random patterns?

This paper tries to answer that question by building a special "detective pipeline" to see what the AI is actually looking at.

The Problem: The AI's "Blurry Vision"

Usually, when we try to see what an AI is thinking, we use tools that highlight a diffuse, fuzzy blob on the image. It's like looking at a photo through a foggy window where a large, indistinct area is glowing. We know the AI is looking somewhere there, but we can't tell if it's looking at a specific car, a street sign, or just the general color of the sky. It's hard to connect that fuzzy blob to a real, recognizable object.

The Solution: Cutting the Image into "Object Puzzles"

The authors created a new method to turn that fuzzy blob into clear, distinct pieces, like cutting a puzzle into individual pieces. Here is how they did it, step-by-step:

Find the Hotspots: First, they used a standard tool (like Grad-CAM) to find the "hotspots" on the image where the AI is paying the most attention. Think of this as the AI pointing a flashlight at the most important parts of the photo.
Cut into Pieces: Next, they took those flashlight-lit areas and used a "segmentation" tool (a digital cutter) to slice them up into object-like chunks. Instead of one big glowing blob, they now have separate pieces that look like a car, a wall, or a sign.
Score the Pieces: They gave each piece a score based on how much it overlapped with the AI's "hotspot." The pieces that matched the AI's attention best were kept.
The "Taste Test" (Deletion & Insertion): This is the most important part. To see if these pieces are actually important, they ran two tests:
- The Deletion Test: They took the specific "object pieces" the AI liked and erased them from the photo. If the AI suddenly got confused and couldn't guess the country anymore, it proved those pieces were crucial.
- The Insertion Test: They took only those specific "object pieces" and put them on a blank background, hiding the rest of the photo. If the AI could still guess the country correctly just from these few pieces, it proved they held all the necessary information.

What They Found

They tested this on photos from France, India, and Japan.

The Result: When they used their "object pieces," the AI performed much better than when they just picked random chunks of the image.
The Analogy: Imagine trying to guess a song by listening to a random 5-second clip versus listening to the specific chorus. The "object pieces" were like the chorus—they contained the real melody the AI needed to make its guess.
The Clues: The pieces the AI focused on were indeed recognizable things: cars, road markings, street signs, and walls. This suggests the AI isn't just guessing randomly; it is actually looking at concrete, human-like clues.

The Catch

The method isn't perfect. Sometimes the "digital cutter" slices things a bit weirdly, creating pieces that are a bit messy or overlap too much. Also, they only tested this on three countries, so we don't know if it works everywhere yet.

The Bottom Line

This paper shows that we can take the "fuzzy" explanations of AI and chop them up into real, understandable objects. By doing this, we can prove that geolocation AI is actually looking at the same kind of visual clues (like signs and cars) that humans use to solve the puzzle, rather than just guessing based on invisible patterns. It's a step toward making AI explain its reasoning in a way that makes sense to us.

Technical Summary: Object-Level Explanations for Image Geolocation Models

Problem Statement
Image geolocation models, which predict the capture location of an image based on visual content, have achieved high performance using deep learning. However, understanding the specific visual evidence driving these predictions remains challenging. While humans rely on concrete, object-level cues (e.g., road signs, vegetation, architectural details) in games like GeoGuessr, standard attribution methods (e.g., Grad-CAM) typically produce diffuse, overlapping heatmaps. These maps highlight broad regions rather than coherent visual entities, making it difficult to link model decisions to specific, interpretable objects or perceptible patterns. This gap limits the ability to interpret, trust, and analyze geolocation models at a concept-level granularity.

Methodology
The authors propose an object-centric analysis pipeline to decompose attribution maps into discrete, object-like elements and evaluate their predictive relevance. The methodology consists of three primary stages:

Model Training and Attribution: A Convolutional Neural Network (ResNet50) is trained for country classification. Attribution maps are generated using gradient-based and score-based methods (Grad-CAM, Grad-CAM++, SmoothGrad, Score-CAM) to identify salient regions influencing the prediction. These maps are thresholded to retain the top- $p$ percentile of pixels, creating "attribution-guided regions."
Object Extraction: Within these salient regions, a segmentation model (specifically variants of the Segment Anything Model: MobileSAM, SAM2, and SAM3) is used to extract discrete visual elements. These elements are not assigned semantic labels but are treated as perceptible units (e.g., a car, a sign).
- Scoring: Each candidate segment is scored based on its overlap with high-saliency pixels, its average attribution value, and the attribution value at its geometric center.
- Filtering: Segments below a relevance threshold are discarded. Overlapping segments are filtered using a containment-based Intersection over Union (IoU) criterion to remove near-duplicates while preserving small details.
- Output: The remaining segments are converted into rectangular bounding boxes with slight padding to preserve context.
Faithfulness Evaluation: The predictive relevance of these extracted elements is tested against randomly selected regions of similar coverage using two standard XAI metrics:
- Deletion Test: The extracted crops are occluded in the original image. A significant drop in classification accuracy indicates the removed regions contained critical information.
- Insertion Test: Only the extracted crops are preserved while the rest of the image is masked. High retained accuracy indicates the crops contain sufficient predictive information.

Key Contributions

Object-Centric Pipeline: The paper introduces a post-hoc framework that bridges the gap between diffuse pixel-level attribution maps and localized, object-like visual evidence without requiring predefined concepts or modifying the underlying classifier.
Quantitative Validation: The study provides a rigorous evaluation showing that attribution-guided crops consistently retain more predictive information than random crops of similar size.
Segmentation Analysis: The work investigates the impact of segmentation quality on interpretability, demonstrating that the choice of segmentation model (e.g., SAM3 vs. SAM2) significantly influences the coverage and effectiveness of the extracted cues.

Results
Experiments were conducted on a benchmark comprising images from three countries: France, India, and Japan (using the OSV-5M dataset).

Predictive Relevance: Across all tested configurations, attribution-guided crops outperformed random baselines. In deletion tests, occluding these regions caused a larger accuracy drop, and in insertion tests, they preserved significantly more predictive information.
Method Comparison:
- Attribution Methods: Grad-CAM++ and other gradient-based methods provided effective saliency maps for guiding the extraction.
- Segmentation Models: SAM3, which generates a higher number of segments, achieved stronger results in both ablation settings compared to SAM2 and MobileSAM. The authors note that SAM2 performed poorly in deletion tests, likely due to generating too few segments, resulting in insufficient spatial coverage to remove meaningful information.
Qualitative Findings: The extracted elements included diverse visual patterns such as vehicles, walls, street signs, and road markings. However, the authors acknowledge limitations, noting that some extracted elements overlap, correspond only to parts of larger structures, or remain amorphous due to segmentation constraints.

Significance and Claims
The paper claims that attribution maps in geolocation models contain localized evidence that can be meaningfully decomposed into perceptible visual regions. This suggests that model predictions can be traced to specific, object-level cues similar to those used by humans. The authors position this work as a step toward object-level interpretability and concept-based explanations in image geolocation.

The authors remain modest regarding the scope of their findings. They acknowledge that the extracted elements are approximate and may be over-segmented. Furthermore, the evaluation is limited to a three-country setting without additional baselines. Future work is proposed to extend the framework to larger, fine-grained benchmarks and to incorporate human evaluation to verify alignment with human reasoning in geolocation tasks.