💻 computer science

SADL: What to Ignore? A Benchmark for Subject-Aware Distractor Localization

This paper introduces SADL, the first real-world benchmark for subject-aware distractor localization, which evaluates Vision-Language Models on identifying and filtering visual distractions while preserving compositionally essential objects, revealing that while these models can detect distractors, they systematically over-apply exclusion rules.

Original authors: Cao-Tri Nguyen, Nguyen-Khoa Luong, Vinh-Tiep Nguyen, Minh-Triet Tran

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Cao-Tri Nguyen, Nguyen-Khoa Luong, Vinh-Tiep Nguyen, Minh-Triet Tran

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a photograph, and you want to edit it. You tell a smart computer, "Make this person the star of the photo." The computer's job is to figure out what to keep and what to delete.

The problem is, photos are messy. They are full of background noise, other people, and random objects that might steal the spotlight. If the computer deletes the wrong thing, the photo looks weird (like deleting the chair someone is sitting on). If it keeps the wrong thing, the photo still looks cluttered.

This paper introduces a new tool called SADL (Subject-Aware Distractor Localization) to teach computers how to make these decisions better. Here is the breakdown in simple terms:

1. The Core Problem: "Who is the Star?"

Most current photo-editing AI works like a generic security guard. It sees a crowd and says, "Everyone except the person in the center is a distraction." It doesn't understand context.

The Analogy: Imagine a party.
- Scenario A: You tell the AI, "Focus on the man in the blue shirt." The AI should delete the loud clown next to him because he's distracting.
- Scenario B: You tell the AI, "Focus on the person in the banana wig." Now, that same clown might be the main star, and the man in the blue shirt becomes the distraction.
- The Failure: Old AI models can't switch hats. They treat the image as a static object, not a story where the "main character" changes based on your instructions.

2. The New Rulebook: "The Inclusion and Exclusion List"

The authors created a strict rulebook for the AI to follow, based on how human photographers think. To decide if an object should be deleted, it must pass two tests:

Test 1: The "Attention Grabber" List (Inclusion Factors)
Is this object trying to steal the show? It checks five things:

Is it bright? (Visual Saliency)
Is it right next to the star? (Spatial Proximity)
Does it look weird here? (Semantic Incongruity - e.g., a cow in a living room)
Is it the same type of thing as the star? (Categorical Similarity - e.g., two people competing for focus)
Is it huge? (Scale Dominance)

Test 2: The "Do Not Touch" List (Exclusion Rules)
Even if an object is loud or bright, sometimes you must keep it.

Is it part of the star? (e.g., The hat the person is wearing).
Is it just background noise? (e.g., Tiny leaves on a tree that don't compete for attention).
Is it necessary for the action? (e.g., The chair the person is sitting on. If you delete the chair, the person falls! This is a "Functional Dependency").

3. The Benchmark: A Tough Exam for AI

The researchers built a massive test set called SADL with 1,000 photos and 1,800 different "what-if" scenarios. They asked seven different AI models (including big names like GPT-4 and Gemini) to take this exam.

The Results: The "Over-Protective" AI
The AI models were actually quite good at spotting the "Attention Grabbers." However, they failed miserably at the "Do Not Touch" list.

The Metaphor: Imagine a bouncer at a club who is too strict. The bouncer sees a loud person (a distractor) and immediately kicks them out, even if that loud person is the birthday girl's best friend holding her cake.
The Finding: The AI models were "over-triggering" the exclusion rules. They were deleting things they should have kept (like chairs or accessories) because they were being too cautious about the scene's logic, rather than focusing on the specific person you asked them to highlight.

4. The "Grounding" Hurdle

There was a second problem. When the AI said, "Delete the red car," it often couldn't point to exactly where the red car was in the photo.

The Analogy: The AI is like a director who can write a great script ("Cut the scene with the dog!") but can't point to the specific dog on the set. When forced to draw a box around the object to delete it, the AI's performance dropped significantly.

Summary of What They Found

AI is smart at reasoning but dumb at context: It can tell you what is distracting, but it often deletes the wrong things because it doesn't understand the specific relationship between the "star" and the "supporting cast."
The "Three-Category" Test: By forcing the AI to choose between "Delete," "Keep (Hard Negative)," and "Ignore," the researchers found that the AI was confusing "Keep" with "Ignore."
The Solution: To fix this, AI needs to be trained to stop being a generic filter and start acting like a director who understands that the "main character" changes the rules of the scene.

The Bottom Line:
SADL is a new diagnostic tool that proves current AI photo editors are too aggressive. They delete things they shouldn't because they aren't paying close enough attention to who you told them to focus on. The paper provides the data and the rules to help build smarter, more careful editing tools in the future.

Technical Summary: SADL – Subject-Aware Distractor Localization

Problem Definition
Photographs frequently contain visual distractors—objects that compete for attention with the intended subject and weaken compositional integrity. While modern editing tools facilitate object removal, the critical preceding step of identifying which objects to remove remains a manual process requiring compositional expertise. Existing approaches, including saliency models and open-vocabulary detectors, suffer from a structural limitation: they generate a fixed distractor map per image without subject awareness. Consequently, they cannot adapt to shifting user intent; an object may be a distractor relative to one subject (e.g., a person) but an essential compositional anchor relative to another (e.g., a dog). Furthermore, context-agnostic removal risks disrupting semantic coherence, such as removing a chair a subject is sitting on. This paper formalizes the task of Subject-Aware Distractor Localization (SADL): given an image $I$ and a textual description of a main subject $M$ , the system must identify which other objects are distractors relative to $M$ .

Methodology and Framework
The SADL framework defines a distractor as an object that satisfies at least one of five inclusion factors (mechanisms driving attentional competition) while triggering none of three exclusion rules (contextual constraints mandating retention).

Inclusion Factors ( $F_1$ – $F_5$ ):
- $F_1$ (Visual Saliency): High color contrast, luminance, or texture novelty.
- $F_2$ (Spatial Proximity): Adjacency to the subject $M$ .
- $F_3$ (Semantic Incongruity): Out-of-context relative to the scene.
- $F_4$ (Categorical Similarity): Sharing the same semantic category as $M$ , causing referential ambiguity.
- $F_5$ (Scale Dominance): Occupying a disproportionately large area relative to $M$ .
Exclusion Rules ( $E_1$ – $E_3$ ):
- $E_1$ (Attribute of $M$ ): Objects held, worn, or attached to $M$ .
- $E_2$ (Neutral Environmental): Expected background elements too small or numerous to compete individually.
- $E_3$ (Functional Dependency): Objects whose removal would render the subject's activity logically incoherent.

Dataset Construction (SADL)
The authors introduce SADL, the first real-world benchmark for this task, comprising:

Scale: 1,000 high-resolution images from diverse public datasets (COCO, Winoground, Visual Genome, Open Images) across eight scene categories.
Cases: 1,800 subject-aware cases (mean 1.8 cases/image), where a single image yields multiple cases based on different specified subjects $M$ .
Annotations: 14,617 annotated candidates, including 1,938 hard negatives (objects that appear to be distractors but must be retained due to exclusion rules).
Pipeline: A hybrid annotation process using agentic pre-labeling (Qwen3-VL-32B) followed by human review and adjudication by professional photographers to resolve disagreements. Inter-annotator agreement (Cohen's $\kappa$ ) reached 0.534 for binary labels, with higher agreement on geometric/categorical factors than perceptual ones.

Evaluation Framework
The paper employs a dual-protocol evaluation to isolate specific failure modes:

Guided Classification (GC): The model receives the image, subject description, and a pre-enumerated list of candidates, predicting a three-class label (Distractor, Excluded, Non-distractor) with factor/rule attributions. This isolates compositional reasoning.
Open Detection (OD): The model receives only the image and subject description, freely enumerating candidates which are then grounded to masks via SAM 3. This measures end-to-end deployment performance.

Key Results
Evaluation of seven Vision-Language Models (VLMs) and four non-reasoning baselines reveals a hierarchy of systematic failures:

Exclusion Over-Triggering: Frontier models (e.g., GPT-4.1, Gemini 3 Pro) demonstrate high capability in identifying distractors but systematically over-apply exclusion rules. When exclusion rules are removed from the prompt, distractor recall (D-Rec) for these models jumps significantly (e.g., GPT-4.1 from 65.1% to 88.9%), indicating the primary bottleneck is exclusion calibration rather than detection failure. Models tend to apply exclusion at the scene level rather than anchoring it to the specific subject $M$ .
Factor Detection Failure: Weaker open-weight models (e.g., Pixtral-12B) fail even when exclusion rules are removed, suggesting a deficit in the perceptual grounding of inclusion factors, particularly for semantic incongruity ( $F_3$ ).
Output Contamination: Some open-weight models produce structurally inconsistent outputs where exclusion criteria fire without triggered inclusion factors, silently suppressing true distractors.
Spatial Grounding Bottleneck: A significant performance gap exists between GC and OD F1 scores ( $\Delta$ F1 is negative for all models). This drop is primarily attributed to label-to-mask conversion failures (e.g., verbose or imprecise text descriptions failing to ground via SAM 3) rather than the segmentation model itself.

Significance and Contributions
The paper claims three primary contributions:

Formal Framework: A rigorous definition of subject-aware distractor localization grounded in visual attention theory, enabling process-level error decomposition by root cause (inclusion vs. exclusion vs. grounding).
Diagnostic Benchmark: SADL provides a unique resource with hard negatives and a dual-protocol design that exposes failure modes (specifically exclusion over-triggering) that binary benchmarks conflate or obscure.
Empirical Analysis: The study identifies four distinct failure modes (exclusion over-triggering, output contamination, factor detection failure, and spatial grounding failure), offering specific diagnostic signatures and remediation directions for multimodal systems.

The authors conclude that while VLMs possess strong compositional reasoning capabilities, their end-to-end deployment is currently hindered by a lack of subject-conditioned exclusion calibration and robust spatial grounding. SADL serves as a foundational tool to advance these specific capabilities in future research.