💬 NLP

Common Objects Out of Context (COOCo): Investigating Multimodal Context and Semantic Scene Violations in Referential Communication

This paper introduces the Common Objects Out-of-Context (COOCo) dataset to investigate how Vision-Language Models (VLMs) balance local object features and global scene context when generating references, revealing that models adaptively utilize context based on semantic congruency and noise levels.

Original authors: Filippo Merlo, Ece Takmaz, Wenkai Chen, Albert Gatt

Published 2026-02-11

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Filippo Merlo, Ece Takmaz, Wenkai Chen, Albert Gatt

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking into a professional office. You see a desk, a chair, and a computer. Everything makes sense. But suddenly, you look closer and realize that instead of a computer, there is a giant, raw ham sitting on the desk.

Your brain immediately goes, "Wait, what?" You’ve just experienced a semantic violation—something that is physically there, but logically doesn't belong.

This research paper explores whether Artificial Intelligence (specifically Vision-Language Models, or VLMs) experiences that same "Wait, what?" moment, or if it just blindly follows the "vibe" of the room.

The Experiment: The "Ham in the Office" Test

The researchers created a massive digital playground called COOCo. They took thousands of normal photos and used AI to swap objects. They didn't just swap them randomly; they created a scale of "weirdness":

The Perfect Fit: A laptop in an office (Normal).
The Slight Mismatch: A stapler in a kitchen (A bit odd).
The Total Chaos: A giant ham in an office (Very weird).

To make it even harder, they played "Hide and Seek" by adding digital noise (blurriness or static) to either the object or the background to see if the AI would rely on the object itself or just guess based on the surroundings.

The Findings: How the AI Thinks

1. The "Guilt by Association" Problem (Context as a Distractor)

When the object was totally out of place (the ham), the background actually tricked the AI. If the AI couldn't see the ham clearly, it would look at the office setting and say, "I see a desk and a chair... it must be a laptop!"

In this case, the context acted like a bad witness in a courtroom, leading the AI to a false conclusion because it was too focused on the "vibe" of the room rather than the actual evidence.

2. The "Safety Net" Effect (Context as a Facilitator)

However, if the object did belong there (the laptop), the background acted like a helpful GPS. If the laptop was covered in digital static and hard to see, the AI would look at the office setting and say, "I can't see it clearly, but since this is an office, it's probably a laptop." Here, the context helped the AI make a smart, educated guess.

3. The "Staring Contest" (Attention Patterns)

The researchers also looked at where the AI's "eyes" (its attention mechanism) were looking. They found something fascinating: The weirder the object, the harder the AI stares at it.

Think of it like a party. If everyone is wearing suits, you don't notice anyone. But if one person walks in wearing a dinosaur costume, everyone’s eyes lock onto them immediately. The AI does the same: when an object violates the "rules" of the scene, the AI shifts its focus intensely to that object to try and make sense of the mistake.

Why does this matter?

If we want AI to drive cars, perform surgery, or help the visually impaired, we can't have it "hallucinating" based on vibes. We don't want a self-driving car to see a blurry shape on a highway and think, "Well, this is a highway, so it must be a car!" when it's actually a fallen tree.

This paper shows that while AI is getting smarter, it still struggles to balance what it sees (the object) with what it expects (the context). Understanding this "tug-of-war" is the key to making AI more reliable and less prone to being fooled by a "giant ham in the office."

Technical Summary: Common Objects Out of Context (COOCo)

1. Problem Statement

Vision-Language Models (VLMs) are increasingly used for tasks like Referring Expression Generation (REG), where they must identify and name objects within a scene. While human perception is heavily influenced by scene semantics (e.g., expecting a laptop in an office but not a piece of ham), it is unclear how VLMs utilize this contextual information. Specifically, the researchers investigate:

To what degree do VLMs rely on scene context when generating object references?
How does the semantic congruency (the "fit" between an object and its scene) affect model performance?
How do models adapt their attention mechanisms when faced with semantic violations or visual noise?

2. Methodology

The authors introduce a rigorous experimental framework involving a new dataset and multi-model evaluation.

A. Dataset Creation (COOCo):
The researchers developed the Common Objects Out-of-Context (COOCo) dataset by manipulating natural images from COCO-Search18.

Semantic Manipulation: Using an inpainting pipeline (LaMa and PowerPaint), they replaced target objects with alternatives of varying semantic relatedness (Low, Medium, and High) to the scene label.
Noise Injection: They applied Gaussian noise ( $\lambda = 0.0, 0.5, 1.0$ ) to three specific areas: the target region, the context region, or the entire image.
Validation: The dataset was cross-validated against the VISIONS dataset, which provides human-validated consistency ratings.

B. Experimental Setup:

Models: Evaluated several state-of-the-art VLMs, including KOSMOS-2, Molmo, xGen-MM-Phi3/BLIP-3, LLaVA-OneVision, and Qwen2.5-VL.
Task: Object naming via structured prompts (e.g., "What is the object in this part of the image [coordinates]? Answer with the object's name only.").
Metrics:
- RefCLIPScore: Measures semantic alignment between the generated text and the image/reference.
- Accuracy: A cosine-similarity-based threshold (0.9) to determine if the output matches the target label.
- Attention Ratio ( $r$ ): The ratio of attention assigned to the target vs. the context, analyzed across Transformer layers.

3. Key Contributions

COOCo Dataset: A large-scale (18,395 images) dataset designed to evaluate the integration of object-level and scene-level information under controlled semantic and noise conditions.
Behavioral Analysis of VLMs: A systematic study showing that VLMs do not just look at the object; they adaptively balance local and global information.
Interpretability Insights: An investigation into how attention patterns change as a function of semantic fit, revealing a non-monotonic relationship.

4. Key Results

Sensitivity to Semantics: Models perform best when the target is congruent with the scene. As semantic relatedness decreases, performance (RefCLIPScore and Accuracy) drops significantly.
Context as a Double-Edged Sword:
- Facilitator: When the target is obscured by noise, a clear scene context helps the model "guess" the correct object (e.g., seeing a blurry shape in a kitchen leads to "knife").
- Distractor: When the target is semantically inconsistent (e.g., a "ham" in an "office"), the scene context acts as a distractor, leading to incorrect predictions. Removing context (via context noise) actually improves performance for low-relatedness objects.
Attention Dynamics:
- Target Focus: Successful categorization is associated with increased attention to the target, primarily in the middle layers (layers 7–15).
- Non-Monotonicity: Attention follows a U-shaped (convex) curve relative to semantic fit. Models allocate the least attention to moderately fitting objects and more attention to both highly expected and highly unexpected (out-of-context) objects. This mirrors human eye-tracking patterns.
- Compensatory Mechanism: Moderate target noise triggers an increase in target-focused attention as the model attempts to recover information.

5. Significance

This work provides a fundamental understanding of the "reasoning" processes in VLMs. It demonstrates that VLMs are not merely pattern-matching local pixels but are performing a dynamic balance between local visual cues and global semantic expectations. The discovery of the non-monotonic attention pattern suggests that VLMs are developing sophisticated, human-like mechanisms for detecting semantic violations. This has significant implications for improving the robustness of VLMs in real-world applications where visual noise or unexpected objects are common.