💻 computer science

SCLARO: A Dataset for Grounded Scenario-Level Scene Understanding and ScenarioCLIP for Benchmarking

This paper introduces SCLARO, a large-scale dataset with 615,805 images annotated with grounded object, relation, and action information across diverse scenarios, and proposes ScenarioCLIP, a tri-level benchmarking model that outperforms existing methods in joint scene understanding and out-of-domain generalization.

Original authors: Advik Sinha, Saurabh Atreya, Aashutosh A V, Sk Aziz Ali, Abhijit Das

Published 2026-07-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Advik Sinha, Saurabh Atreya, Aashutosh A V, Sk Aziz Ali, Abhijit Das

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand a photograph.

Most current robots are like tourists with a camera: they can tell you, "This is a picture of a woman eating broccoli." They get the big picture right. But they struggle with the details. They might not realize which woman is holding the spoon, or that the broccoli is actually inside a bowl, or that the bowl is sitting on a stove. They see the scene as a blur of objects rather than a connected story.

This paper introduces a solution to that problem, consisting of two main parts: a massive new textbook (the dataset) and a new student (the AI model) designed to learn from it.

1. The Textbook: SCLARO

The authors created a giant library called SCLARO (Scene-Contextual Localisation of Actions, Relations & Objects). Think of it as a collection of over 615,000 photos from everyday life—kitchens, streets, parks, and driving scenes.

What makes this textbook special is how it's annotated (labeled). Instead of just writing a sentence at the bottom of the page, the authors broke every image down into three specific layers of understanding:

The Big Story (Global Action): "A woman is eating broccoli."
The Cast of Characters (Objects): "Woman," "Broccoli," "Bowl," "Spoon," "Stove."
The Connections (Relations): This is the secret sauce. The book doesn't just list the items; it draws little "spotlight" boxes around specific interactions.
- It highlights exactly where the woman is holding the spoon.
- It highlights where the broccoli is sitting in the bowl.
- It highlights where the bowl is on the stove.

The authors used a team of AI assistants to read the images, identify the objects, and then draw these "spotlight" boxes to show exactly where the action is happening. They combined data from five different public sources to build this massive library.

2. The Student: ScenarioCLIP

To prove this textbook is useful, the authors built a new AI model called ScenarioCLIP.

Imagine a student taking a test.

Old Students (like PyramidCLIP): These models try to learn the whole picture, the objects, and the relationships all at once using a single "brain." It's like trying to listen to a symphony, a solo violin, and a drum solo simultaneously with one ear. They get the general vibe, but they miss the specific notes.
The New Student (ScenarioCLIP): This model has three specialized ears (encoders):
1. One ear listens to the whole scene (the big story).
2. One ear focuses only on individual objects (the cast).
3. One ear focuses only on the interactions (the connections).

To make sure these three ears don't get confused or contradict each other, the model uses a "teacher" system (called Knowledge Distillation). Imagine a wise teacher who slowly guides the student, saying, "Hey, the detail you see in the spoon must match the story you tell about the woman." This keeps the student's understanding consistent across all levels.

3. The Results: Did the Student Pass?

The authors tested this new student against the old ones using a variety of challenges:

Finding the Right Picture (Zero-Shot Retrieval): If you ask the AI, "Show me a picture of a woman holding a spoon," ScenarioCLIP is much better at finding the exact match than the old models. It improved significantly in identifying specific objects and relationships.
Spotting the Details (Object Detection): When asked to draw boxes around objects, the new model was slightly more accurate.
Understanding the Story (Scene Graph Classification): When asked to describe the relationships (e.g., "What is the broccoli doing?"), the new model was better at connecting the dots.
The "Real World" Test (Generalization): The authors tested the model on pictures it had never seen before (from different datasets like MS-COCO). Even though it was trained on the SCLARO textbook, it didn't forget how to handle general images. In fact, it performed better than the old models, proving that learning specific relationships actually helps it understand the world better, not worse.

The Bottom Line

This paper argues that to truly understand a scene, an AI needs to stop just looking at the "whole picture" and start paying attention to the specific connections between things.

By creating a massive dataset that highlights these connections (SCLARO) and building a model that learns them separately but together (ScenarioCLIP), the authors have shown that AI can move from simply recognizing "what is there" to understanding "how things relate to each other."

Note on Limitations: The authors admit that because they used robots to create the textbook, there might be a few mistakes (like a robot thinking a cloud is a hat). Also, some rare relationships are hard to find because they don't appear often in the photos. But overall, the system works better than what came before.

Technical Summary: SCLARO and ScenarioCLIP

Problem Statement

Current computer vision paradigms for scene understanding face a critical gap in grounded, multi-level reasoning. While large-scale Vision-Language Pretraining (VLP) datasets provide global image-text pairs, they lack the structured, relational grounding necessary to reason about fine-grained scene composition (i.e., specific objects, their inter-object relations, and the actions performed). Conversely, existing scene graph datasets (e.g., Visual Genome, OpenPSG) offer relational structure but lack the scale and annotation diversity required for effective VLP pretraining.

Consequently, models like CLIP struggle to distinguish between semantically similar but contextually distinct scenarios (e.g., "person holding a bag" vs. "person holding an umbrella") because they lack grounded relational supervision during training. Furthermore, hierarchical models like PyramidCLIP, which align global and object-level features, fail to explicitly model inter-object relations, leaving the efficacy of contextual grounded supervision unquantified.

Methodology

1. The SCLARO Dataset

To address the data gap, the authors introduce SCLARO (Scene-Contextual Localisation of Actions, Relations & Objects), a corpus of 615,805 images spanning indoor, outdoor, and driving scenarios. The dataset is constructed from five public sources: CC3M, OpenPSG, Panoptic Video Scene Graph (PVSG), Multi-Moments in Time, and KITTI2015.

The dataset is unique in its three-tiered annotation structure, generated via a three-stage automated pipeline:

Stage 1 (Structured Scene Annotation): An open-weight Vision-Language Model (Ovis-Gemma 9B) generates a global action caption, a list of object instances, and a set of relation triplets (object1, relation, object2).
Stage 2 (Object Localisation): GroundingDINO predicts bounding boxes for all identified objects.
Stage 3 (Focused Region Construction): Using SAM (Segment Anything Model) and Radial Basis Function (RBF) weighting, the pipeline constructs relation-focused regions. These are specific image crops that highlight the spatial context where a specific interaction occurs (e.g., the region where a "woman" is "holding" a "bowl"), providing explicit visual grounding for relation-level supervision.

The final dataset includes 740 action classes, 4,812 object classes, and 225,609 unique relation triplets.

2. The ScenarioCLIP Model

To benchmark SCLARO, the authors propose ScenarioCLIP, a hierarchical tri-level Vision-Language Model designed to jointly encode global context, objects, and relations.

Architecture: Unlike PyramidCLIP, which uses a shared encoder for all levels, ScenarioCLIP employs disentangled encoders:
- Three distinct visual encoders ( $f_{V,G}, f_{V,O}, f_{V,R}$ ) for global images, object crops, and relation-focused regions, respectively.
- Three distinct text encoders ( $f_{T,G}, f_{T,O}, f_{T,R}$ ) for action captions, object names, and relation triplets.
Training Objectives:
- Contrastive Alignment ( $L_{CA}$ ): Aligns visual and textual embeddings at each level independently (Global-Global, Object-Object, Relation-Relation) using a symmetric image-text contrastive loss.
- Knowledge Distillation ( $L_{KD}$ ): An EMA (Exponential Moving Average) teacher-student mechanism enforces semantic consistency across hierarchical levels.
  - Visual Modality: The global visual teacher regularizes local object and relation embeddings (global context $\to$ local).
  - Text Modality: The direction is inverted; fine-grained object/relation text embeddings regularize the global text embedding (local details $\to$ global), as specific descriptions provide more precise content than broad action captions.

Key Contributions

SCLARO Dataset: A large-scale, grounded dataset providing action captions, object bounding boxes, and relation-focused regions across diverse visual domains, bridging the gap between VLP scale and scene graph structure.
ScenarioCLIP: A tri-level model with disentangled encoders and asymmetric intra-modal knowledge distillation, demonstrating that explicit relation-level supervision improves representation quality.
Comprehensive Benchmarking: An evaluation protocol covering zero-shot retrieval, linear probing, object detection, predicate classification, scene graph classification, and out-of-domain generalization, establishing SCLARO as a standard for scenario-level understanding.

Experimental Results

The authors evaluate ScenarioCLIP against PyramidCLIP and various zero-shot baselines (CLIP, SLIP, TripletCLIP, FineCLIP) across multiple tasks:

Zero-Shot Retrieval: ScenarioCLIP substantially outperforms PyramidCLIP and baselines. Notable gains are observed in Object retrieval (Top-1: +14.19% over PyramidCLIP) and Relation retrieval (Top-1: +3.13%). The results indicate that disentangled encoders learn better representations than shared encoders for fine-grained tasks.
Linear Probe: ScenarioCLIP achieves higher Top-1 accuracy across Actions, Objects, and Relations compared to PyramidCLIP.
Predicate & Scene Graph Classification: ScenarioCLIP improves mean Recall@K (mR@K) for both Predicate Classification (PredCls) and Scene Graph Classification (SGCls), confirming improved relational reasoning capabilities.
Object Detection: Using Faster R-CNN heads, ScenarioCLIP shows consistent gains in Average Precision (AP) over PyramidCLIP, suggesting disentangled object representations serve as a superior backbone.
Out-of-Domain Generalization: On external benchmarks (MS-COCO and Visual Genome), ScenarioCLIP matches or exceeds vanilla CLIP and significantly outperforms PyramidCLIP. This indicates that SCLARO pretraining does not cause catastrophic forgetting and that the relation-focused pretraining transfers effectively to unseen domains.

Significance and Claims

The paper claims that SCLARO and ScenarioCLIP demonstrate the value of grounded, relation-level supervision in vision-language pretraining. The authors assert that:

Prior models lacking explicit relation modeling (like PyramidCLIP) are limited in their ability to reason about compositional scene structures.
The SCLARO dataset fills a critical void by providing pretraining-scale data that pairs relation-specific text with spatially grounded regions.
The disentangled architecture of ScenarioCLIP, combined with asymmetric knowledge distillation, yields superior performance in object and relation understanding without sacrificing general-purpose capabilities.

The authors position this work as a foundation for future research in grounded, compositional scene understanding, including scene graph generation and relational reasoning. They modestly acknowledge limitations, noting that the dataset relies on an automated pipeline (potentially retaining some hallucinations) and that the relation vocabulary is naturally long-tailed.