💻 computer science

COCOTree: A Dataset and Benchmark for Open Tree-Structured Visual Decomposition

This paper introduces COCOTree, a massive-scale dataset and benchmark for open tree-structured visual decomposition that leverages an automated LVLM and SAM 3 pipeline to generate 1.8M hierarchical nodes across 21K images, accompanied by a new evaluation metric (OTQ) to assess mask precision, label accuracy, and structural consistency.

Original authors: Junhyub Lee, Seunghun Chae, Hyosu Kim

Published 2026-05-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Junhyub Lee, Seunghun Chae, Hyosu Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a photograph of a busy kitchen.

The Old Way:
Traditional computer vision tools look at that photo and say, "I see a person, a table, and a chair." They might even break the person down a little bit: "Head, torso, arms." But they stop there. They treat the image like a flat list of items. If you ask the computer, "What is that handle attached to the cabinet?", it might just see a "handle" floating in space. It doesn't know that the handle belongs to the cabinet, which belongs to the kitchen, or that the handle is made of a specific knob and a screw. It lacks the "family tree" of the objects.

The New Way (COCOTREE):
This paper introduces COCOTREE, a new way of teaching computers to see the world not as a flat list, but as a giant, branching family tree.

Think of it like a Russian nesting doll or a tree with roots and branches:

The Root: The whole image is the trunk.
The Branches: The trunk splits into big objects (a person, a cabinet).
The Twigs: Those objects split into parts (the cabinet splits into a door, a shelf, and a handle).
The Leaves: The parts split even further (the handle splits into a knob and a screw).

The goal is to map out every visible piece of an object, all the way down to the tiniest detail, and connect them all in a logical hierarchy.

How Did They Build This? (The Robot Chef)

Building a map like this for thousands of photos by hand would be impossible. It would take humans years to label every single screw and hinge.

Instead, the authors built a fully automated "Robot Chef" that does the work for them. This robot uses two powerful AI tools working together:

The Brain (LVLM): A Large Vision-Language Model acts like a curious chef. It looks at the image and says, "I see a cabinet. What's inside? Ah, a shelf and a handle!" It uses its "common sense" to guess what parts exist.
The Hands (SAM 3): A precise segmentation model acts like a pair of steady hands. Once the Brain guesses "handle," the Hands draw a perfect outline around that specific handle in the photo.

The Process:
The robot starts with the whole image. It asks the Brain to find the main parts. The Hands draw lines around them. Then, the robot takes just the "cabinet" part, zooms in, and asks the Brain again: "Now that I'm looking only at the cabinet, what do I see?" The Brain says, "A door and a handle." The Hands draw those. This happens over and over, recursively, until the robot can't find any more smaller pieces.

The Result: A Massive Library of Trees

The result is COCOTREE, a dataset containing over 21,000 images and 1.8 million structural nodes (pieces of objects).

Unconstrained: Unlike old datasets that only know 80 or 100 specific words (like "dog" or "car"), this dataset uses an "open vocabulary." It can label a weird, unique object with a long, specific description if it sees it.
Deep: It goes much deeper than previous methods. While old datasets might stop at "door," this one goes "door -> handle -> knob -> screw."
Verified: The authors didn't just trust the robot. They had 20 human reviewers check the work. The humans agreed that the robot's "family trees" were accurate and matched how humans see the world.

How Do We Grade the Robot? (The OTQ Score)

How do you grade a test where the answers are complex trees? You can't just check if the robot got the "car" right. You have to check:

Did it draw the outline of the wheel correctly? (Mask Precision)
Did it call it a "wheel" and not a "tire"? (Label Accuracy)
Did it correctly put the wheel under the car, and the tire under the wheel? (Structural Consistency)

To do this, they created a new scoring system called OTQ (Open Tree Quality). It's like a report card that gives a single score based on how well the robot drew the picture, named the parts, and organized the family tree.

Summary

In short, COCOTREE is a massive, automatically generated library that teaches computers to see the world in layers. It moves beyond just "what is in the picture" to "how the things in the picture are built and connected," using a team of AI robots to build the map and human reviewers to make sure the map is accurate.

Technical Summary: COCOTREE

Problem Statement

Current image segmentation paradigms, while capable of fine-grained pixel-level partitioning, typically provide only a single-level abstraction of visual components. Existing hierarchical datasets capture compositional relationships but suffer from severe limitations in granularity and flexibility: they are often restricted to shallow, two- or three-tier depths and constrained by closed-set vocabularies. This lack of structural depth and open-vocabulary capability hinders compositional reasoning for autonomous systems (e.g., embodied AI agents) that require understanding the deep, unconstrained physical assemblies of real-world scenes. Furthermore, the manual annotation of such highly granular, open-tree structures is prohibitively expensive and cognitively demanding at scale.

Methodology

The paper proposes Open Tree Decomposition, a task that segments an image into hierarchical trees of visual components with unconstrained granularity and flexibility. To address the annotation bottleneck, the authors developed a fully automated recursive annotation pipeline that synergizes Large Vision-Language Models (LVLMs) with the geometric grounding capabilities of SAM 3.

The pipeline operates through the following stages:

Root Decomposition: An LVLM analyzes the input image to propose semantic labels for major constituent elements based strictly on visible evidence. These proposals are grounded into precise instance masks by SAM 3.
Recursive Decomposition: The pipeline enqueues these semantic nodes. For each node, the image is cropped to the node's spatial extent and fed back into the LVLM to propose sub-components (children). This cycle repeats until no further meaningful subdivisions exist.
Evidence-Based Filtering: To ensure quality, the pipeline employs scale-adaptive filtering. Confidence thresholds for mask extraction are dynamically adjusted based on the parent node's size (relaxed for small regions to capture fine parts, stricter for large regions to avoid noise). Redundant sibling proposals are merged if their geometric overlap exceeds 90%.
Instance-Level Conversion: The intermediate semantic tree is materialized into an instance-level hierarchy where individual masks become distinct nodes, linked by spatial containment heuristics to their parent masks.

Key Contributions

The paper introduces COCOTREE, a massive-scale benchmark and dataset for open tree decomposition, alongside a new evaluation metric:

COCOTREE Dataset: Built upon the COCO dataset, it contains over 21,000 images and 1.8 million structural nodes (instance masks). It achieves an unprecedented annotation density of 85.7 masks per image and an average tree depth of 3.448. Crucially, it utilizes an open-vocabulary space of over 3,500 unique labels, capturing the long-tail distribution of complex physical assemblies without being constrained by predefined taxonomies.
Open Tree Quality (OTQ) Metric: The authors propose OTQ, a Panoptic Quality (PQ)-like metric designed specifically for open-tree structures. Unlike previous metrics that fail on unconstrained hierarchies, OTQ jointly assesses:
- Mask Precision: Intersection over Union (IoU) of matched nodes.
- Label Accuracy: Semantic similarity between open-vocabulary labels (using protocols like BERT, WordNet, or strict matching).
- Structural Consistency: The preservation of visual parent-child relationships in the recovered tree.
Automated Pipeline Validation: The paper demonstrates that a fully automated pipeline can generate annotations that align strongly with human structural judgment, overcoming the scalability limits of manual labeling.

Results and Validation

Scale and Density: COCOTREE significantly outperforms existing hierarchical baselines (e.g., PartImageNet, PACO, CPP) in both mask density and structural depth. While existing datasets are bounded to depths of 2 or 3, COCOTREE supports unconstrained depth, with over 56% of Extra Small (XS) masks grounded at depths of 3 or greater.
Geometric Compatibility: When compared against established flat-hierarchy datasets (COCO-17, LVIS, COCO-ReM), the generated masks show high alignment, achieving Median IoU scores ranging from 0.73 to 0.90. Recall drops for extremely small (XS) regions, attributed to the difficulty of proposing and grounding tiny visual evidence.
Human Evaluation: A rigorous human validation study involving 20 reviewers confirmed the quality of the annotations.
- Node-Level: The pipeline achieved $\ge$ 91% "Good" ratings for label-mask matching, instance separation, child validity, and leaf appropriateness.
- Tree-Level: While full-tree consistency exhibited higher rates of minor issues (expected due to the open-vocabulary nature), major failures remained critically low (1.0% to 7.7%).
Benchmark Performance: Experiments using OTQ on baseline models (flat projections and recursive SAM 3 variants) demonstrated that the metric effectively distinguishes between mask quality, label quality, and structural errors. For instance, "parent rewiring" corruption significantly lowered the Tree Quality (TQ) component while leaving mask and label scores high, validating the metric's sensitivity to structural integrity.

Significance

The paper claims that COCOTREE and the OTQ metric define a new benchmark for models that must recover not only visible regions but also their compositional organization. By moving beyond flat region prediction and fixed object-part templates, this work facilitates the development of algorithms for hierarchical analysis with unconstrained granularity. The authors posit that this setting opens a path toward segmentation systems that better support fine-grained reasoning, interaction, and embodied visual understanding.

The paper maintains a modest tone regarding its limitations, acknowledging that:

Reference Multiplicity: Hierarchical parsing is unconstrained, meaning multiple valid decompositions exist; COCOTREE provides a singular, high-fidelity reference based on human consensus.
Open-Vocabulary Evaluation: Label evaluation relies on similarity protocols which may introduce ambiguity for synonyms or rare words.
Automated Errors: The fully automated pipeline inevitably introduces structural and geometric noise, particularly in highly localized, deep-tier regions, which opens future research directions for self-verifying frameworks.