Technical Summary: ConceptTree

Problem Statement

Long-horizon robotic manipulation requires agents to make complex, sequential decisions based on continuously changing visual scenes. Current approaches, particularly those relying on end-to-end neural policies or vision-language models (VLMs), often function as "black boxes." While effective at capturing complex patterns, these systems lack transparency: the reasoning behind a specific skill selection is implicit, making it difficult to diagnose failures (e.g., distinguishing between perception errors and flawed decision logic) or intervene to correct behaviors without retraining. Existing interpretable methods, such as symbolic policies or decision trees, typically rely on low-dimensional structured states rather than raw visual observations, while post-hoc explanation methods fail to expose the actual decision logic mapping scene properties to actions. There is a critical need for a framework that grounds high-level decisions in human-interpretable concepts derived directly from visual inputs, while maintaining an explicit, traceable decision process.

Methodology: ConceptTree

The authors propose ConceptTree, a framework that unifies semantically grounded concept learning with structured decision making. The approach decomposes the high-level policy into two interpretable stages: a concept layer that maps visual observations to a normalized concept space, and a decision module (a decision tree) that reasons over these concepts to select skills.

1. Concept Supervision via VLM Annotation

To avoid manual annotation, the framework uses a Vision-Language Model (VLM) as an external verifier during dataset construction.

• Concept Generation: A Large Language Model (LLM) generates a predefined set of task-relevant binary concepts (e.g., fridge_door_open, object_inside_fridge, round_yellow_object_on_counter).
• Verification: For each visual observation, the VLM evaluates whether each concept holds true, producing a binary label vector $y_t \in \{0, 1\}^M$.
• Distillation: This process transfers the reasoning capabilities of the large VLM into a lightweight, structured representation, providing explicit binary supervision aligned with task semantics rather than continuous similarity scores.

2. Concept Learning

The visual encoder (ResNet-18) extracts features from RGB observations, which are then projected into the concept space.

• Projection: Features are mapped to $M$ concept dimensions and calibrated using learnable parameters to produce normalized values $c_t \in (0, 1)^M$ via a sigmoid function.
• Training Objective: The concept layer is trained using a composite loss:
  • Concept Loss: Binary cross-entropy between the predicted concept values and the VLM-generated binary labels.
  • Skill-Aware Regularization: A linear classifier head predicts the ground-truth skill from the concept vector, encouraging the representation to retain decision-relevant information.
• Temporal Context: To handle long-horizon dependencies, the policy input $\tilde{c}_t$ can optionally concatenate the current concept vector with the previous step's vector ($[c_{t-1}, c_t]$).

3. Structured Decision Making

The high-level policy is instantiated as a decision tree that maps concept vectors to discrete skills.

• Reasoning Process: Each internal node tests a binary predicate on a single concept dimension (e.g., $c^{(k)}_t \leq \tau$). A decision corresponds to traversing a path from the root to a leaf, resulting in a sequence of explicit concept-level predicates.
• Interpretability & Intervention: Because the decision process is a sequence of logical conditions over human-understandable concepts, errors can be traced to specific nodes. Furthermore, the framework supports fine-grained intervention: by modifying a single concept value (e.g., correcting a misperceived bread_inside_microwave state), the decision path can be redirected to the correct skill without retraining the model.

Key Contributions

1. Framework for Transparent Decision Making: ConceptTree reframes skill selection as reasoning over human-interpretable concepts, bridging the gap between raw visual inputs and structured decision logic.
2. VLM-Grounded Supervision: The method introduces a novel supervision strategy using VLMs to generate binary concept labels, enabling the learning of reliable concept representations without manual annotation and avoiding the ambiguity of similarity-based supervision.
3. Intervenable Policy: By decoupling representation learning from a tree-based decision structure, the system allows for direct inspection and modification of policy behavior. Errors can be localized to specific concept values and corrected via targeted intervention.
4. Empirical Validation: The approach is evaluated on four real-world robotic manipulation tasks with increasing complexity (Fruits-Snacks, Heat-Bread, Cola, Coffee), demonstrating superior performance and interpretability compared to black-box VLM policies and other concept-based baselines.

Experimental Results

The authors evaluated ConceptTree against four baselines: an end-to-end VLM policy, a label-free CBM (CLIP-based), a VLM-supervised CBM with a linear head (VLG-CBM), and a variant using a sparse linear classifier (ConceptSLC).

• Performance: ConceptTree achieved the highest completion rates across all tasks. It significantly outperformed the black-box VLM policy, particularly in complex, long-horizon scenarios (e.g., Coffee task: 37.09% vs. 5.45% with history).
• Comparison with Concept-Based Baselines: While VLG-CBM and Label-free CBM degraded significantly as task complexity increased, ConceptTree maintained robust performance.
• Decision Structure: Ablation studies revealed that ConceptTree consistently outperformed ConceptSLC (which uses the same concept representation but a linear classifier) on complex tasks. This indicates that the performance gain stems from the structured reasoning of the decision tree rather than just the quality of the concept representation.
• Efficiency: ConceptTree produced compact decision trees with short paths (average length < 4 predicates), whereas sparse linear models relied on distributed evidence across many concepts, offering less concise local explanations.
• Intervention Case Studies: Qualitative analysis demonstrated that modifying a single concept value (e.g., correcting bread_inside_microwave from 0.55 to 1.0) successfully redirected the decision path to the ground-truth skill, validating the framework's capacity for targeted debugging.

Significance and Claims

The paper claims that ConceptTree addresses a critical missing interface in robotic manipulation: the ability to ground high-level decisions in visual observations through human-interpretable concepts while maintaining an explicit, traceable decision process.

The authors argue that their approach offers three distinct advantages:

1. Traceability: Decisions are not opaque outputs but sequences of logical predicates, allowing users to inspect exactly why a skill was chosen.
2. Intervenability: The system supports "fine-grained intervention," where specific decision errors can be corrected by modifying individual concept values without the need for costly retraining.
3. Reliability: By separating perception (concept learning) from decision logic (tree structure), the method achieves robust performance in complex, long-horizon tasks where black-box approaches fail.

The authors acknowledge limitations, noting that the quality of concept representations depends on VLM supervision (which may struggle with subtle visual states) and that the method relies on predefined concept and skill sets, potentially limiting scalability to fully open-ended tasks. However, they conclude that combining semantically grounded representations with explicit decision structures is a promising direction for building interpretable and reliable robotic systems.