Technical Summary: BRICKCRAFT

Problem Statement

Autonomous robotic assembly of interlocking bricks represents a formidable challenge in long-horizon manipulation, requiring the seamless integration of high-level task reasoning, spatial grounding, and fine-grained contact-rich manipulation. Existing monolithic end-to-end control paradigms struggle with long-term dependencies and the extensive data requirements needed to model complex structural constraints. Furthermore, bridging the gap between high-level semantic plans and low-level visuomotor execution is particularly difficult in brick assembly due to two confounding factors: (1) visual ambiguity caused by periodic geometric units and repetitive features, and (2) spatial multi-modality, where multiple locally valid but discrete assembly poses exist for interlocking interfaces. Current solutions often rely on explicit kinematic states (vulnerable to perception noise) or implicit goal images (difficult to physically ground), failing to provide robust guidance for robots in tight-clearance assembly scenarios.

Methodology

The paper presents BRICKCRAFT, a compositional framework designed to transform digital brick designs into physical products through three distinct phases:

1. Skill-Oriented Assembly Reasoning

BRICKCRAFT decomposes long-horizon assembly tasks into a sequence of actionable subgoals using an assembly-by-disassembly strategy.

• Relative Formulation: Instead of absolute poses, each assembly step is defined by a tuple $(b_{ref}, b_{tgt}, \tau)$, where a target brick ($b_{tgt}$) is attached to a reference brick ($b_{ref}$) within the existing partial structure.
• Geometric Task Encoding: The relative spatial relationship is parameterized by a 4D vector $\tau = [\tau_x, \tau_y, \tau_z, \tau_\theta]$, representing discrete spatial offsets in stud units and relative planar orientation. This formulation abstracts diverse assembly tasks into a finite set of reusable primitive skills.
• Search Algorithm: A Depth-First Search (DFS) explores viable disassembly paths from the goal state, validating candidate moves against four criteria: Connectivity ($C_{con}$), Observability ($C_{obs}$), Operability ($C_{op}$), and Stability ($C_{stab}$).

2. Assembly Intent Grounding (Situated Manuals)

To bridge the gap between symbolic plans and physical execution, the framework introduces situated manuals. This strategy projects abstract assembly intents onto real-time robot observations to provide explicit spatial guidance.

• Visual Reference Generation: A simulation engine (Blender) renders the expected state of the assembly, generating a photorealistic image and semantic masks for the global structure and the specific reference brick.
• Physical Entity Segmentation: A fine-tuned YOLOv8-seg model segments the physical workspace to identify the assembled structure, the gripper, and the target brick.
• Mask Alignment & Tracking: An affine transformation aligns the simulated reference mask with the physical observation. A tracking module (SAM 2) maintains temporal consistency of these masks.
• Visual Augmentation: The system creates a "situated manual" by overlaying the tracked reference mask onto the raw camera feed. Regions outside the active masks are dimmed (attenuated to 25% brightness) with a gradient transition, creating a spotlight effect that highlights task-relevant entities and suppresses visual distractors.

3. Compositional Visuomotor Execution

The execution pipeline chains learned visuomotor skills under the guidance of situated manuals.

• Policy Architecture: The assembly skill is formulated as a Diffusion Policy conditioned on dual-view visual observations (augmented with the situated manual), proprioceptive state, and the task encoding $\tau$.
• Input Modalities: Observations include workspace and wrist camera views, end-effector pose, gripper state, and 3D external forces estimated from joint torques to enhance contact awareness.
• Skill Chaining: Long-horizon tasks are executed as a sequence of pick and assembly skills: $\{S^{(1)}_{pick} \rightarrow S^{(1)}_{asm} \rightarrow \dots \rightarrow S^{(N)}_{asm}\}$.
• Verification: Pre- and post-conditions verify target visibility, kinematic feasibility, and geometric consistency between the physical state and the symbolic plan to prevent error compounding.

Key Contributions

1. BRICKCRAFT Framework: A compositional framework for long-horizon interlocking brick assembly that decouples task reasoning from execution.
2. Relative Task Formulation: A novel approach mapping diverse assembly tasks to a finite set of primitive skills by anchoring steps to reference bricks via relative geometric encodings.
3. Situated Manuals: A strategy that grounds symbolic intents into real-time observations by segmenting and highlighting task-relevant entities, effectively resolving visual ambiguity and providing explicit spatial guidance.
4. Empirical Validation: Comprehensive real-world experiments demonstrating strong compositional generalization to unseen structures.

Experimental Results

The framework was evaluated on a Kinova Gen3 arm with a Robotiq gripper using LEGO Duplo bricks.

• Dataset: 692 demonstration trajectories across 60+ configurations, augmented with the situated manual pipeline.
• Single-Step Performance: BRICKCRAFT achieved an 86.25% success rate across 240 trials on 48 distinct structures (24 seen, 24 unseen).
  • It significantly outperformed baselines: Goal Image-Conditioned Diffusion Policy (GI-DP) and Goal Image-Conditioned $\pi_0.5$ (GI-$\pi_0.5$).
  • GI-DP suffered a 12.5–15.0% performance drop on unseen structures, while GI-$\pi_0.5$ remained bottlenecked below 50% success.
  • EigenCAM visualizations confirmed that situated manuals concentrate visual attention on geometric relationships, whereas baselines exhibited dispersed activations vulnerable to distractors.
• Long-Horizon Performance: The system successfully completed assembly tasks for four distinct structures (Pyramid, Stairs, House, Castle), including the "Castle" which was entirely unseen during training.
  • Completion rates were high for fully supported structures (Pyramid, Stairs) but showed slight declines for structures with unsupported middle regions (House, Castle) due to accumulated errors and structural deformation.
• Ablation Studies: The background dimming strategy for situated manuals outperformed bounding box alternatives by 10–12.5%, suggesting that smooth transitions preserve geometric contours better than artificial edges.

Significance and Claims

The paper claims that BRICKCRAFT addresses the critical bottleneck of grounding high-level semantic reasoning in physical execution for complex assembly tasks. By introducing situated manuals, the framework effectively resolves visual ambiguity inherent in repetitive geometric structures, enabling robots to learn from limited demonstrations and generalize to novel, unseen configurations.

The authors emphasize that while individual skills are proficient, the compositional execution pipeline is essential for long-horizon reliability. The framework demonstrates that decomposing complex tasks into relative, reusable primitives, combined with explicit spatial guidance, offers a viable pathway toward intelligent embodied systems capable of operating in complex, real-world settings. The work suggests that bridging symbolic reasoning with spatially grounded atomic skills is a promising direction for deploying robust agents in broader manipulation scenarios, though the current system is limited to interlocking brick assembly and lacks reactive self-correction capabilities for failure recovery.