🤖 AI

CoStream: Composing Simple Behaviors for Generalizable Complex Manipulation

CoStream is a novel framework that achieves robust, high-precision long-horizon manipulation by composing three independent behaviors—semantic, predictive, and reactive—into a unified $SE(3)$ control command, thereby overcoming the trade-off between task-specific rigidity and monolithic generalization.

Original authors: Haonan Chen, Yuxiang Ma, Stephen Tian, Xiaoshen Han, Wenlong Huang, Feiyang Wu, Yunzhu Li, Jiajun Wu, Edward H. Adelson, Yilun Du

Published 2026-06-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haonan Chen, Yuxiang Ma, Stephen Tian, Xiaoshen Han, Wenlong Huang, Feiyang Wu, Yunzhu Li, Jiajun Wu, Edward H. Adelson, Yilun Du

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to insert a delicate computer chip into a slot that has less than a millimeter of wiggle room. If you tilt it even slightly, it jams. If it slips in your hand, it jams. If you push too hard, you break it.

Current robots struggle with this because they are built in two extreme ways:

The "Rigid Blueprint" Robot: This robot follows a strict, pre-drawn map. It's great at following lines, but if the map is wrong by a tiny bit, or if the object moves, the robot crashes because it can't change its plan on the fly.
The "Black Box" Robot: This robot tries to learn everything at once by watching thousands of videos. It's very flexible and can learn new things, but it's clumsy. It doesn't have the "feel" to know when it's touching something too hard, and it needs to be retrained from scratch for every single new task.

CoStream is a new way of building robots that combines the best of both worlds by acting like a human team rather than a single machine.

The Three-Headed Team

Instead of one giant brain trying to do everything, CoStream splits the job into three simple, independent "behaviors" that talk to each other constantly. Think of it like a construction crew:

The Architect (Semantic Behavior):
- What it does: This is the "big picture" thinker. It reads your instructions (like "put the GPU in the slot") and uses AI to understand the geometry. It sets the goal: "The chip needs to be here, facing this way."
- Analogy: It's like the foreman who points at the blueprint and says, "We need to build the wall right there." It doesn't worry about the tiny details of how the bricks touch; it just sets the destination.
The Dreamer (Predictive Behavior):
- What it does: This part watches "imagined videos" of the task. It predicts the smooth path the robot should take to get from point A to point B. It generates a nominal motion (a rough draft of the movement).
- Analogy: It's like a dancer rehearsing a routine in their head. They know the general steps and the flow of the dance, but they aren't reacting to the music's sudden beat changes yet.
The Touch Sensor (Reactive Behavior):
- What it does: This is the "feel" expert. It runs super fast (25 times a second) and uses special sensors in the robot's fingers to feel the object. If the chip slips or hits a bump, this part instantly makes tiny corrections.
- Analogy: This is like your own hand when you are threading a needle. Your eyes see the needle, but your fingers feel the thread. If the thread wiggles, your fingers adjust instantly without you having to stop and think.

How They Work Together: The "Right-Hand Rule"

The magic of CoStream is how these three parts combine. They don't take turns; they work simultaneously.

Imagine the robot's movement is a song.

The Architect writes the lyrics (the goal).
The Dreamer composes the melody (the path).
The Touch Sensor is the drummer keeping the beat and adjusting the tempo if the singer stumbles.

In CoStream, these three outputs are mathematically "multiplied" together at every single split-second of the robot's movement.

If the Architect says "Go North," and the Dreamer says "Move forward," but the Touch Sensor feels "Wait, we hit a wall, slide left," the robot instantly combines all three into one perfect movement: "Go North, forward, but slide left."

This allows the robot to have a long-term plan (from the Architect) while having instant reflexes (from the Touch Sensor).

The Results: What They Actually Did

The researchers tested this system on real robots with real tasks. They didn't just simulate it; they built it.

Precision Assembly: They successfully inserted a Drill bit, RAM stick, CPU chip, and GPU card into tight slots. These tasks require millimeter-level precision.
- The Result: The "Rigid Blueprint" robots and the "Black Box" robots failed completely (0 out of 15 tries). CoStream succeeded almost every time (14 or 15 out of 15).
Human Interruption: While the robot was inserting a chip, the researchers physically grabbed the chip and twisted it in the robot's grip to simulate a slip.
- The Result: The robot felt the twist, corrected its grip instantly, and finished the task without dropping the chip or breaking it.
Zero-Shot Transfer: They used the exact same robot code to move from delicate CPU handling (which needs to be gentle) to heavy RAM locking (which needs to be forceful). They didn't have to retrain the robot or change its code; the "team" just adjusted its behavior based on the new object.

In Summary

CoStream proves that you don't need one giant, super-complex brain to do complex robot work. Instead, you can build a system where a planner, a predictor, and a feeler work together in real-time. This allows robots to be both smart enough to understand a task and sensitive enough to handle delicate, messy, real-world physics without crashing.

Technical Summary: CoStream

Problem Statement

Long-horizon, contact-rich manipulation tasks (e.g., inserting a GPU into a PCIe slot with millimeter clearance) present a dual challenge: they require high-precision control to avoid jamming and out-of-the-box generalization to handle new tasks without retraining. Current robotics paradigms struggle to satisfy both simultaneously:

Classical Modular Pipelines: These decouple perception, state estimation, planning, and control. While they can achieve high precision, they are brittle; adapting them to new tasks requires costly, bespoke engineering and manual redesign of modules.
Monolithic End-to-End Policies: Models like Vision-Language-Action (VLA) networks offer better generalization but lack the precision required for complex, out-of-distribution contact tasks unless retrained with extensive new data.

Both paradigms share an implicit assumption that complex manipulation requires a rigid, indivisible control system. This paper challenges that assumption, proposing that complex capabilities can emerge from composing simple, independent behaviors.

Methodology: CoStream Framework

CoStream is a framework that orchestrates foundation models and diverse sensing modalities into multiple composable core behaviors. Unlike conventional pipelines that pass a completed plan through fixed interfaces, CoStream maintains behaviors as independent, parallel streams that operate at their natural rates and compose dynamically.

1. Architecture Overview

The system operates on a shared SE(3) task-space interface. At every control step, three distinct behaviors generate components that are right-multiplied to form a single end-effector pose command ( $W T_{cmd}$ ), executed by a compliant controller.

Semantic Behavior (Task-Frame Anchor):
- Function: Extracts spatial constraints and defines a task frame ( $W T^k_I$ ) relative to the world.
- Mechanism: Uses Large Language Models (LLMs) and Vision-Language Models (VLMs) to parse instructions and scene summaries (objects, affordances, predicates). It minimizes an SE(3) alignment objective to ground the task frame.
- Output: A static or slowly updated anchor frame, not a trajectory. This ensures axis ownership and stiffness remain diagonal.
Predictive Behavior (Nominal Motion):
- Function: Generates a continuous nominal motion trajectory ( $I T^k_{traj}(t)$ ) within the task frame.
- Mechanism: Utilizes a video world model conditioned on observations and step instructions to sample rollouts. A VLM critic selects the best rollout, which is then lifted to a 3D keypoint tracker to form an object-centric reference trajectory.
- Output: A motion prior (nominal path) rather than a precise, rigid plan.
Reactive Behavior (Tactile Residual & Force Loop):
- Function: Provides high-frequency (25 Hz) corrections for in-hand slip and contact forces.
- Mechanism: Uses a GelSight tactile sensor to track object pose relative to the gripper (via NormalFlow) and estimates end-effector wrench. It generates a tactile residual ( $\Delta T^k_{tac}(t)$ ) to cancel sliding and flags contact events (slip, excess force).
- Output: A rigid-body transform that corrects the nominal motion and a force loop signal for the controller.

2. Composition and Control

Action Composition: At each controller tick, the system computes the final command:
$W T^k_{cmd}(t) = (W T^k_I \cdot I T^k_{traj}(t)) \cdot \Delta T^k_{tac}(t)$
This right-multiplication allows the behaviors to remain independent. If a stream is missing or stale, a fallback behavior ( $H_k$ ) ensures graceful degradation rather than stalling.
Controller Compilation: A deterministic compiler maps stage specifications to robot-level parameters (Cartesian stiffness, damping, impedance/admittance axes, force limits).
- In free space, the controller tracks the nominal motion with high stiffness.
- During contact, it switches to Cartesian impedance (virtual spring for force maintenance) or position-based admittance (to bound force), depending on the stage.

Key Contributions

Behavior Composition Interface: A novel interface for contact-rich manipulation that decouples semantic grounding, predictive motion generation, and tactile/force reaction into independent behaviors.
Shared SE(3) Representation: A task-space representation allowing each behavior to run at its natural rate, contributing only the component it is reliable for (task frame, motion prior, or contact correction).
Empirical Validation: Demonstration of the framework on eight real-world tasks, showing high-precision assembly, robust recovery from manual perturbations, and zero-shot transfer to everyday manipulation without retraining.

Experimental Results

The authors evaluated CoStream on two groups of tasks using two Franka Emika Panda robots (one with a GelSight Mini tactile sensor).

Baselines: Compared against VoxPoser (modular vision-language planning) and $\pi_0.5$ (monolithic VLA policy).
High-Precision Assembly Tasks:
- Tasks: Drill insertion (0.5 mm clearance), RAM insertion, CPU placement, GPU insertion.
- Results: CoStream achieved success rates of 14/15 to 15/15. In contrast, VoxPoser and $\pi_0.5$ failed completely (0/15) on these tasks.
- Failure Analysis: VoxPoser failed due to open-loop planning that could not correct geometric deviations before contact. $\pi_0.5$ failed during contact, unable to distinguish contact events from perceptual ambiguity, leading to destabilizing high-frequency motions.
Everyday Manipulation Tasks:
- Tasks: Turning on a lamp, wiping a whiteboard, cup-to-plate transfer, clothes-to-box transfer.
- Results: CoStream outperformed $\pi_0.5$ across all settings, with significant gains in object transfer tasks (e.g., 13/15 vs 4/15 for cup-to-plate).
Ablation Studies:
- Removing the Reactive Behavior on the Drill Insertion task caused success rates to collapse from 15/15 to 3/15, confirming that tactile feedback is essential for sub-millimeter precision.
- Robustness: The system successfully recovered from manual perturbations (e.g., rotating the held object inside the gripper) by reorienting the end-effector based on tactile feedback.

Significance and Claims

The paper claims that CoStream demonstrates a regime where complex manipulation is achieved not by a single complex controller, but by the composition of simple, independent behaviors.

Generalization without Retraining: The same set of behaviors successfully handled diverse physical constraints (from delicate CPU seating to high-force RAM locking) and transferred to everyday tasks without policy retraining or module redesign.
Precision via Composition: By separating the "what" (semantic), the "where" (predictive), and the "how" (reactive), the system achieves millimeter-level precision in contact-rich scenarios where monolithic policies fail.
Modest Scope: The authors acknowledge limitations, noting that the system relies on upstream perception (pose estimates) and is currently scoped to structured rigid-object manipulation. It does not yet handle deformable objects, dexterous in-hand manipulation, or ambiguous goals. The reactive behavior is also limited by tactile coverage and wrench estimation accuracy.

In summary, CoStream proposes that decomposing manipulation into semantic, predictive, and reactive streams, composed via a shared SE(3) interface, offers a robust path toward generalizable, high-precision robotic manipulation.