Technical Summary: DIRECT – When and Where Should You Allocate Test-Time Compute in Embodied Planners?

Problem Statement

Vision-Language Models (VLMs) are increasingly deployed as high-level planners for embodied agents, decomposing abstract instructions into executable sub-skills. A prevailing strategy to enhance capability is scaling test-time compute—expending more resources during inference via techniques like Chain-of-Thought (CoT) reasoning, larger model sizes, or extended memory histories. However, the authors observe that naively scaling compute across all tasks creates significant friction with real-world deployment. Frontier VLM latencies can reach tens of seconds, and uniform scaling often yields diminishing or uneven returns; for instance, a "Thinking" model may be 3.8× slower than a non-thinking counterpart while achieving identical success rates on trivial tasks.

The core problem is that test-time compute is not a uniform lever. Different axes of scaling (reasoning depth, model size, memory) confer qualitatively distinct capabilities that are only beneficial for specific task demands. Allocating expensive compute uniformly is wasteful, while under-allocating leads to failure on complex tasks. The paper argues that determining when and where to spend test-time compute is central to bringing frontier performance to real-world robotic systems.

Methodology: The DIRECT Framework

The authors propose DIRECT (Dynamic Inference Router for Embodied Compute Tradeoffs), a routing framework designed to dynamically allocate test-time compute per task based on multimodal scene context.

Core Architecture

DIRECT operates as a lightweight router that maps a task input $x = (I, \ell)$—consisting of an initial scene observation $I$ (RGB image or multi-view stack) and a natural language instruction $\ell$—to a specific planner $m_k$ from a fixed pool of candidates $\mathcal{M} = \{m_1, \dots, m_K\}$.

• Input Encoding: The router encodes the instruction using a frozen BGE-M3 text encoder and the scene using a frozen SigLIP-family vision encoder. These embeddings are concatenated to form a task feature vector $\phi(x)$.
• Router Models: The framework evaluates various parameter-efficient architectures, including linear models, Multi-Layer Perceptrons (MLPs), k-Nearest Neighbors (KNN), and One-vs-Rest (OVR) classifiers.
• Decision Objective: The router selects the planner that maximizes a utility function $U(q, c)$, balancing quality ($q$, task success) and cost ($c$, latency or FLOPs). The framework supports various utility instantiations, including regression-based heads that predict quality and cost scores to determine the optimal tradeoff.

Data Collection and Training

• Simulation: For benchmarks like VLABench and RoboMME, the authors measure quality (success rate) and cost (latency or FLOPs) for every planner-task pair to construct quality ($Q$) and cost ($C$) matrices.
• Hardware (Physical Robots): Since exhaustive physical rollouts are impractical, the authors employ a synthetic data pipeline. They sample scenes, generate candidate instructions, and use a large VLM to propose skill decompositions. An LLM judge then scores the "steerability" of these plans against reference skills. This allows the router to be trained on physical-style tasks without requiring robot hardware for every training step. The trained router is then deployed zero-shot on physical hardware (Franka arm in a DROID setup).

Scaling Axes Investigated

The framework is evaluated across three dominant scaling axes:

1. Chain-of-Thought (CoT) Depth: Routing between "Thinking" (deliberate reasoning) and "Instruct" (non-thinking) models.
2. Model Size: Routing between VLM variants of varying parameter counts (e.g., 2B to 235B).
3. Memory History: Routing between planners with different memory architectures (e.g., lightweight frame sampling vs. retrieval-based memory like MemER).

Key Results

Experiments were conducted on VLABench, RoboMME, and physical Franka DROID hardware, involving over 270,000 simulated routing decisions and 245 hardware trajectories.

1. Chain-of-Thought Routing

• Finding: CoT reasoning improves average performance but is unnecessary for 44% of tasks, where non-thinking models match or exceed performance with orders of magnitude lower latency.
• Result: DIRECT achieves the highest routing efficiency ($\eta$) across homogeneous and heterogeneous model pairs. On VLABench, it recovers near-expensive model quality while cutting latency by ~30% (e.g., $\eta \approx 75\%$). On physical hardware, DIRECT matches the success rate of a "Thinking" planner at significantly lower latency (e.g., 6.85s vs. 19.58s for a multi-step grocery task).

2. Model Size Routing

• Finding: Scaling model size does not yield a monotonic improvement curve; larger models primarily expand the breadth of skills a planner can command rather than uniformly improving all tasks. Smaller models often suffice for basic skills.
• Result: DIRECT converts erratic scaling curves into monotonic improvements. By routing to the smallest capable model for a given skill set, it recovers the performance of the largest model (e.g., 235B) while significantly reducing average latency. On physical hardware, it matches the success of a 27B planner at near-cheap latency (0.21s vs. 0.36s) by selectively escalating to the larger model only for complex skills like folding or closing.

3. Memory Routing

• Finding: Memory-oriented compute is beneficial only for tasks requiring distant history (e.g., unobservable goals). On easy tasks requiring short recall, lightweight memory schemes outperform heavy retrieval-based models at a fraction of the cost.
• Result: DIRECT identifies tasks requiring long-horizon memory and routes them to memory-augmented planners, while using memory-free planners for observable tasks. On RoboMME and physical hardware, DIRECT achieves success rates close to the oracle (best possible planner) while processing significantly fewer frames than a static memory-augmented planner.

Significance and Claims

The paper claims that naively scaling test-time compute is wasteful and that the benefits of scaling are task-contingent rather than uniform. The primary significance of DIRECT lies in its ability to:

1. Recover Frontier Performance at Lower Cost: By dynamically matching task demands to the appropriate planner, DIRECT achieves success rates comparable to the strongest, most expensive models in the pool but at a fraction of the cost (up to 65% lower average latency in physical experiments).
2. Demonstrate Non-Uniformity of Compute: The work provides empirical evidence that different scaling axes (reasoning, size, memory) yield qualitatively distinct capability gains, necessitating per-task allocation rather than static model selection.
3. Enable Real-World Viability: By reducing latency and token usage without sacrificing success rates, DIRECT addresses the growing friction between frontier VLM capabilities and the real-time constraints of embodied agents.

The authors conclude that treating the high-level planner as a dynamic variable, rather than a fixed capability, is a practical path to achieving frontier-level embodied planning in robotic systems. The framework relies on lightweight, embedding-based features to infer task difficulty, suggesting that effective routing can be achieved without complex, heavy inference mechanisms.