Technical Summary: Elastic Queries Reinforcement Learning (EQRL)

Problem Statement

Vision-Language-Action (VLA) models have emerged as powerful generators for robot manipulation, typically utilizing flow-based or diffusion-based architectures to produce temporally coherent action chunks. However, current execution interfaces are rigid: they employ fixed denoising budgets (sampling steps) and fixed action chunk lengths regardless of the state's difficulty. This rigidity leads to inefficiency; easy states (e.g., free-space movement) often receive excessive computation and frequent replanning, while difficult states (e.g., contact-rich alignment or recovery) may suffer from insufficient feedback and computational resources. Existing post-training methods that adapt VLAs via latent-space steering often leave the execution schedule (sampling budget and chunk length) as manual hyperparameters, preventing the policy from dynamically allocating compute based on state difficulty.

Methodology: EQRL Framework

The authors propose Elastic Queries Reinforcement Learning (EQRL), a framework that treats VLA execution as a joint decision problem over latent steering, denoising budget, and action chunk length, without fine-tuning the underlying frozen VLA model.

1. Latent-Schedule Query Actions

At each query $j$, a lightweight RL adaptor (actor) observes the current state $o_{t_j}$ and outputs a joint action $u_j = (w_j, z^K_j, z^C_j)$:

• $w_j$: A latent steering variable that guides the frozen VLA sampler.
• $z^K_j, z^C_j$: Continuous schedule variables representing residuals around base schedules for the denoising budget ($K$) and chunk length ($C$).

2. Difficulty-Guided Schedule Parameterization

To ensure the schedule is aware of state difficulty, EQRL employs a critic ensemble $\{Q^{(i)}_\psi\}$ trained over the joint latent-schedule action space.

• Difficulty Signal: The system derives a state-difficulty signal $s(o)$ from the disagreement (standard deviation) of the critic ensemble's predictions at the base schedule. High disagreement indicates high difficulty.
• Schedule Prior: This signal generates a "schedule prior" that allocates more denoising steps ($K$) and shorter chunks ($C$) to difficult states, and fewer steps/longer chunks to easy states.
• Final Schedule: The final execution parameters are a combination of this difficulty-derived prior and the actor's learned residuals, clipped to valid integer ranges.

3. Query-Level Learning Objective

EQRL formulates the problem as query-level macro-action Reinforcement Learning:

• Macro-Transitions: Executing a chunk of length $C$ induces a macro-transition in the environment. The Bellman backup uses a chunk-dependent discount factor $\gamma^L$ (where $L$ is the actual executed steps).
• Reward Structure:
  • Task Reward: A terminal success bonus is assigned to the final query of a successful episode.
  • Compute Cost: An episode-level amortized Number-of-Function-Evaluations (NFE) budget is enforced. The cost is defined as the ratio of total denoising steps to total executed actions ($\sum K / \sum C$). Episodes exceeding a budget band incur penalties, while those within the band receive a saving bonus.
• Optimization: The actor is optimized using a maximum-entropy objective (SAC-style) to maximize the critic's Q-value while maintaining entropy. An auxiliary alignment loss encourages the actor's local compute allocation to match the critic-derived difficulty signal.

Key Contributions

1. Formulation of Elastic Execution: The paper formulates frozen VLA execution as a query-level RL problem where the policy jointly selects latent steering, denoising budget, and action chunk length.
2. Critic-Derived Difficulty Guidance: It introduces a mechanism to derive state-dependent compute allocation signals from critic ensemble disagreement, enabling the policy to "know" when to spend more compute.
3. Variable-Length Macro-Action RL: The authors develop a training framework for variable-length chunks using chunk-dependent discounting and an episode-level amortized NFE objective to balance performance and efficiency.

Experimental Results

The authors evaluated EQRL on LIBERO and ALOHA simulation benchmarks, as well as offline and online real-robot manipulation tasks (using DISCOVER Robotics platforms).

• Simulation Performance: EQRL achieved state-of-the-art or improved success rates compared to baselines (DSRL, RLinf-Noise, RLinf-SDE) while significantly reducing inference costs.
  • On LIBERO-4Tasks, EQRL improved Success AUC by 0.04 and reduced relative NFE by 24.3%.
  • On ALOHA-Cube, EQRL achieved 100% final success (vs. 0.94 for DSRL) and reduced relative NFE by 20.0%.
• Real-Robot Performance: In both offline and online settings, EQRL maintained comparable or higher task success while reducing relative NFE by approximately 32% and reducing wall-clock trajectory time by 11–17%.
• Ablation Studies: Experiments on the "Put Cheese" task demonstrated that the gains are not merely due to lowering the average compute budget. Variants that only adjusted steps, only adjusted chunks, or used a fixed budget without difficulty awareness performed worse. This confirms the necessity of joint scheduling and critic-derived difficulty awareness.

Significance and Claims

The paper claims that VLA execution is fundamentally a state-dependent resource-allocation problem. EQRL demonstrates that a frozen VLA policy can be made "elastic" through a lightweight adaptor that learns to allocate computational resources dynamically.

• Efficiency without Fine-Tuning: The method preserves the behavioral priors of the pre-trained VLA while adapting execution efficiency, avoiding the cost of full-model fine-tuning.
• Adaptive Execution: By spending more compute and feedback on difficult phases (e.g., contact alignment) and less on easy phases (e.g., free-space approach), EQRL improves the trade-off between inference cost and task success.
• Practicality: The results suggest that elastic querying is a viable interface for real-world robot learning, translating computational savings into practical speedups and improved contact-time trade-offs.

The authors acknowledge limitations, noting that offline training cannot perfectly observe the causal effects of different schedules without environment feedback, and that current real-robot adaptation requires per-task fine-tuning, limiting generalization across vastly different manipulation skills.