💻 computer science

Retrieve in Time, Correct in Frequency

The paper introduces RTCF, a training-free, low-latency test-time correction framework that enhances frozen vision-language-action policies by causally aligning execution history with successful trajectories to retrieve relevant experiences and transfer only low-frequency motion residuals, thereby significantly improving long-horizon manipulation success without requiring model retraining or additional GPU resources.

Original authors: Yuze Fan, Yue Cao, Pengjie Gao, Haojia Gao, Guangqiu Guo, Ziyue Zhang, Junbo Tan, Bokui Chen, Zhuo Zou, Xueqian Wang

Published 2026-08-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuze Fan, Yue Cao, Pengjie Gao, Haojia Gao, Guangqiu Guo, Ziyue Zhang, Junbo Tan, Bokui Chen, Zhuo Zou, Xueqian Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to make a sandwich. You don't want to program every single muscle twitch; instead, you give the robot a "brain" that looks at the kitchen, reads your instruction ("make a ham sandwich"), and then predicts a whole sequence of moves at once—like a script for the next few seconds. This is called a Vision-Language-Action (VLA) policy. It's like a smart autopilot that can handle complex tasks. However, just like a human who gets distracted, these robots can get confused. If the bread falls slightly, or the lighting changes, the robot might think it's in a different part of the recipe than it actually is. It might try to put the ham on the plate when it's still supposed to be slicing the bread. The problem is that once the robot starts moving, small mistakes can pile up, leading to a messy failure by the end. Scientists want to fix these mistakes in real-time without having to retrain the robot's brain from scratch, which is slow and expensive. They are looking for a way to give the robot a "nudge" or a "hint" when it starts to drift off course, using its own past successes as a guide.

This is exactly what the paper "Retrieve in Time, Correct in Frequency" (RTCF) tackles. The authors propose a clever, free upgrade for these frozen robot brains that doesn't require any new training or extra supercomputers. Think of the robot's memory as a library of successful sandwich-making videos it has watched before. When the robot is currently making a sandwich and starts to wobble, RTCF acts like a super-fast librarian. But here's the trick: it doesn't just look for a video that looks like the current kitchen scene. Instead, it figures out exactly where in the recipe the robot is right now. It asks, "Are we at the 'slicing' stage or the 'toasting' stage?" This is the "Retrieve in Time" part. It aligns the robot's current history with the perfect past videos to find the right moment to borrow from.

Once it finds the right moment, it doesn't just force the robot to copy the whole video, which might be too rigid or wrong for the current situation. Instead, it uses a "frequency" filter. Imagine the robot's movement plan is a song. The low notes are the big, smooth trends (like "move arm forward"), while the high notes are the tiny, quick details (like "wiggle fingers to grip"). RTCF only steals the low notes from the successful memory video to gently correct the robot's overall direction. It leaves the high notes alone, letting the robot's own brain handle the fine details and quick reactions. This is the "Correct in Frequency" part.

The results are quite promising. The researchers tested this on a set of 2,000 robot episodes across four different challenge suites. They found that by using this method, the robot's overall success rate went up from 86.4% to 88.4%. The biggest improvement was on the hardest, longest tasks (called LIBERO-Long), where success jumped from 61.6% to 68.6%. This means the robot successfully finished more complex, multi-step tasks that it would have otherwise failed. Crucially, this didn't require any new GPU power or retraining; the correction happens on a standard computer processor in the blink of an eye, adding only about 10.99 milliseconds of delay per action chunk. The paper explicitly argues against simply replaying old videos or overwriting the robot's entire plan, showing that those methods often make things worse. Instead, by carefully selecting when to borrow and what part of the movement to borrow, RTCF helps the robot stay on track without losing its own reactive spark.

Technical Summary: Retrieve in Time, Correct in Frequency (RTCF)

Problem Statement

Frozen Vision-Language-Action (VLA) policies, which generate temporally extended action chunks, face significant challenges in long-horizon manipulation tasks. While these policies can succeed in isolated episodes, they remain vulnerable to accumulated execution errors and visual aliasing across task stages. A primary difficulty arises because similar visual observations can recur at different stages of a task (e.g., before and after contact), requiring different continuations.

Existing memory-based retrieval methods often fail to address this "progress ambiguity." Simple frame-based retrieval may return actions misaligned with the current execution progress, while direct replay or time-domain fusion can overwrite the reactive structure of the frozen policy proposal. Furthermore, updating the backbone policy online is costly and risky. The paper seeks a solution that improves frozen VLA performance at test time without parameter updates, repeated inference, or additional GPU resources.

Methodology: RTCF

The authors introduce Retrieve in Time, Correct in Frequency (RTCF), a training-free, test-time correction framework. RTCF decouples the retrieval of relevant experience from the application of that experience, separating the "when" (temporal alignment) from the "how" (frequency-domain correction).

1. Retrieve in Time: Progressive Memory Alignment (PMA)

To resolve progress ambiguity without relying on heuristic stage labels, RTCF employs Progressive Memory Alignment (PMA).

Causal Alignment: Unlike standard Subsequence Dynamic Time Warping (S-DTW) which processes a completed query, PMA maintains a causal frontier over monotonic alignments between the growing visual execution history and each complete successful trajectory in a frozen memory bank.
Frontier Mechanism: For every incoming observation, PMA updates an alignment frontier for every memory entry. It preserves competing monotonic hypotheses, allowing later observations to resolve ambiguities introduced by earlier visually similar states.
Retrieval: The method identifies the memory trajectory that best explains the execution history observed so far and determines the precise progress position ( $b^*_t$ ) within that trajectory. This position indexes a future action chunk to be used as a prior, rather than replaying the past action directly.
Memory Representation: To ensure efficiency, visual descriptors are compressed using PCA on frozen SigLIP features, and action records are stored as compact BPE tokens encoding Discrete Cosine Transform (DCT) coefficients.

2. Correct in Frequency: Low-Frequency Residual Correction

Once a relevant action chunk is retrieved, RTCF does not replace the policy's output. Instead, it applies a frequency-selective residual correction.

Frequency Decomposition: Both the retrieved memory action and the frozen policy's proposal are transformed into DCT coefficient space.
Selective Transfer: RTCF transfers only a bounded, low-frequency residual on motion channels.
- Low Frequencies: Captured global motion trends are corrected using the retrieved prior.
- High Frequencies & Gripper: Higher-frequency components (fine local details) and gripper decisions (discrete open/close events) are preserved from the original frozen policy proposal to maintain reactivity and avoid blurring discrete transitions.
Clipping: The residual is clipped to a maximum magnitude ( $\delta_{max}$ ) and scaled by a fixed factor ( $\lambda_0$ ) to prevent instability.

Key Contributions

Low-Overhead Performance Improvement: RTCF improves frozen VLA execution without parameter updates, repeated policy inference, or additional GPU resources. Correction is performed on the client CPU after a single policy invocation.
Progress-Aware and Frequency-Selective Correction: The framework combines causal alignment of execution history with bounded low-frequency residual transfer. This design leverages retrieved experience to correct motion execution while preserving the reactive structure of the frozen policy.
Matched-Memory Evaluation: The authors provide a rigorous evaluation across four LIBERO suites (2,000 episodes per condition) using a matched protocol where the same frozen policy, memory bank, and seeds are used for all conditions.

Experimental Results

The evaluation was conducted on four LIBERO suites: LIBERO-Long, LIBERO-Spatial, LIBERO-Object, and LIBERO-Goal.

Aggregate Performance: RTCF raised the aggregate success rate (SR) of the frozen PI-FAST policy from 86.4% to 88.4% (+2.0 percentage points).
Long-Horizon Gains: The most significant improvement was observed on LIBERO-Long, where SR increased from 61.6% to 68.6% (+7.0 percentage points). This suite, characterized by difficult multi-stage tasks, showed the largest room for improvement.
Task-Level Analysis: Improvements were concentrated in difficult long-horizon tasks (e.g., Task 5 saw an 11-success increase). Tasks already near the performance ceiling (e.g., Spatial, Object) remained largely unchanged, with minor regressions in a small number of cases.
Ablation Studies:
- Removing history-aware alignment (Frame-NN variant) reduced LIBERO-Long SR to 63.6%.
- Removing frequency-selective correction (Time-Domain variant) caused a severe drop to 50.0% SR, indicating that blindly replaying retrieved actions disrupts successful proposals.
- This confirms that both causal alignment and frequency-selective transfer are complementary and necessary.

Deployment and Efficiency

Latency: The correction adds a median latency of 10.99 ms per action chunk for the retrieval and readout stages (6.28 ms for PMA retrieval + 4.71 ms for action readout). The total estimated increase in action-ready latency is approximately 12.20 ms per chunk (a 3.16% increase over the baseline).
Resource Usage: The additional computation is performed entirely on the CPU. The memory footprint for descriptors is minimal (32–50 MiB for an active suite).
Mechanism: The system requires no access to intermediate sampler states or oracle filters, operating solely on the growing execution history and the frozen memory bank.

Significance and Claims

The paper claims that RTCF demonstrates the value of structural priors in robot learning. By recognizing that action chunks are smooth and dominated by low-frequency trends, the method can correct global motion errors using retrieved experience without sacrificing the high-frequency reactivity of the base policy.

The authors emphasize that the gains are most pronounced in long-horizon tasks where accumulated errors and progress ambiguity are most consequential. The work suggests that for frozen VLA policies, test-time correction should focus on aligning execution progress and selectively editing motion trends rather than attempting to overwrite the entire policy output or retrain the model. The results provide evidence that memory-based adaptation can be effective even without updating the policy parameters, provided the retrieval and correction mechanisms respect the temporal and structural constraints of the task.