🤖 machine learning

The Latent That Never Was: A Forensic Re-run of the CVAE Ablation in Action Chunking Transformer

This paper challenges the original findings of the Action Chunking Transformer (ACT) by demonstrating that the reported critical performance drop from removing its conditional variational autoencoder encoder does not replicate in their re-runs, suggesting the encoder's utility is highly sensitive to training and evaluation protocols rather than being a fundamental necessity for the model.

Original authors: Bo Kang

Published 2026-09-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bo Kang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots that learn by watching humans move their arms are no longer science fiction; they are a reality being built in laboratories around the world. To teach a machine how to pick up a block or insert a peg into a hole, researchers often use a method called imitation learning, where the robot studies a library of human demonstrations. A popular tool for this is a system known as the Action Chunking Transformer. It works like a sophisticated prediction engine: it looks at what the robot sees and where its joints are, then guesses a whole sequence of future movements. To handle the fact that humans might move the same object in slightly different ways, the system includes a special internal component, an encoder, designed to capture those subtle differences. During training, this encoder compresses the human's actions into a compact code, a hidden variable that tells the robot how to vary its behavior. However, when the robot actually performs the task on its own, this encoder is turned off, and the robot is told to assume the hidden code is zero. The original creators of this system claimed that this encoder was vital, reporting that removing it caused the robot's success rate to plummet from a respectable 35 percent to a near-total failure of just 2 percent.

A researcher named Bo Kang decided to test this claim from scratch. The original study had not been independently reproduced, and its code did not include a simple switch to turn the encoder off, making it difficult to verify. Kang rebuilt the experiment, running the same robot tasks with the same human demonstrations, but this time with a clear ability to compare the system with and without the encoder. The goal was to see if the dramatic drop in performance was a fundamental truth of the technology or a fluke of how the original experiment was set up. The results were surprising. In Kang's re-runs, the encoder did not save the day. In fact, when the researchers removed the encoder, the robot often performed just as well, or sometimes even better, than when the encoder was present. The massive gap between 35 percent and 2 percent simply did not appear in these new tests.

The investigation went deeper to understand why the original numbers were so different. The researchers found that the outcome of these robot experiments is incredibly sensitive to small details. For instance, the length of time the robot trains matters. If the robot stops training early, the encoder might seem helpful, but if it trains longer, that advantage can vanish or even reverse. Similarly, the method used to pick the best version of the robot's brain for testing can change the results. The original study likely selected a specific moment in training that happened to favor the encoder, while other moments favored the system without it. When the researchers controlled for these factors, using the same training time and the same selection rules, the encoder's supposed superpower disappeared. The difference in success rates became so small that it was impossible to say with certainty whether the encoder helped or hurt.

To understand what the encoder was actually doing, the team looked inside the robot's "mind" to see what information the encoder was storing. They checked if the hidden code carried useful details about the human's style, such as how fast they moved or how smoothly they turned. On the specific tasks tested with the standard training settings, the encoder provided almost no benefit. However, the researchers found that the encoder was not inherently useless; when they removed the penalty that normally restricts the encoder's information, the hidden code improved the reconstruction of actions by up to 84 percent. Furthermore, the encoder was not completely silent; it successfully captured information about the motion within short action chunks, such as speed and acceleration changes, even though it failed to reliably recover the timing or smoothness of entire demonstrations. When they tested the system on a different, simpler task where the encoder was known to be useful, the tools they built successfully detected the encoder's value, proving that their testing methods were sound. But on the complex robot tasks under standard conditions, the encoder provided little measured benefit.

The study also revealed a practical side effect of this finding. Because the encoder is a large part of the robot's software, removing it makes the training process faster and cheaper. In the tests, skipping the encoder calculation increased the speed at which the robot could learn by nearly 25 percent. Since the encoder is not used when the robot is actually performing the task anyway, keeping it during training seems to be an unnecessary expense that offers no clear reward. The researchers concluded that while the original paper claimed the encoder was essential, the evidence suggests it is not. The dramatic failure reported in the original study remains a mystery, likely caused by a specific combination of training length and selection choices that has not been replicated. For now, the most reliable path forward for building these robots appears to be the simpler version, without the encoder, leaving the door open for other researchers to test these findings on different tasks and with different data.

Technical Summary: "The Latent That Never Was: A Forensic Re-run of the CVAE Ablation in Action Chunking Transformers"

Problem Statement
Action Chunking Transformers (ACT) are a dominant architecture for learning robot manipulation from demonstrations, utilizing a Conditional Variational Autoencoder (CVAE) to model differences between demonstrations. The original ACT paper reported a catastrophic performance drop (from 35.3% to 2% mean success rate) when the encoder was removed during training, suggesting the latent variable $z$ is critical for capturing demonstration variance. However, this ablation has never been independently reproduced. The original code lacks a switch to remove the encoder, and the ablation report omits key details such as training length, seed counts, and uncertainty metrics. Furthermore, at inference, ACT sets the latent $z$ to zero (the prior mean), rendering the encoder unused during deployment. This paper investigates whether the reported performance drop is reproducible, what factors might explain the discrepancy, and whether the latent actually carries useful information for reconstructing demonstrated actions.

Methodology
The author conducted a forensic re-run of the ACT CVAE ablation using the original code and a modified implementation based on LeRobot (referred to as "nanoACT"). The study employed a multi-faceted approach:

Reproduction and Comparison: The author re-ran the ablation on the original benchmark tasks (Transfer Cube and Bimanual Insertion) using both human and scripted demonstrations. They compared the original code against their adapted LeRobot implementation to test for implementation-specific effects.
Variable Control: To investigate the unreproducible drop, they systematically varied:
- Training Length: Evaluating checkpoints at 10k, 25k, 50k, 75k, and 100k steps.
- Checkpoint Selection: Comparing final checkpoints against those selected via validation loss (which includes the KL penalty for the encoder but not for the no-encoder variant).
- Execution Strategy: Testing temporal ensembling (averaging overlapping predictions) versus full-chunk execution.
- Data Inputs: Comparing results using converted images (from LeRobot) versus original ACT images.
Latent Probing: Since success rates alone cannot confirm if the decoder utilizes the latent, the author introduced direct probes:
- Reconstruction Benefit: Measuring the reduction in L1 reconstruction error when using sampled latents ( $z$ ) versus a fixed zero latent ( $z=0$ ) in a frozen decoder.
- KL Sweep: Varying the KL penalty weight ( $\beta$ ) across values $\{0.001, 0.01, 0.1, 1, 10\}$ to test if the penalty suppresses latent information, while also running separate experiments with zero penalty ( $\beta=0$ ).
- Positive Controls: Testing the probes on a planar pushing task (PushT) and a "planted-choice" task where specific information was hidden from the observation but present in the action chunk, to verify the probes could detect useful latents when they existed.
- Content Analysis: Attempting to predict demonstration-level properties (timing, speed, smoothness) from the latent mean.

Key Results

Failure to Reproduce the Drop: The reported 33-point drop in success rate (35.3% to 2%) was not reproduced in any tested setting. In the initial 10,000-step re-run using converted images, removing the encoder actually increased the mean success rate (from 26.0% to 45.2%).
Sensitivity to Training and Selection: The relative performance of policies with and without the encoder is highly sensitive to training length and checkpoint selection.
- Training length can reverse which policy scores higher.
- Selecting checkpoints based on validation loss (which differs between the encoder and no-encoder models) can significantly alter the outcome. For instance, at 10,000 steps, validation selection favored the no-encoder policy by a large margin, while final checkpoints showed a smaller or reversed difference.
- Temporal ensembling also influenced results, particularly benefiting the no-encoder policy in specific configurations.
Latent Utility: At the default KL weight ( $\beta=10$ $β = 10$ ) and across a sweep of lower positive weights, the sampled latent provided little to no reconstruction benefit on the ACT benchmark tasks.
- No active latent units were detected (variance of posterior means < 0.01).
- Reconstruction error reduction (z-advantage) was near zero.
- Probes failed to recover demonstration-level style properties (timing, path length) from the latent.
- Conversely, positive controls on PushT and on Transfer Cube with $\beta=0$ successfully detected useful latent information, confirming the probes were sensitive.
Training Efficiency: Removing the encoder reduces training parameters by approximately 20–33% (depending on the implementation) and increases training throughput by 7.8% to 24.5%.

Significance and Claims
The paper concludes that the dramatic performance drop reported in the original ACT paper is not reproducible under the tested conditions, and its cause remains unknown. The author claims that:

The Encoder's Role is Unclear: On the tested benchmarks, the sampled latent provides negligible reconstruction benefit at standard KL weights, and ACT sets this latent to zero at inference anyway.
Uncertainty in Success Rates: Success rate differences between policies with and without the encoder are often within the standard deviation of training seeds, making it impossible to definitively claim the encoder improves or harms task success in these settings.
Methodological Rigor: The study highlights the importance of reporting training lengths, seed counts, and uncertainty in ablation studies. It demonstrates that checkpoint selection criteria and evaluation protocols can drastically alter the perceived efficacy of a model component.
Practical Implication: Removing the encoder reduces computational cost without a clear, reproducible penalty in task success on the tested benchmarks, though the author remains cautious about generalizing this to all tasks or real-world deployments.

The author releases their code, evaluation tools, and results to enable further independent verification and testing on other tasks.

Technical Summary: "The Latent That Never Was: A Forensic Re-run of the CVAE Ablation in Action Chunking Transformers"

More like this