🤖 AI

Continual Field-Adaptive Models (CFAMs) for Post-Deployment Physical AI

This paper introduces Continual Field-Adaptive Models (CFAMs), a complementary learning architecture featuring a frozen slow-learning component and a fast-learning Capsule Field that enables diverse physical robots to efficiently acquire new skills and adapt to novel, verified near-OOD scenarios in the field through autonomous, gradient-free updates while retaining prior competence.

Original authors: Amarjot Singh, Tanmay R. Pancholi, Jainam Kothari, Shrirang Mahajan, Ketan Bansal, Zackory Erickson, Giuseppe Loianno, Alexandre M. Bayen, Jeff Schneider, Vince Nakayama

Published 2026-09-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Amarjot Singh, Tanmay R. Pancholi, Jainam Kothari, Shrirang Mahajan, Ketan Bansal, Zackory Erickson, Giuseppe Loianno, Alexandre M. Bayen, Jeff Schneider, Vince Nakayama

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot sent into a dangerous, uncharted place: a collapsed mine, a radiation-saturated reactor, or the deep ocean floor. Its mission is to clear debris, inspect a structure, or retrieve an object. In the real world, these environments are unpredictable. The ground might be loose gravel instead of solid concrete; a heavy load might shift unexpectedly; a doorway might be narrower than the maps suggest. For a machine to succeed, it cannot simply follow a script written in a safe laboratory. It must be able to learn from its immediate surroundings, adapting to new conditions the moment they appear, without needing a human to send it new instructions or a supercomputer to retrain its brain. This is the holy grail of physical artificial intelligence: a machine that can step into danger, figure things out on the fly, and keep what it learns without forgetting how to do the things it already knew.

For years, the standard approach to building these machines has been to feed them massive amounts of data in a controlled setting, teaching them thousands of variations of a task before they ever leave the lab. But in mission-critical fields like defense or disaster response, such data does not exist. The environments are too hazardous to instrument, and the situations are too unique to predict. If a robot encounters a situation it hasn't seen before, a traditional model often fails, or worse, it tries to "learn" by overwriting its old knowledge, causing it to forget how to perform its basic duties. This creates a dilemma: how do you build a machine that is smart enough to handle the unknown, yet stable enough not to lose its mind when it does?

A team of researchers has proposed a solution called Continual Field-Adaptive Models, or CFAMs. Their work, tested across five different types of robots ranging from walking dogs and flying drones to human-like arms and off-road vehicles, suggests a new way to build machines that can grow smarter after they are deployed. Instead of trying to retrain the robot's entire brain every time it sees something new, the system separates the robot's "slow" knowledge from its "fast" learning. The slow part, which handles the heavy lifting of perception and decision-making, is frozen solid after the initial training in the lab. It acts as a stable foundation. The fast part is a specialized memory bank where the robot stores new, specific lessons as it encounters them in the field.

The core idea is that the robot does not need to relearn how to walk or how to grasp an object every time the ground shifts or the object moves. Instead, it stores a "competence capsule," a compact record of a specific success. Think of it like a single, precise note added to a vast library of knowledge. When the robot encounters a situation that is slightly different from what it knows—say, a heavier load or a slippery surface—it checks its memory. If it finds a stored lesson that is close enough, it uses that lesson to adjust its actions. If it succeeds, it writes a new, slightly different lesson into its memory bank, effectively expanding the range of situations it can handle. This process happens entirely on the robot's own computer, without any human supervision and without any complex mathematical retraining of its main brain.

The researchers tested this system by first teaching the robots a few tasks in a lab using just a handful of demonstrations. They then sent the robots into simulated and real-world environments where the conditions were slightly different from the training data. The results were striking. While standard robots that tried to adapt by retraining their main brains often failed or forgot their original skills, the CFAM robots kept getting better. In one set of tests, the robots improved their success rate by nearly fourteen percentage points simply by capturing and storing these small, verified successes as they worked. They learned to handle heavier loads, navigate denser vegetation, and recover from slips, all while retaining their ability to perform the original tasks perfectly.

Crucially, the system is designed to be safe and bounded. It only learns from situations that are very close to what it already knows, ensuring it doesn't try to invent entirely new behaviors that could be dangerous. It also has a built-in mechanism to clean up its memory, merging similar lessons together so the memory bank doesn't get too full. The researchers found that this approach allowed the robots to learn efficiently, using only forty percent of the data usually required to train a standard robot, and then continue to grow smarter autonomously in the field.

The study confirms that it is possible to build physical AI that does not stop learning when the training ends. By separating the stable, general knowledge of the robot from a fast, flexible memory for new experiences, the researchers have created a system that can adapt to the messy, unpredictable reality of the physical world. This is not a machine that learns everything from scratch every day; it is a machine that builds upon what it knows, adding small, verified improvements to its repertoire as it goes. For missions where humans cannot go, this kind of bounded, autonomous growth offers a path forward: a machine that can step into the unknown, learn from its mistakes and successes, and keep moving forward without forgetting who it is.

Technical Summary: Continual Field-Adaptive Models (CFAMs) for Post-Deployment Physical AI

1. Problem Statement

The paper addresses a critical gap in mission-critical physical AI (e.g., defense, disaster response, deep-sea exploration): the inability of deployed autonomous systems to adapt to novel, slightly out-of-distribution (OOD) field conditions without catastrophic forgetting or requiring extensive retraining.

Current autonomous systems rely on two paradigms: surveillance (observation only) or one-shot strike (pre-committed action). Neither interacts with or adapts to the environment. Building adaptive systems for these domains faces three specific challenges:

Data Scarcity: Mission-critical environments are hazardous and restricted, offering only handfuls of demonstrations rather than the thousands required by modern robot-learning models.
Compute Constraints: Assets (e.g., drones, quadrupeds in mines) operate on edge devices with limited power and memory, disconnected from data centers, precluding centralized retraining or large-model distillation.
Field Novelty: Deployment introduces new object arrangements, loads, and terrain. A frozen model degrades on these cases, while in-place updates risk "catastrophic forgetting" of previously learned skills.

The paper defines the Field Learning Regime, characterized by intermittent connectivity, episodic supervision, and strict real-time control cycle constraints. An architecture for this regime must satisfy four requirements:

R1 (Efficient Learning): Build competence from few-shot demonstrations.
R2 (Typed Growth): Distinguish between correcting familiar mistakes and extending to new within-family instances.
R3 (Bounded Consolidation): Merge redundant competence under a fixed memory footprint to prevent interference.
R4 (Field-Feasible Growth): Write new knowledge in one shot, gradient-free, on-device, within the control cycle.

2. Methodology: CFAM Architecture

The authors propose Continual Field-Adaptive Models (CFAMs), a brain-inspired architecture based on Complementary Learning Systems (CLS). The system separates learning into a slow-learning component (frozen after deployment) and a fast-learning component (continually updated on-device).

2.1 Core Components

The architecture consists of four functional pieces:

Sensor Cortex (Slow-Learning):
- Function: Lifts multi-modal inputs (vision, LiDAR, tactile) into stable, 3D-grounded geometry.
- Implementation: Uses ScatterNet Hybrid Deep Learning (SHDL), a semi-supervised, brain-inspired encoder. It outputs a discriminative embedding ( $z$ ) and sensor-event signals, lifting the scene into a 3D representation (point clouds) to extract task-relevant anchors (object centroids, grasp points).
- Constraint: Frozen after the lab; provides the stable substrate for perception.
Reasoning Cortex (Slow-Learning):
- Function: Decomposes high-level tasks into a sequence of skills and judges outcomes.
- Implementation: A custom Vision-Language Model (VLM), frozen after deployment. It acts as a Capsule Sequencer (retrieving skill plans) and an Outcome Oracle (verifying semantic task progress).
- Constraint: Read-only; does not learn in the field.
Action Cortex (Slow-Learning):
- Function: Executes skills by retrieving stored capsules and warping them onto the current 3D scene.
- Implementation: Uses the Geometric Residual Transform (GRT). Instead of decoding actions frame-by-frame, it retrieves a stored "seed trajectory" and geometrically warps it to match current scene anchors (e.g., object position).
- Efficiency: Reduces action-decoding calls from $T$ (control steps) to $M$ (skill phases), where $M \ll T$ .
Capsule Field (Fast-Learning Memory):
- Function: The "hippocampus" of the system, storing new competence acquired in the field.
- Unit: The Competence Capsule (CC). A compact record containing:
  - Situation Key ( $h_i$ ): A binary code for retrieval.
  - Perception Correction ( $\Delta z_i$ ): Shifts the sensor embedding.
  - Action Correction ( $\Delta a_i$ ): A residual added to the base policy's output.
  - Action Anchor ( $a^*_i$ ): The seed trajectory and 3D anchors.
  - Regime Tag ( $\psi_i$ ): Distinguishes between CORR (correction of familiar errors) and EXT (extension to new within-family instances).
- Constraint: All capsules live in a fixed memory budget on the edge device.

2.2 The Learning Mechanism: CFUR

The Continual Field Update Rule (CFUR) governs how the system learns in the field without gradients:

Trigger: A new capsule is written only when the base policy's confidence drops and the retrieval distance indicates the situation is at the "edge" of a stored skill's support, and the execution is verified as successful by a sensor-grounded phase objective ( $\Phi_{succ}$ ).
Process:
1. Capture: The system identifies a verified near-OOD success (e.g., a grasp on a heavier load or occluded object).
2. Write: A new Competence Capsule is created in one forward pass and one memory insertion. No gradients are computed; the frozen base is untouched.
3. Consolidation: Redundant capsules are merged, and unused capsules decay (persistence mechanism) to maintain a fixed memory footprint.
Growth: The library grows by adding new capsules at the edges of existing skill envelopes, allowing the system to generalize to new variations without retraining the base.

3. Key Contributions

Architecture (CFAM): An end-to-end field-adaptive model class for Physical AI. It introduces a unit-of-inference shift (per-skill capsule emissions via GRT) and a unit-of-learning shift (gradient-free, one-shot capsule writes).
Dataset: A new in-house multi-embodiment dataset (Skylark) containing 2.6+ million trajectories across five physical platforms (manipulator, quadruped, humanoid, quadrotor, off-road vehicle). All baselines are trained on this same dataset for fair comparison.
Efficient Lab Learning: CFAM reaches the operating point of a standard policy trained on 100% of the data using only 40% of the data (2.5× fewer trajectories).
Autonomous Post-Deployment Growth: CFAM autonomously captures verified near-OOD cases, increasing action success by 13.9 percentage points (from 74.0% to 87.9%) on the variation stream, while adaptation baselines (LoRA, MemoryVLA) fail to improve or degrade.
Retention: In sequential simulation, CFAM retains earlier competence with backward transfer of -0.5 percentage points, compared to -11.4 percentage points for LoRA, demonstrating near-zero catastrophic forgetting.

4. Experimental Results

The paper validates CFAM across synthetic benchmarks (SimplerEnv, LIBERO) and five real-world robotic platforms.

Build Phase (Stage A & B):
- CFAM outperforms standard policies ( $\pi_0$ , CogACT, SpatialVLA) trained on the same data.
- On the held-out test set ( $D_{test}$ ), CFAM achieves a +9.7 pp advantage over the best matched-data baseline.
- One-shot Build (one demonstration per task) generalizes across all five embodiments.
Growth Phase (Stage C):
- On the variation stream ( $D_{var}$ ), CFAM autonomously captures verified successes.
- Result: Action success rises from 74.0% to 87.9% (+13.9 pp).
- Baselines: Gradient-based (LoRA) and memory-augmented (MemoryVLA, CronusVLA) baselines show no improvement or performance degradation under the same exposure.
Retention Phase (Stage D):
- In a sequential simulation suite (5 environments), CFAM maintains performance on earlier tasks.
- Forgetting: Only 2.1% of prior tasks lose >5% success.
- Comparison: LoRA suffers 34.7% catastrophic forgetting.
Ablations:
- Removing the capsule field controller costs 8.3 pp.
- Removing consolidation costs 3.0 pp.
- The combination of Correction (CORR) and Extension (EXT) regimes yields synergistic gains exceeding the sum of individual contributions.

5. Significance and Claims

The paper claims that CFAM provides a bounded form of post-deployment physical intelligence. It demonstrates that:

Learning can be continuous without retraining: Systems can grow their competence autonomously after leaving the lab by writing to a separate, fast-learning memory, leaving the slow-learning substrate (the "brain") frozen.
Data efficiency is achievable: High-performance policies can be built with significantly less training data (40%) by leveraging the capsule architecture.
Forgetting can be mitigated by design: By separating learning into distinct, non-interfering subsystems (slow vs. fast), the system avoids the catastrophic forgetting inherent in gradient-based fine-tuning.
Autonomy is possible in the field: The system can identify, verify, and store new skills from its own execution signals without operator labels or human intervention, provided the new cases are within the scope of existing skill families.

Limitations Acknowledged by the Authors:

The system handles near-OOD cases (variations of known skills) but does not handle open-world novelty (entirely new tasks or objects).
It relies on the frozen base's ability to provide a "good enough" starting point; it cannot repair fundamental deficiencies in the base model's perception or actuation.
Retention results are currently validated in simulation; real-platform retention after long-term growth is untested.
The system requires a "Build" phase with human demonstrations; it does not learn from scratch in the field.

In summary, CFAM offers a structural solution to the stability-plasticity dilemma in physical AI, enabling machines to "keep growing" in dangerous, data-scarce environments while preserving what they have already learned.