Runtime-Incremental Transformer for Reinforcement-Learning-Based Adaptive Control

Technical Summary: Runtime-Incremental Transformer for Reinforcement-Learning-Based Adaptive Control

Problem Statement
The paper addresses the challenge of learning-based adaptive control for robotic manipulators subject to non-observable friction memory, specifically Stribeck friction. In such systems, the friction force depends on a hidden internal state with a finite "memory horizon" ($\tau_z$), rendering the closed-loop system non-Markovian with respect to measured states. Previous work by the authors (INCRT-1L) utilized a single-layer transformer encoder with a fixed number of attention heads ($K$) to process a sliding window of past measurements. However, this approach relied on an offline, two-phase procedure (Phase-1/Phase-2) to select $K$ via a surrogate operator analysis. While effective for short and medium memory horizons ($\tau_z \in \{1, 2\}$ s), this fixed-capacity approach suffered from catastrophic failures in 40% of training seeds at long memory horizons ($\tau_z = 5$ s). The failures were attributed to local attractors in the reinforcement learning (RL) optimization landscape that were invisible to the offline surrogate analysis.

Methodology: Runtime-INCRT
The paper introduces Runtime-INCRT, a mechanism that dynamically grows and prunes the attention heads of the transformer encoder during the reinforcement learning training loop, eliminating the need for offline architecture search.

1. Variable-Head Architecture: The meta-controller employs a transformer block with a maximum capacity of $K_{max}$ heads. At any training step $t$, an active mask determines the current number of heads $K(t)$.
2. Runtime Triggers:
   • Growth: Triggered when the effective rank ($\hat{\rho}_\alpha$) of the on-policy context-token distribution exceeds the current capacity scaled by a threshold ($K(t)(1+\epsilon_{grow})$). This indicates that the current representational capacity is insufficient to capture the complexity of the policy's input distribution.
   • Pruning: Triggered when the relative output magnitude of an active head falls below a threshold ($\epsilon_{prune}$), flagging the head as redundant.
3. Operational Constraints:
   • Cooldown: A minimum number of steps ($\Delta_{grace}$) is enforced between events to allow the policy to adapt.
   • Soft Initialization: When a new head is added, its output projection matrix ($W^O$) is initialized to zero. This ensures the new head contributes zero output initially, preserving policy continuity.
   • Pruning Bound: When a head is removed, the change in policy output is bounded by the pruning threshold.

Theoretical Guarantees
The authors establish three analytical guarantees to ensure the stability of the learning process:

• Zero-Loss at Growth (Proposition 7): Due to the soft initialization ($W^O=0$), the addition of a head results in no change to the policy output at the moment of growth.
• Policy Continuity Bound (Theorem 9): The cumulative discontinuity introduced by pruning events is bounded linearly by the number of prune events and the threshold $\epsilon_{prune}$.
• Convergence of Head Count (Proposition 11): Provided the empirical effective rank of the context distribution stabilizes, the active head count $K(t)$ converges to a steady-state interval determined by the growth and prune thresholds.

Experimental Results
The method was evaluated on a 2-DOF manipulator with Stribeck friction across three memory regimes ($\tau_z \in \{1, 2, 5\}$ s) using 10 random seeds per regime (30 total runs).

• Success Rate: Runtime-INCRT achieved a 100% success rate (30/30 runs), successfully eliminating the long-horizon failure mode observed in the fixed-K baseline.
• Performance: It achieved full success across all memory regimes with a single set of hyperparameters, outperforming the fixed-K approach which failed at $\tau_z = 5$ s.
• Capacity Dynamics: Contrary to the expectation that the controller would adapt $K(t)$ to task complexity, the mechanism saturated the maximum head cap ($K_{max}=8$) in all runs. The prune trigger never fired.
• Ablation Insights:
  • Varying $K_{max}$ at $\tau_z = 2$ s showed that smaller caps ($K_{max}=2$) yielded better mean performance and lower variance than larger caps ($K_{max}=16$), suggesting that larger capacity does not inherently improve performance.
  • The performance was non-monotone with respect to the cooldown parameter ($\Delta_{grace}$), with the default value being optimal.
  • Conclusion on Mechanism: The authors conclude that the benefit of Runtime-INCRT stems from the training trajectory rather than the terminal capacity. The gradual activation of heads acts as an implicit capacity curriculum within a single RL run, guiding the optimizer away from poor local attractors.

Significance and Claims
The paper claims that Runtime-INCRT offers a principled alternative to offline architecture search for adaptive control in non-Markovian environments. Its primary contribution is the demonstration that runtime adaptation of model capacity, driven by the effective rank of the on-policy data distribution, can resolve optimization failures associated with long memory horizons. The authors emphasize that the method removes the need for per-task hyperparameter search for the head count and provides analytical guarantees on policy continuity, distinguishing it from standard "restart-and-resize" approaches. The work suggests that the process of capacity growth serves as a critical inductive bias for successful RL training in complex control tasks, rather than the final model size being the sole determinant of success.