Learning Smooth Time-Varying Linear Policies with an Action Jacobian Penalty

1. Problem Statement

Deep Reinforcement Learning (DRL) has proven effective for physics-based character animation and robotics. However, policies optimized naively for task rewards often exploit unnatural high-frequency control signals. These signals result in jittery motions that are unachievable by physical robots due to actuator bandwidth limits and sensor noise, leading to "sim-to-real" failures.

Existing solutions to this problem include:

• Action Change Penalties: Adding reward terms to penalize the difference between consecutive actions ($\|a_t - a_{t-1}\|^2$). This requires significant manual tuning and often fails to balance task performance with smoothness.
• Lipschitz-Constrained Policies: Penalizing the gradient of the action likelihood. While effective for locomotion, this method requires many action samples for accurate estimation and imposes high computational overhead due to additional backpropagation steps.

The core challenge is to learn control policies that are smooth (low high-frequency content) and robust without requiring task-specific hyperparameter tuning or incurring prohibitive computational costs during training.

2. Methodology

The authors propose a two-pronged approach: a new regularization loss function and a novel neural network architecture.

A. Action Jacobian Penalty

Instead of using a reward term, the authors introduce a loss term directly into the Proximal Policy Optimization (PPO) objective to penalize the sensitivity of the action to state changes.

• Definition: The penalty is the squared Frobenius norm of the Action Jacobian ($J$), which represents the derivative of the control action ($a$) with respect to the character state ($s$):
  $$L_{Jac} = \|J\|_F^2 = \left\| \frac{\partial a}{\partial s} \right\|_F^2$$
• Mechanism: By minimizing this norm, the policy is forced to generate actions that change slowly in response to small variations in the state, effectively eliminating high-frequency jitter.
• Advantage: Unlike previous methods that approximate the Jacobian via sampling or penalize specific directions, this method optimizes the full Jacobian norm directly using automatic differentiation.

B. Linear Policy Net (LPN)

Computing the Jacobian for a standard Fully Connected (FF) neural network is computationally expensive (requiring multiple forward/backward passes). To mitigate this, the authors introduce the Linear Policy Net (LPN).

• Architecture: The LPN does not output the action directly. Instead, it takes the reference state ($\hat{s}_t$) as input and outputs:
  1. A Feedback Matrix ($K_t \in \mathbb{R}^{m \times n}$).
  2. A Feedforward Action ($k_t \in \mathbb{R}^m$).
• Control Law: The final control action is computed as a linear combination:
  $$a_t = K_t s_t + k_t + \hat{a}_t$$
  Where $s_t$ is the current character state and $\hat{a}_t$ is the reference joint angle.
• Computational Efficiency: Because the policy is linear with respect to the state ($s_t$), the Jacobian of the action with respect to the state is simply the matrix $K_t$ itself.
  • Computing the penalty becomes a simple forward pass.
  • Computing the gradient for backpropagation becomes a simple backward pass.
  • This reduces the computational overhead of the Jacobian penalty to nearly zero compared to standard FF networks.

3. Key Contributions

1. Action Jacobian Penalty: A regularization technique that directly penalizes the sensitivity of actions to state changes via the Jacobian norm, effectively removing high-frequency control signals without task-specific tuning.
2. Linear Policy Net (LPN): A novel architecture that parameterizes policies as time-varying linear feedback controllers. This architecture makes the computation of the Jacobian penalty trivial, significantly reducing training time and allowing for faster convergence.
3. Empirical Validation: Demonstration that LPNs, despite their restricted linear form, can learn complex, dynamic motions (e.g., backflips, parkour, table tennis footwork) that are comparable to or better than standard FF networks.
4. Sim-to-Real Transfer: Successful application of the learned policies to a physical quadrupedal robot (Boston Dynamics Spot with an arm), demonstrating robustness to real-world noise and actuator limitations.

4. Results

The authors evaluated their method on a simulated humanoid and a physical quadruped across various tasks (walking, running, backflips, cartwheels, parkour, and table tennis footwork).

• Smoothness Metrics:
  • Action Smoothness & High-Frequency Ratio: LPN with the Jacobian penalty achieved the lowest high-frequency ratios (closest to 0, indicating smooth signals) across all tasks, significantly outperforming standard FF networks and Lipschitz-constrained policies.
  • Motion Jerk: LPN produced smoother motions (lower jerk) comparable to heavily tuned reward-based penalties but without the tuning effort.
• Training Efficiency:
  • Convergence: LPN converged faster than FF networks with Jacobian penalties (which were ~1.5x slower per iteration) and often faster than baseline methods.
  • Computation: The overhead of the Jacobian penalty was negligible for LPN, whereas it significantly slowed down FF networks.
• Task Performance:
  • LPNs successfully learned dynamic skills like backflips and wall climbing.
  • Reduced-Order Analysis: Singular Value Decomposition (SVD) revealed that the learned feedback matrices could be compressed to low ranks (e.g., rank 14 for walking, rank 20 for backflips) while maintaining performance, suggesting the underlying control structure is inherently low-dimensional.
  • Sim-to-Real: The LPN controlled a physical robot to perform stable pacing gaits and complex hopping/arm-swing combinations, whereas standard FF policies often failed due to jitter.

5. Significance and Future Work

• Bridging the Gap: This work bridges the gap between the flexibility of DRL and the stability/interpretability of classical linear control. It proves that time-varying linear policies are sufficient for highly dynamic, non-cyclic motions, challenging the assumption that complex non-linear networks are always necessary.
• Efficiency: By removing the computational bottleneck of Jacobian penalties, this method makes smooth policy learning scalable for complex tasks.
• Interpretability: Unlike "black box" neural networks, LPNs produce explicit feedback matrices ($K_t$), offering a path toward explainable AI in robotics.
• Future Directions: The authors suggest exploring piece-wise linear policies (segmenting state space), combining LPNs with Model-Based Control (like DDP) for warm-starting, and scaling the system to generate policies for arbitrary motion sequences using generative models.

In summary, the paper presents a highly efficient framework for learning smooth, robust control policies by combining a mathematically grounded regularization term (Action Jacobian) with a specialized architecture (LPN) that leverages the linearity of the control law to minimize computational cost.