Learning Suffers More Than the Policy Class Under Partial Observability: A Closed-Form Analysis

Technical Summary: Learning Suffers More Than the Policy Class Under Partial Observability

Problem Statement
In partially observable reinforcement learning (RL), agents often fail to find optimal policies. Conventional wisdom attributes this to the policy-class gap: the inability of the agent's representational capacity (e.g., memoryless policies) to capture a good policy because it cannot observe the full state. This paper challenges that view by demonstrating that a larger, distinct failure mode exists: the learning gap. This is the discrepancy between the best policy the agent can represent and the policy the learning algorithm actually converges to. The author argues that even when a good policy exists within the agent's representational class and the value function approximator is theoretically capable of expressing the true value function exactly, the learning dynamics themselves introduce a bias that drives the agent to a suboptimal equilibrium.

Methodology
The author analyzes a solvable, partially observed linear-quadratic Gaussian (LQG) control problem.

• Environment: A two-dimensional system where the agent observes and controls coordinate $v$, while an unobserved coordinate $z$ drives $v$. The unobserved state acts as a memory kernel.
• Learning Algorithm: A standard actor-critic pair.
  • Critic: Uses Temporal Difference (TD(0)) or Least Squares TD (LSTD) with quadratic features $(1, v, v^2)$. Crucially, the true value functions of this problem are quadratic, meaning the feature class is expressive enough to represent the true value function exactly if the full state were observed. The limitation is solely that features are computed from the incomplete observation $v$.
  • Actor: Follows the policy gradient using the critic's value estimates as a baseline.
• Analysis Approach: The author derives closed-form expressions for the critic's fixed point and the actor's expected update direction. They analyze the equilibrium of the expected update (the "averaged dynamics") rather than relying solely on stochastic convergence proofs, verifying these predictions numerically and via deep RL experiments.

Key Contributions and Results

1. Exact Mechanism of Failure (The Aliased Critic):
   The core finding is that the learning gap dominates the policy-class gap. In the default setting, the best memoryless policy costs only 10.4% more than the ideal full-state controller. However, the actor-critic learner converges to a policy that is 35% worse than this best memoryless policy.

   • Mechanism: In the stationary regime, the unobserved state $z$ balances the control input on average, making the observed coordinate $v$ appear to fluctuate like a random walk. The critic, restricted to functions of $v$ alone, cannot attribute this persistence to the hidden state. Consequently, it misattributes the unexplained persistence to the curvature of the value function.
   • Result: The critic reports a value curvature (coefficient of $v^2$) that is roughly 15 times larger than the true conditional curvature. The actor, using this inflated curvature as a baseline, is incentivized to increase the control gain indefinitely until the inflated curvature is balanced by control costs, settling at a gain far higher than optimal.

2. Closed-Form Law for the Learning Equilibrium:
   The author derives an exact formula for the gain $k_{eq}$ at which the expected update vanishes.

   • The equilibrium gain is inversely proportional to the sampling period $h$ and depends on the ratio of exploration noise to the zero-lag intensity of the memory kernel ($\sigma = s_e / \Theta K(0)$).
   • As exploration vanishes, the equilibrium gain approaches the stability boundary of the sampled control system.
   • This location depends on the distribution of the disturbance (specifically the second-order structure of the squared disturbance), not just its autocovariance.

3. Closed-Form Law for the Deployed Cost:
   Despite the incorrect location of the equilibrium, the cost incurred at this fixed point is remarkably robust.

   • The deployed cost at the learning equilibrium converges to $r C(0)$ (where $r$ is the control cost weight and $C(0)$ is the zero-lag autocovariance of the disturbance).
   • This cost law is distribution-free: it holds for any stationary, ergodic, mean-square-continuous disturbance, regardless of whether the disturbance is Gaussian, heavy-tailed, or discrete (e.g., telegraph noise).
   • The author demonstrates that while different disturbance distributions shift the equilibrium location (gain) by a factor of 3.4, the resulting cost remains nearly identical.

4. The Bootstrap Horizon as the Solution:
   The paper identifies a single design choice that eliminates the pathology: the bootstrap horizon (controlled by the $\lambda$ parameter in $\lambda$-returns).

   • Standard TD(0) ($\lambda=0$) looks only one step ahead, leading to the aliased equilibrium.
   • As $\lambda$ increases, the horizon extends. The author shows that the equilibrium migrates continuously back toward the policy-class optimum as the horizon matches the memory of the environment.
   • The transition is sharp: increasing $\lambda$ from 0.98 to 0.99 reduces the excess cost from ~30% to ~1.5%.

5. Deep Reinforcement Learning Validation:
   Experiments using Proximal Policy Optimization (PPO) with Multi-Layer Perceptrons (MLP) confirm the theoretical predictions.

   • Architecture Independence: Adding memory via frame stacking (observing past 32 frames) does not help. Stacked agents exhibit the same aliased behavior as memoryless agents at low $\lambda$ and the same recovery at high $\lambda$. This suggests that having history in the input is insufficient; the learning algorithm must effectively utilize that history through the bootstrap horizon.
   • Pre-registered Tests: The author pre-registered experiments to rule out alternative explanations, such as the collapse of learned exploration noise or a threshold effect in exploration magnitude. These hypotheses were falsified; the failure persists even with fixed exploration noise, and no exploration threshold prevents the displacement.

Significance and Claims
The paper claims to provide the first exact, closed-form characterization of how partial observability biases actor-critic learning, separating the limitations of representation from the limitations of the learning rule.

• Modest Scope: The author explicitly states they do not prove the convergence of stochastic iterates to the fixed point, only that the expected update vanishes there and that stochastic runs follow the averaged trajectory (with a lag).
• Specific Limitations: The exact location law is proven for Gaussian disturbances; the cost law is distribution-free. The paper conjectures that the location law depends on the second-order structure of the squared disturbance for non-Gaussian cases but does not claim to have proven this generally.
• Practical Implication: The primary takeaway is that in partially observable settings, simply increasing model capacity or adding input memory (like frame stacking) is insufficient to fix learning failures. Instead, practitioners must tune the bootstrap horizon ($\lambda$) to exceed the memory of the environment's unobserved dynamics.

The paper concludes that the "learning gap" is a distinct and often larger source of suboptimality than the "policy-class gap," driven by the critic's inability to distinguish unobserved state persistence from value function curvature.