Technical Summary: Informed Asymmetric Actor-Critic

Problem Statement

Reinforcement learning (RL) in partially observable environments is typically formalized as Partially Observable Markov Decision Processes (POMDPs), where agents must make decisions based on incomplete and noisy observation histories. While standard RL methods adapt to this by learning history-dependent policies (often using Recurrent Neural Networks), they assume the same level of observability during both training and execution. This assumption is often unnecessarily restrictive, as many training environments (e.g., simulators) provide additional "privileged" information—such as full state access, expert policies, or multi-view sensory inputs—that is unavailable during deployment.

Existing asymmetric actor-critic methods attempt to leverage this privileged information by conditioning the critic on the full state while the actor relies only on observations. However, these approaches generally assume access to the full environment state during training. This requirement is often unrealistic in practice, where only partial or arbitrary state-dependent signals may be available. Furthermore, the theoretical justification for using arbitrary privileged signals (beyond the full state) in asymmetric settings has been underexplored, leaving a gap in understanding how to select and utilize these signals effectively without introducing bias into the policy gradient.

Methodology

The Informed Asymmetric Actor-Critic (IAAC) Framework

The authors introduce the Informed Asymmetric Actor-Critic (IAAC) framework, which generalizes asymmetric actor-critic methods to allow the critic to be conditioned on arbitrary state-dependent privileged signals ($i_t$) during training, rather than requiring full-state access ($s_t$).

1. Informed POMDP Formalism: Building on the "Informed POMDP" concept, the authors define a framework where an information variable $i_t$ is available during training. The observation $o_t$ is conditionally independent of the true state $s_t$ given $i_t$. This allows the critic to access $i_t$ while the actor remains constrained to the history of observations and actions ($h_t$).
2. Informed History-Based Value Functions: The authors define an informed history-based Q-function, $Q^\pi(h_t, i_t, a_t)$, and a corresponding value function, $V^\pi(h_t, i_t)$. These functions are conditioned on the privileged signal $i_t$ but are proven to be unbiased estimators of the standard history-based counterparts ($Q^\pi(h_t, a_t)$ and $V^\pi(h_t)$).
3. Unbiased Policy Gradient: A central theoretical contribution is Theorem 3.1, which proves that the policy gradient computed using the informed critic (conditioned on $i_t$) is equivalent to the standard policy gradient. Specifically, for any choice of state-conditioned signal $i_t$, the informed asymmetric policy gradient coincides with the standard gradient:
   $$\nabla^{IAAC}_\theta J(\pi_\theta) = \nabla_\theta J(\pi_\theta)$$
   This result holds without requiring the POMDP to satisfy additional assumptions like state-decodability. It implies that using privileged signals reduces the variance of value estimates (by disambiguating states) without biasing the optimization direction.

Informativeness Criteria for Signal Selection

Since any state-conditioned signal yields unbiased gradients, the challenge shifts to selecting the most informative signals. The paper proposes two novel criteria to assess the utility of a privileged signal $i_t$ regarding value estimation:

1. $\alpha$-Residual Informativeness (Pre-training): This criterion tests whether $i_t$ provides information about future returns ($G_t$) beyond what is already encoded in the observable history-action pair $(h_t, a_t)$. It involves:

   • Estimating conditional means $E[G_t|h_t, a_t]$ and $E[i_t|h_t, a_t]$ using regression models (e.g., Random Forests or RNNs).
   • Computing residuals $e_{G_t}$ and $e_{i_t}$.
   • Testing for statistical dependence between these residuals using a measure like the Hilbert-Schmidt Independence Criterion (HSIC).
   • If the null hypothesis of independence is rejected, the signal is deemed informative. This test can be performed on data collected by any exploratory policy (e.g., random) before training the actor.

2. $(\epsilon, \delta)$-Prediction Informativeness (Post-hoc): This criterion evaluates whether including $i_t$ improves the predictive accuracy of the value function. It compares the squared error of a symmetric critic (using only $h_t$) against an informed critic (using $h_t$ and $i_t$). If the inclusion of $i_t$ yields a statistically significant reduction in prediction error (positive gain), the signal is considered informative.

Key Results

Benchmark Performance

The authors evaluated IAAC on 12 partially observable benchmarks (6 navigation tasks from Baisero & Amato, 2022, and 6 from the POPGym suite).

• Performance: The informed asymmetric actor-critic consistently matched or outperformed symmetric baselines (A2C) and often surpassed asymmetric baselines that relied on full-state access (asym-A2C-hs).
• Efficiency: In several environments (e.g., Car-Flag, Shopping-5), the method converged faster and achieved higher asymptotic returns than full-state asymmetric variants, despite relying on strictly less state information.
• Stability: The method demonstrated greater training stability compared to symmetric approaches, which sometimes suffered performance drops or convergence issues in high-dimensional or partially observable settings.

Validation of Informativeness Criteria

Using synthetic informed POMDPs with known reward structures, the authors validated their selection criteria:

• Correlation with Performance: Signals containing features directly relevant to the reward (e.g., specific state dimensions determining the reward) were consistently identified as informative by both criteria.
• Redundancy Detection: Signals containing only redundant or irrelevant features were correctly identified as uninformative.
• Full State vs. Informative Subsets: The experiments showed that full-state access does not always yield the best performance if the state contains noisy or irrelevant features. Carefully selected subsets of privileged signals could outperform full-state baselines.
• Robustness: The criteria remained effective even under noisy observation conditions, correctly ranking signals based on their ability to explain return variance.

Significance and Claims

The paper claims to shift the paradigm of asymmetric reinforcement learning from a focus on access to full state information to the selection of informative training-time signals.

• Theoretical Generalization: The work provides the first theoretical justification for using arbitrary state-conditioned privileged signals in asymmetric actor-critic methods, proving that such usage preserves unbiased policy gradients. This generalizes previous work (e.g., Baisero & Amato, 2022) which was limited to full-state critics.
• Practical Utility: By demonstrating that carefully selected partial signals can match or exceed full-state performance, the paper challenges the assumption that full-state access is essential for effective asymmetric learning.
• Systematic Selection: The introduction of data-driven criteria for signal selection offers a principled approach to handling the "curse of dimensionality" in privileged information, allowing practitioners to identify useful auxiliary signals without manual feature engineering.

The authors conclude that their framework opens new directions for studying how privileged information can be selected, evaluated, and integrated to facilitate learning in partially observable settings, noting that future work could explore end-to-end signal selection and the integration of signal complexity into the learning loop.