Technical Summary: Intrinsic Vicarious Conditioning for Deep Reinforcement Learning

Problem Statement

Reinforcement Learning (RL) has advanced significantly through reward shaping and learning from demonstration (LfD) methods. However, these approaches share a critical limitation: they typically require access to an expert demonstrator's internal representations, action-level trajectories, or knowledge of their reward function. This dependency becomes problematic in environments characterized by sparse extrinsic rewards and catastrophic terminal conditions (e.g., agent death) that provide no feedback signal. In such "single-life" scenarios, biological agents learn to avoid danger and pursue desirable states by observing others without direct experience, a process known as vicarious conditioning. Current RL methods struggle to emulate this capability without requiring the demonstrator's policy or reward function, forcing agents to rely on costly self-exploration or extensive datasets.

Methodology

The authors propose Vicarious Conditioning (VC) as an intrinsic reward mechanism that allows an agent to learn from a few observed demonstrations without access to the demonstrator's policy ($\pi_d$) or reward function ($r_d$). The method is grounded in Bandura's four-step model of observational learning: Attention, Retention, Reproduction, and Reinforcement.

1. Problem Formulation

The environment is modeled as a Partially Observable Markov Decision Process (POMDP). The agent receives a composite reward $\bar{r}(o_t) = r_{ext}(o_t) + \alpha \cdot r_{vic}(o_t)$, where $r_{ext}$ is the sparse environmental reward and $r_{vic}$ is the intrinsic vicarious reward.
The agent observes a demonstrator trajectory $\tau = (o^d_1, \dots, o^d_T, v_\tau)$, where $v_\tau \in \mathbb{R}$ is a communicated value indicating the outcome's valence (positive for approach, negative for avoidance) and intensity. Crucially, the agent does not see the demonstrator's actions or internal states.

2. The Four Steps of Implementation

The authors implement these steps using a Siamese Memory-Augmented Neural Network (MANN) architecture.

• Attention: A shared Controller Encoder (CE), implemented as a Siamese Long Short-Term Memory (LSTM) network, maps observation sequences (from both the demonstrator and the agent) into fixed-dimensional embeddings ($e_\tau$ and $e_t$). This ensures that behaviorally similar sequences produce comparable embeddings, regardless of whether they originate from the demonstrator or the agent.
• Retention: The agent stores the demonstrator's embeddings alongside a communicated value. The vicarious value $V^{vic}$ is calculated via backward discounting from the outcome utility, modulated by a trust coefficient $k$. These embeddings and values are written to an external memory matrix $M_t$ using content-based addressing (Least Recently Used access). The MANN is trained via cross-entropy loss to classify the valence (positive/negative) of the stored trajectories.
• Reproduction: The agent generates an intrinsic reward by comparing its current behavior embedding ($e_t$) against the stored memory. A Generalization Kernel computes the similarity (cosine similarity) between the current state and stored memories. A classification layer outputs confidence scores ($p_t$) for each behavior class. The intrinsic reward $r_{vic}$ is aggregated from these scores, gated by an inhibition threshold ($\theta_{thr}$) to prevent low-confidence matches from triggering rewards.
  • Negative Conditioning: If the agent's state resembles a stored "avoidance" trajectory, it receives a negative intrinsic reward.
  • Positive Conditioning: If the agent's state resembles a stored "approach" trajectory, it receives a positive intrinsic reward.
• Reinforcement: The intrinsic reward is combined with the extrinsic reward. The agent optimizes the policy to maximize the expected return of this composite signal. The extrinsic reward eventually dominates if contradictions arise, ensuring the policy remains optimal for the specific environment.

Key Contributions

1. General Formulation: The paper provides a formal RL framework for vicarious conditioning, explicitly mapping the biological steps of attention, retention, reproduction, and reinforcement to neural network components.
2. Siamese Memory-Augmented Architecture: The authors design a Siamese MANN that enables low-shot learning. By sharing the encoder between the demonstrator and the agent, the system ensures that similar behaviors yield similar embeddings, allowing the agent to recognize and react to its own experiences based on a few stored demonstrations.
3. Valence-Specific Mechanisms: The method introduces distinct mechanisms for negative (avoidance) and positive (approach) conditioning, allowing the agent to learn both what to avoid and what to pursue based solely on communicated outcome values.

Experimental Results

The approach was evaluated in two environments: MiniWorld Sidewalk (testing negative conditioning/avoidance) and Box2D CarRacing (testing positive conditioning/approach and composite conditioning).

• MiniWorld Sidewalk: In an environment with a non-descriptive terminal condition (death yields no reward), standard PPO achieved ~117 steps. The VC method with an optimal threshold ($\theta_{thr} = 0.60$) significantly outperformed the baseline, achieving 142.16 steps (composite reward) and 139.96 steps (intrinsic-only). The results demonstrated that VC successfully discouraged the agent from dangerous states (sidewalk boundaries) without requiring explicit reward signals for death.
• CarRacing:
  • Positive VC: The agent learned to stay in the center lane (demonstrated behavior), achieving nearly three times the episode length of base PPO (88.45 steps vs. 33.27 steps). While this resulted in lower extrinsic reward (due to conservative driving vs. aggressive tile-visiting), the composite reward was significantly higher.
  • Negative VC: When applied to an approach task (avoiding the track boundary), negative conditioning performed worse than the baseline, confirming the method's sensitivity to valence; avoidance signals are detrimental when the task requires approach.
  • Composite VC: Combining positive and negative signals showed mixed results, with high variance at strict thresholds, suggesting the need for careful calibration of valence signals.

Significance and Claims

The authors claim that this work bridges the gap between biological learning capabilities and current RL limitations. By enabling agents to learn from few observed demonstrations without access to policies, large datasets, or reward functions, vicarious conditioning offers a cognitively plausible alternative for single-life learning and continual learning.

The paper emphasizes that this paradigm is particularly suited for real-world environments that are vast, partially observable, and dangerous, where catastrophic failure cannot be learned through direct trial-and-error. The method successfully emulates the biological process where an agent learns from the communicated outcomes of others, allowing it to avoid catastrophic terminal conditions and pursue desirable states with minimal data. The authors acknowledge limitations, such as the inability to distinguish indistinguishable behaviors in the feature space and the lack of memory extinction (once stored, a vicarious value persists), but argue that the reinforcement step eventually allows extrinsic rewards to override intrinsic conditioning if necessary.