🤖 machine learning

A Transdiagnostic Space of Disorder Like Phenotypes in Reinforcement Learning Agents

This paper introduces a transdiagnostic framework for modeling seven psychological disorders in reinforcement learning agents by manipulating cognitive appraisal signals, demonstrating that these induced phenotypes form a controllable, dose-dependent affective space that not only replicates disorder symptoms but also predicts treatment responses and comorbid interactions across different environments.

Original authors: Hari Prasad

Published 2026-07-10

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Hari Prasad

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a robot friend learning to navigate a maze. Usually, we teach robots to be perfect: find the goal, avoid the lava, get the high score. But what if we wanted to understand why humans sometimes get stuck in loops of worry, reckless behavior, or deep sadness? Instead of just watching people, this paper builds a "disorder simulator" inside a robot's brain.

The researchers didn't just tweak the robot's code to make it act weird. Instead, they built a special "emotional dashboard" called Appraisal. Think of this dashboard as a set of six dials that constantly measure how the robot feels about its situation: Is this important? Is it safe? Do I know what's happening? Can I handle it?

The big discovery? By turning just one single knob on this dashboard, they could make the robot develop a specific "disorder-like" behavior. They didn't program the robot to be anxious or addicted; they just adjusted a single number, and the robot's brain figured out the bad behavior on its own.

The Seven "Knobs" of Trouble

The team created seven different "knobs," each grounded in real psychological theories. Here is what happens when you turn them:

The Anxiety Knob: This makes the robot hyper-aware of threats. It stops taking risks and starts avoiding everything, even if it means missing out on the big prize. It's like a person who refuses to leave the house because the street might have a dog.
The Mania Knob: This is the exact opposite of anxiety. The robot becomes reckless, seeking out danger and lava instead of avoiding it. It's the mirror image of anxiety, driven by a feeling that everything is safe and exciting, even when it's not.
The OCD Knob: This makes the robot check things over and over. Imagine a robot that keeps walking back to a door to make sure it's locked, even though it just checked it. The more it checks, the less reassured it feels, so it checks again.
The Depression Knob: This adds a "cost" to every step the robot takes. Moving becomes exhausting. The robot stops trying to reach the goal because the effort feels too heavy, even though it could still do it.
The Impulsivity Knob: This makes the robot care only about now. It grabs a small treat right in front of it instead of waiting for a huge reward just a few steps away.
The Addiction Knob: This introduces a "drug tile" that gives a massive, never-ending reward. The robot ignores the actual goal to chase this tile, eventually forgetting the mission entirely.
The PTSD Knob: This places a "shock" on the shortest path to the goal. The robot learns to take a long, winding detour to avoid that one spot, even if the detour is terrible.

The Magic of the "Dose"

The coolest part is that these aren't just "on" or "off" switches. The researchers could turn the knobs up and down like a volume dial. They ran the simulation 1,375 times with different settings. They found that as they turned the knob up, the bad behavior got worse in a smooth, predictable line. If they turned it down, the robot got better. This proves the behavior wasn't a glitch; it was a direct response to the "dose" of the emotional signal.

The Surprise: The Robot's Brain Organizes Itself

Here is where the paper gets really interesting. The researchers didn't tell the robot how to organize these behaviors. They just turned the knobs. But when they plotted the results, the disorders arranged themselves into a neat map:

Anxiety and Mania were perfect opposites, sitting on opposite sides of the map.
Addiction and Impulsivity both pushed the robot toward "over-pursuing" things.
Depression pulled the robot into "withdrawal."

It's as if the robot's brain naturally sorted these messy behaviors into a logical structure, just by learning from the signals.

The "Cure" Experiment

The team also tried to "cure" the robots by turning the knobs back off. The results were surprising and matched real human therapy logic:

The Easy Fixes: For the "reward distortion" disorders (like Mania, OCD, and Addiction), simply turning off the knob made the robot snap back to normal immediately. The bad habit wasn't deeply rooted.
The Hard Fixes: For the "avoidance" disorders (like Anxiety and PTSD), just turning off the knob didn't work. The robot kept avoiding the scary path because it had learned a habit of staying safe. It needed a "graded exposure" therapy: the researchers had to gently force the robot to face the scary path again, step by step, until it realized it was safe. This mimics how real therapy works for humans with anxiety.

What This Isn't

It's important to know what this paper doesn't say.

These robots are not humans. They don't have feelings, memories, or trauma in the human sense. They are mathematical models.
The researchers didn't program the robot to "be anxious." They programmed a signal, and the anxiety emerged as a side effect.
They didn't prove these are the only causes of these disorders in real people. They just showed that these specific mechanisms are enough to create the behaviors in a simulation.

The Final Test: Does it work in 3D?

To make sure this wasn't just a trick of the simple 2D maze, they tested three of the disorders (Depression, Addiction, Anxiety) on a robot with a camera, navigating a 3D pixel world (like a video game). The same knobs worked exactly the same way. The robot got depressed, addicted, or anxious, even without the special "dashboard" the researchers used in the first test. This suggests the problem isn't the robot's eyes or brain; it's the way the "reward signals" are structured.

The Bottom Line

This paper suggests that we can model complex human struggles like anxiety or addiction by tweaking simple mathematical signals in a learning agent. It shows that these disorders might be different points on a shared map of how we process rewards and threats. Most importantly, it suggests that fixing these problems might require different strategies: sometimes you just need to remove the bad signal, but other times, you have to actively retrain the habit. It turns the study of mental health from just watching what goes wrong into a playground where we can test exactly how to fix it.

Technical Summary: A Transdiagnostic Space of Disorder-Like Phenotypes in Reinforcement Learning Agents

Problem Statement

Reinforcement Learning (RL) agents deployed in healthcare and human-facing systems require affective stability; value estimation errors can lead to catastrophic failure modes such as avoidance, perseveration, or recklessness. While prior work has attempted to model psychological disorders in agents, it suffers from three primary weaknesses:

Lack of Mechanistic Grounding: Manipulations are often hand-tuned heuristics rather than derived from computational psychiatry accounts, resulting in loose mappings to clinical constructs.
Post-hoc Labeling: Behaviors are labeled after observation (e.g., "this looks like OCD") rather than measured against pre-registered criteria, inviting confirmation bias.
Lack of Rigor: Results are typically reported as single runs without variance, controls, or dose-response analysis.

Furthermore, existing computational psychiatry models often fit abstract tasks to behavior but cannot continuously manipulate disorder severity or observe the full trajectory from health to pathology and back.

Methodology

The AG-PPO Architecture

The authors introduce AG-PPO (Appraisal-Guided Proximal Policy Optimization), an agent that integrates Appraisal Theory (Lazarus, 1991; Scherer, 2001) into the RL loop.

Architecture: A shared convolutional encoder processes egocentric symbolic observations (7×7×3). The output feeds both an Actor and a Critic.
Appraisal Vector ( $\zeta$ ): The Critic additionally consumes a six-dimensional appraisal vector $\zeta_t$ $ζ_{t}$ , computed at every step:
1. Motivational Relevance ( $\zeta_{MR}$ ): Function of agent-goal distance.
2. Goal Congruence ( $\zeta_{GC}$ ): Geometric alignment with goals.
3. Certainty ( $\zeta_{C}$ ): Derived from policy entropy.
4. Novelty ( $\zeta_{N}$ ): Derived from KL divergence against a uniform policy.
5. Coping Potential ( $\zeta_{CP}$ ): Fraction of threats outside the agent's view.
6. Anticipation ( $\zeta_{A}$ ): Complement of the next-reward prediction error (supplied by a Next-Reward Estimation network).
Reward Shaping: The environment reward $r_t$ is modified by a shaping function $r'_t = r_t - \sum w_i g_i(\zeta_t) + \text{specific terms}$ . Crucially, each disorder is induced by activating exactly one term in this equation or modifying the discount factor $\gamma$ .

Disorder Mechanisms (The "Knobs")

Seven disorders are modeled as dose-controllable "knobs," each grounded in a specific computational psychiatry account:

Anxiety: Penalizes low coping potential (hypervigilance to threats).
Mania: Penalizes high coping potential (reward hypersensitivity/threat seeking).
OCD (Checking): Grants a diminishing bonus for revisiting checkpoints (modeling diminishing reassurance).
Depression: Imposes a per-step effort cost on movement (reducing behavioral activation).
Impulsivity: Reduces the discount factor $\gamma$ (steep delay discounting).
Addiction: Adds a non-habituating bonus for drug tiles (uncapped incentive salience).
PTSD: Applies a conditioned shock on a trauma tile, causing avoidance over-generalization.

Experimental Protocol

Environments: Four threat/spatial grid worlds (Dynamic-Obstacles, LavaGap, LavaCrossing, Approach-Avoidance, Temporal-Choice, Addiction, Trauma) and a 3D pixel environment (MiniWorld).
Assays: Primary symptom assays were pre-registered for each disorder (e.g., risky-goal choice for anxiety, checking rate for OCD).
Controls: Four control conditions were used: Standard PPO, Critic-noise (random vector), PPO+RND (intrinsic motivation), and Appraisal Critic without shaping.
Statistical Rigor: Experiments ran 1,375 configurations across 10 seeds (95% confidence intervals).

Key Results

1. Graded Dose-Response

Across all seven disorders, increasing the "knob" dose produced a graded, monotone response in the primary symptom assay. Controls failed to reproduce these phenotypes, confirming that the effects arise from the specific shaping mechanisms, not generic intrinsic motivation or noise.

2. Emergent Transdiagnostic Structure

The disorders self-organize into a two-dimensional affective space (Reward-Approach vs. Threat-Avoidance) without this structure being explicitly encoded in the reward:

Mania and Anxiety emerged as exact mirrors across the origin (disinhibited over-pursuit vs. hypervigilant avoidance).
OCD occupied a distinct dimension (compulsivity) orthogonal to the main plane.
Depression mapped to withdrawal; Addiction and Impulsivity to over-pursuit.

3. Treatment Dissociation (Rescue vs. Resistance)

The authors tested "treatment" by removing the pathological knob after training:

Reward Distortion Disorders (Mania, OCD, Addiction): Symptoms remitted immediately upon knob removal, as the goal gradient re-dominated.
Habit/Avoidance Disorders (Anxiety, PTSD, Depression, Impulsivity): Symptoms resisted passive removal. The agents maintained their maladaptive policies because the avoidance behavior prevented them from encountering disconfirming evidence.
Resolution: The resistant disorders (Anxiety, PTSD) were only recovered via a graded exposure curriculum (penalizing the avoidance route), mirroring clinical exposure therapy.

4. Emergent Comorbidity

When two knobs were activated simultaneously, the interaction was non-additive:

Mania × Impulsivity: Impulsivity (steep discounting) suppressed the lethality of mania. The manic agent's reckless approach to lava required multi-step commitment, which the myopic impulsive agent would not undertake.
Anxiety × Depression: Severe depression (high effort cost) overrode the anxiety readout; the agent ceased acting entirely, preventing the expression of avoidance.

5. Generalization to 3D Pixel Environments

Three disorders (Depression, Addiction, Anxiety) were transferred to a 3D pixel environment (MiniWorld) using a standard CNN agent without an appraisal critic.

The same phenotypic signatures emerged (e.g., depression reduced forward action; addiction increased drug occupancy).
Cross-assay dissociation held: The depression knob did not increase drug occupancy, and the anxiety knob did not suppress locomotion. This confirms the mechanisms are properties of the reward structure, not specific to grid worlds or the appraisal critic architecture.

Significance and Claims

The paper claims to establish a controllable, grounded substrate for studying affective phenotypes in artificial agents. Its primary contributions are:

From Description to Manipulation: It shifts disorder modeling from post-hoc labeling to a falsifiable science where severity is continuously tunable via single, interpretable parameters.
Emergent Clinical Parallels: The framework reproduces complex clinical phenomena (e.g., the resistance of avoidance disorders to passive stressor removal, the non-additive nature of comorbidity) without explicitly programming them. These are emergent properties of the interaction between the agent's learning dynamics and the shaped reward.
Unified Framework: It demonstrates that a single agent architecture can model induction, structural organization (transdiagnostic space), and treatment (rescue/extinction) for multiple disorders simultaneously.
Mechanistic Specificity: The dissociation results confirm that specific "knobs" target specific symptom dimensions without cross-contamination, validating the mechanistic grounding of the approach.

The authors conclude that appraisal weights parameterize a space where the same mechanisms that induce a disorder can also model its treatment, offering a new testbed for computational psychiatry and a lens on the failure modes of affective control in RL.