🧬 biology

Fitting Reinforcement Learning Model to Behavioral Data under Bandits

This paper proposes a novel convex relaxation-based optimization method for efficiently fitting reinforcement learning models to behavioral data in multi-armed bandit environments, offering state-of-the-art performance with significantly reduced computation time and an accompanying open-source Python package.

Original authors: Hao Zhu, Jasper Hoffmann, Baohe Zhang, Joschka Boedecker

Published 2026-03-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hao Zhu, Jasper Hoffmann, Baohe Zhang, Joschka Boedecker

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are a detective trying to figure out how a suspect (a human or an animal) makes decisions. You have a video of them playing a game where they have to choose between several doors, and behind some doors are rewards (like candy) and behind others are nothing.

Your goal is to build a mathematical "mind" that explains why the suspect chose the doors they did. This is what scientists call fitting a Reinforcement Learning (RL) model.

However, there's a catch: The math behind these "minds" is incredibly messy and difficult to solve. It's like trying to find the lowest point in a foggy mountain range where the ground is full of hidden pits and sudden cliffs. If you try to walk down blindly, you might get stuck in a small valley (a local minimum) and think you've reached the bottom, when actually there's a much deeper valley nearby.

This paper introduces a clever new way to solve this puzzle. Here is the breakdown in simple terms:

1. The Problem: The "Foggy Mountain"

The authors explain that trying to reverse-engineer a decision-making model from behavior is a non-convex problem.

The Metaphor: Imagine you are trying to find the absolute lowest point in a landscape full of hills, valleys, and caves. Because the landscape is so bumpy, standard computer methods often get stuck in a small dip and stop looking, thinking they found the best answer. They are slow, and they often give you the wrong answer because they can't see the whole picture.

2. The Solution: Flattening the Terrain

The authors realized that while the original "mountain" is bumpy and dangerous, you can create a smooth, flat version of it that is much easier to navigate.

The Analogy: Imagine you have a crumpled piece of paper (the real, messy problem). It's hard to read the text on it. The authors take that paper and iron it out flat (this is called Convex Relaxation).
Why it works: On the flat paper, finding the lowest point is easy and guaranteed to be the true lowest point. You can do this incredibly fast.
The Trade-off: The flat paper isn't exactly the same as the crumpled one. It's an approximation. But, the authors prove that for most real-world situations, this flat version is so close to the real thing that the difference doesn't matter.

3. The Two-Step Detective Process

The authors propose a new method that works in two steps:

Step 1 (The Fast Scan): Use the "flat paper" method to quickly figure out the general pattern of the suspect's behavior (what they value). This takes a fraction of a second.
Step 2 (The Fine-Tune): If you need the specific numbers (the exact "learning rate" or "sensitivity" of the suspect), you take the result from Step 1 and do a quick, targeted check to refine the numbers.

4. The Results: Speed vs. Accuracy

The team tested their method against the old, slow ways of doing things (like guessing and checking many times or using complex probability simulations).

The Old Way: Like trying to find a needle in a haystack by pulling out one piece of hay at a time. It takes hours or even days.
The New Way: Like using a magnet to pull the needle out instantly.
The Outcome: Their new method was just as accurate as the best existing methods but was thousands of times faster.

5. Why This Matters for Everyone

For Scientists: They can now analyze huge amounts of data (like thousands of mouse experiments or human trials) in minutes instead of days.
For the Public: This helps us understand how brains learn. Whether it's a mouse learning to find food or a human learning to navigate a new city, this tool helps us decode the "algorithm" inside our heads much faster.
Accessibility: The authors even released a free software package (like a pre-made tool kit) so that anyone, even without a PhD in math, can use this "magnet" to analyze their own data.

Summary

Think of this paper as inventing a GPS for decision-making. Before, finding the best route through the messy math of human behavior was like driving blindfolded through a maze. This new method gives you a clear, flat map that gets you to the destination instantly, without getting lost in the bumps and holes.

1. Problem Statement

The paper addresses the challenge of fitting Reinforcement Learning (RL) models (specifically Q-learning variants) to observed behavioral data from multi-armed bandit tasks.

Context: In behavioral science (neuroscience, psychology), researchers use RL models to characterize how humans and animals make decisions. The goal is to infer latent model parameters (learning rates $\alpha$ , reward sensitivity $\beta$ ) and value functions ( $x(t)$ ) that best explain observed sequences of actions and rewards.
The Core Difficulty: The standard formulation of this fitting problem involves maximizing the likelihood of observed data subject to recursive update equations (e.g., $x(t) = (1-\alpha)x(t-1) + \alpha\beta u(t)$ ). The authors demonstrate that this results in a non-convex optimization problem.
Consequences: Non-convexity implies that standard local optimization methods (like gradient descent or Nelder-Mead) often get stuck in local minima, while global optimization is computationally intractable (exponential time complexity) for large datasets. Existing solutions are either slow (Bayesian MCMC) or unreliable (heuristic local minimization).

2. Methodology

The authors propose a novel solution based on convex relaxation and convex optimization.

A. Mathematical Formulation

The paper formalizes a general class of RL models (including basic forgetting Q-learning, models with distinct learning rates per arm, and models with sub-reward signals) into a unified optimization problem (Eq. 2.6).

They prove that the problem is non-convex because the relationship between the model parameters ( $\alpha, \beta$ ) and the value functions involves a non-affine transformation $F$ (geometric decay).

B. The Convex Surrogate (Core Innovation)

To overcome non-convexity, the authors introduce a convex relaxation:

Reformulation: They rewrite the recursive update constraints using a matrix $G$ where columns represent the decay of value signals over time.
Relaxation: Instead of enforcing the strict geometric decay structure (which requires non-convex constraints), they relax the constraints to require only that the entries of $G$ are non-negative and monotonically non-increasing along each row.
Result: This transforms the original non-convex problem into a convex optimization problem (Eq. 4.3).
- Objective: Minimize negative log-likelihood (convex).
- Constraints: Affine equality constraints and linear inequality constraints (convex).
- Benefit: This allows the problem to be solved to global optimality in polynomial time using standard solvers (e.g., interior-point methods via CVXPY).

C. Two-Step Recovery Process

Since the convex relaxation estimates the value functions ( $x(t)$ ) directly but does not strictly recover the original parameters ( $\alpha, \beta$ ), the authors propose a two-step workflow:

Step 1 (Convex): Solve the relaxed convex problem (4.3) to obtain the optimal value functions $x^*(t)$ and the matrix $G^*$ . This provides a lower bound on the optimal likelihood.
Step 2 (Parameter Recovery): If specific parameters are needed, solve a secondary, smaller non-convex problem (4.4) to find $\alpha$ and $\beta$ that best approximate the relaxed solution $G^*$ . Crucially, this step can be parallelized across different arms or sub-rewards, significantly speeding up computation.

D. Horizon Truncation

To further improve efficiency for long time-series, the authors introduce a horizon truncation approximation (Eq. 4.5). This assumes that value functions depend only on the most recent $p$ rewards rather than the entire history. This reduces the number of variables from $O(mn)$ to $O(mp)$, drastically lowering computational complexity.

3. Key Contributions

Theoretical Analysis: The first rigorous convexity analysis of the general RL model fitting problem under bandits, proving its non-convex nature and identifying the specific source of non-convexity.
Novel Algorithm: A convex relaxation method that converts the fitting problem into a tractable convex program, guaranteeing global optimality for the value functions.
Efficiency: The method significantly reduces computation time compared to existing benchmarks while maintaining comparable accuracy.
Open-Source Tool: The authors released rlfit, a Python package that implements these methods, allowing researchers without expertise in convex optimization to apply these advanced fitting techniques.

4. Empirical Results

The method was evaluated on simulated data (2-armed and 10-armed bandits) and real-world mouse behavioral data (reversal learning tasks). It was compared against:

MC: Bayesian Monte Carlo sampling (Gold standard for accuracy, but slow).
D-LOC: Direct local minimization with repeated random restarts (Common heuristic).
CVX-based methods: The proposed convex relaxation (CVX, CVX-T, CVX-LOC, CVX-LOC-T).

Key Findings:

Accuracy: The convex surrogate methods (CVX) achieved comparable accuracy to the Bayesian Monte Carlo (MC) method in recovering value functions and action selection probabilities.
Speed: The proposed methods were orders of magnitude faster.
- In 2-armed bandits, CVX-T solved problems in ~0.01 seconds, while MC took ~1.4 seconds and D-LOC took ~0.4 seconds.
- In 10-armed bandits with complex setups (SUB), D-LOC and MC became extremely slow (minutes to hours), whereas the convex methods remained efficient.
Parameter Recovery: While the convex relaxation directly recovers value functions perfectly, the subsequent parameter recovery step (CVX-LOC) yielded results very close to the direct local minimization but with much better stability and speed.
Real-World Application: On the mouse reversal learning dataset, the method successfully fitted complex models (SUB setup) with high log-likelihood and low computational cost.

5. Significance and Impact

Scalability: This work enables the fitting of complex RL models on large-scale behavioral datasets (e.g., thousands of trials, many subjects) which was previously computationally prohibitive using Bayesian methods or unstable using local minimization.
Reliability: By providing a convex lower bound, the method offers a way to judge the quality of other heuristic fits. If a heuristic solution is far from the convex lower bound, it is likely suboptimal.
Accessibility: The rlfit package democratizes access to state-of-the-art model fitting, removing the barrier of needing to implement complex MCMC samplers or debug non-convex optimizers.
Theoretical Bridge: The paper establishes a theoretical link between RL models and generalized linear models (multi-label logistic regression), showing that the latter is a convex relaxation of the former.

In summary, the paper provides a robust, fast, and theoretically grounded framework for analyzing decision-making behavior, solving the long-standing trade-off between computational efficiency and fitting accuracy in behavioral RL modeling.