📈 economics

A Bayesian latent class reinforcement learning framework to capture adaptive, feedback-driven travel behaviour

This paper proposes a Bayesian Latent Class Reinforcement Learning framework to model heterogeneous, adaptive travel behaviors, identifying three distinct traveler classes with unique preference evolution and exploration-exploitation strategies through analysis of driving simulator data.

Original authors: Georges Sfeir, Stephane Hess, Thomas O. Hancock, Filipe Rodrigues, Jamal Amani Rad, Michiel Bliemer, Matthew Beck, Fayyaz Khan

Published 2026-08-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Georges Sfeir, Stephane Hess, Thomas O. Hancock, Filipe Rodrigues, Jamal Amani Rad, Michiel Bliemer, Matthew Beck, Fayyaz Khan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to learn the best way to get to school. Sometimes you take the same route every day because it's fast and reliable. Other times, you try a new street just to see if it's faster, even if it might get you there later. This is the core of how humans make decisions: we balance sticking to what we know (exploitation) with trying new things to learn more (exploration). For a long time, scientists studying how people travel have used models that assume everyone learns the same way or that our preferences are fixed like stone. But in real life, people are messy. We learn at different speeds, we get confused by new information, and we might act totally differently when we are actually driving versus when we are just imagining a trip. This paper dives into that messy reality, using a mix of psychology and math to figure out not just what people choose, but how they learn to make those choices over time.

The researchers behind this study, led by Georges Sfeir and colleagues, decided to build a new kind of "travel brain" model called a Latent Class Reinforcement Learning (LCRL) framework. Think of it as a detective story where the detectives are trying to figure out why different drivers act so differently. Instead of assuming everyone is the same, they used a driving simulator to watch 83 people make 20 route choices each. Half the time, the participants actually drove the routes and felt the time pass (experienced feedback); the other half, they just clicked buttons on a screen with hypothetical times (stated preference). The team then fed this data into their fancy new model, which acts like a sorting machine. It doesn't just say "this person likes fast routes"; it asks, "Is this person a slow learner who sticks to old habits? A risk-taker who tries everything? Or someone who changes their mind depending on whether they are actually driving or just thinking about it?"

The model found that the group of drivers wasn't a single blob of people, but rather three distinct "types" of learners, each with their own personality. About 22% of the people were the "Context Chameleons." These folks acted one way when they were just imagining a trip on a computer screen (preferring the safe, reliable route) but completely flipped their behavior when they were actually behind the wheel in the simulator, suddenly craving the risky, unpredictable route. They learned slowly, taking their time to update their beliefs based on new experiences.

Then there was the largest group, making up nearly half the participants (49%), who were the "Persistent Exploiters." These drivers were the ones who loved the risky, unreliable route no matter what. Whether they were driving or just clicking buttons, they kept choosing the uncertain path, even when it sometimes took them longer. They were the most stubborn exploiters, sticking to their risky choice and adapting somewhat faster to feedback than the slowest group, though they still relied considerably on past experiences.

Finally, there was the third group, about 29% of the sample, who were the "Adaptive Switchers." These drivers were more likely to be female and have higher incomes. They started out liking the safe route when they were actually driving, but when they were just answering questions on a screen, they suddenly became more adventurous and picked the risky route. They displayed the most exploratory tendencies, switching back and forth between safe and risky options. While they had the highest learning rate estimate of the three groups, it was statistically insignificant, suggesting they maintained stable expectations that evolved gradually over time rather than reacting with rapid, volatile shifts.

The paper suggests that by ignoring these different "learning personalities," we might be making mistakes in how we design traffic systems or predict how people will react to new roads. If you treat a "Context Chameleon" like a "Persistent Exploiter," your traffic advice might fail completely. The researchers didn't just guess these groups existed; they used a sophisticated math technique called Variational Bayes to prove that splitting people into these three classes fit the data much better than assuming everyone learns the same way. They even ran simulations to show how each group would react if they kept getting the same bad traffic or the same good traffic over and over, confirming that their "personalities" really did change how they learned.

In short, this paper argues that to understand travel, we can't just look at the map; we have to look at the driver's mind. We need to know if they are the type to stick to the familiar, the type to chase the new, or the type who changes their mind depending on the situation. By building a model that captures these differences, the authors hope to help cities and planners design better systems that work for all types of learners, not just the average one.

Technical Summary: A Bayesian Latent Class Reinforcement Learning Framework

Problem Statement
Traditional travel behaviour research relies heavily on Discrete Choice Models (DCMs) that treat preferences as fixed constructs. However, real-world travel decisions often involve adaptive learning where individuals revise expectations based on feedback (e.g., delays, new routes). While Reinforcement Learning (RL) models from mathematical psychology effectively capture this dynamic, feedback-driven learning, existing applications in transport often assume a single, continuous distribution of learning parameters (e.g., hierarchical Bayesian models with unimodal priors). This approach fails to capture distinct, discrete learning types within a population (e.g., persistent explorers vs. rapid exploiters). Averaging across such divergent behaviours can lead to misleading inferences and poorly targeted policy interventions. Furthermore, conventional travel surveys often lack the longitudinal data required to capture the experienced feedback necessary for modeling dynamic learning.

Methodology
The paper proposes a Latent Class Reinforcement Learning (LCRL) framework that integrates Latent Class Choice Models (LCCMs) with class-specific Reinforcement Learning dynamics. The model is estimated using Variational Bayes (VB) within a fully Bayesian framework to handle high-dimensional parameter spaces and latent variables.

Reinforcement Learning Core: The model utilizes the Rescorla-Wagner learning rule.
- Prediction Error: Expectations ( $Q$ ) for an alternative are updated trial-by-trial based on the difference between the actual outcome ( $r$ ) and the prior expectation, scaled by a learning rate ( $\alpha$ ).
- Choice Rule: Updated expectations feed into a Random Utility Maximization (RUM) framework (Multinomial Logit), where utility depends on the expected outcome and a sensitivity parameter ( $\beta$ ) governing the exploration-exploitation trade-off.
Latent Class Integration: The model assumes the population consists of $K$ $K$ discrete latent classes.
- Class Membership: The probability of an individual belonging to a class is modeled via a multinomial logit function of socio-demographic and contextual attributes.
- Class-Specific Dynamics: Each class $k$ possesses its own set of RL parameters: learning rates ( $\alpha_k$ ), exploration-exploitation sensitivities ( $\beta_k$ ), bias parameters ( $\gamma_{ik}$ ), and initial expectations ( $Q_{i,0,k}$ ).
Estimation: Due to the intractability of the posterior distribution, the authors employ Variational Inference. They approximate the true posterior with a tractable distribution using a mean-field assumption and optimize the Evidence Lower Bound (ELBO) via stochastic gradient descent (Adam optimizer).

Key Contributions

Novel Framework: This is the first application to jointly model discrete behavioural heterogeneity and dynamic reinforcement learning using a fully Bayesian variational approach. It bridges econometric discrete choice models with mathematical psychology.
Handling Heterogeneity: Unlike standard hierarchical RL models that assume continuous variation around a common process, the LCRL explicitly identifies distinct learning types (multimodality), preventing the averaging of divergent behaviours.
Scalable Estimation: The use of Variational Inference allows for scalable estimation of complex dynamic models with high dimensionality, providing full posterior distributions over parameters.
Contextual Adaptation: The model is adapted to handle mixed data environments (driving simulator vs. stated preference), allowing for class-specific sensitivity and bias parameters that shift between experimental contexts.

Results
The model was applied to a dataset of 83 participants completing 20 route choice tasks (10 in a driving simulator, 10 in a stated preference survey) involving a choice between a reliable route (fixed 5 mins) and an unreliable route (variable 2–7 mins).

Model Fit: A three-class LCRL model was selected based on the Bayesian Information Criterion (BIC), significantly outperforming a single-class benchmark RL model in terms of Log-Likelihood, AIC, and BIC.
Identified Classes:
1. Class 1 (Context-Dependent Exploiters, ~22%): Primarily participants who started with Stated Preference (SP) tasks. They exhibit strong context-dependent preferences: they exploit the reliable route in hypothetical SP scenarios but switch to exploiting the unreliable route in the physical driving simulator (DS). They are slow learners (low $\alpha$ ).
2. Class 2 (Persistent Exploiters, ~49%): Dominated by males with lower incomes who started with DS tasks. They display a strong, persistent preference for the unreliable route across both contexts. They are moderate learners and exhibit high exploitation tendencies (significant $\beta$ ).
3. Class 3 (Context-Dependent Explorers, ~29%): More likely to be female and higher-income. They show a preference for the reliable route in DS tasks but reverse this in SP scenarios. They display the highest exploration tendencies (low $\beta$ ) and the highest learning rates, though their expectations evolve gradually.
Behavioral Insights: The results highlight that individuals differ not only in static preferences but in how they learn (learning rates), how they update expectations, and how they balance exploration versus exploitation. The model successfully captures the discrepancy between stated and experienced choices, attributing it to context-dependent shifts in bias and learning dynamics.

Significance and Claims
The paper claims that the LCRL framework provides a richer characterization of travel behaviour by capturing both experience-based learning and discrete forms of heterogeneity.

Theoretical: It offers a more comprehensive understanding of how decision-making preferences evolve dynamically over time, addressing a gap where traditional models overlook the mechanisms of initial preference emergence.
Practical: The findings suggest that policy interventions (e.g., information strategies) must be tailored to specific learning types, as strategies benefiting explorers may hinder exploiters.
Methodological: The integration of econometric choice models with RL and Bayesian inference is presented as an emerging modeling technology. The authors suggest its potential relevance for:
- Autonomous Vehicles (AVs): Designing routing algorithms that anticipate human learning and expectation formation in mixed-autonomy systems.
- Mobility-as-a-Service (MaaS): Identifying segments likely to explore new modes and understanding how incentives accelerate adoption.
- Transport Planning: Anticipating behavioural responses to network disruptions (e.g., closures, unreliability) and assessing how different groups learn to use new infrastructure.

The authors maintain a modest tone regarding future applications, noting that while the empirical focus is on human route choice, the framework offers a tool for designing information strategies that recognize adaptive behaviour rather than assuming static preferences. They also suggest future work could explore within-class heterogeneity by allowing RL parameters to be individual-specific.

Technical Summary: A Bayesian Latent Class Reinforcement Learning Framework

More like this