💰 quantitative finance

Reinforcement Learning and Consumption-Savings Behavior

This paper proposes a reinforcement learning model with neural network approximation to explain how adaptive learning mechanisms, rather than standard rational expectations, simultaneously generate the high marginal propensities to consume among low-asset unemployed households and the persistent consumption "scarring" observed in individuals with past unemployment experiences.

Original authors: Brandon Kaplowitz

Published 2026-08-06

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Brandon Kaplowitz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Brain's GPS and the Mystery of Spending

Imagine your brain as a super-smart GPS navigating the chaotic terrain of daily life. Usually, we assume people make financial decisions like perfect mathematicians: they know exactly how much money they will earn next year, they calculate the perfect balance between spending today and saving for tomorrow, and they never make a mistake. This is the "Rational Expectations" view, the standard map economists have used for decades. But real life is messier. We don't have a crystal ball; we only have our past experiences.

Enter Reinforcement Learning. Think of this not as a super-computer solving equations, but as a video game player learning by trial and error. In a game, you don't know the rules of the level beforehand. You try a move, get a reward (or a "game over"), and your brain updates its internal map of "what works" for next time. If you get a high score, you remember that path. If you lose, you avoid it. This paper asks a fascinating question: What if our spending habits aren't calculated by a perfect mathematician, but are instead shaped by this same "try, fail, learn" process? It suggests that our brains are constantly updating a mental model of how much we should save based on the surprises we've faced in the past, rather than a perfect prediction of the future.

The Puzzle: Why Do Some People Spend More When They're Broke?

Economists have been scratching their heads over two strange patterns in how people spend money during tough times.

The First Mystery: During the pandemic, the government sent out stimulus checks. Researchers found that unemployed people who used to have very little money in their bank accounts spent about 0.53 of every extra dollar they received. But unemployed people who used to have lots of money spent only 0.29 of every extra dollar. This is weird because, at the moment they got the money, neither group was actually "broke" or unable to borrow. Standard economic theory says if you have enough cash, you shouldn't be super eager to spend a sudden windfall. Why would someone's past lack of money make them spend more now, even if they are currently fine?

The Second Mystery: Another study found that people who have been unemployed in the past tend to save more and spend less for years afterward, even if they are currently employed and doing well. This is called "scarring." It's like a ghost from the past that keeps you from enjoying your current paycheck.

The big question is: Can one single explanation solve both mysteries? Usually, economists have to pick one. "Maybe they are just scared of the future," or "Maybe they are just bad at math." But this paper suggests a different culprit: Reinforcement Learning.

The Experiment: Teaching a Digital Agent to Learn

The author, Brandon Kaplowitz, built a computer simulation to test this idea. Instead of giving his digital agents a perfect map of the future, he gave them a "neural network"—a simplified version of a brain that learns by experience.

Here is how the simulation works:

The Setup: The agents start with a rough guess about how to save and spend, similar to what a perfect mathematician would do.
The Game: They live through 50 quarters (about 12.5 years) of simulated life. They get jobs, lose jobs, and get paid.
The Learning: When an agent gets a surprise—like suddenly losing their job—they feel a "temporal difference error." In video game terms, this is like expecting to get a gold coin but getting a rock instead. The brain realizes, "Whoa, my map was wrong!"
The Update: The agent's neural network tweaks its internal weights to fix this mistake. It doesn't just change one number; it reshapes its entire understanding of how valuable saving is for the future.

The Big Reveal: How "Scars" Change the Map

The simulation produced results that looked almost exactly like the real-world puzzles. Here is the magic mechanism:

When an agent experiences unemployment, their brain gets a "negative surprise." To fix this, the neural network updates its mental map of the future. This update does two things simultaneously:

It makes the map steeper: The agent realizes that saving money is more important than they thought. This causes them to spend less overall (the "scarring" effect).
It makes the map curvier: The agent realizes that the value of saving drops off quickly as they get richer. This makes them extremely eager to spend any new money they get, because they feel the "pain" of having low savings is much sharper than before.

The Result:

The Scarring Effect: Agents with a history of unemployment ended up with lower average spending, just like the real-world data.
The High Spending Spree: When these same agents got a stimulus check, they spent 0.50 of it (very close to the real-world 0.53), while agents with a history of stable jobs only spent 0.34 (close to the real-world 0.29).

What This Means (and What It Doesn't)

The paper suggests that we don't need to assume people are "irrational" or have "weird psychology" to explain these behaviors. Instead, our brains are simply doing what they do best: learning from the surprises of the past.

It's not about being pessimistic: The author tested a version where agents just became "pessimistic" (thinking unemployment is more likely). That model failed. It made people spend less, but it also made them spend less of a windfall, which is the opposite of what we see in real life. The reinforcement learning model is the only one that got both the "spend less overall" and "spend more of a bonus" parts right.
It's a simulation, not a crystal ball: The author is careful to note that these are results from a computer simulation. The numbers match the real-world studies (like 0.50 vs 0.34 in the simulation vs 0.53 vs 0.29 in reality), but this is a model of how it could work, not a final proof of how every human brain works.
The "Black Box" Limitation: Because the agents learn through a neural network, we can't easily ask them, "What do you think your chances of getting fired are?" The model works by adjusting values, not by calculating specific probabilities. This is a strength (it's flexible) but also a weakness (we can't see the exact "beliefs" inside the agent's head).

In short, this paper offers a playful but powerful new way to look at our wallets. It suggests that our spending habits are a living history book, written by the surprises we've faced. When we get scared by a job loss, our brain doesn't just get sad; it rewrites the rules of the game, making us save more for a rainy day and spend our windfalls faster, just in case the storm comes back.

Technical Summary: Reinforcement Learning and Consumption-Savings Behavior

Problem Statement
This paper addresses two persistent empirical puzzles in household consumption behavior during economic downturns that challenge standard rational expectations models, even those augmented with borrowing constraints:

High MPCs for Low-Asset Households: Ganong et al. (2024) document that unemployed households with previously low liquid assets exhibit significantly higher marginal propensities to consume (MPCs) out of stimulus transfers (0.53) compared to high-asset households (0.29), despite neither group facing current borrowing constraints. Standard models struggle to explain this heterogeneity without assuming ex-ante differences or binding constraints.
Consumption Scarring: Malmendier and Shen (2024) find that households with more past unemployment experiences maintain persistently lower consumption levels (higher savings rates) even after controlling for current income, wealth, and employment status.

Existing explanations face a reconciliation problem: mechanisms that generate high MPCs (like borrowing constraints) often fail to explain the "scarring" effect (lower average consumption), while mechanisms that explain scarring (like increased subjective unemployment risk) typically predict lower MPCs, contradicting the empirical evidence.

Methodology
The author develops a structural model where agents are economically rational decision-makers subject to informational and computational constraints. Unlike standard models where agents solve the Bellman equation given known income processes, these agents use Reinforcement Learning (RL) to approximate their continuation value.

Agent Setup: Agents face a Markov Decision Problem (MDP) with cash-on-hand ( $x$ ) and labor income ( $y$ ) as state variables. They do not know the true income transition matrix ( $P$ ) and cannot solve the fixed-point Bellman equation analytically.
Learning Mechanism: Agents employ Q-learning with neural network approximation.
- Value Function Approximation: Agents use a two-layer neural network with Rectified Linear Unit (ReLU) activations to estimate the expected value function ( $\hat{EV}$ ) based on current assets and income.
- Policy Derivation: To ensure smooth, monotone consumption policies, the agent fits a polynomial to the neural network's output for counterfactual analysis, though they utilize the raw network output for realized decisions.
- Update Rule: Agents update the neural network parameters ( $\phi$ ) via gradient descent on the Temporal Difference (TD) error. The TD error ( $\epsilon$ ) represents the "surprise" between the realized value of a choice and the agent's prior expectation.
- Initialization: Agents begin with weights initialized to perfectly fit the rational expectations solution (simulating a starting point of rationality). Deviations arise solely from the learning dynamics driven by experienced shocks.
Simulation: The model is calibrated using parameters from Ganong et al. (2024) and Krueger et al. (2016). Agents are simulated over 50 quarters (approx. 12.5 years) with an initial asset distribution drawn from the 2016 Survey of Consumer Finances.

Key Contributions

Unifying Framework: The paper proposes reinforcement learning as a unifying mechanism that simultaneously generates lower average consumption (scarring) and higher MPCs following negative income shocks. This resolves the contradiction found in belief-updating models (e.g., Malmendier and Shen, 2024), where increased perceived risk lowers both consumption and MPCs.
Mechanism Identification: The paper identifies the specific role of the neural network's gradient update rule. When an agent experiences unemployment, the negative TD error causes a localized downward shift in the estimated value function ( $\hat{EV}$ $\hat{E V}$ ). Due to the distance-decay property of neural network updates (related to the Neural Tangent Kernel), this shift:
- Increases the slope of $\hat{EV}$ near current assets, making future savings appear more valuable, which reduces current consumption (scarring).
- Increases the curvature (concavity) of $\hat{EV}$ , causing the marginal return to saving to decline more rapidly as assets rise, which steepens the consumption policy and raises the MPC.
Distinction from Rational Expectations: The model demonstrates that experience-based heterogeneity (ex-post) can replicate empirical patterns without requiring ex-ante heterogeneity or binding borrowing constraints.

Results

MPC Replication: The simulation replicates the findings of Ganong et al. (2024). Agents with low initial assets exhibit an MPC of 0.50, while high-asset agents exhibit an MPC of 0.34. This gap persists even when agents are not borrowing constrained, matching the empirical magnitude (0.53 vs. 0.29).
Scarring Qualitative Agreement: Regarding the findings of Malmendier and Shen (2024), the simulation shows broad qualitative agreement but does not fully match the empirical results. Specifically, the regression coefficient for consumption on the unemployment experience index is negative and statistically significant, confirming the direction of the "scarring" effect. However, the magnitude of this effect in the simulation is approximately an order of magnitude smaller than the empirical estimate. The author attributes this discrepancy partly to frequency differences (quarterly simulation vs. biennial data).
Comparison with Pessimistic Beliefs: The author contrasts the RL agent with a "pessimistic" agent who updates beliefs about unemployment probabilities (Anticipated Utility). The pessimistic agent correctly predicts lower consumption but incorrectly predicts lower MPCs. The RL agent is the only one that matches both empirical directions (lower consumption, higher MPC).

Significance and Claims
The paper claims that adaptive learning through reinforcement learning provides a plausible behavioral foundation for consumption anomalies that standard rational models cannot explain. It suggests that the "scarring" effect and high MPCs are not necessarily due to changes in beliefs about probabilities or binding constraints, but rather due to value function approximation errors that evolve with experience.

The author notes modest limitations:

The magnitude of the scarring effect in the simulation is smaller (by an order of magnitude) than in the empirical data, though the paper attributes this partly to frequency differences (quarterly simulation vs. biennial data).
The model is "model-free," meaning it does not explicitly recover the agent's subjective beliefs about future unemployment probabilities, limiting the ability to use belief surveys for calibration.
The current analysis focuses on single-agent dynamics; the paper does not explore general equilibrium implications where multiple RL agents interact.

Ultimately, the paper argues that RL offers a distinct and unifying explanation for how past experiences shape current consumption behavior beyond what current economic conditions would predict.