Unintended Consequences: Updating Causal Models

Imagine you are playing a video game, but you don't have the rulebook. You only have a guess about how the game works. Maybe you think jumping on a turtle makes you fly, or maybe you think collecting coins heals your character.

This paper is about how we (the players) learn the rules of life, how we make decisions based on our guesses, and why sometimes we get stuck in a loop of bad decisions even when the world is screaming at us that we are wrong.

Here is the breakdown of the paper's ideas using simple analogies:

1. The "Mental Map" (Causal Models)

Every time you make a choice, you are using a mental map of cause and effect.

The Real World: The actual code of the universe. (e.g., If you push a button, a door opens).
Your Map: Your belief about how the code works. (e.g., You believe pushing the button makes it rain).

The authors say that most of us don't know the true map. Instead, we hold a "probabilistic belief." This means we aren't 100% sure; we think, "I'm 80% sure pushing the button opens the door, but maybe it's 20% sure it opens a trap."

2. The "Surprise Box" (Updating Beliefs)

Usually, when you push a button and a door opens, your map gets a little stronger. You learn! This is called updating.

But what happens if you push the button and a dragon appears?

If your map says "Buttons open doors," a dragon is impossible. It's a "probability zero" event.
In the real world, when this happens, you have to throw out your old map and draw a new one.
The paper uses a fancy mathematical tool (called a Conditional Probability System) to describe how we handle these "impossible" surprises. It's like having a backup plan for when your brain breaks because reality didn't follow the script.

3. The Two Types of "Stuck" People

The authors introduce a concept called a Steady State. This is a state where you keep doing the same thing over and over, and your beliefs never change. They show two very different ways this can happen:

Scenario A: The Snake Trap (The Governor)

The Story: A Governor wants to kill snakes. He thinks: "If I pay people to bring me dead snake heads, the snake population will drop."
The Mistake: He doesn't realize that people will start breeding snakes to get the money.
The Result: He pays the bounty, and suddenly, there are more snakes.
The Update: The Governor sees the explosion of snakes. This contradicts his map ("Paying = Fewer Snakes"). Because the evidence is so shocking, he realizes, "Oh no, my map is wrong!" He changes his belief, stops the bounty, and the problem solves itself.
Lesson: Sometimes, a bad outcome is so obvious that it forces you to fix your brain.

Scenario B: The Sun Worshippers (The Aition Tribe)

The Story: A tribe believes their daily prayer is the only reason the sun rises. They think: "If we don't pray, the sun won't come up."
The Mistake: The sun rises because of physics, not prayer.
The Result: They pray every day. The sun rises.
The Trap: Because the sun always rises, their map is never proven wrong. Every morning, they say, "See? My map was right! The sun rose because we prayed."
The Steady State: They are stuck in a delusion. They keep doing the action (praying) because it "works" (the sun rises), but they never realize the action is actually useless. They are trapped in a loop where their bad beliefs are constantly "confirmed" by reality, even though reality is ignoring them.

4. The "Exploration vs. Exploitation" Dilemma

The paper also talks about how smart agents try to learn.

Exploitation: Doing what you think is best right now (e.g., taking the path you think leads to treasure).
Exploration: Trying something weird just to see what happens, even if it might be risky, because you might learn a new rule.

The authors show that if you are too confident in your current map, you will never explore. You will just keep taking the same path. But if you are too unsure, you might waste time trying random things. The "Steady State" is when you find the perfect balance where you are happy with your current path and don't feel the need to test new theories anymore.

5. The "Blind Spot" (Introspective Unawareness)

Finally, the paper asks: What if you don't even know what you don't know?

Imagine you are playing a game, but you don't know that "Magic Spells" exist. You only know about "Swords."
You can't even imagine that a spell could save you.
The authors suggest that sometimes, we are aware that we might be missing something ("I feel like I'm missing a piece of the puzzle"), but we can't imagine what that piece is.
In this case, we might keep doing the same thing, hoping that one day, a "paradigm shift" (a sudden new discovery) will save us, even though we can't plan for it.

Summary

This paper is a mathematical way of saying:

We all have wrong maps of how the world works.
Sometimes, reality shocks us into fixing those maps (The Governor).
Sometimes, reality tricks us into thinking our wrong maps are perfect, trapping us in a loop of bad decisions (The Sun Worshippers).
True wisdom involves knowing when to stick to your plan and when to admit you might be missing a whole part of the world you can't even imagine yet.

It explains why smart people can keep making dumb mistakes for years, and why sometimes, the only way to learn the truth is to be surprised by something your brain couldn't possibly predict.

Here is a detailed technical summary of the paper "Unintended Consequences: Updating Causal Models" by Halpern, Piermont, and Vierø.

1. Problem Statement

The paper addresses the problem of how agents with incorrect causal beliefs make decisions and how they update those beliefs based on feedback. In real-world scenarios, agents often misunderstand causal relationships (e.g., believing a ritual causes the sun to rise, or that paying bounties reduces snake populations).

The core challenge is twofold:

Decision Making under Misspecification: How does an agent choose actions when they hold a probabilistic belief over a set of possible causal models, none of which may be the true model?
Belief Updating and Convergence: How does an agent update their beliefs upon observing data generated by their actions? Specifically, does the agent eventually converge to the truth, or can they get stuck in a "steady state" of perpetual delusion where their incorrect beliefs are consistently rationalized by the data they generate?

The authors investigate whether agents can remain in a state of perpetual delusion (where incorrect beliefs lead to actions that generate data confirming those beliefs) or if they eventually correct their models, even in the face of inexplicable data.

2. Methodology

The authors build a formal framework combining Structural Causal Models (SCMs) with Decision Theory and Conditional Probability Systems (CPS).

A. Structural Causal Models (SCMs)

Variables: The environment consists of exogenous variables ( $U$ ) and endogenous variables ( $V$ ).
Models: A causal model $M = \langle S, F \rangle$ defines structural equations determining endogenous variables based on exogenous ones and other endogenous variables. The authors restrict attention to acyclic models.
Interventions: Agents can intervene on variables ( $do(X=x)$ ), creating counterfactual models $M_{\vec{X} \leftarrow \vec{x}}$ .
Uncertainty: The agent does not know the true model $M^*$ but holds a probabilistic belief over a set of possible models $\mathcal{M}$ . The context (exogenous variables) is drawn from a known distribution $\pi$ .

B. Beliefs and Updating (CPS)

To handle the possibility of observing data that contradicts all currently held models (probability 0 events), the authors equip the agent with a Conditional Probability System (CPS).

A CPS is a family of probability measures $\mu = \{\mu_F\}$ indexed by subsets of models.
Updating: Upon observing an outcome $o$ (resulting from an intervention), the agent updates their beliefs. If the observation is impossible under the current support of beliefs (i.e., $\pi(M, o) = 0$ for all $M$ in the support), the agent uses the CPS to condition on this "inexplicable" event, potentially shifting probability mass to models previously considered impossible.

C. Agency and Utility

The agent maximizes a value function $VAL(a, \mu)$ that balances:

Immediate Utility: The payoff derived from the intervention.
Information Value: The discounted future value of learning more about the causal structure to make better decisions later.

The agent solves a dynamic programming problem: $VAL(a, \mu) = E[UTIL + \delta \max_{a'} VAL(a', \mu')]$ .

D. Steady State Definition

A Steady State is defined as a pair $(\mu, a)$ where:

Optimality: Action $a$ maximizes the agent's value given beliefs $\mu$ .
Stability: The observation generated by action $a$ under the true model $M^*$ does not alter the agent's unconditional beliefs ( $\bar{\mu}$ ).

In a steady state, the agent never revises their beliefs, and the same action is repeated indefinitely.

3. Key Contributions

1. Formalizing "Perpetual Delusion"

The paper provides a rigorous definition of how an agent can remain in a steady state with incorrect beliefs.

Example 2 (The Aition Tribe): Agents believe a ritual causes the sun to rise. They perform the ritual. The sun rises (as it always does). The observation is consistent with their false model. They never update, remaining in a steady state of delusion.
Example 1 (The Cobra Bounty): The Governor believes paying for snake heads reduces the population. He pays, but the population increases (due to breeding). This observation is inexplicable under his model. Consequently, he updates his beliefs, abandons the bounty, and reaches a correct steady state.
Key Insight: Whether an agent corrects their beliefs depends on whether their optimal action generates data that is inexplicable under their current belief support. If the data is consistent with the false model, the error persists.

2. Non-Convergence in Stochastic Environments

The authors demonstrate that in stochastic environments (where context $\pi$ is not trivial), agents may fail to converge to a steady state or even stable behavior.

Example 3: An agent cycles infinitely between two actions. The true model generates data that looks like it supports Model A, causing a shift to Model B. But under Model B, the agent chooses an action that generates data supporting Model A again. The agent oscillates forever, never settling on a single model or action.

3. Introspective Unawareness

The paper extends the model to agents who are aware of their own unawareness.

These agents know they might be missing relevant models but cannot specify what those models are.
They assign a subjective value $v^*$ to the possibility of a "paradigm shift" (discovering something unforeseeable).
This leads to a modified value function $VAL^*$ where agents may choose suboptimal actions with low immediate payoff if they have a high probability of generating "surprising" data that could lead to a paradigm shift.

4. Results

Deterministic vs. Stochastic: In deterministic environments (fixed context), agents eventually converge to a steady state (Remark 2). In stochastic environments, convergence is not guaranteed; agents may cycle indefinitely or fail to stabilize.
Rationalization of Error: An agent's incorrect beliefs are "rationalized" if the data generated by their optimal action is consistent with those beliefs. If the true model $M^*$ generates data that the agent's support $\bar{\mu}$ deems impossible, the agent must update. If the data is possible (even if the model is wrong), the agent may never update.
The Role of Exploration: Agents maximize a trade-off between exploitation (immediate utility) and exploration (gathering information). However, if the agent's belief support excludes the true model, they may never explore the "right" path because they believe the current path is optimal and informative enough within their limited worldview.

5. Significance

Decision-Theoretic Foundation for Causal Reasoning: The paper bridges the gap between causal inference (SCMs) and decision theory. It shows how causal beliefs directly drive utility maximization and how the feedback loop of action $\to$ observation $\to$ belief update functions.
Explaining Persistent Errors: It offers a formal explanation for why societies or individuals persist in superstitious or economically inefficient behaviors (e.g., the Cobra Effect, superstitions) not just due to cognitive bias, but because their actions generate data that falsely validates their incorrect causal models.
Policy Implications: The framework suggests that simply providing data to an agent may not change their behavior if their current actions do not generate "inexplicable" data under their current model. To change a steady state of delusion, an external force might need to force an action that generates data the agent cannot explain, triggering a belief revision.
Awareness and Unawareness: The extension to "introspective unawareness" provides a novel way to model agents who know they don't know everything, influencing their willingness to take risks for the sake of potential discovery.

In summary, the paper demonstrates that steady states of incorrect beliefs are possible and stable if the agent's optimal actions are consistent with their false causal models. Convergence to the truth is not guaranteed and depends heavily on the stochasticity of the environment and the specific structure of the agent's belief support.