📈 economics

Measuring Choice Difficulty

This paper presents a theoretical framework demonstrating that common measures of choice difficulty—such as understanding, choice randomness, and confidence—are generally unrelated, while identifying specific conditions under which they align and highlighting the need for caution when interpreting these metrics across economics and psychophysics.

Original authors: Chris Chambers, Yusufcan Masatolioglu, Paulo Natenzon, Collin Raymond

Published 2026-04-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chris Chambers, Yusufcan Masatolioglu, Paulo Natenzon, Collin Raymond

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a puzzle. Sometimes the puzzle is easy, and you solve it quickly and correctly. Other times, it's hard, and you might guess wrong, feel unsure, or take a long time.

In the world of economics and psychology, researchers have developed three main "thermometers" to measure how difficult a decision is for a person:

Randomness: How often does the person flip a coin and guess? (High randomness = high difficulty).
Confidence: How sure does the person feel about their answer? (Low confidence = high difficulty).
Understanding (Value): How much money or "utility" does the person actually get out of the decision? (Low value = high difficulty).

The big question this paper asks is: Do these three thermometers always move together? If someone is guessing a lot (random), does that mean they are also unsure (low confidence) and getting poor results (low value)?

The Big Surprise: The Thermometers Don't Always Agree

The authors' main finding is a bit of a shocker: No, they don't always move together.

Think of it like a car dashboard. Usually, if your engine is overheating (high difficulty), you expect the temperature gauge to go up, the oil light to flash, and the speed to drop. You expect all three to tell the same story.

This paper shows that in decision-making, the dashboard can be broken. You can have a situation where:

The driver is very confident they are right, but they are actually getting a terrible result (low value).
The driver is guessing randomly, but they are actually getting a great result (high value).
The driver is very unsure, but they are getting a great result.

The "Why" (The Metaphor):
The authors explain that "Understanding" (Value) cares about how much you win. "Randomness" and "Confidence" only care about whether you win or lose.

The High-Stakes Gamble: Imagine a game where you can win $1 or $1,000,000.
- If you pick the $1,000,000 option, you might be very confident because the signal you got was strong.
- However, if the $1,000,000 option only happens 1% of the time, you might end up losing most of the time.
- In this case, your confidence is high, but your understanding (total money earned) is low because you keep picking the rare, high-reward option that usually fails.
- Conversely, you might pick the safe $1 option every time. You might feel unsure (low confidence) because the signals are noisy, but you are winning consistently (high understanding).

When Do They Agree? (The "Magic" Conditions)

The paper isn't just negative; it also says, "Here is when the thermometers do agree." They found two specific scenarios where Randomness, Confidence, and Value all move in the same direction:

1. The "Aligned Shift" (Better Signals)
Imagine you are trying to find a hidden treasure.

Scenario A: Your map is blurry. Sometimes it points to the wrong spot.
Scenario B: You get a "magic upgrade" to your map. Now, whenever the treasure is in the North, the map points North more often than before. It never points South when the treasure is North.
Result: If you only make these specific "aligned" upgrades to your map, then:
- You will guess less (less randomness).
- You will feel more sure (higher confidence).
- You will find more treasure (higher value).
- Note: This only works if the map upgrades are "honest" (indicative) and don't trick you.

2. The "Psychophysics" Game (Right vs. Wrong Only)
In many economics experiments, getting the "right" answer gives you $10, and the "wrong" answer gives you $0. But in some psychology experiments (like judging which line is longer), the reward is binary: 1 point if you are right, 0 if you are wrong. It doesn't matter how much longer the line is; you just get a point for being correct.

Result: In this specific "Right/Wrong" world, Confidence and Value become the same thing. If you are confident, you are getting points. If you are getting points, you are confident. The "how much" factor disappears, so the thermometers align.

What About Other Measures?

The authors also looked at other ways people try to measure difficulty, like:

Willingness to Switch: How much money would you pay to change your answer?
- Finding: If you can measure this in "utils" (pure satisfaction units), it perfectly matches "Understanding." It's a great thermometer.
Attenuation: How much does your choice change when the situation changes slightly?
- Finding: This doesn't fix the problem. You can still have a situation where attenuation looks "good" (low difficulty) but your actual results are terrible.

The Takeaway for Researchers

If you are a scientist trying to figure out if a person "understands" a problem:

Don't just look at how random their choices are.
Don't just ask how confident they feel.
Be careful when comparing Economics experiments (where you care about the amount of money) with Psychology experiments (where you only care about being right). They operate under different rules.

In short: Just because someone is guessing a lot or feels unsure, it doesn't mean they are doing a bad job at maximizing their reward. And just because they feel super confident, it doesn't mean they are actually winning. You have to look at the specific rules of the game to know which thermometer to trust.

1. Problem Statement

The paper addresses a fundamental disconnect in the literature on decision-making under uncertainty. Empirical studies in economics and psychology frequently use various metrics to quantify "choice difficulty," including:

Choice Randomness: The degree of stochasticity in choices (often linked to Fechnerian models).
Confidence: The decision-maker's (DM) belief that their chosen option is ex-post correct.
Understanding (Value of Information): The ex-ante expected utility derived from the decision problem.
Attenuation: The elasticity of choice probabilities with respect to state changes.

The Core Question: Do these metrics move together? Specifically, does a "more difficult" choice (implying less understanding) necessarily manifest as higher randomness and lower confidence? The authors argue that existing literature often assumes a positive correlation between these measures based on specific parametric assumptions (e.g., Gaussian priors/signals), but lacks a general theoretical foundation to support this.

2. Methodology

The authors develop a unifying theoretical framework based on Bayesian Expected Utility (BEU) theory and Blackwell's ordering of experiments.

Environment: A binary choice setting ( $X = \{x, y\}$ ) where the DM faces a state-dependent utility function $u(x, \omega)$ and a prior belief $\pi$ over states $\Omega$ .
The Primitive: The DM does not know the true state but observes a signal $s$ from a Blackwell experiment $\sigma$ . The experiment is the primitive representing the "mental process" or information structure.
Three Primary Orderings: The authors define three partial orders over the set of experiments:
1. Randomness ( $\preceq_R$ ): Defined by the maximum probability of choosing an option in a given state. An experiment is "less random" if the DM is more decisive (higher max probability) in every state.
2. Confidence ( $\preceq_C$ ): Defined as the posterior probability that the chosen option is optimal, conditional on the choice being made.
3. Choice Payoff / Understanding ( $\preceq_W$ ): The ex-ante expected utility of the decision problem given the experiment. This serves as the proxy for "understanding."
Conditioning: The analysis distinguishes between state-by-state comparisons (conditional on $\omega$ ) and expected comparisons (averaged over the prior $\pi$ ).

3. Key Contributions and Results

A. The Negative Result: General Incommensurability

The paper's primary theoretical contribution is a negative result: In a general setting, the three orderings are unrelated.

Proposition 1: Knowing the ranking of two experiments in any two of the three dimensions (e.g., one is less random and more confident) does not imply a specific ranking in the third dimension (e.g., higher expected payoff).
Implication: A DM can exhibit an experiment that is less random and generates higher confidence, yet yields a lower expected utility (worse understanding) than a competing experiment. Conversely, an experiment with higher payoffs can induce higher randomness or lower confidence.
Examples: The authors provide counter-examples where:
- An experiment is more informative (higher payoff) but introduces randomness in a specific state, lowering overall confidence.
- An experiment increases confidence and payoff but increases randomness.
Root Cause: The disconnect arises because "Understanding" (Payoff) relies on cardinal utility values (how much better option A is than B), whereas "Randomness" and "Confidence" rely on ordinal accuracy (whether the choice matches the optimal option).

B. Positive Results: Sufficient Conditions for Co-movement

The authors identify specific conditions under which the measures align (co-move).

1. Restrictions on Signal Structure (Blackwell Experiments)

Aligned and Neutral Shifts: The authors define "aligned shifts" (moving probability mass from signals inducing the "wrong" choice to signals inducing the "right" choice in a specific state) and "neutral shifts" (shuffling mass among signals inducing the same choice).
Proposition 2: If an experiment $\sigma'$ $σ^{'}$ can be constructed from $\sigma$ $σ$ via a sequence of aligned and neutral shifts, then:
- $\sigma'$ has higher expected payoff.
- $\sigma'$ has higher expected confidence.
- If the experiments are indicative (signals generally point toward the correct option), $\sigma'$ is less random.
Application: This class includes Rational Inattention Logit models (where decreasing the cost parameter $\lambda$ corresponds to aligned shifts), explaining why these specific models often show co-movement.
Limitation: Independent repetition of an experiment (e.g., $t$ draws) improves payoff but does not guarantee reduced randomness or increased confidence.

2. Restrictions on Payoff Structure (Psychophysical Experiments)

Psychophysical Payoffs: In many psychology experiments, the payoff is binary: 1 if the correct option is chosen, 0 otherwise (ignoring the magnitude of the utility difference).
Proposition 8: Under psychophysical payoffs, Overall Confidence (averaged across states and options) is mathematically equivalent to Expected Payoff (Understanding).
Significance: This explains why confidence is often a valid proxy for understanding in psychophysics but fails in standard economic experiments where utility magnitudes matter.

3. Alternative Measures

Willingness to Accept (WTA) to Switch: The authors show that if WTA is measured in utils (not just money), it is an affine transformation of the expected payoff. Thus, WTA perfectly recovers the ordering of "understanding."
Attenuation: The paper demonstrates that measuring "attenuation" (elasticity of choice) does not overcome the negative results; it remains unrelated to payoff dominance in general.

4. Significance and Implications

Caution in Interpretation: Researchers must be extremely careful when interpreting "choice difficulty." Observing high randomness or low confidence does not necessarily imply the decision-maker has a poor understanding of the problem (low expected utility), nor does low randomness guarantee high understanding.
Portability between Fields: Insights from psychophysics (where payoffs are binary/correctness-based) cannot be directly ported to economics (where payoffs are cardinal/variable). In psychophysics, confidence is a valid proxy for understanding; in economics, it is not.
Experimental Design:
- To measure "understanding" directly, researchers should elicit WTA in utils rather than relying on proxies like confidence or randomness.
- When comparing experiments, the level of aggregation matters. State-by-state measures and aggregate measures behave differently under various sufficient conditions.
Theoretical Unification: The paper provides the first general non-parametric framework linking stochastic choice, meta-cognition (confidence), and information theory (Blackwell ordering), clarifying why specific functional forms (like Gaussian/Logit) often produce misleadingly consistent results.

Summary Conclusion

The paper establishes that choice randomness, confidence, and understanding are distinct concepts that generally do not align. Their correlation is an artifact of specific structural assumptions (restricted signal families or binary payoff structures) rather than a universal law of decision-making. The authors provide precise conditions under which these measures align and offer WTA (in utils) as a robust alternative metric for measuring decision-maker understanding.