Nondistortionary belief elicitation

1. Problem Statement

The paper addresses a fundamental challenge in experimental economics and decision theory: how to elicit a subject's beliefs about a choice they have made without distorting the incentives of that original choice.

• The Context: Researchers often ask subjects to report beliefs regarding their own performance (e.g., "What is the probability your answer is correct?" or "What is your expected score?"). To ensure truthfulness, researchers typically use incentive-compatible mechanisms (like the Becker-DeGroot-Marschak (BDM) mechanism or quadratic scoring rules).
• The Conflict: If the belief elicitation mechanism rewards the subject based on the reported belief, and that belief is tied to the action chosen in the main task, the subject may alter their initial action to maximize the combined expected payoff of the decision problem and the belief elicitation stage, rather than maximizing the payoff of the decision problem alone.
• The Goal: The authors seek to identify necessary and sufficient conditions under which a researcher can design a payment scheme that:
  1. Incentivizes the subject to truthfully report their subjective expectation of a specific function $X(a, \theta)$.
  2. Leaves the set of optimal actions in the original decision problem unchanged (i.e., the belief elicitation is nondistortionary).

2. Methodology and Model

The authors construct a formal model combining a decision problem with a belief elicitation stage.

• Decision Problem: Defined by a tuple $(\Theta, A, u)$, where $\Theta$ is a finite set of states, $A$ is a finite set of actions, and $u: A \to \mathbb{R}^\Theta$ is the utility function.
• Belief Elicitation: After choosing an action $a$, the subject reports a number $r$ intended to represent their subjective expectation $E_p[X(a; \theta)]$ of a question profile $X: A \to \mathbb{R}^\Theta$.
• Incentivizability: A question $X$ is incentivizable if there exists an elicitation method $V$ such that the set of optimal actions in the combined problem (maximizing $E_p[V]$) is identical to the set of optimal actions in the original problem (maximizing $E_p[u]$), and truthful reporting is the unique optimal strategy for the report $r$.
• Key Concept: Adjacency Graph: The authors introduce the adjacency graph of the decision problem. Two actions $a$ and $b$ are adjacent if there exists a belief $p$ where $\{a, b\}$ are the unique optimal actions. The structure of this graph (trees, cycles, complete graphs) dictates the constraints on incentivizable questions.
• Key Concept: Alignment: A question $X$ is aligned with the utility $u$ if $X(a)$ is an affine transformation of $u(a)$ plus a state-dependent constant. Specifically, $X(a) \sim u(a) + d(\theta)$.

3. Key Contributions and Results

The paper provides a complete characterization of incentivizable questions across three canonical classes of decision problems, defined by the structure of their adjacency graphs.

A. General Necessary Condition: The Adjacency Lemma

The authors derive a fundamental necessary condition for any incentivizable question.

• Lemma 1 (Adjacency Lemma): If $X$ is incentivizable, then for every pair of adjacent actions $a$ and $b$, the question $X$ must be aligned with $u$ on the pair $\{a, b\}$.
• Mathematically, for adjacent $a, b$, there exist scalars $\rho, \sigma$ such that $\bar{X}(b) = \rho \Delta^b_a + \sigma \bar{X}(a)$, where $\Delta^b_a$ is the payoff difference vector and $\bar{v}$ denotes the projection of vector $v$ onto the hyperplane summing to zero.
• This implies that the relationship between the question values and utility values must be consistent across all "indifference points" between actions.

B. Characterization by Problem Structure

The paper classifies decision problems into three types based on their adjacency graphs and provides specific conditions for each:

1. Adjacency Trees (e.g., Monotone problems, Ordered states)

• Structure: The adjacency graph contains no cycles (a tree).
• Result (Theorem 1): A question is incentivizable if and only if it is piecewise aligned with $u$.
• Implication: The alignment parameters (scaling and shifting) can differ between different pairs of adjacent actions. The researcher does not need a single global alignment; local consistency across the tree is sufficient.

2. Complete Adjacency (e.g., Multiple-choice questions, Matching problems)

• Structure: Every pair of actions is adjacent (a complete graph). This occurs when the subject guesses a state and gets a reward only if correct.
• Result (Theorem 2): Under mild richness assumptions (linear independence of payoff vectors), a question is incentivizable if and only if it is globally aligned with $u$.
• Implication: In these highly connected problems, the constraints are tight. Questions like "What is the probability your answer is correct?" (which is often not aligned with utility in complex settings) are generally not incentivizable. However, questions about "Expected Regret" or "Expected Payoff" are incentivizable.

3. Product Problems (e.g., Multi-question tests, Sequential tasks)

• Structure: The decision problem consists of $I$ independent tasks. The adjacency graph is a product of the graphs of individual tasks. Two actions are adjacent only if they differ in exactly one task.
• Result (Theorem 3): A question is incentivizable if and only if it is weighted aligned.
• Definition: $X$ is weighted aligned if it is an affine transformation of a weighted sum of the utilities of the individual tasks: $X(a) \sim d(\theta) + \sum \tau_i u_i(a_i, \theta_i)$.
• Implication: This allows for more flexibility than global alignment. For example, a researcher can incentivize a question about the difference in performance between two parts of a test (where $\tau_i$ has opposite signs), even though this is not aligned with the total score utility. However, questions about the probability of exceeding a threshold (e.g., "Did you score > 50?") are generally not incentivizable because they are non-linear and not weighted aligned.

C. Mechanism Construction

For all incentivizable questions, the authors show how to construct a payment scheme using variants of the Becker-DeGroot-Marschak (BDM) mechanism.

• The mechanism involves normalizing the question to a $[0,1]$ interval and comparing the subject's report to a random draw.
• Crucially, the authors show that by adding the original decision problem's payoff to the belief elicitation payoff (or adjusting the weights), the incentives for the action choice remain undistorted.

4. Significance and Practical Implications

• Resolving the Distortion Dilemma: The paper provides a rigorous "checklist" for experimentalists. Before designing a belief elicitation task, researchers can determine if their specific question (e.g., confidence, regret, probability of success) can be incentivized without corrupting the main task.
• Critique of Common Practices: The authors demonstrate that many standard questions used in literature (e.g., "What is the probability you are correct?" in multiple-choice tests, or "What is the probability your score is above the median?") are not incentivizable in general settings. Using BDM or quadratic scoring rules for these questions creates hidden distortions where subjects may intentionally choose suboptimal actions to manipulate their belief reports.
• Alternative Measures: The paper suggests that researchers should shift toward incentivizable metrics, such as:
  • Expected Regret: Always incentivizable.
  • Expected Payoff: Incentivizable if aligned with utility.
  • Weighted Differences: Incentivizable in product problems (e.g., improvement over time).
• Theoretical Advancement: The work bridges the gap between mechanism design (elicitation) and decision theory (optimality), introducing the concept of the "adjacency graph" as a structural tool to analyze incentive compatibility in action-dependent belief elicitation.

5. Conclusion

Pęski and Stewart establish that nondistortionary belief elicitation is possible only for a specific class of questions determined by the geometry of the decision problem. While simple questions about expected utility or regret are always safe, complex questions about probabilities of thresholds or correctness often distort behavior. The paper offers a constructive framework for designing experiments that preserve the integrity of the primary decision task while accurately measuring cognitive uncertainty.