Technical Summary: Testing Decision Makers without Counterfactuals

1. Problem Statement

The paper addresses the fundamental challenge of evaluating the quality of decision-making in environments where counterfactuals are unobserved. In many critical domains, such as geopolitical strategy or corporate management, a Decision-Maker (DM) acts under uncertainty, and only the outcome of their chosen action is observed. A competing agent, the Adviser (AD), provides recommendations, but the outcomes of these unchosen recommendations remain unobserved counterfactuals.

The central question is: Can an outside observer, relying solely on the sequence of observed decisions, recommendations, and realized outcomes, identify which agent is more informed?

The paper formalizes "more informed" using the Blackwell ordering: one agent possesses the other's information plus additional private information. The agents are modeled as strategic, with the sole objective of being selected as the "winner" by a public test. This induces a zero-sum game between the agents. The paper investigates whether a scoring test—a rule that assigns a numerical score to each observation and selects the agent with the higher average score—can successfully identify the more-informed agent in equilibrium.

2. Methodology and Framework

Model Structure

The authors model the interaction as a repeated bandit environment with $n$ arms (actions).

• Agents: A Decision-Maker (DM) and an Adviser (AD). One holds information $r$ (more informed), the other holds $q$ (less informed), where $r$ is a refinement of $q$.
• Observations: In each period, the DM chooses an arm $a_{DM}$, and the AD suggests an arm $a_{AD}$. Only the realization of $a_{DM}$ is observed.
• Incentives: Both agents are purely strategic, aiming to maximize their probability of being selected by the test. They do not care about social welfare unless it affects the test outcome.
• Test Definition: A scoring test $s: O \to \mathbb{R}$ assigns a score to every outcome triple $(a_{DM}, a_{AD}, u_{DM})$. The winner is determined by the limit average of these scores.

Analytical Approach

The core methodological contribution is the reduction of the complex repeated interaction with incomplete information to a one-shot interaction with complete information.

1. Separation vs. Identification: The authors define a scoring function as separating if it can distinguish between the more and less informed agents in a one-shot zero-sum game.
2. Equivalence Lemmas:
   • Lemma 1: If a scoring function is separating, the corresponding scoring test identifies the more-informed agent in any changing environment.
   • Lemma 2: If no separating scoring function exists, no scoring test can identify the more-informed agent, even in a fixed environment.
3. Complete Information Reduction: To determine if a function is separating, the authors analyze the game $G(s, p)$ where both agents share a common prior $p$ and have no private signals.
   • Lemma 3: A necessary condition for separation is that the value of the game $G(s, p)$ must be zero for all priors $p$.
   • Lemma 4: Under specific assumptions, separation requires that the optimal strategy of the tie-breaking agent in $G(s, p)$ uniquely identifies the welfare-maximizing arm for every prior.

3. Key Results

The paper establishes a dichotomy between simultaneous and sequential decision settings, and a fundamental trade-off between identification and welfare.

A. Simultaneous Decisions

Theorem 1 (Positive Result): In a setting with simultaneous decisions and a changing environment, there exists a scoring test that identifies the more-informed agent.

• Mechanism: The proposed test assigns a score equal to the realized welfare if the agents match ($a_{DM} = a_{AD}$) and a negative score proportional to the welfare if they mismatch.
• Logic: The more-informed agent can guarantee a positive expected score by matching the less-informed agent's action when it is optimal, while the less-informed agent cannot consistently mimic this behavior across all information structures.

Theorem 2 (Negative Result - Welfare Trade-off): In a binary-arm environment with simultaneous decisions, no scoring test can simultaneously identify the more-informed agent and achieve more than half of the optimal welfare.

• Implication: The test that successfully identifies the better agent forces the DM into a zero-sum game where they must randomize their actions to prevent exploitation. This randomization leads to severely suboptimal welfare (approaching the performance of a random guess) in worst-case scenarios.

B. Sequential Decisions

Theorem 3 (Impossibility Result): In a setting with sequential decisions and a fixed environment, no scoring test exists that identifies the more-informed agent.

• Reasoning: In a sequential setting, the follower can always react to the leader's choice. The authors show that for any scoring test, the follower can guarantee a non-negative score without needing any private information, rendering the test incapable of distinguishing information levels.

Nuance in Special Cases (Safe vs. Risky Arms):

• Proposition 1: If the environment consists of one safe arm and one risky arm, and the Adviser acts first, identification is possible. The more-informed agent can exploit the DM's lack of information regarding the risky arm's realization.
• Proposition 2: If the Decision-Maker acts first in the same safe/risky setting, identification is impossible. The DM's choice of the safe arm hides the counterfactual outcome of the risky arm, preventing the AD from proving superior information.

4. Significance and Claims

The paper claims to resolve the tension between evaluating decision quality and incentivizing optimal behavior in the absence of counterfactuals.

1. Feasibility of Identification: The authors demonstrate that, contrary to the intuition that unobserved counterfactuals make evaluation impossible, it is possible to identify the more-informed agent in simultaneous settings using simple scoring rules.
2. The Cost of Identification: The paper highlights a severe limitation: the very mechanism that allows an observer to identify the better agent (a zero-sum scoring game) destroys the incentives for the agent to maximize welfare. The paper claims that achieving identification inevitably leads to a "suboptimal" equilibrium where the DM's actions are driven by the test rather than the actual welfare of the outcome.
3. Strategic Nature of Data: The work emphasizes that in strategic environments, the data observed by the public are shaped by the agents' attempts to pass the test. This distinguishes the problem from non-strategic causal inference, where counterfactuals are simply unobserved but not manipulated.
4. Limitations of Scoring Tests: The results suggest that while scoring tests are sufficient for identification in simultaneous settings, they are fundamentally insufficient in sequential settings (unless specific structural constraints like "safe vs. risky" and order of moves are met).

The paper concludes that while it is possible to design a test to "pick a winner" based on information levels, doing so comes at the cost of the decision-maker's performance, creating an inherent conflict between the public's desire to select a capable leader and the need to ensure good decisions in the present.