📈 economics

The Galileo problem in expert panels: screening calibration under decision loss and blame avoidance

This paper proposes an expected-loss framework for determining the optimal severity of evaluator-level screening in expert panels, arguing that organizers may rationally accept some decision-quality loss to minimize institutional exposure from blame and controversy, while offering a probability-of-exclusion heuristic to operationalize these trade-offs.

Original authors: Krzysztof Kontek

Published 2026-09-25

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Krzysztof Kontek

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

When a group of experts must choose a winner, pick a grant recipient, or hire a new leader, they rarely rely on a single voice. Instead, they gather a panel, ask each member to score the options, and combine those numbers into a final decision. This system is designed to be fair, pooling diverse knowledge to reduce the chance that one person's mistake or bias will ruin the outcome. However, this very strength creates a vulnerability: what happens when one member of the panel is wildly different from the rest? That person might be a saboteur trying to rig the result, a confused participant who misunderstood the rules, or a brilliant maverick who sees a truth the others have missed. The dilemma is known as the "Galileo problem," named after the astronomer who was right while the majority of his peers were wrong. If the group ignores the outlier, they might silence a genius; if they listen to the outlier, they might let a fraud or a mistake dictate the result.

For decades, institutions have tried to solve this by tweaking individual scores. If one judge gives a number that is too high or too low for a specific candidate, the system simply cuts that number off or pulls it closer to the average. This approach treats every candidate separately, ignoring the fact that a judge might be consistently strange across all their evaluations. A newer idea, explored in recent research, suggests looking at the judge's entire record at once. Instead of fixing individual numbers, the system checks the judge's whole profile. If a judge's pattern of scoring is too far from the group's norm, that judge is removed from the process entirely, and their influence is wiped out for every candidate. This method is cleaner and more transparent, but it brings the same old dilemma back with a vengeance: how strict should the rule be?

A new study by Krzysztof Kontek investigates exactly where to draw that line. The research does not claim to have found a perfect, universal formula that works in every situation. Instead, it builds a detailed simulation to understand the trade-offs involved in setting the rules. The author models a typical expert panel and asks what happens when the threshold for removing a judge is set at different levels of strictness. The study finds that there is no single "magic number" that guarantees the best result. Rather, there is a broad, safe zone where the rules work well. If the rule is too loose, the panel keeps judges who are confused or trying to manipulate the result, which hurts the quality of the final decision. If the rule is too strict, the panel starts throwing out judges who are actually right but just happen to think differently, which also hurts the decision. The sweet spot lies in the middle, where the system is strict enough to catch the obvious troublemakers but loose enough to keep the thoughtful dissenters.

The paper also reveals a surprising twist that has nothing to do with math and everything to do with human nature. The person who sets the rules for the panel—the organizer—does not always want the same thing as the panel itself. The panel's goal is simply to make the best possible decision. The organizer, however, has to live with the fallout if things go wrong. If the panel keeps a judge attempting to manipulate the result and the outcome looks rigged, the organizer gets blamed. If the panel throws out a brilliant judge who was right, and the public finds out later, the organizer gets blamed for that too. The study shows that because organizers fear these different kinds of blame, they might choose a rule that is slightly different from the one that would produce the mathematically perfect decision. They might make the rules stricter to avoid the scandal of a rigged result, even if it means occasionally silencing a valuable voice. Or they might make the rules looser to avoid the scandal of firing a famous expert, even if it means accepting a slightly higher risk of a bad decision.

To make these complex choices practical, the study introduces a simple way to talk about the rules. Instead of using confusing technical cutoffs that change depending on how many judges are in the room, the author suggests using a "probability of exclusion." This is a straightforward percentage that tells the organizers: "Under normal circumstances, an ordinary, honest judge has a one-in-twenty chance of being removed by this rule." This number stays the same whether the panel has ten judges or twenty, making it easy to compare different systems and set clear expectations before the judging even begins. The research confirms that while you cannot perfectly tell the difference between a liar and a genius just by looking at their scores, you can design a system that minimizes the damage either one might cause. By accepting that some mistakes are inevitable, institutions can choose a level of strictness that protects the integrity of the process without sacrificing the wisdom of the crowd.

The study relies on computer simulations to test these ideas, running thousands of scenarios with different types of judges and different levels of disagreement. It does not claim to have solved the problem of human judgment forever, but it provides a clear map of the terrain. It shows that the best approach is not to hunt for a perfect filter that catches every bad actor while letting every good one through, because such a filter does not exist. Instead, the best approach is to acknowledge the limits of what can be known and to set a rule that balances the risk of letting a fraud in against the risk of kicking out a genius. The research suggests that for most expert panels, a moderate level of strictness works best, creating a buffer zone where the system is robust enough to handle errors without becoming so rigid that it crushes innovation. Ultimately, the paper argues that the goal of a panel is not just to get the right answer, but to design a process that the public can trust, even when the answer is hard to find.

Technical Summary: The Galileo Problem in Expert Panels

Problem Statement
Institutions relying on expert panels (e.g., grant review, hiring committees, music competitions) face a fundamental tension between controlling strategic manipulation and protecting sincere minority judgment. This is termed the "Galileo problem." While traditional methods like score-level trimming and clipping limit extreme influence, they operate on individual score coordinates, leaving the evaluator's profile intact. A newer alternative, evaluator-level screening (specifically the Excluding Outlier Jurors or EOJ procedure), conditions an evaluator's entire influence on an admissibility rule.

However, a core informational constraint exists: under observational ambiguity, the same submitted score configuration can arise from either strategic manipulation (harmful) or sincere minority judgment (valuable). As formalized by Borowiecki and Law (2026), no deterministic rule can simultaneously guarantee the control of manipulation and the protection of pluralism. Consequently, the design problem shifts from perfect classification to "constitutional choice": determining the optimal severity of screening where the trade-off between suppressing valuable dissent and admitting strategic distortion must be calibrated.

Methodology
The paper develops an expected-loss framework to analyze the calibration of evaluator-level screening. The methodology proceeds through several layers:

Institutional Architecture: The analysis utilizes the EOJ framework (Kontek and Kenner, 2025), where an evaluator is excluded if their Manhattan profile distance from the panel median exceeds a threshold $T_K$ . The threshold is defined by a tolerance multiplier $K$ applied to the median absolute deviation of profile distances.
Latent State Modeling: The paper defines four latent states for a focal evaluator:
- $H$ : Ordinary, competent, sincere judgment.
- $F$ : Sincere but imprecise (harmful due to low precision).
- $M$ : Strategic distortion (harmful manipulation).
- $G$ : Galileo-type informed dissent (valuable but anomalous).
- Note: States $M$ and $G$ are observationally aliased; they generate identical score profiles but have opposite decision values.
Three-Layer Analysis: The paper distinguishes between:
- Statistical Exclusion: The probability an evaluator crosses the boundary.
- Pivotality: The probability that retaining vs. excluding the evaluator changes the collective winner.
- Consequential Exclusion: The intersection of the above, where an exclusion actually alters the outcome.
Loss Functions:
- Decision Quality ( $L_Q$ ): Minimizes expected decision loss (e.g., selecting the wrong winner).
- Organiser Loss ( $L_O$ ): Incorporates decision loss plus "blame avoidance" costs. This includes reputational/procedural costs ( $C_M$ ) from retaining a harmful anomaly and costs ( $C_G$ ) from controversially excluding a valuable dissenter.
Operational Heuristic: The paper introduces Probability of Exclusion (PEx), defined as the ex ante probability that an ordinary evaluator profile crosses the screening boundary under a pre-specified reference model. This translates a theoretical cutoff $K$ into an interpretable operating characteristic.

Key Results

Impossibility Extension: The paper extends the Borowiecki–Law impossibility result to deterministic score-based interventions generally (including trimming and clipping). No deterministic procedure can recover latent information absent from observed scores; thus, the trade-off between manipulation control and pluralism protection is inherent to all such rules.
Decision-Quality Optimum: In the illustrative benchmark ( $N=17, A=6$ ), the decision-quality optimum ( $K_Q^*$ ) lies in a "broad low-loss region" around $K \approx 2.2$ . This is stricter than the statistical reference point of PE5 (5% tail probability, $K \approx 3$ ) but more permissive than the decision-neutral noise boundary ( $K \approx 2$ ). The loss surface is flat, suggesting that a range of cutoffs yields economically indistinguishable decision quality.
The Public Choice Wedge: The paper demonstrates that the organiser's preferred rule ( $K_O^*$ $K_{O}^{*}$ ) often diverges from the decision-quality optimum.
- If the expected institutional cost of retaining a harmful anomaly ( $\pi_M C_M$ ) exceeds the cost of excluding a Galileo-type dissenter ( $\pi_G C_G$ ), the organiser rationally chooses stricter screening ( $K_O^* < K_Q^*$ ), accepting higher decision loss to reduce exposure to blame.
- Conversely, if the cost of excluding a prominent dissenter is higher, the organiser chooses more permissive screening.
- This creates a "Public Choice wedge" where institutional self-protection justifies a rule that is suboptimal for pure decision quality.
PEx as a Calibration Tool: The paper establishes that raw cutoffs ( $K$ ) are not portable across different panel sizes or scorecard widths. PEx provides a stable, ex ante metric for communicating and calibrating screening severity (e.g., "a 5% risk of excluding an ordinary profile") regardless of the specific numerical threshold required for a given panel configuration.

Significance and Claims
The paper claims four primary contributions:

Generalization of the Galileo Problem: It extends the logic of observational equivalence from specific exclusion rules to the broader class of deterministic score-based interventions.
Expected-Loss Framework: It provides a formal method for calibrating screening severity by weighing the consequences of retention and exclusion against the probability of latent states.
Public Choice Analysis: It explicitly separates decision-quality objectives from the organiser's institutional objectives, deriving the conditions under which blame avoidance leads to a divergence between the "best" rule for the decision and the "best" rule for the institution.
Operational Heuristic: It introduces PEx as a practical tool for translating theoretical screening regimes into transparent, pre-announced rules that are comparable across different institutional settings.

The paper concludes that stronger screening can be justified not only by improved decision quality but also by institutional protection. It argues that the "wedge" between decision quality and organiser preference reinforces the case for constitutional commitment: pre-announced, case-neutral rules constrain both evaluators (by making loss of influence credible) and organisers (by limiting post-hoc discretion), thereby managing the inherent trade-offs of the Galileo problem.

Technical Summary: The Galileo Problem in Expert Panels

More like this