📈 economics

Measuring Judgment Quality in Natural-Language Explanations: Evidence from Forecasting Tournaments

This paper introduces Explanation Quality Markers (EQMs), a scalable, LLM-based method for scoring sixty reasoning patterns in natural-language explanations that effectively predicts forecasting accuracy and outperforms traditional text-analysis techniques.

Original authors: Christopher W. Karvetski, Sheldon S. Huang, Simas Kučinskas, Nadja Flechner, Jingyu Hu, Philip Tetlock, Ezra Karger

Published 2026-07-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Christopher W. Karvetski, Sheldon S. Huang, Simas Kučinskas, Nadja Flechner, Jingyu Hu, Philip Tetlock, Ezra Karger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a manager trying to hire the best financial advisor. You have a stack of 55,000 applications. Each application has a specific prediction (e.g., "There is a 20% chance the stock will rise") and a written paragraph explaining why they think that.

Your problem? You can't read all 55,000 paragraphs to see who is actually good at guessing the future. You need a way to quickly scan the text and say, "This explanation looks smart," or "This one looks like a guess."

This paper introduces a new tool called Explanation Quality Markers (EQMs) to solve that problem. Here is how it works, using simple analogies.

The Old Way: Counting Words and Checking Dictionaries

Before this study, researchers tried to judge these explanations using "pre-LLM" tools. Think of these like a spell-checker or a word counter.

They would count how many words were in the paragraph.
They would look for specific "smart-sounding" words (like "therefore" or "however").
They would check if the text was complex.

The Result: These tools were like trying to judge a chef's cooking skills by counting how many spices they used. It didn't really tell you if the food tasted good. The paper found these old methods had almost no connection to whether the person was actually accurate.

The New Way: The "Super-Reader" AI

The authors used a Large Language Model (an advanced AI, like GPT-4o) to act as a Super-Reader. Instead of just counting words, this AI was trained to look for 60 specific "reasoning patterns" that experts know are good or bad.

Think of the AI as a detective looking for clues in a suspect's story.

Good Clues (Green Flags): Does the person look at past data? Do they admit they might be wrong? Do they break a big problem into small, manageable pieces?
Bad Clues (Red Flags): Is the person just guessing? Are they overconfident ("It will definitely happen!")? Are they ignoring evidence that contradicts their view?

The AI reads the explanation and gives it a score based on how many "Green Flags" and "Red Flags" it finds.

The Big Discovery: The "Trash Detector"

The researchers tested this AI against the actual results of the predictions (did the stock go up or down?). Here is what they found:

The AI is a Great "Trash Detector": The AI is incredibly good at spotting bad explanations. If an explanation is full of "Red Flags" (like guessing or overconfidence), the AI flags it, and that person is almost always wrong.
The AI is a Weak "Star Finder": The AI is less good at spotting the absolute best experts. Just because an explanation looks perfect doesn't guarantee the person is a genius.
- Analogy: Imagine a metal detector at a beach. It is amazing at finding the rusty nails and broken glass (the bad stuff). But it isn't very good at distinguishing between a shiny piece of copper and a real gold nugget (the very best stuff).

How It Compares to Humans

The researchers also asked actual humans to read these explanations and rate them.

Humans are easily fooled by length: Humans tended to give higher scores to longer explanations. They thought, "Wow, this person wrote a lot, so they must be smart."
The Reality: The length of the text had almost nothing to do with whether the prediction was correct.
The AI vs. Humans: The AI was much better at predicting who was accurate than the humans were. The humans were distracted by the "fluff" (length and fancy words), while the AI focused on the actual logic.

Why This Matters

This tool is useful because it requires no track record.

Usually, to know if a forecaster is good, you have to wait months or years to see their past results.
With EQMs, you can look at a single written explanation and get a good idea of whether that person is likely to be accurate or not.

Summary

The Problem: We have too many expert explanations to read, and we don't know which ones are trustworthy.
The Solution: An AI that scans for 60 specific reasoning habits (like checking for bias or using data).
The Result: The AI is much better than old computer methods and better than human readers at spotting bad reasoning. It helps decision-makers filter out the "noise" and the "guessers," even if it can't perfectly identify the "geniuses."

Note: The paper strictly tested this on geopolitical and economic forecasting tournaments. It does not claim this method works for medical diagnoses, legal judgments, or other fields yet; it only proves it works for predicting future events in these specific tournaments.

Technical Summary: Measuring Judgment Quality in Natural-Language Explanations

Problem Statement
Decision-makers routinely rely on expert judgments accompanied by written explanations (e.g., intelligence memos, investment theses) to assess which judgments to trust. However, the quality of these explanations is difficult to measure systematically at scale. While prior research has identified hallmarks of sound probabilistic reasoning—such as attention to base rates, openness to disconfirming evidence, and resistance to cognitive biases—measuring these features in unstructured text has historically required labor-intensive expert coding, coarse automated proxies, or tools limited to single theoretical constructs. Furthermore, in most domains, the lack of a clear "ground truth" for judgment accuracy makes it difficult to empirically link explanation quality to performance.

Methodology
The authors leverage forecasting tournaments as a natural testing ground where probabilistic judgments are paired with written rationales and scored against realized outcomes. The study utilizes data from the Aggregative Contingent Estimation (ACE) tournament (2011–2015), comprising over 55,000 forecast-rationale pairs, and validates findings on an independent dataset (Team Dynamics study).

The core methodological innovation is the introduction of Explanation Quality Markers (EQMs), a set of sixty theory-guided reasoning patterns scored by Large Language Models (LLMs).

Pattern Development: The 60 EQM patterns are drawn from judgment and decision-making literature and organized into six families: Bias and Emotion, Analytical Reasoning, Alignment and Discipline, Evidential Sourcing, Integrative Reasoning and Cognitive Framing, and Attitude and Uncertainty Management. Each pattern is defined in natural language with a directional hypothesis regarding its association with accuracy (positive, negative, or neutral).
Scoring Pipeline: An LLM (GPT-4o) scores each rationale on a three-point scale (0: no evidence; 1: some evidence; 2: clear evidence) for all 60 patterns simultaneously in a single API call to provide disambiguating context.
Composite Scoring: The 60 pattern scores are converted into a single composite quality score using LASSO regression, trained to predict realized accuracy.
Benchmarks: The EQM approach is compared against:
1. Pre-LLM methods: Including linguistic dictionaries (LIWC), readability measures, sentiment scores, and specific classifiers for "outside-view" reasoning (Comparison Class Usage).
2. Human Ratings: Usefulness ratings provided by peer forecasters in the ACE tournament.
3. Traditional Indicators: Prior accuracy, update size, and forecast frequency.

Key Results
The study is organized into four analyses:

Predictive Power vs. Pre-LLM Methods:
- EQM composite scores significantly outperform pre-LLM text-analysis methods in predicting accuracy at both the forecast level ( $r = .19$ vs. $r = .06$ ) and the forecaster level ( $r = .51$ vs. $r = .39$ ).
- Over 90% of statistically significant pattern-level correlations match the authors' directional hypotheses.
- The predictive signal is asymmetric: EQMs are more reliable at identifying likely underperformers (low-quality rationales) than at distinguishing the very best forecasters.
Comparison with Traditional Indicators:
- At the forecast level, EQMs are the strongest available predictor, outperforming even prior-year accuracy.
- At the forecaster level, prior-year accuracy remains the strongest predictor ( $r = .70$ ), but EQMs are competitive with behavioral indicators like update size ( $r = .55$ ) and significantly outperform pre-LLM methods.
- In crowd aggregation tasks, ranking forecasters by forecaster-level EQM scores improves aggregate accuracy, whereas ranking individual forecasts by forecast-level EQM scores offers little benefit beyond screening out the weakest judgments.
Comparison with Human Ratings:
- Human ratings of rationale quality are less consistently correlated with accuracy than EQMs (Forecast level: $r = .07$ vs. $r = .23$ ; Forecaster level: $r = .40$ vs. $r = .50$ ).
- Human raters place disproportionate weight on rationale length (square root of word count correlates with human ratings at $r = .62$ ), a feature that is essentially uncorrelated with forecast-level accuracy.
- Inter-rater reliability for human ratings is low (average Cohen's $\kappa = .24$ ), whereas agreement between different LLMs scoring the same patterns is substantially higher ( $\kappa = .56$ ).
Out-of-Sample Transfer:
- EQM models trained on ACE data transfer effectively to an independent dataset (Team Dynamics study) with different questions, forecasters, and elicitation formats (quantiles vs. probabilities).
- Out-of-sample correlations remain robust (Forecast level: $r = .16$ ; Forecaster level: $r = .46$ ), suggesting the framework captures generalizable reasoning patterns rather than memorized data.

Significance and Claims
The paper claims three primary contributions:

Methodological: It introduces EQMs as a scalable, theory-guided measurement framework that leverages LLMs to extract judgment-relevant information from written explanations, overcoming the limitations of pre-LLM NLP tools.
Empirical: It demonstrates that this LLM-based approach is more predictive of judgment accuracy than both traditional text-analysis methods and human ratings.
Practical: It establishes the utility of EQMs for screening judgments. The method requires only a single written rationale and no prior track record, making it applicable in single-shot elicitation settings where historical performance data is unavailable.

The authors note limitations, including that rationales are an imperfect proxy for internal reasoning (due to tacit knowledge or post-hoc justification) and that the findings are correlational. However, they suggest that the framework offers a portable tool for turning unstructured text into explicit, testable features of judgment quality across various domains.