💬 NLP

EvasionBench: A Large-Scale Benchmark for Detecting Managerial Evasion in Earnings Call Q&A

This paper introduces EvasionBench, a large-scale benchmark and taxonomy for detecting managerial evasion in earnings calls, featuring a rigorously annotated dataset and a specialized 4B-parameter classifier that outperforms leading commercial models.

Original authors: Shijian Ma, Yan Lin, Yi Yang

Published 2026-02-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shijian Ma, Yan Lin, Yi Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are sitting in a room with a CEO, asking them tough questions about their company's money. Sometimes, they answer you straight up. Other times, they might say, "That's a great question, and we are looking at many opportunities," without actually giving you a number or a clear "yes" or "no." They are dancing around the question.

This paper introduces EvasionBench, a new tool designed to catch those dance moves. It's like a "lie detector" for corporate meetings, but instead of detecting lies, it detects evasion—when someone answers a question without really answering it.

Here is the breakdown of how they built it and what they found, using simple analogies:

1. The Problem: The "Dodgy" Answer

In the world of politics or law, we know people dodge questions. In the stock market, when a CEO dodges a question about why profits are down, it can be a warning sign for investors. But until now, computers weren't very good at spotting this. Most AI tools are great at telling if a sentence is happy or sad (sentiment), but they struggle to tell if a sentence actually answers the question asked.

2. The Solution: A Massive Library of "Dodges"

The researchers went digging through a massive digital library (S&P Capital IQ) containing 22.7 million question-and-answer pairs from earnings calls. That's like reading every transcript from thousands of years of corporate meetings.

From this mountain of data, they built a three-level "Evasion Scale":

Direct: The CEO gives you the exact number or a clear "Yes/No." (The straight arrow).
Intermediate: The CEO talks about the topic but avoids the specific number or hard truth. They use "hedge words" like "maybe," "we think," or "it's possible." (The foggy mirror).
Fully Evasive: The CEO completely ignores the question, says "we can't say," or changes the subject entirely. (The red herring).

3. The Annotation Challenge: How to Teach the AI

Labeling these answers is hard because "dodging" is subjective. If you ask one human expert, they might say an answer is "Direct." Ask another, and they might say "Intermediate."

To solve this, the researchers didn't just ask one AI or one human. They built a "Multi-Model Consensus" (MMC) system. Think of it like a jury:

The Witnesses: They used two super-smart AI models (Claude Opus and Gemini) to label the answers first.
The Jury: If the two AIs disagreed, they didn't just pick one. They brought in a third AI (GPT-5.2) to act as a judge.
The Verdict: The final label was decided by a majority vote (2 out of 3).

This "jury system" was crucial. The paper found that if you only used one AI, it had a bias (like a judge who always thinks everyone is guilty). By using a jury, they got a much more reliable dataset. They even checked this against human experts and found a very high agreement rate (83.5%), proving their "AI Jury" was doing a great job.

4. The Result: "Eva-4B"

Using this high-quality, jury-approved data, they trained a new AI model called Eva-4B.

The Size: It's a "4-billion parameter" model. In AI terms, that's like a very smart student who is smaller and faster than the massive "super-heroes" (like GPT-5 or Claude Opus) but has been specifically trained on this exact problem.
The Performance: Eva-4B achieved an 84.9% accuracy score.
The Surprise: It actually beat the massive, expensive, top-tier AI models (like Claude Opus 4.5 and GPT-5.2) at this specific task.

5. Why It Matters (According to the Paper)

The paper shows that how you label data matters more than just using the biggest AI.

When they trained the model using just one AI's labels, it struggled and the training process was "noisy" (like trying to learn a language from someone who speaks with a heavy accent).
When they used the "Jury" (Multi-Model Consensus) labels, the model learned perfectly and converged quickly.

The Bottom Line

The researchers created the first large-scale "gym" (benchmark) for teaching computers to spot when a CEO is dodging a question. They proved that by using a "jury" of AI models to label the data, they could build a specialized, smaller AI that is better at catching evasive answers than the giant, general-purpose AI models currently available.

What the paper doesn't claim:

It does not say this AI can predict stock prices on its own (though it notes that evasion correlates with bad stock performance in other studies).
It does not claim this works for political interviews or legal trials yet, though they hope it might in the future.
It does not say this should be used as legal evidence in court.

The paper is strictly about building the data, the method for labeling it, and the model that detects the evasion.

Technical Summary: EvasionBench

Problem Definition

The paper addresses the critical gap in Natural Language Processing (NLP) regarding the detection of managerial evasion in corporate earnings call question-and-answer (Q&A) sessions. While phenomena like sentiment analysis and factual QA have established benchmarks, evasion detection remains underexplored due to the subjective nature of "evasiveness" and the prohibitive cost of expert annotation at scale.

The core task is defined as classifying a management response ( $a$ ) to an analyst question ( $q$ ) into one of three ordinal levels:

Direct: The response explicitly and completely addresses the question core (e.g., specific figures, clear stances).
Intermediate: The response provides related context but sidesteps the specific ask (e.g., hedging, conditional framing, answering adjacent topics).
Fully Evasive: The question is ignored, explicitly refused, or the response is entirely off-topic.

This classification is grounded in Gricean pragmatics, specifically measuring the degree of violation of the Maxim of Relation.

Methodology

1. Data Collection and Filtering

The authors constructed a dataset from 22.7 million Q&A pairs extracted from S&P Capital IQ transcripts spanning 2002–2022. A rigorous three-stage filtering pipeline reduced this to 11.27 million high-quality pairs:

Stage 1: Extraction of sequential analyst questions (Type 3) and management answers (Type 4), removing operator instructions.
Stage 2: Quality checks requiring questions to contain "?", answers to exceed 30 characters, and the exclusion of transcription markers.
Stage 3: Selection of substantial content where the combined Q&A length is $\ge$ 500 characters.

2. Multi-Model Consensus (MMC) Annotation Framework

To overcome the cost of human annotation and mitigate single-model bias, the paper introduces a Multi-Model Consensus (MMC) framework:

Stage I (Dual LLM Annotation): Two frontier Large Language Models (LLMs)—Claude Opus 4.5 and Gemini 3 Flash—independently label samples.
Stage II (Consensus Detection): Samples where both models agree form the "consensus set."
Stage III (Three-Judge Arbitration): For the ~16% of samples where models disagree, a third judge (GPT-5.2) joins the original two. A three-judge majority voting mechanism determines the final label.
Anti-Bias Mechanism: To prevent position bias (primacy effect), the order of model predictions presented to the judges is randomized.

3. Model Development: Eva-4B

The authors fine-tuned Qwen3-4B-Instruct (a 4-billion parameter model) using a two-stage training pipeline:

Stage 1: Full fine-tuning on 60,000 consensus samples (where the two initial LLMs agreed).
Stage 2: Continued fine-tuning on 24,000 samples, incorporating the arbitrated disagreement cases resolved by the three-judge majority.
Ablation Variants: The study compares the full model against variants trained only on consensus data or using single-model (Opus) labels for the second stage.

Key Contributions

EvasionBench Dataset: The first large-scale benchmark for evasion detection, comprising:
- 84,000 balanced training samples (33.3% per class) derived from 8,081 unique companies.
- 1,000 gold-standard evaluation samples with expert human validation.
MMC Annotation Framework: A scalable annotation pipeline that combines dual frontier LLM labeling with three-judge arbitration, achieving a Cohen's Kappa of 0.835 against human annotators.
Eva-4B Model: A specialized 4B-parameter classifier that achieves 84.9% Macro-F1, outperforming frontier closed-source models (Claude Opus 4.5, GPT-5.2, Gemini 3 Flash) on this specific task.
Empirical Validation of Consensus: Ablation studies demonstrating that multi-model consensus labeling significantly outperforms single-model annotation, with a +4.3 percentage point (pp) Macro-F1 improvement over single-model labeling and an 80x reduction in training loss compared to single-model baselines.

Results

Performance: Eva-4B (Full) achieved the highest Macro-F1 (84.9%) on the Gold 1K set, surpassing the best closed-source model (Gemini 3 Flash at 84.6%).
Class Difficulty: All models struggled most with the Intermediate class (F1: 75.0–80.1%), which represents subtle evasion. The Fully Evasive class was the easiest to detect (F1: 91.0–92.4%).
Error Analysis: 95.4% of errors occurred between adjacent classes (e.g., Direct $\to$ Intermediate). The primary error mode was the over-prediction of evasion in responses containing hedging language (e.g., "we do expect... soon") that were actually direct.
Training Dynamics: The model trained with three-judge consensus labels converged to a final loss of 0.007, whereas the model trained with single-model labels plateaued at 0.56, indicating that consensus labels provide significantly cleaner training signals.

Significance and Claims

The paper claims to fill a critical gap in financial NLP by providing the first large-scale, systematically annotated benchmark specifically targeting managerial communication evasion.

Scalability: The MMC framework offers a viable solution to the "prohibitive cost" of expert annotation, demonstrating that high-quality labels can be generated through consensus among frontier LLMs rather than relying solely on expensive human annotators.
Bias Mitigation: The study provides empirical evidence that single-model annotation introduces systematic biases (e.g., Opus favoring "Direct" labels, Gemini favoring "Fully Evasive"), validating the necessity of multi-model consensus and randomized presentation orders.
Practical Utility: The resulting dataset and model enable automated screening for investor relations, regulatory monitoring of disclosure quality, and academic research on strategic communication.

The authors maintain modesty regarding generalization, noting that the dataset is domain-specific to earnings calls and that cross-domain validation (e.g., to political interviews or legal depositions) remains future work. They also acknowledge limitations in temporal coverage (2002–2022) and the reliance on English-language transcripts.