Measuring Faithfulness Depends on How You Measure: Classifier Sensitivity in LLM Chain-of-Thought Evaluation

1. Problem Statement

The field of Large Language Model (LLM) evaluation increasingly treats Chain-of-Thought (CoT) faithfulness (the degree to which a model's reasoning process genuinely leads to its final answer, rather than being influenced by external hints or biases) as an objective, measurable property. Recent studies often report single aggregate numbers (e.g., "Model X is 39% faithful") to compare architectures.

The Core Issue: This paper argues that faithfulness is not an intrinsic, objective property of a model but is highly sensitive to the classification methodology used to measure it. Different classifiers operationalize the concept of "faithfulness" differently (e.g., lexical mention vs. epistemic dependence), leading to vastly different results on the exact same data. Consequently, comparing faithfulness scores across studies using different classifiers is scientifically invalid.

2. Methodology

The authors conducted a controlled experiment to quantify the variance introduced by different classifiers.

• Dataset:

  • Source: 10,276 "influenced" reasoning traces extracted from a larger study involving 12 open-weight reasoning models (ranging from 7B to 1T parameters, including DeepSeek-R1, Qwen, OLMo, etc.).
  • Condition: These are cases where the model changed its answer from a baseline to match a specific injected hint.
  • Hint Types: Five categories based on Chen et al. [1]: Sycophancy (agreeing with an authority), Consistency (matching previous answers), Metadata (using hidden data), Grader (adhering to grading rules), and Unethical (ignoring safety constraints).

• Classifiers Compared:
  Three distinct systems were applied to the identical 10,276 cases:

  1. Regex-Only Detector: A surface-level lexical matcher using 38 regular expressions. It flags a case as "faithful" only if the hint is explicitly mentioned in the text.
  2. Regex + Ollama Pipeline (Two-Stage): Stage 1 uses the regex. Cases not caught by regex are passed to a local panel of three LLMs (GLM-5, Kimi K2, Gemini 3 Flash) for majority-vote classification. This aims to catch implicit references.
  3. Independent Claude Sonnet 4 Judge: A single, high-cost API-based judge (Anthropic) applying a strict "LLM-as-Judge" paradigm. It assesses epistemic dependence: Did the hint serve as a causal, load-bearing reason for the answer, or was it merely mentioned and then overridden by independent reasoning?

• Metrics:

  • Faithfulness Rate (percentage of cases classified as faithful).
  • Cohen's $\kappa$ (inter-classifier agreement correcting for chance).
  • McNemar's Test (statistical significance of disagreement on paired data).
  • Spearman Rank Correlation ($\rho$) to measure stability in model rankings.

3. Key Results

A. Massive Aggregate Disagreement

On the same dataset, the three classifiers produced significantly different overall faithfulness rates:

• Regex-Only: 74.4%
• Pipeline (Regex + LLM): 82.6%
• Claude Sonnet 4: 69.7%
• Gap: A 12.9 percentage point difference between the highest and lowest overall rates.

B. Hint-Type Specific Variance

The disagreement is not uniform; it depends heavily on the type of hint:

• Sycophancy Hints: The largest gap (43.4 percentage points). Regex/Pipeline rate: ~97%; Sonnet rate: ~54%. Cohen's $\kappa$ is 0.06 ("slight" agreement).
• Grader Hints: The smallest gap (2.9 percentage points). Cohen's $\kappa$ is 0.42 ("moderate" agreement).
• Asymmetry: For sycophancy, the Pipeline classified 883 cases as "faithful" while Sonnet classified them as "unfaithful." Only 2 cases went the other way. This suggests the Pipeline often catches "mentions" that Sonnet correctly identifies as non-causal.

C. Ranking Reversals

Classifier choice can completely reverse the perceived performance of models:

• Qwen3.5-27B: Ranked 1st by the Pipeline (98.9% faithful) but 7th by Sonnet (68.3%).
• OLMo-3.1-32B: Ranked 9th by the Pipeline (71.9%) but 3rd by Sonnet (81.0%).
• Correlation: The Spearman rank correlation between the two main classifiers is $\rho = 0.67$ ($p=0.017$), indicating moderate but imperfect agreement.

D. The "Load-Bearing" vs. "Mention" Crux

Qualitative analysis reveals the root cause of disagreement:

• Regex/Pipeline: Operationalizes faithfulness as Mention. If the text says "The professor suggested X," it counts as faithful.
• Sonnet Judge: Operationalizes faithfulness as Epistemic Dependence. If the model says "The professor suggested X, but after my own analysis, I chose Y," it counts as unfaithful because the hint was not the reason for the final answer.
• Conclusion: The classifiers are measuring different constructs, not just different noise levels.

4. Key Contributions

1. Empirical Demonstration of Sensitivity: The paper provides the first systematic quantification of how classifier choice alters CoT faithfulness metrics, showing gaps up to 30.6 percentage points for individual models.
2. Construct Validity Failure: It demonstrates that "faithfulness" is a family of constructs. A single number cannot capture the complexity of whether a model is "faithful" without specifying the operational definition (mention vs. dependence).
3. Methodological Framework: It establishes that future evaluations must report sensitivity ranges rather than single point estimates.
4. Cost-Accuracy Analysis: It highlights that the most expensive classifier (Sonnet) and the cheapest (Regex) are actually closer in agreement than the mid-tier "Pipeline" approach, which tends to over-classify faithfulness by catching non-load-bearing mentions.

5. Significance and Recommendations

Significance:
The findings undermine the validity of current literature that compares faithfulness scores across papers using different evaluation tools. A model ranked "best" in one study might be "average" in another solely due to the classifier used. This mirrors issues in toxicity detection and other subjective NLP tasks where annotator disagreement is often ignored.

Recommendations for the Field:

1. Report Methodology Explicitly: Papers must disclose the full classification prompt and logic, not just the final number.
2. Report Ranges: Instead of "Model X is 80% faithful," report "Model X is 69.7%–82.6% faithful depending on classifier stringency."
3. Cross-Classifier Validation: Standard practice should involve running at least two classifiers (e.g., a surface-level and a semantic-level) to identify sensitivity.
4. Avoid Direct Cross-Paper Comparison: Faithfulness numbers from studies using different classifiers should not be compared directly without sensitivity analysis.

Future Directions:
The authors suggest moving toward classifier-independent signals, such as analyzing internal "thinking tokens" (hidden reasoning before the visible CoT) or using causal interventions, to bypass the ambiguity of textual interpretation.