Technical Summary: Investigating the Interplay between Contextual and Parametric Chain-of-Thought Faithfulness under Optimization

Problem Statement

Chain-of-Thought (CoT) reasoning has proven effective for improving Large Language Model (LLM) performance, yet a critical concern remains: CoT faithfulness. This refers to whether the generated reasoning steps genuinely reflect the model's underlying computational process or merely serve as plausible post-hoc rationalizations.

Current evaluation of CoT faithfulness is fragmented into two disjoint paradigms:

1. Contextual Faithfulness: Assessed by perturbing the input or the reasoning trace (e.g., inserting errors, paraphrasing, or injecting hints) and observing if the model's answer changes accordingly. This measures consistency between the reasoning trace and the output but ignores the model's internal parametric knowledge.
2. Parametric Faithfulness: Assessed by intervening on the model's parametric knowledge (e.g., unlearning the knowledge underlying a specific reasoning step) to see if the answer changes. This targets internal mechanisms but is limited to knowledge explicitly verbalized in the CoT, potentially missing unexpressed factors shaping generation.

Prior work has largely compared these paradigms descriptively on fixed CoTs, leaving a gap in understanding how they interact when models are explicitly optimized for faithfulness. It remains unclear whether optimizing for one paradigm yields gains in the other, or if these paradigms reward conflicting properties.

Methodology: FaithMATE Framework

To address this, the authors introduce FaithMATE (Faithfulness Metric Alignment and Transfer Evaluation), a unified preference-alignment interface designed to optimize models toward either contextual or parametric faithfulness and evaluate the transfer of these gains.

The framework operates in four key steps:

1. CoT Sampling: For each input, the base model samples a set of $m$ CoTs.
2. Faithfulness Evaluation: Each sampled CoT is scored using a suite of metrics ($\mathcal{F}$). The suite includes:
   • Contextual Metrics: Corruption-based (Early Answering, Adding Mistake, Paraphrasing, Filler Token) and Attribution-based (CC-SHAP).
   • Parametric Metrics: Binary ($f_{hard}$) and Continuous ($f_{cont}$) measures based on unlearning reasoning steps.
3. Preference Pair Construction: For a chosen training metric ($F_{train}$), the most faithful CoT is selected as the "chosen" response ($y_w$) and the least faithful as the "rejected" response ($y_l$).
4. Optimization: The model is optimized using either Supervised Fine-Tuning (SFT) on the chosen CoTs or Direct Preference Optimization (DPO) using the constructed pairs.

The core investigation involves Faithfulness Gain Transfer ($\delta$): After optimizing for a specific metric $f_i$, the model is evaluated on all other metrics ($f_j$) to measure $\delta(f_i, f_j) = f_j(c') - f_j(c)$, where $c'$ is the output of the optimized model and $c$ is the base model.

Key Contributions

1. Unified Optimization Interface: FaithMATE provides the first framework to systematically optimize LLMs toward both contextual and parametric faithfulness objectives, enabling controlled study of their interplay.
2. Asymmetric Coupling Discovery: The study reveals that while the two paradigms are positively coupled, they are asymmetric. Optimizing for parametric faithfulness yields consistent gains across both paradigms. In contrast, optimizing for contextual faithfulness yields variable gains; improvements on one contextual metric do not consistently transfer to others.
3. Internal Inconsistency of Contextual Metrics: The authors demonstrate that contextual faithfulness metrics capture disjoint facets of faithfulness. For instance, optimizing for "Adding Mistake" can sometimes degrade performance on "Paraphrasing," suggesting these metrics are not interchangeable proxies for a single latent quantity.
4. Mechanistic Differentiation: Qualitative analysis shows that contextual gains primarily stem from improved reasoning-answer consistency and the mitigation of post-hoc rationalization. Parametric gains, conversely, arise from more explicit grounding in task-relevant facts and knowledge.
5. Model Merging Strategy: The paper explores task-arithmetic merging of models optimized for different contextual metrics. It finds that merging models optimized for "Early Answering" and "Filler Token" amplifies faithfulness gains, whereas including "Paraphrasing" often dilutes gains, likely due to opposing parameter update directions.

Experimental Results

Experiments were conducted on three models (Gemma3-4B, Qwen2.5-7B, Llama3.1-8B) across two datasets (OpenBookQA, LogiQA) using six faithfulness metrics.

• Cross-Paradigm Transfer:
  • Parametric $\Rightarrow$ Contextual: Highly consistent. Optimizing for parametric faithfulness improved contextual faithfulness in 28/30 cases on OpenBookQA and 24/25 on LogiQA.
  • Contextual $\Rightarrow$ Parametric: Positive but highly variable. While 32/48 cases showed positive transfer, the magnitude varied significantly depending on the specific contextual metric (e.g., "Filler Token" transferred well, while "Adding Mistake" transferred poorly).
• Within-Paradigm Transfer (Contextual):
  • Gains did not consistently transfer between different contextual metrics. For example, optimizing for "Paraphrasing" did not reliably improve "Adding Mistake" scores, and in some cases, the two metrics were in tension.
• Training Paradigms:
  • DPO consistently outperformed SFT in generating faithfulness gains, suggesting that contrastive signals (distinguishing high vs. low faithfulness) are more effective than simple likelihood maximization.
• Task Performance:
  • Faithfulness optimization largely preserved or improved overall task accuracy, with no significant degradation observed.

Significance and Claims

The paper claims that CoT faithfulness is not a monolithic objective. The findings imply that:

• Parametric faithfulness is a more reliable target for broad generalization, as it consistently improves both parametric and contextual measures.
• Contextual faithfulness is a family of related but distinct properties. Relying on a single contextual metric for evaluation or optimization is insufficient and potentially misleading, as improvements in one area may not reflect improvements in others.
• Multifaceted Optimization is necessary. Since different metrics reward different underlying behaviors (e.g., reasoning consistency vs. factual grounding), a comprehensive approach to improving CoT faithfulness requires addressing these distinct facets, potentially through techniques like model merging or multi-objective optimization.

The authors conclude that FaithMATE serves as a foundation for developing faithfulness objectives that are both parametrically grounded and contextually robust, enabling more precisely scoped claims about the validity of individual faithfulness metrics.