Compared to What? Baselines and Metrics for Counterfactual Prompting

1. Problem Statement

Counterfactual prompting is a standard method for evaluating Large Language Models (LLMs), where researchers surgically edit an input (e.g., swapping "male" to "female") to isolate the effect of a specific factor (e.g., gender bias) on the model's output. The core assumption is that any observed change in the output is causally attributable to the targeted semantic change.

The authors argue this assumption is flawed because it violates the causal inference principle of Treatment Variation Irrelevance.

• The Issue: Every counterfactual edit is a "compound treatment." It bundles the variable of interest (e.g., gender) with incidental surface-form variations (e.g., specific word choices, token counts, phrasing).
• The Consequence: LLMs are known to be sensitive to arbitrary phrasing and token changes (prompt sensitivity). Therefore, an observed output flip might be caused by the change in text structure rather than the semantic meaning of the perturbation.
• Current Gap: Prior work often interprets raw perturbation effects (e.g., a 15% answer flip rate) as evidence of bias or lack of faithfulness without comparing them to a "null" distribution of changes induced by benign, meaning-preserving modifications.

2. Methodology

The authors propose a rigorous framework to distinguish between targeted effects (caused by the variable of interest) and general model sensitivity (caused by text modification noise).

A. Experimental Design

1. Targeted Perturbation ($D_{targeted}$): The standard intervention (e.g., gender swap, style change).
2. Baseline Perturbation ($D_{benign}$): A control condition designed to mimic the surface-level changes of the target without altering the semantic variable.
   • Token-Adjusted Paraphrase: The primary baseline. An LLM paraphrases the input to preserve semantics but changes a fraction of tokens matching the token edit percentage of the targeted perturbation. This controls for the finding that model sensitivity scales with the number of tokens changed.
   • Fixed Sentence: A secondary baseline where a topically irrelevant sentence is prepended to the prompt.
3. Statistical Testing: Instead of comparing raw metrics, the authors use statistical tests to determine if the divergence in the targeted condition significantly exceeds the divergence in the baseline condition.
   • Per-sample metrics: Paired t-tests on differences between targeted and baseline divergence values.
   • Aggregate metrics: Non-parametric bootstrap tests on the difference of aggregate statistics.
   • Regression: Used for directional hypotheses to estimate the magnitude and sign of the systematic effect.

B. Metrics Evaluated

The paper evaluates three classes of metrics to quantify output divergence:

1. Aggregate Metrics:
   • Flip Rate: Fraction of instances where the categorical answer changes.
   • Mutual Information (MI): Measures consistency of responses across conditions.
   • Phi Coefficient ($\phi$): Distinguishes agreement from systematic disagreement.
   • Limitation: These are low-power and can be degenerate under class imbalance.
2. Per-Sample Distributional Metrics:
   • Kullback-Leibler (KL) Divergence: Measures the shift in the probability distribution over answer options.
   • Jensen-Shannon Divergence (JSD): A symmetric, bounded variant of KL.
   • Advantage: These capture subtle distributional shifts even when the final decision boundary is not crossed.
3. Regression Coefficients:
   • Estimates the directional shift ($\beta$) when perturbations have a signed encoding (e.g., +1 for female, -1 for male). This characterizes both the magnitude and direction of the bias.

3. Key Contributions

1. Methodological Correction: Demonstrates that without a magnitude-matched baseline, counterfactual results are uninterpretable. Observed effects often reflect general prompt sensitivity rather than specific factor sensitivity.
2. Framework Proposal: Introduces a statistical testing framework comparing targeted perturbations against token-matched paraphrase baselines.
3. Metric Analysis: Empirically proves that per-sample distributional metrics (JSD, KL) and regression are significantly more powerful than aggregate metrics (Flip Rate, MI, $\phi$), especially in imbalanced datasets.
4. Open Source: Releases cfprompt, a Python package implementing this framework.

4. Key Results

A. Revisiting MedPerturb (Medical Bias)

• Original Claim: A prior study claimed LLMs are highly sensitive to patient gender and style cues in medical triage.
• New Finding: When compared against token-matched paraphrase baselines:
  • Gender Perturbations: The observed effects (e.g., 14.9% flip rate for gender swap) were statistically indistinguishable from the baseline paraphrase effects (14.1%).
  • Conclusion: Out of 120 tests, only 5 reached significance (all related to "colorful" style perturbations on a smaller model). The claim that models are specifically sensitive to patient gender is largely unsupported; the effects were likely noise from general text sensitivity.

B. Sanity Checks

• Recall (Positive Control): When the task was explicitly changed to "Is this patient male?", the framework correctly detected a massive, significant effect, proving it is not overly conservative.
• Precision (Negative Control): Changing a patient's age by one year (a minimal perturbation) produced no significant effect compared to the baseline, confirming the framework's specificity.

C. Bias-in-Bios (Occupational Bias)

• Context: Testing if models associate "Professor" with men and "Nurse" with women.
• Findings:
  • Aggregate Metrics: Showed small, barely significant flip rates (e.g., +0.7%), leading to a potential conclusion of "no bias."
  • Per-Sample Metrics (JSD/KL): Detected statistically significant distributional shifts ($p \approx 10^{-7}$).
  • Regression: Revealed a clear directional bias ($\hat{\beta} \approx -0.5$), indicating that swapping a biography to female systematically decreases the log-odds of the "Professor" label.
  • Insight: The bias exists but operates below the decision boundary (logits shift without flipping the binary answer). Only regression and distributional metrics could characterize this.

D. Q-Pain Replication (Race Bias)

• Replicating a study on racial disparities in pain management, the authors found that 4 out of 5 significant demographic effects had corresponding significant baseline effects (paraphrasing). This suggests the original findings of racial bias may have been confounded by general text sensitivity.

5. Significance and Practical Guidance

This paper fundamentally shifts how counterfactual prompting should be conducted in LLM evaluation.

• The "Null" Baseline is Mandatory: Researchers must include a baseline perturbation (specifically, a token-matched paraphrase) to establish a null distribution. Without this, "bias" findings may be artifacts of prompt sensitivity.
• Metric Selection Matters:
  • Avoid relying solely on Flip Rate or MI, as they lack power and fail under class imbalance.
  • Use JSD/KL to detect distributional shifts.
  • Use Regression to quantify the direction and magnitude of effects.
• Token Matching: Baselines must match the number of tokens changed by the target perturbation, as model sensitivity scales with edit magnitude.
• Implication: Many previous claims of LLM bias or unfaithfulness may be overstated or misattributed. Real, subtle biases exist (as shown in Bias-in-Bios) but require more sophisticated statistical tools to detect.

In summary, the paper argues that counterfactual prompting is not a "plug-and-play" diagnostic; it requires rigorous statistical controls to separate signal (true factor sensitivity) from noise (general prompt sensitivity).