Surrogate modeling for interpreting black-box LLMs in medical predictions

1. Problem Statement

Large Language Models (LLMs) encode vast amounts of real-world knowledge but operate as "black boxes," where their internal mechanisms and the specific extent of their encoded knowledge are inaccessible. This lack of interpretability poses significant risks, particularly in high-stakes domains like healthcare, where:

• Bias and Inaccuracy: LLMs may perpetuate societal biases (e.g., race-based medical assumptions) or hallucinate incorrect medical facts found in their training data.
• Inconsistency: Direct queries to LLMs often yield non-deterministic responses (inconsistency across runs) and "streaking" (clustering around specific values rather than continuous distributions).
• Lack of Quantification: Current methods struggle to quantitatively measure how an LLM perceives specific input variables in relation to an output, making it difficult to audit for safety or reliability.

2. Methodology: The Surrogate Modeling Framework

The authors propose a surrogate modeling framework to quantitatively explain LLM-encoded knowledge for specific hypotheses derived from domain knowledge. The framework approximates the complex LLM latent space using a simplified, interpretable statistical model based on observable input-output pairs.

The process consists of three stages:

• Stage 1: Simulated Dataset Generation
  • Hypothesis-Driven: A target hypothesis is defined based on domain knowledge (e.g., "How do age and creatinine affect GFR?").
  • Variable Selection: Relevant input variables (risk factors) are selected.
  • Distribution & Sampling: Probability distributions (e.g., lognormal for continuous physiological variables) are assigned to reflect a broad range of physiological states. A large simulated dataset (e.g., 20,000 samples) is generated via random sampling.
• Stage 2: Extensive Prompting
  • The simulated data is converted into natural language prompts.
  • The LLM is queried extensively (e.g., 10 responses per sample) to generate a massive set of input-output pairs.
  • Zero-shot setting: No fine-tuning is performed; the LLM is used as-is.
• Stage 3: Statistical Modeling
  • The input variables and the LLM's predicted outputs are used to train a simple, interpretable statistical model (e.g., linear regression or log-transformed linear regression).
  • The resulting surrogate model (a statistical formula) serves as a proxy for the LLM, quantifying the relationship between inputs and outputs.
  • Analysis: Coefficients are analyzed to determine the directionality and magnitude of the LLM's "perception" of each variable. SHAP (SHapley Additive exPlanations) values are used for local interpretability.

3. Key Contributions

• Quantitative Interpretability: Moves beyond qualitative "prompt engineering" to provide a mathematical formula that explicitly defines how an LLM weights specific medical variables.
• Bias Detection: Demonstrates a method to systematically uncover and quantify hidden biases (specifically race-based biases) that persist in LLMs despite being scientifically refuted in modern guidelines.
• Stability and Efficiency: Shows that surrogate models resolve the "streaking" and inconsistency issues of direct LLM prompting, offering deterministic outputs and inference speeds 900–2,100 times faster than API-based LLM calls.
• Open Tooling: The authors released an online tool (llm-surrogate.com) and code repository to allow other researchers to apply this framework to their own hypotheses.

4. Key Results

The framework was validated through four proof-of-concept experiments in medical predictions using GPT-4, Claude-3.5-Sonnet, and Gemini-1.5-Pro:

• Experiment 1: Cardiovascular Disease (CVD) Risk Prediction

  • Performance: The surrogate models outperformed traditional tools (Pooled Cohort Equations and Framingham Risk Score) in predicting Major Adverse Cardiovascular Events (MACE) on UK Biobank data (C-index ~0.736 vs. 0.727 for PCE).
  • Inconsistency: Direct LLM queries showed clustering ("streaking"), while surrogate models produced smooth, continuous distributions.
  • Variable Perception: While most coefficients aligned with medical knowledge, discrepancies were found. For example, one model showed a statistically insignificant negative association between waist-hip ratio and CVD risk, contradicting established medical consensus.

• Experiment 2: Glomerular Filtration Rate (GFR) Estimation

  • Bias Quantification: The study quantified the persistence of race-based bias.
    • GPT-4: Estimated GFR for Black individuals was 15.3% higher than for White individuals (coefficient 1.153).
    • Claude-3.5-Sonnet: Showed even stronger bias, estimating GFR 22.7% higher for Black individuals (coefficient 1.227), exceeding the bias in the now-refuted 2009 CKD-EPI equation.
    • Gemini-1.5-Pro: Showed minimal bias (coefficient 1.008).
  • This confirms that even advanced LLMs encode and perpetuate scientifically discredited racial adjustments.

• Experiments 3 & 4: Stroke Risk and Pulmonary Function

  • Stroke: Surrogate models achieved performance comparable to the Framingham Stroke Risk Profile.
  • Pulmonary Function: Similar to GFR, the models revealed significant race-based biases in estimating lung capacity (FVC, FEV1), with Black individuals consistently estimated to have lower lung function than White individuals, despite guidelines rejecting such adjustments.

5. Significance and Implications

• Red-Flag Indicator: The framework acts as a diagnostic tool to "red-flag" LLMs before deployment. If a surrogate model reveals coefficients that contradict medical consensus or exhibit harmful biases, the LLM can be flagged for mitigation (e.g., targeted reinforcement learning) or restricted use.
• Safety in Healthcare: By making the "black box" transparent, this approach supports the safe integration of LLMs into clinical decision-making, ensuring that models do not inadvertently exacerbate health disparities.
• Generalizability: The method is not limited to medical tasks; it can be applied to any domain where LLM knowledge needs to be audited against specific hypotheses.
• Efficiency: The drastic reduction in inference time and cost makes it feasible to deploy these models in resource-constrained hospital environments.

In conclusion, the paper establishes that surrogate modeling is a robust, scalable, and necessary technique for auditing, interpreting, and safely deploying Large Language Models in critical domains like medicine.