In-Context Examples Suppress Scientific Knowledge Recall in LLMs

1. Problem Statement

The paper investigates a counter-intuitive phenomenon in Large Language Models (LLMs) applied to scientific reasoning: In-Context Learning (ICL) can suppress, rather than reinforce, a model's pre-trained scientific knowledge.

While ICL is widely used to improve performance by providing examples, the authors hypothesize that for tasks requiring latent structure recovery (inferring hidden parameters from data before applying a domain law), adding examples shifts the model's reasoning strategy. Instead of retrieving and applying the correct scientific formula from its pre-trained weights (Knowledge-Driven Derivation), the model shifts toward Example-Driven Fitting (approximating input-output mappings via pattern recognition).

Crucially, this suppression occurs even when the in-context examples are generated by the exact same scientific formula the model already knows, meaning there is no conflict between the examples and the model's prior knowledge.

2. Methodology

Benchmark Construction

The authors constructed a difficulty-balanced benchmark consisting of 60 latent structure recovery tasks across five scientific domains:

• Domains: Biology, Chemistry, Economics, Physics, and Geoscience (12 tasks per domain).
• Task Structure: Each task follows a specific architecture: $X \to \Theta \to Y$.
  • $X$ (Observables): Raw data provided to the model.
  • $\Theta$ (Latent Variables): Hidden parameters (e.g., reaction constants, elastic moduli) that must be inferred using domain laws.
  • $Y$ (Target): The final quantity to be computed using the inferred parameters.
• Complexity: Tasks vary in the number of observables, latent variables, and equation formats (linear, non-linear, fractional, etc.) to ensure findings are not artifacts of specific equation types.

Experimental Setup

• Models: Experiments were conducted on four models: GPT-5.2 (primary), Claude Sonnet 4.6, Llama 4 Maverick, and DeepSeek V3.2.
• Conditions:
  • Zero-Shot: Model solves the task using only Chain-of-Thought (CoT) reasoning without examples.
  • 10-Shot: Model is provided with 10 input-output pairs generated by the same underlying formula as the test case (no intermediate derivations provided).
• Strategy Classification: An automated classifier (validated by human annotators with 98% agreement) categorizes the model's CoT trajectory into three modes:
  1. Parametric ($S_{know}$): Retrieves a named formula and derives the answer algebraically.
  2. Empirical ($S_{emp}$): Fits a pattern to the examples without invoking domain laws.
  3. Hybrid: Uses a formula but calibrates coefficients based on examples.

Additional Analyses

• Semantic Decoy: Replacing domain-specific vocabulary (e.g., "Consumer Surplus" $\to$ "Gamma Rad") to test if knowledge recall relies on surface cues.
• Internal Representations: Analyzed hidden states of open-source models (Qwen, Mistral, Llama) to observe shifts in representation space.
• Interventions: Tested structural prompting techniques (e.g., forcing formula declaration, negative prompting) to see if displacement can be mitigated.

3. Key Contributions

1. Discovery of Knowledge Displacement: The authors demonstrate that adding consistent in-context examples consistently shifts computation away from knowledge-driven derivation ($S_{know}$) toward example-driven fitting ($S_{emp}$) across all domains.
2. Strategy Competition Framework: They propose a framework explaining why accuracy effects vary. The shift is uniform, but the consequence depends on the quality gap between the displaced strategy (pre-trained knowledge) and the replacement strategy (empirical fitting).
3. Comprehensive Benchmark: A new 60-task benchmark designed specifically to make the competition between scientific reasoning and pattern fitting observable, covering diverse structural complexities.

4. Key Results

A. Uniform Strategy Shift, Divergent Accuracy

Adding 10 examples reduced the share of Parametric ($S_{know}$) trajectories by 6–20 percentage points across all domains. However, the impact on final accuracy varied significantly:

• Economics: Accuracy dropped significantly (−15.7pp). Here, $S_{know}$ was the superior strategy (88.3% accuracy). The shift to empirical fitting (41.7% accuracy) caused a massive performance loss.
• Geoscience: Accuracy appeared to improve (+13.3pp). Here, $S_{know}$ was weak (49.2% accuracy) because the model often recalled the correct formula but used wrong default parameters. The shift to empirical fitting allowed the model to interpolate correctly from examples, masking the loss of scientific reasoning.
• Physics & Biology: Accuracy remained relatively flat. In Physics, the shift was small; in Biology, the empirical strategy happened to perform similarly to the parametric one.

Conclusion: The model does not selectively abandon knowledge where it is wrong; it abandons it everywhere, regardless of whether the new strategy is better or worse.

B. Mechanism: Vocabulary-Triggered Recall

The Semantic Decoy experiment revealed that $S_{know}$ is triggered by surface vocabulary (domain keywords) rather than deep mathematical analysis.

• When domain terms were replaced with pseudonyms, Zero-Shot accuracy collapsed (to 3–14%).
• However, 10-Shot performance partially recovered (to 22–40%), proving that $S_{emp}$ operates on numerical patterns independent of vocabulary.
• Implication: The mere presence of numerical examples activates the empirical pathway, overriding the vocabulary-triggered knowledge pathway.

C. Internal Representation Convergence

Analysis of hidden states showed that under 10-shot conditions, the representation spaces of different domains converged.

• In Zero-Shot, domains occupied distinct regions (activating specific domain circuits).
• In 10-Shot, domains moved closer together, particularly Geoscience, which showed the strongest shift. This suggests the model switches to a domain-agnostic pattern-fitting circuit when examples are present.

D. Failure of Mitigation

Standard prompting interventions (e.g., explicitly naming the formula, negative prompts against fitting) failed to universally prevent displacement. When the pre-trained knowledge was the optimal strategy (Economics), no amount of consistent examples could recover the performance lost by the mere presence of those examples.

5. Significance and Implications

• Caution for Practitioners: The paper warns that in scientific applications, providing in-context examples may displace the very knowledge they are intended to support. Relying solely on end-to-end accuracy is dangerous, as it can mask the loss of scientific reasoning (e.g., in Geoscience, the model got the right answer for the wrong reason).
• Nature of Scientific Reasoning in LLMs: The study suggests that LLMs' scientific reasoning is often recognition-based (triggered by keywords) rather than reasoning-based (mathematical derivation). This makes it fragile and easily overridden by the computational shortcut of pattern matching.
• Broader Shortcut Learning: This phenomenon is framed as an instance of "shortcut learning," where models default to the computationally easiest available strategy (fitting examples) rather than the most principled one (deriving from laws), even when the shortcut is suboptimal.
• Future Directions: The authors call for mechanistic studies (e.g., activation patching) to identify the specific circuits responsible for this competition and for new evaluation metrics that can distinguish between "knowledge-driven" and "example-driven" success.