Task Matters: Knowledge Requirements Shape LLM Responses to Context-Memory Conflict

1. Problem Statement

Large Language Models (LLMs) possess vast parametric memory (knowledge encoded in weights) but are frequently deployed with contextual information (provided in the prompt). A critical issue arises when these two sources conflict (Context–Memory Conflict).

• The Gap: Prior research has largely focused on Contextual Question Answering (CQA), implicitly assuming tasks require exclusive reliance on the provided context. However, real-world tasks vary significantly in their knowledge requirements. Some require only surface extraction, others require integrating background knowledge, and some require ignoring context to rely on internal facts.
• The Question: It remains unclear how LLMs behave under conflict when the task demands different types of knowledge utilization. Does a conflict degrade performance equally across all task types? Do mitigation strategies (like reiteration) work universally?

2. Methodology

The authors propose a model-agnostic diagnostic framework to isolate the effects of conflict from task demands.

A. Data Construction Pipeline (Figure 2)

1. Parametric Knowledge Collection: The framework queries the target model to identify its internal "beliefs" on specific factual questions. It uses a stance-consistency filter (checking consistency across paraphrased prompts) to ensure the model truly holds a specific parametric belief.
2. Evidence Creation:
   • No Contradiction (NC): Context aligns with parametric knowledge.
   • High Plausibility Contradiction (HPC): Context contradicts parametric knowledge but is logically coherent and plausible (e.g., a scientifically sound but incorrect theory).
   • Low Plausibility Contradiction (LPC): Context contradicts parametric knowledge and is implausible (e.g., "grooves on Phobos caused by an alien dance party").
   • HPCE (High Plausibility with Explanation): HPC instances augmented with free-text rationales explaining the contradiction.
3. Task Annotation: The same underlying knowledge and conflict instances are mapped to five distinct task types to vary knowledge reliance:
   • Knowledge Free (KF): Extractive QA (copying text). No reasoning or prior knowledge needed.
   • Contextual Knowledge (CK): Answer must be derived only from the provided context (ignoring parametric memory).
   • Parametric Knowledge (PK): Answer must be derived only from internal knowledge (ignoring context, even if it contradicts).
   • Parametric-Contextual Knowledge (PCK): The model must integrate both internal knowledge and external context, acknowledging the conflict.
   • Retrieval Augmented Generation (RAG): Standard RAG setting where the model receives multiple passages (one NC, one contradictory) and must synthesize an answer.

B. Evaluation

• Models Tested: A diverse set including open-weight models (Mistral-7B, OLMo2-7B/13B, Qwen2.5-7B/14B) and a proprietary frontier model (GPT-5.2).
• Metrics: Accuracy for CK/PK; F1 scores for KF/PCK/RAG.
• Validation: Automated checks by LLMs and human verification on a subsample to ensure data quality.

3. Key Findings

A. Conflict Impact is Task-Dependent

• Knowledge-Intensive Tasks Suffer: Performance degradation is severe in CK and PK tasks when conflicts exist.
  • In CK tasks, models fail to follow instructions to ignore their internal knowledge, leading to hallucinations even when the context is highly plausible.
  • In PK tasks, the presence of conflicting context (even if implausible) degrades performance, suggesting models cannot fully "switch off" context processing.
• Knowledge-Free Tasks are Resilient: In KF tasks (simple extraction), conflict has minimal impact. In some cases, LPC (implausible) contexts actually improved performance by reducing ambiguity, as the model simply copies the text without engaging its parametric memory.
• Plausibility Paradox: Contrary to the intuition that "more plausible = more trusted," in PK tasks, models performed better with implausible (LPC) distractors than plausible (HPC) ones. Plausible distractors successfully hijacked the model's attention, whereas implausible ones were easier to discard.

B. Mitigation Strategies (Rationales vs. Reiteration)

• Rationales (HPCE): Adding explanations to contradictory context improved performance in CK tasks (helping the model follow the context) but harmed performance in PK tasks (acting as strong distractors that pulled the model away from its internal knowledge).
• Reiteration: Simply repeating the context (HPCdub) was often as effective as adding rationales for CK tasks but less detrimental to PK tasks than adding complex explanations. This suggests that reiteration strengthens context reliance without the cognitive "overhead" of processing complex logical arguments that might override internal beliefs.

C. Reliability of Model-Based Evaluation

• The Judge Bias: The paper demonstrates that using LLMs as evaluators (judges) is fundamentally flawed in the presence of conflict.
• Evidence: When an LLM judge evaluates a response that is contextually correct but factually incorrect (contradicting its parametric memory), it frequently rejects the response as "incorrect."
• Implication: Even frontier models (GPT-5.2) exhibit this bias. This undermines the reliability of automated evaluation pipelines where the judge's internal knowledge conflicts with the ground truth provided in the context.

4. Key Contributions

1. Diagnostic Framework: Introduced a unified framework that holds underlying knowledge constant while systematically varying task formulations, allowing for the disentanglement of conflict effects from task demands.
2. Task-Dependent Theory: Established that context-memory conflict is not a monolithic failure mode but is inherently dependent on the task's knowledge requirements.
3. Strategic Insights: Demonstrated that "one-size-fits-all" mitigation strategies (like forcing context reliance via rationales) can be counterproductive depending on the task.
4. Evaluation Warning: Provided empirical evidence that LLM judges are systematically biased by their own parametric knowledge, calling into question the reliability of current LLM-as-a-Judge benchmarks.

5. Significance and Conclusion

The paper argues that context–memory conflict cannot be meaningfully studied or mitigated without explicit consideration of task knowledge requirements.

• For Deployment: Blindly enforcing context adherence (e.g., via RAG or prompt engineering) may improve performance on extraction tasks but degrade performance on tasks requiring factual consistency or internal reasoning.
• For Evaluation: Automated evaluation using LLMs is unreliable when the evaluation task involves factual conflicts. Human oversight or task-aware evaluation mechanisms are necessary.
• Future Directions: The authors motivate the development of task-aware inference mechanisms that can dynamically balance parametric and contextual knowledge based on the specific demands of the task (e.g., selectively attending to context tokens or steering attention heads).

In summary, the paper shifts the paradigm from viewing context-memory conflict as a general model weakness to viewing it as a complex interaction between model architecture, task formulation, and knowledge source reliability.