💬 NLP

Emergence of Context Characteristics Sensitivity in Large Language Models

This paper investigates how large language models' sensitivity to context characteristics evolves across supervised fine-tuning, direct preference optimization, and reinforcement learning stages, revealing that these preferences are actively reshaped at each phase and highlighting the critical importance of balanced instruction fine-tuning datasets for robust context utilization.

Original authors: Nadya Yuki Wangsajaya, Haeun Yu, Isabelle Augenstein

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nadya Yuki Wangsajaya, Haeun Yu, Isabelle Augenstein

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) as a very smart student sitting in a classroom. This student has a massive library of facts memorized in their head (parametric knowledge), but the teacher also hands them a specific textbook page (the context) for every question. The goal is for the student to read that page and answer the question based only on what they just read, ignoring what they already know.

This paper investigates how this student learns to use the textbook during three specific "training camps" (Instruction Fine-Tuning stages): SFT, DPO, and RLVR. The researchers wanted to know: Does the student learn to read the book carefully, or do they start taking shortcuts based on how the book looks?

Here is the breakdown of their findings using simple analogies:

1. The Three Training Camps

Think of the student's education as happening in three phases:

Phase 1: SFT (Supervised Fine-Tuning): This is like the student doing homework with an answer key. They are shown many examples of "Question + Textbook Page = Correct Answer."
Phase 2: DPO (Direct Preference Optimization): This is like a debate club. The student is shown two different answers to the same question. A teacher says, "I like Answer A better than Answer B." The student learns to mimic the "liked" answer.
Phase 3: RLVR (Reinforcement Learning with Verifiable Rewards): This is like a final exam where the student gets points only if they get the answer exactly right. (The paper notes this phase is very expensive to run, so they mostly studied the first two).

2. The "Shortcuts" Learned in Phase 1 (SFT)

During the first training camp (SFT), the student learns a very specific, somewhat lazy habit. They start judging whether to trust the textbook page based on how easy it is to read, rather than whether the information is actually true.

The researchers found the student starts favoring:

Longer Text: They think, "If the text is long, it must be important." (Actually, they often prefer shorter texts because they are easier to digest, but the paper notes a complex relationship here; generally, they prefer texts that are easy to process).
Familiar Words: If the textbook page uses the exact same words as the question, the student thinks, "Aha! This is the right page!" even if the meaning is wrong.
Fluency: If the text is written in smooth, perfect grammar, the student trusts it. If the text is clunky or has typos, the student ignores it, even if the clunky text contains the correct facts.

The Metaphor: Imagine the student is a judge in a contest. Instead of judging the content of the speech, they are judging the font size and the smoothness of the voice. If the speech is loud and clear, they vote "Yes." If it's quiet or stuttering, they vote "No," regardless of the truth.

3. The "Correction" in Phase 2 (DPO)

In the second camp (DPO), the researchers tried to fix these bad habits. They expected the student to unlearn the shortcuts and start reading the content carefully.

The Twist: The outcome depended entirely on what the teacher was teaching.

If the teacher gave the student examples where the "good" answers happened to be long and fluent, and the "bad" answers were short and clunky, the student doubled down on their bad habits. They learned that "Long and Fluent = Good."
However, if the teacher carefully balanced the examples—making sure the "good" and "bad" answers looked the same (same length, same fluency)—the student unlearned the shortcuts. They stopped relying on how the text looked and started relying on the actual content.

The Metaphor: It's like a coach teaching an athlete. If the coach only praises athletes who wear red shoes, the athlete will think red shoes make them faster. If the coach wants the athlete to focus on running technique, they must praise athletes wearing both red and blue shoes equally. If the coach is careless, the athlete keeps focusing on the shoes instead of the running.

4. The Big Takeaway

The paper concludes that a model's ability to use context isn't a fixed trait; it is actively shaped at every step of training.

SFT naturally teaches models to take shortcuts based on how "easy" the text looks (fluency, word overlap).
DPO can either make this worse or fix it, depending entirely on how the training data is curated.

The Lesson: If you want a model that reliably uses the information you give it (and doesn't get distracted by how "pretty" the text looks), you cannot just dump data into the training process. You must carefully curate your datasets to ensure the "good" and "bad" examples are balanced in terms of length, fluency, and word choice. Otherwise, the model will just learn to judge the book by its cover.

Technical Summary: Emergence of Context Characteristics Sensitivity in Large Language Models

Problem Statement

Large Language Models (LLMs) possess the capability to utilize provided context to answer queries, a trait essential for applications like retrieval-augmented generation (RAG) and tool use. However, this usage is often fragile; models may ignore context in favor of parametric knowledge or be distracted by irrelevant information. While prior research has established correlations between specific context characteristics (e.g., length, fluency, similarity) and context usage during inference, the mechanisms by which models acquire these sensitivities during training remain unexplored. Specifically, it is unclear how instruction fine-tuning (IFT) stages shape a model's reliance on these characteristics.

Methodology

The authors investigate the evolution of "context characteristics sensitivity"—defined as the degree to which a characteristic's value influences whether a model uses the provided context—across three distinct IFT stages: Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Reinforcement Learning with Verifiable Rewards (RLVR).

Experimental Setup

Models: The study evaluates four models: Llama-3.1-8B, Llama-3.2-1B, OLMo-2-1B, and OLMo-2-7B. For step-wise analysis (tracking changes within a stage), Llama-3.2-1B is fine-tuned on the same datasets used for OLMo-2-1B.
Datasets: Three datasets are utilized to ensure diversity in task formats and context-knowledge relationships:
- ConflictQA and Context-Reliance: Open-ended QA tasks with counterfactual contexts, ensuring accuracy reflects pure context usage.
- DRUID: A claim verification task reflecting realistic context-knowledge interactions.
Context Characteristics: Three families of characteristics are extracted and measured:
1. Similarity: Jaccard similarity (token overlap) and query-entity overlap.
2. Fluency: Flesch reading ease score and Perplexity.
3. Length: Raw character counts for context and query.
Evaluation Metric: Sensitivity is quantified using the Area Under the Receiver Operating Characteristic (AUROC) curve. This measures how well a specific characteristic value ranks accurate examples (where the model correctly used the context) above inaccurate ones. An AUROC > 0.5 indicates that higher values of the characteristic correlate with correct context usage.

Key Results

1. Sensitivity Formation During SFT

The study finds that SFT consistently instills a specific set of sensitivities across all models and datasets. During SFT, models develop a strong preference for "easy-to-understand" contexts:

Similarity: AUROC for context-query Jaccard similarity increases, indicating models rely more heavily on lexical overlap.
Fluency: AUROC for Perplexity decreases (meaning lower perplexity correlates with accuracy), showing models prefer fluent, natural-sounding text.
Length: AUROC for context length decreases, suggesting models favor shorter contexts, likely because they are easier to integrate.

Crucially, the authors note that while these sensitivities increase during SFT, they do not translate into improved accuracy. This suggests models are exploiting these characteristics as shortcuts to signal context usage rather than genuinely assessing the relevance or truthfulness of the content.

2. DPO and Dataset Characteristics

The impact of DPO is not uniform; it is heavily dependent on the characteristics of the preference dataset (the difference between "chosen" and "rejected" responses).

Resolution vs. Reinforcement: In some cases (e.g., Llama-3.2-1B), DPO resolves the sensitivities learned during SFT, shifting them back toward pre-SFT levels.
Dataset Imbalance: In other cases (e.g., Llama-3.1-8B), DPO amplifies existing sensitivities. The authors attribute this to statistical differences ( $\Delta$ ) in the lexical characteristics of chosen versus rejected responses in the training data. If the chosen responses in the preference dataset differ significantly in length or fluency from the rejected ones, DPO reinforces the model's bias toward those specific characteristics.
Task Dependency: The effect of DPO appears less consistent on the DRUID claim verification task compared to open-ended QA tasks.

3. RLVR Trends

Although RLVR was not analyzed step-wise due to computational costs, the results indicate that RLVR largely preserves the direction of effects established by DPO, consistent with their shared KL-regularized objectives.

Significance and Contributions

The paper's primary contribution is the demonstration that context usage is not a static model property but is actively reshaped at each IFT stage.

SFT as a Shortcut Learner: The authors reveal that SFT teaches models to exploit superficial characteristics (fluency, length, overlap) as proxies for relevance, rather than fostering robust context assessment.
Data Curation Imperative: The findings underscore that the design of the preference dataset in DPO is critical. To counteract the biases formed during SFT and ensure robust context utilization, the chosen and rejected responses in the preference dataset must be carefully curated to share similar context characteristics. Imbalances in these datasets can inadvertently reinforce non-robust behaviors.

Limitations

The authors acknowledge that the study is limited to two model families (Llama and OLMo) at two scales each. While appropriate for analyzing training dynamics, broader coverage across different architectures remains an open area for future work. Additionally, the study focuses on three well-grounded families of characteristics (similarity, fluency, length), though other factors influencing context usage may exist.