💻 computer science

Reconciling Contradictory Views on the Effectiveness of SFT in LLMs: An Interaction Perspective

This paper proposes an interaction-based perspective to explain the inconsistent effectiveness of supervised fine-tuning (SFT) on large language models, revealing that SFT initially removes noise-like interactions but quickly leads to overfitted interactions if training continues beyond a brief denoising phase.

Original authors: Junpeng Zhang, Lei Cheng, Guoxi Zhang, Hua Cai, Qing Xu, Quanshi Zhang

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Junpeng Zhang, Lei Cheng, Guoxi Zhang, Hua Cai, Qing Xu, Quanshi Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Why Does "Practice" Sometimes Make AI Worse?

Imagine you have a brilliant student (a Large Language Model, or LLM) who has already read the entire library of human knowledge. They are smart, but they sometimes speak in a confusing way or use weird slang.

To fix this, teachers give them Supervised Fine-Tuning (SFT). This is like giving the student a specific workbook of "correct" answers and saying, "Practice these examples so you can talk like a helpful assistant."

The Contradiction:

For small students: This practice works wonders. They get better immediately.
For the brilliant student (LLM): Sometimes, this practice works great. But other times, it makes them worse. They start memorizing the workbook too perfectly, forgetting how to talk to real people, or even hallucinating facts.

The Paper's Goal:
The researchers wanted to figure out why this happens. They didn't just look at the final test scores; they looked inside the student's brain to see what was changing while they practiced.

The Tool: "Word Interactions" as Lego Blocks

To peek inside the AI's brain, the researchers used a special tool called Interaction Analysis.

Think of the AI's brain not as a black box, but as a giant set of Lego instructions.

Simple Instructions: "If I see the word 'fire', I should probably say 'hot'." (This is a simple, reliable interaction).
Complex, Messy Instructions: "If I see 'fire' AND 'red' AND 'sky' AND 'cloud' AND 'maybe' AND 'yesterday'..." (This is a complex, messy interaction).

The researchers found that the AI's output is actually just a sum of these Lego instructions. Some instructions are helpful (reliable), and some are just noise (confusing or canceling each other out).

The Discovery: The "Denoising" vs. "Overfitting" Dance

The researchers watched the AI practice step-by-step and discovered that the process happens in two very distinct phases, like a short burst of cleaning followed by a long period of messing things up.

Phase 1: The "Spring Cleaning" (The Good Part)

Duration: Extremely short (only the first few hundred steps of training).

Imagine the AI's brain is a messy room full of old, broken toys that don't fit together.

What happens: As soon as the AI starts practicing, it immediately throws away the broken toys.
The Science: The AI rapidly removes "noise-like interactions." These are the complex, confusing instructions where positive and negative effects cancel each other out (e.g., one rule says "go left," another says "go right").
The Result: The room becomes much cleaner. The AI keeps only the simple, reliable instructions (like "fire = hot"). This is why the AI often gets a quick boost in performance at the very beginning.

Phase 2: The "Over-Decorating" (The Bad Part)

Duration: The rest of the training (the long haul).

Once the room is clean, the AI keeps practicing. But instead of learning new, useful rules, it starts over-decorating.

What happens: The AI starts inventing new, overly complicated rules that only work for the specific practice book, not for the real world.
The Science: The AI begins to learn "overfitted interactions." These are high-complexity patterns that look like they make sense but are actually just memorizing the specific examples in the training data. They are fragile and don't generalize.
The Result: The AI starts to "forget" how to be a general assistant and becomes a robot that only knows the specific workbook. This is why performance can drop or become inconsistent if you train for too long.

The Key Takeaways

The "Sweet Spot" is Tiny: The paper claims that the only time the AI is actually learning something truly useful (by removing noise) is in that very first, brief moment.
More Data Isn't Always Better: If you keep training the AI after that brief cleaning phase, you aren't making it smarter; you are just teaching it to memorize the training data too perfectly (overfitting).
The "Backbone" is Already There: The most reliable rules the AI uses to answer questions were already inside it before it started practicing. The fine-tuning mostly just helped the AI stop using the confusing, broken rules. It didn't build a new brain; it just cleaned up the old one.

The Practical Advice (Early Stopping)

The paper suggests a new way to train AI: Stop early.

Instead of training an AI for days on massive datasets, you should monitor these "Lego instructions." As soon as the AI stops throwing away the "broken toys" (noise) and starts picking up new, weird decorations (overfitting), you should hit the stop button.

In short: The paper argues that for Large Language Models, "less is more." A tiny bit of practice cleans up the mess, but too much practice just creates a new, more complicated mess.

Technical Summary: Reconciling Contradictory Views on the Effectiveness of SFT in LLMs: An Interaction Perspective

Problem Statement

Supervised Fine-Tuning (SFT) is a standard strategy for adapting pre-trained Deep Neural Networks (DNNs) to domain-specific tasks. However, its effectiveness in Large Language Models (LLMs) remains a subject of contradictory viewpoints. While some studies suggest SFT is essential for improving instruction-following and usability, others argue it leads to overfitting, limits generalization, and may even degrade performance. This paper addresses the scientific question: What internal factors account for the inconsistent effectiveness of SFT across different LLMs? Specifically, the authors aim to quantify the inference patterns learned during SFT to explain why it improves performance in some cases while inducing overfitting in others.

Methodology

The paper employs an interaction-based explanation framework to analyze the internal inference patterns of LLMs. This approach treats interactions between input variables (words/tokens) as primitive inference patterns (phrase patterns) that faithfully represent the model's logic.

Core Concepts

Interaction Definition: The authors model the LLM's output score $v(x)$ using a logical model $\phi(x)$ composed of AND-OR interactions.
- AND interactions: Represent a phrase pattern where a set of input variables must all be present to trigger an effect.
- OR interactions: Represent a pattern where the presence of any variable in a set triggers an effect.
- The logical model is proven to be a faithful approximation of the LLM output across all $2^n$ masked states of the input.
Interaction Categorization: During the SFT process, interactions are tracked and categorized into three types based on their evolution from the pre-trained state ( $t=0$ ) to step $t$ :
- Removed Interactions: Encoded before SFT but eliminated during training.
- Preserved Interactions: Encoded before SFT and retained throughout training.
- Newly Emerged Interactions: Acquired during the SFT process.
Quality Metrics: To assess the reliability of these interactions, the authors utilize:
- Order/Complexity: The number of variables involved. Low-order interactions are generally more reliable.
- Generalizability ( $\gamma$ ): Whether an interaction is consistently extracted across different LLM architectures (target vs. baseline).
- Uncancelled Effect Ratio ( $\rho$ ): The ratio of net interaction effects to total absolute effects. A low $\rho$ indicates strong mutual cancellation between positive and negative effects, suggesting noise.
Experimental Setup: The study conducts SFT on multiple LLMs (e.g., Llama-2/3, Qwen2.5, Gemma-3) across diverse datasets (GoEmotions, Unilaw-R1-Data, Databricks-Dolly-15k) using LoRA. The evolution of interaction distributions is tracked across training steps.

Key Findings

The analysis reveals a distinct two-stage evolution of interactions during SFT:

1. The Brief Denoising Stage

Primary Mechanism: In the very early stages of SFT (e.g., within the first 1,000 training steps), the model primarily removes noise-like interactions.
Characteristics of Removed Interactions:
- They are predominantly high-order (complex).
- They exhibit strong mutual cancellation (positive and negative effects cancel out, $\rho \approx 0$ ).
- They show poor generalizability across different model architectures.
Characteristics of Early-Emerged Interactions: A small number of newly emerged interactions occur in this stage. These tend to be low-order, generalizable, and have high uncancelled effects, representing reliable patterns.

2. The Prolonged Overfitting Stage

Primary Mechanism: Once the initial denoising phase concludes, continued fine-tuning leads to the emergence of overfitted interactions.
Characteristics of Later-Emerged Interactions:
- A large volume of new interactions emerges, but they are predominantly high-order.
- They exhibit low generalizability (model-specific) and strong mutual cancellation, behaving similarly to the removed noise.
Preserved Interactions: The set of preserved interactions stabilizes early. These interactions are predominantly low-order, highly generalizable, and contribute consistently to prediction (high $\rho$ ). They form the "backbone" of the model's inference.

Key Contributions

Interaction-Based Diagnosis: The paper introduces interaction evolution as a faithful metric for quantifying the internal dynamics of SFT in LLMs, moving beyond simple loss curves.
Reconciliation of Contradictions: The findings explain the inconsistent effectiveness of SFT:
- Benefit: SFT is effective because it rapidly removes unreliable, noise-like interactions present in the pre-trained model.
- Detriment: SFT becomes detrimental when training continues beyond the brief denoising window, introducing a flood of overfitted, non-generalizable noise patterns.
Identification of the "Backbone": The study demonstrates that a small set of preserved interactions (and a few early-emerged ones) constitutes the core reliable inference logic of the LLM, while removed and late-emerged interactions contribute marginally or act as noise.

Significance and Claims

The paper claims that its findings provide a principled criterion for early stopping in end-to-end SFT.

Practical Guidance: The "effective regime" of SFT is surprisingly short. Collecting massive amounts of SFT data or training for extended periods may offer limited additional value and risks degrading performance by introducing overfitted patterns.
Monitoring Tool: Interaction metrics (specifically the emergence of high-order, cancelling interactions) can serve as diagnostic signals to monitor representation quality and determine the optimal stopping point for fine-tuning.
Theoretical Insight: The work challenges the conventional belief that larger datasets and longer fine-tuning schedules are inherently beneficial for LLMs, suggesting instead that the primary utility of SFT is rapid denoising rather than the acquisition of new, reliable reasoning capabilities.