Explainable Token-level Noise Filtering for LLM Fine-tuning Datasets

This paper introduces XTF, an explainable framework that improves LLM fine-tuning performance by decomposing token contributions into reasoning importance, knowledge novelty, and task relevance to identify and mask noisy tokens, achieving up to a 13.7% improvement across math, code, and medical tasks.

The Gist

Imagine you are trying to teach a brilliant but slightly distracted student (the Large Language Model or LLM) how to solve a specific type of problem, like advanced math or coding. You have a stack of textbooks (the dataset) filled with questions and perfect answers.

Usually, when you teach this student, you hand them the whole textbook page and say, "Memorize this entire answer." The student tries to learn every single word, letter, and punctuation mark on the page.

The Problem: The "Noise" in the Classroom
The authors of this paper realized that not every word on those answer pages is actually helpful. In fact, some words are just "noise."

Think of it like this: If you are teaching someone how to bake a cake, the recipe says: "Mix 2 cups of flour, 1 cup of sugar, and 3 eggs. Then, bake at 350 degrees for 30 minutes."

• Useful words: "2 cups," "flour," "350 degrees."
• Noise words: The word "the" appearing 20 times, or the specific font style of the text, or a random typo that doesn't change the meaning.

If the student tries to memorize every single character, including the boring "the"s and the formatting symbols, they get confused. They waste brainpower on things that don't help them bake the cake. In the world of AI, this is called token-level noise. It slows down learning and can even make the model worse at the final task.

The Solution: XTF (The Smart Filter)
The paper proposes a new method called XTF (Explainable Token-Level Noise Filtering). Instead of handing the student the whole page, XTF acts like a super-smart teaching assistant who reads the answer first, highlights the important parts, and tells the student to ignore the rest.

To decide what to ignore, the assistant uses three simple rules (called Attributes):

1. Reasoning Importance (The "Why" Check):

   • Analogy: If you remove this word, does the sentence stop making sense?
   • Example: In "2 + 2 = 4," the word "4" is crucial. The word "the" in "The answer is 4" is less important. If the model doesn't need a word to figure out the logic, it's noise.

2. Knowledge Novelty (The "New Stuff" Check):

   • Analogy: Does the student already know this?
   • Example: If the student is already an expert at adding numbers, teaching them "1 + 1 = 2" is a waste of time. They need to learn something new. If the model can already guess the word with 99% confidence, it's not learning anything new, so we skip it.

3. Task Relevance (The "Topic" Check):

   • Analogy: Is this word actually about the topic we are studying?
   • Example: If you are studying medicine, a sentence about "how to fix a car engine" is irrelevant, even if it's grammatically correct. The assistant filters out words that drift away from the specific topic (like math or coding).

How It Works in Practice
The XTF system scans the training data, scores every single word based on these three rules, and then masks (hides) the low-scoring words during the training process.

Imagine the student is taking a test. Instead of looking at the whole answer key, the teacher covers up the boring, repetitive, or irrelevant parts with a black marker. The student only focuses on the "meat" of the answer.

The Results
The paper tested this on three difficult subjects: Math, Coding, and Medicine.

• The Outcome: By filtering out the noise, the AI models learned faster and became much better at their jobs.
• The Numbers: In some cases, the models improved their accuracy by up to 13.7%. That's a massive jump, like a student going from a C to an A+ just by cleaning up their study notes.

Why This Matters
Before this, most people tried to fix bad data by throwing away entire sentences or adding more data. This paper says, "No, let's get surgical." We don't need to throw away the whole book; we just need to erase the boring words on the page.

In a Nutshell:
XTF is like a smart editor for AI training data. It looks at the answers the AI is supposed to learn, identifies the "fluff" and "noise," and tells the AI to ignore it. This allows the AI to focus on the truly important information, making it smarter, faster, and more accurate without needing more data.

Technical Summary

Here is a detailed technical summary of the paper "Explainable Token-Level Noise Filtering for LLM Fine-Tuning Datasets" (XTF).

1. Problem Statement

Large Language Models (LLMs) are typically fine-tuned using datasets designed at the sentence level, where the entire output label is treated as a single training target. However, LLMs optimize via token-level loss calculations. This misalignment introduces a fundamental discrepancy:

• Token-Level Noise: Not all tokens within a label sentence contribute positively to the model's learning. Some tokens may be redundant, already known to the base model, or irrelevant to the specific downstream task.
• Negative Impact: Training on these noisy tokens can misguide the convergence direction, dilute the learning signal, and ultimately degrade the performance of the fine-tuned model.
• Research Gap: Existing data optimization methods (e.g., data filtering or augmentation) operate at the sample level (filtering entire sentences) or rely on specific scenarios (like pre-training or preference optimization). There is a lack of methods capable of identifying and filtering noise at the token level specifically for fine-tuning tasks without requiring complex reward models or prior knowledge of labeled pairs.

2. Methodology: XTF Framework

The authors propose XTF (Explainable Token-Level Noise Filtering), a framework that decomposes the contribution of token-level data into three explicit, explainable attributes. The process involves three phases:

Phase 1: Attribute Decomposition

The authors define three attributes that determine a token's value for fine-tuning. A token is considered "noise" if it lacks any of these attributes:

1. Reasoning Importance (RI): Does the token significantly affect the base model's inference logic?
   • Metric: Attention Score. Tokens with low attention scores from the base model (when processing input + label) are deemed less important for reasoning.
2. Knowledge Novelty (KN): Is the token introducing new knowledge the base model hasn't learned?
   • Metric: Probability of Correct Prediction (PCP). If the base model predicts a token with very high probability (e.g., >95%), it implies the knowledge is already mastered. Low novelty (high PCP) indicates noise.
3. Task Relevance (TR): Is the token relevant to the specific downstream task domain?
   • Metric: Semantic Distance. Tokens are embedded using the base model, and their distance from the "domain center" (average embedding of the task dataset) is calculated. Tokens far from the domain center are considered irrelevant.

Phase 2: Scoring and Filtering

XTF employs a conservative strategy to identify noisy tokens based on the scores of the three attributes:

• RI Filtering: Uses the Interquartile Range (IQR) to filter tokens with extremely low attention scores.
• KN Filtering: Uses a heuristic threshold (PCP > 0.95) to filter tokens the model already knows well.
• TR Filtering: Uses the Multi-Otsu method to cluster relevance scores and filter tokens in the cluster with the second-smallest mean value (often representing space replacements or irrelevant filler).
• Union Strategy: The final set of noisy tokens is the union of tokens filtered by any of the three criteria. This ensures that tokens ambiguous in one dimension are caught by another.

Phase 3: Gradient Masking

During fine-tuning, the identified noisy tokens in the output labels are masked by assigning them a default value (e.g., -100 in standard Hugging Face implementations). This excludes them from the loss calculation and gradient updates, effectively forcing the model to focus only on the high-value tokens.

3. Key Contributions

1. Identification of Research Gap: The paper highlights the critical mismatch between sentence-level dataset design and token-level LLM optimization, identifying token-level noise as a key bottleneck in fine-tuning.
2. Explainable Framework (XTF): Proposes a novel, three-attribute decomposition (RI, KN, TR) to quantify token value. Unlike "black box" filtering, XTF provides theoretical justification and interpretability for why specific tokens are removed.
3. Comprehensive Empirical Validation: Conducted extensive experiments across 7 mainstream LLMs (Llama, Mistral, DeepSeek families) and 3 diverse downstream tasks (Math, Code, Medicine).

4. Experimental Results

The authors evaluated XTF against regular fine-tuning and various baselines (Data Filtering, Data Augmentation, Selective Language Model training, Token Cleaning).

• Math Task (GSM8K): XTF improved accuracy by up to 13.3% compared to normal fine-tuning. On DeepSeek-1.5B, it achieved a 9.3% improvement over the best baseline.
• Medicine Task (PubMedQA): XTF achieved up to 13.7% accuracy improvement (specifically on Llama-3.1-8B with LoRA), outperforming all baselines.
• Code Task (HumanEval): While gains were smaller due to the inherent difficulty of code generation, XTF improved pass@1, pass@5, and pass@10 by up to 6.3%. Notably, in cases where normal fine-tuning degraded performance due to noise, XTF maintained or improved results.
• Ablation Studies: Results confirmed that all three attributes are necessary; using them in combination yields the best performance, demonstrating their complementary nature.
• Efficiency: XTF operates at the inference level (requiring only two passes over the data for scoring) rather than training a reference model, making it computationally more efficient than token-level baselines like SLM or TC.

5. Significance and Impact

• Dataset Optimization: XTF shifts the paradigm from "more data" to "better data" at the granular token level, proving that removing specific tokens can significantly enhance model alignment and task performance.
• Explainability: By decomposing noise into reasoning, novelty, and relevance, the method offers a transparent way to understand why certain data points are harmful, bridging the gap between complex training mechanisms and human interpretability.
• Generalizability: The method is model-agnostic and task-agnostic, showing consistent improvements across different model sizes (1B to 14B) and domains (logic, code, specialized knowledge).
• Theoretical Foundation: The paper provides a theoretical proof (in the Appendix) showing that filtering noise improves the alignment between the training gradient and the ideal gradient direction, mathematically justifying the performance gains.

In conclusion, XTF demonstrates that token-level dataset optimization is a crucial, yet underexplored, lever for improving LLM fine-tuning, offering a practical and explainable solution to mitigate the negative effects of noisy training data.