🤖 machine learning

Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation

This paper introduces a scalable, inference-only data auditing pipeline that approximates Shapley values to efficiently identify and remove conflicting, mislabeled, or unsafe records from LLM alignment datasets and benchmarks, thereby significantly reducing manual audit efforts and exposing critical flaws in human-annotated ground truth.

Original authors: Yunting Song, Matthew Watson, Peter Grabowski, Jun Qin

Published 2026-07-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yunting Song, Matthew Watson, Peter Grabowski, Jun Qin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to be a helpful, safe, and honest human assistant. You don't program it with rigid rules; instead, you show it millions of examples of good conversations and bad ones, letting it learn by example. This is how modern AI gets its "personality" and safety filters. But here's the catch: the robot is only as good as the stories you tell it. If your storybook is filled with typos, lies, or confusing contradictions, the robot will get confused and might start acting strangely.

For a long time, checking these storybooks was like trying to find a single bad apple in a mountain of fruit by looking at them one by one. It was slow, expensive, and often missed the subtle problems. Scientists have tried using math to figure out which stories are most important, a method called "Shapley value," but doing the math exactly is so heavy it would take forever for a giant pile of data. The big question is: How can we quickly spot the hidden mistakes, the confusing contradictions, and the bad advice in these massive training books without reading every single page or re-teaching the robot from scratch?

This paper introduces a clever new detective tool called an "influence-based data auditing pipeline." Think of it as a magic magnifying glass that doesn't need to re-read the whole book to find the errors. Instead of retraining the robot, the researchers use a trick: they ask the robot, "If I show you this specific story before you read this next one, does it help you understand, or does it confuse you?"

The researchers built a system that groups similar stories together and then tests them like a game of "spot the difference." They take a story and ask the robot to predict the ending. Then, they show the robot a similar story first (like a hint) and ask it to predict the ending again. If the "hint" story makes the robot's prediction worse, it's a huge red flag. It means the two stories are fighting each other, like one saying "Always tell the truth" and the other saying "It's okay to lie in this case."

The team tested this on two famous collections of training data. First, they looked at a dataset called HelpSteer2. Their tool found that out of thousands of records, they could shrink the list of things humans needed to check by 99.1%. They uncovered hidden errors where humans had given perfect scores to answers that were actually full of made-up facts or ignored important instructions. For example, they found cases where a robot was praised for writing a perfect-looking essay that contained fake dates and fake scientific papers, while a similar essay with the same fake facts was correctly punished. This revealed a "superficiality bias," where human reviewers liked how the text looked but didn't check if the facts were real.

Next, they audited a dataset called HH-RLHF, which is used to teach robots to choose between two answers. They found thousands of hidden contradictions where the "correct" answer chosen by humans was actually dangerous or unhelpful. In one scary example, a human reviewer picked a response that suggested pouring bleach into someone's eye as a prank, while rejecting a safe response that just asked for clarification. The tool spotted these safety risks that standard checks missed.

Perhaps most surprisingly, when they used this tool to check the test questions (the exams used to grade the robots), they found that the exams themselves were broken. The "correct" answers in the test bank were often wrong. In many cases, the smartest robots were actually picking the safer, better answer, but the test marked them as "wrong" because the human-written answer key was flawed. This suggests that some of the biggest problems in AI safety might not be the robots failing, but the tests being unfair.

The authors are careful to say that while their math tool is incredibly fast and effective at narrowing down the search, it still needs a human (or a very smart AI judge) to make the final call on whether a story is truly broken. They didn't fix the data themselves, but they built a highly efficient map that shows exactly where the potholes are, saving researchers from having to drive over every single inch of road to find them. By using this method, they reduced the work of checking a massive dataset from looking at thousands of items to just a few dozen high-priority suspects, proving that sometimes, the best way to find the truth is to see how the pieces of a puzzle push and pull against each other.

Technical Summary: Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation

Problem Statement

The alignment of Large Language Models (LLMs) is increasingly constrained by data quality rather than model architecture. As datasets scale, massive preference and instruction-tuning corpora accumulate hidden structural contradictions, safety risks, and systemic human annotation errors. Current auditing methods face significant limitations:

Semantic Deduplication: Effective at finding textual duplicates but fails to capture the functional utility or predictive impact of individual records.
LLM-as-a-Judge: While scalable, these methods rely on human-defined rubrics, often exhibit biases (e.g., toward response length), and cannot mathematically determine how a record influences a model's predictive distribution.
Data Shapley: Provides a rigorous framework for data valuation but is computationally prohibitive ( $O(2^N)$ ) for large datasets. Existing approximations either lack record-level granularity (requiring coarse grouping) or are restricted to classification/regression tasks, failing to address generative LLM mechanics.

There is a critical gap in the ability to perform point-wise, record-level data valuation for generative LLMs without requiring computationally expensive model retraining or iterative data sampling.

Methodology

The authors propose a scalable, inference-only data valuation pipeline that approximates the Shapley value by measuring the marginal contribution of data points through conditional log-likelihood shifts. The workflow consists of two primary phases:

Phase 1: Influence Measurement

Semantic Neighborhood Construction: Prompts are projected into a dense vector space (using Qwen3-Embedding-4B). Static system instructions are stripped to prevent vector dilution. A $k$ -Nearest Neighbor ( $k$ -NN) graph is constructed based on cosine similarity.
Zero-Shot and One-Shot Scoring: For a target record $(x_i, y_i)$ $(x_{i}, y_{i})$ and a neighbor $(x_j, y_j)$ $(x_{j}, y_{j})$ , the pipeline calculates the conditional log-likelihood of the target response $y_i$ $y_{i}$ under two conditions:
- Baseline ( $LL_{base}$ ): The probability of $y_i$ given only $x_i$ .
- Probe ( $LL_{probe}$ ): The probability of $y_i$ given a composite prompt where $(x_j, y_j)$ serves as a one-shot demonstration prior to $x_i$ .
Influence Calculation: The predictive influence ( $\Delta LL$ ) is defined as the difference: $\Delta LL_{i,j} = LL_{probe}(i, j) - LL_{base}(i)$ . A positive $\Delta LL$ indicates a constructive influence, while a negative value suggests a behavioral contradiction where the neighbor degrades the target's predictive probability.
Directed Graph Construction: A directed graph $G=(V, E)$ is built where edges represent the flow of influence from neighbor to target, weighted by $\Delta LL$ .

Phase 2: Metric Derivation & Auditing

From the graph, three metrics are derived for each node:

Exemplified Advantage (Passive Influence): Measures the aggregate influence a record receives from its incoming neighbors. High values suggest redundancy.
Exemplifying Advantage (Active Influence): Measures the predictive contribution a record exerts on its outgoing neighbors. This approximates the Data Shapley value by treating 1-shot in-context learning as a zero-gradient surrogate for parameter updates.
One of a Kind Score (Topological Uniqueness): Quantifies structural uniqueness based on the asymmetry of $k$ -NN retrieval (records that retrieve neighbors but are rarely retrieved themselves).

Contradiction Detection Paradigms:

Unsupervised (Implicit Labels): Flags nodes where exemplified or exemplifying advantages fall below a threshold ( $\tau_{adv}$ ), specifically looking for Mutual Nearest Neighbors (MNN) that mutually degrade each other's predictive probability.
Supervised (Explicit Labels): A two-stage process for reward modeling datasets. First, it isolates "gold standard" (high human-rated) records with negative exemplifying advantages. Second, it performs "Twin Verification" by finding low-rated neighbors with similar response behaviors to identify inter-annotator inconsistencies.

Key Contributions

Influence-Based Valuation Framework: The paper introduces a highly scalable, inference-only pipeline that approximates Shapley values for generative LLMs. Unlike prior methods, it operates in $O(K)$ forward passes (where $K$ is neighborhood size) without model retraining or clustering, generalizing to open-ended generation and pairwise preference data.
Structural Contradiction Detection: The pipeline uncovers hidden structural contradictions in widely used datasets (HelpSteer2 and HH-RLHF) that standard similarity metrics miss. It identifies human-rewarded records that conceal structural errors (e.g., factual hallucinations, ignored constraints) which degrade model performance.
Exposing Benchmark Vulnerabilities: By auditing the evaluation split of HH-RLHF, the work demonstrates that highly capable models frequently predict the safer or more helpful response but are penalized by objectively flawed human ground-truth labels, revealing vulnerabilities in current benchmark integrity.

Experimental Results

The pipeline was evaluated on two datasets using Qwen3.5-9B as the target model and Gemini 3.1 Pro for expert arbitration.

Case Study 1: HelpSteer2

Process: Applied to 21,362 records, focusing on 8,434 records with perfect human helpfulness scores (4/4).
Efficiency: Reduced the manual audit search space by 99.1% (from 8,434 records to 77 candidate pairs).
Findings: Identified 18 contradiction pairs and 10 unique records falsely labeled as perfect. Errors included factual hallucinations (e.g., incorrect calendar dates), ignored formatting constraints, and fabricated academic citations.
Insight: Revealed a "superficiality bias" in human annotation, where structurally sound but factually fraudulent text received high scores.

Case Study 2: HH-RLHF

Training Data Audit: Identified 4,238 contradictions and 2,841 unique problematic records. Categories included safety violations (e.g., rewarding toxic compliance), hallucinations, unhelpful evasion, and structural failures.
Evaluation Data Audit: Applied to the evaluation split, isolating 108 unique records with confirmed structural contradictions.
Model Disagreement: When tested on these isolated "flawed" records, model disagreement rates with human labels spiked significantly compared to the baseline:
- Qwen3.5-9B (FT): Disagreement rose from 28.9% (full set) to 62.04% (verified subset).
- Qwen3.5-27B: Disagreement rose from 28.67% to 75.00%.
- Gemma3-12B: Disagreement rose from 38.34% to 63.89%.
Conclusion: The high disagreement rates confirm that the models were often predicting the correct behavior, but the benchmark labels were objectively flawed.

Significance and Limitations

Significance:
The paper provides a mathematically grounded, efficient diagnostic tool to uncover human label failures and sanitize evaluation benchmarks. It shifts the paradigm from computationally prohibitive retraining-based Shapley calculations to a tractable, inference-only approach that maintains record-level granularity. This allows for the rapid identification of systemic annotation errors and the validation of benchmark integrity.

Limitations:

Verification Reliance: Final verification relies on LLM evaluators (e.g., Gemini), which may introduce subjective biases and non-determinism.
Filter vs. Detector: The $\Delta LL$ metric acts as an anomaly filter that reduces the search space by ~99% but still requires an arbitrator to confirm semantic rule clashes.
Precision vs. Recall: Strict thresholds prioritize precision, meaning the pipeline does not capture the entirety of dataset errors (e.g., the 108 isolated errors in HH-RLHF do not account for the full ~30% disagreement rate of base models).
Scalability of Human Ground Truth: Establishing large-scale human ground truth for annotation errors remains prohibitively expensive, limiting the ability to fully validate the pipeline's recall.