💬 NLP

Not All Skills Help: Measuring and Repairing Agent Knowledge

The paper introduces ASSAY, a framework that improves LLM agent performance by empirically measuring and selectively suppressing individual skills with negative causal effects on specific tasks, thereby achieving state-of-the-art results without weight updates.

Original authors: Yixuan Wang, Yiyang Zhou, Yiming Liang, Congyu Zhang, Fuxiao Liu, Jiawei Zhou, Huaxiu Yao

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yixuan Wang, Yiyang Zhou, Yiming Liang, Congyu Zhang, Fuxiao Liu, Jiawei Zhou, Huaxiu Yao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Bad Advice" Problem

Imagine you are teaching a very smart but inexperienced assistant (an AI agent) how to do chores. You tell it, "Here is a list of 100 tips I learned from doing these tasks before."

The current way AI works is that it trusts this list completely. It assumes that if a tip was good for one task, it must be good for every task. It reads the whole list before starting every new job.

The paper argues this is a mistake. Just because a tip helps you bake a cake doesn't mean it helps you fix a leaky pipe. In fact, reading the "baking tips" while trying to fix a pipe might distract you and make you fail.

The authors call this "Causal Heterogeneity." In plain English: A skill that helps on one task often hurts on another.

The Solution: ASSAY (The "Skill Filter")

The researchers built a new system called ASSAY. Think of it as a smart editor for the AI's instruction manual. Instead of blindly trusting the AI's judgment on what to keep, ASSAY uses a scientific experiment to test every single tip.

Here is how ASSAY works in three simple steps:

1. The "Randomized Masking" Experiment (The Taste Test)

Imagine you have a soup with 50 ingredients. You want to know if the "cayenne pepper" is actually helping the flavor.

Old Way: You ask the chef, "Do you like cayenne?" The chef says, "Yes, it's spicy!" (But maybe the chef is just used to spicy food).
ASSAY Way: You run a blind taste test.
- You make 12 bowls of soup.
- In some bowls, you include the cayenne. In others, you leave it out.
- You taste them all.
- The Result: You calculate exactly how much the cayenne helped or hurt the soup on average.

In the paper, they do this with AI tasks. They randomly hide different "skills" (tips) and see if the AI succeeds or fails. This gives them a score for every skill:

Green Score: This skill helps.
Red Score: This skill hurts.
The Trap: Some skills have a score of zero because they help on 50% of tasks and hurt on the other 50%. To a simple observer, they look useless. But to ASSAY, they look dangerous because they are unpredictable.

2. The "Offline Restructuring" (Editing the Manual)

Once they have the scores, they clean up the library of skills:

Split: If a skill is "sometimes good, sometimes bad," they rewrite it. They turn "Always check the contacts" into "Check contacts only if you are splitting a bill."
Retire: If a skill is boring and never helps (score near zero), they throw it away.
Merge: If they have two tips that say the same thing, they combine them.

3. The "Per-Task Masking" (The Smart Filter at the Door)

This is the most important part. Even after cleaning the manual, the AI doesn't read the whole thing for every job.

The Scenario: You are about to buy a coffee grinder.
The Problem: Your manual has a tip about "checking Spotify playlists." If the AI reads this while buying a grinder, it gets distracted and fails.
The ASSAY Fix: Before the AI starts the task, ASSAY looks at the task description. It asks, "Based on our past experiments, which skills will hurt this specific task?"
The Action: It silently blocks (masks) the Spotify tip. It only lets the "Amazon size-check" tip through.

Why This Matters (The Results)

The researchers tested this on two big challenges: AppWorld (simulating app usage) and τ-bench (simulating customer service).

The "Before" State: Adding a huge library of skills often made the AI worse at hard tasks because it got confused by irrelevant tips.
The "After" State: By using ASSAY to filter out the bad tips for each specific job:
- DeepSeek-V3 (a powerful AI) went from failing to becoming the best in the world on AppWorld, beating even models that were heavily retrained (which usually takes much more work).
- GPT-4.1 jumped up the leaderboard on the retail test, beating other top models like o1 and o4-mini, without changing a single line of its code.

The Main Takeaway

The paper proves that more skills are not always better.

Think of it like a toolbox. If you give a carpenter a hammer, a saw, and a wrench, they can build a house. But if you also give them a tennis racket, a frying pan, and a pair of skis, they might get confused and drop the hammer.

ASSAY is the system that looks at the job (building a house) and says, "Okay, give them the hammer and saw. Hide the skis and the frying pan." It doesn't need to rebuild the carpenter; it just needs to curate the tools they are holding.

Key Finding: The biggest bottleneck isn't that the AI lacks skills; it's that the AI doesn't know which skills to use for which job. ASSAY fixes that mismatch.

Technical Summary: Not All Skills Help: Measuring and Repairing Agent Knowledge

1. Problem Statement

Current paradigms for improving Large Language Model (LLM) agents without weight updates rely on accumulating natural-language skills from experience. These systems typically delegate the entire skill lifecycle—generation, retention, and retrieval—to the LLM's own judgment. The underlying assumption is that an LLM can sufficiently determine which skills to keep and how to apply them based on single-task contexts.

However, this paper argues that this approach conflates two distinct roles:

Generation: Creating a skill from experience (a creative act where LLM judgment excels).
Curation: Deciding whether a skill actually helps (an empirical question requiring evidence across many tasks).

The authors identify a systematic downside to unchecked accumulation: causal heterogeneity. Individual skills often help on specific task types while actively hurting on others. Because these opposing effects cancel out in aggregate, global curation methods (which evaluate skills based on average performance) fail to detect them. Consequently, "bad" skills become invisible overhead that degrades agent performance, particularly on complex or out-of-distribution tasks.

2. Methodology: The ASSAY Framework

The authors propose ASSAY (Attribution-Based Skill Selection and Assembly), a framework that separates skill generation from curation. It operates in three stages, unified by a single causal attribution matrix $C \in \mathbb{R}^{N \times M}$ , where $N$ is the number of skills and $M$ is the number of development tasks.

Stage 1: Measuring Causal Effects (Randomized Masking)

Instead of relying on LLM judgment to evaluate skills, ASSAY uses randomized masking to compute unbiased causal attribution.

Protocol: For $K$ independent trials, a random mask is constructed by including each skill with probability $f$ (Bernoulli sampling).
Attribution: The causal score $C[j, i]$ for skill $s_j$ on task $d_i$ is calculated as the difference-in-means estimator between trials where the skill is present and trials where it is absent.
Outcome: This produces a matrix revealing whether a skill helps (positive score), hurts (negative score), or has negligible impact on specific tasks. This reveals causal heterogeneity, where skills often reverse sign across different task types.

Stage 2: Offline Library Restructuring

Using the attribution matrix, the framework restructures the skill library offline to resolve heterogeneity before inference.

Split: Skills with high heterogeneity (large variance in causal scores across tasks) are rewritten by the base LLM into conditional variants with explicit trigger conditions (e.g., "apply only when task type is X"). These variants must pass a development gate to ensure they do not degrade overall performance.
Retire: Skills with near-zero global causal scores (negligible impact) are removed to reduce context noise.
Merge: Near-duplicate skills introduced during splitting are clustered, and the highest-scoring member is retained.

Stage 3: Per-Task Causal Masking

At inference time, a static library is insufficient because the optimal set of skills varies per task. ASSAY treats skill selection as a risk-minimization problem.

Prediction: For a new test task, the system identifies the $k$ nearest development tasks based on instruction embeddings. It computes a predicted causal effect $\hat{C}(s_j, t)$ for each skill by weighting the historical causal scores of those nearest neighbors.
Masking Rule: Skills predicted to have a negative causal effect are suppressed, while those with positive or neutral effects are retained. Protected operational templates (e.g., pagination, data validation) are exempt from masking to prevent catastrophic failure.
Fallback: If the filtered set becomes too small, the system reverts to the full library to ensure graceful degradation.

3. Key Contributions

Empirical Characterization: The paper provides the first empirical evidence of pervasive causal heterogeneity in agent skill libraries, demonstrating that skills routinely help on some tasks while hurting on others, rendering them invisible to global curation.
Framework Design: ASSAY operationalizes the separation of generation (LLM judgment) and curation (empirical measurement). It introduces a pipeline that computes causal attribution, restructures libraries offline, and applies per-task masking at inference.
State-of-the-Art Performance: The framework achieves significant improvements across diverse models and benchmarks without modifying model weights.
- AppWorld: On the hardest split, DeepSeek-V3 achieves 69.3% Task Goal Completion (TGC), a 47.4% relative improvement over the baseline, setting a new state-of-the-art among published methods (including weight-tuned approaches).
- $\tau$ -bench: GPT-4.1 improves by 8.7% relative, advancing past o4-mini, o1, and GPT-4.5 on the public leaderboard without weight modification.
Bottleneck Identification: Ablation studies confirm that the primary bottleneck is not the presence of "bad" skills globally, but the failure to match the correct skills to specific tasks at inference time. Per-task masking accounts for the dominant gains.

4. Experimental Results

The framework was evaluated on two benchmarks (AppWorld and $\tau$ -bench) using seven base models from four providers (OpenAI, DeepSeek, Anthropic, Google).

Generalization: ASSAY consistently improved performance across all seven models. On AppWorld's test_challenge split, uncurated skill libraries actually degraded performance for some models (e.g., GPT-5.1 dropped from 52.5% to 49.9%), while ASSAY reversed this and achieved significant gains.
Difficulty Sensitivity: The method provided the largest gains on Level 2 and Level 3 tasks, where uncurated libraries caused the most harm.
Boundary Conditions: Two models (GPT-5.1 and Sonnet 4.5) showed zero gain on $\tau$ -bench. The authors attribute this to high baseline competence saturating the benefits of prompt-time skill injection, suggesting diminishing returns as base model capability increases.

5. Significance and Claims

The paper claims that the current reliance on LLM judgment for the entire skill lifecycle is a fundamental limitation. By introducing an empirical, measurement-driven curation stage, ASSAY resolves the "causal heterogeneity" problem that plagues existing skill-based agents.

The authors emphasize that their approach is complementary to existing methods:

It operates entirely at inference time and requires no weight updates.
It can be layered on top of any existing skill-generation pipeline (e.g., ACE, CUGA, Reflexion).
It shifts the paradigm from "accumulating more skills" to "precisely matching skills to tasks."

The work concludes that while skill generation is a creative act best handled by LLMs, skill curation is an empirical problem that requires randomized measurement to ensure that the knowledge injected into the context window actually aids the agent's performance.