💬 NLP

CaSKG: Counterfactual-Causal Skill Graphs for Scalable Agent Skill Retrieval

CaSKG is a counterfactual-causal framework that enhances scalable LLM agent skill retrieval by constructing and calibrating a directed skill graph through offline evidence aggregation and textual counterfactual probes, thereby significantly improving task success rates and efficiency across diverse benchmarks compared to existing methods.

Original authors: Zhiyuan Li, Linyuan Gao, Xuechun Ding, Hongwei Chen, Yuan Wu, Yi Chang

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhiyuan Li, Linyuan Gao, Xuechun Ding, Hongwei Chen, Yuan Wu, Yi Chang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a computer program that can talk, reason, and act in the real world, much like a human assistant. These programs, known as large language model agents, are becoming increasingly capable of solving complex problems by combining their ability to generate text with access to external tools and environments. They can learn to use APIs, interact with simulated rooms, or follow scientific procedures. However, as these agents accumulate vast libraries of reusable skills—procedures for everything from cooking a meal to testing a chemical—the challenge shifts from simply having the tools to knowing which ones to use. If an agent tries to remember every possible action at once, it becomes overwhelmed by noise. If it looks only for words that match the current task, it might miss a crucial step that doesn't sound similar but is logically necessary. The core problem is how to retrieve the right sequence of actions from a massive library without getting lost in irrelevant details or missing hidden dependencies.

Researchers at Jilin University and Ant Group have developed a new method called CaSKG to solve this retrieval problem. Instead of treating skills as isolated pieces of text or simply connecting them based on how similar they sound, the team built a system that tests the reliability of the connections between skills before the agent ever uses them. They constructed a directed graph, a network where skills are nodes and the arrows between them represent the order in which actions should happen. The innovation lies in how they decided which arrows to keep. Before publishing the final map, the system subjected potential connections to a series of rigorous tests. It asked the artificial intelligence to imagine scenarios where a skill was removed, replaced with a different one, or performed in the wrong order. By observing how the task failed or became confused in these hypothetical situations, the system could determine if a connection was a true dependency or just a coincidence.

The researchers tested this approach on two distinct benchmarks: a set of household tasks involving finding and moving objects, and a collection of science experiments requiring specific sequences of observation and manipulation. They ran these tests using six different large language models, ranging from smaller, more efficient models to massive, powerful ones. The results showed that the new method consistently outperformed existing approaches. In every combination of model and task type, CaSKG achieved the highest success rate. For the science tasks, the average score across all models jumped from roughly 72.6 to 80.5, and for the household tasks, the success rate rose from 80.01% to 86.79%. Perhaps more importantly, the agents using this method reached their goals in fewer steps, suggesting they were not wasting time trying random actions or correcting mistakes caused by bad advice.

The key to this success was the ability of the system to filter out weak links. Previous methods often relied on graphs where connections were based on how often skills appeared together or how similar their descriptions were. The new study found that these methods often led the agent astray, causing it to follow irrelevant paths or miss critical prerequisites. By using counterfactual reasoning—essentially asking "what if this step didn't happen?"—the system could calibrate the confidence of each connection. It kept the strong, necessary links that formed a coherent workflow, such as the need to turn on a power source before testing a circuit, while discarding the weak associations that merely looked related. This allowed the agent to retrieve a compact, executable bundle of skills that preserved the logical order of operations, ensuring that preparation, action, verification, and completion steps were all present and in the right sequence.

The study also explored how this method held up as the library of skills grew larger. When the number of available skills increased from a few hundred to two thousand, the advantage of the new method remained stable, and in some cases, grew stronger. This suggests that as agents accumulate more knowledge, the need for a reliable way to navigate that knowledge becomes even more critical. The researchers found that the system worked best when it could draw on multiple types of evidence, such as the input and output requirements of a skill or its position in a workflow, rather than just the text description. They also discovered that while the method helped most tasks significantly, it was less critical for simple, direct searches where the answer was obvious, confirming that the value of the system lies in handling complex, multi-step procedures.

Ultimately, this work demonstrates that the quality of the connections between pieces of knowledge is just as important as the knowledge itself. By treating the retrieval process as a problem of verifying causal relationships rather than just matching words, the researchers created a framework that allows agents to access their memory more effectively. The method does not change how the agent acts or what tools it can use; it simply ensures that the right instructions are available at the right time. This approach offers a scalable path forward for building agents that can handle increasingly complex tasks, proving that a well-calibrated map of skills is more valuable than a larger, unorganized library.

Technical Summary: CaSKG – Counterfactual-Causal Skill Graphs for Scalable Agent Skill Retrieval

1. Problem Statement

Large Language Model (LLM) agents increasingly rely on reusable skill libraries to execute complex tasks by combining generation with external tools and APIs. However, as these procedural memories grow, the challenge shifts from whether an agent can use a tool to how it can retrieve the correct subset of skills without overwhelming the context window.

Existing retrieval strategies face distinct limitations:

Full-library prompting: Preserves recall but shifts the selection burden to the agent, introducing irrelevant alternatives and high context costs.
Vector retrieval (RAG): Reduces context size via semantic similarity but treats skills as independent text units. This fails to capture procedural dependencies where the utility of one skill relies on prerequisites, state changes, or verification steps performed by another.
Graph-based retrieval (e.g., Graph-of-Skills/GoS): Attempts to recover workflow context by propagating relevance through edges. However, these methods often rely on edges induced by topical similarity, co-occurrence, or loose interface compatibility. If these edges are unreliable (connecting alternatives or unordered neighbors), relevance propagation can pollute the retrieved context with operationally unsuitable skills.

The core bottleneck identified is not merely building a larger graph, but calibrating edge confidence: deciding which candidate relations deserve to influence retrieval and how much weight they should carry.

2. Methodology: CaSKG Framework

CaSKG (Counterfactual-Causal Skill Graph) is an offline framework that separates association discovery from edge reliability assessment. It constructs a state-filtered weighted graph before runtime, leaving the downstream agent policy and task interface unchanged. The process consists of four stages:

2.1 Candidate Skill Graph Induction

CaSKG first induces a high-recall, sparse directed candidate graph ( $C$ ) from heterogeneous evidence channels:

Signals: Lexical similarity, semantic embeddings, input/output (I/O) compatibility, structural workflow roles, and repair evidence.
Scoring: An initial association score ( $A_{ij}$ ) is computed as a weighted average of active signals. A "structural floor" ensures that strong workflow evidence (e.g., explicit I/O dependencies) is preserved even if other signals are weak.
Frontier Selection: A budgeted subset of edges ( $F \subseteq C$ ) is selected for validation, while the remainder remains unvalidated candidates.

2.2 Counterfactual Edge Probing

To distinguish operational dependencies from mere topical associations, CaSKG applies direction-conditioned textual counterfactual probes to the selected frontier ( $F$ ). For a hypothesized edge $s_i \to s_j$ , an LLM evaluates three scenarios:

Removal: What happens if the source skill $s_i$ is removed? (Measures necessity).
Substitution: What happens if $s_i$ is replaced with a dissimilar skill? (Measures specificity).
Reordering: What happens if the order is reversed ( $s_j \to s_i$ )? (Measures directionality).

The LLM returns support scores for each probe, indicating whether the hypothesized dependency holds under these interventions.

2.3 Bayesian Edge Calibration and Publication

The probe scores are aggregated into a smoothed reliability estimate using a Beta-distribution posterior.

Reliability Score ( $c_{ij}$ ): Derived from the Beta mean, summarizing the evidence for the directed relation.
State-Gated Publication: Edges are assigned a state based on the reliability score and a confirmation threshold ( $\tau_c$ $τ_{c}$ ):
- Confirmed: High reliability; retains full weight.
- Uncertain: Moderate reliability; downweighted.
- Rejected: Low reliability; removed.
- Unvalidated: Candidates not in the validation frontier; retained only as a low-weight "scaffold" for coverage if they pass a bounded subset check.
Final Graph ( $G_{pub}$ ): A weighted graph where edge weights reflect both discovery support and counterfactual reliability.

2.4 Task-Conditioned Skill Retrieval

At inference time, CaSKG uses $G_{pub}$ for retrieval:

Seeding: Lexical and semantic matching identifies initial task-relevant skills.
Propagation: A Personalized PageRank algorithm propagates relevance from these seeds over the calibrated graph.
Output: The agent receives a compact bundle of skills shaped by the calibrated procedural structure, rather than raw association strength alone.

3. Key Contributions

Formulation: The paper formulates scalable skill-graph construction as a budgeted edge-confidence calibration problem, balancing relational coverage against the risk of weak associations distorting graph propagation.
Methodology: It introduces CaSKG, an offline construction method combining multi-signal candidate induction with LLM-simulated counterfactual probes and Bayesian state-gated publication.
Evaluation: Comprehensive evaluation across six LLM backbones on two interactive benchmarks (ALFWorld ID-140 and ScienceWorld U211), demonstrating consistent improvements over full-library, vector, and existing graph baselines.

4. Experimental Results

The evaluation was conducted on a fixed Skill1000 library across 12 combinations of models and benchmarks.

Performance Gains: CaSKG achieved the highest task score in all 12 combinations.
- ScienceWorld: The six-model macro-average score improved from 72.62 (GoS) to 80.50.
- ALFWorld: Success rate improved from 80.01% (GoS) to 86.79%.
Efficiency: CaSKG reduced the mean number of environment interaction steps compared to GoS and other baselines in all settings, indicating that better retrieval leads to more direct execution paths.
Task-Type Analysis: Gains were most pronounced in tasks requiring chains of dependent operations (e.g., color mixing, plant growing, conductivity testing). Tasks requiring direct search or measurement showed smaller or negligible gains, suggesting graph expansion is most beneficial when procedural structure is critical.
Qualitative Insights: Case studies (e.g., conductivity testing, cooling and placement) showed that CaSKG successfully retrieved bundles preserving prerequisites, state-changing actions, and verification routines, whereas baselines often missed non-lexical dependencies or included irrelevant alternatives.
Ablation Studies:
- Scale: CaSKG maintained its advantage as library size increased from 200 to 2,000 skills.
- Components: Removing multi-signal induction (semantic-only) or the LLM judge reduced performance, confirming the value of diverse evidence and refined scoring. Publishing all candidates without calibration resulted in lower success rates, validating the necessity of the state-gated publication mechanism.

5. Significance and Claims

The paper positions edge-confidence calibration as an effective route to compact and executable skill retrieval at scale. The authors claim that:

Reliability over Coverage: Simply expanding the graph is insufficient; the quality of the edges carrying relevance is the critical factor.
Procedural Structure: Calibrated edges help retrieval preserve the operational structure (prerequisites, state changes, verification) required for complex tasks, which vector retrieval and uncalibrated graphs often fail to capture.
Generalizability: The improvements are consistent across models of varying capabilities, suggesting that the benefit stems from the retrieval mechanism itself rather than compensating for a specific model's weakness.
Offline Efficiency: The framework constructs the graph offline, requiring no changes to the downstream agent's policy or the task interface, making it a drop-in improvement for existing agent systems.

The work concludes that by separating association discovery from reliability assessment, CaSKG enables agents to access a "compact bundle shaped by validated procedural relations," effectively solving the trade-off between recall and noise in large skill libraries.