💬 NLP

SkillFlow: Scalable and Efficient Agent Skill Retrieval System

SkillFlow introduces a scalable, four-stage retrieval pipeline that significantly improves AI agent performance on coding benchmarks by effectively selecting relevant skills from a large community repository, while highlighting that the system's ultimate success depends heavily on the quality and executability of the available skill corpus.

Original authors: Fangzhou Li, Pagkratios Tagkopoulos, Ilias Tagkopoulos

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Fangzhou Li, Pagkratios Tagkopoulos, Ilias Tagkopoulos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, super-smart robot assistant (an AI Agent) that can do amazing things: write code, solve math problems, and automate tasks. But there's a catch. This robot has a very short attention span (called "context window"). If you try to stuff its brain with a thousand different instruction manuals at once, it gets confused, slows down, and makes mistakes.

However, if you give it the one perfect manual for the specific job at hand, it becomes a superhero.

The problem? There are now 36,000+ different instruction manuals (called "Skills") written by people all over the world, stored in a giant digital library. How do you find the exact right one for a specific task without drowning the robot in useless information?

Enter SkillFlow. Think of SkillFlow as a super-efficient librarian designed specifically for AI agents.

The Problem: The "Too Many Cooks" Dilemma

Imagine you are baking a cake.

The Old Way: You dump the entire library of cookbooks onto your kitchen counter. You have a book on "How to bake a cake," but also 5,000 books on "How to fix a car," "How to knit," and "How to bake a cake using only blueberries." Your robot assistant tries to read them all, gets overwhelmed, and burns the cake.
The Goal: You need a system that instantly grabs only the "Standard Chocolate Cake" book and ignores the rest.

The Solution: The SkillFlow Pipeline

SkillFlow acts like a four-stage security checkpoint at an airport. It starts with a huge crowd of candidates and progressively narrows them down to the VIPs.

Stage 1: The Wide Net (Dense Retrieval)

The Metaphor: Imagine throwing a giant fishing net into the ocean. You don't care if you catch a shark, a tuna, or a plastic bottle; you just want to catch everything that might be a fish.
What happens: SkillFlow uses a fast, broad search engine to pull 1,000 potential skills from the library of 36,000. It's fast but a bit messy. It might grab a book on "baking" when you actually needed "decorating."

Stage 2: The Shallow Check (First Rerank)

The Metaphor: A security guard quickly glances at the bags of the 1,000 people. They don't open the bags fully; they just look for obvious red flags or clear matches.
What happens: A smarter (but still quick) AI looks at the top 1,000 skills and cuts the list down to 100. It realizes, "Hey, that book is about knitting, not baking. Get rid of it."

Stage 3: The Deep Dive (Second Rerank)

The Metaphor: Now, a specialist opens the top 100 bags and reads the first few pages of every book inside. They are looking for deep details.
What happens: A very powerful AI reads the full content of the remaining 100 skills. It cuts the list down to just 10. It finds the specific book that says "How to frost a cake with buttercream," ignoring the one that just says "How to bake a cake."

Stage 4: The Final Decision (LLM Selector)

The Metaphor: The head chef tastes the top 10 dishes and decides which one is actually ready to serve.
What happens: The final AI (a Large Language Model) looks at the last 10 skills and asks: "Is this actually useful for this specific task, or is it just generic advice?" It picks the final 1 to 5 skills to give to the robot agent.

The Big Surprise: It's Not Just About Finding the Book

The researchers tested this system in two different worlds:

World A (SkillsBench): A library full of high-quality, working recipes.
- Result: When SkillFlow found the right skills, the robot's success rate jumped by 78%. It was like giving a chef a perfect recipe instead of guessing.
World B (Terminal-Bench): A library where the "recipes" were mostly empty boxes or instructions that didn't actually work (e.g., "Bake the cake" but no oven instructions).
- Result: Even though SkillFlow found the "right" books, the robot didn't get any better.
- The Lesson: Finding the right book is useless if the book itself is broken. The biggest bottleneck isn't the librarian (SkillFlow); it's the quality of the books in the library.

The "Secret Sauce" of Good Skills

The paper discovered what makes a "good" skill versus a "bad" one.

Bad Skills: Long paragraphs of text, lists of ideas, and vague advice. (Like a book that says "You should bake a cake.")
Good Skills: Short, punchy, and packed with executable code and ready-to-use scripts. (Like a book that says "Here is the exact code to mix the batter and bake at 350°F.")

Summary

SkillFlow is a brilliant system that teaches AI agents how to be picky eaters. Instead of eating everything in the buffet, it uses a smart, multi-step filter to find the single best dish.

However, the paper teaches us a crucial lesson: You can have the best waiter in the world, but if the kitchen only serves burnt toast, the customer will still be unhappy. For AI agents to truly shine, we need more people to write high-quality, working "recipes" (skills) for them to use.

1. Problem Statement

AI agents, particularly those based on Large Language Models (LLMs), can significantly enhance their capabilities by loading reusable "skills" (structured procedural knowledge) into their context at inference time. However, a critical bottleneck exists:

Context Overload: Equipping an agent with too many skills, especially irrelevant ones, degrades both accuracy and efficiency due to context window limitations and distraction.
Scalability Gap: As community-driven repositories (e.g., GitHub) grow to contain tens of thousands of skills (specifically SKILL.md definitions), agents lack a mechanism to selectively retrieve the most relevant skills from a massive library.
Current Limitations: Existing approaches often rely on brute-force enumeration, small manually curated sets, or assume a fixed capability set. There is no scalable, multi-stage retrieval pipeline designed specifically for heterogeneous, community-contributed agent skills.

2. Methodology: The SkillFlow Pipeline

SkillFlow frames agent skill acquisition as an Information Retrieval (IR) problem. It processes a task description against a corpus of ~36,000 community-contributed skills using a four-stage progressive narrowing pipeline designed to balance recall (finding all relevant skills) and precision (filtering out noise) while managing latency.

A. Data Corpus

Source: Skills collected from SkillsMP, an open marketplace, and indexed from GitHub repositories.
Scale: The final library contains 35,866 valid SKILL.md files.
Format: Each skill includes a name, natural language description, YAML metadata, and instructional content (often containing code blocks).

B. The Four-Stage Pipeline

Dense Retrieval (Recall-Oriented):
- Mechanism: Uses a bi-encoder (specifically BAAI/bge-base-en-v1.5) to generate dense embeddings for task queries and skill descriptions.
- Query Strategy: Generates 5 diverse queries per task (covering core domain, language/framework, libraries, and tools) to cast a wide net.
- Output: Retrieves the top 1,000 candidates from the 36K library.
Shallow Reranking (Efficiency-Oriented):
- Mechanism: Uses a lightweight cross-encoder (ms-marco-MiniLM-L-6-v2) on truncated content (512 tokens).
- Query Strategy: Uses a single focused query (M=1) to maximize ranking quality on the reduced set.
- Output: Narrows the set to the top 100 candidates.
Deep Reranking (Precision-Oriented):
- Mechanism: Uses a heavier, more capable cross-encoder (BAAI/bge-reranker-v2-m3) on full content (truncated at 4,096 tokens).
- Query Strategy: Uses a single focused query (M=1).
- Output: Narrows the set to the top 10 candidates.
Final Selection (Semantic Filtering):
- Mechanism: An LLM (gpt-4o-mini) acts as a binary filter to evaluate topical relevance and specificity.
- Output: Returns a variable number of skills (typically ≤5) that are deemed actionable and relevant.

C. Query Generation

The system uses an LLM to decompose a task into specific search queries.

Retrieval Stage: Generates multiple queries to cover different abstraction layers (maximizing recall).
Reranking Stages: Generates a single, precise query to maximize discrimination among candidates.

3. Key Contributions

First IR Framework for Agent Skills: This is the first work to formally frame agent skill acquisition as an information retrieval problem over a large, heterogeneous corpus.
Multi-Stage Pipeline: Proposes a novel architecture (Dense Retrieval $\to$ Shallow Rerank $\to$ Deep Rerank $\to$ LLM Selection) that effectively scales to 36K+ skills while maintaining interactive latency (~35 seconds total).
Open Skill Library: Constructed and released an openly available library of ~36K community-contributed SKILL.md definitions.
Empirical Insight on Bottlenecks: Demonstrated through contrasting benchmarks that retrieval quality is not the primary bottleneck; rather, the quality and coverage of the underlying skill library (specifically the density of runnable code and bundled artifacts) are the limiting factors for agent performance.

4. Experimental Results

The system was evaluated on two coding benchmarks: SkillsBench (87 tasks with known "oracle" skills) and Terminal-Bench (89 tasks without known oracle skills).

A. Downstream Performance (Task Completion)

SkillsBench (High Coverage):
- SkillFlow improved Pass@1 from 9.2% to 16.4% (+78.3% relative increase), reaching 84.1% of the theoretical "oracle" ceiling (19.5%).
- This improvement was statistically significant ( $p_{adj} = 3.64 \times 10^{-2}$ ).
Terminal-Bench (Low Coverage):
- Despite a high skill usage rate (70.1%), SkillFlow showed no performance gain over the baseline.
- Conclusion: The library lacked high-quality, executable skills for the specific domains in Terminal-Bench. Retrieval alone cannot compensate for a poor corpus.

B. Retrieval Quality

Recall: The dense retriever successfully captured 90.5% of oracle skills within the top 1,000 candidates.
Precision: The full pipeline retained 45.5% of oracle skills in its top-5 selection.
Comparison: SkillFlow significantly outperformed a commercial baseline (Vercel's API) in both retrieval rates and agent usage rates (e.g., 70.1% usage vs. 22.3% for Vercel on Terminal-Bench).

C. Structural Analysis of Skill Quality

A comparison between "oracle" skills (high-performing) and "community" skills revealed:

Code Density: Oracle skills had significantly more fenced code blocks (median fraction 0.39 vs. 0.24).
Bundled Artifacts: Oracle skills were 3x more likely to include executable scripts alongside the SKILL.md.
Length/Docs: There was no significant difference in document length or general documentation volume.
Implication: Effective skills are defined by executable content, not just text descriptions.

5. Significance and Future Directions

Paradigm Shift: SkillFlow proves that treating skill acquisition as an IR problem is a viable and effective strategy for scaling agent capabilities.
Practical Constraints: The study highlights that the "ceiling" of agent performance is currently determined by the quality of the skill library (specifically the presence of runnable code and artifacts) rather than the sophistication of the retrieval algorithm.
Latency vs. Quality: The pipeline successfully manages the trade-off between latency and precision, with the deep reranker being the most expensive stage but providing the most significant quality lift.
Future Work: Suggests expanding libraries via automated extraction, enabling peer-to-peer skill sharing, and exploring interleaved retrieval (retrieving skills dynamically during task execution rather than just at the start).

In summary, SkillFlow provides a robust, scalable infrastructure for agent skill retrieval but simultaneously delivers a critical warning: without high-quality, executable skills in the corpus, even the best retrieval system cannot improve agent performance.