Technical Summary: A Single Rewrite Suffices: Empirical Lessons from Production Skill Description Optimization

Problem Statement

Modern enterprise AI assistants rely on Large Language Model (LLM) planners to route user queries to specialized skills based on natural language skill descriptions. A critical failure mode in these systems is skill collision, which occurs when two or more skills possess overlapping or insufficiently discriminative descriptions. This ambiguity causes the planner to misroute queries, particularly during the onboarding of new skills where descriptions must be manually tuned against existing incumbents.

As agents scale to dozens of skills, this manual tuning process becomes a significant engineering bottleneck. It requires iterative loops of writing descriptions, deploying, observing routing errors (from real or synthetic traffic), and editing, demanding expertise in both skill functionality and the planner's decision boundaries.

Methodology

The authors propose and deploy an automated description optimization pipeline designed to replace the manual tuning loop. The pipeline operates in two primary stages:

1. LLM Initialization: An LLM is prompted with a skill name to generate a candidate routing description. This serves as the starting point for refinement and a standalone baseline.

2. Error-Feedback Refinement: The candidate description is evaluated against a labeled training set containing ground-truth skill labels. The system collects:

   • False Positives (FP): Queries incorrectly routed to the skill.
   • False Negatives (FN): Queries the skill should have handled but missed.
   • True Positives (TP): Correctly routed queries (used to balance the prompt context).

   The LLM is then prompted with the current description and a sample of these error cases (up to 5 FP and 5 FN) to identify routing failure patterns and revise the description. This process can run iteratively up to a fixed budget or until a performance threshold is met.

The authors also evaluate a single-shot variant, where the LLM receives the initialized description and all available FP/FN cases in a single prompt to produce one revised description without further iteration.

The pipeline was tested in two environments:

• Production Setting: An enterprise group chat agent with 9 skills (covering people search, calendar, email, etc.) and 372 synthetic regression cases.
• ToolBench Setting: A large-scale evaluation across ~16k RESTful API tools, testing both closed-world (fixed candidate pools) and open-world (retrieval-based) routing regimes.

Key Results

1. Parity with Manual Tuning at Scale

On the production enterprise agent, the automated pipeline achieved routing quality statistically indistinguishable from manually tuned descriptions:

• Automated (Iterative): 79.2% average F1.
• Manual (Human): 79.4% average F1.
• Difference: -0.20%, which falls well within the multi-seed noise floor of ±0.78%.
• Efficiency: The automated pipeline reduced engineering effort from 120 minutes per skill to 3.8 minutes, representing a 32× speedup.

2. The "Single Rewrite" Finding

Systematic ablation studies on both the production system and ToolBench revealed that the complexity of the pipeline is largely unnecessary.

• Single-Shot Sufficiency: A single LLM rewrite using available FP/FN cases captured the vast majority of available improvements. On production, single-shot achieved 79.2% F1 (within 0.12% of human). On ToolBench open-world, it achieved a +4.45% F1 gain, within 0.2% of the best iterative refinement.
• Negligible Impact of Other Factors: Variations in feedback signal composition (FP+FN+TP vs. FP-only), iteration budget (5 vs. 10 vs. 30), dual editing of confused pairs, and training set size (5 vs. 10 vs. 20 examples) each affected the final F1 by less than 0.5%. The per-example error feedback signal is the dominant driver, not the packaging or iteration count.

3. Limitations and Diagnostics

The paper identifies clear boundaries for where description optimization applies:

• Regime Mismatch: Descriptions optimized for closed-world routing do not transfer well to open-world retrieval, and vice versa. LLM initialization helps in one setting but can hurt in the other.
• Genuine Scope Overlap: When two skills have genuinely overlapping intended scopes (rather than just poor wording), no description rewrite can resolve the collision. These cases are flagged by a large train–validation F1 gap, indicating a need for architectural intervention (e.g., scope restructuring) rather than text-level optimization.

Significance and Contributions

The paper makes three primary contributions to the field of LLM agent orchestration:

1. Operational Validation: It demonstrates a deployed production system that replaces manual description tuning at an enterprise scale, validated as non-inferior to human baselines while offering a 32× engineering speedup.
2. Empirical Simplification: Through rigorous ablation, the authors show that the elaborate pipeline mechanisms (iterative loops, complex feedback signals, dual editing) have negligible impact on performance. The core finding is that a single LLM rewrite using objective routing feedback suffices to capture most improvements.
3. Diagnostic Framework: The work characterizes the limits of description optimization, providing a diagnostic signal (the train-validation F1 gap) to distinguish between fixable description collisions and unfixable architectural scope overlaps. It also highlights the necessity of separating optimization regimes for closed-world and open-world routing.

In summary, the paper argues that for skill onboarding, the optimal operational strategy is to perform a single-shot LLM rewrite using error feedback, ensure the optimization regime matches the deployment regime, and use F1 gaps to triage skills requiring architectural changes.