SkillGate: Training In-Policy Skill Selection in Long-Horizon Agents

In the modern landscape of artificial intelligence, large language models have evolved from passive text generators into active agents capable of navigating complex digital environments. These systems do not merely describe the world; they interact with it, using software tools to repair code, manage files, and solve problems through a series of reasoning steps and actions. To handle the vast array of tasks they face, developers have begun packaging procedural knowledge into reusable modules called skills. Imagine a digital library containing thousands of these skills, each with a name and a brief description, waiting to be opened by the agent when needed. The critical challenge for these agents is not just having access to this library, but knowing which specific file to open at the right moment. This decision must be made in the middle of a task, often based on a single line of description, before the agent can see the full content of the file. If the agent picks the wrong skill, the entire effort can fail, yet until now, there has been no reliable way to teach the agent how to make this choice correctly.

Researchers have long assumed that the standard method for training these agents would suffice: present the agent with a list of candidate skills, let it try to solve a task, and reward it only if the final outcome is successful. The idea was that the agent would naturally learn to pick the right skill because that choice leads to a good result. However, a new study reveals that this approach contains a fundamental flaw that prevents agents from learning to select skills effectively, especially in long and complex tasks. The researchers found that when an agent fails to complete a task, the training signal that tells it what went wrong is broadcast equally across every single word the agent generated. In a long sequence of actions, the few words that named the chosen skill are drowned out by the thousands of words used for reasoning and execution. Consequently, the agent receives almost no feedback about its initial choice. Worse, if the agent picked the correct skill but failed later due to a different mistake, the training system punishes the initial choice, teaching the agent that the right decision was actually wrong. This phenomenon, which the authors call "selector credit starvation," means that the most valuable decision in a long sequence is often the one that receives the least and most misleading instruction.

To solve this, the research team developed a new training method called SkillGate. Instead of treating the entire sequence of actions as a single block, SkillGate splits the learning process into two separate channels. One channel evaluates the execution of the task, rewarding or punishing the agent based on whether the final goal was achieved. The other channel focuses exclusively on the moment the agent names a skill. This second channel looks only at the specific words used to identify the chosen file and asks a simple question: was this the single correct file for the job? If the agent picked the right skill and did so only once, it receives positive feedback for that specific choice, regardless of whether the rest of the task succeeded or failed. If it picked the wrong file or picked multiple files, it receives negative feedback. By isolating the decision to name a skill from the outcome of the task, the system ensures that the agent learns to distinguish between a good choice that led to a bad outcome and a bad choice that led to a bad outcome.

The results of this approach were tested across five different benchmarks involving complex agent tasks. The researchers trained a model with 9 billion parameters using this new method and compared it against models trained with the traditional outcome-only approach. The SkillGate-trained model achieved a success rate of 53.2 percent, significantly outperforming the 47.0 percent success rate of the model trained only on final outcomes. This improvement was not just a matter of the model becoming generally smarter; the behavior changed in a specific and measurable way. The SkillGate model was far less likely to read misleading or irrelevant skills, cutting its exposure to bad options by two-thirds, and it read fewer skills overall, indicating a more decisive and accurate selection process. The study also showed that simply making the model larger did not solve the problem; even much larger models failed to learn the skill selection task as effectively as the smaller model trained with SkillGate.

The significance of this work lies in its demonstration that the way an agent learns is just as important as the data it sees. The researchers verified their diagnosis by auditing the training data of models trained with the old method. They found that as tasks got longer, the share of the learning signal that reached the skill-naming words shrank dramatically, becoming nearly invisible. Furthermore, the signal that did arrive was often the opposite of what it should have been, punishing correct choices simply because the task eventually failed. By constructing a system where the choice of a skill is judged on its own merits, separate from the execution of the task, the researchers removed this structural barrier. The new method allows the agent to learn that picking the right tool is a distinct and valuable skill in itself, one that deserves its own clear and direct instruction. This finding suggests that for agents to become truly reliable in complex environments, their training must be precise enough to distinguish between the decision to act and the result of the action.