🤖 AI

LatentSkill: From In-Context Textual Skills to In-Weight Latent Skills for LLM Agents

LatentSkill is a framework that converts textual skills into plug-and-play LoRA adapters via a pretrained hypernetwork, enabling LLM agents to store reusable task procedures in weight space rather than context space, which significantly reduces token overhead while improving performance and enabling modular skill composition.

Original authors: Aofan Yu, Chenyu Zhou, Tianyi Xu, Zihan Guo, Rong Shan, Zhihui Fu, Jun Wang, Weiwen Liu, Yong Yu, Weinan Zhang, Jianghao Lin

Published 2026-08-18

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Aofan Yu, Chenyu Zhou, Tianyi Xu, Zihan Guo, Rong Shan, Zhihui Fu, Jun Wang, Weiwen Liu, Yong Yu, Weinan Zhang, Jianghao Lin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Large language models have become remarkably capable at reasoning through complex problems, often acting as digital agents that can plan, search, and interact with software to complete tasks. To handle specialized jobs, these agents frequently rely on "skills"—essentially written instructions or procedures that tell the model how to solve a specific type of problem, such as organizing a virtual room or answering a multi-step question. Traditionally, these skills are fed directly into the model's input every time it needs to use them, much like reading a manual page by page before taking an action. While this works, it creates a significant bottleneck: the more complex the task and the larger the library of skills, the more text the model must process, which slows it down and consumes vast amounts of computing power. Furthermore, keeping these instructions as plain text in the input leaves them exposed and vulnerable to being misread or hijacked by other parts of the system.

Researchers at Shanghai Jiao Tong University and OPPO Research Institute have proposed a different approach called LatentSkill, which moves these instructions out of the text input and into the model's internal memory structure. Instead of reading a skill description every time, the system uses a specialized generator to convert the text of a skill into a compact set of internal adjustments, known as adapters, that are permanently attached to the model's brain. This allows the agent to "know" the skill without having to read the instructions again. The team found that this method not only speeds up the process and reduces the amount of data the model needs to handle, but also creates a hidden, organized space where different skills can be mixed and matched with mathematical precision.

The core idea behind this work is to treat a skill not as a document to be read, but as a set of weights to be loaded. In their framework, a "skill compiler" takes a written description of a task and instantly generates a unique set of internal modifications for the language model. These modifications are small, efficient, and modular, meaning they can be attached or detached from the main model without retraining the entire system. Once a skill is converted into this internal format, the original text is discarded from the input stream. The model then operates using these hidden adjustments, which guide its behavior just as effectively as the text would have, but without the heavy cost of processing thousands of extra words for every single step of a task.

To test this, the researchers applied their system to two distinct challenges: a text-based simulation of a robot navigating a house to complete chores, and a series of complex question-answering tasks that require searching through multiple sources of information. In the house navigation task, the system had to figure out how to pick up objects, clean them, and place them in specific locations. In the question-answering task, it had to find answers that required connecting several pieces of information. When compared to the standard method of pasting skill instructions into the prompt, the new approach proved significantly more efficient. On the house navigation tasks, the system improved its success rate by over twenty percentage points on familiar tasks and by more than thirteen points on new, unseen tasks, all while using nearly two-thirds fewer input tokens. On the question-answering benchmarks, it achieved a higher accuracy rate while reducing the number of tokens processed per step by more than seventy percent.

Beyond just saving time and resources, the researchers discovered that these internal skill adjustments possess a surprising amount of structure. When they mapped out the mathematical space where these skills live, they found that skills belonging to the same category, such as cleaning tasks or search tasks, naturally grouped together, forming distinct clusters. This suggests that the system is not just memorizing text, but is learning the underlying logic of the procedures. This structure allows for a high degree of control; the researchers found that they could adjust the strength of a skill simply by turning a dial, represented by a scaling number. If the number was too low, the skill had no effect; if it was too high, the model became confused. However, at the right setting, the model performed optimally, and this sweet spot remained consistent across different types of tasks.

Perhaps the most intriguing finding was that these internal skills could be combined. Just as one might mix ingredients to create a new dish, the researchers showed that they could take the internal adjustments for two different skills and add them together to create a new, composite skill. However, this only worked reliably when the skills were broken down into their smallest, most aligned components before being mixed. If they simply added the whole skill blocks together, the result was often less effective. By carefully aligning the parts, they were able to create a skill that performed a complex sequence of actions without losing the ability to perform the individual steps. This suggests that the system can build complex behaviors by assembling modular pieces of knowledge in a way that is both flexible and precise.

The study also highlighted a significant security advantage. Because the skills are stored as internal adjustments rather than visible text, they are much harder to tamper with. When the researchers tested the system against attempts to hijack the instructions or extract the hidden knowledge, the new method held up remarkably well. While the traditional text-based approach often collapsed or revealed its secrets when attacked, the internal skill version maintained its performance and kept its instructions hidden. This indicates that moving skills from the visible prompt into the model's internal weights offers a more robust and secure way to equip artificial agents with specialized knowledge.

The researchers conclude that this approach offers a practical path forward for building more capable and efficient AI agents. By shifting the storage of procedural knowledge from the input stream to the model's internal parameters, they have demonstrated a way to make agents faster, more secure, and better at combining different abilities. The results suggest that the future of AI agents may not lie in feeding them more text, but in teaching them to carry their skills internally, ready to be deployed with a simple, invisible switch.

Technical Summary: LatentSkill

Problem Statement

Large Language Model (LLM) agents increasingly rely on external textual skills—reusable procedures encoding strategies, tool-use patterns, and heuristics—to solve complex, long-horizon tasks. The prevailing design retrieves these skills from a library and injects them as plaintext into the prompt at every decision step. While modular, this approach suffers from three critical limitations:

Context Overhead: Repeatedly inserting skill text consumes significant context window space and increases prefill costs, particularly as skill libraries grow and interactions lengthen.
Robustness: Long inputs can degrade a model's ability to robustly utilize all supplied information.
Security: Keeping skills as readable prompt content exposes proprietary procedures and creates vulnerabilities to instruction hijacking or skill extraction attacks.

Conversely, parametric alternatives like full fine-tuning or curriculum learning fuse skills into model weights, rendering them difficult to update, remove, or combine modularly. This creates a trilemma: existing methods struggle to avoid repeated skill text while maintaining the ability to update and combine skills dynamically during inference.

Methodology: LatentSkill

LatentSkill proposes a framework that converts textual agent skills into modular LoRA (Low-Rank Adaptation) adapters via a pretrained hypernetwork, effectively moving skill knowledge from context space to weight space.

Core Architecture

Skill Compiler (Hypernetwork): A Transformer-based hypernetwork $G_\phi$ maps a textual skill document $s$ to a set of LoRA updates $\Delta_s$ .
$\Delta_s = G_\phi(s)$
The generated update follows the standard LoRA form $\Delta W_s = B_s A_s$ for target modules (query, key, value, output, gate, up, down projections).
Inference Mechanism: Instead of conditioning the model on the skill text within the prompt, the generated adapter is mounted on the frozen backbone LLM $\theta$ . The model predicts outputs based on the task history $h_t$ and the augmented weights:
$p_{\theta, \phi}(y_t | h_t, s) = p_{\theta \oplus \alpha \Delta_s}(y_t | h_t)$
Here, $\alpha$ is a scaling coefficient controlling the injection strength of the skill.

Training Pipeline

The framework employs a two-stage training process for the skill compiler while keeping the backbone LLM frozen:

Document-Level Pretraining: The compiler is trained on a corpus of textual skill documents ( $D_{pre}$ ) using a 1:1 mixture of reconstruction (reproducing the full document) and completion (predicting the truncated suffix) objectives. This initializes the mapping from procedural text to usable adapter weights.
Trajectory-Supervised Fine-Tuning (SFT): The compiler is fine-tuned using teacher agent trajectories ( $D_{sft}$ ). For each skill-document and trajectory pair, the same generated adapter is applied across all decision steps. The objective minimizes the loss between the adapted model's output and the teacher's trajectory, encouraging the adapter to capture stable, skill-level policy information rather than per-step adaptations.

Inference-Time Control and Composition

Modular Loading: Skills are compiled once and cached. At inference, a selector mounts the relevant adapter(s) without including the skill text in the prompt.
Weight-Space Composition: Multiple skills can be combined by summing their LoRA updates: $\Delta_K = \sum \alpha_k C[k]$ .
Component-Level Composition: To avoid over-amplifying shared behaviors, skills can be decomposed into semantic components (e.g., general principles vs. specific strategies), which are compiled independently and recombined with specific injection coefficients.

Key Contributions

Framework Proposal: The introduction of LatentSkill, which converts textual skills into modular, plug-and-play LoRA adapters via a hypernetwork, eliminating the need for skill text in the inference prompt.
Performance and Efficiency: Demonstration that LatentSkill outperforms direct in-context skill prompting on ALFWorld and Search-QA benchmarks while significantly reducing context overhead (prefill tokens).
Structural Analysis: The finding that generated skill LoRAs exhibit a structured semantic geometry, allow for continuous control via scaling coefficients, and support composable arithmetic in parameter space when components are semantically aligned.

Experimental Results

The framework was evaluated on ALFWorld (embodied AI tasks) and Search-QA (multi-hop question answering), using Qwen3-8B as the frozen backbone.

ALFWorld Performance: LatentSkill achieved success rates of 74.3% (seen) and 69.4% (unseen). This represents a 21.4 and 13.4 point improvement over the in-context skill baseline, respectively. It also reduced the average prefill token count by 63.9%.
Search-QA Performance: LatentSkill achieved an average Exact Match (EM) of 35.6%, outperforming RAG by 1.2 points and the in-context skill baseline by 3.0 points, while using 71.8% fewer tokens per step.
Out-of-Domain (OOD) Generalization: On the WebShop benchmark (without training trajectories), LatentSkill (SFT) scored 23.9, significantly outperforming the in-context skill baseline (6.7), suggesting that compiling skills into latent space mitigates the attention dilution caused by long prompts.
Robustness: LatentSkill maintained consistent performance under skill text perturbations (paraphrasing, noise, reordering) and prompt-level attacks (hijacking, extraction), whereas in-context skills suffered severe performance drops (e.g., dropping from 52.9% to 8.57% under hijacking on ALFWorld).

Significance and Claims

The paper posits that LatentSkill offers a practical substrate for extending LLM agents by addressing the efficiency, modularity, and security trade-offs of current skill systems.

Efficiency: By storing skills in weight space, the framework removes the per-step cost of tokenizing and processing skill text, enabling faster inference and lower context usage.
Modularity: Unlike fine-tuning, skills remain separate from the backbone, allowing them to be loaded, unloaded, replaced, or scaled without retraining the base model.
Structured Representation: The generated LoRA weights are not merely compressed prompts; they form a structured semantic geometry where skills cluster by domain. This structure enables:
- Controllability: The injection strength of a skill can be continuously modulated via the scaling coefficient $\alpha$ .
- Composability: Skills can be arithmetically combined in parameter space, provided their semantic components are aligned, allowing for the creation of complex behaviors from simpler modules.

The authors conclude that latent skill weights provide an efficient, modular, and less exposed alternative to plaintext prompts for building robust LLM agents.