Generative Prompt Internalization

1. Problem Definition

Large Language Model (LLM) applications, particularly agent-based systems, often rely on fixed, lengthy, and complex prompts to guide model behavior. While these prompts improve performance, they introduce significant computational overhead:

• Inference Cost: Long prompts increase token counts, leading to higher latency and computational costs (FLOPs/MACs), especially in multi-turn interactions.
• Limitations of Existing Solutions:
  • Prompt Compression: Methods like text compression or embedding caching reduce token counts but often lose critical semantic details or format constraints required for complex agent tasks.
  • Standard Distillation/Fine-tuning: Traditional knowledge distillation trains a student model to mimic a teacher's output given a prompt. However, the student model never sees the prompt content during training, relying only on the output distribution. This leads to performance degradation in information-rich tasks where specific formatting or schema grounding is required.

Goal: To enable an LLM to internalize a specific, predetermined prompt so it can replicate the teacher model's behavior without the prompt being explicitly provided during inference, while maintaining high performance and efficiency.

2. Methodology: Generative Prompt Internalization (GenPI)

GenPI is a lightweight training approach that forces the model to generate the prompt content and the reasoning behind behavioral changes alongside the task output.

A. Core Training Objective

GenPI employs a joint loss function combining two components:

1. Supervised Fine-Tuning (SFT) Loss ($L_{SFT}$): The student model mimics the teacher model's output ($y^T$) given the input ($x$). This ensures the model learns the correct task behavior.
2. Prompt Generation Loss ($L_{PG}$): The model is trained to generate:
   • The Prompt ($p$): The actual content of the instructions.
   • The Reason ($r$): An explanation of why the student's initial output (without the prompt) differs from the teacher's output (with the prompt).
   • Formula: $L_{PG} = -\log P(p, r | y^S, y^T, x)$, where $y^S$ is the "AS-IS" (student without prompt) and $y^T$ is the "TO-BE" (teacher with prompt).

The final objective is: $L = (1 - \lambda) L_{PG} + \lambda L_{SFT}$.

B. Data Synthesis: Self Role-Playing Conversation

A key challenge is that internalization often occurs in scenarios where only the prompt is available, and no conversational training dataset exists. GenPI introduces a technique to autonomously generate training data:

1. Pseudo User Input: A large language model generates hypothetical user queries ($x$) based on the prompt's context.
2. Self Role-Playing: To generate multi-turn conversation data ($y$) without interacting with a real environment, the model swaps roles. The same model acts as both the Agent and the Environment (e.g., the OS or Web Browser) by reversing the roles in the prompt context. This simulates a full dialogue trajectory.
3. Reason Generation: A separate LLM generates the "Reason" ($r$) by comparing the "Bad Output" (student without prompt) and "Good Output" (teacher with prompt), explaining the necessary behavioral shift.

C. Implementation

• Model: Uses QLoRA for efficient fine-tuning, requiring only 0.5% additional parameters (adapters) per prompt.
• Scale: Trained on just 1,000 pseudo-samples generated from the prompt itself.

3. Key Contributions

1. Generative Internalization: Unlike prior methods that treat prompts as static inputs or compress them, GenPI trains the model to generate the prompt content and rationale, allowing it to "remember" the instructions implicitly.
2. Self-Role-Playing Data Synthesis: A novel method to create high-quality, multi-turn training datasets for prompt internalization without requiring access to real-world environment interactions or existing datasets.
3. Efficiency vs. Performance Trade-off: Demonstrates that prompts can be fully internalized, eliminating the need for explicit prompt tokens during inference, thereby drastically reducing computational overhead.

4. Experimental Results

The method was evaluated on AgentBench across three tasks: OS Interaction, Web Browsing, and Web Shopping.

• Performance Retention:
  • OS Interaction: Achieved 100% of the upper-bound performance (matching the teacher model with the full prompt).
  • Web Browsing & Shopping: Maintained >82% performance retention despite prompts exceeding 1,000 tokens.
  • Comparison: Significantly outperformed prompt compression baselines (LLMLingua-2, ICAE) and standard distillation methods, which suffered severe performance drops when prompts were removed or compressed.
• Efficiency Gains:
  • GenPI reduced computational overhead by 39% in MACs and FLOPs compared to baselines using prompt caching.
  • Reduced inference latency by 17%.
  • Unlike compression methods, GenPI's efficiency gains do not degrade as conversation turns increase, as it requires zero prompt tokens.

5. Significance and Impact

• Scalability for Agents: GenPI solves the bottleneck of long context windows in agent applications, enabling efficient, multi-turn interactions without the cost of re-sending complex system instructions.
• Cost-Effectiveness: By removing the need for explicit prompts during inference, service providers can significantly reduce token usage costs and latency.
• Generalizability: The "Self Role-Playing" data synthesis technique offers a pathway to internalize prompts for any application where only the instruction set is known, bypassing the need for large-scale, curated interaction datasets.
• Future Directions: The authors suggest extending this approach to long chat histories, Retrieval-Augmented Generation (RAG), and multimodal contexts.

In conclusion, GenPI represents a paradigm shift from "prompt engineering" (managing external inputs) to "prompt internalization" (embedding instructions into the model's weights), offering a highly efficient solution for deploying complex LLM agents in production.