Technical Summary: Steering Black-Box LLMs with ADVISOR MODELS

Problem Statement

Frontier large language models (LLMs) such as GPT-5 and Claude 4.5 are increasingly deployed as black-box API services where model weights are inaccessible. This architecture renders traditional fine-tuning impossible. While static prompt optimization (e.g., finding a single fixed prompt for all instances) offers a workaround, it fails to adapt to diverse inputs, is constrained by context-length limitations, and cannot effectively handle deep specialization (e.g., complex rule-following, low-resource languages) or dynamic personalization (e.g., heterogeneous user preferences). Existing methods for training lightweight models to modify prompts often rely on expert labelers, produce static outputs for entire problem classes, or utilize uninterpretable soft prompts. There is a critical need for a method to parametrically optimize black-box models without weight access, capable of generating instance-specific guidance that improves performance while maintaining robustness.

Methodology: ADVISOR MODELS

The paper introduces ADVISOR MODELS, a framework that trains a lightweight, open-weight "advisor" model to generate dynamic, per-instance natural language advice. This advice is injected into the context of a frozen black-box "student" model to steer its behavior.

Core Architecture

1. Pipeline: The advisor sits between the user input and the black-box student. It receives the task prompt (and optionally, initial student attempts or environmental observations) and generates specific guidance.
2. Training via Reinforcement Learning (RL): The system treats prompt engineering as an RL problem.
   • The advisor samples candidate advice.
   • The black-box student generates an output conditioned on this advice.
   • A task-specific reward is computed based on the final output (e.g., correctness, efficiency, preference alignment).
   • The advisor is updated using Group Relative Policy Optimization (GRPO) based solely on these observed rewards. No gradients from the black-box model are required.
3. Design Variants:
   • 2-Step: Advisor generates advice based on the input prompt; student generates the final response.
   • 3-Step: Student generates an initial attempt; advisor provides verification/correction advice; student generates a revised response. This variant simplifies the advisor's role to that of a verifier, proving effective for complex tasks like tax rule-following.
   • Multi-Turn: For agentic tasks (e.g., software engineering), the advisor can provide iterative advice based on the student's actions and environmental observations over multiple turns.

Key Mechanisms

• Parametric Optimization: Unlike static prompts, the advisor learns a policy to generate custom advice for every instance.
• Transferability: Advisors are trained on low-cost student models (e.g., GPT-4o mini, Gemini 2.5 Flash) and can be transferred to improve frontier models (e.g., GPT-5, Gemini 3 Pro) without retraining.
• Robustness: Because the black-box model's weights remain unchanged, the system avoids catastrophic forgetting and maintains performance on untrained tasks.

Key Contributions

1. Novel Paradigm: Introduction of a trainable policy (the advisor) to dynamically steer frozen black-box models, reframing prompting from a static search problem to a learning-to-advice problem.
2. Performance Gains: Demonstration that ADVISOR MODELS significantly outperform static prompt optimizers (like GEPA and Profile-Augmented Generation) in specialized reasoning and personalization tasks.
3. Transferability and Cost-Efficiency: Proof that advisors trained on inexpensive models transfer substantial improvements to frontier models, enabling cost-effective optimization without direct interaction with expensive APIs during training.
4. Robustness: Empirical verification that the method improves specific capabilities without degrading general performance on untrained benchmarks.

Experimental Results

Specialized Reasoning

The authors evaluated ADVISOR MODELS on three domains where frontier models lack specific domain knowledge:

• RuleArena (Taxes): An advisor trained on GPT-4.1 mini improved GPT-5.2's accuracy on tax filing from 67.2% to 85.6% (+27.4%). Static optimizers failed to match the standalone baseline.
• SWE Agent Efficiency: Using Gemini 2.5 Flash as the student, the trained advisor reduced the average steps taken by Gemini 3 Pro to resolve software issues from 31.7 to 26.3 (a 24.6% reduction) while maintaining the same resolution rate.
• MTOB (Machine Translation): For translating the low-resource Kalamang language, the advisor lifted GPT-4o mini's performance from a chrF score of 33.1 to 45.8, approaching human expert performance (51.6).

Personalization

In tasks requiring the learning of hidden user preferences (Review Length, Review Level, Math Solutions):

• ADVISOR MODELS achieved near-perfect alignment with latent preferences (e.g., 0.94 reward for review length, 99.6% accuracy for review level).
• Static optimizers (GEPA, PAG) and even in-context oracle prompts (providing full preference details) failed to match this performance, often achieving only marginal improvements over baselines.
• Advisors trained on GPT-4o mini successfully transferred to personalize GPT-5 outputs.

Ablation Studies

• Transferability: Advisors trained on one model family (e.g., GPT) successfully improved models from another (e.g., Claude), demonstrating the interpretability of the generated advice.
• Robustness: Pipelines trained on personalization tasks showed no degradation in accuracy on unrelated math benchmarks (MATH-500) when applied to frontier models.
• Limits: The method provides no significant gain on domains where frontier models are already "maxed-out" (e.g., standard math problem solving), confirming that the advisor learns only where knowledge gaps exist.

Significance and Claims

The paper posits that ADVISOR MODELS offer a practical and cost-effective solution for the growing challenge of customizing black-box frontier models. By decoupling the optimization process (training the advisor) from the inference target (the black-box student), the method enables:

• Specialization: Enhancing capabilities in domains with sparse pretraining data (e.g., specific tax laws, low-resource languages).
• Personalization: Learning complex, instance-specific user preferences that static prompts cannot capture.
• Safety and Stability: Avoiding the risks associated with direct fine-tuning, such as catastrophic forgetting, by keeping the base model immutable.

The authors emphasize that while direct reinforcement learning on open-weight models remains the "gold standard" for performance, ADVISOR MODELS recover the majority of these gains in black-box settings, making high-level optimization accessible without API-level weight access. The work suggests a new paradigm where "learning to advise" becomes the primary mechanism for adapting frontier AI systems to specific, evolving needs.