Technical Summary: FAPO (Fully Autonomous Prompt Optimization)

Problem Statement

Multi-step Large Language Model (LLM) pipelines, which integrate retrieval, reasoning, and formatting steps, frequently fail due to complex interactions between these components. Traditional prompt optimization methods often fail to address these issues because they treat the pipeline as a black box or focus solely on single-turn prompt tuning. Consequently, bottlenecks arising from structural flaws, retrieval gaps, or formatting errors propagate through the chain, limiting overall performance. Existing tools like DSPy or GEPA optimize prompts within fixed program structures but lack the ability to inspect intermediate step-level failures and dynamically alter the pipeline's architecture within a standardized codebase.

Methodology: The FAPO Framework

FAPO (Fully Autonomous Prompt Optimization) is a framework designed to optimize LLM pipelines by treating them as inspectable workflows. It utilizes Claude Code as an autonomous orchestrator to drive a closed-loop optimization process within a standardized, isolated code workspace.

Core Architecture

The system operates around a "tenant" model, where each task (e.g., QA, security classification) has its own isolated workspace containing instructions, prompts, scoring rules, and allowed change scopes. The optimization loop consists of the following stages:

1. Evaluation: The system runs the current pipeline variant on training cases, recording final outputs, scores, and crucially, intermediate step artifacts (logs, retrieval results, reasoning traces).
2. Failure Attribution: A subagent analyzes the recorded evidence to diagnose the root cause of failures. It classifies errors as:
   • Prompt-addressable: Issues like verbose answers, abstention, or weak instructions.
   • Structural: Issues like insufficient retrieval coverage, missing pipeline steps, or formatting constraints that prompts cannot fix.
3. Scoped Proposal: Based on the attribution, the optimizer proposes a single, scoped change.
   • Prompt-First Policy: The system prioritizes prompt edits.
   • Escalation: It only escalates to changing chain parameters or pipeline structure (e.g., adding nodes, modifying retrieval depth) when the attribution report identifies a structural bottleneck that prompt edits are unlikely to resolve, and the tenant's scope contract permits such changes.
4. Review and Validation: A separate reviewer agent checks proposals for scope compliance, data leakage, and scorer compatibility. If approved, the variant is evaluated against aggregate validation scores.
5. Iteration: The process repeats, keeping improvements and iterating until performance plateaus or the budget is exhausted.

Guardrails

To prevent overfitting and ensure safety, FAPO employs:

• Split Access Controls: The optimizer sees individual training cases but only aggregate scores for validation/test sets.
• Scope Constraints: Tenant playbooks strictly define allowed and forbidden changes.
• Variant Immutability: Every attempt is recorded as a distinct file to maintain a full history.

Key Contributions

The paper presents three primary contributions:

1. A Claude Code-based Pipeline Optimization Technique: Unlike previous methods that optimize prompts in isolation, FAPO integrates prompt editing with structural pipeline changes, escalating to the latter only when evidence supports the necessity.
2. A Reproducible Workspace Procedure: The framework records final outputs, intermediate steps, configurations, and variant histories, ensuring full visibility and reproducibility of optimization attempts.
3. Empirical Validation: Extensive experiments demonstrate the technique's performance advantages across diverse domains, including QA, fact verification, instruction following, math, and security classification.

Experimental Results

FAPO was evaluated against GEPA (a state-of-the-art prompt optimizer) across six benchmarks and three task models (GPT-4.1-mini, GPT-5.4-mini, and Gemma 3-12B).

• Overall Performance: FAPO outperformed GEPA in 15 out of 18 model-benchmark comparisons. The mean gain of FAPO over GEPA was +14.1 percentage points (pp).
• Statistical Significance: In 11 comparisons, FAPO's wins were statistically significant (non-overlapping mean ± standard deviation ranges).
• Impact of Structural Changes: The most dramatic improvements occurred on HoVer and IFBench, where prompt-only search was insufficient. FAPO escalated to structural changes (e.g., extending retrieval chains from 3 to 4–5 hops, adding deterministic post-processing nodes). In these six comparisons, FAPO won all six with a mean gain of +33.8 pp.
• Prompt-Only Performance: Even in comparisons restricted to prompt edits, FAPO won 9 of 12 cases, attributed to the deep, iterative reasoning of the Claude Code orchestrator.
• Security Tasks: On the CTIBench-RCM (CVE-to-CWE classification) task, where the scope was restricted to prompt-only edits, FAPO improved test accuracy by +4.0 pp on GPT-5, +7.1 pp on Foundation-Sec-8B-Instruct, and +2.0 pp on Foundation-Sec-8B-Reasoning.
• Exceptions: FAPO did not outperform GEPA on the AIME benchmark across all models. The authors attribute this to the small sample size relative to the problem space and potential overfitting, treating the result as inconclusive rather than a failure of the method.

Significance and Claims

The paper positions FAPO as a state-of-the-art technique for optimizing multi-step LLM systems. Its significance lies in shifting the optimization paradigm from "finding a single successful prompt" (common in adversarial jailbreaking) to "improving the mean performance of a deployable pipeline."

By combining step-level attribution with a prompt-first escalation policy, FAPO addresses the specific failure modes of complex pipelines that single-turn prompt tuning misses. The authors claim that this evidence-grounded, pipeline-aware approach provides a practical path toward more reliable LLM systems for both general-purpose and security-focused tasks, moving beyond the limitations of fixed-structure prompt optimization.