PerfGuard: A Performance-Aware Agent for Visual Content Generation

Imagine you are the director of a massive, high-tech film studio. You have a script (your idea) and a huge warehouse full of special effects tools (AI image generators and editors). Your goal is to create a movie scene exactly as you imagined it.

In the past, the "director" (the AI Agent) had a major problem: it didn't know which tool actually worked best.

The director would just read a tool's name tag, like "Magic Paintbrush," and assume it was perfect for painting a dragon. But in reality, that specific brush might be great at painting water but terrible at drawing fire. The director would pick the wrong tool, the scene would look weird, and the whole project would fail.

PerfGuard is a new, super-smart "Production Manager" that solves this problem. Here is how it works, broken down into simple concepts:

1. The Problem: The "Blind" Director

Current AI agents are like directors who only read the menu description of a restaurant dish.

The Menu says: "Delicious Burger."
The Reality: It might be dry, burnt, or missing the cheese.
The Result: The director orders the wrong dish, and the customer (the user) is unhappy.

In the world of AI, tools are described with vague text like "generates images." But one tool might be great at drawing faces, while another is better at drawing landscapes. The old AI didn't know the difference, so it often picked the wrong tool, leading to messy results.

2. The Solution: PerfGuard (The "Quality Control" Manager)

PerfGuard changes the game by giving the director a detailed scorecard for every single tool in the warehouse, not just a menu description.

It uses three main "superpowers" to ensure the movie gets made perfectly:

A. The Scorecard System (Performance-Aware Selection Modeling)

Instead of just reading the tool's name, PerfGuard gives every tool a report card with specific grades.

Analogy: Imagine you are hiring a carpenter. Instead of just asking, "Can you build a chair?", you check their report card:
- Wood Grain Accuracy: A+
- Painting Smoothness: B-
- Leg Stability: A
How it works: If your task is to "paint a smooth wooden chair," PerfGuard looks at the scorecards and picks the tool with the "A" in painting, ignoring the one that is only good at building legs. It matches the specific need to the specific strength.

B. The "Learning from Mistakes" Loop (Adaptive Preference Updating)

Sometimes, a tool's report card is outdated. Maybe a tool got a software update and is now amazing at drawing cats, but the scorecard still says it's bad.

Analogy: Think of a food critic who tastes a dish and realizes, "Wow, this chef actually improved their soup since last year!"
How it works: PerfGuard tries a few different tools. If a tool that was supposed to be "average" actually produces a "perfect" result, PerfGuard updates its internal scorecard immediately. It learns in real-time, getting smarter with every single task it does.

C. The Smart Scriptwriter (Capability-Aligned Planning)

This is the most important part. PerfGuard doesn't just pick the right tool; it rewrites the script to fit the tools it has.

Analogy: Imagine you want to film a scene with a dragon breathing fire.
- Old Director: "Okay, we need a dragon, then we need fire, then we need smoke." (Picks tools blindly).
- PerfGuard Director: "Wait, our 'Fire Tool' is great at making smoke but bad at making dragons. Let's use the 'Dragon Tool' first to make the creature, then use the 'Fire Tool' to add the smoke around it."
How it works: PerfGuard breaks your big request into tiny steps. It looks at the scorecards and says, "Step 1 needs to be done by Tool A, Step 2 by Tool B." It plans the order of operations so that the weaknesses of one tool don't ruin the work of the next.

3. The Result: A Perfect Movie

Because PerfGuard knows exactly what every tool can and cannot do, and it plans the steps carefully:

Fewer Mistakes: It doesn't waste time trying to paint a dragon with a tool meant for landscapes.
Better Quality: The final image looks exactly like what you asked for.
Faster: It doesn't get stuck in loops trying to fix errors because it picked the right tool the first time.

Summary

Think of PerfGuard as the ultimate matchmaker between your imagination and the AI tools available.

Old Way: "I have a hammer and a screwdriver. I'll use the hammer to screw in this nail." (It fails).
PerfGuard Way: "I have a hammer and a screwdriver. I know the screwdriver is 95% accurate for screws and the hammer is 90% accurate for nails. I will use the screwdriver for the screw and the hammer for the nail, and I'll do it in that specific order."

It turns a chaotic, guessing-game process into a precise, reliable, and highly successful operation.

1. Problem Statement

Current Large Language Model (LLM)-powered agents for visual content generation (AIGC) operate under an idealized assumption: that tool invocations are invariably successful. Existing frameworks typically rely on coarse, generic textual descriptions of tool capabilities (e.g., "can generate images based on text"). This approach suffers from three critical limitations:

Lack of Precision: Textual descriptions fail to distinguish the nuanced performance boundaries of different models (e.g., one model may excel at color accuracy while another excels at spatial relationships).
Planning Uncertainty: Without precise performance data, agents cannot reliably plan complex, multi-step tasks, leading to execution failures or suboptimal results.
Static Adaptation: Existing systems cannot adapt to iterative tool updates or deviations between theoretical benchmark scores and actual execution outcomes in real-world scenarios.

2. Methodology: The PerfGuard Framework

PerfGuard is a performance-aware agent framework designed to systematically model tool performance boundaries and integrate them into task planning and scheduling. It builds upon a standardized agent architecture (Analyst, Planner, Worker, Self-Evaluator) and introduces three core mechanisms:

A. Performance-Aware Selection Modeling (PASM)

Instead of relying on text descriptions, PASM replaces them with a multi-dimensional scoring system based on fine-grained performance evaluations.

Tool Performance Boundaries: Tools are evaluated across specific dimensions derived from authoritative benchmarks (e.g., T2I-compbench for generation and ImgEdit-Bench for editing).
- Generation Dimensions: Color, Shape, Texture, 2D/3D Spatial, Numeracy, Non-spatial semantics.
- Editing Dimensions: Addition, Removal, Replacement, Attribute Alteration, Motion Change, Style Transfer, Background Change.
Selection Mechanism: The Worker dynamically calculates a preference weight vector ( $W_{task}$ ) for a given subtask. It computes a suitability score ( $S_{tools}$ ) by combining this weight with the tool's performance boundary matrix ( $M_p$ ). The tool with the highest score is selected, ensuring the choice is driven by intrinsic performance characteristics rather than vague descriptions.

B. Adaptive Preference Updating (APU)

Recognizing that static benchmark scores may not reflect real-time execution, APU introduces a dynamic feedback loop.

Exploration-Exploitation Strategy: During tool selection, the system chooses the top $m$ tools based on theoretical scores but also samples $n$ random tools to explore potential high-performers.
Ranking Correction: The system compares the theoretical ranking ( $R_{theory}$ ) with the actual execution ranking ( $R_{actual}$ ), determined by a multimodal large model evaluating the outputs.
Matrix Update: The performance boundary matrix is updated iteratively using the difference ( $\Delta$ ) between theoretical and actual rankings, allowing the system to adapt to real-world task requirements and tool updates.

C. Capability-Aligned Planning Optimization (CAPO)

To align the Planner's decision-making with the performance-aware strategy, PerfGuard extends Step-aware Preference Optimization (SPO).

Decision Performance Estimator: The Self-Evaluator assesses the output of each subtask against global and local evaluation goals.
Optimization Objective: The Planner generates multiple candidate subtask plans. The system treats the highest-scoring plan as a "winning sample" and the lowest as a "losing sample." The Planner is then optimized via a preference learning objective (similar to DPO) to align its future planning patterns with the performance-aware strategy.
Memory Retrieval: To improve efficiency, the system retrieves optimal subtask sequences from past successful tasks to guide new planning iterations.

3. Key Contributions

Shift from Text to Performance: The paper proposes a paradigm shift from relying on ambiguous textual tool descriptions to a rigorous, multi-dimensional performance scoring system (PASM).
Dynamic Adaptation: The introduction of APU allows the agent to continuously refine its understanding of tool capabilities based on actual execution feedback, bridging the gap between benchmark data and real-world performance.
Planning-Execution Alignment: CAPO ensures that the high-level planning logic is explicitly trained to respect the performance constraints and strengths of the underlying tools, creating a closed-loop system for robust reasoning.
Comprehensive Evaluation: The framework is validated across basic generation, advanced generation, and complex editing tasks, demonstrating superior alignment with user intent.

4. Experimental Results

PerfGuard was evaluated against state-of-the-art methods (including FLUX, SD3, GoT, GenArtist, T2I-Copilot) on three benchmarks: T2I-CompBench, OneIG-Bench, and Complex-Edit.

Quantitative Performance:
- Basic Generation: PerfGuard achieved the highest scores in Attribute Binding (0.8753), Object Relationship (0.7366), and Complexity (0.5007) on T2I-CompBench, outperforming both diffusion models and other agent-based systems.
- Advanced Generation: On OneIG-Bench, PerfGuard led in Reasoning metrics (0.350) and Alignment (0.834), demonstrating superior handling of complex, cross-domain information.
- Complex Editing: On Complex-Edit (Level-3), PerfGuard achieved the highest scores in Instruction Following (8.95) and Perceptual Quality (9.02).
Qualitative Analysis: Visual comparisons show that PerfGuard successfully handles complex prompts involving multiple entities and detailed attributes (e.g., specific spatial arrangements, textures) where other methods fail to include key elements or misinterpret semantics.
Ablation Studies:
- Removing PASM resulted in a high tool selection error rate (77.8% with text-only descriptions).
- Adding APU reduced the error rate to 14.2%, proving the efficacy of adaptive updates.
- CAPO significantly improved the Planner's ability to sequence operations correctly (e.g., editing backgrounds before objects to avoid artifacts).
Efficiency: PerfGuard demonstrated lower inference time and significantly lower token consumption for tool selection compared to text-based methods, especially as the tool library size increased.

5. Significance

PerfGuard addresses a fundamental bottleneck in the deployment of AI agents for creative tasks: the reliability of tool execution. By explicitly modeling performance boundaries and creating a feedback loop for continuous improvement, the framework:

Enhances Reliability: Reduces the uncertainty in planning, making agents more robust for complex, multi-step AIGC tasks.
Improves User Intent Alignment: Ensures that the generated content strictly adheres to nuanced user requirements (e.g., specific counts, spatial relationships, styles).
Scalability: The performance-driven selection mechanism scales efficiently with large tool libraries, avoiding the token explosion associated with text-based reasoning.
Future Impact: This work provides a blueprint for building "performance-aware" agents in other domains beyond visual generation, where tool reliability and capability matching are critical for autonomous execution.