🤖 AI

PluginEval: A Diagnostic Benchmark for Fine-Grained Error Attribution in Function Calling

This paper introduces PluginEval, a diagnostic benchmark that employs a two-stage framework combining LLM generation with deterministic API execution to systematically address data distribution gaps and lack of adversarial testing, thereby enabling fine-grained error attribution for evaluating tool routing in large language models.

Original authors: Dongjie Xu, Julius, Hanchi Dong, Minghua Tang, Yuxuan Sun, Ziwei Nie, Zicheng Liu, Dujun Qing, Jiajie Xu

Published 2026-08-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Dongjie Xu, Julius, Hanchi Dong, Minghua Tang, Yuxuan Sun, Ziwei Nie, Zicheng Liu, Dujun Qing, Jiajie Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart, very eager robot butler how to use a giant toolbox filled with thousands of different gadgets. You tell the robot, "I'm hungry," and it has to decide three things: first, does it actually need to use a tool, or can it just think of a snack? Second, if it needs a tool, which specific gadget is the right one (a blender, not a hammer)? And third, can it actually grab that gadget and use it correctly without breaking it? This is the world of "function calling" in Artificial Intelligence. It's the skill that lets AI models stop just chatting and start actually doing things, like booking flights or checking the weather. But here's the tricky part: just because a robot says the right words doesn't mean it picked the right tool or used it correctly. We need a way to test if the robot is truly smart or just lucky.

That's exactly what the researchers behind this paper, PluginEval, are trying to fix. They noticed that most current tests for these AI robots are like a driving test where everyone gets an easy, empty parking lot. The tests don't have enough tricky situations, they don't catch the specific ways robots fail, and they often rely on other robots to grade the answers, which can be biased. So, the team built a new, much tougher "driving test" specifically for Chinese language queries. They created a system that doesn't just look at the final score but acts like a detective, figuring out exactly why a robot failed: did it forget to bring a tool, did it grab the wrong one, or did it try to use the right tool but spill the ingredients?

The Detective's Toolkit: How They Built the Test

To build this new benchmark, the authors didn't just write down a bunch of questions and hope for the best. They invented a "Closed-Loop Construction Framework" (PCCF), which is a fancy way of saying they built a self-correcting machine to make the test.

Think of it like a video game level designer that keeps replaying a level until it's perfect.

Stage One (The Reality Check): First, they take a question (like "Find me a cheap flight to Tokyo") and ask several AI models to try to solve it. But here's the twist: they don't just trust the AI's word. They actually run the code. If the AI says "I'll call the flight API," the system tries to make that call. If the API says "Error: Missing date," the system knows the AI failed. This separates the "idea" from the "reality."
Stage Two (The Gap Filler): The system then looks at the results and asks, "Where are we missing tricky questions?" Maybe the test has too many easy questions about weather and not enough hard questions about booking complex travel. The system then generates new, harder questions specifically to fill those holes. It creates "adversarial" questions—traps designed to trick the AI into making mistakes.
The Loop: These new, tricky questions go back to Stage One to be tested again. If they are too easy, the system makes them harder. If they are too confusing, it uses recognition evidence to strengthen the negative examples before re-verifying them. This loop keeps spinning until the test covers every possible way an AI could get confused, from simple mistakes to complex logic traps.

The Verdict: What the Test Revealed

Once they had their perfect test (which included 3,000 human-verified questions across 54 different tools), they put five of the world's smartest AI models to the test. These included big names like GPT-5.4, Claude 4.6, and Gemini 3.1 Pro.

The results were eye-opening. The authors found that aggregate scores (the final grade) are misleading.

The Difficulty Trap: When the test was mostly easy questions, all the AIs looked like geniuses, scoring over 80%. But as soon as they hit the "Hard" questions, the scores plummeted. For the hardest questions, even the best AI only got about 10% right. This suggests that current models are very fragile; they work great on simple tasks but crumble when things get complicated.
Different Flaws for Different Models: The paper didn't just say "Model A is better." It broke down the errors like a mechanic checking a car engine.
- GPT-5.4 was the worst at both identifying required tools and avoiding unnecessary calls, exhibiting the highest rates of both "miss recall" (forgetting to use a needed tool) and "over recall" (using a tool when it shouldn't).
- Claude Opus 4.6 was the best at remembering to use tools (lowest "miss recall"), but it sometimes used them when it shouldn't have (high "over recall").
- Gemini 3.1 Pro had the most balanced error profile, achieving the lowest "over recall" rate, which helped it maintain competitive accuracy despite having a lower recall than Claude.
The "Time" Problem: One specific type of error stood out for every model: Temporal Errors. Whether it was asking for "next Tuesday" or "last month," the AIs consistently struggled to get the dates and times right. This was the biggest source of failure across the board.

Why This Matters

The paper argues that we can't just look at a single percentage to judge an AI's ability to use tools. A model might have a high score because it's good at easy questions, or because it's lucky. By using their new "Gold-Anchored" judging system—which compares the AI's answer against a human-verified "gold standard" rather than another AI's guess—they could pinpoint exactly where the robot failed.

The authors suggest that to make truly reliable AI agents, we need to stop treating them like black boxes that either work or don't. Instead, we need to diagnose them like a doctor: checking if they missed a step, if they grabbed the wrong tool, or if they just got the timing wrong. Their work shows that while today's AI is powerful, it still has significant blind spots, especially when the task gets hard or the timing gets tricky. The paper doesn't claim to have "solved" AI tool use, but it provides the first real map of where the robots are stumbling, so we can help them learn to walk more steadily.

Technical Summary: PluginEval

Problem Statement

As Large Language Models (LLMs) increasingly function as autonomous agents, reliable evaluation of tool routing (function calling) is critical. Current benchmarks suffer from three structural limitations:

Data Distribution Bias: Power-law distributions leave rare scenarios and edge cases underrepresented.
Lack of Adversarial Rigor: The absence of hard negatives obscures performance differences between models.
Unvalidated Annotation Pipelines: Many benchmarks rely on LLM judgments for annotation that have not been validated through actual execution, leading to potential semantic errors despite structural correctness.

Furthermore, existing benchmarks often focus on aggregate success rates or call syntax, failing to diagnose specific failure modes such as missed tool calls, spurious calls, or parameter grounding errors. There is a specific need for a benchmark that combines reliable, execution-grounded annotation with structured coverage of trigger and exclusion boundaries, particularly for Chinese queries which are often ambiguous or context-dependent.

Methodology: The PluginEval Closed-Loop Construction Framework (PCCF)

The paper introduces PluginEval, a benchmark constructed via a two-stage, closed-loop framework (PCCF) designed to separate annotation reliability from coverage requirements.

Task Formulation

Tool routing is formulated as a hierarchy of three decisions:

Necessity ( $f_1$ ): Is a tool required?
Relevance ( $f_2$ ): Is the target plugin relevant?
Invocation ( $f_3$ ): Can the request be grounded into a valid invocation with correct arguments, or must it be rejected?

Stage I: Execution-Grounded Ensemble Annotation

This stage focuses on establishing trustworthy annotations for both real and synthetic queries.

Generation: Multiple generators independently propose complete invocations.
Verification: Proposals undergo a multi-step verification process:
- Deterministic Schema Validation: Checks structural validity.
- Argument Evaluation: An LLM evaluator checks if arguments are supported by the query and consistent with plugin semantics.
- Execution: Proposals are executed via MCP (Model Context Protocol).
- Result Evaluation: A separate evaluator checks if the execution response satisfies the user request.
Refinement Loop: Proposals that fail verification (due to schema violations, execution errors, or semantic mismatches) are retained as traces. These traces, along with independent evaluator scores, are fed back to generators for refinement. This process repeats for a maximum of $R_{max}$ rounds until a proposal meets a confidence threshold $\tau$ or is rejected as a justified abstention.

Stage II: Scenario-Grounded Synthesis and Behavioral Calibration

This stage addresses coverage gaps and difficulty calibration.

Scenario Decomposition: Plugins are decomposed into trigger scenarios (supported requests) and exclusion scenarios (requests requiring no tool, other tools, or exceeding capabilities).
Targeted Synthesis: The system identifies under-covered units (combinations of capability, intent, and boundary) and synthesizes new queries to fill these gaps.
Difficulty Calibration:
- Positive Examples: Difficulty is based on inference depth (0, 1, or $\ge$ 2 steps).
- Negative Examples: Difficulty is calibrated based on behavioral evidence. A frozen reference router evaluates the synthesized negative $K$ times. If the router consistently fails to reject the negative (high false positive rate), the example is deemed "Hard." If it is easily rejected, it is "Easy."
Closed Loop: Synthesized examples are returned to Stage I for verification. The loop continues until coverage budgets are met or improvement plateaus.

Evaluation Framework: Gold-Anchored Judge

To move beyond aggregate accuracy, the paper employs a Gold-Anchored LLM Judge:

Input: Predicted calls, human-verified gold calls, and the user query.
Constraint: The judge treats the gold call list as the sole anchor. It cannot extend or override gold specifications.
Hierarchical Verification:
1. Tool Alignment: Matches tool names between prediction and gold.
2. Argument Filtering: Ensures non-empty, comparable argument sets.
3. Semantic Verification: Determines if predicted arguments are semantically equivalent to gold arguments under the query intent (allowing for aliases and formatting variations but strictly evaluating temporal scopes).
Error Attribution: Failures are classified into mutually exclusive categories:
- Miss Recall: Missing required tools.
- Parameter Error: Semantically incorrect arguments (sub-classified into temporal errors, missing arguments, incorrect values).
- Over Recall: Spurious tool activation on negative queries.

Key Contributions

PluginEval Benchmark: A Chinese function-calling benchmark with 3,000 human-verified instances covering 54 plugins, 63 tools, and 9 domains. It features explicit trigger/exclusion scenarios, three types of boundary negatives, and behaviorally calibrated difficulty.
PCCF Framework: A closed-loop construction method that decouples annotation trustworthiness (via execution-grounded verification) from coverage requirements (via scenario synthesis). It utilizes schema validation, independent evaluation, and tool execution to repair annotations.
Fine-Grained Error Attribution: An evaluation framework that attributes failures to specific causes (miss recall, parameter errors, over recall) rather than relying on aggregate success metrics.
Empirical Analysis: Evaluation of five frontier LLMs (including proprietary and open-weight models) revealing that aggregate scores are highly sensitive to difficulty composition and that models exhibit distinct failure profiles.

Results

The paper evaluates five models: GPT-5.4, Claude 4.6, Gemini 3.1 Pro, DeepSeek-V4-Pro, and Qwen3-235B-A22B.

Performance Sensitivity: Aggregate performance is highly dependent on difficulty distribution. On the original distribution (40% Easy, 40% Medium, 20% Hard), Claude Opus 4.6 achieved the best performance (67.60% Recall, 68.17% Accuracy). However, performance drops significantly on Hard instances (Recall drops to 2.8%–10.8% across models).
Difficulty Impact: Shifting the distribution to be Easy-weighted (60% Easy) substantially improved all models (Recall increased by ~15–21 points), demonstrating that current benchmarks may overestimate capabilities if not rigorously calibrated.
Comparison with Existing Benchmarks: Models generally performed worse on PluginEval than on BFCL, APIBench, or NexusRaven. For instance, Claude Opus 4.6's Recall dropped from 73.04% on BFCL to 67.60% on PluginEval. This suggests PluginEval captures more challenging, realistic, and adversarial scenarios.
Error Profiles:
- GPT-5.4: Exhibited the highest Miss Recall (33.72%) and the highest Over Recall (33.20%), indicating substantial difficulty in both identifying required tools and avoiding unnecessary calls.
- Claude Opus 4.6: Had the lowest Miss Recall (13.95%) but a high Over Recall rate (29.10%), which remains higher than Gemini's but lower than GPT-5.4's.
- Gemini 3.1 Pro: Achieved the lowest Over Recall (18.55%), contributing to competitive accuracy despite lower recall.
- Temporal Errors: Across all models, temporal grounding errors were the most frequent parameter error subtype, indicating a shared weakness in handling time-based constraints.
Reliability: Human audits confirmed high agreement (Fleiss' $\kappa$ up to 0.9451) on reference annotations and judge attributions, validating the benchmark's reliability.

Significance and Claims

The paper claims that PluginEval addresses the critical gap in diagnostic evaluation for tool routing. By moving beyond aggregate success rates, it reveals that:

Aggregate scores obscure distinct failure modes: Models with similar overall scores may fail for entirely different reasons (e.g., tool selection vs. parameter grounding).
Difficulty composition matters: Performance rankings can shift based on the ratio of easy to hard examples, necessitating calibrated difficulty labels derived from model behavior.
Execution is essential for trust: Validating annotations through actual API execution and schema checks is necessary to ensure semantic correctness, not just syntactic validity.

The authors conclude that structured error attribution is necessary to guide the development of more robust agentic systems, and that their closed-loop framework provides a reproducible method for constructing such diagnostic benchmarks.