Technical Summary: Attributing Structured-Output Gains in Function Calling

1. Problem Statement

Prompt-based interventions, particularly trajectory-derived skill injection, often improve Large Language Model (LLM) scores on function-calling benchmarks like the Berkeley Function Calling Leaderboard (BFCL) and API-Bank. However, a raw score increase does not distinguish between two distinct mechanisms:

1. Procedural Transfer: The model genuinely learned a reusable procedure for selecting the correct tools and arguments.
2. Interface Alignment: The model merely learned to conform to the specific output format (e.g., JSON wrapper keys, schema syntax) preferred by the evaluator.

Current evaluation practices often conflate these mechanisms. A skill prompt might improve scores simply by instructing the model to use a specific JSON key (e.g., "name" vs. "function") that the scorer accepts, even if the underlying tool selection remains unchanged or degrades. Conversely, semantically correct calls may be penalized for minor formatting deviations. The paper argues that without attributing gains to their specific causes, claims of "transferable agentic skill" are unjustified.

2. Methodology: The Four-Layer Attribution Protocol

The authors propose a rigorous, four-layer protocol to audit prompt-prepended skill gains. This protocol moves beyond simple score comparison to isolate the source of improvement.

Layer 1: Canonicalized Rescoring

The authors introduce canonicalized rescoring to separate "wrapper-key aliasing" from actual task-policy changes.

• Mechanism: Standard BFCL scoring treats specific JSON keys (e.g., "name") as correct and others (e.g., "function", "function_name") as errors, even if arguments are identical. The protocol normalizes these keys to a canonical form before evaluating AST (Abstract Syntax Tree) accuracy.
• Goal: To determine if a gain is merely a result of the model switching to the "correct" key alias rather than selecting the correct tool.

Layer 2: Format-Only Controls

This layer tests whether an explicit interface contract alone can reproduce the observed gains.

• Mechanism: The authors compare "Full Skills" (containing both procedural instructions and format constraints) against "Format-Only" prompts (containing only the expected JSON schema/wrapper instructions, stripped of task logic).
• Goal: If a format-only prompt matches or exceeds the performance of a full skill prompt, the gain is attributed to interface alignment, not procedural learning.

Layer 3: Repaired and Balanced Induction

This layer investigates whether gains are artifacts of the extraction pipeline (e.g., biased demonstration selection or specific extractor styles).

• Mechanism:
  • Repaired Extraction: Skills are re-extracted using a different model family (Qwen3-32B) and a balanced pool of high/low-scoring demonstrations, rather than the original GPT-4o extraction from high-scoring only.
  • Balanced Induction: The protocol tests if gains persist when the demonstration pool is balanced, removing the bias toward format-compliant examples.
• Goal: To determine if the "skill" is robust or if the original gains were driven by specific, non-generalizable demonstration biases.

Layer 4: Portability Checks

This layer tests the generalizability of the gains across different contexts.

• Mechanism:
  • Cross-Family: Testing skills extracted from one model family (e.g., Qwen) on others (e.g., Llama, Gemma).
  • Cross-Benchmark: Testing on API-Bank (using bracket syntax) and reasoning benchmarks (MATH-500, MultiHop-RAG).
  • Generic Controls: Comparing extracted skills against length-matched generic procedural prompts.
• Goal: To verify if the remaining gains represent a transferable procedure or are specific to the source model and contract.

3. Key Results

Applying this protocol to BFCL and API-Bank across 8 Qwen3 models (0.6B–235B) and other families yields several counter-intuitive findings:

• Interface Alignment Dominates: In many cases, format-only prompts match or exceed full skill prompts. For example, on Qwen3-8B, a format-only prompt yielded a +21.3 percentage point (pp) gain, surpassing the full skill@235B gain of +11.3 pp. This suggests the "skill" was primarily teaching the model how to wrap the output, not how to solve the task.
• Canonicalization Reveals Sensitivity: Standard scoring showed large gains and harms. After canonicalized rescoring, several "harmful" cells flipped to neutral, and positive gains shrank significantly. For instance, the 14B model's skill@235B gain dropped from -11.7 pp to -1.0 pp under canonicalization, indicating the original harm was largely due to key-alias sensitivity.
• Induction Sensitivity: The largest observed gains (e.g., +17.7 pp for 1.7B, +11.3 pp for 8B) largely disappeared or reversed when using repaired/balanced induction. When extraction was performed from a balanced pool of demonstrations rather than high-scoring ones, the gains vanished, suggesting the original improvements were driven by demonstration bias rather than robust skill extraction.
• Generic Prompts Replicate Gains: On API-Bank, length-matched generic procedural prompts nearly reproduced the gains of extracted skills (within 0.5 pp), further indicating that the specific wording of the "skill" was not the primary driver of performance.
• Cross-Family Failure: Skills extracted from Qwen3 often failed or degraded performance when applied to Llama or Gemma models, whereas format-only cues sometimes worked better. This lack of portability undermines claims of universal procedural transfer.

4. Contributions

The paper makes three primary contributions:

1. Formulation: It frames function-calling skill-gain attribution as a reporting problem, distinguishing between task-policy changes and interface alignment in structured-output audits.
2. Protocol: It proposes and instantiates a four-layer attribution protocol (canonicalized rescoring, format-only controls, induction-robust extraction, and portability checks) to rigorously evaluate skill injection.
3. Empirical Findings: It demonstrates that in BFCL/API-Bank-style settings, many apparent gains are better attributed to interface alignment than robust procedural transfer. It releases BFCL-CANONICAL (a dataset/metric set) and a practical reporting recipe for future claims.

5. Significance and Claims

The paper does not claim that format compliance is unimportant; rather, it argues that format compliance is an engineering capability distinct from procedural reasoning.

• Clarification of Metrics: The authors assert that a structured-output score alone cannot certify that a model has learned a transferable procedure. A score improvement may simply reflect that the model learned to "speak the evaluator's language."
• Reporting Standards: The paper recommends that future claims of "transferable skill" must survive the four-layer protocol. If a gain disappears under canonicalization, format-only controls, or balanced induction, it should be reported as interface alignment or prompt-contract adaptation, not as a reusable agentic skill.
• Constructive Diagnosis: The findings are presented as a constructive diagnosis for the field. By separating interface alignment from procedural transfer, researchers can better understand what their benchmarks actually measure and avoid overclaiming the capabilities of skill-injection methods.

The authors conclude that while interface alignment is valuable for deployment, attributing gains to procedural transfer without passing these rigorous checks is an attribution error. The protocol preserves the engineering value of format alignment while reserving the stronger claim of procedural transfer only for residual evidence that survives all controls.