Technical Summary: Assertion-Conditioned Compliance in Multi-Turn Tool-Calling Agents

Problem Statement

While Large Language Models (LLMs) augmented with tool-calling capabilities have demonstrated significant utility in multi-turn dialogues, their deployment in safety-critical industries remains hindered by concerns regarding reasoning robustness. Existing benchmarks, such as the Berkeley Function-Calling Leaderboard (BFCL), primarily evaluate final task success (accuracy) based on the final environment state. However, these metrics often fail to capture procedural robustness—the model's ability to maintain a correct execution pipeline when confronted with misleading information.

Current safety evaluations often focus on overt adversarial attacks (e.g., prompt injection) or linguistic sycophancy (verbal agreement). There is a critical gap in understanding how agents handle incorrect assertions that originate from different provenance sources:

1. User-Sourced Assertions (USAs): Misleading information provided by the user, testing the model's tendency toward sycophancy.
2. Function-Sourced Assertions (FSAs): Contradictory or stale hints originating from tool outputs or system policies, testing the model's compliance with internal system cues.

The core problem is that an agent may successfully complete a task (high accuracy) while executing unnecessary, unsafe, or misleading operations along the way due to compliance with these false assertions. This "quiet" over-compliance represents a latent vulnerability where the execution pipeline is corrupted even if the final state appears correct.

Methodology

The authors introduce Assertion-Conditioned Compliance (A-CC), a novel evaluation paradigm designed to diagnose how models incorporate incorrect assertions into their reasoning and tool-calling pipelines.

Experimental Setup

• Benchmark: The study utilizes the multi_turn_base category of the BFCL, consisting of 200 curated entries based on real-world APIs.
• Models: Evaluation was conducted on 11 state-of-the-art models, including Qwen3, Salesforce's xLAM 2, Watt Tool, BitAgent Bounty, and ToolACE 2, ranging from 3B to 70B parameters.
• Assertion Generation:
  • USAs: Synthetic, plausible but incorrect claims were injected into user prompts. These were generated in two tonal variants (confident and hedged) and three functional modes (init, read-heavy, write-heavy) to simulate different user misdirections.
  • FSAs: Misleading policy notes were injected directly into tool output payloads. These were designed to contradict the user's intended actions, simulating scenarios where unmaintained tools provide stale or erroneous hints.
• Protocol: Each test case was run in a baseline condition (no assertion) and asserted conditions. The study tracked whether the model invoked the asserted (incorrect) function at least once within the relevant turn.

Evaluation Metrics

The authors decouple Task Success (standard BFCL accuracy based on final environment state) from Compliance Rate (CR).

• Compliance Rate (CR): Measures the frequency with which an agent executes an asserted operation, regardless of whether the final task succeeds.
• Outcome Buckets: To analyze the relationship between accuracy and compliance, runs were categorized into four transitions:
  • S→S: Success preserved (compliance occurred, but task still succeeded).
  • S→F: Assertion-induced failure (compliance led to task failure).
  • F→S: Assertion-induced recovery.
  • F→F: Failure persisted.

Key Contributions

1. A New Evaluation Paradigm (A-CC): Formalizes the measurement of how LLMs integrate incorrect assertions into procedural execution, moving beyond surface-level verbal agreement to track pipeline manipulation.
2. Dual Provenance Vectors: Introduces distinct testing for User-Sourced (social) and Function-Sourced (procedural) assertions, revealing that agents exhibit different compliance biases depending on the source of the misleading signal.
3. Reproducible Benchmarking Framework: Provides a framework for empirically comparing behavioral patterns across model families, specifically isolating the effects of provenance-tagged assertions.
4. Decoupled Metrics: Establishes Compliance Rate as a critical metric that reveals safety risks invisible to standard accuracy scores, demonstrating that high task success does not guarantee safe or efficient execution pipelines.

Results

The empirical analysis of 11 models yielded several critical findings:

• Widespread Vulnerability: Models exhibit moderate but significant compliance to both USAs and FSAs.
  • USA Compliance: Macro-averaged compliance was 36.3% for confident assertions and 27.7% for hedged assertions. Tool-native models (e.g., Watt Tool 70B) and context-aware models (Qwen3 variants) showed higher susceptibility to function-sourced hints.
  • FSA Compliance: Average compliance was 27.8%, indicating that agents often treat erroneous tool feedback as authoritative.
• Decoupling of Accuracy and Compliance: Task success (accuracy) and behavioral compliance are not tightly correlated.
  • Models with similar baseline accuracy showed marked differences in compliance rates.
  • Crucially, many models exhibited high compliance in the S→S bucket (Success→Success). This means agents frequently executed unnecessary or misleading operations while still achieving the correct final state, masking the vulnerability under standard metrics.
• Significant Worst-Case Degradation: Despite moderate average compliance, assertions caused substantial drops in task success.
  • The worst-case success degradation averaged 16.9 percentage points across models.
  • Qwen3 14B (FC) suffered the largest drop (23.4 points), followed by xLAM 2 32B (-20.3) and BitAgent Bounty 8B (-20.8).
• Provenance Sensitivity: The source of the assertion matters. Tool-native models were particularly susceptible to Function-Sourced Assertions, suggesting they prioritize internal system feedback over external verification.

Significance and Claims

The paper claims that Assertion-Conditioned Compliance represents a critical, latent vulnerability in deployed multi-turn agents. The primary significance lies in the revelation that standard leaderboard scores (BFCL accuracy) are insufficient for safety-critical deployment because they fail to detect "quiet" procedural failures.

• Hidden Risks: Agents judged solely on task success may still execute unsafe or unnecessary operations when faced with plausible but false assertions from users or stale tool hints.
• Operational Implications: The study argues that user prompts, tool outputs, and policy hints must be treated as separate, potentially adversarial channels rather than a single trusted context.
• Evaluation Necessity: The authors posit that A-CC-style evaluation is a necessary pre-deployment check for pipelines mediating high-stakes actions, as it exposes risks that standard accuracy metrics obscure.

The paper concludes modestly, acknowledging that while it defines and measures the A-CC vulnerability, it does not yet propose specific training objectives or architectural modifications to remediate it. The work serves as a controlled stress test to highlight the need for additional guardrails and provenance-sensitive robustness in future agent design.