Technical Summary: Set-shifting Behavioral Test for Harnessed Agents

Problem Statement

Modern Large Language Model (LLM) agents operate within "agentic harnesses" where they maintain persistent session contexts, including prior tool calls, results, and skill libraries. In such environments, agents often converge on preferred subsets of tools to solve tasks. A critical vulnerability arises when the reliability of these tools changes silently (e.g., a backend API shift) while the tool names, descriptions, and schemas remain identical.

This paper investigates whether harnessed agents can adapt their action routes when the reliable tool group shifts mid-trajectory without explicit notification. Drawing from cognitive psychology, the authors frame this as a set-shifting problem, analogous to the Wisconsin Card Sorting Test (WCST) and the Einstellung effect, where agents may exhibit perseveration—sticking to a learned rule or toolset even after it becomes suboptimal or broken.

Methodology

Benchmark Design: WCST-Tool-Bench

The authors introduce a benchmark built on redundant tool-skill sets.

• Tool-Skill Structure: A "tool" is a callable function; a "tool group" consists of functionally equivalent tools with different names/descriptions but identical schemas. A "skill" is a markdown procedure linking the agent to a specific tool group.
• Redundancy: Multiple tool groups exist for the same task, differing only in hidden reliability. The interface presented to the agent (names, descriptions, schemas) remains invariant across shifts.
• Domains: Three domains were instantiated:
  1. Scheduling/Coordination: Groups presented as competing slot providers.
  2. DevOps Incident Triage: Groups presented as complementary observability stacks.
  3. Multi-cloud Storage: Groups presented as competing storage providers.
• Set Framing: The relationship between tool groups is manipulated via descriptions to be either competing (alternatives) or complementary (parts of a whole).

Experimental Setup

• Environment: The Hermes Agent harness (open-source) with persistent session context.
• Models: Two open-weight models: mimo-v2.5 (Xiaomi) and deepseek-v4-pro (DeepSeek).
• Schedule: A branched tree structure with $M$ tool groups and $K$ branching layers.
  • Shifts: At predetermined boundaries, the reliable tool group changes silently.
  • Controls: Every shift is paired with a "no-shift" control where the previous group remains reliable.
  • Trajectories: Paths include reversals (returning to a previously used group), novel shifts (to a never-used group), and no-shift controls.
• Feedback: A deterministic verifier provides binary feedback (success payload vs. failure indicator) with no diagnostic messages, forcing the agent to infer reliability solely from outcomes.

Evaluation Metrics

• Set-shifting Accuracy ($\Phi$): The joint probability that the agent routes to the correct target tool group in every post-shift window along a trajectory.
• Route Classes: Trajectories are classified by the mix of tool groups called:
  • $R_1$: Target only.
  • $R_2$: Target + one other.
  • $R_3$: Target + both others.
  • $R^*$: No target.
• Task Completion Rate ($F$): Proportion of turns where the agent successfully completes the task using at least one reliable tool.

Key Contributions

1. Benchmark: A longitudinal evaluation framework for harnessed agents using redundant tool-skill sets with hidden reliability shifts and paired no-shift controls.
2. Descriptive Framework: A characterization of post-shift behavior as a recurrence on a small set of discrete routines (attractor states).
3. Intervention Case Study: An analysis of two intervention channels:
   • Instructional: Policy prompts (e.g., "adaptive" vs. "polymath" personas).
   • Structural: Set framing (competing vs. complementary descriptions).

Results and Analysis

1. Prior-Use Roles and Commitment Patterns (RQ1)

Agents do not adapt uniformly; their behavior depends on the target's prior usage role (reversal, reinforced, or novel).

• Discrete Routines: Post-shift behavior quickly locks into a small set of discrete routines. Call shares concentrate on specific values (e.g., 1.0, 0.5, 0.33, 0).
• Model Differences:
  • mimo-v2.5: Exhibits cell-level lock-in. It settles on a stable, mid-range routine (often a mix of previously used groups) within a few turns. Its behavior is sensitive to the target's prior use (e.g., it struggles most with "novel" targets it has never seen).
  • deepseek-v4-pro: Exhibits prefix-level lock-in. The route chosen early in a trajectory predicts the entire segment's behavior. It tends to bifurcate into extreme states ($s \approx 1$ or $s \approx 0$), showing high variance but strong persistence in its initial choice regardless of the target's prior role.

2. Depth and Path Dependence (RQ2)

• Depth: Deeper shifts (Tier-2) generally reduce accuracy ($\Phi$) for both models. However, deepseek shows less degradation because its early commitment persists through layers, whereas mimo's accuracy drops significantly as it struggles to update its recent history.
• Path: For mimo, the path taken to reach a target matters (recent history dictates performance). For deepseek, the path matters less once the initial commitment is made; the prefix-level lock-in overrides path variations.

3. Policy Prompts (RQ3)

• Adaptive Prompt: Instructing the agent to "track success and switch immediately upon failure" successfully overrides deepseek's bimodal lock-in, achieving high set-shifting accuracy ($\Phi \ge 0.86$) and reducing variance. It did not reliably change mimo's behavior in a small sample.
• Polymath Prompt: Instructing the agent to "vary choices and avoid fixating" increased tool diversity but did not consistently improve correct routing, sometimes leading to under-weighted routines.

4. Set Framing (RQ4)

• Structural Intervention: Changing the framing from complementary (groups as parts of a whole) to competing (groups as alternatives) significantly shifted routing dynamics.
• Effect: The competing framing consistently improved set-shifting accuracy ($\Phi$) across both models and domains compared to the complementary framing.
  • Complementary: Agents tended to lock into a "mixed routine" (calling one tool from every group), resulting in low accuracy.
  • Competing: Agents were more likely to commit to a single reliable provider, resulting in a bimodal distribution of high or low accuracy, but with a higher probability of high accuracy.

Significance and Claims

The paper claims to establish a new paradigm for evaluating LLM agents that moves beyond single-turn tool selection to longitudinal adaptation in dynamic environments.

• Failure Modes: The study identifies that agents do not merely "fail" to shift; they settle into stable, recurring routines (attractors) that can be either efficient (if the environment is stable) or rigid (if the environment shifts).
• Context Dependence: The results mirror cognitive psychology findings: agents inherit context dependence where "perseveration" is efficient in stable settings but costly in shifting ones.
• Intervention Efficacy: The paper demonstrates that both instructional (policy prompts) and structural (set framing) interventions can alter these lock-in patterns. Specifically, framing tools as competing rather than complementary can encourage agents to commit to a single reliable source rather than diluting effort across a mixed routine.
• Modesty: The authors acknowledge limitations, including the use of only two open-weight models and the lack of a quantitative metric for "framing." They note that whether these behaviors constitute "failures" or "features" depends on the specific task requirements (decisive workflows vs. exploratory workflows).

The work suggests that current agentic harnesses may inadvertently encourage rigidity, and that designing the presentation of tool sets (framing) or explicit policy instructions can significantly alter an agent's ability to adapt to hidden environmental changes.