When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors

1. Problem Statement

The rapid proliferation of Large Language Model (LLM) agents has led to a phenomenon of behavioral homogenization. Despite diverse origins, many agents exhibit nearly identical reasoning steps, redundant tool-calling habits, and shared failure modes. This suggests that many models are "distilled echoes" of a few dominant teacher models rather than independent breakthroughs.

Key Challenges:

• Existing Metrics are Insufficient: Current evaluation methods focus on static dialogue similarity or overall task success. They fail to distinguish between mandatory behaviors (actions strictly required for task success, e.g., calling a specific API to cancel a flight) and non-mandatory behaviors (autonomous preferences, such as how a model phrases a response or which optional tools it chooses).
• The "Bad Habit" Convergence: Without isolating non-mandatory patterns, it is impossible to determine if two models converge because there is only one correct path, or because one is blindly shadowing the other's "bad habits" (e.g., verbose reasoning, redundant verification).

2. Methodology

The authors propose a framework to quantify agent distillation by isolating non-mandatory behavioral patterns using two complementary metrics: Response Pattern Similarity (RPS) and Action Graph Similarity (AGS).

A. Response Pattern Similarity (RPS)

RPS measures verbal alignment by analyzing how models phrase responses, structure explanations, and express reasoning.

• Stage Annotation: Trajectories are segmented into five canonical stages to handle varying trajectory lengths:
  1. Authentication (Identity verification)
  2. Elicitation (Requesting missing parameters)
  3. Execution (Invoking tools)
  4. Verification (Confirming critical actions)
  5. Notification (Reporting results)
• Scoring: An LLM judge compares responses at shared stages across three dimensions:
  • Style: Wording habits and vocabulary.
  • Structure: Sentence patterns and templates.
  • Alignment: Consistency between reasoning (thought) and execution (action).
• Output: A score from 1 to 5, where 5 indicates high similarity.

B. Action Graph Similarity (AGS)

AGS measures tool-use habits by modeling tool call sequences as directed graphs. It specifically isolates optional behaviors from mandatory ones.

• Graph Construction: Nodes represent tool calls ($v = \{name, args, result\}$). Edges represent:
  • Sequential Edges: Temporal order of execution.
  • Dependency Edges: Data flow where an argument in a later call is derived from the result of an earlier call (verified by an LLM to avoid false positives).
• Sub-metrics:
  1. $S_{node}$ (Optional Tool Agreement): Measures the consistency of optional tool choices. Mandatory tools (required for all successful paths) are excluded to prevent score inflation. It calculates the agreement rate on tools that some models use but others skip.
  2. $S_{seq}$ (Sequential Pattern Similarity): Measures local execution habits (e.g., post-write verification, pre-write confirmation, error retry rates).
  3. $S_{dep}$ (Dependency Pattern Similarity): Measures data dependency structures (e.g., output reuse rate, depth of dependency chains, fan-out of outputs).

3. Key Contributions

1. First Framework for Distillation Quantification: The paper introduces the first systematic method to disentangle mandatory task behaviors from non-mandatory patterns in tool-use trajectories.
2. Novel Metrics (RPS & AGS):
   • RPS provides interpretable metrics for verbal similarity (Style, Structure, Alignment).
   • AGS provides behavioral fingerprints for tool invocation, specifically isolating optional choices ($S_{node}$) which are strong indicators of distillation.
3. Comprehensive Evaluation: Evaluated 18 models from 8 major providers (Anthropic, OpenAI, DeepSeek, Moonshot, ByteDance, Google, Qwen, Zhipu) against Claude Sonnet 4.5 (thinking) as a reference "oracle."
4. Controlled Validation: A controlled distillation experiment (fine-tuning Qwen2.5 on Claude trajectories) confirmed that AGS specifically detects teacher-student convergence, whereas baseline metrics (like Graph Edit Distance) fail to distinguish between teacher and non-teacher controls.

4. Key Results

• Within-Family vs. Cross-Family: Models from the same provider (e.g., Anthropic's Opus and Sonnet) showed high internal consistency. However, cross-family pairs with high similarity suggest behavioral inheritance.
• The Kimi-K2 Anomaly:
  • Kimi-K2 (thinking) exhibited the highest similarity to the reference model (Claude Sonnet 4.5) among all non-Anthropic models.
  • AGS Score: 82.7% (highest non-Anthropic).
  • $S_{node}$: 82.6% (exceeding Anthropic's own Opus 4.1).
  • $S_{dep}$: 94.7% (exceeding Anthropic's Opus 4.1).
  • RPS Score: 3.65 (highest non-Anthropic).
  • Implication: Kimi-K2 shares more optional tool choices and dependency patterns with Claude than Anthropic's own models do, suggesting strong distillation from Claude.
• Distinct Dimensions: RPS and AGS capture largely independent behavioral dimensions (Pearson $r = 0.491$), meaning a model can be verbally similar but structurally different, or vice versa.
• Controlled Experiment: When a student model was distilled from a teacher:
  • AGS increased toward the teacher (+0.13) and decreased toward a non-teacher control (-0.05).
  • Baseline Metrics (GED) increased toward both the teacher and the control, failing to distinguish the source of convergence.

5. Significance and Implications

• Ecosystem Transparency: The findings provide empirical evidence for the "déjà vu" feeling in the agent ecosystem. Many models are not independent but are converging on the specific (and sometimes suboptimal) habits of a few dominant teachers.
• Robustness Concerns: If multiple "independent" models share the same failure modes and redundant tool habits, the ecosystem lacks true robustness. They will fail in the exact same way, undermining the value of ensemble methods.
• Diagnostic Tool: RPS and AGS serve as critical diagnostic signals for researchers and regulators to detect hidden distillation relationships and benchmark contamination that traditional performance metrics miss.
• Future Directions: The paper highlights the need for diverse training signals to prevent the collapse of behavioral diversity in the agent ecosystem.

Code Availability: The authors have released their code at https://github.com/Syuchin/AgentEcho.