Technical Summary: FailureScope – Cross-Regime Behavioral Diagnosis of Language Model Weaknesses

1. Problem Statement

Standard Large Language Model (LLM) benchmarks typically report aggregate accuracy scores (e.g., MMLU, GSM8K), which obscure the specific capabilities a model lacks. As models are deployed in complex, multi-turn, and agentic scenarios, the gap between a single aggregate number and the specific diagnostic needs of practitioners widens. Furthermore, the research community has fragmented into separate subcommunities for single-turn benchmarks, multi-turn dialogue, and adversarial agent evaluations, each with distinct metrics and diagnostic frameworks that do not transfer across regimes.

The paper argues that the fundamental unit of diagnosis should not be the task itself, but the cross-model behavioral pattern: specifically, which models fail on a given probe. Current methods often focus on compressing benchmarks to reduce evaluation costs while preserving scalar rankings, often discarding the granular information about types of failures that practitioners need.

2. Methodology: FAILURESCOPE

The authors introduce FAILURESCOPE, a behavioral-diagnosis methodology that constructs failure taxonomies by clustering evaluation probes based on their cross-model pass/fail signatures. The approach is designed to be portable across different evaluation regimes.

Core Pipeline

The methodology follows a five-step algorithm that remains consistent across single-turn, multi-turn, and adversarial regimes:

1. Matrix Construction: Build a probe-by-model pass/fail matrix.
2. Imputation: Impute missing entries (common when models are not evaluated on all benchmarks) using column means. The authors note that using sentinel values (e.g., -1) injects false negative signals into distance calculations.
3. Embedding & Normalization:
   • Tasks are represented by a combined vector: $z_{combined} = [\alpha \hat{z}_{sem} \parallel (1-\alpha) \hat{z}_{beh}]$.
   • $\hat{z}_{sem}$ is a semantic embedding (Voyage-3-Large).
   • $\hat{z}_{beh}$ is an 18-dimensional behavioral vector encoding pass/fail status across models.
   • The authors find $\alpha=0.1$ (behavioral-dominant) optimal, as a small semantic component helps separate borderline clusters without drowning out the behavioral signal.
   • Vectors are L2-normalized.
4. Clustering: Reduce dimensionality via UMAP (or PCA fallback) to 30 dimensions, then cluster using HDBSCAN.
5. Labeling: Clusters are labeled via LLM-prompted exemplars (using the 5 nearest neighbors).

Validation Protocol: Leave-One-Model-Out (LOMO)

To validate the taxonomy without circularity, the authors use a LOMO protocol:

• One model ($m^*$) is held out.
• The taxonomy is rebuilt using the remaining $N-1$ models.
• The held-out model's weakness profile is then evaluated against this new taxonomy using two metrics:
  1. Diagnostic Efficiency: How well does taxonomy-conditioned sampling (selecting tasks to cover all clusters) recover the held-out model's weakness ranking compared to random or stratified sampling? (Measured by Kendall's $\tau$).
  2. Cross-Model Prediction: Can the failure profile of the held-out model be predicted based on its similarity to the training models? (Measured by AUC).

3. Key Contributions and Results

The paper validates FAILURESCOPE across three distinct regimes, demonstrating that the same primitive yields stable, interpretable clusters in each.

A. Single-Turn Benchmarks

• Dataset: 2,664 tasks from 6 benchmarks (GSM8K, ARC, MMLU, HumanEval, MBPP, IFEval) evaluated across 18 models.
• Clustering: The behavioral-dominant setting ($\alpha=0.1$) yields 25 stable clusters with a silhouette score of 0.12. Crucially, 86.7% of clusters span multiple benchmarks, indicating that failure signatures transcend specific dataset boundaries.
• Diagnostic Efficiency: Under LOMO, taxonomy-conditioned sampling achieves a Kendall's $\tau$ of 0.81 at 50 tasks. This is significantly better than random selection ($\tau=0.34$) and benchmark-stratified sampling ($\tau=0.64$).
• Prediction: Cross-model failure prediction reaches an AUC of 0.88 using a nearest-model approach.

B. Multi-Turn Dialogue

• Dataset: 363 frontier-hard conversations across 5 frontier models, covering 11 skill axes and 7 conversational depths.
• Clustering: The method recovers 6 clusters with a high silhouette of 0.92.
• Key Findings:
  • Deep-Context Collapse: A large cluster (162 tasks, mostly depth-6) where all models fail universally, a pattern not visible in single-turn analysis.
  • Model-Selective Resilience: One cluster (n=11) shows Claude Sonnet 4.6 failing at ~70% while other models fail at 85–100%.
  • DeepSeek-Immune Candidate: A small cluster (n=10) where DeepSeek succeeds while GPT and Claude families fail. The authors treat this as a hypothesis-generating signal requiring larger-scale replication.

C. Adversarial Agent Attacks

• Dataset: 630 multi-turn agent traces against a code-execution sandbox, involving three attack families and three frontier models.
• Clustering: The method identifies 3 distinct behavioral profiles:
  1. Meta-Failure of Evaluation: A 73–100 percentage-point gap between LLM-judge Attack Success Rates (ASR) and real network-execution ASR on cognitive-load attacks. The cluster reveals that the evaluation instrument (the judge) is unreliable, not the model.
  2. Selective Compositional Refusal: Claude Sonnet 4.6 exhibits a unique profile where it refuses the malicious payload at step 2 (21/30 cases) despite passing the first step, whereas GPT-5.4 executes the attack 92% of the time.
  3. Compositional Compliance: GPT-5.4 and GPT-5.4-mini show high execution rates, forming a "genuinely compromised" cluster distinct from the instrument failure and refusal clusters.

4. Significance and Claims

The paper claims that behavioral clustering is a portable diagnosis primitive that generalizes beyond any single benchmark.

• Portability: The same five-step algorithm produces stable, interpretable taxonomies across single-turn, multi-turn, and adversarial regimes. The authors emphasize that while silhouette scores differ across regimes (0.12, 0.92, 0.71) due to varying cluster counts and dimensions, the qualitative coherence and downstream utility (high $\tau$ and AUC) remain consistent.
• Efficiency: The method enables efficient structured diagnosis. Practitioners can evaluate a new model on approximately 50 taxonomy-conditioned tasks to recover a weakness profile with high fidelity ($\tau=0.81$), or predict the profile without evaluation using cross-model prediction (AUC 0.88).
• Discovery of Hidden Patterns: The approach uncovers failure modes invisible to single-benchmark analysis, such as depth-dependent universal collapse in multi-turn dialogue and the unreliability of LLM judges in specific adversarial contexts.

5. Limitations and Scope

The authors maintain a modest stance on their claims:

• Extrapolation: The LOMO validation holds for models within the same capability range. They do not claim a taxonomy built on 7B–70B models would transfer to significantly stronger systems (e.g., GPT-5 class) without rebuilding.
• Data Sparsity: The method relies on imputation for missing benchmark data; richer evaluation data would likely improve clustering quality.
• Labeling Subjectivity: Cluster labels are generated by LLMs. While the structure is robust, specific failure mode labels (e.g., reasoning vs. knowledge gap) can be ambiguous.
• Scope: The single-turn corpus does not cover multimodal reasoning or long-context tasks, though the multi-turn case studies address some of these gaps with smaller model pools.

The paper concludes by releasing the pipeline, three annotated corpora, cross-regime taxonomies, and 1,203 generated adversarial tasks to facilitate further research into structured model diagnosis.