Technical Summary: XDomainBench: Diagnosing Reasoning Collapse in High-Dimensional Scientific Knowledge Composition

1. Problem Statement

While Large Language Models (LLMs) are increasingly deployed for knowledge synthesis, their capacity for compositional generalization in scientific knowledge remains under-characterized. Existing benchmarks primarily focus on single-turn, restricted scenarios (e.g., static, one-shot queries) or isolated domain evaluations (e.g., ScienceQA, GPQA). These fail to capture the dynamic, interactive, and multi-constraint nature of real-world scientific discovery (AI4S), where research trajectories require interleaving constraints from disparate domains (e.g., combining analytical chemistry, solid-state physics, and financial cost assessment).

Consequently, there is a lack of systematic understanding regarding how and why LLMs degrade when multiple scientific constraints collide. Current evaluations cannot distinguish whether failures stem from missing domain knowledge or limitations in joint reasoning across domains.

2. Methodology: XDomainBench Framework

To address this gap, the authors introduce XDomainBench, a diagnostic benchmark designed to stress-test compositional generalization in interactive interdisciplinary scientific reasoning. The framework is built on two controllable dimensions:

1. Composition Order ($k$): The number of distinct domains involved in a session, measuring the breadth of integration ($k \in \{1, 2, 3, 4\}$).
2. Mixture Structure: The logical dependency and distribution of domains within a session, measuring the depth of joint reasoning.

Dataset Construction

• Scale: The benchmark comprises 8,598 interactive sessions across 20 scientific domains (organized into 6 major categories: Natural Science, Engineering, Social Science, Humanity, Art, and General) and 4 task types (Multiple Choice, Factual QA, Reasoning, Code).
• Domain Selection: Domains are combined using embedding-based semantic similarity to ensure realistic interdisciplinary connections, followed by human sanity checks.
• Trajectory Patterns: Sessions are constructed with 8 realistic trajectory patterns covering difficulty and domain-mixture dynamics:
  • Difficulty Patterns: Stable, Gradual Increase, Gradual Decrease, Spike, Fluctuate.
  • Mixture Patterns: Stable, Gradual Shift, Fluctuate.
• Quality Control: A multi-stage pipeline involving LLM generation, a three-model ensemble for validation (checking difficulty and domain weights), and detailed human auditing ensures that realized sessions match intended diagnostic targets.

Diagnostic Mechanisms

The benchmark attributes performance degradation to two root causes:

1. Direct Mechanism (Turn-Level): Immediate difficulty increases induced by domain composition, raising cognitive load and joint reasoning requirements before any multi-turn error propagation.
2. Indirect Mechanism (Session-Level): Interaction-amplified failures where specific trajectory patterns trigger:
   • Error Accumulation: Sustained performance decline after an onset turn.
   • Reasoning Breaks: Sudden accuracy drops indicating a failure to maintain continuity.
   • Domain Confusion: Realized domain reliance deviating from the intended mixture.

3. Key Results

Extensive evaluations across frontier proprietary models (e.g., GPT-5.2, Claude-4.5), open-weight models (e.g., Qwen, Llama), and MoE models (e.g., Mixtral) reveal a systematic reasoning collapse as composition order increases.

• Non-Linear Degradation: As $k$ increases from 1 to 4, performance drops significantly. For large models, turn-level accuracy (Recall) falls from 38.7% ($k=1$) to 27.1% ($k=4$). The degradation accelerates non-linearly at higher orders.
• Model Sensitivity: Small models exhibit steeper degradation and higher variance compared to large models. MoE models demonstrate superior performance, consistently outperforming others across all metrics, particularly in session success rates.
• Mechanism Validation:
  • Direct Effect: Even at Turn 1 (before multi-turn interaction), composing domains induces an immediate performance drop compared to single-domain baselines, confirming an integrative cognitive load.
  • Indirect Effect: Specific trajectory patterns (e.g., "Fluctuate" or "Spike" difficulty) correlate strongly with failure phenomena like error accumulation and reasoning breaks, leading to session collapse.
• Robustness: Semantic evaluation via LLM judges preserves the same monotonic degradation trend as token-level Recall, confirming the collapse is not an artifact of lexical overlap but a fundamental reasoning limitation.

4. Significance and Claims

The paper claims that scaling alone is insufficient to solve compositional scientific reasoning. The systematic collapse observed suggests that current architectures struggle with the cognitive load of high-dimensional domain compositions and the maintenance of coherence across multi-turn interactions.

XDomainBench provides a controllable testbed for:

• Diagnosis: Moving beyond simple accuracy scores to identify exact failure mechanisms (direct vs. indirect).
• Training Strategies: Enabling pattern-aware training where models learn to adapt to difficulty spikes and mixture shifts.
• Architecture Design: Motivating specialized approaches such as expert routing for domain mixtures (supported by MoE performance) and memory-augmented architectures to prevent error propagation.

The authors position XDomainBench not as a complete simulator of scientific discovery, but as a diagnostic tool to isolate intrinsic multi-turn interdisciplinary reasoning capabilities, paving the way for more robust and adaptive AI4S assistants.