Disentangling Intrinsic Importance from Emergent Structure in Multi-Expert Orchestration

Technical Summary: Disentangling Intrinsic Importance from Emergent Structure in Multi-Expert Orchestration

Problem Statement
Multi-expert systems, where multiple Large Language Models (LLMs) collaborate via an orchestrator to solve complex tasks, are increasingly adopted for high-performance reasoning. However, the orchestration policies governing expert interaction, sequencing, and selection remain largely opaque. Current frameworks often treat routing as a black box optimized for performance, lacking tools to distinguish between relational importance (how frequently an expert is selected or acts as a hub) and intrinsic importance (the functional necessity of an expert's internal representations). This opacity makes it difficult to diagnose failure modes, such as brittle routing, redundancy, or the reliance on experts that are frequently selected but functionally marginal.

Methodology: The INFORM Framework
The authors introduce INFORM, an interpretability analysis framework that treats orchestration as an explicit, analyzable computation. INFORM does not modify the collaboration protocol but probes a learned orchestrator ($O_\theta$) to decouple three key dimensions: interaction structure, execution order, and functional attribution.

The framework operates on a canonical orchestrator managing a consortium of frozen, instruction-tuned experts ($E = \{E_1, \dots, E_N\}$). The orchestrator utilizes two primary modules:

1. Interaction Module: Computes a conditional transition matrix $C(x)$ using query-key attention and a semantic prior to model the compatibility between experts.
2. Selection Module: Determines the marginal selection distribution $s(x)$ using a Gumbel-Softmax trick to allow differentiable sampling.

INFORM extracts insights through three specific probes:

• Probing Interaction Structure: Analyzes the entropy and rank of the transition matrix $C(x)$ to define Relational Importance ($u_j$), measured as the total incoming routing mass to an expert. This identifies interaction hubs.
• Probing Sequencing Decisions: Analyzes the entropy of the initial selection distribution $s(x)$ to understand how the orchestrator learns to order experts, distinguishing between global competence and context-specific necessity.
• Functional Attribution via Gradient Sensitivity: Computes Intrinsic Importance ($I(E_i)$) by backpropagating selection decisions to expert representations. Specifically, it calculates the gradient norm of the log-probability of the selected expert with respect to the expert's encoded representation ($\|\nabla_{h_i} \log P(E_i | x)\|^2$). This measures how sensitive the routing decision is to the semantic content of an expert, distinguishing content-dependent selection from heuristic selection.

Experimental Setup
The authors evaluated INFORM on a homogeneous consortium of ten instruction-tuned experts (drawn from LLaMA-3.1 8B, Qwen3 8B, and DeepSeek-R1 8B) with controlled decoding-temperature variations to induce behavioral diversity. A secondary heterogeneous consortium (spanning 1B–7B parameters) was used to test generalizability. Experiments were conducted on GSM8K, HumanEval, and MMLU. The orchestrator was trained to maximize task performance and align with a larger oracle LLM (absent during inference).

Key Results and Findings

1. Divergence Between Relational and Intrinsic Importance: The study reveals a significant misalignment between routing frequency and functional necessity. Frequently selected experts often act as "interaction hubs" with limited intrinsic influence, while sparsely routed experts can be structurally critical. For instance, masking the most intrinsically important expert (via gradient attribution) induced a 5.5× higher routing KL divergence on MMLU compared to masking frequently selected peers, confirming that routing mass is a poor proxy for structural dependency.
2. Asynchronous Emergence of Orchestration Behaviors: Orchestration dynamics emerge asynchronously. The system establishes "who to trust" (centralization of routing mass) before resolving "how confidently to route" (stabilization of routing entropy). Furthermore, expert ordering remains structured but non-deterministic, with the orchestrator learning soft constraints rather than rigid sequences.
3. Task-Dependent Sensitivity: Prompt perturbation studies show that the orchestrator's sensitivity is task-specific. On GSM8K, the system is highly sensitive to numerical token removal; on HumanEval and MMLU, it is more sensitive to sentence shuffling and reasoning cue removal.
4. Homogeneous vs. Heterogeneous Dynamics: In the homogeneous setting, routing converges rapidly to confident, centralized patterns. In the heterogeneous setting (1B–7B models), routing is more volatile and less centralized, with gradient attribution spreading across a wider, more diverse subset of models, though a persistent alignment gap remains where frequently selected models do not necessarily possess the highest gradient influence.
5. Ablation and Baseline Comparisons:
   • Ablations: Removing relational structure or intrinsic scoring degrades performance, particularly on heterogeneous tasks like MMLU. The full INFORM setup balances structural stability with semantic precision.
   • Baselines: Compared to the rigid, role-based MetaGPT framework, INFORM achieves a ~3.5× speedup on HumanEval with a ~1.4% performance gain by adaptively pruning unnecessary expert invocations.

Significance and Claims
The paper claims that INFORM provides a practical tool for diagnosing structural dependencies and interaction hubs in multi-expert systems that are invisible to accuracy metrics alone. By disentangling intrinsic importance from emergent routing structure, the framework exposes:

• Redundancy: Experts that are frequently routed but functionally unnecessary.
• Brittleness: Systems that rely on specific interaction topologies which collapse when key intrinsic experts are masked.
• Efficiency: The ability to identify minimal functional trajectories for task solving, reducing computational overhead compared to rigid protocols.

The authors emphasize that their approach captures local computational sensitivity rather than establishing a formal causal graph. Consequently, while the framework requires white-box access to the orchestrator's internal representations and gradients, it allows the constituent experts to remain black-box. The work positions interpretability-driven analysis as essential for improving the reliability, efficiency, and safety of learned orchestration policies beyond what performance metrics alone can capture.