Technical Summary: ReM-MoA: Reasoning Memory Sustains Mixture-of-Agents Scaling

Problem Statement

Mixture-of-Agents (MoA) architectures aim to improve inference-time scaling by organizing multiple Large Language Model (LLM) agents into layered reasoning pipelines. While increasing the depth (number of layers) and width (number of agents per layer) theoretically offers a path to scalable collective reasoning, existing MoA variants fail to sustain performance gains as architectures deepen. Empirical observations reveal three distinct failure modes:

1. Degradation: Standard MoA systems often peak at shallow depths and subsequently degrade, as raw inter-layer concatenation propagates both useful reasoning and accumulated errors.
2. Early Plateauing: Variants using local residual propagation (e.g., RMoA) fail to retrieve high-quality reasoning generated in non-adjacent earlier layers, causing performance to plateau.
3. Saturation and Diversity Collapse: Systems relying on intra-layer interaction (e.g., AttentionMoA) or identical accumulated context cause agents to converge toward redundant solution paths, collapsing exploration diversity.

Existing methods attempt to address these issues through information consolidation, inter-agent interaction, or quality-guided selection, but none successfully maintain the balance between quality preservation (retaining high-quality reasoning trajectories) and diversity preservation (preventing agents from converging to identical paths) as the system scales.

Methodology: ReM-MoA

The authors propose ReM-MoA, a memory-augmented MoA framework designed to sustain scaling by treating the problem as a joint quality-diversity preservation task. The framework consists of three interconnected components:

1. Ranked Reasoning Memory ($M$)

Unlike episodic memory systems that accumulate experience across tasks, ReM-MoA maintains a structured, cross-layer reasoning memory within a single problem instance.

• Mechanism: At each layer $l$, $N$ proposer agents generate reasoning traces. A Reviewer Agent (an LLM) comparatively evaluates these traces, assigning an integer score and a targeted rationale to each.
• Storage: The memory stores a union of entries from all previous layers ($M_{\le l} = \bigcup_{k=1}^l M_k$), where each entry is a triple of (trace, score, rationale).
• Access: The memory is append-only and supports read access from any subsequent layer, allowing deeper layers to revisit and build upon reasoning paths from arbitrary depths, not just adjacent ones.

2. Curated Diversified Memory Routing ($\Pi$)

To prevent diversity collapse, the framework ensures that agents within the same layer do not consume identical context.

• Mechanism: A routing scheme selects distinct subsets of the memory for each proposer agent. These subsets are drawn from three families of routing functions:
  • Contrastive ($Conn$): Combines high-scoring (Top) and low-scoring (Bot) traces to provide contrastive signals.
  • Positive Reinforcement ($Top$): Focuses on successful patterns.
  • Failure-mode Focus ($Bot$): Surfaces common pitfalls.
• Distribution: For $N$ agents, the routing functions are distributed across these families (e.g., one agent receives contrastive, another positive, another failure-focused). This ensures the layer as a whole explores the memory's content while individual agents receive distinct, quality-aware reference sets.

3. Reviewer Distillation (Optional Enhancement)

To improve the ranking quality of the Reviewer Agent, the authors introduce an optional pipeline using frontier-model distillation.

• Process: A strong teacher model (e.g., GPT-5.5) supervises the scoring and rationales of traces generated by ReM-MoA itself on a multi-domain training set.
• Fine-tuning: A smaller base Reviewer model is fine-tuned using parameter-efficient LoRA to imitate the teacher's judgments. This creates a general-purpose Reviewer capable of transferring across reasoning domains.

Key Contributions

1. Empirical Characterization of Scaling Failures: The paper identifies and characterizes three specific failure modes in existing MoA systems (degradation, plateauing, and diversity collapse) driven by raw concatenation, limited cross-layer access, and homogenized context.
2. ReM-MoA Framework: The proposal of a framework that sustains scaling through a Ranked Reasoning Memory and Curated Diversified Memory Routing. This approach preserves high-quality reasoning trajectories while maintaining exploration diversity by exposing agents to distinct combinations of successful and failed traces.
3. Optional Distillation Pipeline: The introduction of a multi-domain Reviewer distillation pipeline that upgrades the ranking module, enabling the system to generalize to held-out domains.
4. Comprehensive Evaluation: Extensive experiments across five reasoning benchmarks (Math, Formal Logic, Code, Knowledge, and Commonsense) demonstrating that ReM-MoA outperforms prior variants across both depth and width scaling.

Experimental Results

The authors evaluated ReM-MoA against Standard MoA, RMoA, and AttentionMoA across five benchmarks: MATH, MMLU-redux, Formal Logic, CRUX (code), and HellaSwag.

• Depth Scaling ($L=1$ to $9$, $N=3$):

  • Standard MoA degraded significantly (e.g., a ~9.1 point drop on MMLU-redux by $L=9$).
  • RMoA and AttentionMoA plateaued or saturated early.
  • ReM-MoA consistently improved as depth increased. The performance gap between ReM-MoA and the strongest baseline widened monotonically with depth (e.g., a +7.3 point lead on Formal Logic at $L=9$).
  • ReM-MoA$^*$ (with distilled Reviewer) further extended sustained scaling to all benchmarks, flipping saturation trends to continued growth on previously plateauing tasks.

• Width Scaling ($N=3$ to $8$, $L=3$):

  • ReM-MoA outperformed all baselines across all width configurations.
  • While width scaling showed diminishing returns compared to depth, ReM-MoA maintained a consistent lead, with the gap widening from +2.30 to +3.68 points as $N$ increased.

• Ablation Studies:

  • Removing cross-layer access (restricting reads to the immediate previous layer) caused the largest performance drop, confirming the necessity of accessing non-adjacent layers.
  • Removing routing diversification (sharing one reference set among all agents) triggered a regression similar to the full model's failure, proving that within-layer diversity is critical.
  • Removing rationales resulted in a smaller performance drop but preserved the scaling trend, indicating rationales act as refinement rather than essential scaffolding.

• Distillation Transfer:

  • A Reviewer distilled on four domains (holding out the evaluation domain) recovered 44% of the gain achieved by a multi-domain distilled model, demonstrating transferability to unseen reasoning domains.

Significance and Claims

The paper claims that structured cross-layer reasoning memory is a critical missing mechanism for scalable multi-agent inference. By decoupling the preservation of high-quality reasoning from the maintenance of exploration diversity, ReM-MoA overcomes the scaling collapse observed in prior architectures.

The authors assert that ReM-MoA establishes a new paradigm where scaling gains are not merely a function of aggregation or interaction, but of how reasoning traces are stored, ranked, and selectively routed. The framework demonstrates that with the correct memory architecture, multi-agent systems can sustain performance improvements as they scale in both depth and width, a property previously unattained in the field. The optional distillation pipeline further suggests that these benefits can be generalized across domains without task-specific verifiers.