MMoA: An AI-Agent framework with recurrence for Memoried Mixure-of-Agent

Technical Summary: MMoA – An AI-Agent Framework with Recurrence for Mixture-of-Agents

Problem Statement
The Mixture-of-Agents (MoA) framework has demonstrated success in enhancing Large Language Model (LLM) performance by aggregating outputs from multiple independent agents. However, existing MoA systems rely on static routing mechanisms that aggregate agent outputs without explicitly considering temporal or contextual dependencies across successive aggregation layers. This limitation prevents the system from adapting agent contributions based on historical routing decisions or the evolving context of the generation process, potentially leading to suboptimal efficiency and aggregation quality.

Methodology
The paper proposes MMoA, a novel recurrent MoA architecture that integrates an LSTM-based gating mechanism into the agent selection process. The core methodology involves:

1. Recurrent Gating Module: Instead of statically aggregating agent outputs, MMoA fuses the outputs of $n$ agents ($A_1(x), \dots, A_n(x)$) into a representation $z_t$. This representation is processed by an LSTM alongside the hidden state from the previous layer ($h_{t-1}$) to capture historical context, producing an updated hidden state $h_t$.
2. Dynamic Weighting: A gating vector $g$ is computed via a linear transformation and softmax activation applied to $h_t$. This vector dynamically weights the contributions of each agent for the current layer: $y = \sum g_i \cdot A_i(x)$.
3. Iterative Refinement: The architecture operates over multiple layers ($L$). At each layer, the module receives candidate responses and the aggregated output from the previous layer, allowing the model to iteratively refine its output by leveraging both current input features and historical routing information.
4. Training Objective: A specific loss function is defined for the router to minimize task-specific losses while maintaining exploration through entropy regularization ($H(p)$) and optionally enforcing load balancing across agents.

Key Contributions

• Router Design for MoA: The authors introduce a router adapted from Mixture-of-Experts (MoE) mechanisms, specifically utilizing LSTM/RNN-based gating to perform dynamic, context-aware agent selection within the MoA framework.
• Recurrent Gating Integration: This work is the first to integrate a recurrent gating mechanism into the MoA framework, enabling the system to make routing decisions that are informed by both current inputs and historical routing states.
• Efficiency-Accuracy Trade-off: The approach offers a scalable solution that dynamically activates fewer agents, significantly reducing computational overhead while maintaining high accuracy.

Experimental Results
The authors evaluated MMoA against standard baselines (including static MoA and ablated models) on instruction-following benchmarks: AlpacaEval 2.0, MT-Bench, and Arena-Hard.

• Accuracy: MMoA achieves accuracy comparable to traditional MoA with only minor drops.
  • On AlpacaEval 2.0, MMoA achieved a win rate of 58.0% compared to MoA's 59.8% (a 1.8 point drop), and an LC win rate of 61.5% vs. 65.1%.
  • On MT-Bench, the average score declined by only 0.05 points (from 9.25 to 9.20).
• Efficiency: The recurrent router significantly improves runtime efficiency by reducing the effective time complexity from $O(nL)$ to roughly $O(n+L)$ by reusing hidden states rather than re-computing gating networks for every agent at every layer.
  • On AlpacaEval 2.0, MMoA demonstrated up to a 4.6% improvement in runtime efficiency compared to non-recurrent baselines.
  • Inference time reductions were consistent across different agent counts ($n=2, 3, 6$), with MMoA completing tasks in roughly 54–56% of the baseline runtime.

Significance and Claims
The paper claims that MMoA represents the first work to introduce a router for agent selection specifically within the MoA framework, bridging the gap between static aggregation strategies and dynamic, recurrent routing mechanisms. The authors position this as a step toward "tuning the Mixture of Agent system in a Language Model way."

The significance of the work lies in its ability to provide a scalable and efficient solution for adaptive multi-agent LLM systems. By enabling context-aware and temporally informed routing, MMoA allows for the activation of fewer agents without sacrificing substantial performance, making it a viable approach for real-world applications where computational cost is a constraint.

Limitations
The authors modestly acknowledge several limitations:

• Evaluation is primarily focused on instruction-following benchmarks; generalization to long-context reasoning, tool use, or safety-critical tasks is unproven.
• The recurrent router is instantiated specifically with an LSTM-style mechanism; other designs (e.g., Transformer-based or RL-based) were not explored.
• The system introduces additional hyperparameters and may be sensitive to training data and agent pool composition, aspects not fully analyzed.
• The focus remains on inference-time efficiency and win-rate metrics, with qualitative failure modes and calibration left for future work.