🤖 AI

MARFT: Multi-Agent Reinforcement Fine-Tuning

This paper introduces Multi-Agent Reinforcement Fine-Tuning (MARFT), a comprehensive framework featuring a new Markov Game formulation called Flex-MG and a universal algorithmic approach designed to overcome the unique challenges of applying reinforcement learning to Large Language Model-based Multi-Agent Systems.

Original authors: Junwei Liao, Muning Wen, Jun Wang, Weinan Zhang

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Junwei Liao, Muning Wen, Jun Wang, Weinan Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Teaching a Team of AI Experts to Work Together

Imagine you have a team of highly intelligent robots (Large Language Models, or LLMs) that are great at talking and writing. Individually, they are smart. But when you ask them to solve a complex problem together—like planning a trip, writing a complex computer program, or conducting scientific research—they often struggle. They might talk over each other, forget who is supposed to do what, or get confused about the goal.

This paper introduces a new training method called MARFT (Multi-Agent Reinforcement Fine-Tuning). Think of MARFT as a specialized "team coaching" system designed to teach these AI agents how to collaborate effectively without losing their individual smarts.

The Problem: Why Old Methods Fail

The authors explain that we can't just use the old rules for training robot teams (called Multi-Agent Reinforcement Learning, or MARL) on these new AI teams. Here is why:

The "Simultaneous" vs. "Sequential" Mismatch:
- Old Way: Traditional training assumes everyone acts at the exact same time, like a soccer team kicking a ball simultaneously.
- Real Life: AI agents usually work in a sequence. One agent might say, "I need the date," and the next agent waits to hear that before saying, "Okay, I'll book the flight."
- The Fix: MARFT treats the team like a relay race or a conversation where one person speaks, then the next, rather than everyone shouting at once.
The "Identity Crisis":
- Old Way: In traditional training, agents are often identical twins.
- Real Life: In a real AI team, one agent is the "Planner," one is the "Coder," and one is the "Checker." They have different jobs and different "personalities" (prompts).
- The Fix: MARFT teaches the system to respect these specific roles. It ensures the "Coder" doesn't try to act like the "Planner."
The "Forgetting" Risk:
- Old Way: If you train a robot too hard to win a game, it might forget how to speak human language.
- Real Life: We want these AI agents to get better at solving tasks without forgetting how to write good sentences or understand context.
- The Fix: MARFT is a "Fine-Tuning" method. It's like a gentle nudge rather than a heavy overhaul. It improves their teamwork while keeping their original language skills intact.

The Solution: How MARFT Works

The paper proposes a new framework called Flex-MG (Flexible Markov Game). Here is how it works in plain English:

The "Dependency Map": Imagine a flowchart. MARFT creates a map that says, "Agent B cannot act until Agent A finishes." This handles the fact that AI agents often depend on each other's outputs.
The "Coach" (Central Critic): MARFT uses a central "coach" (a neural network) that watches the whole team. Instead of just praising an agent for doing its own job well, the coach praises the agent for helping the whole team win.
Token-Level Training: Instead of just rewarding the final answer (like "Did you get the flight booked?"), MARFT looks at every single word (token) the agents generate. It's like a teacher grading a student's essay sentence by sentence, not just the final grade. This helps the agents learn how to think, not just what to say.

The Experiments: Does It Actually Work?

The authors tested this method on two difficult tasks: Math and Coding.

The Setup: They created teams of AI agents (2, 3, or 4 agents) with different roles (e.g., Planner, Solver, Verifier).
The Comparison: They compared MARFT against the standard method (Independent PPO), where agents are trained separately and then thrown together.
The Results:
- Better Scores: MARFT teams consistently scored higher on math problems and coding benchmarks than the standard teams.
- Stability: The standard method sometimes crashed or got confused during training. MARFT improved steadily and smoothly.
- The "Anonymous" Surprise: In a cool experiment, they tried training teams without giving them specific job titles (like "Planner" or "Solver"). Surprisingly, the anonymous teams learned to figure out their own roles and performed even better than the pre-assigned teams in some cases. This suggests MARFT is flexible enough to let the AI figure out its own organization.

The "Flex-MG" Concept

The paper introduces a new mathematical model called Flex-MG. Think of this as a new rulebook for a board game.

In the old rulebook, everyone moves at once.
In the Flex-MG rulebook, the game changes as you play. Sometimes you move alone; sometimes you have to wait for a teammate. Sometimes the rules change based on what happened in the previous turn. This rulebook is designed specifically for the messy, dynamic way real AI teams work.

What's Next? (Challenges)

The authors admit that while MARFT is a great step forward, there are still hurdles:

Hard to Find Practice Fields: It's difficult to build realistic, dynamic environments (like a simulated city or a complex office) where these AI teams can practice.
Data Hungry: Like all AI training, it requires a lot of high-quality examples to learn from.
No Standard Tool: There isn't a single, easy-to-use "toolbox" yet that combines all these features for everyone to use.

Summary

In short, MARFT is a new way to train teams of AI agents. Instead of treating them like identical robots playing a synchronized game, it treats them like a diverse group of human experts working in a sequence. It uses a "coach" to guide them, respects their specific roles, and ensures they get better at teamwork without forgetting how to speak human language. The results show that this method makes AI teams smarter, more stable, and better at solving complex problems like math and coding.

Technical Summary: MARFT: Multi-Agent Reinforcement Fine-Tuning

Problem Statement

Large Language Model (LLM)-based Multi-Agent Systems (LaMAS) have demonstrated significant potential in solving complex, agentic tasks requiring multifaceted reasoning and collaboration. However, optimizing these systems remains challenging. While Reinforcement Learning (RL) has proven effective for single-agent LLMs through Reinforcement Fine-Tuning (RFT), directly applying conventional Multi-Agent Reinforcement Learning (MARL) to LaMAS introduces significant friction.

The core problem lies in the mismatch between traditional MARL assumptions and the unique characteristics of LaMAS. Traditional MARL often assumes synchronous agent execution, homogeneous agent structures, and static system organizations. In contrast, LaMAS operates with:

Asynchronous interactions: Agents often act sequentially or depend on the outputs of others.
Heterogeneity: Agents may possess different base models, vocabularies, external tools, and input/output formats.
Profile-aware design: Agents require specific system prompts (profiles) to define their roles, which are not present in standard MARL benchmarks.
Dynamic organization: Task decomposition and agent coordination can vary dynamically based on the problem context.

Naive application of single-agent RFT or standard MARL (e.g., Independent PPO) to these settings leads to unstable training, inactive agents, and inefficient communication. There is a lack of a unified framework that integrates RFT principles with the specific dynamics of multi-agent LLMs.

Methodology

1. Theoretical Formulation: Flex-MG

To address these discrepancies, the authors propose Flex-MG (Flexible Markov Game), a new game-theoretic formulation denoted as $\langle V, N, S, A, T, R, \gamma, D \rangle$ .

Dependency Function ( $D$ ): A novel component $D: A \times A \rightarrow \{0, 1\}$ explicitly models inter-agent dependencies. If $D(a_i, a_j) = 1$ , agent $j$ 's decision is conditionally dependent on agent $i$ 's action. This allows the framework to capture the asynchronous and sequential nature of LaMAS workflows.
State and Action Spaces: The framework accommodates heterogeneous agents where action spaces and observation spaces can vary across agents, incorporating agent profiles into the observation vector.

2. The MARFT Framework

The paper introduces Multi-Agent Reinforcement Fine-Tuning (MARFT), a paradigm that adapts RFT to the multi-agent domain. Key methodological components include:

Sequential Decision-Making Reformulation: Leveraging the Multi-Agent Advantage Decomposition Theorem, MARFT reframes the multi-agent problem as a sequential decision-making process. Agents generate actions in a specific order (or permutation), conditioning their policy on the actions of predecessors ( $a_{1:m-1}$ ).
Agent-by-Agent Update: To ensure monotonic improvement and stability, MARFT employs a sequential update scheme where agents are updated one by one, rather than simultaneously. This mitigates non-stationarity issues common in concurrent updates.
Action and Token Normalization: To handle the varying lengths of LLM-generated outputs (e.g., Chain-of-Thought vs. direct answers), the framework introduces normalization techniques that divide joint probabilities by token or word length, preventing bias toward shorter or longer sequences.
Two Instantiations:
- Action-Level: Optimizes the policy at the level of complete agent actions (sequences of tokens).
- Token-Level: Treats every token as an individual action. This reduces optimization complexity from $O(|V|^L)$ to $O(|V| \times L)$ but introduces sparsity in rewards. The authors derive specific Bellman backups for this token-level MG to handle the delayed reward signal.
Implementation Strategy: The framework utilizes LoRA (Low-Rank Adaptation) for efficient fine-tuning, allowing agents to share a base model while maintaining distinct adapters for specific roles. It employs a centralized critic for global value estimation while maintaining decentralized execution.

Key Contributions

Conceptualization of MARFT: The paper formally defines Multi-Agent Reinforcement Fine-Tuning, distinguishing it from both single-agent RFT and traditional MARL by highlighting the unique constraints of LaMAS (asynchronicity, heterogeneity, and profile dependence).
Flex-MG Formulation: The introduction of the Flexible Markov Game provides a mathematical foundation that explicitly models inter-agent dependencies, bridging the gap between static MARL environments and dynamic LaMAS workflows.
Algorithmic Framework: The authors present a scalable, modular algorithm (Algorithm 3) that integrates sequential advantage decomposition, agent-by-agent updates, and token-level adaptations.
Open-Source Implementation: A complete, open-source implementation of the MARFT framework is provided to facilitate adoption and further research.

Experimental Results

The authors evaluated MARFT on mathematical reasoning (DeepScaler, AIME2024, MATH500, etc.) and coding tasks (DeepCoder, LiveCodeBench, CodeForces) using Qwen2.5 base models.

Performance vs. Independent PPO (MAPoRL): MARFT consistently outperformed Independent PPO (the standard baseline for multi-agent LLMs) across various benchmarks. The performance gap widened as the number of agents increased (from Duo to Quad configurations), suggesting MARFT scales better with system complexity.
Training Stability: Training curves demonstrated that MARFT achieves monotonic improvement and avoids the "crashes" or instability observed in Independent PPO during training.
Role Specification Ablation: An ablation study compared pre-configured role descriptions (e.g., "Planner," "Solver") against anonymous agents. Surprisingly, the anonymous setting (where agents are not explicitly assigned roles via system prompts) outperformed the pre-configured setting in early and middle training stages. This suggests that allowing agents to self-evolve role divisions may be more effective than rigid human-defined roles during the fine-tuning process.
Scaling Observations: While increasing the number of homogeneous agents did not yield proportional performance gains (due to diminishing returns and redundant reasoning paths), MARFT consistently improved the absolute capability of the system at every scale.

Significance and Claims

The paper positions MARFT as a necessary evolution in the field of agentic systems. Its significance lies in:

Bridging Theory and Practice: It provides a principled approach to applying RL fine-tuning to the complex, real-world dynamics of multi-agent LLMs, moving beyond the limitations of synchronous, homogeneous MARL assumptions.
Enabling Resilient Collaboration: By preserving the foundational language capabilities of pretrained LLMs while enhancing their collaborative reasoning, MARFT aims to create systems that are robust, adaptive, and human-aligned.
Roadmap for Future Research: The authors identify critical open challenges, including the lack of dynamic training environments, low sample efficiency, the absence of unified communication protocols, and the need for standardized frameworks. They propose MARFT as a foundational step toward realizing scalable, collaborative AGI systems capable of tackling complex open-world problems.

The work concludes that while significant challenges remain, the MARFT paradigm offers a robust mechanism for advancing the collective intelligence of LLM-based multi-agent systems, particularly in domains requiring distributed reasoning and dynamic coordination.