🤖 AI

CoRe-Code: Collaborative Reinforcement Learning for Code Generation

CoRe-Code is a collaborative reinforcement learning framework that employs role-specialized Planner and Coder agents enhanced by Group Relative Policy Optimization (GRPO) to overcome the limitations of autoregressive decoding and improve the accuracy and efficiency of code generation across complex tasks.

Original authors: Zhihao Dou, Qinjian Zhao, Zhongwei Wan, Xiaoyu Xia, Sumon Biswas

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhihao Dou, Qinjian Zhao, Zhongwei Wan, Xiaoyu Xia, Sumon Biswas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a complex piece of furniture, like a bookshelf, but you don't have a blueprint. You just start hammering nails and gluing wood together, hoping it looks right as you go. This is how most current AI code generators work. They write code word-by-word (or "token-by-token"), which often leads to a structure that looks okay at first glance but falls apart when you try to use it. It might be locally coherent (the words make sense) but globally broken (the shelf won't hold books).

The paper introduces CoRe-Code, a new way to teach AI how to write code by acting like a construction crew with two distinct roles: a Planner and a Coder.

The Problem: The "Hammer-First" Approach

Current AI models are like enthusiastic apprentices who grab a hammer immediately. They might build a sturdy-looking leg for the table, but then realize they forgot to make the tabletop. Or, they might build a table that works, but it takes 100 years to assemble because they didn't plan the steps efficiently.

The Solution: The Architect and the Builder

CoRe-Code splits the work into two specialized agents:

The Planner (The Architect): Before any code is written, this agent draws a detailed blueprint. It doesn't write the code itself; it writes a step-by-step recipe called an "Algorithmic Thought." This recipe breaks the problem down into clear parts:
- Inputs/Outputs: What are we starting with, and what do we need to end up with?
- Linear Steps: The straight path from start to finish.
- Conditions: "If this happens, do that."
- Loops: "Repeat this action until we're done."
The Coder (The Builder): This agent takes the Architect's blueprint and actually builds the furniture (writes the code). Its only job is to follow the plan faithfully and efficiently.

The Secret Sauce: Learning by Doing (Reinforcement Learning)

The real magic of CoRe-Code isn't just having two agents; it's how they learn to work together.

Imagine a training camp where the Architect and the Builder practice together.

The Architect draws a plan.
The Builder tries to build it.
The Test: They put the finished product in a "test lab" (running the code against real-world problems).
The Feedback:
- If the code works perfectly and is fast, both get a high score.
- If the code fails, the system looks at why. Did the Builder mess up the instructions? Or did the Architect give a bad blueprint?

The paper uses a special training method called GRPO (Group Relative Policy Optimization). Think of this as a coach who compares different teams of Architect/Builder pairs. If Team A's plan leads to a sturdy table and Team B's plan leads to a wobbly mess, the coach tells Team A, "Great job, keep doing that," and tells Team B, "Your blueprint was the problem; try a different approach."

Crucially, the Architect learns indirectly. It doesn't get a score for the drawing itself; it gets a score based on how well the Builder could execute that drawing. This forces the Architect to write plans that are actually useful, not just pretty.

What They Found

The researchers tested this "Planner-Builder" team on several difficult coding challenges (like sorting lists of numbers or solving complex math puzzles).

Better Accuracy: The CoRe-Code team solved more problems correctly than other AI methods, including those that use "Chain of Thought" (talking to itself) or other multi-agent systems.
Efficiency: The code they wrote wasn't just correct; it was faster and used less computer memory.
Flexibility: The team showed that this "Planner-Builder" learning style isn't just for one type of task. They successfully applied it to other AI teams that include "Retrieval Agents" (who find information) and "Debugging Agents" (who fix errors), proving the method is a versatile tool for any AI construction crew.

In Summary

CoRe-Code is like upgrading an AI from a solo worker who guesses the steps, to a professional construction crew with a dedicated Architect and Builder who train together. By rewarding them based on the final, working result, they learn to communicate better, plan smarter, and build code that actually works.

Technical Summary: CoRe-Code

Problem Statement

Large Language Models (LLMs) have demonstrated strong capabilities in code generation but often rely on autoregressive decoding without global planning. This sequential, token-level generation mechanism frequently produces outputs that are locally coherent but globally suboptimal, leading to failures in test cases or inefficient algorithmic complexity. While recent approaches like Chain-of-Thought (CoT) and Multi-Agent Systems (MAS) attempt to introduce planning, they often suffer from limited role specialization and poor coordination. Existing MAS frameworks, where auxiliary agents (e.g., planners, retrievers) provide guidance to a central Coder, often yield only marginal performance gains or even negative "collaboration gains," indicating that the auxiliary agents fail to effectively support the Coder.

Methodology

The authors propose CoRe-Code (Collaborative Reinforcement Code), a framework designed to enhance inter-agent coordination and role specialization through a collaboration-aware reinforcement learning approach. The framework operates under a Planner–Coder paradigm and utilizes Group Relative Policy Optimization (GRPO) with verifiable rewards.

1. Algorithmic Thought

To structure the planning process, the framework introduces Algorithmic Thought, a structured representation for the Planner agent's output. Inspired by prior work, this representation decomposes reasoning into four core components:

Input-Output Definition: Clarifies functional objectives.
Linear Progression: Outlines sequential computational steps.
Conditional Logic: Specifies branch decisions.
Iteration: Describes repeated computations.
This structure aligns the Planner's reasoning with programming semantics, providing clearer guidance for the Coder.

2. Collaborative GRPO Framework

CoRe-Code employs a two-stage reinforcement learning process to jointly optimize the Planner and Coder agents, using downstream code execution results as verifiable rewards rather than unverifiable proxy rewards.

RL for Planner Specialization:
- The Planner generates $N$ distinct algorithmic thoughts.
- The Coder (frozen during this stage) translates these thoughts into code.
- Rewards: The quality of a thought is evaluated based on the accuracy (pass rate of test cases) and time complexity of the generated code. A sigmoid-based weighting scheme assigns higher rewards to thoughts leading to code that passes more tests and has lower complexity.
- The Planner is updated to maximize these rewards, learning to generate plans that better support downstream code synthesis.
RL for Coder Specialization:
- The specialized Planner generates a high-quality algorithmic thought.
- The Coder generates multiple code snippets based on this thought.
- Rewards: The Coder is rewarded for accuracy (passing test cases) and space efficiency. A space efficiency reward is assigned if code passes all tests, encouraging implementations that approach a target memory usage ( $O_{target}$ ) derived from the best-performing snippet.
- The Coder is updated to better align with and faithfully execute the Planner's instructions.

3. Collaboration Gain (CG)

The paper defines Collaboration Gain (CG) as a metric to quantify the effective contribution of auxiliary agents to the Coder's performance. CoRe-Code aims to maximize this gain by ensuring the Planner's output directly improves the Coder's success rate.

Key Contributions

CoRe-Code Framework: A novel collaboration-aware RL framework that optimizes role-specialized agents (Planner and Coder) using verifiable execution feedback.
Algorithmic Thought: A structured planning representation that decomposes algorithmic reasoning into input-output definitions, linear progression, conditional logic, and iteration, improving the interpretability and executability of plans.
Collaborative GRPO: An extension of GRPO that jointly optimizes the Planner and Coder. It uses downstream code execution (accuracy and complexity) as verifiable rewards to train the Planner, solving the challenge of optimizing intermediate plans that are difficult to verify in isolation.
Generalizability: Demonstration that the framework extends beyond the Planner–Coder paradigm to optimize other agents in multi-agent systems, such as Retrieval and Debugging agents within the MapCoder framework.

Experimental Results

The authors evaluated CoRe-Code on four benchmarks (LiveBench, MBPP, CodeContests, CodeForces) using three base models (Qwen2.5-7B-Coder-Instruct, Qwen2.5-14B-Coder-Instruct, and Qwen3-4B).

Performance: CoRe-Code consistently outperformed existing RL-based baselines (GRPO, Focused-DPO, CURE, CodeRL+) and multi-agent systems (SCoT, Reflexion, MapCoder) across all metrics, including Pass@1, Pass@5, and Average Pass Rate (APR).
Efficiency: The framework achieved higher efficiency in terms of execution time and memory usage compared to baselines.
Ablation Studies: Removing either the Planner RL or Coder RL components resulted in performance degradation, confirming that joint optimization provides complementary benefits.
Extension: When applied to MapCoder, independently reinforcing the Retrieval or Debugging agents with CoRe-Code improved performance over the original MapCoder baseline, particularly on complex competitive programming tasks.
Training Dynamics: Analysis showed a consistent upward trend in Collaboration Gain during training, indicating improved coordination between agents.

Significance and Claims

The paper claims that CoRe-Code represents one of the first verifier-guided RL-based multi-agent frameworks for code generation. Its significance lies in:

Addressing the "coordination gap" in multi-agent systems where auxiliary agents often fail to provide substantial support to the Coder.
Demonstrating that using verifiable execution results (rather than proxy rewards) allows for the effective training of planning agents.
Showing that role-specialized collaboration, when optimized via RL, leads to more accurate, efficient, and maintainable code generation across varying difficulty levels.
Highlighting flexibility, as the approach is not limited to the Planner–Coder paradigm but can be adapted to optimize diverse agents in broader multi-agent workflows.

The authors note limitations, including the reliance on specific open-source models and a fixed interaction format, suggesting future work could explore more diverse agent communication styles and model families.