🤖 AI

Cooperative Coevolution for Resource-Constrained Agentic LLM Post-Training

The paper introduces Cooperative Parameter-subspace Evolution Strategy (CoPES), a memory-efficient method that decomposes the parameter space to enable effective resource-constrained post-training of agentic LLMs, achieving 92% of full-parameter GRPO's performance gains with significantly lower GPU memory requirements.

Original authors: Zhiyuan Wang, Shengcai Liu, Jiahao Wu, Ning Lu, Hui Ouyang, Shaofeng Zhang, Haoze Lv, Ke Tang

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhiyuan Wang, Shengcai Liu, Jiahao Wu, Ning Lu, Hui Ouyang, Shaofeng Zhang, Haoze Lv, Ke Tang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers don't just chat, but actually do things. They can browse the web, run code, and solve complex puzzles by taking many steps, checking their work, and trying again. This is the exciting frontier of "agentic" AI. But there's a catch: teaching these digital helpers to get better is incredibly expensive. The usual way to train them is like a high-stakes video game where the computer tries to figure out the perfect move by calculating the "gradients" (the mathematical slope of success) for every single part of its brain at once. This requires a massive amount of computer memory, like trying to hold a library of books in your head while solving a riddle. If the task is long and complicated, the memory needed explodes, making it impossible for most people to train these agents without a supercomputer.

Enter a different approach called "Evolution Strategies." Instead of calculating slopes, this method is more like a game of "Hot or Cold." You make tiny, random changes to the AI's brain, see if it gets better, and if it does, you keep the change. It's much lighter on memory because you don't need to remember the path you took to get there. However, there's a new problem: because you're guessing randomly, you need to try a lot of times to get good results, which takes a huge amount of time and computer power. It's like trying to find a needle in a haystack by checking one straw at a time; you'll eventually find it, but you might be there for a century. The big question for scientists is: Can we keep the memory savings of the "Hot or Cold" method but make it fast enough to be useful on just a few regular computers?

This paper introduces a clever new trick called CoPES (Cooperative Parameter-subspace Evolution Strategy) to solve exactly that problem. The researchers realized that trying to change the entire AI brain at once is inefficient. So, they broke the brain down into smaller, manageable chunks. Imagine you are trying to tune a massive orchestra to sound perfect. Instead of asking every single musician to change their instrument randomly at the same time (which creates chaos), you ask the violin section to try a new setting, then the brass section, then the percussion, while everyone else stays still. By testing these smaller groups one by one, but keeping the whole orchestra playing together, you can figure out what works much faster.

In the paper, the authors tested this on a specific AI model (Qwen3.5-4B) trained to solve math problems. They compared their new "orchestra tuner" method against the standard "Hot or Cold" method and the heavy-memory "gradient" method. They found that CoPES was a game-changer for resource-constrained settings. While the standard random method needed about 480.60 GPU-hours (a measure of computer time) to reach a good level of performance, CoPES achieved nearly the same results in just 68.65 GPU-hours. Even more impressive, under a strict time limit where the heavy-memory method could only train for 16 steps, CoPES recovered 92% of the performance gains that the heavy method achieved, whereas the standard random method only managed 67%.

The study also showed that CoPES is much more practical for everyday hardware. While the heavy-memory method required a massive setup of 8 high-end GPUs to even start training on long tasks, CoPES could run successfully on a single, standard 24GB GPU. This means that instead of needing a data center, a researcher with a single powerful computer could now train advanced AI agents. The authors also tested this on question-answering tasks, not just math, and found the method worked well there too, consistently beating the standard random approach.

The paper suggests that by breaking the problem into cooperative subgroups and carefully combining the results, we can get the best of both worlds: the low memory cost of evolutionary methods and the high speed of gradient-based methods. It doesn't claim to be a magic bullet that solves everything instantly, but it demonstrates a significantly better trade-off between memory and time. The results indicate that with this new strategy, training powerful AI agents becomes feasible for many more people, turning a task that once required a supercomputer into something achievable on a single workstation.

Technical Summary: Cooperative Coevolution for Resource-Constrained Agentic LLM Post-Training

Problem Statement

Tool-using Large Language Model (LLM) agents generate long, multi-turn trajectories to solve complex tasks. While post-training with verifiable rewards (e.g., via Reinforcement Learning) is a scalable method to improve agent behavior, it faces a critical bottleneck in resource-constrained environments.

Gradient-based methods, such as Group Relative Policy Optimization (GRPO), require backpropagation through these long trajectories. This necessitates storing model weights, gradients, optimizer states, and, crucially, intermediate activations. As context length increases, activation memory becomes prohibitive, often requiring high-end GPU clusters that are unavailable to many researchers. Conversely, Evolution Strategies (ES) offer a memory-efficient alternative by avoiding backpropagation entirely, requiring only forward generation and reward evaluation. However, standard ES is computationally expensive, typically requiring significantly more GPU-hours than gradient-based methods to achieve comparable performance. In resource-constrained settings with limited GPU counts, this high computational demand translates into impractically long wall-clock training times.

The core challenge addressed is how to improve the optimization efficiency of ES without sacrificing its memory advantages, specifically for full-parameter agentic LLM post-training.

Methodology: Cooperative Parameter-Subspace Evolution Strategy (CoPES)

To address the efficiency bottleneck, the authors propose Cooperative Parameter-subspace Evolution Strategy (CoPES), a cooperative coevolutionary method adapted for full-parameter ES.

Core Mechanism

CoPES decomposes the high-dimensional full parameter space ( $\mathbb{R}^d$ ) into $K$ lower-dimensional, mutually disjoint subspaces. Instead of perturbing the entire parameter space simultaneously for every evaluation, CoPES performs cooperative searches within these subspaces:

Random Partitioning: At each training step, the parameter indices are randomly partitioned into $K$ equally sized subspaces ( $S_1, \dots, S_K$ ).
Cooperative Subspace Search: The total perturbation budget $N$ is distributed across subspaces ( $N_k = N/K$ ). For each subspace $k$ , $N_k$ perturbed models are generated by adding noise only to parameters in $S_k$ , while all other parameters remain fixed at their current values. This provides a shared full-model context for evaluating the impact of changes in a specific subspace.
Dimension-Aware Perturbation Scaling: To ensure the magnitude of subspace perturbations matches the expected squared norm of a full-space perturbation, the perturbation scale $\sigma_k$ is adjusted: $\sigma_k = \sqrt{K}\sigma$ .
Joint Reward Standardization: Unlike standard ES which standardizes rewards within a single population, CoPES pools the rewards from all $N$ perturbed models across all $K$ subspaces. It computes a global mean ( $\mu_c$ ) and standard deviation ( $s_c$ ) to standardize all rewards ( $\hat{r}_{k,i}$ ). This provides stable normalization statistics and a common reward scale for combining subspace updates.
Synchronous Update: The subspace directions are estimated and composed into a full-parameter update: $\theta_{t+1} = \theta_t + \alpha \sum_{k=1}^K P_k g_k$ , where $g_k$ is the estimated gradient direction for subspace $k$ .

Implementation Details

To maintain memory efficiency, CoPES employs:

Seed Replay: Perturbations are regenerated from random seeds rather than storing full perturbation vectors.
Weight Backup: Pre-perturbation weights are backed up in CPU memory to ensure every evaluation starts from the exact same model state, avoiding floating-point residual errors.
Chunked Processing: Parameters are processed in chunks to manage memory usage.

Key Contributions

Novel Algorithm: CoPES is presented as the first cooperative coevolutionary method designed specifically for full-parameter agentic LLM post-training. It combines parameter-space decomposition with joint reward standardization.
Efficiency-Memory Trade-off: The method improves fixed-budget optimization efficiency while preserving the memory-efficient, full-parameter nature of standard ES. It avoids the backpropagation overhead of gradient-based methods and the high wall-clock time of standard ES.
Empirical Validation: Extensive experiments on math and multi-hop question-answering (QA) tasks validate the method across multiple metrics, including hardware feasibility tests and controlled ablations.

Experimental Results

The authors evaluated CoPES on a Qwen3.5-4B model across five math benchmarks (AIME 2024, AIME 2025, GSM8K, MATH-500, MATH-Test) and three QA benchmarks (HotpotQA, 2Wiki, MuSiQue).

Performance vs. Gradient-Based RL (GRPO)

Memory Efficiency: Theoretical analysis shows CoPES requires less than 1/8th of the GPU memory of full-parameter GRPO. In practice, CoPES can be trained on a single 24GB GPU, whereas full-parameter GRPO requires 8x 48GB GPUs for 128K context lengths.
Fixed-Budget Performance: Under a GPU-hour budget equivalent to the best validation checkpoint of full-parameter GRPO (16 training steps):
- CoPES recovered 92% of the validation-accuracy gain achieved by full-parameter GRPO.
- Standard ES recovered only 67%.
- CoPES consistently outperformed both standard ES and LoRA-based GRPO across all pass@k metrics on the five math benchmarks.
Scaling: On the challenging AIME 2024 and AIME 2025 benchmarks, CoPES surpassed full-parameter GRPO at larger $k$ values (pass@k) despite using the same fixed training budget.

Performance vs. Standard ES

CoPES significantly outperformed standard ES across all evaluated metrics.
Ablation Studies:
- Subspace Count ( $K$ ): Performance improved as $K$ increased from 1 to 4, but dropped sharply at $K=8$ due to insufficient population per subspace ( $N_k=5$ ).
- Joint Standardization: Using joint reward standardization across all subspaces yielded higher pass@1 scores compared to independent standardization within each subspace.
- Perturbation Scale: The performance gain was not solely due to the larger per-coordinate perturbation scale ( $\sigma_k = \sqrt{K}\sigma$ ); standard ES with the same increased scale performed worse.

Multi-Hop QA

In question-answering tasks, CoPES outperformed standard ES on all benchmark-metric combinations and remained competitive with full-parameter GRPO, demonstrating the method's applicability beyond mathematical reasoning.

Significance and Claims

The paper claims that CoPES offers a practical solution for agentic LLM post-training under resource constraints. By decomposing the search space and standardizing rewards jointly, CoPES achieves a superior trade-off between memory requirements and training time. It enables the use of full-parameter evolution strategies on consumer-grade or limited GPU hardware (e.g., single 24GB GPUs) where gradient-based full-parameter methods are infeasible due to memory limits, and where standard ES is too slow due to computational costs. The authors emphasize that their approach retains the "forward-only" advantage of ES while mitigating its primary computational disadvantage.