🤖 AI

ESSAM: A Novel Competitive Evolution Strategies Approach to Reinforcement Learning for Memory Efficient LLMs Fine-Tuning

The paper introduces ESSAM, a memory-efficient full parameter fine-tuning framework that combines Evolution Strategies with Sharpness-Aware Maximization to achieve competitive mathematical reasoning performance on LLMs while significantly reducing GPU memory usage compared to traditional reinforcement learning methods like PPO and GRPO.

Original authors: Zhishen Sun, Sizhe Dang, Guang Dai, Haishan Ye

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhishen Sun, Sizhe Dang, Guang Dai, Haishan Ye

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Heavy Backpack"

Imagine you have a brilliant student (a Large Language Model or LLM) who is good at math but needs extra tutoring to get really good at solving tricky word problems.

Usually, the best way to tutor this student is using a method called Reinforcement Learning (RL). Think of RL as a very strict, high-tech coach who watches every move the student makes, calculates exactly how to improve, and gives immediate feedback. It works great, but it has a massive downside: it requires a gigantic backpack of computer memory (GPU memory) to carry all the calculations.

For a standard math problem, this "backpack" might weigh as much as a small car (hundreds of gigabytes). Most people and small labs don't have a car-sized computer to carry this load, so they can't use this powerful coaching method.

The Old Solution: The "Guess and Check" Team

To avoid the heavy backpack, researchers tried a different method called Evolution Strategies (ES).

How it works: Instead of one strict coach, imagine a team of 100 explorers. They all start at the same spot, but each one takes a slightly different, random path (a "perturbation"). They all try to solve the math problem. The ones who get the right answer get a "reward." The team then looks at the winners and says, "Okay, let's all move a little bit in the direction the winners went."
The Benefit: This method is incredibly light. It doesn't need the heavy backpack because it doesn't do complex backward calculations. It just needs enough memory to run the model once (like a standard user).
The Flaw: While it's light, the explorers often get stuck in "shallow valleys." They find a solution that works okay, but it's not very robust. If you give them a slightly different math problem, they might get confused. They lack generalization (the ability to apply what they learned to new situations).

The New Solution: ESSAM (The "Smart Hiker")

The authors propose a new method called ESSAM (Evolution Strategies with Sharpness-Aware Maximization). They wanted to keep the light backpack of the "Guess and Check" team but add the "smartness" of the strict coach to improve the results.

Here is how ESSAM works, using a hiking analogy:

The "Sharpness" Problem: Imagine the explorers are hiking toward a peak (the best answer). Sometimes, they find a peak that looks high but is actually a sharp, jagged rock. If the wind blows (a new math problem), they fall off easily. This is a "sharp minimum."
The ESSAM Trick: Before the team commits to moving in a specific direction, ESSAM sends a scout to check the terrain around that direction.
- The scout asks: "If we move this way, is the ground flat and stable, or is it a jagged cliff?"
- If the ground is jagged (sharp), the team reverses course slightly. They look for a direction where the ground is flat and wide (a "flat minimum").
- A flat peak is safer. Even if the wind blows or the problem changes slightly, the team stays on top.

In technical terms: ESSAM takes the standard "Guess and Check" method and adds a "neighborhood probe." It temporarily moves the model parameters to a nearby spot, checks if the reward is stable there, and uses that information to guide the main update. This forces the model to find solutions that are robust, not just lucky.

The Results: Light as a Feather, Strong as an Ox

The paper tested this on a popular math dataset called GSM8K (a collection of grade-school math word problems).

Performance: ESSAM performed just as well as the heavy, memory-hungry Reinforcement Learning methods (like PPO and GRPO). In fact, on some models, it was even better.
Memory: This is the big win. While the heavy methods needed hundreds of gigabytes of memory, ESSAM used the same tiny amount of memory as the basic "Guess and Check" method.
- The Analogy: If the old methods needed a semi-truck to carry their equipment, ESSAM fits in a bicycle.
- The Stats: It used 18 times less memory than the standard heavy method (PPO) and 10 times less than the other popular method (GRPO).
Generalization: When tested on math problems they hadn't seen before (different datasets), the ESSAM-trained models were much better at solving them than the standard "Guess and Check" models. They had learned the concept of math, not just memorized the specific answers.

The "Turbo" Version (ESSAM-F)

The authors also created a faster version called ESSAM-F.

How it works: It uses a smaller team of scouts for the "checking" phase.
The Result: It runs twice as fast as the original ESSAM while using the same tiny amount of memory and keeping almost the same high accuracy.

Summary

The paper introduces ESSAM, a new way to teach AI math. It combines the low cost of "Guess and Check" methods with a smart stability check (Sharpness-Aware Maximization).

Before: You had to choose between "High Performance but Heavy/Expensive" (RL) or "Light/Cheap but Weak" (Standard ES).
Now: With ESSAM, you get the High Performance of the heavy methods with the Light/Cheap footprint of the simple methods. It allows smaller labs and open-source communities to train powerful math-reasoning AI without needing supercomputers.

Technical Summary: ESSAM – Evolution Strategies with Sharpness-Aware Maximization

Problem Statement
Reinforcement Learning (RL), particularly algorithms like Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), has become a cornerstone for enhancing the mathematical reasoning capabilities of Large Language Models (LLMs). However, these methods impose prohibitive GPU memory requirements due to their reliance on backpropagation, value networks, and policy updates. For instance, fine-tuning an 8B parameter model with PPO can require over 300 GiB of GPU memory, rendering it inaccessible for resource-constrained open-source communities. While Evolution Strategies (ES) offer a memory-efficient, zero-order alternative by relying solely on forward generation and reward evaluation, they often suffer from poor generalization. Standard ES tends to drive models toward "sharp minima" in the parameter space, leading to suboptimal performance on mainstream reasoning tasks like GSM8K compared to RL methods.

Methodology: ESSAM
The authors propose ESSAM (Evolution Strategies with Sharpness-Aware Maximization), a full-parameter fine-tuning framework that integrates the zero-order search capabilities of ES with the generalization-enhancing principles of Sharpness-Aware Maximization (SAM).

Core Mechanism: Unlike standard ES, which updates parameters directly based on reward-weighted noise directions, ESSAM introduces a two-stage update process:
- Stage 1 (Neighborhood Probing): The algorithm computes a reward-weighted aggregation of noise directions to move the current parameters $\theta_t$ in the opposite direction of the reward gradient (toward a "sharp" region). This creates a perturbed neighborhood point $\theta^{SAM}_t$ .
- Stage 2 (Reverse Update): The algorithm evaluates the reward at this perturbed point $\theta^{SAM}_t$ and uses the resulting reward signals to update the original parameters $\theta_t$ . By effectively "looking ahead" to the sharp region and updating based on the perturbed landscape, the method guides the optimization toward flatter minima, which are associated with better generalization and robustness.
Memory Efficiency: ESSAM retains the zero-order nature of ES, requiring no backpropagation or storage of gradients/activations. It utilizes memory-saving techniques (such as in-place updates and seed replay) to maintain GPU memory usage at "inference-level" scales, significantly lower than RL methods.
Accelerated Variant (ESSAM-F): To improve training speed, the authors propose ESSAM-F. This variant employs a smaller population size for the initial SAM neighborhood probing stage and utilizes Frobenius norm normalization for the perturbation direction. This reduces computational overhead while maintaining the same low memory footprint.
Training Protocol: The authors emphasize a standardized training procedure, including standard train/evaluation splits, data shuffling, and multi-step updates with small mini-batches, addressing previous inconsistencies in ES literature.

Key Contributions

Novel Framework: Introduction of ESSAM, the first zero-order full-parameter fine-tuning method to successfully integrate SAM mechanisms for LLMs, guiding optimization toward flat minima without gradient computation.
Standardization: Extension of zero-order fine-tuning to the mainstream GSM8K mathematical reasoning task using a rigorous, standard training pipeline (shuffling, mini-batches) to ensure stability and reproducibility.
Accelerated Variant: Development of ESSAM-F, which achieves nearly a twofold speedup over ESSAM while preserving memory efficiency and competitive accuracy.
Resource Efficiency: Demonstration that full-parameter fine-tuning can be achieved with inference-level GPU memory, drastically reducing the barrier to entry for LLM reasoning research.

Experimental Results
Experiments were conducted on the GSM8K dataset across various model families (Qwen2.5 and LLaMA3) ranging from 0.5B to 8B parameters.

Performance on GSM8K: ESSAM achieved an average accuracy of 78.27% across all models. This significantly outperforms standard ES (75.97%) and surpasses PPO (77.72%). It is comparable to GRPO (78.34%) and exceeds it on specific models.
Generalization: On six out-of-distribution benchmarks (including SVAMP, MATH, and AMC), models fine-tuned with ESSAM achieved the best average performance on 5 out of 6 datasets, demonstrating superior transferability compared to ES, PPO, and GRPO.
Memory Usage: ESSAM reduces GPU memory usage by approximately 18× compared to PPO and 10× compared to GRPO. For an 8B model, ESSAM uses roughly 21.45 GiB, whereas PPO requires ~314.44 GiB.
Efficiency: The accelerated variant, ESSAM-F, achieved an average accuracy of 78.02% (outperforming PPO) with nearly 2× speedup in training time compared to the standard ESSAM, while maintaining identical memory usage.

Significance and Claims
The paper claims that ESSAM provides a practical, resource-efficient alternative to RL for fine-tuning LLMs on mathematical reasoning tasks. By combining the memory efficiency of Evolution Strategies with the generalization benefits of Sharpness-Aware Maximization, ESSAM bridges the performance gap between zero-order methods and gradient-based RL. The authors assert that their approach makes high-performance LLM reasoning fine-tuning feasible in resource-constrained settings, offering a viable path for the open-source community to develop robust reasoning models without access to massive computational clusters. The work highlights that memory efficiency does not necessitate a trade-off in model performance or generalization capability.