💻 computer science

Improving CMA-ES Convergence Speed, Efficiency, and Reliability in Noisy Robot Optimization Problems

This paper introduces Adaptive Sampling CMA-ES (AS-CMA), a novel algorithm that dynamically assigns evaluation time to candidates based on predicted sorting difficulty, demonstrating superior convergence speed, efficiency, and reliability over standard CMA-ES and Bayesian optimization in noisy robot optimization tasks.

Original authors: Russell M. Martin, Steven H. Collins

Published 2026-01-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Russell M. Martin, Steven H. Collins

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find the absolute best setting for a new, high-tech walking suit (an exoskeleton) that helps people walk with less effort. The problem is that the human body is messy and unpredictable. Every time you test a new setting, you have to ask a person to walk on a treadmill for a while to see how much energy they use.

Here is the catch:

If you test a setting for too short a time (say, 30 seconds), the data is noisy and unreliable. It's like trying to guess the average temperature of a day by checking the thermometer for only one second. You might get a lucky guess, or you might get a wildly wrong number.
If you test a setting for too long a time (say, 10 minutes), the data is very accurate, but you waste a huge amount of time. If you do this for every single setting you try, the whole optimization process takes forever.

Traditionally, researchers picked one "middle ground" time (like 2 minutes) and stuck with it for every single test. This paper introduces a smarter way to do this called AS-CMA (Adaptive Sampling CMA-ES).

The Problem: The "One-Size-Fits-All" Mistake

Think of the optimization process like a detective trying to find a suspect in a crowded room.

Static Sampling (The Old Way): The detective decides to spend exactly 5 minutes interviewing every single person in the room, regardless of how suspicious they look.
- If the person is clearly innocent, 5 minutes is a waste of time.
- If the person looks very suspicious but is hard to distinguish from a neighbor, 5 minutes might not be enough to be sure.
- This approach is inefficient. It's either too slow or not accurate enough.

The Solution: AS-CMA (The Smart Detective)

The authors created a new method, AS-CMA, which acts like a detective who adjusts their interview time based on how hard it is to tell people apart.

Easy Decisions Get Short Interviews: If the detective sees two people who look very different (one is wearing a clown suit, the other a business suit), they don't need to spend much time to know who is who. AS-CMA says, "This candidate is clearly better or worse than its neighbor; let's test it quickly."
Hard Decisions Get Long Interviews: If the detective sees two people who look almost identical, they need to spend more time to be sure who is the suspect. AS-CMA says, "These two candidates are very close in performance; let's test this one longer to get a precise answer."

How It Works in Practice

The algorithm looks at the "landscape" of possibilities.

Early in the search: The candidates are very different from each other. The algorithm is like a sprinter, testing many options quickly with short durations to narrow down the field.
Late in the search: The candidates are all very similar (they are all "good" options, but which is the best?). The algorithm slows down, acting like a marathon runner, spending more time on the few remaining top contenders to ensure it picks the absolute winner.

What the Paper Found

The researchers tested this "Smart Detective" against the old "One-Size-Fits-All" method and other advanced methods (like Bayesian Optimization) in four different simulated worlds:

Real-world Exoskeleton Data: A simulation based on actual human walking data.
Complex Math Problems: Simulations with many bumps, valleys, and tricky interactions between variables.

The Results:

Speed: AS-CMA found the best solution 24% to 65% faster than the best fixed-time method.
Reliability: It succeeded in 98% of all attempts without needing any manual tweaking. The old methods often failed or got stuck in "good enough" solutions that weren't actually the best.
Efficiency: It saved a massive amount of "energy cost" (in the simulation, this meant the total time spent walking) because it didn't waste time on easy decisions.

Comparison to Other Methods:

Vs. Fixed Time: AS-CMA was almost always better.
Vs. Bayesian Optimization: In simple, smooth problems, Bayesian optimization was faster. But in complex, bumpy, or noisy problems (which are common in real robots), Bayesian optimization often got confused or stuck, while AS-CMA kept moving steadily toward the solution.

The Real-World Test

The team didn't just stop at computer simulations. They actually used AS-CMA to optimize an ankle exoskeleton on a real human subject.

The algorithm quickly figured out the right direction to improve.
It successfully reduced the person's energy cost by 42% (very close to the expected 39%).
It did this by automatically switching between short tests (when the options were obvious) and long tests (when it needed to be precise), without the researchers having to tell it how long to wait.

The Bottom Line

The paper argues that in noisy, real-world experiments (like robotics or human testing), you shouldn't treat every test the same. You should be flexible. AS-CMA is a tool that automatically decides "how long to listen" based on how confusing the current situation is. It makes the search for the best robot settings faster, more reliable, and less wasteful of time and energy.

Technical Summary: Improving CMA-ES Convergence in Noisy Robot Optimization

Problem Statement
Experimental optimization of robotic systems, particularly in human-in-the-loop contexts like exoskeleton assistance, faces a critical speed-accuracy tradeoff. Evaluating candidate control policies often requires seconds to minutes to allow human or system adaptation to reach a steady state. Shorter evaluation times enable faster iteration but introduce significant measurement noise, potentially leading to incorrect sorting of candidates and suboptimal convergence. Conversely, longer evaluation times reduce noise but slow the optimization process and increase the total cost (e.g., metabolic energy expenditure) of the experiment. Traditional approaches often rely on a single, static sampling time for all candidates, which fails to account for the varying difficulty of sorting candidates as the optimizer progresses through different regions of the cost landscape.

Methodology: Adaptive Sampling CMA-ES (AS-CMA)
The authors propose AS-CMA, a modification to the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) that dynamically allocates sampling time to candidates based on predicted sorting difficulty. The core logic relies on the observation that CMA-ES selection depends on the rank ordering of candidates rather than their absolute cost values.

Mechanism: AS-CMA estimates the difficulty of distinguishing a candidate from its nearest neighbor in the current generation. It uses the Euclidean distance between candidate parameters ( $d_{nearest}$ ) as a proxy for cost difference, scaled by an estimate of the local landscape slope ( $k_{avg}$ ).
Noise Modeling: The algorithm assumes a noise model $E(t)$ where measurement error decreases as sampling time $t$ increases. It calculates a target noise level ( $\epsilon_i$ ) required to achieve a desired signal-to-noise ratio ( $\beta$ ) for the estimated cost difference.
Adaptive Allocation: By inverting the noise model, AS-CMA determines the minimum necessary sampling time $t_i$ $t_{i}$ for each candidate to ensure reliable sorting.
- Candidates with large parameter distances from neighbors (easy to sort) receive shorter sampling times.
- Candidates with small parameter distances (hard to sort, often occurring near the optimum) receive longer sampling times.
Constraints: The method avoids "doubling back" (re-evaluating previously measured candidates) and does not rely on historical data from distant generations, making it suitable for time-varying contexts where the cost landscape may evolve (e.g., human learning).

Experimental Framework
The authors evaluated AS-CMA against standard CMA-ES with various static sampling times, Bayesian Optimization (BO), and Kullback-Leibler Knowledge Gradient CMA-ES (KL-KG CMA-ES).

Simulations: Four cost landscapes were tested: an empirical 4D ankle exoskeleton metabolic cost landscape, a 4D Rosenbrock function (variable interaction), a 4D Levy function (local minima), and a 20D Sphere function (high dimensionality). Noise was modeled as multiplicative and dependent on sampling time, derived from empirical metabolic data.
Metrics: Performance was measured by convergence reliability (percentage of runs reaching a threshold), time to convergence, and cumulative cost to convergence.
Laboratory Pilot: A single-subject experiment optimized ankle exoskeleton assistance to minimize metabolic cost, using AS-CMA to select condition durations between 0.5 and 5.5 minutes.

Key Results

Convergence Speed and Efficiency: Across all four simulated landscapes, AS-CMA converged 24–65% faster and with 29–76% lower total cost than the best-performing static sampling time for standard CMA-ES. It achieved fine convergence (within 5% of the true minimum) in 98% of all runs without parameter tuning.
Sorting Accuracy: AS-CMA maintained high Spearman's rank correlation ( $\rho$ ) between noisy and true costs throughout the optimization run, outperforming static strategies which either started with low accuracy (short times) or wasted time on unnecessary precision (long times).
Comparison to Bayesian Optimization: In simpler, unimodal landscapes (Ankle, Sphere), BO with optimal static sampling was more efficient than AS-CMA. However, in complex landscapes with local minima and parameter interactions (Rosenbrock, Levy), BO struggled with reliability, while AS-CMA remained robust and efficient.
Comparison to KL-KG CMA-ES: KL-KG CMA-ES showed marginal benefits over static CMA-ES in some cases but was generally outperformed by AS-CMA. In complex landscapes, KL-KG required significantly more time and cost than AS-CMA to achieve similar convergence.
Laboratory Validation: In the exoskeleton experiment, AS-CMA rapidly identified the correct direction of the optimum and achieved a 42% reduction in metabolic cost (close to the expected 39%), with the algorithm dynamically varying sample times across the allowable range.

Significance and Claims
The paper claims that AS-CMA offers a practical, low-overhead solution for noisy experimental optimization. Its primary significance lies in:

Removing a Tuning Parameter: It eliminates the need for researchers to manually select a static sampling time, a process that often requires multiple preliminary runs.
Robustness in Complexity: It provides a more reliable alternative to Bayesian Optimization in complex, high-dimensional, or non-convex search spaces common in real-world robotics.
Efficiency in High-Cost Environments: By minimizing unnecessary sampling time for easy-to-sort candidates and maximizing it only when necessary, AS-CMA reduces the total experimental burden (time and energy) without sacrificing solution quality.

The authors conclude that AS-CMA is particularly well-suited for domains where switching between candidates incurs high overhead (e.g., human adaptation, system reconfiguration), as it avoids re-evaluating candidates and adapts sampling effort to the immediate needs of the optimization process.