Redesigning the Eterna100 RNA design benchmark for the Vienna 2 folding engine

Technical Summary: Redesigning the Eterna100 RNA Design Benchmark for the Vienna 2 Folding Engine

Problem Statement
The rational design of RNA molecules relies on "inverse folding" algorithms that generate sequences folding into a target secondary structure. These algorithms are typically evaluated against the Eterna100 benchmark, a collection of 100 difficult target structures originally curated by the Eterna citizen science community. However, these puzzles were designed using ViennaRNA 1 (specifically version 1.8.5 with Turner 1999 parameters). As the field has shifted toward ViennaRNA 2 (with updated thermodynamic parameters, e.g., Turner 2004), a critical gap emerged: it was unknown whether the Eterna100 puzzles remained solvable under the newer thermodynamic model, and whether algorithmic performance rankings held true when the folding engine changed. Furthermore, recent claims of "solving" the Eterna100 benchmark were found to rely on inconsistent folding engine settings, potentially inflating performance metrics.

Methodology
The authors employed a multi-stage approach combining crowdsourced human design, algorithmic benchmarking, and rigorous re-evaluation of prior literature:

1. Identification of Unsolvability: The authors hypothesized that thermodynamic differences between Vienna 1 and Vienna 2 would render some Eterna100 puzzles unsolvable. They crowdsourced the solution of all 100 Eterna100-V1 puzzles using the Vienna 2 folding engine via the Eterna platform.
2. Benchmark Redesign (Eterna100-V2): For the 19 puzzles identified as unsolvable in Vienna 2, Eterna participants were challenged to redesign the target structures with minimal modifications (insertions/deletions) to make them solvable under Vienna 2 while preserving the original structural complexity. This resulted in the Eterna100-V2 benchmark.
3. Algorithmic Evaluation: Six state-of-the-art RNA design algorithms were tested on both Eterna100-V1 (using Vienna 1) and Eterna100-V2 (using Vienna 2). The algorithms included:
   • Stochastic/Monte Carlo methods: RNAinverse, NEMO, DesiRNA.
   • Deep learning/Reinforcement learning methods: EternaBrain, LEARNA, SentRNA.
   • Evaluations were conducted using the specific folding engine intended for each benchmark version.
4. Re-scoring of Prior Solutions: The authors re-evaluated sequences published by recent studies claiming to solve the Eterna100-V1 benchmark (specifically DesiRNA and Montparnasse). They re-folded these sequences using the strict, original Vienna 1 executable and parameters (specifically checking the --dangles setting, which defaults to 2 in Vienna 2 but was 1 in Vienna 1).

Key Results

• Thermodynamic Sensitivity: 19 of the original 100 Eterna100 puzzles were found to be unsolvable in Vienna 2. The unsolvability was attributed to specific thermodynamic shifts, including increased free energy penalties for large internal loops and changes in the stability of multi-helix junctions and isolated base pairs.
• Successful Redesign: Eterna participants successfully redesigned the 19 problematic puzzles with minimal structural alterations (e.g., adding 3 base pairs to "Shooting Star" or deleting a single base from "Teslagon"). The resulting Eterna100-V2 benchmark is solvable by humans in Vienna 2 and remains largely solvable in Vienna 1 (99/100 puzzles), ensuring comparability between versions.
• Algorithm Performance Shifts:
  • Rankings remained largely consistent, but absolute performance varied significantly based on the folding engine.
  • Algorithms heavily dependent on Vienna 1-specific heuristics or training data (e.g., EternaBrain, NEMO, SentRNA) showed decreased performance when evaluated on Eterna100-V2 with Vienna 2.
  • DesiRNA showed improved performance on Eterna100-V2, becoming the top performer in that configuration, though it still failed to solve all 100 puzzles.
  • No algorithm solved all 100 puzzles in either benchmark configuration. Even if solutions are aggregated across the 6 algorithms, 7 problems remain unsolved in Eterna100-V2 with Vienna 2.
• Correction of Prior Claims: The re-scoring revealed that previously published "solutions" to Eterna100-V1 were incorrect.
  • When re-scored against the Vienna 1 executable, the published DesiRNA designs fold correctly for only 69 of 100 targets (down from the claimed 100).
  • Montparnasse's solutions solved only 27/100.
  • The discrepancy arose because these studies used Vienna 2 executables with Turner 1999 parameters but failed to set the --dangles flag to 1 (the Vienna 1 default), leading to different minimum free energy (MFE) structures.

Significance and Claims
The paper establishes that the difficulty of RNA design is intrinsically linked to the thermodynamic model used. The authors claim that:

1. The Eterna100 benchmark requires version-specific definitions: Both the original Eterna100-V1 and the modernized Eterna100-V2 remain unsolved by current algorithms when evaluated with their intended folding engine settings.
2. Algorithmic dependence on thermodynamic models: Performance of design algorithms is sensitive to the folding engine. Algorithms relying on heuristics derived from specific thermodynamic models (Vienna 1) do not generalize well to updated models (Vienna 2), suggesting a need for algorithms that are either agnostic to thermodynamic models or rapidly retrainable.
3. Standardization is critical: The study highlights that inconsistent settings (specifically the --dangles parameter) in previous literature have led to overestimates of algorithmic performance. The authors provide a standardized repository of benchmarks, solutions, and validator scripts to ensure future evaluations are reproducible and accurate.

The work concludes that while human players can adapt to thermodynamic changes, current automated methods struggle to do so, and the field must move toward benchmarks and algorithms that account for the evolving landscape of RNA thermodynamic prediction.