Library Reachability in LSR-Synth: How Anti-Memorization Design Changes the Measurement of Symbolic Discovery

Technical Summary: Library Reachability in LSR-Synth

1. Problem Statement

Existing benchmarks for scientific equation discovery, such as LSR-Synth, are designed to mitigate "formula memorization" by constructing target equations that combine established scientific mechanisms with novel synthetic terms. This ensures the complete target expression is unlikely to exist verbatim in a language model's (LLM) training corpus.

However, a critical gap remains in evaluating whether an LLM's success stems from genuine scientific priors or merely from the ability to recombine familiar mathematical building blocks. A target equation may be novel as a whole, yet its constituent operators (e.g., powers, exponentials, trigonometric functions) and structural patterns may be fully contained within a fixed, semantics-free search space. If a conventional search algorithm using a documented vocabulary can recover the target without accessing variable meanings or domain context, the LLM's contribution is indistinguishable from standard operator search. The paper addresses the question: Do current LSR-Synth tasks effectively distinguish between LLM-supplied scientific priors and conventional search within a fixed vocabulary?

2. Methodology

The authors introduce a diagnostic framework centered on the concept of library reachability: the property that a target can be recovered by a semantics-free search under a prescribed vocabulary and computational budget.

Experimental Setup:

• Dataset: 129 tasks from the LSR-Synth benchmark (covering population growth, physical oscillations, chemical reactions, and material relationships).
• Candidate Sources:
  • BANK: A fixed candidate library constructed without access to task semantics (variable names or domain descriptions). Two variants are used: BANK-LaSR (based on the PySR operator set) and BANK-SRBench-PySR.
  • LLM: Candidates proposed by a language model under two conditions:
    • L1: Full context (variable names, domain description, data ranges).
    • L2: Semantic blinding (anonymous variables, no domain description).
  • UNION: The union of the BANK and LLM candidate sets.
• Search Procedure: A paired greedy search is executed on each candidate pool (BANK, LLM, UNION) using identical training subsamples, fitting procedures (nonlinear least squares with joint parameter re-estimation), and stopping criteria. This isolates the candidate source as the sole variable.
• Interventions:
  • Semantic Blinding: Comparing L1 vs. L2 to assess the value of domain context.
  • Library Weakening: Replacing the full library with a BANK-Poly (polynomial-only) library to create explicit coverage gaps.
  • Matched Knockouts: Removing target-required operator families vs. removing an equal number of irrelevant operators to test coverage sensitivity.

Evaluation Metrics:

• In-Distribution Accuracy: Relative error thresholds ($\tau=0.01, 0.001$).
• Strict Joint Recovery: Success on both in-distribution (ID) and out-of-distribution (OOD) data with $NMSE < 10^{-8}$.
• Symbolic Accuracy (SA): A strict unanimity rule (10/10 "Yes" votes from GPT-OSS-120B) determining if the predicted skeleton is structurally equivalent to the ground truth.

3. Key Results

• High Coverage by Fixed Libraries: Under the full LaSR vocabulary, the fixed BANK-LaSR library achieves high performance (77.71% Acc0.01, 19.77% SA), outperforming many existing LLM-based methods reported in literature.
• Marginal Contribution of LLMs: When the fixed library is complete, adding LLM candidates (UNION) yields negligible or non-monotonic improvements over BANK alone. The LLM rarely discovers solutions that the fixed library cannot find.
• Impact of Library Weakening: When the fixed library is restricted to polynomials (BANK-Poly), performance drops significantly. In this setting, adding LLM candidates (UNION-Poly) produces substantial gains across all metrics, demonstrating that LLMs are effective at filling genuine structural gaps (e.g., exponential or trigonometric terms) missing from the restricted vocabulary.
• Semantic Context: LLM performance is higher with full context (L1) than with blinded context (L2), but this advantage only translates to end-to-end gains when the fixed library has coverage gaps.
• Decomposition of Success: Analysis of paired outcomes reveals extensive overlap between BANK and LLM successes. Under strict criteria (SA, OOD), the number of tasks solved only by the LLM is small compared to those solved only by BANK.

4. Key Contributions

• Concept of Library Reachability: The paper defines and operationalizes "library reachability" as a diagnostic tool to separate the novelty of a complete equation from the novelty of its search space.
• Diagnostic Benchmarking Protocol: It proposes a rigorous evaluation protocol using paired BANK, LLM, and UNION runs with fixed seeds and budgets to isolate the contribution of LLM priors from candidate coverage.
• Empirical Evidence on LSR-Synth: The study demonstrates that the current aggregate scores of LSR-Synth are insufficient to claim that LLMs provide indispensable scientific priors beyond a fixed search space, as most tasks are reachable by semantics-free libraries.

5. Significance and Claims

The paper does not claim that LLMs are useless for symbolic regression or that anti-memorization benchmarks are invalid. Instead, it offers a limited but precise conclusion:

"The current aggregate benchmark score is insufficient, on its own, to demonstrate that a language model contributes scientific knowledge beyond what is already encoded in a fixed library."

The authors argue that while LSR-Synth successfully prevents the recall of complete formulas, it does not currently control for the reachability of the search space. Consequently, high accuracy on these benchmarks may reflect the model's ability to recombine familiar operators rather than the application of novel scientific priors.

Implications for Future Work:

• Benchmarks should stress-test generated tasks against documented fixed libraries before release.
• Evaluations should report BANK, LLM, and UNION results jointly to distinguish candidate recall from effective search-boundary expansion.
• Claims of scientific discovery should be supported by evidence that the LLM expands the search boundary beyond what a fixed, semantics-free vocabulary can achieve under the same computational budget.