🤖 machine learning

Are you sure? A Comprehensive and Comprehensible Survey of Uncertainty Quantification in Symbolic Regression

This survey addresses the critical gap in uncertainty quantification (UQ) for symbolic regression by introducing essential concepts, reviewing existing literature across frequentist, Bayesian, and model selection approaches, and highlighting the need for further research to enable reliable real-world decision-making.

Original authors: Julia Reuter, Fabricio Olivetti de Franca

Published 2026-06-08

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Julia Reuter, Fabricio Olivetti de Franca

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery using a set of clues (data). Your goal is to find the perfect "rule" or "formula" that explains how those clues fit together. This is what Symbolic Regression (SR) does: it searches through a giant library of mathematical functions to find the one that best describes your data.

However, in the real world, clues are rarely perfect. They might be smudged, measured with a shaky hand, or just inherently random. This is where Uncertainty Quantification (UQ) comes in. Think of UQ as the detective's "confidence meter." Instead of just saying, "The rule is $y = 2x$ ," a good detective with UQ says, "The rule is likely $y = 2x$ , but I'm only 95% sure, and the real answer could be anywhere between $1.8x$ and $2.2x$ ."

This paper is a comprehensive survey (a big review) of how researchers are currently trying to build these "confidence meters" for Symbolic Regression. The authors argue that while SR is getting better at finding rules, it's still terrible at telling us how much we should trust those rules.

Here is a breakdown of the paper's main ideas using simple analogies:

1. The Two Types of "Doubt"

The paper explains that there are two main reasons a detective might be unsure:

The "Messy Clues" (Aleatoric Uncertainty): Imagine trying to measure the height of a bouncing ball. No matter how many times you measure it, it will bounce differently every time. This is noise you can't get rid of, even with more data. UQ helps you realize, "Hey, the ball is just chaotic; I can't predict the exact bounce, only the average."
The "Missing Knowledge" (Epistemic Uncertainty): Imagine you are trying to guess a secret code, but you only have three clues. You don't know the rule because you haven't seen enough examples yet. This is a lack of knowledge. If you get more clues, you can reduce this doubt. UQ helps you realize, "I'm guessing here because I haven't seen enough data."

2. The Three Ways to Measure Confidence

The paper organizes the current methods into three "schools of thought" on how to calculate that confidence meter:

A. The Frequentist Approach (The "Repeat the Experiment" Club)

The Analogy: Imagine you bake a cake and it tastes perfect. To know if it's really perfect or just lucky, you bake it 100 times. If it tastes great 95 times, you are confident.
How it works: These methods assume there is one "true" rule out there, but our data is noisy. They use math (like Fisher Information) to look at the shape of the "hill" where the best answer lies.
- Sharp Peak: If the hill is a sharp spike, you know exactly where the top is (high confidence).
- Flat Plateau: If the hill is a flat table, you could be standing anywhere on it and still be "correct" (low confidence).
Tools: They use Confidence Intervals (a range for the rule's numbers) and Prediction Intervals (a range for future guesses). They also use Conformal Prediction, which is like a safety net that guarantees your guess will be right a certain percentage of the time, without needing to assume the data follows a specific pattern.

B. The Bayesian Approach (The "Belief Update" Club)

The Analogy: Imagine you start with a hunch (a prior belief) about what the rule might be. As you get new clues, you update your hunch. You don't just look for one best rule; you keep a whole "cloud" of possible rules in your head, each with a probability of being true.
How it works: Instead of finding a single answer, these methods try to map out the entire "cloud" of possibilities.
- Credible Intervals: This is the Bayesian version of a confidence interval. It says, "There is a 90% chance the true rule is inside this box."
- The Challenge: Calculating this "cloud" is hard. The paper discusses methods like MCMC (Markov Chain Monte Carlo), which is like a drunkard's walk that eventually maps out the whole territory, and SMC (Sequential Monte Carlo), which uses a swarm of particles to explore the territory more efficiently.
Innovation: Some researchers are trying to use these methods not just to find the numbers in the rule, but to find the shape of the rule itself (e.g., is it a line? a curve? a tree?).

C. Model Selection (The "Choosing the Best Detective" Club)

The Analogy: You have 10 different theories about the mystery. Some are simple (one suspect), some are complex (a conspiracy involving 50 people). How do you pick the right one without falling for a conspiracy theory that fits the data too perfectly (overfitting)?
How it works: This section looks at methods like MDL (Minimum Description Length). Think of it as a "compression" game. The best rule is the one that explains the data using the fewest words (bits). If a rule is too complex, it's like trying to compress a book into a single sentence—it loses meaning. These methods penalize overly complex rules to ensure the detective doesn't get fooled by random noise.

3. What the Survey Found

The authors looked at many research papers and found a few key things:

It's a New Frontier: Only a handful of papers (mostly from the last few years) are actually doing this. Most Symbolic Regression research focuses only on finding the "best" answer, ignoring how unsure we should be about it.
Bayesian is Popular: Because Symbolic Regression is so flexible, it's hard to use the "Frequentist" math (which assumes simple, stable rules). So, most new research is using Bayesian methods to handle the complexity.
The "Structure" Problem: It's easy to calculate uncertainty for the numbers in a formula (like the "2" in $2x$ ). It is much, much harder to calculate uncertainty for the shape of the formula (is it $x^2$ or $x^3$ ?). The paper highlights that while some researchers are trying to solve this, it's still a major challenge.
The "Noise" Problem: Sometimes the data itself is messy (the x-axis has errors, not just the y-axis). Most standard methods ignore this, but a few advanced papers are starting to fix that.

The Bottom Line

The paper concludes that Symbolic Regression is currently "blind" to its own uncertainty. It's like a GPS that tells you to turn left but doesn't tell you if the road is actually closed or if the map is outdated.

The authors are calling on the scientific community to stop just looking for the "best" mathematical rule and start building tools that tell us how much we can trust that rule. They believe that without this "confidence meter," we can't safely use these powerful AI tools for real-world decisions where mistakes are costly.

Technical Summary: A Comprehensive and Comprehensible Survey of Uncertainty Quantification in Symbolic Regression

Problem Statement

Symbolic Regression (SR) is a search algorithm class designed to discover mathematical functions that accurately capture underlying relationships in datasets. Despite recent advancements, the adoption of SR in real-world decision-making processes is limited by a lack of support for Uncertainty Quantification (UQ). In regression analysis, UQ is critical for assessing model reliability, preventing overfitting by accounting for data noise (aleatoric uncertainty), and providing insights for decision-making.

SR faces two primary sources of uncertainty:

Aleatoric Uncertainty: Intrinsic randomness or measurement imprecision in the data generation process. This is irreducible but can be quantified to estimate the noise-induced lower bound of error.
Epistemic Uncertainty: A lack of knowledge regarding the true model. This includes:
- Model Uncertainty: The risk that the true model $h^*$ is not present in the hypothesis space $\mathcal{H}$ .
- Approximation Uncertainty: Uncertainty about a specific hypothesis $h$ being the best within $\mathcal{H}$ , often decomposed into parameter uncertainty (uncertainty in adjustable parameters $\theta$ ) and structural uncertainty.

While traditional SR literature focuses on efficient traversal of the multimodal search space, uncertainties are often not explicitly addressed. This survey aims to introduce essential UQ concepts and review current literature to raise awareness and promote advancements in reliable UQ methods for SR.

Methodology and Theoretical Foundations

The paper structures the review around three research directions: Frequentist, Bayesian, and Model Selection. It establishes a theoretical foundation using likelihood functions and Fisher information.

1. Statistical Foundations

Likelihood and Fisher Information: The survey defines the likelihood function $L_N(\theta)$ and its log-transformation. It utilizes the Hessian matrix of the log-likelihood to estimate parameter certainty. The negative Hessian corresponds to the observed Fisher Information, where small information values indicate flat likelihood surfaces (high uncertainty) and sharp peaks indicate precise estimation.
Frequentist Approaches: These treat parameters as fixed but unknown. UQ is achieved via Confidence Intervals (CI) for parameters and Prediction Intervals (PI) for outputs. The paper details the calculation of standard errors using the inverse Hessian and discusses Conformal Prediction (CP) as a distribution-free method that guarantees coverage for exchangeable data, contrasting it with traditional PIs which rely on Gaussian assumptions.
Bayesian Methods: These treat parameters as random variables with distributions. UQ is achieved via Credible Intervals (CrI) derived from the posterior distribution $p(h|D) \propto p(h)p(D|h)$ . The paper discusses Markov Chain Monte Carlo (MCMC) and Sequential Monte Carlo (SMC) as methods to approximate intractable posteriors. It also covers Bayesian Model Averaging (BMA) to handle approximation uncertainty.
Model Selection: The survey reviews principles like the Bayes Factor, Fractional Bayes Factor (FBF) (to handle improper priors), and the Minimum Description Length (MDL) principle, which balances model complexity and data fit by minimizing the codelength.

2. Survey of SR-Specific UQ Literature

The authors surveyed 18 relevant articles (11 Bayesian, 4 non-Bayesian, 3 model selection) from peer-reviewed journals, conferences, and preprints.

Bayesian Approaches in SR

Sequential Monte Carlo (SMC): A series of works (Bomarito et al., Leser et al.) integrates SMC into Genetic Programming (GP) for SR. They propose a probabilistic reformulation of local optimization and selection. The method estimates the Normalized Marginal Log Likelihood (NMLL) to compare models. It uses SMC to approximate integrals over parameters, handling multimodal posteriors better than Laplace approximations. Recent work extends this to marginalize over discrete model structures directly, treating the population of symbolic models as a weighted posterior.
Structural Priors: Guimera et al. and Bartlett et al. introduce informative priors over model structures. Guimera et al. use operator frequencies from a corpus (Wikipedia) to define priors, while Bartlett et al. use language models (n-grams) to model node probabilities conditional on ancestors, aiming to capture context in expression trees.
X-axis Uncertainty: Bartlett and Desmond formulate likelihoods that incorporate measurement errors in independent variables ( $x$ ), marginalizing out the true values to derive a modified likelihood function.
Ensemble Methods: Agapitos et al. propose Bayesian Model Averaging over a population of diverse models to account for approximation uncertainty.
Linear Bayesian Forests: Roy et al. propose HierBOSSS (a forest of linearly combined symbolic trees) and VaSST (using variational inference with soft symbolic trees). These approaches exploit conjugacy for linear coefficients but rely on mean-field approximations for tree structures. The authors note limitations regarding the linearity assumption and potential inconsistencies in experimental replication.
Post-hoc MCMC: Zhao and Zhao apply MCMC to the final model found by SR to calculate credible intervals for coefficients and predictions.

Non-Bayesian Approaches

Quantile Regression: Hoekstra and Hengst replace the SR fitness function with a pinball loss to estimate conditional quantiles, effectively generating prediction intervals.
Profile Likelihood: de Franca and Kronberger use profile likelihood to calculate asymmetric confidence intervals for parameters and predictions, capturing dependencies between parameters that delta-method approximations miss.
Bootstrapping: Manzi and Vasile apply bootstrapping to generate multiple samples, fit SR models, and visualize parameter distributions via histograms to assess structural and parametric uncertainty.
Terminal Symbols for Noise: Schmidt and Lipson introduce a random terminal symbol to model nonlinear noise relationships directly within the expression, optimizing the range of prediction intervals.
Conformal Prediction in SR: Thuong et al. integrate CP either post-hoc on the best model or by incorporating interval width into the fitness function, finding that post-hoc application yielded better results.

Model Selection

MDL in SR: Iba et al. and Nikolaev apply MDL to polynomial models (STROGANOFF) and exhaustive searches, using codelength to penalize complexity. Bartlett et al. propose a detailed DL metric for enumerated expressions that accounts for functional complexity and parameter precision via discretization.

Key Results and Findings

Recency: Only 3 of the 18 surveyed papers were published before 2020, indicating that UQ in SR is a rapidly emerging field.
Methodological Diversity: While traditional UQ methods (like CIs) exist, their direct application to SR is rare. Most surveyed works (approx. 60%) utilize Bayesian approaches, leveraging sampling methods (MCMC, SMC) to inherently provide UQ.
SMC vs. Laplace: SMC-based methods for SR (Bomarito et al.) demonstrate superior ability to capture multimodal posteriors compared to Laplace approximations, though at a higher computational cost.
Structural Priors: Incorporating structural priors (e.g., based on language models or operator frequencies) helps in model selection and reduces overfitting, though they may restrict the exploration of novel expressions not present in the training corpus.
Limitations of Current Methods:
- Many Bayesian SR methods (e.g., VaSST) rely on mean-field approximations that ignore dependencies between nodes, potentially leading to suboptimal global structures.
- Linear Bayesian forests are limited to models linear in their parameters.
- Some proposed methods (e.g., HierBOSSS, VaSST) face challenges in experimental replication and code consistency.
- Frequentist approaches struggle with SR because flexible SR models can easily overfit, making them unsuitable as unbiased estimators required for certain variance calculations.

Significance and Claims

The paper claims to be the first survey to clearly address the issue of UQ in Symbolic Regression. Its primary significance lies in:

Bridging the Gap: It connects established statistical UQ concepts (frequentist and Bayesian) with the specific challenges of SR, such as the discrete nature of model structures and multimodal search spaces.
Categorization: It organizes the fragmented literature into three coherent research directions: frequentist, Bayesian, and model selection.
Highlighting Under-explored Areas: The authors conclude that UQ in SR remains largely underexplored and its importance is systematically underestimated.
Future Direction: The paper argues that future work must adopt established UQ methods for SR, coupled with careful analysis of data and noise, to develop reliable models for real-world decision-making. It emphasizes that while UQ can come "for free" in sampling-based approaches, current implementations have significant limitations regarding computational cost, structural priors, and the handling of multimodality.