Multi-Modal Learning meets Genetic Programming: Analyzing Alignment in Latent Space Optimization

1. Problem Statement

Symbolic Regression (SR) aims to discover interpretable mathematical expressions from data. Traditionally, this is solved via Genetic Programming (GP) using combinatorial search over symbolic structures. Recent approaches, known as Latent Space Optimization (LSO), transform this discrete search into continuous optimization by mapping symbolic expressions into a latent space using neural encoders.

The specific problem addressed in this paper concerns SNIP (Meidani et al., 2024), a state-of-the-art multi-modal LSO method inspired by CLIP. SNIP uses contrastive pre-training to align two modalities:

1. Symbolic (Genotype): The mathematical expression structure.
2. Numeric (Phenotype): The input-output behavior of the expression.

The core hypothesis of SNIP is that by aligning these modalities in a shared latent space, optimizing in the numeric latent space will implicitly guide the search toward the correct symbolic structure. However, literature on similar models (like CLIP) suggests that contrastive alignment is often coarse-grained, capturing high-level semantics but failing at fine-grained distinctions. The authors investigate whether SNIP's alignment is sufficiently fine-grained to support effective symbolic search and whether the optimization algorithm actually exploits this alignment.

2. Methodology

The authors conducted a systematic empirical investigation to test two hypotheses regarding SNIP's performance:

• H1 (Algorithm Exploitation): Does the LSO algorithm actively exploit the learned cross-modal alignment during optimization?
• H2 (Alignment Granularity): Is the learned alignment fine-grained enough to distinguish between structurally similar symbolic expressions?

Experimental Setup

The study utilized the Feynman and Strogatz benchmark suites from SRBench. The experiments were divided into two complementary studies:

Study 1: Alignment Evolution During Optimization

• Procedure: The authors ran SNIP's standard LSO algorithm (using a Grey Wolf Optimizer) for 80 iterations.
• Metrics Tracked:
  1. Fitness ($R^2$): To confirm the algorithm was successfully improving data-fitting accuracy.
  2. Cross-Modal Alignment: The cosine similarity between the candidate's symbolic embedding ($Z_S$) and the target's numeric embedding ($Z^*_V$).
• Goal: To determine if improving fitness correlates with increasing alignment to the target, which would indicate the algorithm is using the alignment signal to guide the search.

Study 2: Retrieval Task for Granularity

• Procedure: A retrieval task was designed to test the discriminative power of the alignment.
  • A base expression $f$ and its numeric observations $(X, y)$ were provided.
  • $K-1$ perturbed variants of $f$ were generated by making minimal structural changes (e.g., swapping operators, changing constants, substituting variables).
  • The model had to rank the correct expression against the perturbations based solely on the cosine similarity between the symbolic embedding of the candidate and the numeric embedding of the target.
• Perturbation Types: Unary operator swaps, binary operator swaps, constant changes, and variable substitutions.
• Goal: To verify if the model can distinguish between an expression and its structurally similar but semantically different variants.

3. Key Contributions

The paper makes three primary contributions:

1. Empirical Evidence of Disconnected Optimization: The authors demonstrate that SNIP's LSO procedure does not actively exploit the learned cross-modal alignment. As numerical fitness ($R^2$) improves, the alignment score between the candidate and the target remains flat or even decreases.
2. Quantification of Coarse-Grained Alignment: Through retrieval tasks, the authors prove that SNIP's alignment is too coarse to support principled symbolic search. The model fails to distinguish between structurally similar expressions, often ranking perturbed variants higher than the ground truth.
3. Roadmap for Future Work: The paper identifies fine-grained alignment as the critical bottleneck for multi-modal LSO in SR, moving beyond the assumption that contrastive pre-training alone is sufficient for symbolic optimization.

4. Key Results

Results of Study 1 (Optimization Dynamics)

• Fitness Improvement: The algorithm successfully improved data-fitting accuracy, with $R^2$ increasing from 0.73 (initial) to 0.99 (final) on the Feynman dataset.
• Alignment Stagnation: Despite the fitness gains, the cross-modal alignment score remained essentially flat (starting at ~0.038 and dropping to ~0.001).
• Conclusion: The optimization process operates independently of the cross-modal relationships learned during pre-training. The algorithm finds better numeric fits without necessarily finding symbolically "closer" representations to the target.

Results of Study 2 (Retrieval & Granularity)

• Performance on Feynman Dataset:
  • Retrieval Accuracy: 18.3% (significantly below the random baseline of 27.3%).
  • Interpretation: The model performs worse than random chance, actively misranking perturbed expressions as more similar to the target than the correct expression.
• Performance on Synthetic Data:
  • Retrieval Accuracy: 23.0% (matching the random baseline of 22.9%).
  • Interpretation: Even on in-distribution data, the model offers no advantage over random selection.
• Perturbation Analysis:
  • Binary Operator Swaps: The most problematic, fooling the model 65.8% of the time. This indicates the model captures general functional forms but fails to distinguish specific arithmetic operations.
  • Variable Substitutions: The easiest to detect (24.4% fooling rate), likely because variable indices are explicitly encoded in the input tensor.
• Conclusion: The alignment is too coarse-grained to support symbolic search. It captures broad semantic similarities but fails at the structural precision required for SR.

5. Significance and Implications

• Redefining Multi-Modal SR: The paper challenges the assumption that CLIP-style contrastive learning is a "plug-and-play" solution for Symbolic Regression. It highlights a fundamental gap between coarse-grained semantic alignment (sufficient for image captioning) and fine-grained structural alignment (required for mathematical expression discovery).
• Bottleneck Identification: The research isolates two specific bottlenecks:
  1. Algorithmic: Current LSO algorithms do not leverage alignment signals effectively.
  2. Model-Level: The alignment itself lacks the necessary granularity.
• Future Directions:
  • Model Improvements: Future work must focus on training objectives that enforce fine-grained discrimination (e.g., hard negative mining, structured contrastive losses) rather than just high-level semantic matching.
  • Algorithmic Integration: Once fine-grained alignment is achieved, optimization algorithms must be redesigned to explicitly optimize for both fitness and alignment simultaneously.
• Broader Impact: These findings apply to any multi-modal approach in SR, including one-shot prediction and neural-guided GP. The paper argues that without solving the granularity issue, multi-modal methods will remain limited in their ability to discover complex, interpretable symbolic laws.

In summary, while multi-modal LSO holds significant potential for transforming Symbolic Regression, the current state-of-the-art (SNIP) fails to realize this potential because the learned alignment is too coarse to guide the search, and the optimization process does not utilize the alignment even if it were perfect. The path forward requires a dedicated focus on fine-grained cross-modal alignment.