Technical Summary: OrthoReg for Hybrid Symbolic-Neural Dynamical Systems

1. Problem Statement

Dynamical systems modeling faces a fundamental trade-off between interpretability and expressiveness. Manually prescribed mechanistic (symbolic) models are interpretable but often misspecified or overly simplistic for complex natural phenomena. Conversely, flexible data-driven neural methods excel at fitting data but lack physical insight. Hybrid modeling attempts to bridge this gap by combining a prescribed or symbolic physics-based component ($f_{phy}$) with a flexible neural residual ($f_{aug}$).

A critical challenge arises when the symbolic component is not fixed a priori but is discovered from data via sparse library-based methods (e.g., SINDy-style approaches). In this regime, standard hybrid approaches often fail to maintain a clean division of labor. Specifically, the neural augmentation may "relearn" or overlap with the symbolic structure, leading to redundant and uninterpretable models.

The paper identifies that existing methods relying on standard $L_2$ regularization (such as the APHYNITY framework) rely on a projection argument that assumes a fixed symbolic subspace. When the symbolic component is learned jointly with sparsity constraints (e.g., $L_1$ regularization), the shrinkage of coefficients leaves residual structure within the symbolic library span. Consequently, the neural augmentation can overlap with the symbolic part even if the total norm is small, breaking the orthogonality guarantee that $L_2$ regularization provides in fixed-class settings.

2. Methodology: OrthoReg

The authors propose OrthoReg (Orthogonal Regularization), a method designed to enforce a complementary decomposition between the symbolic and neural components during joint training.

Core Objective

OrthoReg modifies the training objective to explicitly penalize the empirical overlap between the neural augmentation and the symbolic library. Given a dataset $D = \{x_i\}_{i=1}^N$, a fixed symbolic library $\{\phi_j\}_{j=1}^M$, and a neural augmentation $\hat{f}_{aug}(\cdot; \vartheta)$, the objective is:

$$\min_{w, \vartheta} \mathcal{L}_{fit}(w, \vartheta) + \mu \|w\|_1 + \lambda \sum_{j=1}^M \langle \hat{f}_{aug}(\cdot; \vartheta), \phi_j \rangle_D^2$$

Where:

• $\mathcal{L}_{fit}$ is the data-fit loss (vector-field regression or one-step prediction).
• $\mu \|w\|_1$ enforces sparsity on the symbolic coefficients $w$.
• The third term is the Orthogonality Penalty, where $\langle \cdot, \cdot \rangle_D$ denotes the empirical inner product over the observed states.
• $\lambda$ controls the strength of the orthogonality constraint.

Theoretical Mechanism

Unlike $L_2$ regularization, which penalizes the total norm of the augmentation ($\|\hat{f}_{aug}\|_D^2$), OrthoReg penalizes the squared inner products of the augmentation with every library function.

• Theoretical Guarantee: The authors prove that as $\lambda \to \infty$, the empirical orthogonality violations converge to zero.
• Error Decomposition: Under the empirical inner product, enforcing orthogonality yields a clean error decomposition:
  $$\|f - (\hat{f}_{phy} + \hat{f}_{aug})\|_D^2 = \|P_D^{F_{phy}}(f) - \hat{f}_{phy}\|_D^2 + \|f - P_D^{F_{phy}}(f) - \hat{f}_{aug}\|_D^2$$
  This ensures the symbolic part captures what the library can express (the in-library projection), while the neural part captures the orthogonal complement (the residual).

3. Key Contributions

1. Theoretical Analysis of Overlap: The paper demonstrates that $L_2$ regularization fails to prevent symbolic-neural overlap when the symbolic component is learned via sparse discovery (Theorem A.2). A counterexample shows that sparsity forces the residual to remain within the symbolic span, allowing the neural network to relearn symbolic directions.
2. OrthoReg Algorithm: The authors introduce an algorithmic solution that integrates an orthogonality penalty directly into the joint sparse-discovery objective, ensuring the neural residual remains orthogonal to the symbolic library under the empirical inner product.
3. Empirical Validation: Extensive experiments on benchmark dynamical systems (Damped Pendulum, Lotka-Volterra, SIR, Duffing oscillator) show that OrthoReg improves symbolic recovery (higher F1 scores, fewer terms) and out-of-distribution (OOD) generalization compared to pure symbolic, pure neural, and $L_2$-regularized hybrid baselines.

4. Experimental Results

The evaluation focuses on systems with partial library mismatch, where the true dynamics contain terms outside the symbolic dictionary.

• Symbolic Recovery: OrthoReg consistently achieves the highest F1 scores for recovering the correct symbolic support and the lowest number of terms, indicating a cleaner, more interpretable model. For example, on the Damped Pendulum, OrthoReg achieved an F1 of $0.93 \pm 0.15$ compared to $0.61$ for the $L_2$ baseline.
• Out-of-Distribution Generalization: OrthoReg significantly reduces OOD derivative errors, particularly in regimes of moderate library mismatch. On the Damped Pendulum, OOD derivative error dropped by more than two orders of magnitude relative to the $L_2$ baseline.
• Trade-offs: The method accepts a slight increase in in-distribution (ID) state MSE in exchange for better OOD behavior and symbolic separation. This is consistent with the theoretical view that enforcing a strict orthogonal split sacrifices some raw fit to preserve structural integrity.
• Robustness: The benefits persist under irregular sampling but diminish under heavy observation noise or when missing dynamics are strongly entangled with library features.

5. Significance and Claims

The paper claims that OrthoReg addresses a specific fragility in hybrid modeling: the loss of interpretability when the symbolic component is learned rather than fixed. By making the complementarity between symbolic and neural components explicit through orthogonality regularization, the method ensures that:

• The symbolic component captures all dynamics expressible by the library.
• The neural component captures only the structured residuals that the library cannot express.

The authors position OrthoReg not as a universal accuracy booster, but as a decomposition regularizer. Its primary value lies in preserving a meaningful symbolic explanation while assigning non-symbolic effects to the residual, thereby enabling hybrid scientific models where approximate mechanistic vocabularies and flexible residuals coexist without erasing the symbolic explanation.

Limitations Acknowledged:

• Theoretical guarantees are currently restricted to the empirical inner product and additive models with fixed differentiable libraries.
• Population-level guarantees require uniform convergence arguments over the augmentation hypothesis class, which are left for future work.
• The method does not recover governing laws absent from the library; it only separates library-expressible dynamics from the rest.
• Performance degrades under high noise levels where derivative estimation becomes unreliable.