Technical Summary: Certification-Driven Reinforcement Learning for Neutrino Flavor Model Discovery

Problem Statement

Scientific discovery in fields like particle physics often requires searching vast, combinatorial hypothesis spaces under complex domain constraints. In the specific context of neutrino flavor model discovery, the hypothesis space exceeds $10^{26}$ possible models. Constructing a viable model involves selecting particle content, symmetry groups, and representation assignments, then verifying if the resulting Lagrangian reproduces observed neutrino masses and mixing angles.

Existing approaches, such as the AMBer framework, utilize Reinforcement Learning (RL) to navigate these spaces. However, traditional RL relies on scalar reward signals that indicate whether a candidate solution failed but provide little information about why. Consequently, agents frequently waste computational resources repeatedly exploring invalid regions of the search space that share the same structural flaws. While external reasoning tools (e.g., theorem provers, constraint solvers) can identify the specific causes of failure, standard RL algorithms do not fully exploit this structured feedback.

Methodology: Certification-Driven Reinforcement Learning (CDRL)

CDRL introduces a framework that integrates structured feedback from symbolic reasoning tools directly into the RL loop. Instead of treating a failure merely as a negative scalar reward, CDRL extracts a "certificate"—a structured explanation identifying the specific subset of decisions responsible for the violation.

Core Components

1. Policy-Value Network & MCTS: The agent constructs models sequentially using a Monte Carlo Tree Search (MCTS) guided by a neural network. The state is represented as a matrix of particle assignments (irreducible representations and charges).
2. Symbolic Referee (Constraint Layer): Before the search commits to a move, a lightweight symbolic reasoning layer enforces known domain constraints. This layer uses Boolean Constraint Propagation (BCP) to immediately prune partial assignments that violate hard constraints, ensuring the agent only explores feasible regions.
3. Certificate Analyzer: When a candidate model fails a physics evaluation (e.g., yielding a rank-deficient mass matrix or violating symmetry rules), a dedicated analyzer extracts the specific assignments causing the failure.
4. Constraint Database & Clause Injection: The analyzer converts the failure cause into a reusable symbolic conflict clause (a logical constraint). This clause is added to a global database and enforced via BCP in all subsequent search steps. This effectively eliminates entire classes of invalid solutions, not just the single candidate that failed.
5. Knowledge Discovery: Post-hoc analysis of search trajectories extracts interpretable rules (decision patterns) which can be reused as soft constraints to further guide future exploration.

The Feedback Loop

The system operates on two complementary signals:

• Scalar Reward (Soft Constraint): Scores the quality of a valid model (based on $\chi^2$ fit and parameter count), shaping the policy network's weights to prefer high-quality solutions.
• Certificate (Hard Constraint): Identifies the exact components responsible for a failure and forbids them via a logical clause. This prunes the search space, making specific failure patterns logically unreachable for all agents sharing the constraint database.

Key Contributions

1. CDRL Framework: A novel paradigm that transforms symbolic failure certificates into reusable search constraints. This allows the agent to progressively eliminate large invalid regions of the combinatorial space, shifting from "learning to avoid" to "learning to prune."
2. State-of-the-Art Performance: Application of CDRL to neutrino flavor model discovery, achieving significantly higher discovery rates than the previous state-of-the-art (AMBer) while evaluating fewer candidates.
3. Interpretable Knowledge Extraction: A mechanism to extract 40 interpretable rules from search trajectories, demonstrating that the search process uncovers reusable structural dependencies within the theory space.

Experimental Results

The authors evaluated CDRL across three distinct theory spaces: $A_4 \times Z_4$, $A_4 \times Z_N$ (where $N \in [2, 10]$), and $T_{19} \times Z_4$.

• Discovery Rates: CDRL achieved up to 1.95× higher valid model rates and up to 6.33× higher neutrino model rates compared to AMBer.
  • Example: In the $A_4 \times Z_N$ space, CDRL found 0.19% neutrino models versus AMBer's 0.03% (a 6.33× improvement).
• Sample Efficiency: CDRL discovered more valid models while evaluating up to 4× fewer candidates than AMBer. For instance, in the $A_4 \times Z_N$ space, CDRL found 2,343 neutrino models with 1 million evaluations, whereas AMBer found 1,394 with 4 million evaluations.
• Ablation Studies: Removing any single component (neural guidance, MCTS, or symbolic constraints) resulted in substantial performance degradation, with neutrino discovery collapsing to near zero in some ablated configurations.
• Rule Reuse: Reusing the 40 extracted interpretable rules as soft constraints yielded additional gains of up to 2× in valid model rates and 3× in neutrino model discovery.

Significance and Claims

The paper claims that CDRL provides a general framework for scientific model discovery by leveraging the complementary nature of scalar rewards and symbolic certificates. The authors argue that while scalar rewards shape the agent's preferences, certificates fundamentally alter the feasible search space by ruling out impossible regions.

The significance of this work lies in its ability to:

• Uncover Reusable Structure: CDRL demonstrates that combinatorial search spaces in physics contain latent, reusable structure that can be captured via certificates and decision rules.
• Accelerate Discovery: By preventing the rediscovery of equivalent failure modes, CDRL allows for more efficient navigation of massive hypothesis spaces ($>10^{26}$ models) where exhaustive search is infeasible.
• Bridge Neuro-Symbolic AI: The approach successfully unites neural learning (for exploration and value estimation) with symbolic reasoning (for constraint enforcement and failure analysis), offering a practical path toward more interpretable and efficient AI-driven scientific discovery.

The authors conclude that this approach is particularly valuable in domains where external reasoning tools can produce structured feedback, suggesting broad applicability beyond particle physics to other scientific discovery tasks involving complex constraints.