🤖 machine learning

Pretrain on Small Synthetic Data, Scale Large for Free: Symmetry-Aware Foundation Model for Logic Rule Induction

This paper introduces a symmetry-aware foundation model for logic rule induction that, by enforcing exact equivariance through a novel canonical export mechanism, enables a model pretrained on small synthetic data to generalize interpretable and accurate rules to significantly larger schemas without retraining.

Original authors: Yin Jun Phua

Published 2026-08-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yin Jun Phua

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Detective's Dilemma: Finding Rules in a Noisy World

Imagine you are a detective trying to solve a mystery, but instead of fingerprints, you are looking for hidden patterns in a pile of clues. In the world of computer science, this is called "rule induction." The goal is to teach a computer to look at a bunch of examples—like a list of who bought what, or how a chemical reacts—and figure out the simple "if-then" rule that explains everything. The catch? The computer needs to be smart enough to ignore the noise and the specific names of the people or things involved. If the rule is "If it rains, the grass gets wet," it shouldn't matter if you call the rain "water falling from the sky" or "precipitation," or if you swap the order of the examples in your notebook.

For a long time, computers were terrible at this. They would memorize the specific names of the clues (like "John" or "Atom 5") instead of learning the actual logic. If you gave them a new case with different names, they would get confused. This is where "foundation models" come in. Think of these as super-smart students who study thousands of tiny, made-up puzzles to learn the concept of logic itself, rather than just memorizing the answers to specific puzzles. The big question researchers are asking is: Can we build a computer that learns the shape of logic so well that it can solve brand-new, much bigger puzzles it has never seen before, without needing to study for them again?

The Paper's Big Idea: Teaching Logic to Ignore the "Who" and "When"

This paper introduces a clever new way to build one of these logic-learning computers, called the Symmetry-Aware Foundation Model. The author, Yin Jun Phua, realized that the best way to teach a computer to generalize is to force it to respect "symmetries." In everyday language, symmetry here means that the computer should treat the world fairly: it shouldn't care if you shuffle the order of the examples, rename the variables, flip a switch from "on" to "off," or swap the labels "yes" and "no."

The researchers started with an existing model called the Neural Rule Inducer (NRI). This model was already pretty good; it could learn rules from small, noisy datasets. However, it had a flaw: it relied on shortcuts based on the order of the data or the specific names of the atoms (the building blocks of the logic). If you gave it a puzzle with 1,000 atoms instead of the 12 it was trained on, it would fail because it was relying on those specific names rather than the underlying logic.

To fix this, the author didn't retrain the model from scratch. Instead, they built a "symmetry-aware" wrapper around it. They added a few architectural tweaks and a special "export" step that acts like a translator. Here is how it works in simple terms:

The Architectural Fixes: They removed the parts of the model that cared about the order of the data or the specific names of the atoms. They made the model "blind" to these irrelevant details, forcing it to focus only on the relationships between the clues.
The "Canonical Export": This is the paper's star invention. When the model guesses a rule, it produces a set of scores. The new export method takes these scores and translates them into a final rule in a very strict, standardized way. It ensures that if you swap the names of the atoms in the input, the output rule swaps the names in exactly the same way. If you flip a switch, the rule flips back. It does this without needing to learn anything new; it's a mathematical guarantee built into the translation process.

What They Found: Scaling Up for Free

The team tested their new model, which they call G-NRI, on some very tough challenges.

The "Stress Test": They trained the model on tiny puzzles with only 6 to 12 variables (atoms). Then, they froze the model and asked it to solve puzzles with up to 1,024 variables. That is 85 times larger than what it was trained on.
The Result: The original model (the baseline) fell apart as the puzzles got bigger, dropping to the level of random guessing. But the new G-NRI model stayed strong. It kept its accuracy high and, most importantly, the rules it produced were mathematically consistent. If you shuffled the input, the output rule shuffled perfectly to match.
Real-World Proof: They also tested it on 19 real-world datasets, like medical records and chemical data. While it didn't beat models that were specifically trained on each dataset (which is expected for a "zero-shot" model), it performed significantly better than the original model, especially on the larger datasets. In fact, on some large datasets, it even beat the "majority class" guess (just guessing the most common answer).

The Takeaway: A Mathematical Guarantee, Not Just a Guess

The most exciting part of this paper isn't just that the model got better; it's why it got better. The author proved that their "canonical export" method is a mathematical guarantee. As long as the model's internal scores respect the symmetries, the final rule must respect them too. This isn't a lucky accident; it's a property of the design.

They found that by enforcing these symmetries "by construction" (building them into the system rather than hoping the model learns them), they turned a small-data model into a reusable tool that can handle massive, complex problems. The model didn't need to be retrained for the big puzzles; it just needed the right "translator" to read its mind.

In short, this paper shows that if you teach a computer to ignore the irrelevant details (like names and order) and focus only on the logical structure, it can scale up to solve problems far beyond its training. It's like teaching a detective to recognize the pattern of a crime rather than memorizing the faces of the suspects, allowing them to solve cases in cities they've never visited. The author suggests this approach makes the model reliable for "zero-shot" transfer, meaning it can jump from small, synthetic training data to large, real-world applications with a high degree of confidence, all while keeping the rules it generates simple and easy for humans to read.

Technical Summary: Symmetry-Aware Foundation Model for Rule Induction

Problem Statement

Logical rule induction aims to learn interpretable hypotheses (specifically Disjunctive Normal Form, or DNF, rules) that generalize across different propositional schemas. Existing approaches, such as Inductive Logic Programming (ILP) and Learning from Interpretation Transition (LFIT), often struggle with noisy, partially observed, or unseen data. While neuro-symbolic methods offer noise tolerance, they typically require per-task training, failing to act as true foundation models that can transfer zero-shot to new schemas.

A critical barrier to scaling these models is the violation of structural symmetries. A robust rule inducer should respect four key symmetries inherent to logical problems:

Example Order ( $S_M$ ): Permuting the order of training examples should not change the learned rule.
Atom Renaming ( $S_N$ ): Renaming atoms (variables) should result in a correspondingly renamed rule.
Polarity Flips ( $Z_2^N$ ): Flipping the polarity of an atom (e.g., $x_j \to \neg x_j$ ) should flip the corresponding literal in the rule.
Label Swap ( $Z_2^y$ ): Swapping the output labels (0/1) should result in the Boolean complement of the learned rule.

Previous foundation models, such as the Neural Rule Inducer (NRI), respect example order but fail to maintain equivariance under atom renaming, polarity flips, or label swaps. Consequently, their performance degrades significantly when applied to schemas larger than those seen during training, and their exported rules do not transform correctly under symmetry operations.

Methodology

The paper proposes G-NRI, a symmetry-restored version of the Neural Rule Inducer. The core contribution is not a new learning algorithm, but a canonical export mechanism and specific architectural modifications that enforce exact equivariance by construction, without requiring retraining or additional learnable parameters.

1. Symmetry Group Definition

The problem is framed under the symmetry group $G = S_M \times B_N \times Z_2^y$ , where $B_N = S_N \ltimes Z_2^N$ represents signed atom permutations. A map is $G$ -equivariant if it commutes with every transform in $G$ .

2. Architectural Fixes (Restoring $B_N$ )

To ensure the NRI's internal score generation is equivariant under atom permutations and polarity flips, three specific modifications are made to the encoder and decoder:

Zero Literal Sign Feature: The constant feature distinguishing positive and negative literals is zeroed out, preventing the model from relying on fixed sign representations.
Disable Example-Attention Content Keys: Learned keys for examples are disabled to prevent aggregation from depending on specific atom orderings.
Tie-Complete Selection: Instead of breaking ties between literal pairs by atom index (which breaks permutation invariance), the model uses a "tie-complete" selection that retains all pairs with scores tied to the $k$ -th best.
Polarity-Averaged Embeddings: For $Z_2^N$ equivariance, embeddings of selected pairs are averaged over their polarity flips.

3. Canonical Rule Export (Restoring $Z_2^y$ and Full $G$ )

The paper introduces a canonical export that decodes discrete rules from continuous literal scores ( $p^+, p^-$ ) in a way that is mathematically guaranteed to be equivariant:

Presence Score ( $s_j$ ): Defined as $\max(p^+_j, p^-_j)$ , invariant to polarity swaps.
Signed Contrast ( $d_j$ ): Defined as $p^+_j - p^-_j$ , which flips sign under polarity swaps.
Decision Logic: An atom is included if $s_j \ge 0.5$ and $d_j \neq 0$ . The polarity is determined by the sign of $d_j$ . If $d_j = 0$ (exact tie), the atom is omitted (abstention) rather than arbitrarily assigned a polarity.
Dual-Rail Selection: To handle label swaps, the model averages predictions from two "rails" (positive and negative classes) and selects the final rule or its De Morgan complement based on a label-swap-invariant comparator.
Abstention: The export abstains (returns an empty rule) on exact ties in the selection process rather than breaking them arbitrarily.

4. Training and Inference

Symmetrised Training: The loss function is symmetrized by averaging predictions over label swaps, though the paper notes the inference-time-only variant (components 1–4) achieves nearly identical accuracy.
Zero-Shot Scaling: The model is trained on small synthetic schemas ( $N=6$ to $12$) and evaluated on schemas up to $N=1024$ without retraining.

Key Contributions

Formalization of Symmetry in Rule Induction: The paper casts binary rule induction under the group $G = S_M \times B_N \times Z_2^y$ , defining equivariance as the requirement that the induced rule transforms exactly as the input data transforms.
Canonical Export with Exact Guarantees: The central contribution is a decoding mechanism that lifts equivariance from continuous scores to the discrete exported rule. Theorem 2 proves that if the input scores are $G$ -equivariant, the canonical export yields a rule that is exactly $G$ -equivariant (up to logical equivalence) without retraining.
Empirical Validation of Scaling: The author demonstrates that enforcing these symmetries by construction allows a model trained on small schemas to scale reliably to much larger schemas ( $N=1024$ ), whereas the baseline model degrades to chance performance.

Results

The evaluation uses synthetic stress tests and 19 real-world datasets (including UCI benchmarks, MONK's problems, MUTAG, and CLEVR-Hans3).

Exact Equivariance (RQ1): On synthetic data, the full G-NRI achieves 100% rule equivalence across 13 different symmetry transforms. On real data, the model maintains exact equivariance (1.000) for 17 of the 19 datasets when restricted to schema-valid transforms (where the transform preserves the validity of the encoding). However, on raw transforms that break encoding validity (specifically on the Mushroom and MUTAG datasets), the rule equivalence drops to 0.785, confirming that the guarantee holds specifically for encoding-preserving subgroups.
Scaling Reliability (RQ2): When evaluated on schemas up to $N=1024$ (85 $\times$ the training maximum), the G-NRI maintains stable support-set accuracy and rule fidelity. In contrast, the baseline NRI degrades significantly, approaching random chance at large $N$ . The architectural fixes alone account for the majority of this gain.
Real-World Transfer (RQ3): On 19 real datasets, G-NRI improves zero-shot accuracy over the baseline, with the improvement growing as the schema size increases (e.g., +5.67 percentage points on schemas with $N > 100$ ).
Rule Quality: The exported rules are compact. On average, G-NRI induces DNFs with ~2 clauses and ~6 literals, significantly sparser than Decision Trees or RIPPER baselines, though with slightly lower accuracy than per-task trained models (as expected for zero-shot transfer).

Significance and Claims

The paper claims that enforcing symmetry by construction transforms a small-data pretrained model into a reusable, interpretable foundation model capable of transferring to larger schemas.

Mathematical Property: The author emphasizes that the exact equivariance of the exported rule is a mathematical property of the canonical export mechanism, not specific to the NRI architecture. Any inducer providing $G$ -equivariant literal scores can utilize this export.
No Retraining Required: The method achieves these guarantees without retraining or per-dataset fitting, adding only a small constant factor to inference time (1.2–1.8 $\times$ ).
Modest Scope: The author is modest about the model's general competitiveness; it does not outperform per-task trained learners on all metrics (e.g., accuracy on multi-class problems like "car"). However, it successfully demonstrates that symmetry constraints are essential for the reliability of variable-schema foundation models in rule induction.

The work validates that respecting domain symmetries (atom naming, polarity, label swap) is not merely a learning objective but a structural necessity for scaling logical induction beyond the training distribution.