Reinforcing Numerical Reasoning in LLMs for Tabular Prediction via Structural Priors

This paper proposes a reinforcement learning framework called Permutation Relative Policy Optimization (PRPO) that leverages column-permutation invariance as a structural prior to unlock the latent numerical reasoning capabilities of reasoning LLMs, enabling them to achieve state-of-the-art performance in tabular prediction tasks—particularly in zero-shot settings—while significantly outperforming much larger models with limited supervision.

The Gist

Imagine you have a giant, super-smart librarian (the Large Language Model, or LLM) who has read every book in the world. This librarian is amazing at writing stories, answering trivia, and solving riddles. However, if you hand them a spreadsheet full of numbers—like a list of house prices, patient records, or stock market data—they often get confused. They might guess the answer, but they can't explain why they guessed it, and they struggle to learn new patterns without reading a whole new library of books first.

This paper introduces a new training method called PRPO (Permutation Relative Policy Optimization) to teach this librarian how to become a Data Detective.

Here is the breakdown of how they did it, using simple analogies:

1. The Problem: The "Shuffled Deck" Confusion

Traditional computer programs (like XGBoost or TabPFN) are like specialized calculators. They are great at crunching numbers but can't "talk" about their reasoning. They are also bad at learning new tasks without lots of examples.

The super-smart librarian (LLM) can talk and reason, but when looking at a spreadsheet, they get tripped up by the order of the columns.

• The Analogy: Imagine you are looking at a recipe card. It says: "Flour: 2 cups, Sugar: 1 cup, Eggs: 3."
• If you shuffle the card to say: "Eggs: 3, Sugar: 1 cup, Flour: 2 cups," the recipe is still the same.
• But a standard AI might get confused, thinking the order matters, or it might not realize that shuffling the ingredients doesn't change the cake. Because the AI doesn't "get" that the order doesn't matter, it struggles to learn the underlying math logic.

2. The Solution: The "Shuffle and Compare" Game

The authors created a special training game called PRPO. Here is how it works:

• Step 1: The Shuffle. They take a single data row (like one patient's record) and shuffle the columns (the features) in many different ways.
  • Analogy: Imagine you have a deck of cards representing a patient's data. You shuffle the deck 10 different times. The "Patient" is still the same, but the order of the cards on the table is different every time.
• Step 2: The Guess. The AI looks at all 10 shuffled versions and tries to guess the answer (e.g., "Will this patient recover?").
• Step 3: The "Aha!" Moment (The Reward).
  • In normal training, the AI only gets a "Yes/No" at the very end. If it guesses wrong, it gets a "0" and doesn't know why. This is like a student getting a test back with just a red "F" and no corrections.
  • With PRPO, the AI gets dense feedback. The system compares the AI's guesses across all the shuffled versions.
  • The Magic: If the AI guesses "Yes" for the original order but "No" for a shuffled order, the system says, "Wait! You know the answer is the same regardless of the shuffle! You are being inconsistent!"
  • This turns a single "Yes/No" grade into a rich conversation about why the answer should be the same, teaching the AI to focus on the math and logic rather than the order of the words.

3. The Result: A Detective Who Needs No Manual

After this training, the AI becomes a Tabular Reasoning Expert.

• Zero-Shot Superpower: Usually, to solve a new problem (like predicting house prices in a new city), you need to show the AI hundreds of examples first. This new AI, however, can look at a brand new dataset it has never seen before and solve it almost as well as if it had studied it for weeks.
  • Analogy: It's like a detective who, after training on one type of crime, can walk into a completely different crime scene and solve it immediately because they understand the principles of deduction, not just the specific clues.
• Beating the Giants: The paper shows that their small model (8 billion parameters) actually beats much larger, famous models (like DeepSeek-R1 with 685 billion parameters) on these tasks.
  • Why? Because the big models are like generalists who know a little about everything but aren't trained specifically for this "shuffled deck" logic. The small model is a specialist who learned exactly how to think about numbers.

Summary

Think of this paper as teaching a brilliant but confused student how to read a spreadsheet.

1. Old Way: Show the student the spreadsheet once and say, "Guess the answer." (They guess wrong and don't learn).
2. New Way (PRPO): Show the student the same spreadsheet 10 times, but shuffle the columns every time. Ask them to explain why the answer is the same every time.
3. Outcome: The student stops memorizing the order of the columns and starts understanding the mathematical relationships between the numbers. They become a reasoning machine that can solve new problems instantly, explain their work clearly, and outperform much larger, more expensive models.

Technical Summary

tags and the final answer within` tags.

B. Permutation Relative Policy Optimization (PRPO)

PRPO is the core innovation, designed to densify reward signals by exploiting the column-permutation invariance of tabular data (i.e., shuffling column order does not change the ground-truth label).

1. Data Augmentation via Permutation: For a single training sample, the model generates $m$ column-permuted variants. Each variant preserves the original label but presents features in a different order.
2. Two-Level Advantage Estimation:
   • Intra-permutation Advantage: Computes the advantage of a rollout within a specific permutation group (comparing the rollout against the mean reward of that specific permutation's rollouts).
   • Inter-permutation Advantage: Computes the advantage across all permutations (pooling all rollouts from all $m$ permutations to calculate a global mean and standard deviation).
3. Combined Objective: The final advantage $\hat{A}^{PRPO}$ is a weighted sum of the intra- and inter-permutation advantages:
   $$\hat{A}^{PRPO}_{k,i} = \alpha \cdot \hat{A}^{(1)}_{k,i} + (1-\alpha) \cdot \hat{A}^{(2)}_{k,i}$$
   This mechanism transforms sparse outcome-level feedback into dense learning signals, providing effective guidance even when the model initially produces incorrect answers.

C. Theoretical Guarantees

The paper provides theoretical proofs showing that PRPO:

• Reduces Variance: The two-level advantage estimation yields lower gradient variance compared to standard GRPO.
• Densifies Rewards: It effectively increases the probability of receiving a positive reward signal by a factor of approximately $m$ (the number of permutations), accelerating the escape from cold-start phases.
• Unbiasedness: The gradient remains an unbiased estimator of the true policy gradient.

3. Key Contributions

1. First Reasoning LLM for Tabular Data: The authors introduce the first LLM specifically tailored for tabular prediction that integrates multi-step reasoning with tabular semantics, producing interpretable predictions.
2. PRPO Algorithm: A novel RL strategy that leverages structural priors (column permutation invariance) to solve the sparse reward problem in tabular RL, stabilizing training and activating latent numerical reasoning capabilities with limited supervision.
3. Comprehensive Dataset: Construction of a large-scale reinforcement learning dataset comprising 139 diverse tabular datasets (OpenML) with verifiable rewards, serving as a foundation for future tabular reasoning research.

4. Experimental Results

The model (based on Qwen3-8B) was evaluated on 139 datasets across classification and regression tasks under fully supervised, few-shot, and zero-shot settings.

• Fully Supervised Performance:
  • Classification: Achieved a mean accuracy of 0.8436 (Rank 2.08), outperforming TabPFN (0.8413) and XGBoost (0.8234).
  • Regression: Achieved a Normalized Mean Absolute Error (NMAE) of 0.1499, competitive with TabPFN (0.1236) and better than XGBoost (0.1682).
• Zero-Shot and Few-Shot Generalization (The Core Breakthrough):
  • Zero-Shot: The model achieved 0.7021 accuracy on unseen classification datasets, significantly outperforming general-purpose LLMs (e.g., DeepSeek-R1 at 0.5313) and surpassing the 16-shot performance of strong baselines like XGBoost and TabPFN.
  • 32-Shot: With 32 in-context examples, the model reached 0.7542 accuracy, surpassing all baselines trained with 32 shots.
  • Efficiency: The zero-shot performance of the 8B model matches or exceeds the performance of much larger models (e.g., DeepSeek-R1 685B) and approaches the performance of 32-shot baselines without any task-specific fine-tuning.
• Transferability: The model trained on tabular data showed improved performance on general mathematical reasoning benchmarks (e.g., GSM8K, MATH), suggesting that PRPO activates fundamental numerical reasoning priors.

5. Significance

• Bridging the Modality Gap: The paper demonstrates that tabular reasoning is not just a data formatting issue but a reasoning capability that can be unlocked in LLMs through structural priors.
• Solving the Sparse Reward Problem: PRPO offers a generalizable solution for RL in domains where process-level rewards are unavailable, by leveraging structural invariance to create dense supervisory signals.
• Efficiency and Accessibility: The results show that a relatively small model (8B parameters) trained with this specific RL method can outperform massive foundation models (685B+) and specialized tabular models in zero-shot settings. This democratizes high-performance tabular prediction, making it accessible without massive computational resources or large labeled datasets.
• Interpretability: Unlike traditional black-box models, this approach provides transparent reasoning traces, enhancing trust in AI-driven decision-making for critical domains like finance and healthcare.