🤖 machine learning

Knowledge Distillation from Large Reasoning Models to Compact Student Models: A Case Study on the John O Bryan Mathematics Competition

This paper demonstrates that fine-tuning a compact Qwen2.5-7B student model with Chain-of-Thought data distilled from the DeepSeek-R1 teacher using a dual-agent framework and early stopping significantly improves its accuracy on the John O'Bryan Mathematics Competition and MATH-500 benchmarks, while also revealing that shorter response lengths correlate with reduced reasoning quality.

Original authors: Gaurab Baral, Aaditya Khanal, Yangyang Tao, Junxiu Zhou

Published 2026-07-01

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Gaurab Baral, Aaditya Khanal, Yangyang Tao, Junxiu Zhou

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, world-class math tutor (let's call him Professor DeepSeek) who can solve incredibly difficult competition problems. He's amazing, but he's also huge, slow, and requires a massive supercomputer to run. You want to teach a smaller, faster, and cheaper student (let's call her Student Qwen) to think like the professor so she can do the same job on a regular laptop.

This paper is the story of how the researchers tried to teach the student using a specific set of math problems from the John O'Bryan Mathematics Competition (a real contest held at Northern Kentucky University from 2011 to 2025).

Here is the story of their experiment, broken down simply:

1. The Training Method: "Show Your Work"

The researchers didn't just give the student the final answers. They used a special technique called Chain-of-Thought (CoT) Distillation.

The Teacher: First, the "Professor DeepSeek" solved 671 past competition problems. But instead of just writing the answer, he wrote out every single step of his thinking, like a student showing their work on a test.
The Verifier: A second AI checked the Professor's work to make sure the steps were actually correct. Only the perfect solutions were kept.
The Student: The "Student Qwen" (a smaller model) was then trained to mimic these step-by-step solutions.

2. The "Over-Practice" Trap

The researchers faced a classic problem: Overfitting.
Imagine a student who memorizes the exact answers to a practice test instead of learning the math concepts. If you give them a slightly different question, they fail.

The researchers initially let the student train for 1,000 "rounds" (iterations).
The Result: The student started memorizing the specific words of the training problems rather than learning the logic. Her performance actually got worse on new problems.
The Fix: They realized the student peaked at 200 rounds. After that, she was just copying. So, they stopped training early (at 200 rounds) to keep her "fresh" and capable of generalizing.

3. The Results: A Solid Improvement

By stopping early, the student improved significantly:

Before Training: The student got about 65% of the competition problems right.
After Training: She jumped to about 69.4%.
The Bonus: She didn't just get better at these specific problems; she also got better at a different, famous math benchmark called MATH-500, proving she actually learned the skill of reasoning, not just the answers.

4. The "Word Count" Experiment

The researchers wanted to know: How much "thinking space" does the student need?
They forced the student to solve problems with strict limits on how many words she could write (like a strict word count on a test).

No Limit (R1): She wrote about 220 words and got 69% right.
Moderate Limit (R2): She wrote about 120 words and dropped to 62%.
Tight Limit (R6): She was forced to write only about 31 words. Her accuracy plummeted to 42%.

The Analogy: It's like asking someone to solve a complex puzzle. If you give them a small notepad (few words), they have to skip steps and guess. If they have a full notebook (many words), they can write out the logic and get it right. The study found that for these hard problems, you need roughly 50 to 100 words of "thinking space" to get a good answer.

5. Where She Struggled

Speed vs. Logic: The student was weakest on the "Two-Person Speed" section of the competition. These problems required fast, multi-step calculations. When the word count was tight, she couldn't fit the necessary calculation steps in, and her accuracy crashed harder than in other sections.
Formatting Errors: Interestingly, the researchers found that about 40% of the student's mistakes weren't because she was bad at math. She actually got the right number but wrote it in a messy way (like leaving a fraction unsimplified or forgetting to put the answer in a box). If a human or a simple script cleaned up her writing, her score would have been even higher.

The Bottom Line

This paper proves that you can take a giant, powerful AI and teach a smaller, cheaper AI to reason through math problems effectively, but only if you stop training before it starts memorizing.

However, there is a catch: Thinking takes space. If you force the AI to be too concise (too few words), it loses its ability to solve hard problems. The "sweet spot" for these types of math competitions seems to be allowing the AI enough room to write out about 50 to 100 words of reasoning.

Technical Summary: Knowledge Distillation from Large Reasoning Models to Compact Student Models

Problem Statement

Large Language Models (LLMs) with advanced mathematical reasoning capabilities, such as DeepSeek-R1, often require significant computational resources, making local deployment impractical. While Knowledge Distillation (KD) offers a pathway to transfer reasoning abilities to smaller, more efficient "student" models, existing research predominantly relies on generic benchmarks (e.g., MATH-500, GSM8K) that may suffer from pretraining data contamination. There is a lack of understanding regarding how distillation performs on domain-specific, expert-curated problem sets with lower contamination risks, and how specific constraints, such as token budgets, affect the quality of distilled reasoning.

Methodology

The authors propose a pipeline to distill reasoning capabilities from a large teacher model to a compact student model using the John O'Bryan Mathematics Competition (2011–2025) as a specialized corpus.

1. Dataset Construction

Corpus: 671 problems spanning 15 years, categorized into Freshman/Sophomore, Junior/Senior, and Two-Person Speed sections.
Split: Chronological splitting was employed to prevent data leakage: 2011–2021 (Training), 2022–2023 (Validation), and 2024–2025 (Test).
Dual-Agent Framework: A two-step process using the DeepSeek-R1 API:
- Agent 1 (Solver): Generates step-by-step Chain-of-Thought (CoT) reasoning traces.
- Agent 2 (Verifier): Validates the solver's output against the ground-truth answer, returning a CORRECT/INCORRECT verdict.
- Only traces verified as correct (594 out of 651 attempted) were retained for training.

2. Model Architecture and Training

Teacher: DeepSeek-R1 (accessed via API).
Student: Qwen2.5-7B-Instruct (4-bit quantized).
Fine-Tuning: Low-Rank Adaptation (LoRA) was applied using the MLX framework on Apple Silicon hardware.
- Configuration: Rank $r=8$ , $\alpha=16$ , dropout $0.05$, targeting query, key, value, and output projection modules.
- Parameters: 11.534M trainable parameters (0.151% of total).
Training Protocol:
- Diagnostic Run: A 1,000-iteration run identified severe overfitting, with validation loss minimizing at iteration 200 before rising.
- Final Runs: Five independent training runs were executed, each limited to 200 iterations with different random seeds (42, 123, 256, 512, 999) to ensure stability.
- Early Stopping: Models were fused and evaluated based on the 200-iteration checkpoint.

3. Evaluation Strategy

Metrics: Answer accuracy (judged by DeepSeek-R1 to handle equivalent mathematical forms) and perplexity.
Generalization: Out-of-domain evaluation on the MATH-500 benchmark.
Token Budget Experiment: The fine-tuned model was evaluated under six reasoning levels (R1–R6) with progressively restricted token limits (from unlimited down to 100 tokens) to assess the impact of response length on reasoning quality.

Key Results

Performance Gains

Base Model: Qwen2.5-7B achieved 64.67% accuracy on the competition test set.
Distilled Model: The 200-iteration LoRA fine-tuned model achieved a mean accuracy of 69.43% ± 0.17%, a 4.76 percentage-point improvement.
Stability: The narrow standard deviation (0.17 pp) across five seeds confirms the improvement is robust and not an artifact of initialization.
Overfitting: Continuing training to 1,000 iterations resulted in lower accuracy (67.59%) and higher perplexity, confirming that early stopping is critical for small-corpus distillation.

Generalization

On the MATH-500 benchmark, the distilled model improved from a base of 70.1% to 73.1% ± 0.18%.
Performance gains were uneven across subjects: significant improvements were seen in Algebra (93.5%) and Number Theory (82.3%), which are well-represented in the training corpus, while Intermediate Algebra, Geometry, and Precalculus showed weaker performance.

Impact of Token Budget

Length vs. Accuracy: There is a consistent decline in accuracy as token budgets shrink.
- R1 (Unlimited, ~220 words): 69.43%
- R6 (100 tokens, ~31.2 words): 41.9%
Critical Threshold: The most significant drop occurred between R3 (~~96 words) and R4 (~~55 words), suggesting a practical lower bound of 50–100 words for multi-step competition problems.
Section Sensitivity: The "Two-Person Speed" section was most sensitive to token reduction, dropping 34.5 percentage points from R2 to R6, whereas conceptual sections (Freshman/Sophomore, Junior/Senior) were more robust.

Error Analysis

Qualitative analysis of 197 incorrect responses revealed that 40% of failures were due to formatting or extraction issues (e.g., unrendered LaTeX, unreduced fractions) rather than mathematical reasoning errors. This suggests that post-processing could recover a significant portion of these errors without retraining.

Significance and Claims

The paper claims to be the first study to apply CoT distillation to the John O'Bryan Mathematics Competition corpus, offering a controlled environment with low contamination risk. Its primary contributions are:

Demonstrating Viability: Proving that CoT distillation via LoRA with early stopping can effectively transfer reasoning capabilities from a large model (DeepSeek-R1) to a compact model (Qwen2.5-7B) on domain-specific data.
Quantifying Constraints: Establishing that response length is a critical factor in mathematical reasoning quality, identifying a practical token budget floor for multi-step problems.
Methodological Rigor: Highlighting the necessity of early stopping in small-corpus fine-tuning to prevent overfitting and the value of multi-seed validation for stability.

The authors maintain a modest tone, noting that the model is not yet ready for fully autonomous educational use due to the ~30% error rate and potential biases from the single-institution corpus. They position the model as a drafting tool for human-in-the-loop workflows and suggest that future work should focus on diverse corpora and training on compressed CoT traces to improve performance under tight token constraints.