💻 computer science

AraCoT-Agent: Arabic Chain-of-Thought Distillation for Agentic Reasoning

The paper introduces AraCoT-Agent, a framework that distills structured reasoning capabilities into Arabic models using a high-quality, filtered dataset (AraCoT) and a reward-weighted training objective, achieving a 55.7% relative improvement on the new AraAgent-Bench and significantly narrowing the reasoning gap between Arabic and English LLMs.

Original authors: Mohamed Ali Farag

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mohamed Ali Farag

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, hyper-intelligent tutor who speaks perfect English and can solve the world's hardest math puzzles and logic riddles by writing out every single thought step-by-step. Now, imagine you want to teach that same tutor to speak Arabic and solve those same puzzles for 400 million Arabic speakers. The problem? The tutor is great at English, but when it tries to speak Arabic, it gets stuck, skips steps, or just gives up.

That's the gap this paper, AraCoT-Agent, is trying to fix. The researchers found that Arabic Large Language Models (the AI brains) are significantly worse at multi-step reasoning and planning than their English counterparts. They aren't just "dumber"; they lack the specific training data that teaches them how to think through a problem in Arabic.

Here is how they fixed it, using a recipe that sounds a bit like a high-tech cooking show.

The Secret Sauce: Quality Over Quantity

The team didn't just dump a million random Arabic math problems into the AI's brain. Instead, they acted like strict food critics. They took problems from English sources (like grade-school math and competition puzzles), translated them into Arabic, and then asked the super-smart "teacher" models (DeepSeek-R1 and QwQ-32B) to solve them, writing out their thoughts in Arabic.

But here's the twist: They threw away 95% of the results.

They built a five-stage "quality filter" pipeline. Think of it as a bouncer at an exclusive club.

The Math Check: Did the answer actually match the correct number? (If not, out).
The Length Check: Was the explanation too short (lazy) or too long and rambling? (If so, out).
The Fluency Check: Did the Arabic sound natural, or did it sound like a robot trying to speak? They used a tool called AraBERT to measure this.
The Structure Check: Did the AI actually break the problem into at least three clear steps?
The Copy-Paste Check: Did this solution look exactly like another one? (If so, out).

After all that filtering, they were left with just 823 high-quality examples. The paper suggests that for teaching an AI how to reason, quality is king. Having fewer, perfect examples is better than having thousands of messy ones.

The Training: Teaching the Student to Think

Once they had their 823 perfect examples, they didn't just tell the AI to "memorize" them. They used a special training method called reward-weighted supervised fine-tuning.

Imagine a student taking a test. If they get an easy question right, the teacher doesn't need to spend much time on it. But if they struggle with a hard question but eventually get it right, the teacher focuses extra attention there. This method does the same thing: it tells the AI, "Don't waste energy on the easy stuff you already know; focus your learning power on the tricky steps where you almost messed up."

They tested this on three different AI models:

Qwen3-8B (an 8-billion-parameter model)
Llama-3.1-8B (another 8-billion-parameter model)
Qwen3-4B (a smaller 4-billion-parameter model)

The Results: A Giant Leap Forward

The results were impressive. The best model (Qwen3-8B with this new training) jumped from a 30.0% success rate on a new, tough Arabic reasoning test (called AraAgent-Bench) to 46.7%. That's a 55.7% relative improvement.

The paper explicitly rules out a few things:

It's not just about making the model bigger. They found that a smaller, well-trained model (8 billion parameters) actually beat a much larger Arabic-native model (13 billion parameters) that hadn't been taught to reason this way.
It's not just about having more data. They showed that adding more examples beyond a certain point (around 400) didn't help much; the quality of the examples mattered way more.
It's not a magic bullet for everything. The models still struggled with the hardest problems (Tier 5, which require 11+ steps) and with "causal reasoning" (figuring out why things happened).

The Hidden Bottleneck: The "Word-to-Token" Translator

One of the most interesting findings was about how the AI "sees" words. The researchers discovered that the Qwen model is much more efficient at reading Arabic than the Llama model.

Imagine you are reading a book.

Qwen sees Arabic words efficiently: It takes about 1.8 tokens (digital building blocks) to represent one Arabic word.
Llama is clunky: It takes about 3.2 tokens to represent the same word.

This means that for the same amount of "screen space" (context window), Qwen can hold 1,138 Arabic words in its memory, while Llama can only hold 640. The paper suggests this inefficiency is a major reason why Llama struggled more, even with the same training. It's like trying to solve a complex puzzle with half the table space available.

What This Means for the Future

The paper concludes that we can build powerful Arabic reasoning AI without needing to train massive models from scratch. By using a "teacher" to generate perfect reasoning steps, filtering them ruthlessly, and training smaller models to mimic that thinking, we can close the gap between English and Arabic AI capabilities.

However, the authors are careful to note that this is just the beginning. Their work only covers Modern Standard Arabic (the formal language used in news and books), not the many dialects people speak at home (like Egyptian or Gulf Arabic). They also admit they relied on AI to help create the test questions, which might introduce some bias.

In short: The paper suggests that if you want an AI that can think in Arabic, don't just feed it more data. Teach it how to think, give it the best examples possible, and make sure it can "read" Arabic efficiently. It's a promising step, but the journey to perfect Arabic reasoning is still ongoing.

Technical Summary: AraCoT-Agent

Problem Statement

Arabic Large Language Models (LLMs) exhibit a significant deficit in multi-step reasoning, mathematical problem-solving, and agentic planning compared to their English counterparts. The authors attribute this gap ( $\Delta_{gap} > 15$ percentage points) not merely to a lack of data volume, but to the scarcity of high-quality Arabic Chain-of-Thought (CoT) training data and the suboptimal tokenization of Arabic script in existing multilingual models. Current Arabic LLMs struggle to provide the structured, step-by-step reasoning required for complex tasks in education, clinical diagnostics, and legal reasoning.

Methodology

The paper proposes AraCoT-Agent, a framework designed to distill structured reasoning capabilities from frontier teacher models into smaller, deployable Arabic-focused models. The methodology consists of three core components:

1. Dataset Construction (AraCoT)

The authors constructed AraCoT, the first curated, quality-filtered Arabic reasoning trace dataset.

Source: Seed problems were assembled from GSM8K, MATH (Levels 1–3), ArabicMMLU STEM subsets, and BIG-Bench Hard.
Translation: English-origin problems underwent a two-stage translation pipeline (Helsinki-NLP MT followed by LLM post-editing) with round-trip BLEU verification ( $q_{trans} \ge 0.45$ ).
Teacher Distillation: Reasoning traces were generated using frontier models DeepSeek-R1 and QwQ-32B. For each problem, $K=3$ traces were generated via nucleus sampling, and the optimal trace was selected based on answer correctness and length (as a proxy for reasoning depth).
Five-Stage Filtering: A rigorous pipeline filtered the raw traces ( $\sim17,500$ $\sim 17, 500$ ) down to 823 high-quality examples (retaining $\sim4.7\%$ $\sim 4.7%$ ). The filters enforced:
1. Answer Correctness: Numerical tolerance $\epsilon \le 10^{-3}$ .
2. Length Bounds: 150 to 3,500 tokens.
3. Fluency: Perplexity bounds via AraBERT ( $PPL \le \tau_{ppl}$ ).
4. Structural Completeness: Minimum 3 steps and $<5\%$ English content.
5. Deduplication: MinHash-based Jaccard similarity ( $<0.85$ ).

2. Training Strategy

The study employed Reward-Weighted Supervised Fine-Tuning (SFT) inspired by Group Relative Policy Optimization (GRPO).

Objective: The loss function is weighted by a composite reward $r(x,c) = \lambda_1 \cdot \mathbb{1}_{[correct]} + \lambda_2 \cdot |c|/L_{max} + \lambda_3 \cdot \text{fluency}(c)$ . This down-weights gradients for already high-quality examples, focusing learning on harder or less fluent traces.
Architectures:
- Track A: QLoRA (rank 64, 4-bit NF4) fine-tuning on Qwen3-8B.
- Track B: QLoRA fine-tuning on Llama-3.1-8B.
- Track C: Full fine-tuning on Qwen3-4B.
Masking: Loss was applied exclusively to assistant response tokens to prevent the model from wasting capacity on instruction following.

3. Evaluation Benchmark (AraAgent-Bench)

A new native Arabic benchmark comprising 500 problems across five task types (arithmetic planning, logical deduction, spatial navigation, resource allocation, causal reasoning) and five difficulty tiers (defined by minimum solution steps).

Key Contributions

AraCoT Dataset: A high-quality, filtered Arabic reasoning trace dataset constructed via teacher distillation and a formalized five-stage filtering pipeline.
Systematic Cross-Architecture Study: A controlled comparison of CoT distillation across Qwen3-8B, Llama-3.1-8B, and Qwen3-4B using both QLoRA and full fine-tuning.
AraAgent-Bench: A native Arabic evaluation benchmark with formal difficulty stratification for agentic reasoning.
Empirical Insights: Evidence that data quality dominates quantity in CoT distillation, tokenization efficiency is a critical predictor of cross-lingual reasoning transfer, and reward-weighted training yields significant gains on complex problems.

Results

Performance Gains: The best model (Qwen3-8B + CoT via QLoRA) achieved 46.7% accuracy on AraAgent-Bench, a 55.7% relative improvement over the strongest baseline (Llama3-8B base at 30.0%). This improvement was statistically significant ( $p < 0.05$ , McNemar's test).
Scaling Laws: Data scaling analysis revealed a sublinear power law ( $\beta \approx 0.18$ ), indicating diminishing returns beyond $\sim400$ examples. This confirms that a small, highly curated dataset is more effective than a larger, unfiltered one.
Architecture Comparison: Qwen3-8B consistently outperformed Llama-3.1-8B. The authors attribute this to tokenization efficiency: Qwen3 requires $\approx 1.8$ tokens per Arabic word, whereas Llama-3.1 requires $\approx 3.2$ . This results in a 78% larger effective context for Arabic reasoning in Qwen3 (1,138 words vs. 640 words in a 2048-token window).
Error Analysis: CoT distillation shifted the nature of errors. Baseline models failed primarily through hallucination and early termination. Distilled models failed more often through answer extraction formatting and arithmetic errors, indicating a shift from "not reasoning" to "reasoning with occasional errors."
Difficulty Tiers: The performance gap between distilled models and baselines widened significantly at higher difficulty tiers (Tier 4–5), suggesting CoT distillation is most valuable for complex, multi-step problems where pattern matching fails.

Significance and Claims

The paper claims that targeted CoT distillation is a more effective strategy for improving Arabic reasoning than simply scaling pretraining data or using larger Arabic-native models. Specifically:

Quality over Quantity: Fewer than 1,000 curated examples are sufficient to achieve substantial reasoning improvements.
Tokenization as a Bottleneck: The efficiency of the tokenizer is a significant predictor of reasoning performance; models with inefficient Arabic tokenization (like Llama-3.1) struggle to maintain long reasoning chains due to context truncation.
Reward-Weighted Training: GRPO-style weighting provides consistent gains over standard SFT, particularly on the most challenging problems, by acting as an implicit curriculum.

The authors conclude that their approach substantially closes the Arabic-English reasoning gap, enabling smaller, more deployable models to perform complex agentic tasks previously reserved for frontier English models. However, they note limitations regarding the exclusive focus on Modern Standard Arabic (MSA), reliance on API-based teacher models, and the lack of large-scale human evaluation.