💬 NLP

ZAYA1-8B Technical Report

The paper introduces ZAYA1-8B, a highly efficient 8B-parameter MoE reasoning model trained entirely on AMD infrastructure that achieves state-of-the-art performance on mathematics and coding benchmarks through a specialized four-stage RL cascade and the novel Markovian RSA test-time compute method, effectively narrowing the gap with much larger models.

Original authors: Robert Washbourne, Rishi Iyer, Tomas Figliolia, Henry Zheng, Ryan Lorig-Roach, Sungyeon Yang, Pritish Yuvraj, Quentin Anthony, Yury Tokpanov, Xiao Yang, Ganesh Nanduru, Stephen Ebert, Praneeth Medepal

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Robert Washbourne, Rishi Iyer, Tomas Figliolia, Henry Zheng, Ryan Lorig-Roach, Sungyeon Yang, Pritish Yuvraj, Quentin Anthony, Yury Tokpanov, Xiao Yang, Ganesh Nanduru, Stephen Ebert, Praneeth Medepalli, Skyler Szot, Srivatsan Rajagopal, Alex Ong, Bhavana Mehta, Beren Millidge

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: A Small Brain with a Superpower

Imagine you have a tiny, incredibly smart student named ZAYA1-8B. This student has a brain with only 700 million active neurons (a very small size for an AI), but they are part of a larger "school" with 8 billion total neurons.

Usually, to solve hard math problems or write complex code, you need a giant brain (like a massive AI with billions of parameters). ZAYA1-8B is special because, despite being small, it can solve these hard problems just as well as, or even better than, much larger "genius" students.

How? By using three secret weapons: a new way of thinking, a special training camp, and a unique method for checking its own work.

1. The Architecture: A Specialized Team

Most AI models are like a single person trying to do everything alone. ZAYA1-8B is built like a specialized team using a "Mixture of Experts" (MoE).

The Team: Imagine a classroom with 16 different teachers (experts). When a question comes in, a "router" (the class monitor) decides which teacher is best suited to answer it.
The New Monitor (ZAYA1 Router): In older models, the monitor was a simple rule-follower. ZAYA1-8B uses a smarter, multi-layered monitor that makes better decisions about who should teach, ensuring the right expert handles the job every time.
The Compressed Library (CCA): To read long books or remember long stories, the model uses a "compressed library" system. Instead of carrying every single book page in its head, it stores a summarized, compressed version that saves space and energy but keeps all the important details. This lets it handle very long conversations without getting tired.

2. The Training: Learning to Think Before Speaking

The paper describes a unique training process designed to turn this student into a reasoning machine.

The "Answer-Preserving" Trim: Teachers often have long, detailed explanations (reasoning traces) that are too long to fit on a single page of a textbook. Instead of cutting off the middle of the story (which would ruin the logic), ZAYA1-8B uses a special trick: it cuts off the end of the explanation but keeps the final answer. This teaches the model how to plan and think, even if the full thought process is too long to fit in one go.
The Four-Stage Boot Camp: After learning the basics, the model went through a rigorous "Reinforcement Learning" (RL) boot camp:
1. Warm-up: Solving easy puzzles and math problems to get used to thinking hard.
2. The Gym: A custom environment with 400 different puzzle generators that get harder as the model gets better.
3. The Main Event: Tackling real math and coding challenges, including a special "Test-Time Compute" (TTC) phase where it learns to generate multiple possible answers and pick the best one.
4. Polishing: A final stage to learn how to chat politely and follow instructions, like a good assistant.

3. The Secret Sauce: "Markovian RSA" (The Group Think)

This is the most innovative part of the paper. Usually, when an AI solves a hard problem, it tries to think of the answer in one long, continuous stream. If the stream gets too long, the AI gets confused.

ZAYA1-8B uses a method called Markovian RSA. Think of it like a relay race with a twist:

The Race: Instead of one runner trying to run the whole marathon, the model sends out 16 runners (candidates) at the same time.
The Handoff: Each runner runs for a short, safe distance (a "tail" of 4,000 words). They don't carry the whole history of the race; they just pass their last few steps (the tail) to the next round.
The Aggregation: A "coach" looks at the last few steps of all 16 runners, combines the best ideas, and sends them out for another round.
The Result: By breaking the thinking process into small, manageable chunks and having the team vote on the best path, ZAYA1-8B can solve incredibly hard math problems (like the AIME and HMMT competitions) that usually require much larger brains.

The Analogy: Imagine trying to solve a giant jigsaw puzzle.

Old Way: One person tries to hold the whole puzzle in their head at once. They get overwhelmed.
ZAYA1-8B Way: You send 16 people to work on small sections. They only pass a few pieces of their section to the next group. The group combines these small pieces to build the whole picture. It's faster, uses less memory, and gets a better result.

4. The Hardware: Built on AMD

The entire model was trained using AMD computer chips (specifically the MI300X GPUs). This is a big deal because it proves that you don't need the most expensive, proprietary chips to train top-tier reasoning models. The team showed that with the right software and hardware setup, AMD chips can handle the heavy lifting of training a complex AI.

5. The Results: Small but Mighty

The paper claims that with this setup:

ZAYA1-8B (with only 0.7 billion active parameters) beats or matches DeepSeek-R1 (which has 37 billion active parameters) on math and coding tests.
When using the "Markovian RSA" group-thinking method, it gets scores on hard math competitions that are very close to massive, expensive models like Gemini-2.5 Pro and GPT-5-High.

Summary

ZAYA1-8B is a small, efficient AI that proves you don't need a giant brain to be a genius. By using a smart team structure, a special training method that keeps the answers safe, and a "group relay race" strategy for thinking, it can solve the hardest math and coding problems while using a fraction of the computing power of its larger competitors.

Technical Summary: ZAYA1-8B

1. Problem Statement

The paper addresses the challenge of achieving high-level mathematical and coding reasoning capabilities in large language models without relying on massive parameter counts. While frontier models often utilize hundreds of billions of parameters, there is a need to determine if competitive reasoning can be achieved with significantly fewer active parameters through architectural innovation, reasoning-heavy training, and efficient test-time compute (TTC) strategies. Specifically, the authors investigate whether a model with under 1 billion active parameters can match or exceed the performance of substantially larger open-weight and proprietary reasoning models on benchmarks like AIME and HMMT.

2. Methodology

Architecture: Zyphra MoE++

ZAYA1-8B is a Mixture-of-Experts (MoE) model with 8 billion total parameters and 700 million active parameters. It introduces three key architectural modifications over standard Transformer MoE designs:

Compressed Convolutional Attention (CCA): Replaces standard attention with a lightweight convolutional downprojector to perform sequence mixing in a compressed latent space. This reduces FLOPs and KV-cache size, enabling efficient long-context training and inference.
ZAYA1 Router: Replaces the standard linear router with a multi-layer MLP-based design enhanced by Exponential Depth Averaging (EDA). This increases routing expressiveness, improves expert specialization, and stabilizes load balancing without significant parameter overhead.
Learned Residual Scaling: Applies learned bias and gating coefficients to the residual stream and layer outputs to control residual-norm growth through depth, preventing gradient vanishing with negligible parameter cost.

Training Pipeline

The model was trained from scratch on a full-stack AMD compute platform (MI300X GPUs and Pensando Pollara networking). The training process consists of:

Pretraining & Midtraining: Includes reasoning data (long Chain-of-Thought) from the start. A novel Answer-Preserving (AP) Trimming scheme is used to handle reasoning traces longer than the context window. This method truncates the tail of the reasoning trace while preserving the final answer, allowing the model to learn from long CoT data even at shorter context lengths (4K, 32K, and 131K).
Supervised Fine-Tuning (SFT): Conducted at 131K context length, focusing on chat, instruction following, code, math, and aggregation examples for test-time compute.
Reinforcement Learning (RL) Cascade: A four-stage post-training pipeline:
1. Reasoning Warmup: Math, puzzles, and TTC traces.
2. RLVE-Gym Curriculum: 400 adaptive puzzle-like environments with difficulty calibrated to a 0.5 pass rate using Thompson Sampling and IRT.
3. Math + Code + TTC: Two phases combining olympiad math, competitive programming, and test-time compute prompts. Includes synthetic code environments (I/O prediction, reconstruction, falsification).
4. Behavioral RL: Final stage for chat style and instruction following.

The RL pipeline utilizes PipelineRL for asynchronous training, DPPO Binary-TV for trust-region masking, Dr-GRPO for loss aggregation, and MaxRL for advantage estimation. Crucially, the authors implement Router Replay, where the trainer reuses expert routing decisions from the rollout engine to prevent MoE-specific logit mismatches that destabilize training.

Test-Time Compute: Markovian RSA

The paper introduces Markovian RSA, a test-time compute method combining Recursive Self-Aggregation (RSA) with the bounded-workspace principle of Markovian Thinking.

Mechanism: The model generates $N$ candidate reasoning traces in parallel. Instead of passing full chains to the next aggregation round (which grows context linearly), it passes only the final $\tau$ tokens (the "tail") of each trace.
Efficiency: This bounds the aggregation context size ( $C \times \tau$ ) regardless of the per-rollout reasoning budget ( $\beta$ ), allowing for deep reasoning without exploding attention costs.
Training Integration: The model is trained on aggregation examples constructed from expert rollouts (SFT) and self-rollouts (RL), ensuring it learns to aggregate candidate tails effectively.

3. Key Contributions

ZAYA1-8B Model: A 0.7B active / 8B total parameter MoE model that achieves competitive reasoning performance.
Markovian RSA: A novel TTC method that enables recursive reasoning refinement with bounded context, decoupling reasoning depth from context window size.
Answer-Preserving Trimming: A data construction technique enabling the use of long CoT data in pretraining/midtraining contexts shorter than the trace length.
Stable MoE RL Training: Demonstrates the necessity of Router Replay and specific precision settings (BF16 with selective FP32 operations) to stabilize PipelineRL in MoE models.
AMD Stack Validation: Validates the AMD MI300X and Pollara networking stack for large-scale MoE pretraining and long-context training.

4. Results

Evaluated on the Zyphra harness, ZAYA1-8B demonstrates the following performance:

Single-Rollout Performance: Matches or exceeds DeepSeek-R1-0528 (37B active / 671B total) on several benchmarks, including AIME'26 (89.1% vs 87.5%) and HMMT'26 (71.6% vs 79.4% - note: DeepSeek-R1-0528 is 79.4% in Table XI, but 79.4% in text vs 71.6% for ZAYA1; however, the paper claims ZAYA1 matches or exceeds DeepSeek-R1-0528 on several benchmarks, and Table VIII shows ZAYA1 at 89.1 vs DeepSeek-R1-0528 at 87.5 for AIME'26). It is competitive with larger open-weight models like Nemotron-3-Nano-30B-A3B and Qwen3-Next-80B-A3B-Think.
Test-Time Compute (TTC) Performance: With Markovian RSA (40K reasoning budget, 4K tail), ZAYA1-8B reaches 91.9% on AIME'25 and 89.6% on HMMT'25. This narrows the gap to much larger proprietary models, including Gemini-2.5 Pro, DeepSeek-V3.2, and GPT-5-High.
Scaling Efficiency: The model achieves these results with only 0.7B active parameters, suggesting that active parameter count and inference-time tokens are complementary scaling axes.

5. Significance and Claims

The paper claims that competitive mathematical reasoning can be achieved with under 1B active parameters when model architecture, reasoning-heavy training, verifiable RL, and test-time aggregation are co-designed.

Efficiency: ZAYA1-8B demonstrates that small-active-parameter MoE models can leverage test-time compute to approach the performance of frontier-scale models, offering a favorable trade-off in active-parameter $\times$ generated-token cost.
Training Design: The results highlight the importance of training for the specific inference workflow (Markovian RSA) and the critical role of stability mechanisms like Router Replay in MoE RL.
Hardware: The work serves as a validation of the AMD full-stack platform for training complex MoE reasoning models.
Limitations: The authors note that while the model excels in reasoning and code, it trails models with dedicated multi-turn agentic RL on benchmarks like BFCL-v4 and $\tau^2$ . Additionally, while it is strong for its scale on knowledge benchmarks (MMLU-Pro, GPQA), it does not fully close the gap with much larger models, consistent with the intuition that reasoning and factual storage scale differently.

The paper concludes that test-time compute is a promising avenue for smaller reasoning-focused models, provided they are trained to utilize aggregation workflows and possess architectures that support efficient long-context processing.