🤖 AI

Wnuan: Staged Post-Training for Question Answering over Proprietary Enterprise Knowledge

The paper introduces Wnuan, a three-stage post-training pipeline that effectively adapts large language models to proprietary enterprise knowledge by achieving a 91.51% acceptable-answer rate on WnuanBench while quantifying the associated trade-off in general instruction-following capabilities.

Original authors: Xiaofeng Shi, Xiaosong Qiu, Wenxin Ma, Qian Kou, Yiming Pan, Longbin Yu, Ying Liu, Haiping Wang, Hua Zhou

Published 2026-08-04

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiaofeng Shi, Xiaosong Qiu, Wenxin Ma, Qian Kou, Yiming Pan, Longbin Yu, Ying Liu, Haiping Wang, Hua Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant, well-read teenager how to run a very specific, complicated business. This teenager has already read almost every book in the public library and can chat about anything from history to coding. However, they have never seen the company's internal rulebooks, safety manuals, or secret project files. If you just ask them to guess the answers based on what they know from the outside world, they might sound confident but get the details wrong, or worse, make up facts that sound real but aren't true (a problem experts call "hallucinating").

This is the challenge of Enterprise Question Answering. Companies have mountains of private documents that hold the "real" answers, but their AI models don't know them. The goal is to teach the AI these private secrets without making it forget how to be a helpful, polite, and logical conversationalist. It's a delicate balancing act: if you teach it too much about the company, it might stop being a good general assistant; if you don't teach it enough, it can't do its job. The paper you are about to read explores a clever, three-step recipe to solve this puzzle, turning a general-purpose AI into a company expert without losing its mind.

The Wnuan Recipe: Teaching an AI to Be a Company Expert

Meet Wnuan, a new training pipeline designed to turn a general AI into a specialist who knows a company's private documents inside out. Think of the AI not as a student cramming for a test, but as a new employee who needs to learn the company handbook, the safety protocols, and the project history, all while remembering how to be a polite human being.

The authors of this paper built a three-stage training camp for their AI, which they call Wnuan. Here is how the journey unfolds, step by step.

Step 1: Turning the Handbook into a Quiz Show

First, the team had to turn thousands of dry, boring company documents (like PDFs of safety rules or technical specs) into something the AI could actually learn from. You can't just feed the AI a 500-page manual and hope it memorizes it. Instead, they used a process called Document-to-QA.

Imagine taking a dense rulebook and having a smart editor turn every single rule into a "Question and Answer" flashcard. If the rule says, "All employees must wear hard hats in Zone B," the system creates a card that asks, "What must you wear in Zone B?" and provides the answer. They did this for over 220,000 examples! They even had the AI rewrite some answers to make sure they sounded exactly like the specific AI they were training. This created a massive, custom-made quiz deck tailored to the company's secrets.

Step 2: The "Replay" Break

Now comes the tricky part. If you only study the company handbook, you might forget how to talk to people or solve general problems. This is called "catastrophic forgetting." To stop this, the team used a technique called General-Data Replay.

Think of this like a study session where the student studies the company handbook for a while, but then takes a break to chat about general topics like sports, science, or logic puzzles. The paper found that mixing in about 48% general conversation data with the company data was the "sweet spot." It kept the AI smart and polite while it learned the company rules. Without this break, the AI would have become a great company expert but a terrible conversationalist.

Step 3: The "Residual Error" Drill

After the first two steps, the AI was pretty good, but it still made mistakes. It got the easy questions right but stumbled on the hard ones. This is where the third stage, Residual-Error Reinforcement Learning (RL), kicks in.

Instead of re-studying the whole quiz deck, the team looked specifically at the questions the AI got wrong (the "residual errors"). They treated these mistakes like a coach focusing on an athlete's weak spots. They used a special reward system to teach the AI how to fix those specific errors. It's like saying, "You got the easy math problems right, but let's drill on these three tricky algebra problems until you master them."

The Results: A Big Win with a Small Cost

The results of this three-stage training were impressive. Before any training, the AI (called Wnuan-Base) could only give acceptable answers to about 52.76% of the questions on their test set, called Wnuan-Bench.

After Step 2 (SFT with Replay): The score jumped to 80.06%. The AI was now a solid employee.
After Step 3 (Residual-Error RL): The score climbed even higher to 91.51%. The AI had mastered the company's secrets.

The team also checked to see if the AI had forgotten how to be a general assistant. They found that while the AI got slightly worse at following very specific, tricky instructions (a drop of about 5 points on a general benchmark), it didn't lose its general smarts. The trade-off was worth it: the AI became a much better company expert.

What the Paper Rules Out (and What It Doesn't)

The authors were very careful to test their ideas. They didn't just guess that focusing on mistakes was better; they proved it.

Focusing on mistakes works: They compared their "Residual-Error" method (focusing only on wrong answers) against two other methods: studying everything (the full pool) and studying a random mix of questions. The "Residual-Error" method won, beating the random method by 2.97 points and the full pool by 3.11 points. This suggests that once an AI knows the basics, drilling on its specific weaknesses is the most efficient way to learn.
Retrieval isn't a magic fix: The team also tested a common trick called RAG (Retrieval-Augmented Generation), where the AI is allowed to look up answers in a database during the test. Surprisingly, for the fully trained AI, looking up answers actually made it worse in some cases. The AI had already learned the material so well that trying to look it up confused it. This suggests that for this specific type of training, it's better to have the knowledge "baked in" rather than looked up on the fly.
It's not a universal cure-all: The paper admits that this method works great for this company's internal documents. It doesn't claim to solve every AI problem in the world. Also, the AI is still designed to be a helper that humans check, not a robot that makes final safety or hiring decisions on its own.

The Bottom Line

The Wnuan paper shows us a clear path for making AI experts. By turning documents into quizzes, mixing in general conversation to keep the AI balanced, and then drilling specifically on the mistakes it makes, you can turn a general-purpose robot into a highly skilled company insider.

The authors suggest that this "staged" approach is the key: first teach the basics, then refine the weak spots. While the AI does lose a tiny bit of its ability to follow complex instructions, the massive gain in accuracy (from 52.76% to 91.51%) makes it a winning strategy for companies that need their AI to know the rules, the procedures, and the secrets of the business. It's a reminder that sometimes, the best way to learn isn't to study everything again, but to focus intently on what you got wrong.

2.  **Stage II: SFT with General-Data Replay**
    *   **Base Model:** The pipeline starts with **Qwen3-32B**, which the authors denote as **Wnuan-Base**.
    *   **Training:** The model undergoes full-parameter SFT on the domain QA set mixed with public general examples (Soren 2025).
    *   **Replay Strategy:** The authors empirically search for an optimal "replay" ratio (0% to 100% of domain examples) to balance domain performance with general capability retention. The selected **50% nominal setting** (actual ratio ~48.3%) maximizes the unweighted average of MMLU, IFEval, and C-Eval benchmarks. This produces **Wnuan-Inst**.

3.  **Stage III: Residual-Error Reinforcement Learning (RL)**
    *   **Selection:** Instead of training on the full pool, the pipeline identifies "residual errors"—examples that **Wnuan-Inst** still answers incorrectly. This yields a pool of 56,147 examples.
    *   **Optimization:** The model undergoes Group Relative Policy Optimization (GRPO) with a semantic reward function ( $r = 0.6r_{acc} + 0.3r_{quality} + 0.1r_{format}$ ). The reward components score correctness, logical soundness, professional expression, concision, and format compliance.
    *   **Comparison:** A controlled experiment compares this **residual-error sampling** against **full-pool sampling** and **size-matched random sampling** under a fixed 100-update budget.

**Key Contributions**
*   **End-to-End Pipeline:** A recipe for converting proprietary documents into a closed-book QA model, explicitly addressing the trade-off between domain specialization and general capability retention.
*   **Wnuan-Bench:** A new evaluation benchmark consisting of 707 questions across eight business domains (e.g., operational scenarios, standards, safety supervision), curated by internal experts and distinct from the training data.
*   **Residual-Error RL Strategy:** Evidence that concentrating a fixed update budget on examples still judged incorrect after SFT yields superior performance compared to random or full-pool sampling.
*   **Calibrated Evaluation:** An automatic evaluation ensemble (using multiple LLM judges) calibrated against a domain expert, achieving 90.5% agreement on a stratified sample.

**Results**
Evaluated on the 707-question Wnuan-Bench using the primary 32B route:
*   **Performance Gains:**
    *   **Wnuan-Base:** 52.76% Acceptable-Answer Rate (AAR).
    *   **Wnuan-Inst (Post-SFT):** 80.06% AAR (+27.30 points).
    *   **Wnuan-RL (Post-RL):** 91.51% AAR (+11.45 points over Inst).
*   **Residual-Error Sampling Efficacy:** Under a matched 100-update protocol, residual-error sampling achieved **89.39% AAR**, outperforming full-pool sampling (86.28%) and size-matched random sampling (86.42%) by 3.11 and 2.97 points, respectively. Source-cluster bootstrap intervals remained positive for these contrasts.
*   **General Capability Trade-offs:**
    *   The general benchmark average (MMLU, IFEval, C-Eval) decreased from 88.61% (Base) to 83.44% (RL).
    *   This decline was concentrated in **instruction following (IFEval)**, which dropped by 6.52 points from Inst to RL, while MMLU and C-Eval saw slight increases.
*   **Retrieval Analysis:** Applying a fixed RAG pipeline improved the unadapted Base model but was non-additive (and slightly detrimental) for the specialized Inst and RL models, suggesting retrieval should be gated rather than unconditionally concatenated.

**Significance and Claims**
The paper claims to characterize both the gains and the costs of staged enterprise adaptation.
*   **Staged Efficiency:** The pipeline demonstrates that after SFT resolves easy examples, focusing RL on residual errors is an effective policy for maximizing domain accuracy under budget constraints.
*   **Explicit Trade-offs:** The authors modestly note that the gains in domain mastery come with a "general-capability cost," specifically a reduction in instruction-following performance. They do not claim a universal solution but rather a specific operating point selected via empirical grid search.
*   **Evaluation Rigor:** The results are supported by source-cluster sensitivity analysis and expert-calibrated automatic evaluation, distinguishing between aggregate policy improvements and exact row-level invariance.
*   **Limitations:** The study is confined to a single enterprise's in-domain data; it does not claim cross-enterprise generalization or source-held-out performance. The retrieval diagnostic is limited to a single pipeline and does not establish a general training-retrieval interaction.

In conclusion, Wnuan provides a validated recipe for enterprise adaptation that prioritizes residual-error learning in the final stage, offering a clear path to high-accuracy closed-book QA while explicitly acknowledging the associated degradation in specific general capabilities.