💬 NLP

FM SO.P: A Progressive Task Mixture Framework with Automatic Evaluation for Cross-Domain SOP Understanding

FM SO.P introduces a progressive task mixture framework and an automatic multi-agent evaluation system to enhance cross-domain Standard Operating Procedure (SOP) understanding by sequentially building terminology precision, sequential ordering, and constraint reasoning capabilities.

Original authors: Siyuan Huang, Ziyu Wang, Chao Pan, Han Zhao

Published 2026-02-11

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Siyuan Huang, Ziyu Wang, Chao Pan, Han Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a new employee at a massive, global corporation. This employee needs to follow Standard Operating Procedures (SOPs)—the "rulebooks" for everything from how to refund a customer in a bank to how to process a driver's license at the DMV.

The problem? Most AI models (like ChatGPT) are like "smart but scatterbrained" interns. They know a lot of facts, but when you give them a complex rulebook, they get confused. They might skip a step, mix up two similar-sounding words (like "refund" vs. "reimbursement"), or try to give a refund before they’ve even checked if the customer is logged in.

The researchers created FM SO.P, a specialized training program designed to turn these "scatterbrained interns" into "expert professionals."

Here is how they do it, using three simple metaphors:

1. The Training: "The Three-Step Ladder"

Instead of throwing the whole rulebook at the AI on day one, the researchers use Progressive Task Mixture. Think of it like teaching a child to read:

Step 1: Learning the Vocabulary (Concept Disambiguation). Before you can write a novel, you have to know the difference between "there," "their," and "they're." The AI learns to distinguish tiny but vital differences in professional jargon so it doesn't confuse a "reimbursement" with a "refund."
Step 2: Learning the Sequence (Action Sequence). Now that it knows the words, it learns the order. It’s like learning to bake: you can't frost the cake before you've baked it. The AI is trained on "broken" recipes so it learns exactly why skipping a step ruins the result.
Step 3: Learning the Logic (Graph Reasoning). This is the advanced level. It’s like playing chess. The AI learns that "If X happens, then do Y, but only if Z is true." It learns to navigate complex "decision trees" where one wrong turn leads to a dead end.

2. The Evaluation: "The Multi-Agent Jury"

Usually, testing an AI is like a teacher grading a multiple-choice test. It’s rigid and misses the nuance. FM SO.P uses a "Jury of Experts" (an Automatic Multi-Agent Evaluation System) to grade the AI:

The Rule-Maker (Agent 1): This agent looks at the specific field. If the AI is being tested on Banking, the Rule-Maker says, "Focus on security and math!" If it's the DMV, it says, "Focus on time limits and ID checks!"
The Examiner (Agent 2): This agent creates a "stress test." It doesn't just ask easy questions; it creates tricky, complex scenarios to see if the AI actually understands the rules or is just guessing.
The Judge (Agent 3): This agent looks at the AI's answer and compares it to the "Gold Standard," grading it on specific dimensions like "Did it stay professional?" or "Did it protect user privacy?"

3. The Result: "The Small Giant"

The most impressive part of this paper is the Efficiency.

In the AI world, "bigger is usually better," but big models are expensive and slow—like a massive cargo ship that takes miles to turn. The researchers proved that by using this specialized training, they could take a small, fast model (a 7B parameter model) and make it perform just as well as a massive, heavy model (a 72B parameter model).

In short: They didn't just make the AI "smarter"; they taught it how to think like a professional by teaching it the right things, in the right order, and grading it with a jury that actually understands the job.

Technical Summary: FM SO.P Framework

Title: FM SO.P: A Progressive Task Mixture Framework with Automatic Evaluation for Cross-Domain SOP Understanding

1. Problem Statement

Standard Operating Procedures (SOPs) are essential for enterprise operations but present unique challenges for Large Language Models (LLMs). Existing models struggle with cross-domain generalization and procedural reasoning due to three primary limitations:

Conflicting Optimization in Joint Training: Current models attempt to learn terminology, sequential ordering, and conditional logic simultaneously. This "joint training" causes interference, as models often attempt complex reasoning before mastering basic domain terminology.
Inflexible Evaluation Metrics: Standard evaluation metrics (like BLEU or fixed accuracy) fail to capture domain-specific nuances, such as regulatory compliance in banking versus temporal constraints in DMV (Department of Motor Vehicles) workflows.
Deployment Inefficiency: While massive models (70B+) perform well, they are too resource-intensive for real-world enterprise deployment, which requires efficient, smaller-scale models (e.g., 7B).

2. Methodology

The authors propose FM SO.P, a framework consisting of two core components: a progressive training phase and an automatic multi-agent evaluation phase.

A. Progressive Task Mixture (Training Phase)
Instead of standard curriculum learning (which orders samples by difficulty), FM SO.P introduces a staged task mixture that builds hierarchical capabilities using cumulative data. Each stage retains all data from previous stages to prevent catastrophic forgetting and optimization conflict:

Stage 1: Concept Disambiguation: Focuses on terminology precision (e.g., distinguishing "refund" from "reimbursement") using contrastive learning with term substitution.
Stage 2: Action Sequence Understanding: Focuses on procedural correctness by injecting errors (reordering, omitting, or inserting spurious steps) into workflows to teach the model to recognize sequential dependencies.
Stage 3: Scenario-Aware Graph Reasoning: Focuses on conditional logic by representing SOPs as Directed Action Graphs (DAGs). The model learns to identify constraint violations such as cyclic dependencies, missing preconditions, or invalid edges.

B. Automatic Multi-Agent Evaluation (Evaluation Phase)
To solve the problem of rigid evaluation, the framework employs three specialized agents:

Agent 1 (Adaptive Rubric Generator): Analyzes the domain (e.g., Healthcare) to generate custom scoring dimensions (e.g., patient safety, privacy) and weights.
Agent 2 (Test Generator): Creates stratified test sets ranging from basic factual recall to complex multi-hop reasoning.
Agent 3 (Rubric Scorer): Compares model outputs against ground truth using the domain-specific rubrics generated by Agent 1.

3. Key Contributions

Hierarchical Task Design: A novel training paradigm that structures learning through distinct cognitive stages (Terminology $\rightarrow$ Sequence $\rightarrow$ Logic).
Contrastive Error Injection: A systematic method for generating negative samples (incorrect workflows and invalid graphs) to provide explicit signals of what not to do.
Scalable, Domain-Adaptive Evaluation: An automated system that eliminates the need for manual human annotation by adapting evaluation criteria to the specific requirements of different industries.

4. Results

The framework was evaluated on SOPBench across seven domains (Bank, DMV, Healthcare, Market, University, Library, Hotel).

Superior Performance: The 32B model achieved a 48.3% pass rate, significantly outperforming the Qwen-2.5-72B-Instruct baseline (34.4%).
Extreme Parameter Efficiency: The 7B model (34.3% pass rate) matched the performance of the 72B model (34.4%) while using 10× fewer parameters.
Quality Improvement: Borda count analysis confirmed that the framework doesn't just increase "pass rates" but fundamentally improves the relative quality and reasoning depth of the model's outputs.
Ablation Insights: Results showed that Stage 3 (Graph Reasoning) provided the largest gains for complex domains (e.g., University), while Stage 1 and 2 were critical for sequential domains (e.g., Healthcare).

5. Significance

FM SO.P demonstrates that task architecture is as important as model scale. By carefully structuring the training mixture and the evaluation process, it is possible to equip small, deployable models with the sophisticated reasoning capabilities previously reserved for massive, computationally expensive models. This has profound implications for the practical deployment of AI agents in highly regulated and complex enterprise environments (Finance, Healthcare, Government).