Technical Summary: SOP-Maze

Problem Statement

While Large Language Models (LLMs) have demonstrated significant progress in instruction-following and natural language generation, their ability to operate within complex, real-world business environments remains under-evaluated. Existing benchmarks often focus on generic instruction following, single-turn dialogues, or tool manipulation. However, real-world business scenarios are driven by Standard Operating Procedures (SOPs) that involve conditional branches, complex constraints, nested execution logic, and noisy multi-turn conversations. Current benchmarks fail to capture the specific challenges of faithfully traversing these long logical chains, handling conversational nuances (e.g., sarcasm, intent reversal), and performing context-aware calculations required in enterprise settings.

Methodology

The SOP-Maze Benchmark

To address this gap, the authors introduce SOP-Maze, a benchmark constructed from real-world business data.

• Data Source: The dataset was derived from 300,000 internal API router logs from a business department, capturing actual LLM invocations for business tasks. After semantic clustering and manual selection, 23 distinct business scenarios were identified.
• Scale: The final dataset comprises 397 instances and 3,422 sub-tasks, with an average specification length of 5,040 tokens.
• Task Structure: Each task prompt includes four components:
  1. Objective: Background, model role, and task goals.
  2. Standard Operating Procedures (SOP): Detailed execution logic and operation steps.
  3. User Input: Real-world, often noisy, multi-turn dialogue data.
  4. Output Requirement: Strict adherence to a predefined JSON schema.
• Evaluation Protocol: To ensure reliability and avoid the instability of LLM-as-a-judge, the authors employ a JSON schema-based evaluation. Models must output both the user-expected response and a corresponding reference index from the SOP guidance. Scoring is tiered:
  • 0.2 points: Valid JSON format.
  • 1.0 points: Valid format + correct reference index (decision).

Task Categorization

The authors innovatively categorize SOP tasks into two classes based on the structural complexity of the procedures, borrowing terms from plant root morphology:

1. Lateral Root System (LRS): Represents "wide" tasks with relatively shallow branching structures (max 3 levels) but high branching factors (often >10 child nodes per parent). These tasks demand precise selection among many alternatives while maintaining procedural compliance.
2. Heart Root System (HRS): Represents "deep" tasks characterized by long, intricate logical chains. Successful completion requires the model to traverse multi-step reasoning paths and preserve contextual consistency over long horizons.

Experimental Setup

The study evaluates 18 prominent LLMs (11 API-based and 7 open-source), including models from Anthropic, OpenAI, DeepSeek, Bytedance, and Google. All models were tested with default hyperparameters.

Key Findings and Results

Performance Gap

Extensive experiments reveal that nearly all state-of-the-art models struggle significantly with SOP-Maze. Even the top-performing models (e.g., DeepSeek-V3.1-Thinking, Claude-Opus-4-Thinking) achieve scores far below perfect, indicating a substantial gap between current LLM capabilities and the rigorous demands of business SOPs.

Error Taxonomy

Through a comprehensive analysis of model failures (specifically on responses that satisfied the format requirement but failed the decision), the authors identified three dominant error categories:

1. Route Blindness:
   • Definition: The model fails to follow the SOP procedure correctly, deviating to incorrect branches, skipping prerequisite checks, or applying less specific rules when exceptions are required.
   • Manifestation: In LRS, models struggle to maintain and prune large candidate sets, often committing early to wrong branches. In HRS, models skip intermediate nodes, incorrectly assuming preconditions are met, and jump to later steps.
2. Conversational Fragility:
   • Definition: The inability to handle nuances in real-world dialogue, including strong contextual reliance, disfluent phrasing, and subtle intent (e.g., sarcasm).
   • Manifestation: Models often anchor on initial instructions and fail to update intent when users reverse their stance (e.g., threatening a complaint then saying "let it slide"). They also fail to isolate decisive affirmations amidst noisy, interleaved speech or misinterpret sarcasm as positive feedback.
3. Calculation Errors:
   • Definition: Unreliable discrete reasoning in time-related and arithmetic/statistical calculations embedded in SOPs.
   • Manifestation: Models make substantial mistakes in computing reply latencies or medians, even when timestamps and data are explicitly provided. Ablation studies suggest the bottleneck is often evidence selection under noisy context rather than the arithmetic operation itself.

Reasoning vs. Non-Reasoning Models

Reasoning models (those with "Thinking" capabilities) substantially outperform non-reasoning models, particularly in LRS tasks where managing multiple plausible branches is critical. However, even reasoning models exhibit weaknesses in HRS tasks regarding global organization and verification.

Significance and Contributions

The paper claims the following contributions:

1. First Real-World SOP Benchmark: SOP-Maze is the first benchmark specifically designed to evaluate the practical instruction-following capability of LLMs in complex business SOP scenarios, derived from real enterprise data rather than synthetic generation.
2. Novel Taxonomy: The introduction of Lateral Root System (LRS) and Heart Root System (HRS) provides a structured framework for evaluating model capabilities across both the breadth (wide branching) and depth (long logical chains) of complex procedures.
3. Diagnostic Insights: The identification of route blindness, conversational fragility, and calculation errors offers new insights into the specific limitations of current SOTA models, moving beyond simple format compliance to analyze substantive procedural failures.

The authors position SOP-Maze as a testbed for developing more reliable instruction-following systems and isolating the bottlenecks preventing LLMs from faithfully executing complex, nested business procedures. The work is open-sourced to facilitate further research in this domain.