Runtime Burden Allocation for Structured LLM Routing in Agentic Expert Systems: A Full-Factorial Cross-Backend Methodology

This paper reframes structured LLM routing as a systems-level burden-allocation problem and demonstrates through a comprehensive cross-backend benchmark that optimal routing strategies are highly dependent on specific model interactions rather than universal rules, offering a practical framework for balancing correctness, latency, and cost in agentic AI systems.

The Gist

The Big Idea: It's Not Just About the "Brain," It's About the "Delivery"

Imagine you are running a busy hospital emergency room. You have a highly intelligent Doctor (the AI Model) who can diagnose any problem. But before the Doctor can treat a patient, a Triage Nurse (the Router) must decide which department the patient goes to: Surgery, Pediatrics, Cardiology, or General Medicine.

For years, researchers have been obsessed with making the Doctor smarter. They ask, "Which Doctor is the best?"

This paper argues that we are asking the wrong question. The real problem isn't just who the Doctor is; it's how the Triage Nurse writes the referral note.

The authors call this "Runtime Burden Allocation." In plain English: Who does the heavy lifting of formatting the instructions?

1. Option A: The Doctor writes the full, perfect, machine-readable referral note themselves.
2. Option B: The Doctor writes a quick, messy shorthand note, and a Computer Script (local software) cleans it up and turns it into a perfect referral note.

The paper asks: Does it matter which option we choose?

The Experiment: A Massive "What-If" Game

To find the answer, the researchers set up a giant experiment. They didn't just test one thing; they tested every possible combination of:

• 3 Different Doctors: OpenAI (GPT), Google (Gemini), and Llama (an open-source model).
• 4 Different Ways of Writing Notes: From "Perfect JSON" (Option A) to "Shorthand + Cleanup" (Option B).
• Different Constraints: Like giving the Doctor more time or less time.

They ran over 15,500 test cases. It was like running a full-scale simulation of a hospital for a month to see what happens when you change the paperwork rules.

The Shocking Discovery: There is No "One Size Fits All"

If you asked a typical tech expert, "Which way of writing notes is best?" they would say, "The one that is fastest and cheapest!"

The paper says: Nope.

The best method depends entirely on which Doctor you are using.

• For the "Smart & Stable" Doctors (OpenAI & Google):

  • If you let them write the messy shorthand and clean it up later (Option B), they get faster and cheaper, but they start making terrible mistakes in their diagnosis. They get confused by the shorthand.
  • Analogy: It's like asking a brilliant surgeon to write a prescription in crayon. They might be faster, but the pharmacist (the computer script) might misread it, and the patient gets the wrong medicine.
  • Verdict: Let these Doctors write the full, perfect note themselves. It's safer.

• For the "Fast but Fragile" Doctor (Llama):

  • If you ask this Doctor to write the full perfect note, they are actually quite good at it.
  • But if you try to use the "Shorthand + Cleanup" method, the system completely collapses. The Doctor gets so confused by the shorthand that the cleanup script can't fix it. The referral note becomes garbage.
  • Verdict: This Doctor is actually the fastest at writing the full note, but the "cleanup" method breaks their brain.

The "Streaming" Myth

The paper also tested Streaming. This is when the Doctor starts talking to you before they finish their whole thought (like seeing the first few words of a text message before the rest arrives).

• The Finding: For a Triage Nurse, streaming is useless.
• Analogy: Imagine the Triage Nurse starts handing you a piece of paper that says "Send to..." but stops there. You can't send the patient anywhere until the whole sentence is written. Seeing the first few words doesn't help you move the patient faster. You have to wait for the full sentence anyway.
• Lesson: Don't waste energy trying to make the AI "stream" its answers for control tasks. Just wait for the full answer.

The Three Golden Rules for Developers

Based on this study, here is how you should build your AI system:

1. Don't assume the "Fastest" method is the best.
   Just because a method saves money or time doesn't mean it works. If you switch to a "shorthand" method, you might save 50% on costs but lose 50% of your accuracy.
2. Match the Method to the Model.
   You cannot use the same "paperwork rules" for every AI. What works for Google's AI might break OpenAI's AI or Llama. You have to test them separately.
3. Protect the "Specialist" Routes.
   The study found that while the AI might be good at sending patients to "General Medicine," it might fail miserably at sending them to "Neurosurgery" if you use the wrong paperwork method. If a mistake is expensive (like sending a patient to the wrong specialist), stick to the safe, full-note method.

The Bottom Line

This paper is a wake-up call for AI engineers. We have been treating AI routing like a simple chatbot conversation. But in real-world systems (like expert systems), how you package the AI's answer is just as important as the answer itself.

You can't just pick the "best" AI and the "best" speed setting and hope for the best. You have to find the perfect match between your specific AI and the way you ask it to write its instructions. It's not about finding the fastest car; it's about knowing which car handles best on your specific road.

Technical Summary

1. Problem Statement

The paper addresses a critical gap in the deployment of Agentic Expert Systems: the assumption that structured Large Language Model (LLM) routing is primarily a prompt-engineering challenge. The authors argue that it is fundamentally a systems-level runtime burden-allocation problem.

In production agentic systems, the "Director" component must convert natural language requests into machine-readable control records (e.g., deciding whether to call a tool, query memory, or route to a specific subsystem). The core problem is that the correctness, latency, and cost of this routing are not determined solely by the model's intelligence but by how the "structural work" (serialization, schema enforcement, and state retention) is distributed between the LLM generation loop and the surrounding deterministic software.

Current literature lacks a framework to evaluate how different runtime packaging strategies (e.g., direct JSON emission vs. compressed code with local reconstruction) interact with different backend families (e.g., OpenAI, Gemini, Llama) to affect system reliability.

2. Methodology

The study employs a Full-Factorial Cross-Backend Benchmark designed to isolate the effects of runtime packaging from model identity.

• Experimental Design:
  • Scale: 48 deployment configurations tested across 15,552 total requests (324 requests per configuration).
  • Factors:
    1. Backend Families: OpenAI, Gemini, and Llama.
    2. Runtime Modes (Burden Allocation):
       • MJ (Minimal JSON): Low token budget, model emits final JSON.
       • SJ (Standard JSON): High token budget, model emits final JSON.
       • MJS (Streaming JSON): JSON target with streaming transport.
       • MCLR (Compressed Local Reconstruction): Model emits compact plain-text code; deterministic local software reconstructs the final JSON schema.
    3. Constraints: Limited vs. Unlimited generation budgets.
    4. Transport: Streaming vs. Non-streaming.
• Metrics:
  • Correctness: Format Compliance (FC), Routing Accuracy (RA), State Retention (SR).
  • Efficiency: Response Latency (p50, p95), Token Consumption (TOK).
  • Workflow Proxy: Workflow Lower-Bound Completion (WLC), a conservative metric estimating the rate at which a usable, correctly dispatched control record is produced.
• Statistical Analysis: Type-II ANOVA was used to test main effects and, crucially, the Backend × Mode interaction, moving beyond simple ranking to analyze conditional compatibility.

3. Key Contributions

1. Theoretical Framework (Runtime Burden Allocation): The paper proposes a framework defining routing quality as a function of three dimensions: Serialization Burden (how much schema construction the model does), Transport Semantics (streaming vs. batch), and Locus of Structure Realization (model vs. local software).
2. Methodological Shift: It reframes structured routing evaluation from a "model leaderboard" approach to a compatibility-based systems design approach. It argues there is no universal "best" mode; performance is dictated by the interaction between the backend and the runtime profile.
3. Empirical Evidence: The study provides the first large-scale, full-factorial evidence that Backend × Mode interaction is a first-order determinant of routing quality, often outweighing the main effects of the model or the mode individually.
4. Deployment Guidelines: It offers concrete, data-driven rules for engineers on when to use compressed reconstruction versus direct JSON emission based on the specific backend and the criticality of the routing task.

4. Key Results

• No Universal Optimum: There is no single routing mode that performs best across all backends.
  • Gemini & OpenAI: High-fidelity direct JSON (MJ/SJ) modes maintain the highest correctness (RA ~86%, SR ~75%). Compressed modes (MCLR) offer significant efficiency gains (lower tokens/latency) but suffer substantial correctness degradation (RA drops to ~58–63%).
  • Llama: Exhibits extreme incompatibility with compressed modes. While MCLR is the fastest, it causes a collapse in correctness (RA drops to 23%, FC to ~53%). Direct JSON modes (MJ/SJ) remain robust (82% RA).
• Interaction Dominance: Statistical analysis reveals that the Backend × Mode interaction has a massive effect size (Partial Eta Squared > 0.93 for correctness metrics). This confirms that the "best" packaging strategy is entirely backend-dependent.
• Efficiency vs. Correctness Trade-off: Compressed local reconstruction (MCLR) consistently reduces token usage (by ~60–75k tokens) and latency, but this efficiency is "purchased" with a loss of route correctness and state retention, particularly on Llama.
• Streaming is Secondary: Transport semantics (streaming vs. non-streaming) have negligible effects on core correctness metrics (RA, SR, FC) and only minor effects on latency. For control tasks where the full record is required before downstream execution, streaming offers little operational value.
• Route-Level Fragility: Aggregate accuracy can mask critical failures. Specialist routes (e.g., "developer" or "tool" branches) are significantly more fragile under compressed modes than general conversational routes. On Llama, MCLR caused a 100% failure rate for the "dev" route.
• Workflow Impact: The Workflow Lower-Bound Completion (WLC) metric shows that while MJ/SJ maintain a ~50–60% workflow completion floor, MCLR on Llama drops to 0%, rendering the system unusable for critical workflows despite its speed.

5. Significance and Implications

• For System Designers: The paper provides a decision framework for selecting runtime packaging.
  • Rule 1: Use direct JSON (MJ/SJ) when correctness and state retention are paramount, especially on Llama or for critical specialist routes.
  • Rule 2: Use compressed local reconstruction (MCLR) only when efficiency is the primary driver and backend-specific validation confirms semantic stability.
  • Rule 3: Do not rely on streaming for control tasks; prioritize full-response latency.
• For Research: It establishes that "structured output" is not a static capability of a model but a dynamic property of the model-software interface. It calls for future research to move beyond isolated model benchmarks toward compatibility-aware evaluation.
• For Industry: The findings warn against "one-size-fits-all" deployment strategies. A routing configuration optimized for OpenAI may catastrophically fail on Llama if the same burden-allocation strategy is applied without adjustment.

In conclusion, the paper argues that runtime packaging is a first-class design variable in agentic systems. The optimal strategy is not to find the "smartest" model, but to find the compatible burden-allocation profile that places the specific backend on the best possible correctness-cost-latency frontier for the target application.