Technical Summary: Classifier-Gated Multi-Tier LLM Routing for Regulated Industries

Problem Statement

The deployment of Large Language Models (LLMs) in regulated industries (e.g., financial services, healthcare) faces a critical trade-off between capability, cost efficiency, and regulatory compliance. Current serving architectures fail to address three specific constraints simultaneously:

1. Data Residency and Compliance: Regulations such as GDPR, SEBI, RBI, and India's DPDPA mandate that Personally Identifiable Information (PII) must not cross jurisdictional boundaries. Single-model deployments cannot enforce this because data enters the model endpoint before any routing decision is made.
2. Inefficiency of Single-Model Serving: Routing all queries to a single large model consumes full GPU capacity regardless of query complexity. This creates infrastructure contention and ignores the reality that most queries in regulated sectors are straightforward lookups requiring only small models.
3. Limitations of Existing Routing Approaches:
   • Mixture-of-Experts (MoE): While MoE routes tokens between expert layers, all experts remain loaded in memory, and routing occurs after data has arrived at the endpoint. This prevents geographic enforcement of data residency.
   • Prior Routing Methods (FrugalGPT, RouteLLM, Hybrid LLM): These approaches focus on cost or quality but lack integrated PII detection and geographic routing. They do not enforce compliance by design, often allowing sensitive data to traverse cross-region infrastructure before a decision is made.

Methodology

The paper proposes ComplianceGate, a classifier-gated routing architecture that enforces compliance and optimizes cost at the pre-inference layer. The system operates within a private network and consists of two primary components:

1. Encoder Classifier

A bidirectional transformer encoder (12 layers, 768 hidden dimensions, ~134M parameters) sits before any decoder inference.

• Training: Pre-trained on 2 billion tokens of English text and fine-tuned on 50,000 domain-specific examples.
• Function: It evaluates raw query text for complexity (simple, medium, complex) and sensitivity (PII presence vs. non-sensitive).
• Output: A probability distribution over six labels (the cross-product of 3 complexity levels and 2 sensitivity dimensions) with a confidence score.
• Performance: Inference time is under 10ms (specifically 7ms in experiments).

2. Routing Logic and Fallback

The routing layer interprets the classifier's output to make a deterministic decision before any LLM computation begins:

• Geographic Enforcement: Queries containing PII are routed to local endpoints (within the jurisdiction). Non-sensitive queries are routed to cross-region infrastructure.
• Model Tier Selection:
  • Simple queries $\rightarrow$ Small, fast dense models (e.g., Nova Lite, Gemma 4B).
  • Medium queries $\rightarrow$ Mid-sized models (e.g., Gemma 27B, Ministral 14B).
  • Complex queries $\rightarrow$ Large, capable models (e.g., Qwen3-480B).
• Confidence-Based Fallback: If the classifier's confidence falls below a configurable threshold (default 80%), the system defaults to the most restrictive label (complex_pii). This ensures that under uncertainty, queries are routed to local, full-capacity models, structurally preventing data residency violations.

3. Deployment Architecture

The system is containerized and deployed via infrastructure-as-code. Each model tier operates on independent infrastructure with no shared memory or failure domains, allowing for independent scaling and upgrades without affecting other tiers.

Key Contributions

1. Compliance-by-Design: The architecture makes data residency violations structurally impossible by detecting PII and routing to local endpoints before any LLM computation occurs.
2. Multi-Tier Cost Optimization: Unlike binary routing or MoE, this system supports multiple model tiers. It routes simple queries to models costing 6–8× less than frontier models, significantly reducing serving costs proportional to actual traffic complexity.
3. Performance and Resiliency: By avoiding the overhead of processing simple requests through massive models, the system reduces latency and increases throughput. Independent tiers ensure that failures in one model do not cascade to the entire system.
4. Formal Cost Model: The paper derives a mathematical model for fractional savings ($S$), demonstrating that savings are maximized when query distributions are skewed toward simple requests (typical in financial services).

Experimental Results

The system was evaluated on 600 queries using Amazon Bedrock (us-east-2) with a baseline of Qwen3-480B-A35B (MoE) versus the gated routing configuration.

• Accuracy: The encoder classifier achieved 99.2% accuracy with near-perfect PII recall.
• Latency Reduction:
  • Simple Queries: 2–4× faster (490–840ms vs. ~1850ms baseline).
  • Medium Queries: 1.7–3.3× faster (1140–2250ms vs. ~3750ms baseline).
  • Complex Queries: No difference (same model used).
  • Median Latency: 39% reduction overall.
• Throughput: Gated routing achieved 122–200 tokens/second for simple/medium queries, compared to 50–64 tokens/second for the MoE baseline.
• Predictability (Variance):
  • MoE baseline showed high latency variance (Max/Min ratio up to 34.5×).
  • Gated routing reduced variance to 1.1–2.0×, making SLA guarantees feasible.
• Cost Savings:
  • 33% savings under an equal query distribution (33% simple/medium/complex).
  • 52% savings under a skewed distribution typical of financial services (60% simple, 25% medium, 15% complex).
  • 74% savings projected under highly skewed distributions (70% simple).

Significance and Claims

The paper claims that ComplianceGate provides a production-ready alternative to single-model and MoE deployments for regulated industries. Its significance lies in the architectural shift from "post-hoc" policy enforcement to pre-inference structural enforcement.

• Structural Advantage: Unlike MoE, which cannot enforce geographic routing once data enters the model, this architecture enforces jurisdictional boundaries at the routing layer.
• Scalability: It allows for independent scaling of model tiers, whereas MoE requires scaling the entire cluster.
• Practicality: With negligible routing overhead (<0.1% of total cost) and high classification accuracy, the system demonstrates that a trained encoder can simultaneously satisfy strict compliance mandates (data residency) and aggressive cost-efficiency goals without sacrificing response quality for complex queries.

The authors conclude that the value of this work is not in the specific models chosen, but in the routing framework that enables matching the right model to the right query based on measurable complexity and sensitivity per query, per jurisdiction, and per constraint.