🤖 AI

Position: Avoid Overstretching LLMs for every Enterprise Task

This paper argues that enterprises should avoid over-relying on large language models for all tasks, advocating instead for modular architectures where LLMs serve strictly as interfaces for structured extraction while dedicated knowledge bases and symbolic procedures handle computation and storage to ensure reliability, scalability, and transparency.

Original authors: Kuldeep Singh, Anson Bastos, Isaiah Onando Mulang'

Published 2026-05-12

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Kuldeep Singh, Anson Bastos, Isaiah Onando Mulang'

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Idea: Stop Asking the "Super-Brain" to Do Everything

Imagine you run a massive, high-stakes factory. You have a brilliant, all-knowing genius (the Large Language Model or LLM) who can write poetry, tell jokes, and solve complex riddles.

Currently, many companies are trying to force this genius to run the entire factory. They ask the genius to:

Remember every single rule in the employee handbook.
Calculate the exact price of a widget based on fluctuating raw material costs.
Check if a specific invoice matches a specific contract.
Decide who gets approved for a loan.

The authors of this paper say: "Stop doing this."

They argue that asking a general-purpose genius to do these specific, repetitive, rule-based factory jobs is inefficient, expensive, and dangerous. Instead, they propose a new way to organize the factory.

The Problem: The "Swiss Army Knife" vs. The "Specialized Tool"

The paper claims that treating an LLM as a "monolithic engine" (a single giant brain that does everything) is a mistake for enterprise tasks.

The Analogy:
Think of an LLM like a Master Chef who can cook anything in the world.

The Current Approach: You ask the Master Chef to not only cook a gourmet meal but also to count the exact number of grains of rice in a sack, check the expiration dates on every can in the pantry, and file the tax returns.
The Result: The Chef gets tired, makes mistakes (like counting the rice wrong), costs a fortune to hire, and you can't easily check why they made a mistake because their brain is a "black box."

The Paper's Solution:
Don't use the Master Chef for the counting and filing. Use them only to talk to the workers.

Hire a Specialized Assistant (a Small Language Model or SLM) whose only job is to listen to a customer and write down exactly what they need (e.g., "Customer wants 500 units of Widget A").
Send that note to a Database (the Knowledge Base) which holds the exact price and inventory.
Send that note to a Calculator (the Symbolic Procedure) which does the math.
The Master Chef is only used offline to train the assistants or write the rules, not to run the factory every second.

The Three-Layer Factory (The Proposed Architecture)

The paper suggests breaking the AI system into three distinct, separate layers:

1. The Interface Layer (The "Translator")

What it is: A small, cheap, specialized AI (SLM).
Job: It takes messy, unstructured human input (like an email saying "I need a refund for my broken toaster") and turns it into a clean, structured list of facts (e.g., Action: Refund, Item: Toaster, Reason: Broken).
Why: It's fast, cheap, and only has to do one thing well.

2. The Knowledge Layer (The "Library")

What it is: A traditional database or knowledge graph.
Job: It holds the facts. It knows the refund policy, the price of the toaster, and the customer's history.
Why: Facts change often. If you update a database, the new rule is there instantly. If you try to "teach" the AI new rules by retraining it, it takes days and might forget old rules.

3. The Computation Layer (The "Calculator")

What it is: Standard computer code and logic rules.
Job: It does the math and checks the rules. "If the toaster is broken and under warranty, approve the refund."
Why: Computers are perfect at math and logic. They don't "hallucinate" (make things up) like AI sometimes does.

Why This is Better (The "Why" Behind the "What")

The paper uses some heavy math to prove three main points, which we can translate simply:

1. The "Memory" Problem

The Claim: An AI model has a limited amount of "brain space" (parameters). It cannot memorize every single rule, policy, and fact of a huge company.
The Analogy: You can't fit the entire Library of Congress into one person's head. If you try, they will forget things or mix them up.
The Fix: Keep the library (Knowledge Base) separate. The AI just looks up the book when it needs to.

2. The "Math" Problem

The Claim: AI is great at guessing the next word in a sentence, but it is bad at doing exact, step-by-step logic or complex math.
The Analogy: Asking an AI to do a long division problem is like asking a poet to build a bridge. They might write a beautiful poem about a bridge, but the bridge will collapse.
The Fix: Let the AI write the request, but let a calculator (code) build the bridge.

3. The "Blame" Problem

The Claim: If a giant AI makes a mistake, it's hard to know why. Did it forget a rule? Did it guess wrong?
The Analogy: If a giant robot crashes, you have to take the whole robot apart to find the broken gear.
The Fix: In the new system, if the "Translator" makes a mistake, you fix the translator. If the "Calculator" makes a mistake, you fix the code. You don't have to retrain the whole system.

When Should You Use This?

The paper provides a clear guide on when to use this "Specialized Team" approach versus when to just use the "Master Chef" (Big LLM).

Use the Specialized Team (SLM + Database + Code)	Use the Master Chef (Big LLM)
High Volume: You have thousands of requests a day.	Low Volume: You have a few unique, weird questions.
Strict Rules: You need 100% accuracy (e.g., banking, compliance).	Creative Work: You need brainstorming or open-ended ideas.
Changing Facts: The rules change often (e.g., new laws, new prices).	No Facts: The answer depends on opinion or creativity, not facts.
Auditing: You need to prove exactly why a decision was made.	Exploration: You are just testing ideas.

The "Offline" Secret Weapon

The paper also mentions that the big, expensive "Master Chef" (Frontier LLM) still has a job, but it's an offline job.

Instead of running the factory every second, the Master Chef sits in a quiet room.
It helps write the rules for the "Calculator."
It helps design the forms for the "Translator."
Once the rules are written, the Master Chef goes back to sleep, and the cheap, fast, specialized team runs the show.

Summary

The paper argues that for business tasks, stop trying to stuff the whole world's knowledge and logic into one giant AI brain.

Instead, build a team:

A small AI to listen and translate.
A database to hold the facts.
A calculator to do the logic.

This makes the system cheaper, faster, more accurate, and much easier to trust.

Technical Summary: Avoid Overstretching LLMs for Every Enterprise Task

Problem Statement

The paper addresses a critical misalignment between current Large Language Model (LLM) deployment strategies and the structural realities of enterprise workloads. While enterprises increasingly adopt AI for critical operational workflows (e.g., compliance, incident triage, structured diagnostics), these tasks are characterized by deterministic patterns, explicit constraints, and heavy reliance on evolving, proprietary knowledge.

The authors argue that the prevailing trend of deploying frontier LLMs as monolithic reasoning engines—or distilling them into smaller language models (SLMs) to mimic this behavior—is fundamentally flawed for enterprise settings. This approach conflates distinct computational roles (knowledge storage, algorithmic reasoning, and interface mapping) within a single parametric system. The consequences include:

Inefficiency: High computational costs for tasks that do not require full generative capacity.
Unreliability: An "irreducible error floor" where models fail to access up-to-date or proprietary knowledge not present in their training weights.
Governance Gaps: Difficulty in auditing, debugging, and updating systems where knowledge and logic are entangled in model parameters.
The Pilot-to-Production Gap: Despite widespread experimentation, few organizations achieve measurable enterprise-scale value due to these structural barriers.

Methodology and Theoretical Framework

The paper employs a theoretical and information-theoretic approach to formalize the limitations of parametric models in enterprise contexts. It does not present new empirical benchmarks but rather derives rigorous proofs regarding model capacity and task complexity.

1. Information-Theoretic Formalization

The authors define enterprise tasks where the output $Y$ depends on input $X$ and external knowledge $Z$ (from a knowledge base $K$ ), such that $Y = h(X, Z)$ .

The Mismatch: They prove that if the conditional mutual information $I(Y; Z | X) > 0$ (meaning $X$ alone is insufficient to determine $Y$ ), any purely parametric model $f_\theta(X)$ faces an irreducible Bayes risk.
The Hybrid Solution: A modular system using an extraction function $\phi$ (implemented by an SLM) and a deterministic reasoner $R$ over external knowledge $K$ computes $\hat{Y} = R(\phi(X), K)$ . The authors prove that if the extraction is sufficiently reliable, this hybrid system can strictly outperform a rationale-free distilled SLM by bypassing the irreducible error floor.

2. Computational Capacity and Expressivity

The paper introduces a capacity condition where the information complexity of a task ( $I_{task}$ ) must be less than or equal to the model's effective capacity ( $I_{model}$ ).

The Projection Bottleneck: The authors argue that standard distillation techniques (training a student SLM on teacher outputs without rationales) are "information-losing." They create a bottleneck that prevents the student from inheriting the teacher's complex algorithmic reasoning structures.
Strict Dominance: Theorem 2 demonstrates that for tasks requiring specific algorithmic procedures (e.g., exact counting, constraint satisfaction) that exceed the expressivity of a bounded SLM, a hybrid system delegating computation to symbolic reasoners strictly dominates a monolithic SLM.

3. Entropic Decomposition

The authors decompose the conditional uncertainty of a task into knowledge ( $H_{KNOW}$ ), computation ( $H_{COMP}$ ), and language interface ( $H_{LANG}$ ) components.

In monolithic LLMs, all uncertainty must be resolved implicitly within the model.
In the proposed modular architecture, retrieval and tools reduce $H_{KNOW}$ and $H_{COMP}$ , leaving the SLM to solve only the lower-entropy interface problem ( $H_{LANG}$ ). This leads to significant efficiency gains, as specialized SLMs are sufficient for the remaining interface task, avoiding the linear cost scaling of large LLMs.

Key Contributions

The paper makes four primary contributions:

A Unifying Viewpoint: It redefines the role of language models in enterprise AI. Instead of acting as the primary policy and reasoning engine, LLMs (and SLMs) should serve as bounded interface layers that map unstructured inputs to governed enterprise infrastructure. Knowledge, computation, and verification are elevated to first-class, external system components.
Theoretical Foundation for Reliability: The paper provides formal proofs demonstrating that:
- Purely parametric systems face an irreducible error floor when tasks depend on external knowledge.
- Rationale-free distillation creates a "projection bottleneck," preventing SLMs from inheriting complex algorithmic reasoning.
- Hybrid architectures (SLM extraction + external knowledge + symbolic reasoning) strictly dominate monolithic approaches on knowledge-dependent and structured tasks.
Architectural Separation: It advocates for a specific three-layer architecture:
- Interface Layer: Task-specific SLMs for structured extraction.
- Knowledge Layer: External, inspectable knowledge bases and document indices.
- Computation Layer: Deterministic algorithms, symbolic rules, and verifiers.
Decision-Theoretic Framework: The authors provide a framework for practitioners to select between monolithic LLMs and modular SLM-tool architectures based on task characteristics (e.g., volume, knowledge dependence, and the nature of uncertainty).

Results and Claims

The paper's "results" are theoretical proofs and logical deductions rather than experimental data:

Strict Dominance: The authors prove that for tasks where $I(Y; Z | X) > 0$ or where the task complexity exceeds model capacity, hybrid systems are strictly superior to rationale-free distilled SLMs.
Efficiency Gains: By externalizing knowledge and computation, the system reduces the task to a low-entropy interface problem. This allows the use of specialized SLMs, which are significantly cheaper and faster than frontier LLMs, while maintaining or improving accuracy.
Failure Mode Localization: Modular architectures localize errors to specific components (e.g., extraction errors vs. knowledge retrieval failures), making systems more diagnosable and auditable compared to monolithic "black boxes."

Significance and Position

The paper positions itself as a corrective to the current "monolithic" paradigm in enterprise AI. Its significance lies in shifting the focus from "how to make LLMs bigger or better" to "how to architect systems that respect the structural constraints of enterprise workloads."

Paradigm Inversion: The core argument is an inversion of control: runtime decision authority should shift from a monolithic LLM to explicitly governed system components.
Addressing the Value Gap: The authors claim this approach directly addresses the "pilot-to-production" gap by solving structural barriers (reliability, cost, governance) rather than just algorithmic ones.
Role of Frontier LLMs: The paper does not advocate for eliminating LLMs. Instead, it proposes using frontier LLMs offline to synthesize symbolic assets (schemas, rules, ontologies) and using them selectively for open-ended, low-volume tasks, while relying on specialized SLMs and external systems for high-volume, deterministic enterprise workflows.

In summary, the paper argues that for enterprise AI to achieve scale and reliability, language models must be treated as interfaces to formal infrastructure, not as the infrastructure itself.