💬 NLP

Making Prompts First-Class Citizens for Adaptive LLM Pipelines

This paper introduces SPEAR, a framework that treats prompts as first-class citizens to enable structured management, dynamic runtime refinement based on feedback, and policy-driven control, thereby overcoming the limitations of brittle, opaque prompts in modern LLM pipelines.

Original authors: Ugur Cetintemel, Shu Chen, Alexander W. Lee, Deepti Raghavan, Duo Lu, Andrew Crotty

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ugur Cetintemel, Shu Chen, Alexander W. Lee, Deepti Raghavan, Duo Lu, Andrew Crotty

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef running a very busy, high-tech kitchen. In this kitchen, the "chefs" are Artificial Intelligence (AI) models, and the "recipes" they follow are called prompts.

Right now, most AI kitchens operate with a broken system. The recipes are written on sticky notes that are taped to the wall. They are messy, hard to read, and if the food tastes bad, the chef has to stop, peel the sticky note off, rewrite it by hand, and tape it back up. Worse, the kitchen manager (the computer program) can't see the recipe; it just sees a black box. It doesn't know why the chef made a mistake, so it can't fix the recipe automatically.

This paper introduces a new system called SPEAR (Structured Prompt Execution and Adaptive Refinement). It wants to turn those messy sticky notes into living, breathing digital cookbooks that the kitchen manager can read, edit, and optimize in real-time.

Here is how SPEAR works, broken down into three simple ideas:

1. Prompts as "Living Recipes" (Structured Management)

The Old Way: Prompts are just long strings of text. If you want to change a recipe for "soup," you have to rewrite the whole thing. You can't easily reuse the "chopping vegetables" part of the soup recipe for the "salad" recipe.
The SPEAR Way: SPEAR treats prompts like modular LEGO blocks.

Instead of one giant recipe, you build a "Soup View" that says: Take the "Chopping Block," add the "Boiling Block," and then add the "Seasoning Block."
If you want to change the seasoning, you just swap out the "Seasoning Block." You don't have to rebuild the whole soup.
Because these blocks are saved in a library, you can see exactly how the recipe changed over time (version control), just like tracking changes in a Google Doc.

2. The Recipe That Fixes Itself (Adaptive Refinement)

The Old Way: If the AI chef burns the soup, the human has to go in, stop the process, and manually rewrite the recipe to say, "Don't burn the soup next time." This is slow and happens after the mistake.
The SPEAR Way: The recipe is alive. It can watch itself while it cooks.

If the AI chef starts to burn the soup, the recipe can instantly say, "Oh, the heat is too high! Let me add a step to lower the temperature."
It can also say, "I'm not sure about this ingredient; let me ask a senior chef (a bigger AI model) for help."
This happens while the work is being done, not after. The system adapts on the fly.

3. The "If-Then" Rulebook (Policy-Driven Control)

The Old Way: The kitchen manager has no rules. It just hopes the chef does a good job.
The SPEAR Way: You can write a simple rulebook for the kitchen manager:

Rule: IF the customer is a VIP (Input Condition), THEN use the "Premium Recipe."
Rule: IF the chef takes too long to cook (Output Condition), THEN switch to a faster, simpler recipe.
Rule: IF the chef seems confused (Runtime Feedback), THEN ask for a second opinion.

This rulebook allows the system to make smart decisions automatically without a human needing to press a button every time something changes.

Why Does This Matter? (The Results)

The authors tested this system using a medical example: asking an AI to summarize a patient's medication history.

Without SPEAR: The AI often forgot details or gave messy answers. Fixing it required hours of manual tweaking.
With SPEAR: The system realized, "Hey, this question is easy; let's use a small, fast, cheap AI." But if the question was hard, it said, "This is tricky; let's switch to a big, smart, expensive AI."
The Result: They got the same high-quality answers as the expensive AI, but for half the cost and much faster.

The Big Picture

Think of SPEAR as upgrading from a paper map to a GPS navigation system.

Paper Map (Current AI): Static. If you take a wrong turn, you have to stop, unfold the map, and figure out a new route manually.
GPS (SPEAR): Dynamic. It knows where you are, sees traffic ahead, and instantly reroutes you to the best path, all while you are driving.

By making prompts "first-class citizens" (meaning they are treated as important, manageable data rather than just text), SPEAR makes AI pipelines smarter, cheaper, and much easier for humans to manage.

1. Problem Statement

Modern Large Language Model (LLM) pipelines have evolved into complex, data-centric applications involving data retrieval, error correction, tool calling, and multi-agent coordination. However, the central control element—the prompt—remains a significant bottleneck:

Brittleness and Opacity: Prompts are typically treated as static, opaque strings disconnected from the surrounding program logic.
Lack of Systematic Management: They are often crafted via manual trial-and-error, lacking systematic versioning, tracking, or provenance.
Static Nature: Existing frameworks (e.g., LangChain, DSPy) largely treat prompts as static templates tuned offline. They lack the ability to dynamically adapt prompt logic based on real-time execution feedback (e.g., latency, token budget, or result confidence).
Optimization Limits: Because prompts are "black boxes" to the execution engine, opportunities for runtime optimization, reuse, and adaptive refinement are severely limited.

2. Methodology: SPEAR

The authors propose SPEAR (Structured Prompt Execution and Adaptive Refinement), a system that treats prompts as first-class citizens within the execution model. SPEAR elevates prompts from simple strings to structured data objects, integrating them directly into the system state.

Core Architecture

SPEAR operates on a system state defined by three components:

Prompt Views ( $P$ ): Named, versioned logical views of prompts (similar to SQL views) that encapsulate reusable program logic.
Context Store ( $C$ ): Runtime data including inputs, outputs, and intermediate results.
Runtime Metadata ( $M$ ): Control signals and diagnostics (e.g., token usage, latency, confidence scores).

Key Mechanisms

Structured Prompt Management:
- Prompts are defined as views that can be parameterized, composed, and versioned.
- A Prompt Algebra is introduced, where operators consume and produce the $(P, C, M)$ triple.
- Core operators include GEN (generate LLM output), RET (retrieve external data), and SWITCH (conditional logic).
Adaptive Prompt Refinement:
- Prompts are not static; they can be modified dynamically during execution.
- Refinements are operators that update the set of prompt views (e.g., appending examples, rewriting instructions).
- Refinement Modes:
  - Manual: Developer explicitly edits prompt text.
  - Assisted: Developer provides high-level intent; the system compiles a refinement operator.
  - Automatic: The system monitors runtime state and triggers refinements based on conditions.
Policy-Driven Control:
- SPEAR introduces When-Then policies (WHEN condition THEN refinement).
- Input Conditions: Trigger refinements before execution (e.g., selecting a specific prompt view based on user input).
- Output Conditions: Trigger refinements after execution based on feedback (e.g., retrying with a different prompt if confidence is low).
- Policies themselves can be prompts, allowing the control plane to be self-improving.

3. Key Contributions

Structured Prompt Management: A paradigm shift treating prompts as versioned, composable data objects with an executable algebra, enabling modularity and introspection.
Adaptive Prompt Refinement: A mechanism for prompts to evolve dynamically at runtime based on execution signals, bridging the gap between offline optimization and real-time adaptivity.
Policy-Driven Control: A declarative framework for specifying automatic refinement logic, unifying static planning with runtime adaptivity.
Optimization Opportunities: The authors identify new optimization strategies unlocked by this model, including:
- Prefix Caching & Reuse: Caching stable prompt segments to reduce latency.
- Input/Output Batching: Processing multiple inputs or similar prompts in single LLM calls.
- Operator Fusion: Merging tightly coupled operators to reduce intermediate storage and overhead.

4. Experimental Results

The authors evaluated SPEAR using the EHRNoteQA benchmark (clinical note question answering) across four Google Gemma 3 models (1B, 4B, 12B, 27B).

Comparison Baselines:

Baseline: Minimal prompt with basic instructions.
DSPy: Automated offline optimization.
DSPy-CoT: DSPy with Chain-of-Thought.

SPEAR Strategies Tested:

R1-R3 (Static/Assisted): Manual and assisted refinements focusing on output formatting and prompt structure (e.g., reordering clinical evidence before questions).
R4 (Automatic Input): Routing questions to appropriate model sizes based on difficulty.
R5 (Automatic Output): Cascading to larger models only when the smaller model signals uncertainty.

Key Findings:

Performance vs. DSPy: SPEAR refinements consistently outperformed DSPy and DSPy-CoT. While DSPy struggled with strict output constraints in long contexts (leading to parsing errors), SPEAR's transparent, structured approach allowed for targeted fixes.
Impact of Structure: The most significant performance leap came from R3 (reordering prompt components), demonstrating that explicit, interpretable structural changes are highly effective.
Efficiency of Automation:
- R4 (Routing): Achieved 83.5% accuracy using the 1B model as a router, matching the standalone 12B model's performance but with lower cost.
- R5 (Cascading): A 12B $\to$ 27B cascade achieved 86.7% accuracy (comparable to standalone 27B at 87.3%) at nearly half the cost ($0.046 vs. $0.087).
Conclusion: SPEAR enables complex optimization strategies (like dynamic routing and cascading) to be implemented simply and modularly, significantly improving the cost-accuracy trade-off.

5. Significance

Paradigm Shift: SPEAR moves prompt engineering from a manual, static craft to a dynamic, data-driven engineering discipline.
Complementarity: It complements existing offline optimization tools (like DSPy) by addressing the critical dimension of runtime adaptivity.
Developer Productivity: By treating prompts as structured views with versioning and algebra, SPEAR improves code modularity, reusability, and maintainability in LLM applications.
Future Potential: The framework lays the groundwork for self-improving systems where the logic governing prompt refinement can itself be optimized based on accumulated runtime data, potentially leading to fully autonomous, adaptive LLM pipelines.