Technical Summary: Prompt Governance? On Governing Technologies Governed by Natural Language

Problem Statement

Generative Artificial Intelligence (GenAI) systems, particularly Large Language Models (LLMs), are increasingly operated via natural language instructions (prompts) rather than formal code. Stakeholders—including developers, deployers, and end-users—utilize system-level instructions (e.g., "You are a helpful assistant" or "Avoid generating content involving real public figures") to define constraints, priorities, and compliance rules. Policymakers and regulators (e.g., in the US and EU) have begun treating these system prompts as accessible, governable artifacts, assuming they function as reliable behavioral controls.

The core problem identified is a significant misalignment between the policy assumption that system prompts are stable, interpretable control mechanisms and the technical reality of their reliability. There is a lack of systematic evidence regarding whether natural language instructions operate consistently enough across contexts and adversarial settings to support governance frameworks. Relying on prompt text as a proxy for behavioral control risks creating a "false sense of control" where governance targets legible artifacts rather than operational outcomes.

Methodology

The authors employ a mixed-method approach combining a systematic literature review with a policy analysis:

1. Systematic Literature Review (PRISMA):

   • Scope: A search of academic literature from 2022–2025 focusing on Large Language Models (LLMs) to isolate text-to-text instruction effects.
   • Process: Starting with 923 identified records, the authors screened 746 abstracts and conducted full-text reviews of 373 papers, resulting in a final corpus of 287 papers.
   • Analysis: Using inductive thematic analysis, the authors extracted claims regarding the goals of system-level instructions. These claims were coded as either "supportive" (furthering a goal) or "critical" (hindering a goal) and synthesized into a typology.

2. Policy Document Analysis:

   • Scope: A scoping search of governance documents across jurisdictions (US, EU, UK, Australia, Singapore) that explicitly address system-level instructions.
   • Case Studies: Two specific frameworks were analyzed in depth:
     • US: Executive Order 14319 ("Preventing Woke AI in the Federal Government") and the OMB Memorandum on Unbiased AI Principles.
     • EU: The General-Purpose AI Code of Practice (under the AI Act).
   • Goal: To identify implicit theories of control within these policies and compare them against the state of technical evidence.

Key Contributions

1. A Typology of System-Level Instruction Goals

The literature review reveals a fragmented landscape with divergent claims. The authors distill these into eight distinct goal categories, divided into System Goals (targeting AI behavior) and Prompt Goals (targeting the artifact itself):

• System Goals:

  • Alignment: Shaping behavior to reflect stakeholder values (e.g., safety, neutrality). Evidence is mixed; while some studies show prompts reduce unsafe outputs, others highlight "alignment faking," bias amplification, and the inability of natural language to guarantee specific value adherence.
  • Accessibility: Making control accessible via natural language. While prompts are easy to edit, the literature warns that readability does not equate to predictability, often leaving users "operating blind."
  • Adaptability: Reconfiguring models for domains or personas. Claims of high adaptability are countered by evidence of "brittle" personas and instruction forgetting in long or conflicting contexts.
  • Performance: Improving task accuracy. While specific prompts can boost performance, gains are context-dependent and often outperformed by fine-tuning; long prompts can introduce latency and degrade quality.
  • Stability: Maintaining consistent behavior. Research indicates instructions often fail to produce stable effects across multi-turn interactions or when conflicting guidelines are present.
  • Security: Protecting against or enabling attacks. Prompts are dual-use: they can serve as defense mechanisms (guardrails) but are also primary vectors for jailbreaks and extraction attacks.

• Prompt Goals:

  • Implementation: Methods for authoring and maintaining prompts (e.g., human vs. automated generation). The literature notes risks in both: automated prompts may drift, while human engineering lacks systematic improvement paths.
  • Auditability: Making instructions visible and traceable. Challenges include the lack of transferability of audit findings across models and the difficulty of reconstructing active prompt versions due to frequent iterations.

2. Policy Analysis and Misalignment

The analysis of US and EU frameworks reveals that regulators treat system prompts as legible artifacts that can be disclosed and inspected to ensure compliance.

• US Approach: Treats prompt disclosure as permissible evidence for alignment with "Unbiased AI Principles" (truth-seeking, ideological neutrality). It assumes that writing normative commitments into prompts translates into consistent behavior.
• EU Approach: Incorporates system prompts into "model specifications" for evaluators, assuming the prompt is a meaningful representation of intended behavior.
• The Gap: Both frameworks rely on assumptions of stability and control that are not consistently supported by the technical evidence. Policies often treat prompts as static specifications rather than evolving, layered mechanisms subject to supply chain conflicts and adversarial manipulation.

Results

• Fragmented Evidence: The research landscape is characterized by contradictory claims. For every study demonstrating that a prompt successfully aligns a model, another often shows the same prompt failing under different conditions or being easily bypassed.
• The "False Sense of Control": Governance approaches that rely on inspecting prompt text risk regulating what is legible rather than what is operational. The translation between human linguistic interpretation and machine processing is non-trivial and prone to stochasticity.
• Supply Chain Complexity: System instructions exist in a "prompt stack" involving foundation model providers, application developers, and end-users. Conflicts between these layers (e.g., a developer's safety constraint vs. a user's override) are often invisible to regulators, complicating accountability.
• Malleability: Prompts are frequently updated, optimized, or re-injected. Static disclosure requirements fail to capture the dynamic nature of these instructions, rendering audit trails incomplete.

Significance and Claims

The paper argues that governing GenAI through natural language instructions requires a fundamental shift in approach. The authors do not claim that prompt governance is impossible, but rather that it is currently overstated and under-evidenced.

• Cautionary Stance: Writing rules that govern machines requires different approaches than writing rules that govern humans. The accessibility of natural language does not guarantee its reliability as a control mechanism.
• Implications for Governance:
  • Beyond Textual Compliance: Governance must move beyond mere disclosure of prompt text. It requires linking disclosed artifacts to behavioral evidence and evaluating the "prompt stack" as a whole.
  • Socio-Technical Considerations: Governance must account for the layered, multi-stakeholder nature of AI supply chains and the malleability of instructions.
  • Avoiding Function Creep: There is a risk of expanding prompt-based governance to tasks for which there is no evidence of reliability (e.g., complex value alignment or censorship).
• Conclusion: The paper concludes that while system-level instructions are a pervasive intervention in the AI ecosystem, treating them as stable, interpretable control mechanisms without robust behavioral evidence risks creating a compliance illusion. Effective governance must prioritize actual behavioral outcomes over easily produced textual artifacts.