LLM-enabled Applications Require System-Level Threat Monitoring

1. Problem Statement

The rapid integration of Large Language Models (LLMs) as core reasoning components in software systems (e.g., AI agents) has fundamentally altered the software ecosystem. While these systems offer advanced capabilities, they introduce unique reliability and security challenges that traditional software engineering principles cannot fully address:

• Non-determinism & Unverifiability: LLMs are statistical, learning-driven, and non-deterministic, making formal verification and static analysis insufficient for guaranteeing safety.
• Expanded Attack Surface: The interaction between LLMs, external tools (via Model Context Protocol - MCP), retrieval systems (RAG), and user interfaces creates complex, multi-stage workflows. This expands the attack surface beyond traditional software vulnerabilities.
• Limitations of Current Defenses: Existing defenses like Red Teaming (episodic testing), Guardrails (localized input/output filtering), and Model Alignment (training-time safety) are insufficient. They often fail to detect stealthy, distributed, or emergent threats that manifest only during runtime across multiple interaction stages.
• The Core Gap: There is a lack of System-Level Threat Monitoring (analogous to Endpoint Detection and Response - EDR in traditional IT) that treats security failures as expected operational conditions rather than exceptional events requiring post-hoc forensic analysis.

2. Methodology

The authors propose a systematic, taxonomy-grounded monitoring framework designed for the entire lifecycle of LLM-enabled applications. The methodology involves:

• Workflow Decomposition: The paper defines a representative 8-stage workflow for LLM applications (from user prompt submission to final response delivery), identifying specific points where threats can manifest (e.g., prompt injection at Stage 1, tool execution at Stage 5).
• Threat Taxonomy: The authors categorize 14 distinct threat categories relevant to LLM applications, ranging from Prompt Injection and Adversarial Inputs to Model Drift and Data Leakage.
• Artifact-Based Monitoring: For each threat category, the paper identifies specific monitoring artifacts (logs, metrics, hashes, embeddings) that can be captured at different workflow stages to detect anomalies.
• Unified Audit Logging Pipeline: The core methodology is a unified audit logging system that aggregates telemetry across the entire execution path. It moves beyond simple logging to context-aware correlation, mapping attack vectors to specific monitoring artifacts to enable detection, forensic analysis, and incident response.

3. Key Contributions

A. Comprehensive Threat Taxonomy & Monitoring Schema

The paper provides a detailed mapping of 14 threat categories to specific attack vectors and monitoring artifacts. Key examples include:

• Prompt Injection: Monitoring raw prompt text for suspicious imperative verbs, checking RAG document provenance, and validating service API output schemas to detect "instruction flooding."
• Adversarial Inputs: Using Unicode normalization diffs to detect invisible characters, embedding outlier scores to identify semantic drift, and cross-modal consistency checks for multimodal attacks.
• Response Manipulation: Tracking "safety scores" across conversation turns, monitoring memory mutation logs for poisoned RAG data, and analyzing feedback loops for "reward gaming."
• DoS & Resource Exhaustion: Monitoring token burn-down rates, tool-call fan-out ratios, and cache hit-rate degradation to detect recursive loops or cache-bypass attacks.
• Live Data/Model Poisoning: Validating model update signatures, monitoring index update diffs for anomalies, and tracking telemetry integrity to prevent self-correction loops from being hijacked.
• Data Leakage & Cross-Context Disclosure: Enforcing strict access control logs for RAG, monitoring cache key entropy to prevent tenant collisions, and scanning logs for PII leakage.
• Model Theft & Drift: Detecting "latent space walking" (systematic probing for model extraction), monitoring side-channel latency jitter, and tracking input distribution shifts (KL divergence) to detect model drift.

B. The Audit Logging Framework

The paper outlines a multi-stage audit logging pipeline that operates as a defensive filter:

1. Ingestion: Capturing raw inputs, tool outputs, and intermediate states.
2. Analysis: Applying deterministic pre-filters (e.g., schema validation) and selective LLM-based analysis for semantic anomalies.
3. Correlation: Linking events across stages (e.g., connecting a specific user prompt to a subsequent tool execution and final output) to trace the root cause of an incident.
4. Response: Enabling automated containment (e.g., blocking a specific tool, rolling back a model version) based on severity.

C. Post-Monitoring Incident Response

Drawing from traditional EDR practices, the paper argues for a complete incident response lifecycle:

• Root Cause Analysis: Reconstructing the temporal progression of an attack.
• Severity Prioritization: Ranking alerts based on potential impact.
• Recovery: Specific to LLMs, this includes instructional tuning, prompt sanitization, and retraining, rather than just patching code.

4. Results and Analysis

While this is a position paper and does not present empirical experimental results (e.g., detection rates on a specific dataset), it provides:

• A Structured Analysis: A rigorous breakdown of how 14 major threat categories manifest in modern agent workflows.
• Feasibility Assessment: The paper demonstrates that the required monitoring artifacts (e.g., embedding distances, token counts, cache hit rates) are technically feasible to collect using existing infrastructure.
• Gap Identification: It highlights that current defenses (Red Teaming, Guardrails) are reactive or localized, whereas the proposed monitoring is continuous and system-wide.

5. Significance and Impact

• Paradigm Shift: The paper advocates for a fundamental shift in how LLM applications are secured: moving from a "prevention-only" mindset (testing/guardrails) to an "assume breach" operational mindset where continuous monitoring and incident response are mandatory.
• Standardization: It calls for standardized observability interfaces from cloud providers to enable black-box compatible monitoring, addressing the "limited observability" challenge in closed-source models.
• Foundation for Future Research: The proposed taxonomy and artifact list serve as a blueprint for researchers and engineers to build specialized EDR tools for AI agents.
• Trustworthy Deployment: By treating security anomalies as expected operational conditions, the framework aims to make LLM-enabled applications viable for high-stakes domains (finance, healthcare, legal) where reliability is non-negotiable.

6. Challenges Identified

The paper acknowledges significant hurdles in implementation:

• Semantic Ambiguity: Distinguishing between complex legitimate instructions and malicious overrides remains difficult, creating a trade-off between false positives and false negatives.
• Latency: Deep context inspection using LLMs for monitoring can introduce inference latency, requiring tiered strategies (lightweight filters + selective deep analysis).
• Observability: Closed-source models limit the ability to audit internal states, necessitating new protocols for controlled observability without exposing proprietary weights.

Conclusion

The paper concludes that system-level threat monitoring is a prerequisite for the reliable deployment of LLM-enabled applications. It argues that without a dedicated, workflow-aware monitoring framework capable of detecting and contextualizing anomalies across the entire execution lifecycle, the inherent non-determinism and expanded attack surface of AI agents will remain a critical barrier to their safe adoption.