📄 other

Symbolic Log Shield - an adversarial resistant neurosymbolic framework for network log classification

This paper introduces Symbolic Log Shield, a neurosymbolic framework that significantly mitigates the vulnerability of Large Language Models to adversarial jailbreak attacks in network log classification, effectively restoring and even enhancing their anomaly detection performance from severely degraded levels.

Original authors: Franciszek Górski, Mateusz Krzysztoń, Max Landauer

Published 2026-07-24

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Franciszek Górski, Mateusz Krzysztoń, Max Landauer

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the digital world as a massive, bustling city where every computer, server, and router is a citizen leaving a trail of footprints. These footprints are called "logs." They tell the story of what happened: who entered a building, what they touched, and if they tried to pick a lock. For years, security teams have used smart computer programs to read these logs and spot the bad guys. Recently, we've started using a new kind of super-smart computer brain called a "Large Language Model" (LLM). Think of an LLM as a detective who has read every book in the library; it's incredibly good at understanding the story behind the footprints and figuring out if a suspicious activity is a real attack or just a clumsy janitor.

But here's the catch: just like a human detective can be tricked by a clever liar, these AI detectives can be fooled. A "jailbreak" attack is like a criminal whispering a secret code or telling a fake story to the detective, convincing them that the crime scene is actually a harmless birthday party. If the detective gets tricked, the alarm never rings, and the city is left vulnerable. This paper dives into exactly how easily these AI detectives can be fooled and, more importantly, builds a new kind of shield to protect them.

The Paper: Building a "Log Shield" for AI Detectives

In this study, the researchers set out to test how fragile these AI log detectives really are. They took several different AI models—ranging from small, nimble ones with 2 billion "brain cells" (parameters) to massive, heavyweights with 128 billion—and put them to work analyzing network logs. Their first job was to see if they could spot real cyber attacks. When the logs were clean and honest, the big models did a great job, with the largest one (Mistral-Medium-3.5-128B) getting an accuracy score (AUROC) of about 84.54%. The smaller models were a bit more confused, with the tiniest one (Qwen3.5-2B) scoring around 60.00%.

Then, the researchers played the role of the villain. They crafted a special "adversarial dataset" using 7 different types of jailbreak tricks. These weren't just random typos; they were sophisticated psychological manipulations. Some tricks involved wrapping the log data in a fake story (like a "DeepInception" where the AI is asked to imagine a sci-fi scene where the attack is actually normal maintenance). Others involved hiding the log text inside secret codes (like Caesar ciphers or Base64) or speaking to the AI in a language it wasn't fully trained on (like Swahili or Zulu) to bypass its safety filters.

The result? The AI detectives were completely duped. The attacks were devastatingly effective. The massive Mistral model's accuracy plummeted from 84.54% down to 48.83%. The smaller models fared even worse, with some dropping below 30%. The researchers found that simply making the AI bigger didn't make it safer; even the giant 128-billion-parameter model was just as vulnerable to these clever tricks as the smaller ones. The attacks successfully convinced the AI that dangerous attacks were actually "Certain: Normal" or "Almost Certain: Normal," effectively blinding the security system.

The Solution: The "Symbolic Log Shield"

Realizing that the AI alone wasn't enough, the team built a new framework they call Symbolic Log Shield. Imagine this as a two-step security checkpoint before the AI detective ever sees the evidence.

Step 1: The Pre-Processing "Sanitizer" (The Pre-Check)
Before the log reaches the AI, it goes through a strict, rule-based filter. This isn't a smart brain; it's a rigid, unyielding robot that knows exactly what a real log looks like.

It strips out invisible characters (like zero-width spaces) that hackers use to hide their tricks.
It fixes weird letter swaps (like turning "chMod" back into "chmod").
It re-organizes the logs so they are in the correct time order, undoing any attempts to scramble the timeline.
It translates secret codes back into plain text.
This layer acts like a translator who speaks "Hacker" and "Normal" fluently, ensuring that by the time the log reaches the AI, it's clean and structured.

Step 2: The Post-Processing "Reasoner" (The Reality Check)
After the AI makes its guess, the log goes to a second layer: a symbolic rule engine. This is like a senior supervisor who double-checks the detective's work using a strict rulebook (based on the MITRE ATT&CK framework).

If the AI says "This is normal," but the supervisor sees that the log was heavily scrambled or encoded, the supervisor overrides the AI and says, "Wait, that's suspicious!"
If the AI says "This is an attack," but the supervisor sees no actual network scanning or weird IP addresses, it might say, "Maybe you're just being paranoid."
It looks for specific patterns, like a huge number of domain names being checked in a short time (a sign of a hacker mapping out a network), which the AI might have missed.

The Results: The Shield Works Wonders

When the researchers tested the AI models with this new Symbolic Log Shield in place, the results were dramatic. The shield didn't just patch the holes; it almost completely restored the AI's vision.

Recovery: The massive Mistral model, which had been knocked down to 48.83% by the attacks, bounced back to 82.10% with the shield. That's almost as good as it was with clean data!
Improvement for Small Models: The small models saw even bigger gains. The Qwen3.5-2B model, which was struggling at 40.41% under attack, jumped to 77.59%. The Llama-3.2-3B model went from a terrible 26.49% to a solid 72.86%.
Beating the Original: In some cases, the shield actually made the models better than they were on clean data. For example, the Qwen3-14B model scored 81.00% with the shield, which was higher than its original score of 75.41%.

The researchers also measured how "confused" the models were. Without the shield, the AI models would often give up and say "Neutral" (meaning "I don't know") or guess wildly. With the shield, the models became much more confident and accurate, reducing their "ignorance" significantly.

What This Means

The paper concludes that while large AI models are powerful, they are surprisingly fragile when faced with clever, targeted tricks. Relying on the AI alone is risky because even the biggest models can be tricked into ignoring real attacks. However, by wrapping these AI models in a "neurosymbolic" framework—combining the AI's pattern recognition with a rigid, rule-based shield—the system becomes robust again.

The authors suggest that this approach is a crucial step forward. It proves that we don't need to wait for AI to become magically immune to attacks; instead, we can build a protective layer around it today. The Symbolic Log Shield acts as a guardian, cleaning up the noise and checking the logic, ensuring that the AI detective sees the truth, even when the criminals are trying to tell a lie. While the study focused on specific types of jailbreaks and log data, it offers a promising blueprint for making our digital cities safer, one log entry at a time.

Technical Summary: Symbolic Log Shield

Problem Statement

Large Language Models (LLMs) are increasingly deployed in Security Operations Centers (SOCs) to analyze network and system logs for anomaly detection and attack classification. While effective, these systems face a critical vulnerability: they are susceptible to adversarial attacks, specifically "jailbreak" techniques designed to manipulate the model's output. Existing research has demonstrated that prompt injection and jailbreak attacks can force LLMs to violate safety guidelines or generate prohibited content, but few studies have addressed their impact on the specific task of network log classification.

The core problem is that adversarial perturbations—such as subtle modifications to log entries or the injection of crafted context—can drastically reduce an LLM's ability to distinguish between actual system attacks and normal activity. This leads to a degradation in anomaly detection capabilities, potentially allowing cybercriminals to evade detection. Furthermore, purely neural approaches lack interpretability and formal guarantees, making it difficult to articulate why a specific log sequence is malicious.

Methodology

1. Adversarial Threat Model and Dataset Generation

The authors define a black-box threat model where an attacker aims to downgrade the severity classification of a malicious log trace (e.g., from "Certain: Attack" to "Certain: Normal") without altering the underlying malicious functionality. The attacker possesses limited knowledge of the target system but can modify log content via injection.

To evaluate this, the authors constructed a specialized adversarial dataset using a surrogate model approach (Llama3.1-8B) to generate attacks against target models, simulating a realistic scenario where attackers do not have direct access to the target's parameters. The dataset includes 7 distinct types of Jailbreak attacks, adapted from existing literature to target log classification:

In-Context Attack (ICA): Injecting "benign" few-shot examples to calibrate the model's priors toward normalcy.
Rewriting Nesting LLM (ReNeLLM): Rewriting prompt framing and nesting logs within benign scenarios (e.g., SRE runbooks).
DeepInception: Embedding logs in recursive, nested fictional narratives to bypass safety guardrails.
Cipher: Encoding log content (Caesar, ASCII) to obscure attack-indicative tokens from the tokenizer.
Jailbroken: Applying random character-level perturbations (leetspeak, Base64) to degrade pattern recognition.
Multilingual: Translating system instructions into low-resource languages to exploit gaps in safety alignment.
GPTFuzzer: Automatically mutating prompt templates (e.g., policy overrides, false-positive audits) to find vulnerabilities.

2. The Symbolic Log Shield Framework

The paper proposes Symbolic Log Shield, a neurosymbolic architecture designed to wrap the LLM in deterministic symbolic layers for pre- and post-processing. This "sandwich" approach aims to neutralize perturbations before inference and validate outputs afterward.

Symbolic Preprocessing (Sanitization):
- Text Sanitizer: Neutralizes character-level perturbations using Unicode normalization (NFKC), homoglyph mapping, zero-width character stripping, case normalization for known commands, and character deduplication. It outputs a sanitized log and a Perturbation Score ( $S_1$ ).
- Structural Reconstructor: Reconstructs the log structure destroyed by attacks. It performs temporal sorting, noise filtering (identifying injected benign logs via entropy and process whitelists), and domain pattern analysis (detecting DNS enumeration patterns regardless of subdomain names). It outputs a structured Log AST and an IP graph.
Neural Core:
- The LLM Classifier processes the cleaned, structured data. It outputs a vector of MITRE ATT&CK techniques and a confidence score (ranging from "Certain: Attack" to "Certain: Normal").
Symbolic Postprocessing (Defense):
- Symbolic Rule Engine: A rule-based system that adjusts the LLM's confidence score based on two inputs: the perturbation score ( $S_1$ ) and a Jailbreak Score ( $S_{jb}$ ) derived from semantic analysis of the prompt.
- Rules 1–4: Address network-level and syntactic indicators (e.g., DNS enumeration, IP dispersion, log-to-TTP mapping) to boost confidence toward "Attack" if perturbations are detected but the model predicts "Normal."
- Rules 5–9: Countermeasure rules specifically targeting jailbreak strategies (e.g., detecting encoding artifacts, narrative framing, few-shot injection, and non-log content inflation). These rules shift confidence toward "Attack" if manipulation is detected.
- Rule 10: A "Smart False-Positive Guard" that only shifts confidence toward "Normal" if all adversarial signals (syntactic and prompt-level) are zero, preventing the suppression of legitimate attack detections.
- Constraint Checker: Validates the logical consistency of the detected techniques against the MITRE ATT&CK ontology and the log source.

Key Results

The authors evaluated six LLMs ranging from 2B to 128B parameters (including Qwen, Llama, Mistral, and Ministral models) across four conditions: Original, Adversarial, Sanitized, and Defended.

Vulnerability of LLMs: Adversarial attacks caused a drastic reduction in performance. AUROC scores for all models dropped significantly, falling from a range of 46–84% (Original) to 26–51% (Adversarial). The largest models (e.g., Mistral-128B) suffered the most severe absolute drops (nearly 36 percentage points).
Effectiveness of Symbolic Log Shield:
- Sanitized Phase: Applying only the preprocessing layers improved AUROC by 11–24 percentage points, demonstrating the value of neutralizing character-level perturbations.
- Defended Phase: The full neurosymbolic framework (pre- and post-processing) nearly recovered the original effectiveness for most models, achieving AUROC scores between 72% and 82%.
- Improvement over Original: For smaller models (e.g., Qwen3.5-2B and Llama-3.2-3B), the defended architecture actually surpassed their original performance, improving AUROC by 18 and 26 percentage points respectively. This suggests the symbolic layers helped correct inherent biases (e.g., excessive "Neutral" responses) in smaller models.
Robustness: The framework successfully mitigated the negative impact of all 7 jailbreak strategies, restoring the models' ability to distinguish between attacks and normal activity.

Significance and Claims

The paper claims to be one of the first works to:

Systematically evaluate the vulnerability of LLMs of varying sizes to adversarial jailbreak attacks specifically in the context of network log classification.
Demonstrate that model size does not guarantee robustness; even large models (128B) are highly susceptible to these attacks without defense.
Propose and validate Symbolic Log Shield, a neurosymbolic framework that leverages the complementary strengths of neural pattern recognition and symbolic reasoning to achieve adversarially resilient intrusion detection.

The authors conclude that while LLMs are powerful for log analysis, their deployment in security-critical environments requires hybrid architectures. The proposed neurosymbolic approach not only defends against adversarial manipulation but also enhances the interpretability and reliability of the detection process, making it a viable solution for robust SOC operations. The work acknowledges limitations regarding generalization to other attack types and dependence on system prompts, framing this as a foundational step for future research in adversarially robust log analysis.