💻 computer science

Security Document Classification with a Fine-Tuned Local Large Language Model: Benchmark Data and an Open-Source System

This paper introduces TorchSight, an open-source local system utilizing a fine-tuned Qwen 3.5 27B model trained on over 78,000 samples, which achieves significantly higher accuracy (95.0%) in security document classification compared to commercial cloud-based alternatives while ensuring data remains under local control.

Original authors: Ivan Dobrovolskyi

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ivan Dobrovolskyi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Cloud" vs. The "Safe Room"

Imagine you work for a bank or a hospital. You have thousands of documents, and you need to find the ones that contain secret passwords, credit card numbers, or classified military plans.

Currently, companies have two main ways to do this:

The "Rule Book" Method: They use a computer program that looks for specific patterns, like a string of numbers that looks like a credit card. It's fast, but it's dumb. If you write a story about a fake credit card number in a tutorial, the program screams "ALERT!" even though it's safe. It misses things that don't look like patterns but are still dangerous.
The "Cloud" Method: They send all their documents to a giant, smart AI in the cloud (like a super-intelligent assistant at a big tech company) to read them. This AI is very smart and understands context. But, to use it, you have to mail your secret documents out of your building to someone else's building. For a bank or the military, this is a huge risk. They can't let their secrets leave the building.

The Paper's Solution: The author built a system called TorchSight. It's like hiring a super-smart security guard who lives inside your building. This guard is smart enough to understand context (unlike the rule book) but never leaves the building (unlike the cloud service).

The "Guard" (The AI Model)

The author took a very large, pre-trained AI brain (called Qwen 3.5) and gave it a special "training camp."

The Training: They didn't just show it random documents. They fed it 78,358 examples of documents. Some were real leaks, some were fake examples made by another AI, and some were tricky "traps" (documents that look dangerous but are actually safe, or vice versa).
The Result: This created a specialized version of the AI called Beam. It learned to spot the difference between a real secret and a harmless example, something the "Rule Book" and even the "Cloud" AIs struggled with.

The Big Test: Who is the Best Guard?

The author put the new Beam guard against the best "Cloud" guards (like GPT-5, Claude, and Gemini) and the old "Rule Book" guards. They gave them all the same 1,000 documents to sort.

The Results:

The Cloud Guards: They got about 75% to 80% right. They were okay, but they made a lot of mistakes.
The Rule Book: It only got about 53% right. It was too easily confused.
The Beam Guard (Local): It got 95% right.

The "False Alarm" Problem:
The most important part of a security system is not just catching bad guys, but not screaming "Fire!" when someone is just holding a candle.

When the Cloud guards looked at safe documents (like a cooking recipe or a public news article), they often panicked. They thought 22% to 63% of safe documents were dangerous. This would make a security team crazy because they'd have to check every single safe file.
Beam only panicked on 2% of safe documents. It knew the difference between a real threat and a harmless file much better than the others.

Why Was Beam So Good?

The paper explains that the secret sauce wasn't just the AI's size, but how it was taught.

The author created a specific "rule set" (a taxonomy) with 51 different types of secrets (like "Medical Records," "Military Plans," "Passwords").
The Cloud AIs were told to "find secrets," but they made up their own categories or got confused.
Beam was fine-tuned to strictly follow the 51 categories. It learned to speak the same language as the security team. It was like teaching a dog to sit, stay, and fetch specific items, rather than just telling it "be good."

The "Outside World" Test

To make sure Beam wasn't just memorizing the practice test, the author tested it on a completely different set of 500 documents it had never seen before (like real phishing emails and medical records from public databases).

Beam still scored 93.8%.
The Cloud AIs dropped significantly, with some scoring as low as 65%.
This proves that Beam actually learned the concept of security, not just the specific answers to the practice test.

The Bottom Line

The paper shows that you don't need to send your secrets to the cloud to get a smart AI to protect them. By training a local AI on a massive, carefully curated dataset, you can get a system that is:

More Accurate: It finds more real threats.
Less Annoying: It makes far fewer false alarms on safe files.
Private: The documents never leave your computer.

The author released the whole system (the code, the training data, and the AI model) for free so anyone can use it to build their own "Safe Room" security guard.

Technical Summary: Security Document Classification with a Fine-Tuned Local Large Language Model

Problem Statement

Organizations face a critical dilemma in Data Loss Prevention (DLP): existing rule-based tools (e.g., regex, keyword lists) fail to classify sensitive content that depends on context, while cloud-based Large Language Models (LLMs) introduce data exposure risks by requiring documents to be uploaded to external infrastructure. Current solutions often suffer from high false-positive rates when distinguishing between instructional/harmless content and actual threats, or they lack the ability to operate within strict data-residency and air-gapped environments. There is a practical gap for a system that combines the contextual understanding of LLMs with the privacy guarantees of on-premise processing.

Methodology

1. Taxonomy and Dataset Construction

The study defines a hierarchical security taxonomy comprising 7 top-level categories and 51 subcategories, drawing from NIST, MITRE ATT&CK, OWASP, and compliance frameworks (GDPR, HIPAA, etc.). The categories include Malicious, Confidential, Credentials, PII, Safe, Financial, and Medical.

To train the model, a dataset of 78,358 samples was curated from 13 permissively licensed or public-domain sources (e.g., NVD, SEC EDGAR, Enron Corpus) and augmented with synthetic data.

Synthetic Augmentation: GPT-4 was used to generate 33,100 synthetic samples and 6,400 "hard-negative" samples (e.g., dangerous-looking safe documents) to test boundary cases.
Rebalancing: The dataset was rebalanced to cap subcategories at 5,000 samples, ensuring coverage across the full taxonomy.
Format: Data was formatted in the Alpaca instruction style with a fixed system prompt requiring JSON output.

2. Model Architecture and Training

The system, named TorchSight, utilizes Qwen 3.5 27B as the base model.

Fine-Tuning: The model was fine-tuned using LoRA (Low-Rank Adaptation) with rank $r=128$ and $\alpha=256$ .
Hardware: Training utilized 8× NVIDIA A100 GPUs. Inference is designed for consumer hardware, specifically running on a 32 GB Apple Silicon Mac with q4_K_M quantization (approx. 17 GB memory).
System Implementation: The scanner is written in Rust, supporting ~50 file formats. It queries a local Ollama instance via HTTP, ensuring no data leaves the local environment. Outputs are generated in JSON, SARIF 2.1.0, HTML, and PDF.

3. Evaluation Protocol

The study employed a controlled comparative design using two benchmarks:

TorchSight-Eval-1000: A stratified synthetic benchmark covering all 7 categories, including hard-negative boundary cases.
TorchSight-Eval-500-External: A held-out set of 500 real-world documents from public datasets (NVD, NIST, MTSamples, etc.) explicitly excluded from training.

Models were evaluated under identical prompting protocols (temperature = 0 for deterministic inference, except for GPT-5) against:

The fine-tuned local model (Beam).
The base Qwen 3.5 27B model (untuned).
Commercial APIs (GPT-5, Claude Sonnet/Opus 4, Gemini 2.5 Pro).
A rule-based baseline (48 compiled regex patterns).

Key Results

Performance on Primary Benchmark (Eval-1000)

Accuracy: The fine-tuned Beam q4_K_M achieved 95.0% category-level accuracy (95% CI: 93.5–96.2).
Comparison: This outperformed all tested commercial models by 15–20 percentage points (Commercial range: 75.4%–79.9%) and the base Qwen 3.5 27B by 8.7 percentage points.
False Positives: On "Safe" documents, Beam maintained a 2.0% false-positive rate. In contrast, commercial models ranged from 22.4% to 63.2%, rendering them operationally difficult for routine screening.
Subcategory Adherence: Beam achieved 48.2% subcategory accuracy, significantly outperforming the base model (19.0%) and commercial models (11.6%–23.0%), indicating that fine-tuning was crucial for enforcing the strict 51-subcategory schema.

Performance on External Validation (Eval-500)

Beam achieved 93.8% accuracy on the external held-out set, outperforming the strongest commercial baseline (Claude Sonnet 4 at 86.4%) by 7.4 percentage points.
Beam demonstrated greater stability across benchmarks (only a 1.2% drop from primary to external), whereas GPT-5 showed a significant performance drop (11.1%).

Ablation and Baseline Comparisons

Fine-Tuning Impact: The jump from 86.3% (base) to 95.0% (fine-tuned) highlights the necessity of domain-specific adaptation for schema adherence.
Regex vs. LLM: A regex-only baseline achieved only 52.7% overall accuracy, failing completely on "Confidential" and "Medical" categories (0%), whereas Beam achieved 100% and 68% respectively. This demonstrates that context-dependent classification cannot be solved by pattern matching alone.

Significance and Claims

The paper claims that TorchSight successfully bridges the gap between the high accuracy of LLMs and the privacy requirements of sensitive environments.

Local Control: The system proves that a fine-tuned local model can achieve superior accuracy to frontier commercial APIs while keeping document processing entirely within local infrastructure, eliminating data egress risks.
Operational Viability: The low false-positive rate on safe documents (2.0%) suggests the model is practically viable for DLP screening, a domain where commercial LLMs currently struggle due to over-sensitivity.
Open-Source Contribution: The study provides a complete, reproducible pipeline, including a 78k-sample dataset, model weights, and a Rust-based CLI/desktop application, addressing the lack of actively maintained open-source LLM-based DLP tools.

The authors modestly note limitations, including the use of synthetic data for categories lacking public benchmarks (credentials, financial, confidential), the use of a single prompt for commercial baselines, and the focus on English-language, single-label classification. The results are presented as evidence that domain-specific fine-tuning is a viable path for secure, on-premise security document classification.