🤖 AI

Towards Small Language Models for Security Query Generation in SOC Workflows

This paper proposes a cost-effective, three-knob framework leveraging Small Language Models with error-aware prompting, rationale distillation, and a two-stage refinement architecture to achieve high-accuracy natural-language-to-KQL translation for Security Operations Centers, significantly reducing token costs compared to large language models while maintaining robust performance across syntax and semantic metrics.

Original authors: Saleha Muzammil, Rahul Reddy, Vishal Kamalakrishnan, Hadi Ahmadi, Wajih Ul Hassan

Published 2026-02-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Saleha Muzammil, Rahul Reddy, Vishal Kamalakrishnan, Hadi Ahmadi, Wajih Ul Hassan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective in a massive, high-tech police station (a Security Operations Center, or SOC). Every day, millions of digital "witness statements" (logs) pour in from cameras, computers, and networks. To find a specific criminal, you need to ask the database a very specific question using a secret, complex code called KQL.

The problem? Writing this code is like speaking a foreign language that only a few experts know. If you ask the wrong question, you get no answers. If you ask a slightly wrong question, you might accidentally lock yourself out or get the wrong suspect.

This paper asks a simple question: Can we teach a small, cheap, and fast computer brain (a "Small Language Model" or SLM) to translate our plain English questions into this secret code, so regular detectives can solve crimes faster?

Here is the breakdown of their solution, using some everyday analogies:

1. The Problem: The "Expert" Bottleneck

Currently, you need a senior code-wizard to translate your question ("Show me all computers that connected to that suspicious IP address yesterday") into KQL.

The Old Way: You hire a super-expensive, slow, and energy-hungry "Super-Brain" (a Large Language Model or LLM) to do the translation. It's great, but it costs a fortune and takes time.
The New Idea: Can we use a "Junior Detective" (a Small Language Model)? They are fast, cheap, and can run on a regular laptop, but they aren't very smart on their own. They often hallucinate (make things up) or get the grammar wrong.

2. The Solution: A Three-Step Training Camp

The authors didn't just throw the Junior Detective at the problem. They built a three-step training camp to turn them into a pro.

Step A: The "Cheat Sheet" (Prompting)

Imagine giving the Junior Detective a cheat sheet that says, "Hey, don't forget to put a semicolon at the end, and don't mix up the table names!"

What they did: They figured out the most common mistakes these small models make (like missing parentheses or wrong table names) and added specific "error-aware" instructions to the prompt.
The Result: The Junior Detective stopped making silly grammar mistakes without needing more brainpower.

Step B: The "Mentorship" (Fine-Tuning with Distillation)

Imagine the Junior Detective is studying for a test. Instead of just memorizing the answer key, a Mentor (a powerful LLM) explains how they solved the problem.

What they did: They used a powerful AI to generate thousands of practice questions and answers, but they added a "thought process" explanation for each one. They then taught the small model to mimic this reasoning.
The Result: The small model learned how to think, not just what to output.

Step C: The "Two-Stage Assembly Line" (The Architecture)

This is the paper's biggest innovation. Imagine a factory line:

Stage 1 (The Fast Worker): The cheap, fast Junior Detective (SLM) quickly drafts 5 different answers. It's fast and cheap, but maybe a bit sloppy.
Stage 2 (The Quality Inspector): A slightly smarter, but still affordable, "Inspector" (a small LLM) looks at those 5 drafts. It picks the best one, fixes any remaining errors, and checks it against the rulebook (the database schema) to make sure it makes sense.

The Result: You get the speed and low cost of the Junior Detective, but the high accuracy of the Quality Inspector.

3. The Results: Fast, Cheap, and Accurate

They tested this system against the "Super-Brains" (like GPT-5).

Accuracy: The new system was almost as good as the expensive Super-Brains. It got the grammar right 98.7% of the time and the meaning right 90.6% of the time.
Cost: This is the big win. The new system cost 10 times less than the expensive Super-Brains.
Speed: It was much faster, which is crucial when you are trying to stop a cyber-attack in real-time.

The Big Picture

Think of this like cooking.

The Old Way: You hire a world-famous, Michelin-star chef (LLM) to cook every single meal for your restaurant. It tastes amazing, but it's incredibly expensive and slow.
The New Way: You hire a fast, local cook (SLM) to prep the ingredients and cook the basics. Then, you have a head chef (the Oracle) who just tastes the dish, adds a pinch of salt, and fixes the plating.
The Outcome: The food tastes just as good, but you can feed 10 times more people for the same price.

In summary: This paper proves that we don't need to rely on expensive, slow AI giants to handle security tasks. By combining a fast, small AI with a smart "quality check" step, we can make cybersecurity faster, cheaper, and accessible to everyone.

1. Problem Statement

Security Operations Centers (SOCs) face an overwhelming volume of telemetry data (logs, network traffic, alerts), requiring analysts to write complex queries in Kusto Query Language (KQL) to investigate threats.

The Bottleneck: Writing correct KQL requires specialized expertise. Translating natural language (NL) intent into accurate KQL over large, evolving schemas is difficult for non-experts and creates a scalability bottleneck.
The Limitation of Current Solutions: While Large Language Models (LLMs) can automate this translation, they suffer from high latency, significant memory requirements, and prohibitive operational costs (token usage), making them difficult to deploy at scale under enterprise data-governance constraints.
The Gap: There has been no systematic study on whether Small Language Models (SLMs) (defined as models with $\le$ 10 billion parameters) can effectively handle NL-to-KQL translation in realistic, schema-rich security environments.

2. Methodology

The authors propose a framework to enhance SLM performance using three orthogonal "knobs": Prompting, Fine-Tuning, and Architecture Design.

A. Prompting Enhancements

Adaptation of NL2KQL: The authors re-implemented the state-of-the-art NL2KQL framework but replaced heavy components with lightweight alternatives (e.g., using Google's text-embedding-004 instead of ada-002 for embeddings).
Error-Aware Prompting: Instead of generic prompts, they introduced specific instructions targeting common KQL parser failures (e.g., missing delimiters, malformed parentheses, incorrect operator usage). This improves syntactic correctness without increasing token count.
Schema Context: The system retrieves relevant tables and columns based on cosine similarity between the NL query and the schema, providing the SLM with necessary context to avoid hallucinations.

B. Fine-Tuning with Rationale Distillation

LoRA (Low-Rank Adaptation): The authors applied Parameter-Efficient Fine-Tuning (PEFT) using LoRA to a DeepSeek Coder 6.7B Instruct model. This freezes original weights and trains only low-rank adapter matrices, keeping the model compact.
Rationale Distillation: To transfer reasoning capabilities from a "teacher" LLM (Gemini 2.0 Flash) to the SLM, they augmented the training dataset. Each Natural Language Query (NLQ)–KQL pair was accompanied by a short Chain-of-Thought (CoT) explanation generated by the teacher. The SLM was trained to output both the reasoning and the final query, allowing it to learn intermediate logical steps.

C. Two-Stage SLM-Oracle Architecture

This is the core architectural contribution. It divides labor between efficiency and correctness:

Stage 1 (Generation): An SLM (DeepSeek Coder 6.7B) efficiently generates candidate KQL queries based on the NLQ and retrieved schema context.
Stage 2 (Refinement/Judgment): A low-cost LLM acts as an Oracle. It receives the SLM's candidates, validates them against the schema, and refines the best option.
- The Oracle operates in two modes: General Refinement (using internal knowledge) and Schema-Aware Refinement (using explicit schema context).
- This approach leverages the SLM for speed and the LLM for high-fidelity validation, avoiding the cost of running the LLM for the entire generation process.

3. Key Contributions

First Systematic Evaluation: The paper provides the first comprehensive benchmark of SLMs for NL-to-KQL translation, comparing five SLMs against four LLMs across syntax, semantics, latency, and cost.
Novel Framework: Introduction of a three-pronged enhancement strategy (Error-Aware Prompting, LoRA with Rationale Distillation, and Two-Stage Architecture).
Cost-Efficient Architecture: Demonstration that a two-stage SLM-Oracle pipeline can achieve near-LLM performance at a fraction of the cost.
Generalizability: The framework is designed to be adaptable to other query languages (e.g., EQL, SPL) used in cybersecurity.

4. Experimental Results

The study evaluated nine models on Microsoft's NL2KQL Defender Evaluation Dataset (230 queries) and Sentinel-Queries (83 queries for generalization).

Baseline Performance:
- LLMs: Achieved high syntax accuracy (>0.90) but variable semantic accuracy in zero-shot settings.
- SLMs (Zero-Shot): Performed poorly, with semantic scores near 0 and table selection scores below 0.1, indicating an inability to independently map NL to complex schemas.
Impact of Enhancements:
- Prompting: Error-aware prompting significantly improved syntax scores for SLMs.
- Fine-Tuning: LoRA with rationale distillation improved semantic understanding but did not outperform the best prompting strategies alone.
- Two-Stage Architecture (Best Result):
  - Syntax Accuracy: 0.987 (on Defender dataset).
  - Semantic Accuracy: 0.906 (on Defender dataset).
  - Generalization: On unseen Sentinel data, achieved 0.964 syntax and 0.831 semantic accuracy.
Cost and Latency:
- The proposed SLM-Oracle approach was up to 10x cheaper in token costs compared to GPT-5.
- It maintained low latency, making it suitable for real-time SOC workflows.

5. Significance and Conclusion

This paper establishes that Small Language Models are a practical, scalable foundation for security analytics when combined with lightweight scaffolding.

Operational Impact: The two-stage architecture offers a viable deployment pattern for SOCs: use local/private SLMs for fast, low-cost draft generation and a narrow, low-cost LLM for final validation and refinement.
Economic Viability: By reducing token costs by an order of magnitude while maintaining near-LLM accuracy, this approach removes the economic barrier to automating threat hunting and log analysis at scale.
Future Outlook: The framework is extensible to other security query languages (EQL, SPL), suggesting a broad applicability for automating security operations beyond just KQL.