💻 computer science

Seclens: Role-specific Evaluation of LLM's for security vulnerablity detection

The paper introduces SecLens-R, a multi-stakeholder evaluation framework that reveals significant variations in LLM vulnerability detection performance across different organizational roles, demonstrating that single-metric benchmarks fail to capture the distinct priorities of stakeholders like CISOs and engineering leaders.

Original authors: Subho Halder, Siddharth Saxena, Kashinath Kadaba Shrish, Thiyagarajan M

Published 2026-04-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Subho Halder, Siddharth Saxena, Kashinath Kadaba Shrish, Thiyagarajan M

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a massive ship (your company's software), and you need to hire a new First Mate (an AI model) to scan the horizon for hidden reefs (security vulnerabilities).

For years, the shipping industry has used a single number to rate these First Mates: "The Overall Score." If a mate has a score of 90/100, everyone assumes they are the best.

But this paper, SecLens, argues that this single number is a lie. It's like rating a chef solely on how fast they chop onions. A chef might be lightning-fast (great for a busy kitchen) but terrible at seasoning (disastrous for a fine-dining restaurant).

Here is the simple breakdown of what the authors discovered, using a few creative analogies.

1. The Problem: One Size Does Not Fit All

The paper says that different people in your company care about different things. A single "Overall Score" cannot satisfy them all.

The CISO (Chief Information Security Officer): Think of them as the Safety Inspector. They don't care if the First Mate is fast or cheap. They only care: "Did you miss a shark?" If the AI misses a critical vulnerability, it's a disaster. They want Recall (catching everything), even if it means the AI screams "Shark!" at a floating log (False Positives).
The Head of Engineering: Think of them as the Kitchen Manager. They care about Speed and Noise. If the AI screams "Shark!" at every piece of driftwood, the developers stop listening. They want Precision (only shouting when there's a real shark) so their team can keep cooking without panic.
The Chief AI Officer (CAIO): They are the Budget Manager. They ask: "Is this First Mate worth the salary?" They want a balance of skill and cost.
The Security Researcher: They are the Forensics Expert. They want to know how the AI found the shark. Did it see the fins? Did it smell the water? They need deep reasoning, not just a "Yes/No."
The "AI-as-Actor": This is the Robotics Engineer. They just want to know: "Does the robot crash when it tries to open a door?" They care if the AI can follow instructions without breaking.

2. The Solution: The "Role-Specific Glasses"

The authors created a new way to test AI called SecLens-R. Instead of giving one score, they put on five different pairs of colored glasses (one for each role) to look at the same AI.

They built a scoreboard with 35 different metrics (like speed, cost, accuracy, and reasoning).

The Safety Inspector puts on red glasses that only highlight "Missed Sharks" and "Critical Errors."
The Kitchen Manager puts on yellow glasses that highlight "False Alarms" and "Time Wasted."

The Shocking Result:
When they tested 12 top AI models, the scores changed wildly depending on who was looking!

Model A (Qwen3-Coder):
- To the Kitchen Manager: It's an A+. It rarely screams "Shark!" at driftwood. It's precise and fast.
- To the Safety Inspector: It's an F. It missed almost all the real sharks because it was too cautious.
- The Gap: A 31-point difference on the same model.
Model B (GPT-5.4):
- To the Kitchen Manager: A. Great precision, low cost.
- To the Safety Inspector: D. It missed critical vulnerabilities.
Model C (Gemini 3 Flash):
- To the Safety Inspector: B. It caught almost everything.
- To the Kitchen Manager: B. Good, but maybe a bit noisy.

3. The "Tool-Use" Twist

The paper also tested the AI in two modes:

Code-in-Prompt (CIP): You hand the AI a single page of code and ask, "Is this broken?" (Like looking at a photo of a reef).
Tool-Use (TU): You give the AI a map and a boat, letting it sail around the whole database to find the reef. (Like actually sailing the ship).

The Catch: The "Tool-Use" mode is 10 to 100 times more expensive and slower. It's like hiring a full dive team instead of just looking at a photo. The paper found that for most companies, just looking at the photo (CIP) is enough to make a decision, unless you specifically need the AI to navigate the whole ship autonomously.

4. The Big Takeaway

The paper concludes that there is no "Best AI" for security.

If you are a Security Chief, you should pick the AI that catches the most sharks, even if it's a bit noisy.
If you are an Engineering Lead, you should pick the AI that is quiet and fast, even if it misses a few small fish.

The Metaphor:
Imagine you are buying a car.

The Safety Inspector wants the car with the most airbags and the slowest top speed (safest).
The Racing Driver wants the car with the fastest top speed and the lightest weight (fastest).
The Budget Buyer wants the car with the best gas mileage.

If a car magazine says, "This car is the Winner because it has a score of 90," they are lying. That car might be the worst choice for a family road trip (Safety) and the worst choice for a race (Speed).

SecLens simply says: "Stop looking for the single 'Winner.' Look at the score that matters to your specific job."

Summary in One Sentence

Don't let a single "Overall Score" trick you; the best AI for your security team might be the worst AI for your engineering team, and you need to choose the right tool for your specific job, not the one with the highest number on the leaderboard.

1. Problem Statement

Current benchmarks for Large Language Models (LLMs) in security vulnerability detection suffer from a critical limitation: they collapse complex performance profiles into a single aggregate score (e.g., a single F1 score or leaderboard rank). This "one-size-fits-all" approach fails to address the divergent needs of different organizational stakeholders:

CISOs prioritize low false-negative rates on critical vulnerabilities and severity-weighted recall.
Heads of Engineering prioritize high precision (low false positives), fast execution, and actionable findings to maintain developer velocity.
Chief AI Officers (CAIOs) balance capability against cost and autonomous tool-use effectiveness.
Security Researchers require deep reasoning, evidence chains, and precise CWE classification.
AI-as-Actors need to evaluate parse reliability, format compliance, and graceful degradation in autonomous agents.

The paper argues that a model ranked first on a general leaderboard may be the worst choice for a specific stakeholder (e.g., a model with high recall but low precision is bad for engineering but good for a CISO).

2. Methodology: The SecLens-R Framework

The authors introduce SecLens-R, a multi-stakeholder evaluation framework designed to transform raw benchmark data into role-specific "Decision Scores."

A. Evaluation Layers

The framework builds upon the SecLens benchmark, evaluating models across two distinct layers:

Code-in-Prompt (CIP): The model receives a vulnerable function directly in the prompt. It tests pure reasoning without external tools.
Tool-Use (TU): The model interacts with a sandboxed repository using tools (read_file, search, list_dir). This tests real-world auditing capabilities and multi-turn navigation.

B. Dimension Taxonomy

The framework defines 35 shared evaluation dimensions organized into 7 categories:

Detection: MCC, Recall, Precision, F1, CWE Accuracy, Location IoU, Actionable Finding Rate.
Coverage & Consistency: CWE coverage breadth, worst-category floor, cross-language consistency.
Reasoning & Evidence: Evidence completeness, reasoning presence, reasoning quality.
Operational Efficiency: Cost per task, cost per TP, MCC per dollar, wall time, throughput, token usage.
Tool-Use & Navigation: Tool calls, turns, navigation efficiency, tool effectiveness.
Risk & Severity: Severity-weighted recall, critical miss rate, severity coverage.
Robustness: Parse success, format compliance, error rate, autonomous completion, graceful degradation.

C. Role-Specific Weight Profiles

Five stakeholder roles are defined, each selecting 12–16 dimensions from the shared pool and assigning integer weights that sum to 80.

CISO: Heavily weights Detection (34) and Risk/Severity (18).
CAIO: Balances Robustness (20), Efficiency (19), Detection (16), and Tool-Use (15).
Security Researcher: Concentrates on Detection (39) and Reasoning (21), with the highest weight on CWE Accuracy (12).
Head of Engineering: Prioritizes Detection (35) and Efficiency (22), heavily weighting Precision (12) and Actionable Findings (10).
AI-as-Actor: Focuses on Robustness (36) and Tool-Use (18), emphasizing autonomous completion (12).

D. Scoring Mechanism

The Decision Score is a weighted composite calculated as:
$D^{(r)}(m) = \frac{\sum_{i \in A(r)} w_i^{(r)} \cdot s_i^{(m)}}{\sum_{i \in A(r)} w_i^{(r)}} \times 100$

Dynamic Exclusion: If a dimension is unavailable (e.g., Tool-Use metrics in CIP mode), the denominator automatically adjusts to the sum of available weights, ensuring the score remains comparable.
Normalization: Raw values are mapped to [0, 1] using four strategies (Ratio, MCC, Lower-is-better, Higher-is-better) with fixed reference caps to eliminate cohort-relative artifacts.
Grading: Scores map to letter grades (A ≥ 75, B ≥ 60, etc.).

3. Key Contributions

Multi-Stakeholder Framework: A novel scoring layer that applies 5 distinct weight profiles to a shared pool of 35 dimensions, moving beyond single-score leaderboards.
Dynamic Scoring Logic: A methodology that handles missing data (e.g., tool-use metrics in CIP) by dynamically adjusting the scoring denominator.
Empirical Validation: Evaluation of 12 frontier models (from Anthropic, Google, OpenAI, and OpenRouter) on 406 tasks derived from 93 open-source projects across 10 languages and 8 OWASP-aligned categories.
Discovery of Divergence: Demonstration that model rankings are highly dependent on the stakeholder lens, with scores diverging by up to 31 points for the same model.

4. Key Results

The study evaluated models including GPT-5.4, Claude Opus/Sonnet/Haiku, Gemini 3.x, and Qwen3-Coder.

Significant Score Divergence:
- Qwen3-Coder: Earns an A (76.3) for the Head of Engineering but a D (45.2) for the CISO (31.1-point divergence).
- GPT-5.4: Earns an A (76.7) for Engineering but a D (48.4) for CISO (30.8-point divergence).
- Claude Haiku 4.5: Ranks 8th on the aggregate leaderboard but 2nd for the CISO (71.2), outperforming top leaderboard models in severity-weighted recall.
Role Divergence Index (RDI):
- Models with conservative prediction strategies (high precision, low recall) perform well for Engineering but poorly for CISOs.
- Models with high recall but lower precision perform well for CISOs but poorly for Engineering.
- The AI-as-Actor lens is universally lenient (all models score A), indicating that parse reliability is largely solved, whereas CISO scores show the widest variation, highlighting that severity-aware detection remains a challenge.
Category-Specific Weaknesses:
- No single model dominates all 8 vulnerability categories.
- Authentication Failures proved the hardest category (best F1: 0.585; worst: 0.000 for Opus 4.6).
- SSRF was a strength for Claude models.
Cost vs. Performance:
- GPT-5.4 offered the best quality-per-dollar ratio ($0.007/task).
- Claude Haiku 4.5 was the cheapest option ($0.005/task) while maintaining a "B" grade for CISOs.
- Tool-Use (TU) evaluation is 10–100x more expensive than CIP and significantly lowers scores for Engineering roles due to latency and cost penalties.

5. Significance and Implications

No Universal "Best" Model: The paper conclusively proves that model selection for security is not a single-objective problem. A model's suitability depends entirely on the specific organizational goal (e.g., risk mitigation vs. developer velocity).
Actionable Decision Support: The framework provides a mechanism for organizations to generate custom evaluation reports based on their specific risk appetite and operational constraints (via customizable YAML weight profiles).
Beyond Leaderboards: Aggregate leaderboard ranks are shown to be poor predictors of role-specific performance. Relying solely on a single score can lead to deploying models that are either too noisy for engineering or too blind for security teams.
Framework Reusability: The SecLens-R scoring layer is designed to be orthogonal to the underlying benchmark. It can consume per-task results from other benchmarks (like SecVulEval or TOSSS) to generate stakeholder-specific insights without requiring new data collection.

In conclusion, SecLens-R shifts the paradigm from "which model is the best?" to "which model is best for my specific stakeholder needs?", providing a rigorous, quantitative method for aligning LLM deployment with organizational security and engineering priorities.