Technical Summary: Cross-Vendor Sola ISPM Benchmark

Problem Statement

The rapid proliferation of multi-cloud and SaaS architectures has transformed Identity Security Posture Management (ISPM) into a fundamentally cross-vendor challenge. Critical security risks, such as privilege escalation paths and misconfigurations, increasingly span disparate Identity Providers (IdPs), Cloud Service Providers (CSPs), and productivity suites that were never designed to interoperate. While Large Language Models (LLMs) are increasingly deployed for security operations, the field lacks a standardized, data-grounded benchmark to assess whether agentic AI can perform reliable reasoning across these fragmented boundaries.

Existing benchmarks, such as Spider 2.0 for text-to-SQL, operate over well-defined, single-database schemas with explicit foreign keys. In contrast, cross-vendor ISPM requires navigating disjointed systems lacking shared schemas. This creates a "Correlation Gap," where an agent must perform multi-hop entity resolution (e.g., inferring that an Okta user identity corresponds to a specific AWS IAM role) without explicit relational metadata. Current evaluations fail to measure an agent's ability to reconstruct the precise supporting entities, join relationships, and cross-platform evidence required to act on security conclusions.

Methodology

The Cross-Vendor Sola ISPM Benchmark

To address this evaluation void, the authors introduce a production-grade benchmark consisting of 50 data-grounded tasks. These tasks are deployed over a live, integrated enterprise environment comprising eight distinct platforms:

• Identity & HR: Okta, Azure Active Directory, HiBob (HRIS).
• Cloud & Infrastructure: AWS, Google Cloud Platform (GCP), MongoDB Atlas.
• Productivity & DevOps: Google Workspace, GitHub.

The benchmark covers critical cross-vendor use cases, including MFA bypass vulnerabilities, dormant account proliferation across federated tenants, and multi-hop privilege escalation vectors.

Question Generation Pipeline:
The 50 tasks were constructed via an iterative multi-agent pipeline:

1. Generation: A schema-aware generative agent, equipped with web-search capabilities, proposes candidate natural-language questions and SQL queries based on expert-defined security themes.
2. Filtering: An automated execution filter discards candidates where SQL fails to execute or cannot be deterministically answered by the live environment.
3. Refinement: A "Critic-Composer" loop iteratively refines questions and queries against expert-defined criteria (technical correctness, operational realism, PII compliance).
4. Validation: A human cybersecurity expert performs final validation to ensure semantic uniqueness and operational realism.

Evaluation Framework

The authors propose a comprehensive evaluation framework that measures not only final answer correctness but also the quality of the reasoning chain. The framework utilizes a LLM-as-a-Judge approach with a dual-judge consensus mechanism (Anthropic Claude Sonnet 4.6 and OpenAI GPT-4.1) and deterministic structural metrics.

Key Metrics:

• Answer Quality: AnswerCorrectnessVsGT (semantic equivalence), AnswerVerdictCorrectnessVsGT (binary yes/no verdict), SuccessRate, and FailureRate.
• Retrieval & Context Use: ContextualRelevancy, RetrievalUseQuality, and EvidenceCompletenessVsGT.
• Reasoning Quality: ReasoningUtility (progress toward resolution).
• SQL Quality: SQLSemanticAppropriateness and SQLEquivalenceVsGT.
• Deterministic Structural Metrics: Precision, Recall, and F1 scores for Tables and Joins compared against ground truth, ensuring the agent correctly identified the necessary data sources and cross-vendor join paths.

Experimental Setup

The study evaluates the Sola AI Agent across three frontier LLMs (GPT-5.5, Gemini 3.1 Pro, Claude 4.6 Sonnet, and Claude 4.8 Opus) under five distinct context configurations:

1. No Context: Blind autonomous discovery.
2. Schema Only: Raw table definitions.
3. Schema + Graph: Schemas plus Sola's Security Graph (explicit cross-vendor join paths).
4. Schema + Examples: Schemas plus few-shot text-to-SQL examples.
5. Full Context: Schemas, Security Graph, and Examples.

The agent operates in full-path exploration mode, performing iterative reasoning and step-journaling to isolate the impact of contextual grounding on explicit reasoning trajectories.

Key Contributions

1. Cross-Vendor ISPM Question Suite: A standardized set of 50 tasks requiring multi-hop entity resolution and cross-system data correlation across eight integrated enterprise platforms, representing real-world federated identity investigation workflows.
2. Evidentiary Grounding Metric Framework: A novel evaluation suite that assesses the full reasoning chain, including structural join reconstruction, multi-schema entity resolution, and retrieval utilization, moving beyond simple answer correctness.
3. Ablation Study on Contextual Grounding: A systematic analysis demonstrating how structured relational context (specifically graph topology and schema metadata) shapes cross-vendor reasoning fidelity and execution efficiency.

Results

Performance Gains from Context

The study demonstrates that enriching agents with structured relational context significantly improves performance:

• Answer Correctness: Under the Full Context configuration, the best-performing model (Claude 4.8 Opus) achieved an AnswerCorrectnessVsGT score of 78%, a relative improvement of approximately 34% over the No Context baseline (58%).
• Failure Reduction: The Full Context configuration reduced the FailureRate (complete inaccuracies) to 4%, compared to 28–38% in the No Context baseline.
• Efficiency: Enriching the agent with context reduced the average number of exploration queries by approximately 70% across all models. This indicates that explicit relational context transforms inefficient trial-and-error exploration into targeted, graph-aware execution strategies.

Structural Fidelity

The inclusion of the Security Graph (cross-vendor topology) was identified as a primary driver of gains.

• Join Fidelity: Models showed clear upward trajectories in JoinsF1 scores when graph knowledge was injected. For example, Gemini's join fidelity surged from 0.55 (Schema Only) to 0.68 (Full Context).
• Reasoning vs. Evidence: A critical finding is the gap between AnswerVerdictCorrectness (high-level security conclusion) and AnswerCorrectness (full semantic evidence). Even in No Context, models often achieved high verdict scores (e.g., correctly identifying a risk existed) but failed to reconstruct the precise supporting entities and join paths. Full Context significantly narrowed this gap, proving that the primary limitation of frontier LLMs in this domain is not abstract reasoning capability, but the availability of explicit relational context for entity resolution and evidentiary grounding.

Model Comparisons

• Claude 4.8 Opus consistently outperformed other models, achieving the highest accuracy and lowest failure rates under Full Context.
• GPT-5.5 demonstrated high execution efficiency (lowest query counts) but lower semantic accuracy compared to top-tier Claude models.
• Gemini 3.1 Pro showed strong sensitivity to contextual enrichment but appeared to reach a mild context-saturation threshold where additional metadata layers yielded diminishing returns.

Significance and Claims

The paper claims that the Cross-Vendor Sola ISPM Benchmark provides the first standardized means of evaluating whether agentic systems can reliably navigate the "Correlation Gap" in federated identity security.

The authors assert that:

1. Unassisted LLMs are insufficient for reliable enterprise data exploration across fragmented systems.
2. Frontier LLMs possess substantial latent security reasoning capability, but their ability to perform reliable cross-vendor analysis is fundamentally constrained by the lack of structured relational context.
3. Context orchestration is critical: Injecting rich metadata—specifically relational schemas, semantic graphs, and few-shot exemplars—fundamentally stabilizes text-to-SQL execution, acting as both an accuracy multiplier and an operational accelerator.
4. Structural graphs are essential: The integration of structural graph topologies is universally essential for achieving optimal balance between text-to-SQL accuracy and runtime efficiency in cross-vendor environments.

The work concludes that future ISPM evaluation must move beyond isolated platform checks to assess the agent's ability to reconstruct evidence across heterogeneous ecosystems, a capability that is currently dependent on explicit, domain-aware context orchestration.