Agentic SPARQL: Evaluating SPARQL-MCP-powered Intelligent Agents on the Federated KGQA Benchmark

This paper explores the potential of SPARQL-MCP-powered intelligent agents for federated Knowledge Graph Question Answering by extending an existing benchmark to evaluate agentic capabilities in endpoint discovery, schema exploration, and query formulation across multiple data sources.

The Gist

Imagine you are a detective trying to solve a complex mystery. In the past, you had to visit every single library, archive, and police station in the city one by one, ask the same question, and hope someone had the answer. If you asked the wrong librarian the wrong way, you'd get confused or sent away.

This paper is about teaching a super-smart AI detective (called an "Agentic AI") how to do this job automatically, but with a twist: instead of just one library, the AI has to search through a giant, interconnected network of thousands of different libraries (Knowledge Graphs) that all speak slightly different languages.

Here is the breakdown of their adventure, explained simply:

1. The Problem: The "Tower of Babel" of Data

The internet is full of data. Some of it is in a format called SPARQL (think of this as a very strict, formal language used by digital libraries to answer questions).

• The Challenge: There are thousands of these digital libraries. Some are big (like Wikidata), some are small. Some are open, some are closed. They don't all agree on how to answer questions.
• The Old Way: Humans had to manually figure out which library to ask, how to phrase the question, and then combine the answers. It was slow and hard.
• The New Way (Agentic AI): We want an AI that can look at a question like "Show me all of Tim Berners-Lee's books that are also on Wikidata," and automatically figure out:
  1. Which libraries have this info?
  2. How to ask them nicely?
  3. How to glue the answers together?

2. The Tool: The "Universal Translator" (MCP)

The researchers built a special tool called SPARQL-MCP.

• The Analogy: Imagine the AI is a traveler who speaks "English" (Natural Language). The digital libraries speak "Latin" (SPARQL).
• The Bridge: The MCP is like a Universal Translator and Tour Guide standing between the traveler and the libraries. It doesn't just translate words; it helps the traveler plan the trip. It says, "Hey, Library A is closed today, but Library B has what you need. Let's ask Library C first, then check Library B."

3. The Big Test: The "Federation" Benchmark

To see if this AI actually works, the researchers couldn't just test it on one library. They had to create a giant simulation.

• The Setup: They took 19 different datasets and chopped them up like a pizza, spreading the slices across 118 different "virtual libraries" (endpoints).
• The Twist: They made it so the AI didn't know which library had which slice. The AI had to discover the slices itself.
• The Goal: Can the AI find the right slices, ask the right questions, and put the pizza back together to answer the original question?

4. The Results: The "Smart" vs. The "Brute Force"

They tested two types of AI detectives:

• The Super-Genius (GPT-5.2): This AI was like a seasoned detective. It looked at the clues, figured out which libraries were likely to have the answer, and asked them specifically.
  • Result: It got the answer right about 45% of the time. This is impressive because it's as good as the best human-written systems, even though it had to do much more work (finding the libraries itself).
• The Hard-Worker (Qwen3-8B): This AI was smaller and less experienced. It tried to solve the mystery by shouting the question at every single library at once (a "brute force" approach).
  • Result: It got the answer right only 13% of the time. It also made a lot of grammar mistakes when writing the questions because the language (SPARQL) is very strict.

5. Key Discoveries & Surprises

• Less is More: The researchers found that giving the AI a one-sentence description of a library (e.g., "This library has car data") worked better than giving it a massive, technical manual (called a "VoID description"). The AI got overwhelmed by the technical details and made mistakes.
• The "Trivial" Trap: The smaller AI often tried to ask every library the same question, even when only one was needed. It was like asking 100 people for the time when you only needed to ask one. This wasted time and resources.
• The "Smart" Explorer: The smarter AI actually "explored." It would ask one library, see what it got, and then decide, "Okay, that wasn't it, let's try the next one."

6. Why This Matters

This paper is a big step forward because it proves that AI agents can handle complex, distributed data without needing a human to hold their hand.

• Before: You needed a human expert to write the code to connect different databases.
• Now: You can just ask the AI a question in plain English, and it figures out the connections.

The Catch: The AI still needs to be "smart" (large and powerful) to do this well. Smaller, cheaper AI models still struggle with the strict grammar of the data languages and tend to be inefficient.

The Bottom Line

The researchers have built a GPS for data. Instead of you driving around looking for the right database, you tell the AI your destination, and it navigates the complex web of digital libraries to find the answer for you. It's not perfect yet (the GPS sometimes takes a wrong turn), but it's a massive leap toward a future where we can ask the internet anything and get a real answer.

Technical Summary

Here is a detailed technical summary of the paper "Agentic SPARQL: Evaluating SPARQL-MCP-powered Intelligent Agents on the Federated KGQA Benchmark."

1. Problem Statement

The paper addresses the challenge of integrating Large Language Models (LLMs) with Federated Knowledge Graphs to solve complex Question Answering (KGQA) tasks. While LLMs have advanced significantly, they struggle with:

• Hallucinations and Knowledge Cut-offs: Requiring access to external, structured data.
• Federated Query Complexity: Existing "Agentic AI" approaches (using tools like the Model Context Protocol, or MCP) typically interact with single databases. They lack the capability to dynamically discover, select, and query multiple distributed SPARQL endpoints simultaneously.
• Specific Challenges in Agentic SPARQL:
  • Interface Heterogeneity: Endpoints vary from simple triple-pattern fragments to full SPARQL 1.1 endpoints.
  • Uneven SPARQL Support: Variations in supported features (e.g., SERVICE, VALUES, aggregates) across endpoints.
  • Metadata Heterogeneity: Lack of standardized, accessible metadata (VoID descriptions) for endpoint discovery and schema exploration.
  • Latency & Availability: Unpredictable endpoint performance and timeouts affecting query planning.
  • Query Formulation: Difficulty in generating syntactically correct and semantically accurate federated SPARQL queries from natural language without prior knowledge of endpoint URLs or schemas.

2. Methodology

A. SPARQL-MCP Server Implementation

The authors developed SPARQL-MCP, an extension of the Model Context Protocol designed specifically for federated SPARQL querying. Key architectural features include:

• Toolset: Provides agents with tools for run_sparql_query, get_void_descriptions (schema exploration), and endpoint discovery.
• Proxy Federation Engine: To handle SPARQL 1.1 SERVICE clauses (which many public endpoints like Wikidata block), the system routes multi-endpoint queries through a local proxy federation engine. This engine decomposes queries, forwards subqueries to remote endpoints, and joins results.
• Dynamic Schema Exploration: The system dynamically retrieves or computes VoID (Vocabulary for Interlinking Datasets) descriptions to provide agents with schema information (classes, properties, statistics) when not explicitly provided.

B. The FKGQA Benchmark

To evaluate these agents, the authors created the Federated Knowledge Graph Question Answering (FKGQA) benchmark by extending the existing Spider4SPARQL dataset:

• Partitioning Strategy: They applied three sharding strategies to 19 RDF datasets to create 118 distinct shards (endpoints):
  1. Class Sharding: Splitting data based on subject types (requires conjunctive splits).
  2. Predicate (Vertical) Sharding: Splitting based on predicates (requires conjunctive splits).
  3. Horizontal Sharding: Splitting instances of a class across shards (requires disjunctive splits).
• Set-Cover Optimization: Used Answer Set Programming (ASP) to select a minimal set of shardings that ensures every query in the benchmark requires federation.
• Evaluation Conditions: Agents were tested in three scenarios:
  1. Baseline: Only endpoint URLs and query execution tools.
  2. High-Level: URLs + one-sentence natural language descriptions of endpoints.
  3. VoID Tool: URLs + access to the get_void_descriptions tool for detailed schema exploration.

C. Experimental Setup

• Models: Evaluated GPT-5.2 (high-capability frontier model) and Qwen3-8B (compact open-weight model).
• Agent Architecture: A ReAct-style agent (Reason + Act) that alternates between LLM reasoning and tool calls.
• Metrics: Syntactic validity of queries, overall pipeline accuracy (matching ground truth), endpoint consultation accuracy, and behavioral analysis (exploration vs. redundancy).

3. Key Contributions

1. Agentic SPARQL Framework: A novel implementation of SPARQL-MCP that enables LLM agents to perform federated querying, including dynamic endpoint discovery and schema exploration, rather than just single-endpoint interaction.
2. FKGQA Benchmark: The first general-purpose benchmark for end-to-end Federated KGQA, featuring 118 dynamically generated endpoints and queries requiring complex federation strategies (conjunctive and disjunctive splits).
3. Comprehensive Evaluation: A large-scale study (5,814 agent runs) comparing different architectural options and model capabilities in a federated setting.

4. Results

Performance Accuracy

• GPT-5.2: Achieved accuracy rates of 42.1% – 45.4% across the three scenarios. This is comparable to state-of-the-art approaches on the original non-federated Spider4SPARQL benchmark (45%), demonstrating that agentic approaches can handle federation complexity.
• Qwen3-8B: Significantly lower accuracy (13.1% – 13.8%), highlighting that smaller models struggle with the combined tasks of schema understanding, federation logic, and SPARQL syntax generation.

Syntactic Validity

• GPT-5.2: High success rate (97.4% – 98.0%) in generating syntactically correct SPARQL queries.
• Qwen3-8B: Low success rate (41.5% – 61.1%), with common errors including prefix issues, unmatched braces, and malformed SERVICE syntax.

Behavioral Analysis & Endpoint Selection

• Exploration vs. Brute Force:
  • GPT-5.2 exhibited an exploration strategy, switching endpoints and refining queries based on results. It effectively used high-level descriptions to reduce "trivial queries" (querying all endpoints unnecessarily) from 90.2% (baseline) to 11.0% (high-level).
  • Qwen3-8B showed no exploration, rarely switching endpoints, and frequently querying all endpoints simultaneously (trivial federation), leading to high redundancy and inefficiency.
• Metadata Utility: Surprisingly, high-level natural language descriptions were more effective for guiding source selection than detailed VoID metadata. The VoID tool did not consistently improve accuracy over high-level descriptions, suggesting that for current agents, concise semantic summaries are more actionable than complex technical metadata.

5. Significance and Future Work

• Viability of Agentic Federation: The paper proves that sufficiently capable LLMs can autonomously manage the complexities of federated SPARQL querying, bridging the gap between natural language and distributed linked data.
• Limitations: Current smaller models lack the SPARQL-specific reasoning and syntax generation capabilities required for production use. Additionally, agents currently lack the "cost-awareness" of traditional federated optimizers, often generating inefficient query plans.
• Future Directions:
  • Model Specialization: Fine-tuning smaller models specifically for SPARQL syntax and federation logic.
  • Metadata Optimization: Investigating how to structure metadata (beyond VoID) to be more effective for agentic planning.
  • Registry Development: Building searchable, up-to-date registries of endpoint availability and quality to assist agents in real-world scenarios where endpoints are unstable.

In conclusion, this work establishes a foundational step toward "Agentic SPARQL," demonstrating that while current frontier models can effectively navigate federated knowledge graphs, significant improvements in model scale and tool design are necessary for robust, scalable deployment.