Towards Neural Graph Data Management

This paper introduces NGDBench, a comprehensive benchmark designed to evaluate neural models' ability to manage and query structured graph data across diverse domains using the full Cypher language, revealing significant current limitations in reasoning and robustness despite the models' success with unstructured text.

The Gist

Imagine you have a brilliant, super-smart librarian (an AI) who has read every book, article, and website on the internet. This librarian is amazing at understanding messy, unstructured stories. But now, you ask them to manage a giant, high-speed train schedule or a complex financial ledger stored in a strict database.

Suddenly, the librarian gets confused. They can't handle the rigid rules, the specific numbers, or the fact that the schedule changes every second.

This paper, "Towards Neural Graph Data Management," introduces a new tool called NGDBench to fix this problem. It's like a "driving test" for AI, specifically designed to see if it can handle complex, structured data (like graphs) as well as it handles text.

Here is the breakdown using simple analogies:

1. The Problem: The Librarian vs. The Ledger

• The Status Quo: AI models are great at reading a novel (unstructured text). But when you give them a database (structured data), they struggle.
• The Three Big Hurdles:
  1. Too Simple: Current AI can only do basic "find the red ball" tasks. Real life needs "Find the average price of all red balls sold in the last hour." AI often fails at the math and logic parts.
  2. The "Fake News" Problem: In the real world, data is messy. Sometimes a graph (a map of connections) has fake links or missing pieces. AI often treats the messy map as the truth, leading to bad decisions (like thinking a fraudster is innocent because the data looked "normal").
  3. The Moving Target: Databases change constantly. If you ask, "What's the stock price right now?", the AI can't just re-read its entire training book every second. It needs to update its memory instantly.

2. The Solution: NGDBench (The Ultimate Driving Test)

The authors created NGDBench, a massive testing ground with five different "driving courses" (domains):

• Social Networks: Like tracking who knows whom in a huge city.
• Finance: Like a bank ledger tracking money transfers.
• Medicine: Like a complex map of diseases and genes.
• AI Tools: Tracking how AI agents use different software tools.
• Business Reports: Connecting companies to their financial data.

What makes this test special?

• The "Noise" Injection: They don't just give the AI clean data. They deliberately break the data (add typos, remove connections, swap labels) to see if the AI can still find the truth. It's like asking the librarian to find a book in a library where some shelves are missing and some books have the wrong titles.
• The "Full Cypher" Language: Previous tests only let AI ask simple questions. NGDBench lets them ask complex questions using the full language of graph databases (Cypher). This includes things like "Find the shortest path between A and B, but only if the total cost is under $500."
• Dynamic Updates: The test includes a "live" section where the AI has to add or delete items from the database and immediately answer questions about the new state.

3. The Results: The AI is Still Learning

The authors tested the smartest AI models available (like GPT-5, DeepSeek, and Qwen) on this new benchmark. The results were a wake-up call:

• Good at Text, Bad at Math: The AI models were okay at simple lookups but terrible at complex calculations (like averaging numbers) or spotting errors in the data.
• The "RAG" Struggle: Methods that try to "search and retrieve" information (RAG) often missed the big picture. They found the right "neighborhood" but couldn't navigate the whole "city."
• The "Oracle" Gap: Even when the AI was given the perfect instructions, the data itself was so noisy that the AI couldn't get the right answer. This proves that the problem isn't just the AI's intelligence; it's that the data is too messy for current models to handle reliably.

4. Why This Matters

Think of NGDBench as the "Crash Test Dummy" for the future of AI.

• Before this, we didn't have a standard way to measure if an AI could actually manage a real-world database.
• Now, researchers have a clear target. They know exactly where AI fails (noise, math, updates).
• The goal is to build "Neural Graph Databases"—systems where AI doesn't just read the database but understands it, fixes its own mistakes, and updates itself in real-time, just like a human expert would.

In a nutshell: We are building AI that can read a novel perfectly. This paper says, "Great, but can it also manage a bank, fix a broken map, and update a train schedule in real-time?" The answer is "Not yet," and NGDBench is the tool we'll use to teach it how.

Technical Summary

Here is a detailed technical summary of the paper "Towards Neural Graph Data Management":

1. Problem Statement

Despite the rapid growth of structured data in databases (e.g., graph databases, relational systems), modern AI systems, particularly Large Language Models (LLMs), struggle to effectively utilize this data. Current neural approaches face three critical limitations:

• Limited Expressiveness: Existing methods are often restricted to elementary logical operations (Existential First-Order logic) and simple pattern matching. They fail to handle complex analytical reasoning common in real-world databases, such as numerical aggregations (AVG, SUM), variable-length path traversals, and value filtering.
• Factual Discrepancy: Real-world graphs often contain noise or adversarial edges (e.g., in fraud detection). Models tend to overfit to the observed (noisy) topology rather than recovering the underlying ground truth or valid logic.
• Dynamic Updatability: High-frequency data environments (e.g., finance, real-time recommendations) require instant graph updates (Create, Update, Delete). Current neural methods often rely on expensive retraining, lacking the ability to perform dynamic, in-context state management.

Furthermore, there is a lack of high-quality benchmarks that evaluate these capabilities. Existing datasets are often static, homogeneous, or limited to specific modalities, failing to reflect the heterogeneity, noise, and dynamic nature of production graph databases.

2. Methodology: NGDBench

The authors introduce NGDBench, a unified benchmark designed to evaluate neural graph database capabilities across five diverse domains: Social (NGD-BI), Finance (NGD-Fin), Medicine (NGD-Prime), AI Agent Tooling (NGD-MCP), and Economics (NGD-Econ).

A. Data Construction & Unification

• Diverse Sources: The benchmark integrates structured data (from LDBC benchmarks, PrimeKG) and unstructured data (annual reports, AI tool trajectories) into a unified Labeled Property Graph (LPG) representation.
• Perturbation Generation: To simulate real-world imperfections, a Perturbation Generator injects controlled noise at three levels:
  • Topological: Stochastic edge addition/removal.
  • Schema: Relation type swapping or node mislabeling.
  • Attribute: Typos, OCR errors, and numerical deviations.
  • Note: Unstructured datasets (NGD-MCP, NGD-Econ) inherently contain extraction noise, so no artificial injection is applied.

B. Query Workload

Unlike prior works limited to simple logic, NGDBench supports the full Cypher query language (the de facto standard for graph databases like Neo4j).

• Analytical Queries: A template library covers complex patterns, including variable-length paths, set operations (Union, Cartesian Product), and numerical aggregations (COUNT, AVG, MAX).
• Management Queries: A "Batch" system simulates dynamic transactions. Each batch consists of $K$ steps (e.g., $K=5$), where each step is an Operation-Validation Pair (e.g., a CREATE or DELETE operation followed immediately by a read query to verify the state change).
• Format: Queries are provided in both Cypher and natural language (via translation) to test Text-to-Cypher and GraphRAG capabilities.

C. Evaluation Tasks

The benchmark evaluates models on two primary tasks:

1. Robust Analytical QA: Measuring the model's ability to answer queries over noisy graphs, minimizing the discrepancy between the model's output and the execution on the clean ground truth.
2. Dynamic Graph Management: Evaluating the model's ability to maintain an internal graph state through sequential "in-context" editing (Create/Update/Delete) and verify the state after each step.

3. Key Contributions

1. Comprehensive Benchmark: NGDBench is the first unified benchmark spanning five diverse domains, integrating structured and unstructured data with realistic noise injection.
2. Advanced Workloads: It moves beyond simple logical reasoning to support the full spectrum of Cypher (complex patterns, aggregations) and dynamic data management (CUD operations), bridging the gap between neural models and industrial database standards.
3. Systematic Evaluation: The paper provides a rigorous evaluation protocol and reveals significant gaps in current state-of-the-art (SOTA) LLMs and RAG methods.

4. Experimental Results

The authors evaluated SOTA LLMs (e.g., GPT-5.1-Codex, DeepSeek-V3.2, Qwen3-Coder) and GraphRAG approaches.

• Analytical Queries (Structured Domains):
  • Text-to-Cypher methods generally outperformed GraphRAG in non-aggregation and aggregation scenarios. This is because structured query mechanisms allow for more complete information retrieval compared to RAG's dense vector search, which often truncates candidates.
  • However, performance dropped significantly on "inconsistent" subsets where the noise altered the ground truth, highlighting that models struggle to infer valid logic from corrupted data.
  • Aggregation: Models showed poor precision in numerical reasoning (high sMAPE and MSLE), often failing to correctly aggregate values or filter by numerical thresholds.
• Boolean Queries: GraphRAG performed competitively here, as the candidate-based formulation narrows the retrieval space, making verification easier than generating complex Cypher logic.
• Dynamic Management:
  • Text-to-Cypher methods could generate explicit edit operations but suffered from error accumulation; a mistake in an early step propagated to subsequent states.
  • GraphRAG, reasoning over the full edit history, showed better resilience against cascading errors but struggled with precise state tracking.
• Unstructured Domains: GraphRAG significantly outperformed text-first baselines (like HippoRAG2) on unstructured datasets (NGD-MCP, NGD-Econ). The structured graph representation provided a higher signal-to-noise ratio for local, structural queries compared to extracting entities and running multi-hop searches on raw text.

5. Significance

• Bridging the Gap: NGDBench establishes a critical testbed for advancing Neural Graph Data Management, moving the field from theoretical logical reasoning to practical, production-grade database capabilities.
• Identifying Weaknesses: The results expose that current AI systems lack noise robustness and analytical precision, particularly in handling dynamic updates and complex aggregations.
• Future Directions: The benchmark sets a standard for developing next-generation systems that can handle noisy, dynamic, and complex graph data, paving the way for AI agents that can reliably interact with enterprise knowledge graphs and databases.

The code and data for NGDBench are publicly available at https://github.com/HKUST-KnowComp/NGDBench.