Technical Summary: Natural Language Access to Domain-Specific Metadata

Problem Statement

Researchers and domain experts frequently possess large archives of domain-specific data (e.g., neuroimaging metadata) but lack the expertise to construct structured queries (SQL or SPARQL) against the underlying metadata schemas. While Large Language Models (LLMs) can translate natural language (NL) into structured queries, their accuracy is heavily dependent on how the underlying data schema is presented. On complex, real-world enterprise schemas with hundreds of tables, top-tier models often fail (dropping to 0% accuracy in some benchmarks), whereas performance on simple schemas remains high. Existing solutions often rely on fine-tuning, retrieval augmentation (RAG), or multi-agent orchestration to compensate for opaque schemas or poor domain representation, adding significant complexity and cost.

Methodology

The authors propose an ontology-first development process and a reusable framework called NLKGQ (Natural Language Knowledge Graph Query). The core hypothesis is that if domain vocabulary and semantics are captured in a well-designed Web Ontology Language (OWL) ontology, LLMs can generate accurate structured queries zero-shot, without fine-tuning or complex orchestration.

The Development Process

1. Ontology Design: Domain vocabulary is formalized in an OWL ontology following specific design principles to maximize LLM comprehension:
   • Use of full English words (e.g., hasAcquisitionTime instead of acqT).
   • Consistent naming patterns (e.g., has<Property>).
   • Descriptive relationships indicating direction.
   • Explicit domain and range declarations.
   • Rich natural language annotations (rdfs:comment, rdfs:label, skos:altLabel).
   • Avoidance of opaque codes in favor of meaningful URIs.
2. ETL and Knowledge Graph (KG) Construction: A domain-specific Extract-Transform-Load (ETL) pipeline extracts metadata from sources (e.g., DICOM headers, NIfTI files) and maps it to the ontology, producing an RDF Knowledge Graph. The pipeline materializes implied relationships (e.g., inverse properties) as concrete triples to avoid the need for runtime reasoning.
3. Iterative Refinement: The ontology, a set of competency/regression test cases, and a domain-specific prompt rider co-evolve. Testing reveals gaps that lead to refinements in the ontology (renaming confusing properties), the prompt rider (addressing recurring LLM mistakes), or the test cases.

The NLKGQ Framework

The framework is a domain-agnostic infrastructure supporting the above process:

• Query Server: Receives NL questions and constructs a system prompt containing the full OWL ontology (in Turtle format), generic SPARQL instructions, and a domain-specific rider.
• LLM Integration: An LLM generates the SPARQL query. The system handles post-processing (stripping chain-of-thought blocks, fixing prefixes) and can retry failed queries.
• Web Interface: Allows end-users to pose questions, view generated SPARQL for transparency, and download results. It includes features like prefix compression for readability and an "Explain" button.
• Comparative Backend: The framework automatically converts the OWL ontology into a relational SQL schema (PostgreSQL) to enable a controlled comparison between Text-to-SPARQL and Text-to-SQL.

Experimental Setup

The system was evaluated on a large-scale neuroimaging archive (MRI Research Ontology, MRO) containing metadata for 80 studies and 2,000 experiments.

• Models: Eight locally deployed Qwen3 variants (8B to 35B parameters, including dense, MoE, and quantized versions) were tested to ensure privacy compliance (GDPR) and feasibility on modest institutional hardware.
• Ablation Study: Eight different ontology representations were tested, ranging from the full annotated Turtle to abstract graphs with anonymized labels.
• Metrics: Accuracy was measured by exact match of query result sets against reference SPARQL queries across 21 competency questions.

Key Contributions

1. NLKGQ Framework & Process: A reusable framework and ontology-first development process for enabling NL access to domain-specific metadata, demonstrated on neuroimaging data.
2. Systematic Evaluation: A comprehensive evaluation of LLM-driven SPARQL generation across 8 models, 8 ontology representations, and 768 configurations (over 16,000 runs), identifying ontology design as the dominant factor in accuracy.
3. OWL-to-SQL Comparison: A generic OWL-to-SQL schema generator and data transformer enabling a direct comparison of Text-to-SPARQL vs. Text-to-SQL using the same data, ontology, and LLMs.
4. Structural Analysis: An analysis demonstrating that OWL's structural features (explicit domain/range, inverse properties, readable naming) provide a substantial advantage over SQL DDL for LLM-driven query generation.

Results

• Accuracy: The best configuration (Qwen3.6-27B dense model with the full, annotated ontology) achieved 100% accuracy on the competency question set for SPARQL generation, zero-shot.
• Ontology Impact: Ontology design was the most critical factor. Stripping semantic annotations (rdfs:comment, rdfs:label) reduced SPARQL accuracy from 100% to 81%. Replacing readable names with abstract identifiers collapsed accuracy to 10–14%.
• Model Performance: Dense models (27B) outperformed larger Mixture-of-Experts (MoE) models (35B), which peaked at 57–62%. Quantized models (Q8) running on older institutional hardware (Quadro RTX 5000) achieved the same 100% accuracy as full-precision models on HPC hardware.
• SPARQL vs. SQL: The same best configuration achieved only 57% accuracy on the auto-generated SQL backend. The gap is attributed to SQL's lack of structural scaffolding (e.g., explicit inverse relationships and domain/range grouping) compared to OWL.
• Prompting: Elaborate prompting strategies (guardrails, procedural instructions) offered no significant advantage over a simple baseline prompt when the ontology was well-designed.

Significance and Claims

The paper claims that the primary barrier to effective NL querying is not the LLM's capability, but the representation of domain knowledge. By co-designing the ontology specifically for LLM consumption (prioritizing readability and semantic annotation), the need for fine-tuning, RAG, or multi-agent systems is eliminated.

Key takeaways include:

• Ontology as Single Source of Truth: A well-designed ontology serves as the schema for the ETL pipeline, the context for the LLM, and the basis for the SQL conversion.
• Hardware Feasibility: High-accuracy NL querying is achievable on modest, local hardware (e.g., older GPUs or consumer-grade Apple Silicon with 128GB RAM), making it viable for institutions with strict data privacy requirements (e.g., human subject data).
• Structural Advantage of OWL: OWL provides structural advantages over SQL DDL that significantly aid LLMs in generating correct queries, particularly regarding relationship traversal and type safety.
• Iterative Co-Evolution: The process of refining the ontology, test cases, and prompt rider together is essential for resolving domain ambiguities and achieving high accuracy.

The authors conclude that for new domain metadata projects where the ontology can be designed, the "simpler path" of a well-structured ontology and zero-shot LLM generation is superior to complex pipelines designed to cope with opaque, pre-existing schemas.