Technical Summary: Open Ontologies

Problem Statement

Current ontology engineering faces two distinct gaps. First, while Large Language Models (LLMs) can generate syntactically valid OWL and orchestrate multi-step workflows, they cannot guarantee logical consistency or verify that generated axioms are mutually coherent. Second, existing ontology alignment systems achieve strong pairwise matching on benchmarks but operate as standalone pipelines, disconnected from construction workflows and lifecycle management. Furthermore, there is a lack of understanding regarding how LLMs interact with ontological data: specifically, whether providing raw syntax files improves performance compared to unaided inference or structured tool access.

Methodology

The authors present Open Ontologies, an open-source system implemented in Rust (~17,400 lines of code) that integrates ontology construction, reasoning, alignment, and lifecycle management into a single LLM-orchestrated workflow. The system is distributed as a single binary with no JVM or Python dependencies.

System Architecture

The system exposes tools via the Model Context Protocol (MCP), allowing an LLM to generate OWL, validate it against formal constraints, and refine it based on symbolic feedback.

• Core Engine: Utilizes an Oxigraph triple store for SPARQL 1.1 queries, a native reasoner supporting OWL-RL forward chaining for triple materialization and a partial SHIQ tableaux for consistency checking, a SHACL validator, and a pattern enforcer.
• Alignment Module: Computes a weighted similarity score for candidate class pairs using six signals: label similarity (Jaro-Winkler + token Jaccard), property overlap, parent overlap, instance overlap, restriction similarity, and neighbourhood similarity.
  • Stable Matching: A critical component where candidates are sorted by confidence, and a stable 1-to-1 matching algorithm is applied to eliminate many-to-many spurious matches.
  • Penalty Mechanism: If structural signals are zero, confidence falls back to label similarity with a 15% penalty to prevent structurally unsupported matches.
• Lifecycle Management: Implements an infrastructure-as-code approach involving planning (diff with blast-radius scoring), enforcement, safe application, monitoring, and drift detection, all recorded in an append-only lineage trail.

Evaluation Setup

The system was evaluated using:

1. OAEI Benchmarks: The Anatomy track (2,737 mouse classes, 3,304 human classes) and the Conference track (7 ontologies).
2. OntoAxiom Benchmark: Used to evaluate three LLM access modalities: (B) unaided inference from name lists, (D) reading raw OWL/Turtle files, and (C) using structured MCP tools (SPARQL).
3. Reasoning Performance: Compared OWL-RL forward chaining against the HermiT reasoner on the LUBM benchmark.
4. Construction Task: Automated construction of the Manchester Pizza Tutorial ontology.

Key Contributions and Results

1. Stable Matching Dominates Alignment Quality

The paper presents a counter-intuitive finding regarding ontology alignment: the matching constraint is the dominant factor, rendering signal weights largely irrelevant when stable matching is applied.

• OAEI Anatomy Track: The system achieved an F1 score of 0.832 (Precision = 0.963, Recall = 0.733). This precision exceeds all published systems (e.g., AML at 0.950, BERTMap at 0.940).
• Ablation Study: When stable matching was enabled, varying the weights of the six similarity signals resulted in F1 variations of less than 0.004 (ranging from 0.830 to 0.834). Conversely, removing stable matching dropped the F1 score to 0.728, regardless of the weighting scheme.
• Conference Track: The method achieved a micro-averaged F1 of 0.438. The lower recall (0.320) is attributed to heterogeneous modeling styles where label overlap is minimal.

2. Tool Structure vs. Information Availability

The study disentangles the effect of tool structure from information availability in LLM interactions.

• Raw File Performance (Condition D): An LLM reading a raw OWL file achieved an F1 of 0.323, performing significantly worse than unaided inference (Condition B, F1 = 0.431). The raw syntax introduced parsing noise (e.g., confusion between property and class IRIs, missed multi-line blocks) that overwhelmed the model's extraction ability.
• Structured Tool Performance (Condition C): Using MCP tools (SPARQL access) achieved an F1 of 0.717.
• Conclusion: Structured tool access provides a qualitatively different mode of access that raw syntax cannot replicate. The improvement from raw files to structured tools (+122%) outweighs the benefit of simply providing more information.

3. System Performance and Construction

• Reasoning: The OWL-RL reasoner operates at interactive speeds (e.g., 15 ms for 50,000 triples) compared to HermiT (24,490 ms), though it covers a different, incomplete inference profile suitable for engineering workflows.
• Construction: The system constructed a 91-class ontology based on the Manchester Pizza Tutorial in under 5 minutes, achieving 96% class coverage, compared to the ~4 hours required for manual work.

Significance and Claims

The paper claims two primary empirical findings that challenge existing assumptions in the field:

1. Matching Strategy over Signal Engineering: For ontologies with limited structural metadata, the assignment constraint (stable 1-to-1 matching) is the binding factor for alignment quality. Efforts to engineer complex similarity functions yield diminishing returns compared to enforcing a stable matching constraint.
2. Qualitative Difference in Tool Access: Providing an LLM with raw syntax is not merely "more information"; it can be actively harmful. Structured, typed tool access (via MCP) is essential for high-quality extraction, suggesting that the structure of tool access is more critical than the content of the information provided.

The authors note limitations, including that OAEI results have not yet been independently evaluated in an official campaign, signal ablation was limited to the Anatomy track, and the system relies on a single primary developer and specific LLM models (Claude). The system is released under the MIT license.