💻 computer science

Ontology-Amplified Distillation and Contextuality Auditing for Sovereign Enterprise Language Models: A Combined Proof-of-Mechanism and Negative-Results Method Study

This paper presents a combined study demonstrating that ontology-amplified distillation of a Qwen3.6-27B model on a single Apple M5 Max achieves Vietnamese financial task grounding comparable to a GPT-5 baseline but lacks statistical power to prove superiority, while a concurrent contextuality-audit pilot yields negative results indicating zero residual contextuality, collectively failing to establish deployability, safety, or statistical equivalence for sovereign enterprise language models.

Original authors: Thanh Luong Tuan

Published 2026-07-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Thanh Luong Tuan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're running a very strict bank in Vietnam. The rules say you can't send your customers' secret account files to a giant, super-smart AI cloud server in another country. You have to keep everything inside your own building. But here's the problem: the "local" AI you can run on your own computers is usually not as smart as the giant cloud one.

This paper asks a big question: Can we teach our local, smaller AI to be just as good at following the bank's specific rulebook as the giant cloud AI?

The researchers tried a clever trick called "Ontology-Amplified Distillation." Think of it like this: Imagine the giant cloud AI is a master chef who knows how to cook everything. The local AI is a junior chef. Instead of just telling the junior chef to "copy the master," they gave the junior chef a special, highlighted recipe card (the "ontology") that lists exactly which ingredients (terms) must be mentioned for a specific dish. They then trained the junior chef by showing them the master's answers with the recipe card, and the master's answers without it, rewarding the junior chef only when they used the card correctly.

The Result: A Tie, Not a Victory
After training the junior chef (a model called Qwen3.6-27B) on a single Apple M5 Max computer using just 47 made-up practice questions, they put it to the test. They gave it 40 real-world banking questions in Vietnamese.

Here is the score:

The Local Junior Chef: Got 36 out of 40 questions right (meaning it mentioned the required rulebook terms).
The Giant Cloud Chef (GPT-5): Also got 36 out of 40 questions right.

They tied! The local model managed to match the giant one on this specific test.

But Wait, Don't Pop the Champagne Yet
The paper is very careful not to call this a "win" or a "breakthrough." Here's why:

It's a Small Sample: The test only had 40 questions. The authors say this is too small to prove they are exactly equal. It's like flipping a coin 40 times and getting 20 heads and 20 tails; it looks even, but you can't be 100% sure the coin is fair. The math shows the real difference could be as much as 4 questions either way.
The Test Was Easy: The test didn't ask if the answers were correct or complete. It only asked, "Did you mention at least one of the magic words from the rulebook?" It's like grading a student only on whether they wrote the word "apple" in an essay about fruit, not on whether the essay made sense.
The Training Was Fake: The practice questions used to teach the junior chef were made up by a computer, written in English, and about healthcare and insurance—not the actual banking questions they were tested on. The paper admits this is a "proof of mechanism" (showing the idea works in theory) rather than a "deployment claim" (proving it's ready for real life).
No "Super" Power: Before the study, the researchers hoped the local model would actually beat the giant one because the rulebook helps small models more than big ones. That didn't happen. They just matched.

The Second Part: The "Confusion" Detector
The paper also tested a second idea: a "Contextuality Audit." Imagine you ask the AI the same question but change the role it plays (e.g., "Act as a risk manager" vs. "Act as a salesperson"). Sometimes the answer changes. Is that because the AI is confused and unreliable? Or is it just reacting normally to the new role?

The researchers ran a pilot test with 576 outputs. They found that what looked like "confusion" was actually just the AI reacting to the new role or the way the question was phrased. Once they accounted for that, there was zero leftover "confusion" or weird quantum-like behavior.

The Finding: The paper rules out the idea that the AI is secretly "contextual" in a spooky, unpredictable way.
The Lesson: If an AI gives different answers, don't panic and send it to a human immediately. First, check if the change was just because you changed the role or the wording. If it was, that's normal. Only call a human if the answer is still weird after you fix those things.

The Bottom Line
This paper is a "proof of concept" report. It says: "Hey, we showed that a small, local AI can be trained to match a giant AI on a specific, narrow test using a special rulebook method. We also showed that we can tell the difference between 'normal reaction' and 'actual confusion' in AI answers."

However, the authors are very clear: This is not ready for the real world yet. They haven't proven the AI is safe, they haven't proven it works on harder questions, they haven't proven it works on the smaller models they actually want to use, and they haven't proven it works with real human experts. It's a promising first step, like a pilot flying a plane for the first time and landing safely, but before you let it carry passengers, you need a lot more tests.

Technical Summary: Ontology-Amplified Distillation and Contextuality Auditing for Sovereign Enterprise Language Models

Problem Statement
Regulated financial institutions in Vietnam operate under strict data-residency rules that prohibit sending customer data to foreign frontier-model APIs. Consequently, a locally deployable model running within the institution's perimeter is a compliance requirement, not merely a cost-saving alternative. The core challenge is determining whether a smaller, locally deployable "student" model can achieve the same domain grounding quality as a frontier model (e.g., GPT-5) when restricted to on-premise hardware. Previous work established that supplying a domain ontology improves a frontier model's fidelity, but it remained unproven whether ontology grounding could lift a student model to parity with a frontier model on regulated tasks, particularly given the student's thinner parametric coverage of specialized domains.

Methodology
The study employs a two-part approach: a reduced-power proof-of-mechanism for model distillation and a consolidated contextuality audit for agent routing.

Ontology-Amplified Distillation:
- Model: A Qwen3.6-27B student model was adapted locally on a single Apple M5 Max (128 GB unified memory) using the MLX framework.
- Training Pipeline:
  - Capture: Frontier teachers (GPT-5 and GPT-5-mini) generated 451 grounded trajectories by injecting a specific "Foundation AgenticOS" ontology slice (covering role, domain concepts, regulations, and interaction patterns) into prompts.
  - Supervised Fine-Tuning (SFT): A low-rank adapter (LoRA) was trained on these trajectories, reducing loss from 2.370 to 0.496.
  - Ontology-Grounded Direct Preference Optimization (DPO): The student was further tuned on 47 synthetic preference pairs (42 training, 5 validation). In each pair, the ontology-grounded frontier answer was marked "preferred," and the base model's answer without the ontology slice was "dispreferred." This isolated the learning of ontology-grounded behavior from general frontier imitation.
- Evaluation: The student was evaluated against GPT-5 on a held-out set of 40 Vietnamese financial-domain tasks (banking, insurance, capital markets, real-estate technology).
- Metric: A local, deterministic "ontology-compliance" metric ( $r_{onto}$ ) was used. It scores 1.0 if an answer mentions at least one of the task's ontology terms and 0.5 otherwise. This metric acts as a degenerate proxy for a broader four-component compliance composite, focusing strictly on term presence rather than answer correctness or completeness.
Contextuality Audit:
- A separate pilot (RA-15) tested the "Contextuality-by-Default" (CbD) framework for enterprise-agent routing.
- The audit aimed to distinguish between "direct influence" (where prompt changes naturally alter answers), "construct coupling," and "residual contextuality" (inconsistency remaining after accounting for direct influence).
- The study analyzed outputs across multiple phases (1.0 to 1.3) involving role frames, prompt orders, and construct wording, using both local Qwen and Gemma models.

Key Results

Distillation Parity: On the 40 held-out tasks, the distilled student grounded 36 tasks (90% grounded rate; mean $r_{onto} = 0.95$ $r_{o n t o} = 0.95$ ), achieving the exact same count as the GPT-5 frontier baseline.
- Disagreement Pattern: The models missed different tasks. The student missed two capital markets and two real-estate technology tasks, while the frontier missed one capital markets and three real-estate technology tasks. Two tasks were missed by both.
- Statistical Limitation: With $n=40$ , the 95% confidence interval for the paired difference spans ±4 tasks. The result establishes an equal point estimate but does not demonstrate statistical equivalence or superiority. The study explicitly notes it failed to confirm the pre-registered "amplification" hypothesis (that the student would exceed the frontier).
Training Mechanism: The DPO objective successfully separated the preference pairs. Validation preference accuracy shifted from 0 to 1.0, and the reward margin increased to 0.307 in a single epoch, indicating the student learned to surface ontology terms when presented with the slice.
Contextuality Audit: The pilot yielded a negative result regarding residual contextuality. After correcting for probability-drift residuals and applying expectation-scale direct-influence corrections, the canonical CbD degree was zero for all Phase 1.3 groups in both the local-Qwen and Gemma runs. The useful signal identified was direct influence and construct coupling, not surviving residual contextuality.

Significance and Claims
The paper positions itself as a "mechanism-and-control" study rather than a deployment-ready solution. Its primary contributions are:

A Training Recipe: It specifies "ontology-amplified distillation" as a method to transfer specific ontology-adherence behaviors to sovereign models, separating this behavior from general frontier imitation.
A Governance Diagnostic: It proposes a contextuality audit method to determine when model disagreement should trigger escalation (e.g., debate, synthesis, human review). The finding that residual contextuality is zero in the pilot suggests that apparent disagreement is often explainable by direct influence or construct coupling, advocating for a "safer precursor rule" to audit these factors before escalating tasks.
Honest Scope Limitation: The authors explicitly state that the evidence does not support claims of deployability, safety, superiority, or statistical equivalence. The study is "underpowered" to rule out real gaps, uses synthetic English-only preference data for a Vietnamese evaluation, and relies on a coarse term-coverage metric.
Dual-Use Transparency: The paper acknowledges the researcher's dual role as a platform co-founder and academic, noting that the study serves both as an internal proof-of-concept for a sovereign model and an academic contribution.

Conclusion
The study demonstrates that a locally deployable Qwen3.6-27B model, tuned via ontology-grounded distillation, can match a frontier model's term-coverage performance on a specific set of Vietnamese financial tasks. However, this parity is a reduced-power proof of mechanism. The paper concludes that full-power evaluation is required to assess cost, safety (abstention on out-of-distribution prompts), cross-scale performance (e.g., 14B models), and cross-lingual grounding before any deployment claims can be made. The contextuality audit further suggests that enterprise routing should prioritize diagnosing direct influence and construct coupling before assuming residual contextuality requires complex multi-agent intervention.

Technical Summary: Ontology-Amplified Distillation and Contextuality Auditing for Sovereign Enterprise Language Models

More like this