🤖 AI

Toten: Knowledge-Based Ontological Tokenization Of Physical Quantities And Technical Notation In Brazilian Portuguese

This paper introduces TOTEN, a knowledge-based ontological tokenization framework for Brazilian Portuguese that replaces statistically derived subwords with a declarative, ontology-driven approach to preserve the semantic and structural integrity of physical quantities and technical notation, demonstrating superior performance over state-of-the-art baselines in unit atomicity and numerical reconstruction.

Original authors: Antonio de Sousa Leitão Filho; Allan Kardec Duailibe Barros Filho; Fabrício Saul Lima; Selby Mykael Lima dos Santos; Rejani Bandeira Vieira Sousa

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Antonio de Sousa Leitão Filho; Allan Kardec Duailibe Barros Filho; Fabrício Saul Lima; Selby Mykael Lima dos Santos; Rejani Bandeira Vieira Sousa

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to read a complex engineering manual written in Brazilian Portuguese. The manual is full of precise measurements like "5.2 kN/m²" (kilonewtons per square meter) or "1.5 × 10⁻³ m".

The Problem: The "Pixelated" Translator
Most modern AI models use a method called Byte-Pair Encoding (BPE) to read text. Think of this like a translator who only speaks in tiny, broken fragments. To save space, this translator chops words into arbitrary pieces based on how often they appear in general books.

If the text says "5.2 kN/m²", a standard BPE tokenizer might chop it into: 5, ., 2, k, N, /, m, ².
To the computer, "5.2 kN/m²" isn't a single concept; it's just a random pile of eight separate puzzle pieces. The computer has to guess later how to put them back together to understand that it's a specific physical force. This is like trying to understand a sentence by looking at individual pixels of the letters rather than the letters themselves.

The Solution: TOTEN (The "Smart Librarian")
The authors created a new system called TOTEN. Instead of guessing how to chop words based on statistics, TOTEN acts like a knowledge-based librarian who has a strict, formal rulebook (an "ontology") for what engineering terms actually are.

Here is how TOTEN works, using simple analogies:

The Rulebook (The Ontology): Before reading a single word, TOTEN has a pre-written dictionary of engineering rules. It knows that "kN" is a unit of force, "m²" is an area, and "5.2" is a number. It doesn't guess; it knows the definitions by heart.
The Three Expert Assistants (Oracles): TOTEN doesn't try to memorize every possible typo or symbol. Instead, it calls on three trusted experts for help:
- Pint: The expert on units of measurement (knows exactly what a "kilowatt" is).
- Unicode Database: The expert on symbols and fonts (knows that a superscript "2" is a square, not just a number).
- RSLP: The expert on Portuguese grammar (knows that "potências" means "powers" in a technical context).
The Sorting Process: When TOTEN sees the text "5.2 kN/m²", it doesn't chop it up. It looks at its rulebook and the experts, and it says: "Ah, this whole thing is one single, atomic block called 'Physical Quantity'." It wraps the whole phrase in a single, labeled tag, preserving the exact meaning.

The Results: Why It Matters
The authors tested TOTEN against eight other top-tier systems (including the standard "pixelated" translators and other number-finding tools) using a benchmark of 800 engineering cases and four other real-world Portuguese text collections.

Atomicity (Keeping things whole): TOTEN was perfect at keeping physical quantities as single, unbroken units. The other systems often broke them apart.
Reconstruction (Putting it back together): When the computer needed to extract the actual number and unit later, TOTEN could do it correctly 78% to 90% of the time. The best competing system only managed about 63% to 70%.
Accuracy: The difference wasn't just a little bit better; it was statistically significant. TOTEN didn't just guess; it used its rulebook to get the right answer consistently.

The Bottom Line
TOTEN proves that for technical, scientific text, you don't need to rely on "statistical guessing" (which works well for casual conversation but fails at engineering). Instead, you can use a structured, rule-based approach that treats technical terms as complete, meaningful objects.

The paper claims that this method allows computers to "see" the structure of numbers and units exactly as engineers intend, without breaking them into confusing fragments. It is a specialized tool designed specifically to make Brazilian Portuguese technical text readable for machines, ensuring that "5.2 kN/m²" is treated as a single, coherent idea rather than a jumbled mess of characters.

Technical Summary: TOTEN – Knowledge-Based Ontological Tokenization of Physical Quantities in Brazilian Portuguese

1. Problem Statement

Current statistical tokenization algorithms (e.g., Byte-Pair Encoding, WordPiece, SentencePiece), which dominate Large Language Model (LLM) preprocessing, are optimized for vocabulary compression rather than semantic preservation. When applied to technical text in Brazilian Portuguese (PT-BR), these methods fragment semantically atomic entities—such as physical quantities, locale-specific numbers, compound dimensional units, and symbolic expressions—into arbitrary subword sequences. This fragmentation forces downstream models to reconstruct the original semantic structure a posteriori, a process that empirical studies suggest degrades numerical reasoning and arithmetic accuracy. Furthermore, existing quantitative extraction tools for English do not adequately model the specific technical vocabulary and normative identifiers (e.g., NBR, ABNT) of PT-BR, while generic Named Entity Recognition (NER) models ignore technical-scientific categories.

2. Methodology

The paper proposes TOTEN (Typed Ontological Tokenization), a framework that replaces statistical derivation with declarative classification grounded in a formal ontology.

2.1 Formalization

TOTEN is formalized as the triple $\langle O, \text{classify}, \{inst_\tau\} \rangle$ :

$O$ (Ontology of Engineering Entities - OEE): A formal ontology comprising a set of primary types ( $T$ ), structural principles ( $P$ ), composition relations ( $R$ ), and invariants ( $I$ ). Types include physical quantities, technical identifiers, formal operators, and hierarchical references. The ontology is governed by eight axioms (e.g., Intrinsicity, Mediated Composition, Categorical Error) ensuring that types are defined by intrinsic properties rather than corpus frequency.
$\text{classify}$ : A function that maps raw text ( $\Sigma^*$ ) into a set of typed regions $R_\Sigma$ , identifying spans of text and assigning them a type based on the ontology.
$\{inst_\tau\}$ : An indexed family of instantiators that map classified regions into a structured output language $M$ . This layer formats the data without re-deciding the type, adhering to the "single authority" principle.

2.2 Architecture and External Oracles

The framework couples the OEE with three consolidated external oracles to ensure robustness without manual enumeration:

Pint: Provides the dimensional authority for units of measure and SI prefixes.
Unicode Character Database (UCD): Handles typographic markers (superscripts, subscripts, mathematical operators) via character properties rather than character-by-character enumeration.
RSLP: A standard Portuguese morphological algorithm used to reduce inflections to lemmatic roots, helping to disambiguate single-letter units (e.g., distinguishing "K" in "temperatura" from "K" in "Kelvin").

2.3 Output Language

The output is a structured sequence of tags interleaved with unclassified residual text. Each tag contains a type identifier and attributes (e.g., value, unit, dim, locale). Crucially, the system preserves the literal form of the source text outside typed regions, ensuring that numerical representation (locale, separators, scientific notation) remains intact for downstream processing.

3. Key Contributions

The paper identifies three primary scientific contributions:

(C1) Formalization of Ontological Tokenization: Defining tokenization as a triple $\langle O, \text{classify}, \{inst_\tau\} \rangle$ that is implementation-independent and evaluable via verifiable properties, distinguishing it categorically from statistical subword tokenization.
(C2) The OEE Ontology: A formal ontology of engineering entities with primary types defined by intrinsic properties, eight structural axioms, and declared composition relations, designed to be extensible under the "open-for-extension, closed-for-modification" principle.
(C3) Intrinsic Evaluation Protocol: A computationally inexpensive evaluation based on four properties verifiable by construction:
1. Ontological Atomicity: Entities are emitted as single, indivisible tags with correct types.
2. Dimensional Equivalence: Dimensional vectors of equivalent quantities match.
3. Typographic Robustness: Semantically equivalent notational variants receive identical types.
4. Numerical Reconstruction: The ability to programmatically extract value and unit with high fidelity.

4. Experimental Results

The evaluation utilized an internally generated, physically validated benchmark (EngQuant, $N=800$ ) and four external PT-BR corpora (MMMLU PT_BR, BLUEX, ENEM Maritaca, Alvorada-Bench; total $N=1,771$ eligible cases). TOTEN was compared against eight state-of-the-art systems, including statistical tokenizers (BPE variants), specialized extractors (Quantulum3, CQE, GLiNER), dimensional libraries (Pint, udunits-2), and generic NER (spaCy).

Ontological Atomicity: TOTEN achieved 1.000 unit atomicity on the internal benchmark, significantly outperforming the best baseline (Quantulum3 at 0.297) and statistical tokenizers (approx. 0.48).
Numerical Reconstruction:
- Internal Benchmark: TOTEN scored 0.780, significantly higher than the best baseline CQE (0.340) and Quantulum3 (0.220).
- External Corpora: TOTEN maintained leadership across all four external datasets, achieving scores between 0.775 and 0.904. In contrast, the best baseline (Quantulum3) ranged from 0.627 to 0.703.
- Statistical significance was confirmed via McNemar tests with Holm correction ( $p < 0.001$ ).
Dimensional Equivalence: TOTEN achieved 0.968 conditional accuracy, showing statistical parity with the Pint oracle (0.985), confirming that the system inherits dimensional authority without distortion.
Typographic Robustness: TOTEN achieved 0.326 (global) and 0.325 (scoped), outperforming other systems that fragment entities upon encountering unanticipated notational variants.
Ablation Studies: Removing the Pint oracle caused a significant drop in recall and dimensional accuracy, confirming its role as the dimensional authority. Removing UCD or RSLP had negligible effects on the specific engineering benchmark used, validating their targeted utility for typographic and morphological ambiguities respectively.
Concurrent Validity: A strong Spearman rank correlation ( $\rho \geq 0.856$ ) between internal and external corpus rankings confirmed that the EngQuant benchmark is a faithful proxy for real-world performance.

5. Significance and Claims

The paper claims that TOTEN offers a categorical advantage over statistical tokenization by preserving the semantic structure of technical entities before they reach consumer models. Unlike statistical methods that rely on distributional properties, TOTEN is semantically committed to a conceptualization of the domain defined by the OEE.

Key claims include:

Recovery of Intrinsic Properties: The framework recovers properties (numerical structure, locale, dimensional vectors) that purely statistical pipelines systematically lose.
Neurosymbolic Compatibility: TOTEN positions itself as a symbolic pre-processing layer compatible with neurosymbolic architectures, producing a domain-semantically committed representation that can be consumed by both statistical models and symbolic agents.
Modest Scope: The authors explicitly state that the downstream effect of this representation on consumer model performance (e.g., improved arithmetic accuracy) is outside the scope of this study and left as future work. The evaluation is strictly intrinsic, focusing on the quality of the input representation itself.
Limitations: The system does not actively normalize typographically degraded notation (preserving literal form instead) and has partial coverage for dimensionless units by construction to avoid lexical collisions with Portuguese words.

In conclusion, the paper demonstrates that a knowledge-based, ontological approach to tokenization can achieve statistically significant improvements in atomicity and numerical reconstruction for Brazilian Portuguese technical text, providing a robust alternative to statistical compression methods for domains requiring high semantic fidelity.