Specializing Large Models for Oracle Bone Script Interpretation via Component-Grounded Multimodal Knowledge Augmentation

1. Problem Statement

Oracle Bone Script (OBS) is the earliest known mature writing system in China, holding immense historical value. However, deciphering it is exceptionally difficult due to:

• Low Decipherment Rate: Only ~33% of the 4,500+ identified characters have been deciphered.
• Structural Complexity: OBS is a pictographic system where characters are composed of recurring semantic components (radicals), but existing AI approaches treat decipherment as a closed-set image recognition problem.
• The "Interpretation Gap": Current Vision-Language Models (VLMs) fail to bridge the gap between visual recognition and linguistic reasoning. They often suffer from "visual hallucinations," lack domain-specific knowledge, and cannot generalize to unseen characters because they ignore the structural logic of component composition.

2. Methodology

The authors propose an Agentic Retrieval-Augmented Generation (Agentic RAG) framework that shifts the paradigm from pure visual recognition to component-grounded reasoning. The pipeline consists of four integrated modules:

A. Component Identification Module

• Architecture: Utilizes a Vision Transformer (ViT) based on DINOv2 to extract visual features, followed by a Prototypical Network classifier.
• Function: Identifies radical components within an input OBS image by mapping them to a prototype feature space. This is crucial because the system relies on recognizing known components to infer the meaning of unknown characters.
• Advantage: The prototype-based approach is robust in low-resource settings, handling the scarcity of labeled component data better than standard classifiers.

B. Agent-Driven Graph Knowledge Retrieval

• Knowledge Graph (KG): Built upon the new OB-Radix dataset, the KG connects characters, their constituent components, and expert-verified semantic explanations.
• Agent Orchestration: A tool-using LLM agent dynamically queries the KG.
  • Component-Centric Retrieval: Queries explanations for predicted components and finds other characters containing them.
  • Constrained Synthesis: If evidence is weak, the agent performs internal variant lookups and modern-OBS mapping without external tools, summarizing all evidence into a concise bundle.
• Caching: A semantic-similarity cache is used to improve efficiency for repeated queries.

C. Component Relationship Inference

• Task: The system uses a VLM to analyze the structural relationships between identified components (e.g., determining if a character is ideographic, pictographic, or phono-semantic).
• Output: Generates a reasoning trace explaining how components interact to form meaning (e.g., "A hand comforting a woman suggests 'comfort'").

D. Interpretation Generation

The framework supports two inference modes:

1. VLM Inference: Directly fuses visual embeddings, component predictions, and retrieved semantic prompts to generate an interpretation.
2. Multi-Agent Inference: Decouples the process into two specialized agents:
   • Knowledge Retrieval Agent: Executes graph queries.
   • Semantic Reasoning Agent: Synthesizes retrieved evidence with visual cues to produce the final human-interpretable explanation. This separation reduces error propagation and enhances reasoning fluency.

3. Key Contributions

A. OB-Radix Dataset

A novel, expert-annotated dataset designed specifically for component-level analysis:

• Scale: 1,022 character images (934 unique) and 1,853 fine-grained component images (478 distinct components).
• Annotation Quality: Annotated by archaeology PhD students based on paleographic function rather than just visual salience. It includes bounding boxes, semantic explanations, and verified inter-component relations.
• Significance: Unlike previous datasets (e.g., HUST-OBC, EVOBC) that focus on whole-character recognition, OB-Radix provides the structural decomposition necessary for reasoning.

B. Framework Innovation

• Reformulated OBS decipherment as a structure-aware reasoning task rather than a classification task.
• Introduced a Component-Grounded Multimodal approach that integrates visual cues with graph-based knowledge retrieval.

C. Evaluation Benchmarks

Designed three progressive tasks to evaluate the system:

1. Component-level retrieval.
2. Component relationship inference (type classification + reasoning).
3. Full OBS interpretation generation.

4. Experimental Results

The framework was evaluated against baseline VLMs (GPT-5, Claude Opus 4.1, GLM-4.5V, Qwen3-VL) across three tasks:

• Component Identification: Achieved Top-1 accuracy of 77.95%, demonstrating effective recognition of structural units.
• Relationship Inference: The proposed method significantly outperformed baselines. For example, Qwen3-VL improved classification accuracy from 35.0% (baseline) to 59.9%. Reasoning quality (measured by BERTScore and LLM-as-a-Judge) also saw substantial gains (e.g., +0.206 LLM-Judge score for Qwen).
• Interpretation Generation:
  • RAG vs. Baseline: The Agentic RAG pipeline consistently outperformed direct generation. Qwen3-VL improved ROUGE-1 from 0.264 to 0.354 and LLM-Judge scores from 0.076 to 0.339.
  • Multi-Agent vs. Single-Agent: The multi-agent configuration (Retriever + Reasoner) achieved the highest scores, with GPT-5 as the Reasoner reaching an LLM-Judge score of 0.657.
• Human Evaluation: Two archaeology PhDs evaluated the outputs using a 5-point Likert scale. The Multi-Agent pipeline scored 3.43, significantly higher than the KG-RAG (2.13) and Baseline (1.37) pipelines. Inter-rater reliability was high (ICC3 = 0.71).
• Ablation Study: Removing the retrieval component caused consistent performance drops across all metrics, confirming that the Knowledge Graph provides essential semantic context beyond visual recognition.

5. Significance and Impact

• Bridging the Gap: The work successfully bridges the gap between AI image recognition and expert paleographic reasoning, moving beyond "black-box" classification to interpretable, logic-driven decipherment.
• Data-Centric AI: By introducing OB-Radix, the paper addresses the critical lack of structured, component-level data in the field, enabling future research in structural decomposition.
• Scalability: The component-grounded approach allows the system to infer meanings for unseen characters by analyzing their known constituent parts, a capability essential for deciphering the remaining ~66% of undeciphered OBS.
• Methodological Blueprint: The Agentic RAG framework serves as a blueprint for applying Large Models to other low-resource, highly specialized domains where structural logic and external knowledge are paramount.

Limitations & Future Work

• Component Recognition: The system is not yet perfect; occasional spurious component detection can occur.
• Knowledge Coverage: Many characters still lack widely accepted interpretations, limiting the ground truth for automated analysis.
• Dependency: The current system relies heavily on external retrieval; future work aims to reduce this dependency as base models acquire stronger intrinsic domain knowledge.
• Phono-semantic Compounds: Handling complex phonetic-semantic compounds remains a challenge.