Technical Summary: OracleAnalyser

1. Problem Statement

The decipherment of Oracle Bone Scripts (OBS), China's earliest mature writing system, remains a significant challenge in archaeology and philology. While over 4,500 characters have been identified, only approximately 1,600 are deciphered. Existing artificial intelligence approaches to this domain, including deep learning models and current benchmarks, are predominantly focused on recognition tasks (identifying the character). These methods often overlook the equally crucial aspect of analytical reasoning—the process of interpreting the pictographic structure and semantic meaning of the script. Current benchmarks lack metrics to evaluate a model's ability to generate intermediate analytical reasoning, limiting the interpretability and effectiveness of AI in deciphering unknown characters.

2. Methodology

The authors propose OracleAnalyser, a reasoning framework for oracle bone analysis based on post-training techniques applied to Multimodal Large Language Models (MLLMs). The framework operates through three primary stages:

A. Dataset Construction

• Oracle Bone Reasoning Dataset: Constructed from the EVOBC dataset, filtered to include 2,807 valid characters. The authors utilize Qwen-VL-MAX to generate analytical descriptions based on pictographic perspectives, conditioned on the corresponding modern Chinese character to ensure accuracy. These descriptions are refined by Qwen-3 and manually verified by experts, resulting in a dataset of approximately 5,500 samples (5,000 for training, 500 for in-domain testing, and 500 for out-of-domain testing).
• Oracle Bone Preference Dataset: Generated by using the supervised fine-tuned model (OracleAnalyser-SFT) to perform multiple inference traces on oracle bone images. Positive samples are ground-truth analyses, while negative samples are selected from the model's own systematic errors (incorrect recognition traces). This yields over 3,400 high-quality preference pairs after manual filtering.

B. Training Pipeline

1. Supervised Fine-Tuning (SFT): The base model, Qwen2.5-VL-3B-Instruct, is fine-tuned on the reasoning dataset. This stage equips the model with fundamental visual pattern recognition and analytical reasoning abilities, serving as a stable initialization for subsequent steps.
2. Stable Focal Preference Optimization (SFPO): To address the limitations of standard Direct Preference Optimization (DPO) in this domain (specifically, the risk of overfitting noisy preferences and the tendency to over-interpret hard-to-rank pairs), the authors propose SFPO.
   • Focal Component ($L_F$): Adapts the Focal Loss concept to down-weight "hard-to-rank" preference pairs where the model struggles to distinguish between correct and incorrect reasoning, preventing the model from learning spurious distinctions.
   • Stability Component ($L_S$): Introduces a reverse Kullback-Leibler (KL) divergence term to constrain the divergence between the policy model ($\pi_\theta$) and the reference model ($\pi_{ref}$), ensuring stable training and preventing excessive drift.
   • The final loss function combines these components: $L_{SFPO} = L_F + \lambda L_S$.

C. Evaluation Benchmark

The authors introduce a new benchmark comprising in-domain and out-of-domain evaluations. Unlike previous benchmarks that focus solely on recognition accuracy, this benchmark evaluates:

• Analytical Quality: Measured via BLEU-1, BLEU-4, ROUGE-1, and ROUGE-L scores against ground-truth analysis.
• Recognition Accuracy: For in-domain tasks.
• Sentence-BERT Score (SBS): For out-of-domain tasks, measuring the semantic similarity between the model's generated analysis and the ground truth to evaluate generalization on unseen characters.

3. Key Contributions

• Datasets and Benchmark: The release of the first oracle bone reasoning dataset, the first oracle bone preference dataset, and a new benchmark designed to systematically evaluate analytical capabilities rather than just recognition.
• OracleAnalyser Framework: A novel framework integrating explicit reasoning with multiple post-training strategies (SFT and SFPO) specifically tailored for oracle bone analysis.
• SFPO Algorithm: A new preference optimization algorithm that stabilizes training and focuses on valuable samples by mitigating the negative effects of noisy and ambiguous preference pairs.
• Performance: Demonstration that a 3B parameter model can achieve superior analytical performance compared to significantly larger models (up to 241B parameters) and specialized recognition models.

4. Experimental Results

Extensive experiments were conducted on both in-domain and out-of-domain test sets:

• In-Domain Performance: OracleAnalyser achieved a recognition accuracy of 39.6% and significantly outperformed all competing models (including InternVL-3.5-241B and specialized models like OBSD) on analytical metrics (BLEU and ROUGE). For instance, it achieved a BLEU-4 score of 72.77 compared to the next best general MLLM's score of ~4.32.
• Out-of-Domain Generalization: On unseen characters, OracleAnalyser achieved the highest Sentence-BERT Score (70.35), indicating its generated analyses are semantically closer to ground truth than those of larger models.
• Ablation Studies:
  • Data Filtering: Removing ambiguous preference pairs improved performance, confirming that data quality is more critical than quantity.
  • Training Strategy: The combination of SFT, Focal Loss ($L_F$), and Stability Loss ($L_S$) (i.e., SFPO) yielded the best results. Standard DPO provided limited improvement, and omitting the stability term ($L_S$) led to training collapse in later stages.
• Visual Analysis: Qualitative results show the model can accurately decompose glyphs and associate components with real-world entities, demonstrating learned pictographic analysis capabilities.

5. Significance

The paper claims that OracleAnalyser represents a shift from pure recognition to analytical reasoning in the field of oracle bone script research. By leveraging post-training techniques and a novel optimization algorithm, the framework enables a relatively small model (3B parameters) to surpass much larger models in decipherment tasks. The authors argue that this approach provides a more interpretable and effective solution for cultural heritage preservation, offering a method to narrow the search space for paleographers when facing unknown scripts. The work is presented as the first to integrate explicit reasoning with multiple post-training strategies specifically for oracle bone analysis.