Technical Summary: A Reproducible Multi-Architecture Baseline for Token-Level Chinese Metaphor Identification

Problem Statement

While metaphor is pervasive in language, token-level computational identification of metaphor-related words in Chinese under the Metaphor Identification Procedure VU (MIPVU) framework remains under-explored compared to English. Existing English research has established strong baselines using Transformer-based sequence labelers and specialized architectures like MelBERT. However, for Chinese, the PSU Chinese Metaphor Corpus (PSU CMC)—the only widely available MIPVU-annotated corpus—has not been systematically evaluated with modern encoder fine-tuning, MelBERT-style lexical fusion, or instruction-tuned Large Language Models (LLMs) under a shared, reproducible protocol. Furthermore, a critical resource gap exists: MelBERT relies on basic-meaning representations (typically WordNet glosses for English), but no equivalent open Chinese resource was available.

Methodology

The paper establishes a reproducible baseline by comparing three distinct model families on the PSU CMC, using a fixed file-level 70/10/20 train/dev/test split across five random seeds.

1. Resource Construction: MCD7 Basic-Meaning Embeddings

To enable MelBERT adaptation for Chinese, the authors constructed a basic-meaning resource from the Modern Chinese Dictionary, 7th edition (MCD7).

• Pipeline: 74,823 dictionary entries were parsed to extract the most concrete/primary sense (typically the first definition). Cross-references were recursively resolved.
• Encoding: Basic-meaning texts were encoded into 1024-dimensional vectors using Chinese RoBERTa-wwm-ext-large.
• Coverage: The resource covers 71.51% of the unique tokens in PSU CMC. Out-of-vocabulary tokens receive a zero-vector fallback.
• Release: Embeddings and pipelines are released; original gloss text is not redistributed due to copyright.

2. Model Architectures

The study evaluates four configurations across three model families:

• Encoder Fine-tuning (RoBERTa): Standard token classification using Chinese RoBERTa-wwm-ext-large with a linear head.
• MelBERT Adaptation: An extension of RoBERTa incorporating two parallel channels grounded in metaphor theory:
  • MIP Channel: Contrasts contextual representation against the MCD7-derived basic-meaning vector.
  • SPV Channel (Selectional Preference Violation): Contrasts local context with token representation to flag anomalies.
  • Variants: Full (MIP + SPV + Fusion), MIP-only, and SPV-only.
• Instruction-Tuned Generation (Qwen): Qwen3.5-9B fine-tuned with QLoRA (4-bit quantization + low-rank adapters). The task is framed as a generation problem where the model outputs a structured JSON list of metaphor tokens. Six task formulations were tested, with "Generative JSON Extraction" (Q2) selected as the best.

3. Experimental Protocol

• Metric: Positive-class F1 (Test pos-F1) is the primary metric due to class imbalance (9.16% positive tokens).
• Seeds: Five fixed seeds ({42, 123, 2024, 7, 31415}) are used for all models to ensure fair comparison.
• Hardware: Single NVIDIA RTX 5090.

Key Results

Performance Comparison

• Best Performer: MelBERT (MIP-only) achieved the highest performance with a test positive F1 of 0.7281 ± 0.0050.
• Comparison:
  • MelBERT Full (MIP+SPV): 0.7270 ± 0.0069 (marginally lower than MIP-only).
  • Plain RoBERTa: 0.7142 ± 0.0121.
  • Qwen3.5-9B (QLoRA): 0.6157 ± 0.0113.
• Generative Gap: The Qwen generative baseline trails encoder-based models by approximately 11 absolute F1 points.

Key Findings

1. SPV Channel Ineffectiveness in Chinese: Contrary to English results where SPV aids performance, the SPV channel in Chinese provides no reliable positive signal. The MIP-only configuration outperforms the Full configuration and exhibits significantly lower seed variance (0.0050 vs. 0.0069). The authors hypothesize this is due to the dominance of conventional metaphors in Chinese, where metaphor-context pairings are frequent and do not trigger selectional violations.
2. Precision-Recall Asymmetry in LLMs: The Qwen baseline suffers primarily from low recall (0.5526) compared to encoders, while maintaining comparable precision. This suggests generative models make discrete commitments that cause them to drop ambiguous or borderline cases, a limitation exacerbated by low-rank adaptation (QLoRA).
3. Format Sensitivity: Several Qwen task formulations failed due to design rather than capacity. Specifically, BIO tagging (Q4) failed because Chinese metaphors in PSU CMC are predominantly single-token (making the "I-" tag useless), and structured generation with short context windows (Q8 v1) failed due to supervision-token truncation.
4. Register Difficulty: Performance varies by register, with fiction being the most difficult (lowest F1, highest variance) and academic being the easiest. This correlates with metaphor density and novelty; fiction contains more novel, context-dependent metaphors.

Contributions and Significance

The paper claims three primary contributions:

1. A Reproducible Baseline: It provides the first systematic, multi-architecture baseline for token-level Chinese metaphor identification on PSU CMC, including full code, split manifests, and per-seed outputs.
2. MCD7 Basic-Meaning Resource: It introduces a novel resource derived from the authoritative Modern Chinese Dictionary, enabling MelBERT-style lexical fusion in Chinese for the first time.
3. Empirical Insights:
   • It demonstrates that the SPV channel is not beneficial for Chinese metaphor identification under MIPVU.
   • It highlights the specific limitations of current instruction-tuned LLMs (specifically QLoRA-adapted Qwen) for fine-grained token-level tasks, noting a precision-favoring bias.
   • It identifies that generative task formulation failures are often due to format design (e.g., BIO tags for single-token phenomena) rather than model capacity.

The authors position this work as a "common reference point" for future research, urging that future systems compare against these specific configurations and seed protocols rather than informal estimates. They modestly note that while they do not expect QLoRA-tuned LLMs to close the gap with encoders immediately, future work could explore multi-sense MelBERT integration or hybrid paradigms combining encoder precision with LLM recall.