Technical Summary: How Much Do LLMs Know About Chinese Zero Pronouns?

Problem Statement

Zero Pronouns (ZPs) represent a pervasive linguistic phenomenon in pro-drop languages like Chinese, where overt pronouns are frequently omitted but semantically recoverable from discourse context. While Large Language Models (LLMs) have demonstrated strong proficiency in various Chinese language tasks, their specific capability to process ZPs remains poorly understood. Prior computational research largely focused on downstream tasks such as ZP resolution (selecting antecedents) and translation, often neglecting upstream tasks like identifying the positions of ZPs or determining their referentiality. This study addresses the gap in understanding the extent to which current LLMs can comprehensively handle the full pipeline of Chinese ZP processing, from detection to translation.

Methodology

The authors conduct a systematic investigation using the OntoNotes 5.0 dataset (Chinese portion), which provides rich annotations for zero pronouns and coreference chains. The study decomposes ZP understanding into five linguistically motivated tasks arranged in a sequential pipeline:

1. ZP Identification: Predicting the positions of all zero pronouns within a discourse.
2. Referentiality Classification: Determining whether an identified ZP is referential (having an antecedent or being deictic) or non-referential (e.g., arbitrary omission).
3. Referential Type Classification: For referential ZPs, classifying them as anaphoric (antecedent precedes), cataphoric (antecedent follows), or deictic (context-dependent).
4. ZP Resolution: Linking referential ZPs to their antecedents, evaluated under three settings:
   • Multiple-Choice Question (MCQ): Selecting from candidate antecedents.
   • Question-Answering (QA): Generating the antecedent directly.
   • Clustering: Grouping all mentions and ZPs into coreference clusters.
5. ZP Translation: Translating Chinese discourse containing unmarked ZPs into English, evaluating the accuracy of pronoun insertion.

The evaluation covers a diverse set of LLMs, ranging from small models (4B parameters) to large models (235B+ parameters), including base models (e.g., Qwen3, GPT-4.1) and reasoning-oriented models (e.g., DeepSeek-R1, GPT-o4). Evaluation metrics include Precision, Recall, F1-score, Accuracy, and Borda counts for cross-task comparison. Special attention is paid to evaluation protocols tailored for character-based languages and human evaluation for translation accuracy due to the limitations of automatic metrics.

Key Contributions and Results

1. Systematic Evaluation Pipeline

The paper introduces a comprehensive framework that treats ZP processing as a multi-stage pipeline rather than a single task. This allows for a granular analysis of where LLMs fail, distinguishing between upstream difficulties (detection) and downstream failures (resolution/translation).

2. Performance Findings

• Overall Difficulty: Chinese ZPs remain highly challenging for current LLMs. Across all tasks, performance generally hovers between 20% and 40%, significantly lower than performance on other NLP tasks.
• Upstream Challenges: Upstream tasks, particularly ZP Identification and Referentiality Classification, are the most difficult. Smaller models (e.g., Qwen3-4B) achieve F1 scores below 1% for identification. Most models tend to over-predict referentiality, labeling nearly all ZPs as referential (F1 > 70% on referential instances but < 10% on non-referential).
• Resolution Performance: LLMs achieve only 20–30% accuracy across resolution tasks.
  • MCQ-based resolution performs surprisingly poorly, especially on conversational genres (Broadcast Conversation, Telephone Conversation), due to a bias toward selecting proper names over pronouns.
  • Clustering-based resolution yields very low recall, as models struggle to link all mentions within a chain.
• Translation Accuracy: Even state-of-the-art reasoning-oriented models correctly translate fewer than half of Chinese ZPs into English. Performance on translation does not significantly improve with larger model capacity or stronger reasoning capabilities when given plain text.
• Impact of Explicit Information: When ZP locations are explicitly marked (Oracle setting), translation accuracy improves substantially (up to ~79% for reasoning models). This indicates that the primary bottleneck is often the failure to locate and identify the ZP rather than the inability to resolve it once found.

3. Correlations and Analysis

• Task Interdependence: There is a strong correlation (r = 0.964) between ZP Identification and ZP-aware Translation, suggesting that the ability to locate ZPs is a prerequisite for accurate translation.
• Reasoning Capabilities: Reasoning-oriented models (e.g., GPT-o4-mini, DeepSeek-R1) generally outperform base models, particularly in identification and referential type classification. However, this advantage does not fully close the performance gap.
• Context Size: Increasing context window size (up to 4 sentences) has mixed effects; it improves identification for capable models but can degrade performance for others or in clustering tasks due to increased noise.

Significance and Claims

The paper claims that current LLMs, despite their general language proficiency, lack the robust discourse-level reasoning required to comprehensively process Chinese zero pronouns. The study highlights that:

• Upstream tasks are the bottleneck: The inability to accurately identify ZPs and distinguish referential from non-referential omissions severely limits downstream performance.
• Translation is not a proxy for understanding: High translation scores in general benchmarks do not guarantee the ability to handle implicit pronouns, as evidenced by the low ZP-specific translation accuracy.
• Need for specialized evaluation: The proposed task suite serves as a benchmark for evaluating discourse-level reasoning in Chinese, revealing that "reasoning" capabilities in current models are insufficient for the specific syntactic and pragmatic demands of pro-drop languages.

The authors conclude that while larger and reasoning-oriented models show promise, the gap between current LLM capabilities and human-level understanding of Chinese ZPs remains significant, necessitating further research into discourse-level reasoning mechanisms.