Technical Summary: BeDiscovER

Problem Statement

While Large Language Models (LLMs) have demonstrated remarkable capabilities in reasoning across mathematics, coding, and common-sense tasks, their ability to process and understand discourse knowledge—the interplay of multi-sentence and long-context information—remains an open question. Previous evaluations of discourse understanding have been fragmented, utilizing disparate models, metrics (e.g., probing, prompting, acceptability judgment), and narrowly defined tasks (e.g., specific theoretical frameworks like PDTB or SDRT, or specific domains). This fragmentation prevents a holistic assessment of how state-of-the-art reasoning models handle discourse phenomena ranging from lexical particles to document-level coherence. Furthermore, existing benchmarks often lack coverage of novel challenges, such as the disambiguation of polyfunctional discourse particles, or fail to unify multilingual and multi-framework perspectives.

Methodology

The authors introduce BeDiscovER (Benchmark of Discourse Understanding in the Era of Reasoning Language Models), a comprehensive evaluation suite designed to assess discourse-level knowledge across three hierarchical levels: Lexicon, (Multi-)Sentences, and Document.

Benchmark Composition

BeDiscovER aggregates 52 publicly available datasets (totaling ~30,000 test instances) spanning 16 languages and 6 theoretical frameworks. The benchmark is organized into five core tasks:

1. Discourse Marker Understanding: Evaluates sensitivity to polyfunctional particles (e.g., "just," "otherwise") using newly curated datasets (Just-Manual, Just-Subtitle, Otherwise).
2. Temporal Reasoning: Tests event-event reasoning across adjacent and distant sentences, including arithmetic-style temporal inference (TimeBank-Dense, TDDiscourse, Test-of-Time).
3. Discourse Relation Recognition: Utilizes the DISRPT 2025 shared task data, which unifies over 300 relation types into 17 categories across 38 datasets and multiple frameworks (PDTB, RST, SDRT, etc.).
4. Sentence Ordering: Assesses the ability to reconstruct coherent text from shuffled sentences across scientific abstracts and narrative stories (e.g., AAN, ROC Stories, Wiki Movie Plots).
5. Dialogue Discourse Parsing: Focuses on constructing discourse structures (SDRT-style) for multi-party dialogues (STAC, Molweni, MSDC).

Evaluation Protocol

The authors evaluate a suite of reasoning-oriented LLMs (Qwen3 series, DeepSeek-R1, GPT-5-mini) alongside non-reasoning baselines (GPT-4o-mini, LLaMA-4-Scout, Qwen2.5-72B).

• Prompting Strategy: All tasks are cast into a unified open-ended Question Answering (QA) format. For classification tasks, label sets are explicitly provided in the system prompt.
• Reasoning Effort: The study distinguishes between "thinking" (high reasoning effort) and "non-thinking" (low reasoning effort) modes for models that support them.
• Metrics: Performance is measured primarily via accuracy or micro-F1 scores. The authors also analyze the correlation between sub-task performance and the overlap of underlying discourse knowledge types (lexical, temporal, rhetorical, etc.).

Key Results

The evaluation reveals distinct strengths and weaknesses in current reasoning models:

• Temporal Reasoning: Models exhibit strong performance in arithmetic-style temporal reasoning (e.g., Test-of-Time), where reasoning resembles mathematical calculation. However, they struggle with long-dependency reasoning and logical temporal inference in full documents (e.g., TimeBank-Dense, TDDiscourse), often underperforming compared to supervised baselines trained on smaller models.
• Discourse Relations: While reasoning models outperform non-reasoning counterparts, they still lag significantly behind supervised systems (by ~20 points) in rhetorical relation classification. Performance varies by framework, with newer frameworks (eRST) yielding lower scores, likely due to limited exposure in pretraining data.
• Discourse Markers: Providing richer context (definitions and examples) generally improves performance for large reasoning models (>32B), suggesting they benefit from in-context learning. However, excessive context can degrade performance for smaller models.
• Document Structure: Larger models show a clear scaling trend in sentence ordering and dialogue parsing, achieving competitive results on narrative coherence tasks, though full structure prediction in dialogue parsing remains a bottleneck.
• Reasoning Effort: Increased reasoning effort (longer "thinking" traces) consistently improves performance on arithmetic temporal tasks but does not guarantee better outcomes on other tasks; in some cases, it leads to verbosity without improved accuracy.

Key Contributions

1. Unified Benchmark: BeDiscovER is the first comprehensive benchmark to evaluate discourse understanding across lexical, sentential, and document levels, covering 52 datasets and multiple frameworks.
2. Novel Tasks: It introduces evaluations for discourse particle disambiguation and aggregates the multilingual/multi-framework DISRPT 2025 relation classification task.
3. Empirical Analysis: The paper provides a detailed analysis of how reasoning effort, model size, and knowledge overlap influence performance, identifying that current models excel in data-rich, arithmetic-like reasoning but struggle with subtle semantic phenomena and long-range dependencies.
4. Open Resources: The authors release a unified evaluation pipeline to facilitate future assessments and continuous benchmarking.

Significance and Claims

The authors position BeDiscovER as a necessary tool to diagnose the specific limitations of reasoning-oriented LLMs in discourse understanding. They claim that while state-of-the-art models show significant reasoning abilities, these strengths are often concentrated in phenomena well-represented in training data (e.g., frequent particles, arithmetic logic). The benchmark highlights persistent gaps in logical temporal reasoning, rhetorical relation classification, and long-context coherence.

The paper concludes that BeDiscovER serves as a baseline for future research, suggesting that improving discourse-aware LLMs requires addressing the transfer of abstract reasoning skills and potentially leveraging high-quality reasoning traces in pre- and post-training strategies. The authors explicitly note that BeDiscovER currently assesses discourse knowledge (answer correctness) rather than reasoning quality directly, acknowledging this as a limitation for future work.