Technical Summary: MMLongBench-Doc-V2

Problem Statement

MMLongBench-Doc (Ma et al., 2024) is a benchmark for long-document question answering (QA) comprising 1,082 questions across 135 PDFs. While designed to test document understanding and suppress hallucination through "unanswerable" questions, the original benchmark (V1) suffers from two critical flaws that distort performance measurement:

1. String-Metric Limitations: V1 evaluates answers by comparing extracted strings against reference keys. This penalizes correct semantic answers for minor formatting differences (e.g., 1,358,000 vs. 1358000, or Operating Activities vs. Operations activities). This issue disproportionately affects systems answering list-based or free-text questions, distorting relative rankings rather than just absolute scores. Additionally, six questions were technically unscoreable because their reference answers violated their own declared answer_format constraints (e.g., parsing errors).
2. Systematic Annotation Errors: A non-trivial portion of ground-truth annotations are incorrect, ambiguous, or incomplete. These errors are not random; they are concentrated in questions that capable systems answer correctly but are scored as zero because the system's defensible answer conflicts with a flawed key. This creates an artificial ceiling on benchmark performance that mimics a model limitation rather than a dataset issue.

Methodology

MMLongBench-Doc-V2 addresses these issues through a rigorous revision process involving metric replacement, annotation correction, and a formal decision procedure for edge cases.

1. Metric Revision: The Pinned LLM Judge

The string-identity metric is replaced with a "pinned LLM judge."

• Input: The judge receives the question, the reference answer, the expected format, and the system's full response text (not a pre-extracted string).
• Constraint: The judge is explicitly instructed not to read the source document. It treats the reference answer as ground truth and determines only if the system's response conveys the same meaning.
• Rubric: Wording, case, units, separators, and list order are flexible. However, missing/extra list members, different values, or a refusal to answer (when an answer exists) are penalized. For "Not answerable" references, only a clean decline is accepted; volunteering a nearest figure counts as an error.
• Stability: The judge model, reasoning effort, and JSON schema are fixed ("pinned") to ensure reproducibility, though the model choice remains a free parameter for the metric.

2. Annotation Corrections

The author audited 1,082 questions, identifying and correcting 106 annotations.

• Correction Types: Changes include fixing wrong answers (26), repairing defective questions (19), clarifying ambiguous questions (19), completing incomplete answers (16), and reclassifying "Not answerable" items as answerable (14).
• Corpus Defects: Ten questions targeting documents shipped under incorrect filenames were removed entirely, as no system could answer them. One duplicate question with contradictory keys was also removed.
• Documentation: Every correction is published with the original annotation, the corrected version, and specific evidence (page numbers or arithmetic) to allow independent verification.

3. Decision Procedure for "Empty Set" Keys

A specific challenge involved questions keyed as 0 (e.g., "how many cats?") versus "Not answerable." The author established a decision procedure to determine when an "empty set" key should be widened to "Not answerable" without destroying the deliberate negative sample nature of the benchmark.

• Widening Criteria: A key is widened only if the container exists (e.g., a specific page range) and the target entity is verifiably absent. In this case, "none" or "0" is a valid reading of the document.
• Non-Widening Criteria: Widening is prohibited if:
  • The container itself is missing (e.g., non-existent page ranges).
  • The entity never appears in the document's scope (e.g., a specific year not covered).
  • The question asks for a complement against an unbounded universe (e.g., "icons not found").
  • The set is short but not empty (e.g., asking for two items when only one exists).
• Verification: This process relies on executable emptiness checks rather than reviewer intuition to prevent silent errors.

Key Contributions and Results

• Corrected Corpus: V2 contains 1,071 questions over 134 documents (down from 1,082/135).
• Annotation Record: A public record of 106 corrections with supporting evidence is provided.
• Evaluation Harness: An open-source evaluation harness is released that implements the pinned LLM judge.
• Unanswerable Slice: 208 questions (19.4%) remain unanswerable, with 14 keys widened according to the new decision procedure.
• Scoring Impact: V2 scores are not comparable to V1 numbers. Scores are generally higher for the same systems, with the largest gains observed in list and free-text tasks where the string metric previously penalized formatting variations.

Significance and Limitations

The paper positions MMLongBench-Doc-V2 as a necessary repair rather than a replacement for the original benchmark. Its primary significance lies in:

• Restoring Measurement Validity: By decoupling scoring from string identity and correcting systematic annotation errors, the benchmark better reflects a model's actual document understanding capabilities.
• Preserving the "Unanswerable" Design: The benchmark remains one of the few suites where confidently answering an unsupported question incurs a penalty, a feature the author argues is essential for detecting hallucination.
• Methodological Rigor: The introduction of a formal decision procedure for "empty set" keys and the transparent publication of correction records set a standard for future benchmark maintenance.

Limitations:

• Audit Bias: Corrections were derived from questions that a specific system answered confidently but scored zero. Consequently, the overall error rate of the corpus remains unmeasured; a uniform random audit was not performed.
• Task Type Coverage: Verification was thorough for lookup, derive, compare, and verify tasks but less complete for count and enumerate tasks, where roughly 95 rows remain where a capable system disagrees with the key but the error could not be proven.
• Automated Labels: Task-type and requires_visual labels were generated by a single model pass and are not human-verified ground truth.
• Judge Fallibility: While the judge is pinned and explicitly ruled, it is still a model and not infallible.

The author concludes that the corrected corpus, per-entry records, and evaluation harness are available under Apache 2.0, inviting further corrections via issues that cite specific pages and entries.