Double Triangle Annotation: A Scalable Human-in-the-Loop Framework for High-Precision Historical Document Annotation

Technical Summary: Double Triangle Annotation Framework

Problem Statement
The extraction of structured information from historical documents (e.g., medical directories, census records) is critical for digital humanities research but faces a bottleneck in producing high-precision ground-truth annotations. Traditional independent manual annotation is prohibitively expensive and difficult to scale. Conversely, fully automated pipelines relying on single Multimodal Large Language Models (MLLMs) are prone to hallucinations and lack the reliability required for gold-standard datasets. Existing human-in-the-loop approaches often rely on single-model confidence scores, which can introduce anchoring bias and fail to detect systematic errors when the model is overconfident. The core challenge is verifying model outputs efficiently without incurring the full cost of manual verification for every data point.

Methodology: Double Triangle Annotation
The authors propose Double Triangle Annotation, a two-layer human-in-the-loop framework that leverages cross-model consensus to automate the majority of annotation work while reserving human effort strictly for ambiguous or conflicting cases. The framework operates under a single assumption: error independence between architecturally distinct models.

• Layer 1: The Annotation Triangle

  • Architecture: Two independent machine annotators ($M_1, M_2$) process each document in parallel. A human jury ($H$) acts solely as a conflict resolver, not a primary producer.
  • Workflow:
    1. Parallel Inference: $M_1$ and $M_2$ generate provisional labels.
    2. Consensus Filtering: If $M_1$ and $M_2$ agree, the label is auto-accepted. The framework posits that the probability of two independent models making the same error on the same field is negligible ($P \le \epsilon_1 \cdot \epsilon_2$).
    3. Human Adjudication: If the models disagree, the case is routed to the human jury, who reviews the image and both predictions to determine the final label.
  • Independence Strategy: To ensure error independence, the framework employs "cross-family" (different providers, e.g., Anthropic vs. Qwen) or "cross-modal" (OCR vs. MLLM) model pairings.

• Layer 2: The Validation Triangle

  • Architecture: Two independent Layer 1 systems ($S_a, S_b$) are treated as single annotator units. A final domain expert ($R$) acts as the ultimate arbiter.
  • Workflow:
    1. Dual Generation: $S_a$ and $S_b$ independently process the data to produce refined labels ($L_a, L_b$).
    2. Meta-Comparison: If $L_a$ and $L_b$ match, the label is designated as "Gold Standard."
    3. Expert Resolution: Disagreements between the two systems are escalated to the domain expert.
  • Objective: This layer mitigates human fallibility (fatigue, subjectivity) by catching errors that a single human jury might miss, further reducing the residual error rate toward zero.

Key Contributions

1. Framework Design: Introduction of a two-layer paradigm that shifts the human role from producer to verifier, automating the majority of annotation via consensus while strictly reserving human effort for ambiguous cases.
2. Minimal Assumption Design: The framework requires no task-specific calibration or distributional priors, relying only on the independence of model errors. It is applicable to any structured extraction task where parallel model execution is feasible.
3. Future-Proof Scalability: The architecture is designed to improve autonomously as model capabilities increase; higher model accuracy leads to higher consensus rates, further reducing human effort.
4. Benchmark Release: Construction and release of the first structured-extraction ground truth for the Guides Rosenwald (French medical directories, 1887–1906), comprising 60 document columns and 13,595 annotated fields.

Experimental Results
The framework was evaluated on the Guides Rosenwald corpus.

• Accuracy: On a controlled evaluation of 380 fields, the framework achieved a final Word Error Rate (WER) of 0.003 and a Character Error Rate (CER) of 0.001.
  • Layer 1 juries corrected 18.4% and 26.3% of fields respectively.
  • The final reviewer (Layer 2) corrected an additional 4.2% of fields.
• Efficiency:
  • In the controlled evaluation, model consensus auto-accepted over 73% of fields.
  • In the scaled benchmark (13,595 fields), the auto-accept rate rose to over 85%.
  • The final reviewer only needed to correct 7.2% of fields in the scaled benchmark.
• Error Analysis: The two residual errors in the final output were traced to genuine visual ambiguity in the source images (e.g., faint diacritics or line-break artifacts). In these cases, the degraded input induced correlated errors across both humans and machines, violating the independence assumption. The authors note this as an irreducible failure mode for consensus-based mechanisms.

Significance
The paper claims that Double Triangle Annotation offers a scalable solution to the high-cost bottleneck of historical document annotation. By replacing single-model confidence filtering with cross-model consensus, the framework effectively mitigates hallucination and anchoring bias. It demonstrates that high-precision ground truth can be achieved at a fraction of the cost of manual annotation, with the system becoming increasingly autonomous as underlying models improve. The release of the Rosenwald Guides benchmark provides a new resource for future research in historical document processing.

Limitations Acknowledged by the Authors

• Single-Page Evaluation: Detailed error analysis relied on a single page (380 fields), though the benchmark construction demonstrated scalability.
• Source Ambiguity: The framework cannot detect errors when the source image is so degraded that all annotators (human and machine) converge on the same incorrect interpretation.
• Annotator Overlap: In the experimental setup, the final reviewer was also one of the Layer 1 juries, which may weaken the independence between layers (though the authors argue this provides a conservative upper bound on error rates).
• Filtering Bias: The benchmark construction involved filtering out columns with low model agreement, potentially biasing the dataset toward easier cases.
• Model Scope: Experiments utilized only MLLM pairs; cross-modal configurations (MLLM + traditional OCR) were proposed but not evaluated.