💬 NLP

Cross-Examination Framework: A Task-Agnostic Diagnostic for Information Fidelity in Text-to-Text Generation

This paper introduces the Cross-Examination Framework (CEF), a reference-free, multi-dimensional diagnostic tool that evaluates text-to-text generation fidelity by treating source and candidate texts as independent knowledge bases to generate interpretable scores for coverage, conformity, and consistency, thereby outperforming traditional metrics in identifying semantic errors across diverse tasks.

Original authors: Tathagata Raha, Clement Christophe, Nada Saadi, Hamza A Javed, Marco AF Pimentel, Ronnie Rajan, Praveenkumar Kanithi

Published 2026-01-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tathagata Raha, Clement Christophe, Nada Saadi, Hamza A Javed, Marco AF Pimentel, Ronnie Rajan, Praveenkumar Kanithi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher grading a student's essay. Traditionally, you might use a ruler to measure how many words the student used that match the textbook (like BLEU or ROUGE scores). But what if the student wrote the exact same ideas using completely different words? The ruler says "bad grade," even though the meaning is perfect. Or, what if the student copied the words perfectly but invented a fake fact that wasn't in the textbook? The ruler says "good grade," even though the student lied.

This paper introduces a new way to grade: The Cross-Examination Framework (CEF). Instead of just counting matching words, CEF acts like a detective or a courtroom lawyer.

The Core Idea: The "Question and Answer" Game

The authors realized that if two texts (the original source and the new AI-generated version) contain the same information, they should be able to answer the same questions.

Here is how the "detective" works in three simple steps:

The Interrogation (Question Generation):
The system takes the original text and asks a computer brain (an LLM) to generate a list of simple "Yes/No" questions that can be answered only by reading that text.
- Example: If the text says, "The doctor prescribed 50mg of aspirin," the question might be: "Was the dosage 50mg?" (The answer must be "Yes").
The Cross-Examination:
The system then takes those questions and asks the new AI-generated text to answer them.
- If the new text says "Yes," it passed.
- If it says "No," it's a contradiction (a lie).
- If it says "I don't know" (IDK), it means the information was left out (an omission).
The system does this in reverse too: it generates questions from the new text and checks if the original text can answer them. This ensures nothing was added that shouldn't be there (hallucinations).
The Verdict (Three Scores):
Instead of one single grade, CEF gives three specific scores:
- Coverage: Did the new text remember all the important details? (Did it leave anything out?)
- Conformity: Did the new text contradict the original? (Did it say the opposite?)
- Consistency: Did the new text make up facts that weren't in the original? (Did it hallucinate?)

Why This is Better Than Old Methods

The paper argues that old methods are like checking a recipe by only counting the number of "cups" and "teaspoons." If you swap "sugar" for "honey," the old method gets confused. CEF, however, tastes the dish to see if it still works.

The researchers tested this "detective" on three different jobs:

Translation: Turning English news into French, Spanish, Arabic, or Japanese.
Summarization: Condensing long news articles into short summaries.
Clinical Notes: Turning doctor-patient conversations into medical records.

The "Judge" and the "Stability" Test

To make sure the "detective" isn't biased or confused, the authors had to pick the best computer brain to do the questioning. They tested five different powerful AI models (like Llama, Qwen, and DeepSeek) to see which one was the most consistent.

They found that DeepSeek-V3 was the most reliable "judge." It was the least likely to change its mind about whether a question was valid or not.

They also figured out the perfect number of questions to ask. Asking too few (like 3) made the results jump around wildly (unstable). Asking too many (like 20) was a waste of time and money. They found that 10 questions was the "Goldilocks" number—enough to be accurate, but not too expensive.

What They Found

It catches lies: CEF is very good at spotting when an AI invents a fact (hallucination) or leaves out a crucial detail, which old methods often miss.
It works without a "Gold Standard": Usually, to grade a translation, you need a human-written perfect version to compare against. CEF is special because it can grade the work without needing that perfect human version. It just compares the source to the output.
It speaks the truth: When the researchers compared CEF's "detective work" against human experts, they found that CEF was much better at catching errors that changed the meaning of the text (like getting a name or a relationship wrong) than errors that were just about grammar or style.

The Bottom Line

The Cross-Examination Framework is a new tool that stops AI from "lying" or "forgetting" by treating the text like a witness in court. It asks the text questions to prove it knows the facts. It doesn't just count words; it checks if the story is still true.

Important Note: The paper strictly limits its claims to these evaluation tasks (translation, summarization, and note generation). It does not claim this tool can diagnose patients, make medical decisions, or be used for anything other than checking the quality of text generation.

Technical Summary: Cross-Examination Framework (CEF) for Information Fidelity

Problem Statement

Traditional evaluation metrics for generative text-to-text tasks (such as machine translation, summarization, and clinical note generation) fail to adequately capture information fidelity. Overlap-based metrics like BLEU and ROUGE penalize valid paraphrasing and exhibit surface-form bias, while embedding-based metrics like BERTScore remain largely insensitive to factual contradictions and entity-level hallucinations. Furthermore, these methods often rely heavily on costly gold-standard human references, limiting their applicability in domain-specific settings where the core goal is semantic faithfulness to the source rather than alignment with a specific reference text.

Methodology

The authors adapt the Cross-Examination Framework (CEF) as a task-agnostic, reference-free diagnostic tool. The framework operates on the intuition that if two texts contain the same information, they should answer the same questions. CEF treats the source document ( $D$ ) and the generated candidate ( $T$ ) as independent knowledge bases and executes a three-stage process:

Question Generation: An LLM generates closed-ended questions independently from both the source ( $Q_D$ ) and the candidate ( $Q_T$ ). Crucially, questions are constrained to be "YES-only" (verifiable facts where the correct answer is "YES"), ensuring they are grounded in the originating document's content.
Cross-Examination: The questions generated from the source ( $Q_D$ ) are presented to the candidate ( $T$ ) to generate answers ( $\hat{A}_{Q_D|T}$ ), and vice versa ( $Q_T$ presented to $D$ ). Answers are categorized as "YES", "NO", or "IDK" (I Don't Know).
Score Computation: Three interpretable scores are derived by comparing predicted answers against the ground-truth "YES" labels:
- Coverage: Measures content retention (percentage of source questions the candidate can answer, penalizing "IDK").
- Consistency: Measures hallucination (percentage of candidate questions the source can answer, penalizing "IDK").
- Conformity: Measures contradiction (percentage of source questions the candidate answers "NO").
- Conciseness: (Specific to summarization) Measures brevity via word count reduction.

The framework operates in two modes: Reference-Free (evaluating fidelity against the source) and With-Reference (evaluating alignment against a human reference).

Key Contributions

Task-Agnostic Framework: CEF is established as a reference-free, multi-dimensional assessment applicable across translation, summarization, and clinical tasks.
Multi-Task Validation: Empirical validation on NTREX (translation), CNN/DailyMail (summarization), and ACI-Bench (clinical notes) demonstrates that CEF uncovers semantic errors (omissions, factual contradictions) missed by traditional overlap and embedding metrics.
Robustness & Reliability Analysis: A systematic study identifies DeepSeek-V3 as the most stable "judge" model across diverse tasks and languages. The study also establishes $N=10$ questions per document as the optimal configuration, balancing statistical reliability with computational efficiency.
Reproducibility: The authors release prompts, code, and detailed ablation studies.

Experimental Results

Judge Stability: Across five candidate LLMs, DeepSeek-V3 consistently achieved the lowest Disagreement Rates (ADR) and Answer Deviation Scores (ADS) across all language pairs and tasks, particularly in morphologically complex languages (Arabic, Japanese) and clinical note generation.
Performance Sensitivity: CEF successfully differentiated between specialized translation models (e.g., NLLB-3.3B, commercial APIs) and general-purpose LLMs, showing that performance gaps widen in morphologically complex languages.
Reference-Free Reliability: Strong positive correlations were found between reference-free and with-reference modes for Coverage ( $r=0.845$ ) and Consistency ( $r=0.846$ ), validating CEF's ability to measure fidelity without gold references. Conformity showed lower correlation ( $r=0.457$ ) due to the rarity of contradictions (scores clustering near 100%).
Human Validation:
- In translation (WMT'25 Japanese subset), CEF mismatching questions aligned with human-annotated semantic errors at a rate of 78.40%, significantly higher than the 23.32% alignment with non-semantic errors.
- In summarization (FRANK benchmark), CEF excelled at identifying relational and entity-based factual distortions (e.g., Relation errors: 92.1%, Entity errors: 86.2%) but showed lower alignment with grammatical or coreference errors, confirming its focus on global semantic grounding over surface fluency.
Ablation Studies:
- Question Count: Low question counts ( $N=3, 5$ ) resulted in severe score instability. $N=10$ provided near-optimal stability, with doubling to $N=20$ yielding marginal gains for double the cost.
- Constraint: Enforcing "YES-only" questions proved critical. Unconstrained prompts allowing "NO" or "IDK" answers resulted in high instability, as models struggled to formulate verifiable negative claims or genuinely unanswerable questions.

Significance and Claims

The paper positions CEF not as a replacement for existing metrics, but as a complementary diagnostic tool that enriches evaluation by providing deep semantic insights. Its primary significance lies in its ability to:

Identify critical failure modes (content omissions, factual contradictions, hallucinations) that surface-level metrics miss.
Operate effectively without gold-standard references, making it suitable for domain-specific applications where references are unavailable.
Provide interpretable, fine-grained error analysis through specific question-answer mismatches.

The authors conclude that CEF offers a principled, scalable approach to evaluating information fidelity, particularly excelling in detecting entity-based and relational distortions in text-to-text generation.