💬 NLP

CRAFT: A Unified Counterfactual Reasoning Framework for Tabular Question Answering and Fact Verification

The paper introduces CRAFT, a unified framework that enhances table reasoning and fact verification in large language models by employing a bidirectional counterfactual reasoning process to overcome the limitations of single-direction inference and achieve superior performance on complex datasets.

Original authors: Chenshuo Pan, Yu Zhao, Jie Zhang, Changzai Pan, Zhenhe Wu, Jiayi Liang, Yujie Mao, Shuangyong Song, Yongxiang Li, Zhongjiang He

Published 2026-06-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chenshuo Pan, Yu Zhao, Jie Zhang, Changzai Pan, Zhenhe Wu, Jiayi Liang, Yujie Mao, Shuangyong Song, Yongxiang Li, Zhongjiang He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery using a giant, messy spreadsheet filled with facts about sports, money, or history. This is what computers (specifically Large Language Models, or LLMs) struggle with when asked to answer questions based on these tables. Usually, they act like a detective who only looks at the evidence from one angle. They read the question, guess an answer, and stop there. If they make a mistake in their first guess, they often stick with it, even if the evidence doesn't quite add up.

The paper introduces a new system called CRAFT (Counterfactual Reasoning Framework). Think of CRAFT not as a single detective, but as a team of detectives working together to challenge each other's theories.

Here is how CRAFT works, broken down into simple steps using a detective analogy:

1. The Setup: Turning a Question into a Claim

The Problem: The computer is given a question like, "Which country won more bronze medals than China?"
CRAFT's Move (The Rewriter): Instead of just guessing, the system first turns that question into a specific statement, like a hypothesis. It says, "Okay, let's assume the answer is Japan." Now, instead of a vague question, we have a concrete claim to test.

2. The Twist: The "What If?" Scenario

The Problem: If the computer only checks if Japan is the answer, it might miss the fact that South Korea also won more medals.
CRAFT's Move (The Reverser): This is the magic step. The system creates a "Counterfactual"—a "What if?" scenario. It takes the original claim and flips it or changes it to create a different path of thinking.

Original Path: "Japan won more medals than China."
Counterfactual Path: "What if Japan didn't win more? Or what if we look at the statement: 'Both Japan and South Korea won more medals'?"

Think of this like a lawyer presenting two different theories to a jury:

Theory A: The suspect is guilty.
Theory B: What if the suspect is innocent, or what if a different person did it?
By forcing the computer to explore the "What if" path, it is forced to look for evidence it might have ignored in the first path.

3. Gathering Evidence (The Extractor)

Now, the system sends both theories (the original and the "What if") back to the table. It digs through the rows and columns to find proof for both sides.

It finds proof that Japan has 77 medals.
It finds proof that South Korea has 65 medals.
It realizes that the "Japan only" theory was incomplete because the "What if" path revealed South Korea was also a valid answer.

4. The Final Decision (The Rethinker)

Finally, a "Judge" (the Rethinker module) looks at the evidence from both paths.

If both paths agree, the answer is easy.
If they disagree, the Judge weighs the evidence. It asks, "Which path had stronger proof?" or "Can we combine these findings?"
In our example, the Judge sees that the "What if" path uncovered South Korea, so the final answer becomes: "Japan AND South Korea."

Why is this better?

The paper claims that previous methods were like a person walking in a straight line; if they hit a wall, they might just keep walking into it. CRAFT forces the computer to walk in two directions at once.

It catches more mistakes: By checking the opposite or alternative view, the system is less likely to miss important details.
It works on any computer brain: The paper tested this on different types of AI models (some smart, some less smart) and found that CRAFT helped them all get better, often closing the gap between the "smart" and "less smart" models.
It's not just guessing more: The authors proved that simply asking the computer to guess the answer 10 times and picking the most common one doesn't work as well as CRAFT's method of looking at the problem from two distinct, logical angles.

The Bottom Line

CRAFT is a framework that teaches AI to play devil's advocate with itself. Instead of just accepting its first idea, it creates a "What if" version of that idea, checks the facts for both, and then combines the best evidence to give a more accurate, complete answer. This makes the AI much better at reading complex tables and answering questions correctly.

Technical Summary: CRAFT – A Unified Counterfactual Reasoning Framework

1. Problem Statement

Table reasoning remains a significant challenge for Large Language Models (LLMs), particularly when tasks require multi-step inference over long, structured tabular data. Existing approaches predominantly rely on single-direction reasoning, where models generate a hypothesis and verify it against the table. This unidirectional approach limits the model's ability to explore alternative hypotheses, often leading to brittle reasoning that fails to detect errors or consider counter-scenarios. Furthermore, current methods often suffer from performance gaps between different backbone LLMs and struggle to unify Table Question Answering (QA) and Table Fact Verification (FV) under a single paradigm.

2. Methodology: The CRAFT Framework

The authors propose CRAFT (Counterfactual Reasoning Framework), a unified, bidirectional verification process that reformulates table reasoning into a general framework applicable to both QA and FV. The framework operates through four cooperative modules:

2.1 Rewriter

The Rewriter transforms open-ended table QA tasks into informative, verifiable declarative statements.

Mechanism: It infers the expected answer type (e.g., number, time, boolean) from the question semantics without accessing ground truth. It then samples a type-consistent candidate value from the table to instantiate a declarative statement ( $S$ ).
Goal: This bridges the gap between QA and FV, allowing both tasks to be processed as statement-based verification problems.

2.2 Reverser

The Reverser constructs an optimal counterfactual statement ( $R^*$ ) based on the declarative statement $S$ .

Mechanism: Instead of a simple binary negation, the Reverser explores multiple counterfactual directions. It generates candidate counterfactuals using rule-based templates and selects the optimal one ( $R^*$ ) by evaluating which candidate induces the largest structural expansion in the reasoning space (proxied by the complexity of a generated SQL-style verification program).
Goal: To create a complementary reasoning path that challenges the initial premise and explores alternative scenarios.

2.3 Extractor

The Extractor performs step-by-step reasoning grounded in the table for both the original statement ( $S$ ) and the counterfactual statement ( $R^*$ ).

Mechanism: It generates two reasoning traces: $Trace(S)$ and $Trace(R^*)$ . From these traces, it extracts evidence items ( $e$ ) and evaluates their logical consistency with the table and question context. Valid evidence is accumulated to form evidence sets $E(S)$ and $E(R^*)$ .
Goal: To distill essential supporting evidence and generate candidate answers ( $A_1$ and $A_2$ ) under both factual and counterfactual assumptions.

2.4 Rethinker

The Rethinker aggregates the evidence and candidate answers to make a final decision.

Mechanism: If $A_1$ and $A_2$ coincide, the answer is returned. If they differ, the module computes self-consistency scores for each (answer, evidence) pair. It employs a weighted decision rule and, in ambiguous cases, re-evaluates candidates under opposing evidence to test reasoning consistency.
Goal: To synthesize a final, verified decision by weighing the strength of evidence from both reasoning paths.

3. Key Contributions

Unified Framework: CRAFT is the first framework to unify Table QA and Table Fact Verification under a shared counterfactual reasoning paradigm, eliminating the need for task-specific pipeline engineering.
Bidirectional Reasoning: By explicitly constructing and reasoning over both declarative and counterfactual statements, the framework overcomes the limitations of single-direction inference, guiding LLMs toward more heuristic and discerning reasoning.
Robustness Across Models: The framework significantly mitigates performance gaps between different backbone LLMs, demonstrating that counterfactual reasoning acts as a general heuristic rather than a model-specific optimization.

4. Experimental Results

The authors evaluated CRAFT on two standard datasets: WikiTQ (Table QA) and TabFact (Fact Verification), using diverse backbone LLMs (Deepseek-R1-14B, Qwen2.5-72B, Llama3.3-70B, and GPT-5-mini).

Performance Gains: CRAFT consistently outperformed representative baselines (including Chain-of-Table, Table-Critic, and Binder).
- On WikiTQ, it achieved an average accuracy of 82.4%, surpassing the strongest baseline by 4.7 points.
- On TabFact, it achieved 94.6%, exceeding the strongest baseline by 1.1 points.
Model Agnosticism: Notably, the framework reversed the performance hierarchy between models; for instance, Llama3.3-70B surpassed Qwen2.5-72B when using CRAFT, whereas Qwen typically outperformed Llama in standard settings.
Ablation Studies:
- Both the Rewriter and Reverser modules contributed positively, with the combination yielding the best results.
- The improvements were not merely due to repeated sampling; CRAFT outperformed Self-Consistency and Confidence-Weighted voting methods, indicating that the direction of reasoning (exploring diverse semantic regions) is more critical than the quantity of samples.
- The framework showed superior robustness across different table sizes, particularly on large tables where single-direction reasoning often degrades.

5. Significance and Claims

The paper claims that CRAFT establishes a more principled paradigm for structured reasoning tasks. By explicitly integrating counterfactual analysis, the framework helps LLMs move away from fixed inference patterns toward more robust, evidence-based reasoning.

The authors emphasize that counterfactual reasoning effectively addresses the brittleness of unidirectional inference. While the framework shows significant promise, the authors modestly acknowledge limitations:

Experiments were limited to larger models (14B+ parameters) due to the long-context requirements of tabular data.
The current implementation is tailored to table reasoning; generalizing to other domains (like text-based QA) remains challenging due to the difficulty of ensuring semantic consistency.
The Rethinker module, while effective, has not yet fully realized the potential of counterfactual reasoning and requires further refinement.

In conclusion, CRAFT demonstrates that guiding LLMs to explore alternative hypotheses through counterfactual reasoning is a viable and effective strategy for enhancing structured data comprehension.