Analyzing LLM Instruction Optimization for Tabular Fact Verification

1. Problem Statement

Tabular fact verification involves verifying natural language claims against structured data (tables). While numerous methods exist, many are specialized to specific datasets or fail to outperform simple prompting strategies. Recent advancements in Large Language Models (LLMs) have introduced tool-augmented agents (e.g., ReAct) and program-aided reasoning (e.g., CodeAct), but it remains unclear whether these complex approaches consistently outperform language-based reasoning (e.g., Chain-of-Thought) when paired with optimized instructions. Furthermore, there is a lack of systematic understanding regarding how instruction optimization (tuning prompts without gradient updates) affects different reasoning paradigms, model scales, and tool usage behaviors in this domain.

2. Methodology

The authors conducted a comprehensive comparative study using the DSPy optimization framework. The study evaluated the intersection of three variables:

• Prompting Techniques:
  1. Direct Prediction: Standard zero-shot prompting.
  2. Chain-of-Thought (CoT): Encouraging intermediate reasoning steps.
  3. ReAct: Interleaving reasoning with SQL tool calls to query table data.
  4. CodeAct: Using executable Python code for data parsing and mathematical operations.
• Instruction Optimizers (DSPy):
  1. COPRO: Uses beam search to explore candidate instructions based on evaluation scores.
  2. MiPROv2: A multi-stage framework using Bayesian optimization to refine instructions and few-shot demonstrations.
  3. SIMBA: A stochastic introspective algorithm that leverages self-reflection on challenging instances to generate heuristic rules and refine instructions.
     Note: The study restricted MiPROv2 and SIMBA to instruction-only tuning to isolate the effect of instructions from few-shot selection.
• Model Families & Scales:
  • Qwen3: 8B and 32B parameters.
  • Gemma3: 12B and 27B parameters.
  • GPT-4o: Mini and full versions (evaluated on a subset due to budget).

Datasets: The evaluation spanned four benchmarks covering diverse domains:

• TabFact: Large-scale binary fact verification on Wikipedia tables.
• PubHealthTab: Public health claims with noisy, complex table structures.
• SciTab: Scientific claims requiring compositional reasoning and numerical analysis.
• MMSci: Multimodal scientific reasoning (tables converted to text).

Experimental Setup: All experiments were conducted in a zero-shot setting. A hybrid training set (100 instances sampled from TabFact, SciTab, and PubHealthTab) was used for instruction optimization, while performance was measured on development and test sets using Accuracy and Macro-F1.

3. Key Contributions

• First Systematic Comparison: This is the first study to systematically compare instruction optimization across different prompting paradigms (Direct, CoT, ReAct, CodeAct) and model families for tabular fact verification.
• Optimizer Specificity: The paper identifies that different optimizers excel in different contexts. MiPROv2 is most effective for CoT reasoning, while SIMBA provides the most significant gains for tool-augmented agents (ReAct/CodeAct), particularly in larger models.
• Behavioral Analysis: The study moves beyond accuracy metrics to analyze how optimization changes model behavior, specifically regarding tool usage frequency, reasoning paths, and error taxonomies.
• Generalization Insights: It reveals that optimized instructions for smaller models often generalize well to unseen datasets (like MMSci), whereas GPT-4o's optimized instructions showed less consistent generalization.

4. Key Results

Performance Trends

• Instruction Optimization Impact: Optimization consistently improved verification accuracy across all models and techniques.
• Model Scale: Larger models (e.g., Qwen3-32B, Gemma3-27B) benefited more substantially from optimized instructions than their smaller counterparts.
• Optimizer Performance:
  • MiPROv2: Delivered the most stable and consistent gains for CoT reasoning across all datasets.
  • SIMBA: Proved superior for ReAct and CodeAct agents, particularly with larger models. It significantly improved tool-use efficiency and accuracy.
  • COPRO: Showed moderate benefits but was less consistent than the other two, especially for tool-based agents.

Behavioral Shifts

• Tool Usage (ReAct):
  • SIMBA encouraged more direct reasoning paths, reducing unnecessary tool calls. For Qwen3-32B, tool usage frequency dropped from 90% to 35.4%, while successful execution rates increased.
  • MiPROv2 also reduced tool usage but sometimes led to higher error rates in tool execution compared to SIMBA.
  • COPRO increased tool calling frequency but resulted in more execution errors.
• Reasoning Paths:
  • Optimized instructions (especially from SIMBA) introduced specific heuristics regarding ordinal information (e.g., "first," "second") and numerical comparisons.
  • For CoT, SIMBA instructions explicitly focused the model on comparing values in the table, improving performance on claims involving numerical differences.
  • For Direct prediction, optimization improved the model's ability to detect contradictions in the order of elements.

CoT vs. Tool-Augmented Agents

• CoT: Remains a highly effective and robust choice, especially for smaller models. It avoids the overhead and error risks associated with tool execution.
• ReAct/CodeAct: Can achieve competitive or superior performance on complex, multi-hop claims if the model is large and instructions are carefully optimized (specifically via SIMBA). However, without optimization, ReAct agents often suffer from tool schema violations and execution errors.
• Error Analysis: ReAct errors were primarily due to reasoning failures (45%) and data-related issues (35%). CodeAct errors were dominated by data-related issues (70%), often stemming from the dataset's ambiguity, followed by code-related syntax errors (30%).

5. Significance

This paper provides critical insights for deploying LLMs in structured data verification:

1. Lightweight Optimization: It demonstrates that instruction optimization is a lightweight, model-agnostic method to significantly boost performance without the computational cost of fine-tuning.
2. Strategic Tool Use: It challenges the assumption that tool-augmented agents are always superior. The study shows that for tabular fact checking, CoT is often sufficient and more robust, while tool use (ReAct) requires specific instruction tuning (via SIMBA) to avoid unnecessary complexity and errors.
3. Heuristic Learning: The findings highlight that optimizers like SIMBA effectively teach models domain-specific heuristics (e.g., how to handle ordinal terms or specific metric comparisons) that are crucial for accurate fact verification.
4. Practical Guidance: For practitioners, the paper suggests using MiPROv2 for CoT-based systems and SIMBA for tool-based agents, with a strong emphasis on optimizing instructions for larger models to maximize gains.

In conclusion, the work establishes that while tool-augmented agents hold promise, their success is heavily dependent on high-quality, optimized instructions that guide them to use tools judiciously and reason directly when appropriate.