HIPPO: Enhancing the Table Understanding Capability of LLMs through Hybrid-Modal Preference Optimization

1. Problem Statement

Table understanding and reasoning are critical tasks in NLP, yet current approaches face significant limitations due to their reliance on unimodal representations:

• Text-based limitations: Linearizing tables into text (e.g., Markdown, CSV) often loses structural semantics (layout, cell alignment, colors) and is sensitive to formatting noise, leading to unstable performance.
• Image-based limitations: While screenshots preserve layout and visual cues (colors, borders), Multi-modal Large Language Models (MLLMs) often struggle with symbolic reasoning (e.g., arithmetic, precise data lookup) when relying solely on visual input.
• The Gap: Existing studies rarely combine these modalities effectively. Prior work focuses on optimizing models for either text or image inputs, failing to leverage the complementary strengths of both to mitigate modality-specific weaknesses.

2. Methodology: The HIPPO Framework

The authors propose HIPPO (HybrId-modal Preference oPtimizatiOn), a training framework designed to enhance MLLM table reasoning by constructing preference signals from multiple modalities. The core innovation lies in Direct Preference Optimization (DPO) using a modality-consistent negative selection strategy.

Key Components:

1. Hybrid-Modal Input Sampling:
   The model is prompted with three distinct input configurations for the same table $T$ and query $Q$:

   • Text-only: $L(T)$ (Linearized Markdown).
   • Image-only: $V(T)$ (Table screenshot).
   • Hybrid: $L(T) + V(T)$ (Both text and image).
     The model samples $K$ candidate responses for each modality, creating a diverse candidate set $\tilde{Y}$.

2. Modality-Consistent Negative Selection:
   Unlike standard DPO which might use random negatives, HIPPO introduces a specific strategy to avoid modality bias:

   • Positive Sample ($\tilde{y}^+$): The ground truth answer $y^*$.
   • Negative Sample ($\tilde{y}^-$): Instead of random selection, HIPPO identifies the most frequent incorrect answer across all modalities (text, image, and hybrid) within the sampled set $\tilde{Y}$.
   • Rationale: This "most frequent incorrect answer" represents a stable, modality-invariant error pattern. By penalizing this specific error, the model learns to correct systematic reasoning flaws that persist regardless of the input modality, rather than just learning to exploit cues specific to one modality.

3. Optimization Objective:
   The model is fine-tuned using the DPO loss function (Eq. 3 in the paper), maximizing the likelihood of the ground truth while minimizing the likelihood of the representative negative. This aligns the model toward correct reasoning paths that are robust across modalities.

3. Key Contributions

• HIPPO Framework: A novel training framework that jointly leverages text and image representations to optimize MLLMs for robust table reasoning.
• Modality-Consistent Strategy: The introduction of a negative selection strategy based on cross-modal self-consistency. By selecting the most frequent error across modalities as the negative, the method effectively mitigates modality bias and enhances response diversity.
• Comprehensive Evaluation: Extensive experiments on diverse benchmarks (Table QA and Fact Verification) demonstrating that HIPPO not only improves multi-modal performance but also enhances unimodal reasoning capabilities.

4. Experimental Results

The authors evaluated HIPPO on Table Question Answering (TQA) and Table Fact Verification (TFV) tasks using various backbones (MiniCPM-V-2.6, Qwen2.5-VL, Table-LLaVA).

• Overall Performance: HIPPO achieved a 4% average improvement over various strong baselines (including zero-shot and standard SFT models).
  • On MiniCPM-V-2.6, it improved TQA by 3.6% and TFV by 2.8%.
  • On Table-LLaVA-7B, it improved TQA by 4.2% and TFV by 5.1%.
• Ablation Studies:
  • vs. Random Negatives: HIPPO outperformed a version using random negative selection by over 1%, proving the efficacy of the modality-consistent sampling.
  • vs. Standard DPO: HIPPO showed significant gains over standard DPO (which uses only text+image inputs), confirming that hybrid-modal sampling is crucial.
  • Sampling Size ($K$): Performance peaked at $K=10$; increasing to $K=20$ introduced noise and reduced gains.
• Cross-Modal Consistency:
  • HIPPO generated more consistent responses across unimodal and multimodal inputs (higher Jaccard similarity) compared to DPO.
  • It demonstrated reasoning diversity, showing that the model could engage in different reasoning patterns depending on the input while maintaining accuracy.
• Unimodal Failure Recovery: In scenarios where a model failed on text-only or image-only inputs, HIPPO successfully integrated complementary cues from the other modality to recover the correct answer, outperforming SFT and DPO significantly.

5. Significance

• Bridging the Modality Gap: HIPPO demonstrates that table reasoning is not just about having more data, but about how modalities interact. It proves that text and image modalities provide complementary semantics (text for arithmetic/symbolic logic, image for layout/visual context).
• Robustness to Error: By targeting the "most frequent error" across modalities, the method trains models to be less susceptible to specific modality biases (e.g., over-relying on visual layout or text formatting).
• Generalization: The framework improves performance even in unimodal settings, suggesting that the reasoning capabilities learned through hybrid-modal optimization transfer effectively to single-modality tasks.
• Practical Impact: The approach offers a scalable way to enhance table understanding in real-world applications (e.g., document analysis, database querying) where tables vary in format and quality.

In conclusion, HIPPO represents a significant step forward in multi-modal table reasoning by moving beyond simple concatenation of modalities to a sophisticated preference optimization strategy that leverages cross-modal consistency to eliminate systematic errors.