HyperClaim: Fine-Grained Cross-Modal Hypergraph Reasoning for Video Misinformation Detection

Technical Summary: HyperClaim

Problem Statement
Video misinformation detection is frequently approached through global multimodal fusion or free-form multimodal reasoning. The authors argue that both paradigms often fail to capture localized authenticity cues arising from coupled interactions among query phrases (titles), contextual text, and short temporal spans of frames. While pairwise graph formulations are insufficient to capture these multi-way, higher-order cross-modal dependencies, existing methods that aggregate evidence into modality- or sample-level representations tend to dilute inconsistencies confined to specific tokens, frames, or brief temporal transitions. The core challenge is to represent and aggregate this localized evidence structure without flattening it through global fusion, particularly in a closed-input setting where the relevant signal is distributed across the title, context, and temporally ordered frames.

Methodology: HyperClaim
HyperClaim is a discriminative temporal hypergraph framework designed for sample-level authenticity classification. It treats the video title or benchmark-provided paired text as a "claim-like query" ($q$) and organizes the remaining textual context ($E$) and video frames ($v$) as evidence. The architecture consists of four primary stages:

1. Claim and Evidence Encoding:

   • The model uses Qwen3-VL-Embedding-2B as a unified multimodal encoder.
   • Text: Token-level hidden states are retained for both the query and evidence tokens to preserve fine-grained entities, actions, and relations.
   • Video: Frames are sampled at a rate of one every 30 original frames (up to 16 frames) to maintain chronological order while bounding visual nodes.

2. H-Forge: Claim-Oriented Sparse Hypergraph Formation:

   • Constructs a heterogeneous hypergraph $G = (V, H)$ where nodes include claim tokens ($V_c$), evidence tokens ($V_e$), and sampled frames ($V_v$).
   • Hyperedge Families:
     • $H_c$: Intra-claim hyperedges connecting all query tokens.
     • $H_e$: Source-specific hyperedges for each evidence item (e.g., comments, ASR, OCR).
     • $H_v$: Local temporal hyperedges using overlapping windows of three consecutive frames.
     • $H_{tv}$: Sparse cross-modal verification units connecting textual anchors to their strongest frame matches.
   • Filtering & Budgeting: A confidence-aware filtering process selects textual anchors based on cross-modal matching strength and margin (to reduce ambiguity). A "claim-aware source budgeting" mechanism reserves ~35% of anchor slots for claim tokens, prioritizing the query without duplicating it into every hyperedge.

3. Aether: Adaptive Temporal Hypergraph Reasoning:

   • Bidirectional Text-Video Calibration: A lightweight cross-attention block recalibrates textual and visual states at each reasoning layer, mixing attention distributions with a uniform residual to ensure robust message passing.
   • Soft Incidence & Propagation: Instead of hard assignments, the model learns soft node–hyperedge memberships. It computes seed states for hyperedges and uses attention mechanisms to propagate messages between nodes and hyperedges, allowing for context-dependent weighting.
   • Inter-hyperedge Refinement: A weighted line-graph layer is applied to refine interactions among overlapping evidence units without collapsing their individual states.

4. Cred: Claim-Conditioned Discrepancy Readout:

   • Aggregates textual, visual, and hyperedge states using learned linear attention.
   • Discrepancy Modeling: The readout explicitly models claim–video agreement and discrepancy. It computes the difference ($d_{cv}$) and element-wise product ($p_{cv}$) between normalized claim and video representations.
   • The final prediction is generated by an MLP taking the concatenated summaries and discrepancy features as input.

Training Strategy
The model employs a two-stage optimization strategy:

• Stage 1: The multimodal encoder is frozen. Hypergraph reasoning, pooling, and prediction modules are optimized on precomputed features to establish a stable evidence-routing function.
• Stage 2: Downstream modules are frozen. Only LoRA adapters within the visual and textual attention/feed-forward projections of the encoder are optimized to adapt representations to the fixed downstream structure.

Key Contributions

1. Sparse Temporal Hypergraph Representation: Introduces a claim-oriented representation that preserves localized query–text–frame and short-range temporal relations, avoiding the collapse of heterogeneous evidence into global summaries.
2. Integrated Framework (HyperClaim): Develops a system combining confidence-aware hypergraph formation, adaptive soft-incidence reasoning, residual text–video calibration, and discrepancy-aware readout.
3. Structural Evidence Tracing: Unlike methods generating free-form rationales, HyperClaim supports token- and frame-level structural evidence tracing via learned incidence and attention weights, revealing high-weight information paths.

Experimental Results
Evaluated under the FactGuard temporal protocol on three benchmarks: FakeSV, FakeTT, and FakeVV.

• Performance: HyperClaim achieved state-of-the-art results, outperforming strong discriminative baselines and reasoning-centric systems (Fact-R1, FactGuard).
  • FakeSV: 83.7% Accuracy, 84.2% F1.
  • FakeTT: 82.0% Accuracy, 79.5% F1.
  • FakeVV: 87.3% Accuracy, 86.1% F1.
• Ablation Studies:
  • Removing video evidence dropped F1 significantly (from 80.5% to 70.1% on FakeSV), confirming the necessity of joint multimodal modeling.
  • Replacing learned aggregation with global mean pooling reduced performance, highlighting the importance of source- and claim-dependent relevance.
  • Both soft incidence (adaptive routing) and text-video calibration were shown to be complementary; removing either degraded performance.

Significance
The paper posits that effective video misinformation detection requires preserving localized cross-modal evidence rather than assuming every misleading sample reduces to a single explicit contradiction. By utilizing hypergraphs to represent higher-order dependencies and employing a discriminative, closed-input reasoning approach, HyperClaim successfully captures fine-grained structural evidence. The model demonstrates that explicit local evidence structures can provide strong decision paths without relying on generated rationales or external tool calls, offering a robust alternative to both global fusion and open-ended reasoning paradigms.