Technical Summary: EVIDENCELENS

Problem Statement

Large Language Models (LLMs) are increasingly deployed to answer questions over complex financial documents (annual reports, earnings decks, analyst notes). However, in high-stakes financial workflows, the outputs of these models are difficult to verify. A fluent response often blends directly grounded statements, weak synthesis, and unsupported claims across narrative text, tables, and charts. Conventional chat interfaces typically provide only coarse citation links or no provenance at all, making it impossible for analysts to distinguish between a claim that is rigorously supported and one that is merely plausible-looking speculation. Existing visualization and AI research has addressed chart understanding or text reasoning separately, but few interfaces exist to audit multimodal financial answers where evidence is simultaneously textual, tabular, and visual.

Methodology

The authors propose EVIDENCELENS, a visual analytics prototype that reframes financial question answering (QA) not as a text-generation task, but as a claim-evidence alignment problem.

1. Data Model and Normalization

The system decomposes a generated answer into a set of atomic claims ($C$) and aligns them with evidence items ($E$).

• Claims: Normalized as tuples $(y_i, o_i, v_i, q_i)$, where $y_i$ is the claim type (value, comparison, trend, explanation), $o_i$ is an operator template, $v_i$ is a numeric payload, and $q_i$ is a text gloss.
• Evidence: Normalized as tuples $(m_j, u_j, t_j, a_j)$, where $m_j$ is the modality (text, table, chart), $u_j$ is a modality-specific anchor (e.g., cell path, chart mark group), $t_j$ is a surface description, and $a_j$ stores typed attributes (e.g., slope, extrema).
• Alignment Scoring: A lightweight compatibility score ($s_{ij}$) is computed using weighted semantic, numeric, structural, and local proximity features. Links are categorized as support, partial support, contradiction, or context based on thresholds and polarity tests.

2. Visual Design

The interface centers on a multimodal claim-evidence matrix rather than a chat transcript.

• Claim Panel: Decomposes the answer into cards, displaying a status glyph and a stacked bar showing the relative contribution of text, table, and chart evidence. It visualizes a confidence–support gap (high model confidence vs. weak aligned support) to prioritize review.
• Claim-Evidence Matrix: Rows represent claims; columns represent evidence items grouped by modality and source order. Cells encode the relationship (support/contradiction). This layout exposes support coverage, modality imbalance (e.g., an explanation supported only by prose), and contradictions (mixed green/red patterns) at a glance.
• Coordinated Source View: Selecting a matrix cell highlights the corresponding region in synchronized text, table, and chart views, preserving local context (e.g., headers, surrounding sentences, specific chart intervals).
• Trace View: Summarizes the order of evidence retrieval and reasoning steps.

3. Reproducibility and Pipeline

To ensure reproducibility, the system specifies a JSON-based artifact schema containing claims, evidence, and links. A deterministic review-priority ranking algorithm aggregates per-modality support to calculate a risk score ($r_i$). This score prioritizes claims with large confidence–support gaps, contradictions, or reliance on a single modality. The prototype uses a semi-automatic backend where an LLM proposes claims and rationale spans, mapped to source regions via rules, but the visualization logic remains backend-agnostic.

Key Contributions

The paper outlines four specific contributions:

1. Claim-Centered Provenance: A representation for financial QA that treats answers as sets of atomic claims linked to heterogeneous evidence (text, tables, charts) rather than a single block of text.
2. Multimodal Claim-Evidence Matrix: A visualization that makes support coverage, modality imbalance, and contradictions immediately visible, enabling analysts to audit answers at speed.
3. Reproducible Alignment Pipeline: A specification for a lightweight alignment and review-priority system with explicit weights, thresholds, and a JSON artifact schema that allows deterministic reconstruction of the visual state.
4. Scenario Validation: A demonstration of the design through representative report-auditing scenarios, distilling lessons for trustworthy financial AI interfaces.

Results and Observations

The paper does not present a large-scale user study with statistical significance but provides representative scenario walkthroughs (Case 1: Margin improvement; Case 2: Guidance revision) to illustrate the system's utility.

• Failure Mode Detection: The visualization makes three recurrent failure modes legible:
  • Unsupported explanatory leaps: Identified by sparse rows where support is concentrated only in prose.
  • Hidden contradictions: Identified by mixed support/contradiction cells within a single row (e.g., narrative vs. footnotes).
  • Overconfident synthesis: Identified by a large confidence–support gap despite fluent language.
• Workflow Impact: The system allows analysts to distinguish between claims that should be accepted, qualified, or rejected outright, a granularity that conventional chat interfaces flatten. It shifts the workflow from manual document searching to an "overview-first" triage of risky statements.

Significance and Claims

The paper argues that trustworthy financial QA depends as much on interface design as on model accuracy.

• Representation Problem: The authors posit that beyond retrieval (finding evidence) and language (generating answers), there is a missing "representation problem." Users need a structure to compare claims, notice imbalances, and preserve source context without losing the thread of the answer.
• Auditability over Narrative: The matrix design prioritizes auditability (rapid comparison, density patterns) over narrative flourish, treating charts as first-class evidence rather than screenshots.
• Editorial Utility: The interface supports an editorial workflow where professionals can refine answers to be precise enough for reuse in notes or memos, rather than just finding binary errors.
• Generalizability: While focused on finance, the authors suggest the approach generalizes to other domains requiring provenance over heterogeneous data (policy, medicine, scientific reporting).

The paper concludes that EVIDENCELENS reframes financial QA as a visual auditing task, turning opaque multimodal QA into an auditable visual structure suitable for inspectable, defensible, and time-pressured financial AI tools.