Technical Summary: CLBench-V

Problem Statement

While large language models increasingly rely on task-specific context provided at inference time rather than solely on parametric knowledge, existing evaluations of this "context learning" capability are predominantly restricted to unimodal textual scenarios. In real-world applications—such as analyzing scientific figures, interpreting financial reports, navigating maps, or synthesizing web-based information—context is inherently multimodal. Current benchmarks fail to address the specific difficulties models face when they must localize, bind, and reason over evidence distributed across text, images, tables, and layouts. Furthermore, existing evaluations often struggle to distinguish where the context learning process breaks down: whether the model fails to find the evidence, misapplies the found information, or ignores context-defined rules in favor of prior knowledge.

Methodology

The paper introduces CLBench-V, a benchmark designed to evaluate multimodal context learning through a structured, three-level capability hierarchy. This hierarchy isolates specific failure modes and diagnostic dimensions:

1. L0: Context Grounding: Tests the model's ability to locate, read, and combine evidence from multimodal contexts without needing to learn new rules. Tasks include visual search, spatial localization, and cross-image binding (e.g., transit-map reasoning, high-resolution visual search).
2. L1: New Information Application: Tests the ability to apply newly provided facts, values, or cases using existing reasoning skills, while avoiding the override of context-specific data with outdated prior knowledge. Tasks include visual fact QA and financial report analysis (e.g., ROE calculation).
3. L2: New Knowledge Learning: Tests the ability to acquire new rules, empirical patterns, or scientific conclusions defined only within the context and apply them to answer tasks. Tasks include inferring paper conclusions from medical figures and following context-defined rules in image-rendered tables.

Benchmark Construction:
CLBench-V is constructed from two complementary sources to ensure diversity and diagnostic depth:

• Data Integration: Public benchmarks (e.g., ReasonMap, Insight-O3, ZeroBench, MIRBench, PRISMM-Bench) were converted into a unified input-output format. Strict curation criteria were applied to ensure visual dependency, minimize irrelevant context, prioritize multi-hop reasoning, and exclude video tasks.
• Data Construction: Two new domain-specific tasks were created using automated construction and filtering procedures to reduce annotation costs while maintaining quality:
  • Financial Report ROE Analysis: Models must compute Return on Equity from converted PDF reports using standard DuPont decomposition, requiring document-level grounding and entity-value binding.
  • Paper Conclusion Inference: Models are given the introduction and methods of recent medical papers (excluding results) and must infer scientific findings supported by the paper's figures and tables, testing the synthesis of multimodal evidence.

Evaluation Protocol:
The benchmark utilizes a unified framework with task-specific evaluators. Structured outputs (e.g., multiple-choice, numeric) use deterministic matching or topology judges. Open-ended tasks use LLM-based semantic judging, with specific metrics like list-level semantic scoring for paper conclusions to penalize over-prediction.

Key Results

Experiments were conducted on 3,443 instances across six recent multimodal models (including InternVL3.5, Qwen3.5/3.6, Doubao-Seed, Kimi-K2.6, and GPT-5.4).

• Overall Performance: Multimodal context learning remains far from saturated. The best overall score achieved by any model was 0.2847 (InternVL3.5-30B-A3B), indicating significant room for improvement.
• Capability Disparities: Models exhibit distinct strengths across the hierarchy:
  • InternVL3.5-30B-A3B performed best on L0 (Context Grounding) and L2 (New Knowledge Learning), particularly in paper conclusion inference.
  • Qwen3.5-Plus and Qwen3.6-27B performed best on L1 (New Information Application), showing stronger abilities in applying document-specific values.
  • No single model dominated across all datasets; performance varied significantly based on task type (e.g., route topology vs. spatial intelligence).
• Diagnostic Insights:
  • Context Length & Image Count: Performance was not strictly correlated with token length within a model's context window. However, extremely long inputs (specifically financial reports) caused failures when they exceeded model limits. Image count showed model-dependent effects, acting as richer evidence for some models but visual distraction for others.
  • Failure Modes: Analysis identified six common failure types: evidence missing, evidence misbinding, context misuse, prior knowledge override, incomplete induction, and format failure.
  • Judge Reliability: The choice of LLM judge for semantic evaluation significantly impacted reported scores, highlighting the need for transparent reporting of evaluation metrics.

Significance and Claims

The paper claims that CLBench-V provides a necessary diagnostic framework for multimodal context learning by moving beyond simple retrieval or long-context capacity tests. Its primary significance lies in:

1. Granular Diagnosis: By separating grounding, application, and learning, the benchmark reveals that accepting multimodal input is insufficient for reliable context use; models may excel at one level while failing at another.
2. Real-World Relevance: The inclusion of financial and scientific domains addresses the gap where real-world contexts are inherently multimodal and require synthesizing evidence from figures, tables, and text.
3. Scalable Construction: The paper demonstrates that automated construction and filtering can effectively generate high-quality, domain-specific context-learning tasks, reducing the cost of benchmark expansion.

The authors position CLBench-V not as a final solution, but as an initial step to guide the development of multimodal systems that can not only "see" context but effectively "use" and "learn" from it.