Technical Summary: MERA Multi – A Multimodal Evaluation Framework for Russian-Language Architectures

Problem Statement

Multimodal Large Language Models (MLLMs) are rapidly advancing, yet their capabilities, limitations, and risks remain insufficiently understood, particularly in the context of the Russian language. Existing benchmarks predominantly focus on English and general domains (e.g., MultiBench, MMBench), neglecting the linguistic and cultural nuances of Slavic languages. Russian presents specific challenges due to its Cyrillic script, rich cultural context (e.g., folklore, Soviet media), and morphological complexity, which render direct translation of English-centric benchmarks inadequate. Furthermore, while text-based Russian benchmarks exist (e.g., Russian SuperGLUE, MERA), there is a critical gap in multimodal evaluation frameworks that cover image, audio, and video modalities.

Methodology

The authors introduce MERA Multi, the first open multimodal evaluation framework designed specifically for Russian-spoken architectures. The framework is instruction-based and encompasses four modalities: default text, image, audio, and video.

1. Benchmark Structure and Data

• Scope: The benchmark comprises 18 evaluation tasks distributed across 11 image datasets, 4 audio datasets, and 3 video datasets.
• Data Composition: To balance reproducibility and novelty, the benchmark integrates 7 tasks from publicly available open-source datasets and 11 tasks constructed from scratch. The private datasets are designed to incorporate Russian cultural specificities, target underexplored skill categories, and mitigate data contamination.
• Task Formats: Tasks are formulated as either multiple-choice questions (MC) or open-ended questions requiring short free-form answers.
• Taxonomy: The evaluation is grounded in a unified taxonomy of multimodal abilities, categorized into three broad cognitive groups: Knowledge (common, domain, expert), Perception (fine-grained single/cross-instance, cross-modal alignment), and Reasoning (inductive, deductive, abductive, quantitative).

2. Evaluation Pipeline

• Prompting Strategy: The framework employs a block-prompting scheme to construct universal prompts. Instead of hard-coding task-specific prompts, it uses a set of 10 predefined prompt blocks (e.g., attention hooks, reasoning requests, answer formats) to ensure consistency while allowing controlled variation to prevent benchmark saturation. Statistical analysis confirms that prompt formulation significantly impacts model scores, justifying the use of multiple prompt variants.
• Scoring Metrics:
  • Exact Match (EM): A generative analog of accuracy using normalized string comparison.
  • Judge Score (JS): A semantic similarity metric calculated using a dedicated LLM-as-a-judge model (based on RuModernBERT). This model is trained on human-annotated data to assess semantic equivalence, achieving a 0.96 F1 score and 99.6% agreement with EM on identical answers.
  • Final Score: The task score is the average of EM and JS.
• Aggregation: The Total Score combines the Attempted Score (quality on tasks the model attempted) and Coverage (breadth of the benchmark attempted). This approach allows for fair ranking without imputing arbitrary zeros for missing modalities.
• Data Protection: To prevent benchmark leakage, the authors employ data watermarking (using AudioSeal for audio and visual overlays for images/video) and a Multimodal Semantic Membership Inference Attack (MSMIA) detector to identify if models were trained on the benchmark data.

Key Contributions

1. Unified Taxonomy: A proposed taxonomy and evaluation methodology specifically designed for MLLM assessment, covering knowledge, perception, and reasoning across modalities.
2. Novel Datasets: The creation of 18 datasets, many built from scratch, incorporating Russian cultural and linguistic specificities, unified prompts, and metrics.
3. Baseline Results: Comprehensive baseline performance results for over 50 publicly available models (ranging from 1B to 110B parameters) and proprietary models (GPT-4.1), covering both open- and closed-source architectures.
4. Leakage Protection: A robust methodology for data leakage analysis and watermarking to protect private evaluation datasets, ensuring the validity of future evaluations.
5. Infrastructure: A standardized codebase for reproducibility and a submission platform with automated scoring and a public leaderboard.

Results

The evaluation of over 50 models reveals several trends:

• Modality Gaps: There remains a significant gap between modalities. While image understanding is relatively mature, audio and video understanding are underrepresented in terms of both model capabilities and dataset coverage, reflected in lower scores.
• Omni-Models vs. Specialists: Omni-models (e.g., Qwen3-Omni-30B, Qwen2.5-Omni-7B) generally achieve higher Total Scores due to broader coverage, even if their per-modality scores are not the absolute highest. Specialist models (e.g., Ultravox for audio, LLaVA-NeXT-Video for video) often show strong performance in their specific domain but suffer from low coverage scores.
• Top Performers: The top performer, Qwen3-Omni-30B-A3B-Instruct, achieved a Total Score of 0.5, driven by high Attempted Score (0.563) and Coverage (0.889). GPT-4.1 led in the image modality but had a lower Total Score (0.159) due to low coverage (0.333).
• Metric Discrepancy: Across most models, the Judge Score (JS) is higher than the Exact Match (EM), indicating that models often produce semantically correct answers that violate strict output formatting instructions.

Significance

The paper positions MERA Multi as the first comprehensive framework for transparent and culturally grounded multimodal evaluation in Russian. Its significance lies in:

• Filling a Critical Gap: Addressing the absence of multimodal benchmarks for Russian, a major Slavic language.
• Methodological Blueprint: Providing a replicable methodology for constructing multimodal benchmarks in typologically diverse languages, particularly within the Slavic family, where direct translation of English benchmarks is insufficient.
• Community Foundation: Establishing a foundation for transparent evaluation practices, encouraging community contributions, and promoting the development of culturally aware multimodal models for non-English languages.

The authors acknowledge limitations, noting that the 18 tasks may not cover all model abilities or specialized domains, and that inference results can vary based on the hardware-software stack. They also note that the current work focuses on semantic abilities and does not explicitly evaluate model biases toward underrepresented minorities, identifying this as a direction for future work.