MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks

This paper introduces MCIF, the first crosslingual, human-annotated benchmark based on scientific talks that systematically evaluates multimodal large language models across four languages, three modalities, and varying input lengths to address gaps in existing assessments of instruction-following capabilities.

The Gist

Imagine you are trying to teach a new, super-smart robot assistant how to be a multilingual, multi-sensory tour guide for a global science conference.

Right now, most robot assistants are like monolingual librarians. They are great at reading books (text) in English, but if you hand them a video with a German speaker, or ask them to summarize a 2-hour lecture in Chinese, they often get confused, speak the wrong language, or just make things up.

This paper introduces MCIF (Multimodal Crosslingual Instruction-Following), which is essentially a giant, rigorous "driver's license" test for these new AI assistants.

Here is the breakdown of what they did, using some everyday analogies:

1. The Test Track: "The Science Talk Gauntlet"

Instead of testing the robots with simple questions like "What is the capital of France?", the researchers built a test track using real scientific talks from a major computer science conference (ACL).

• The Content: They took 21 real presentations (about 2 hours of video) and added 79 more summaries.
• The Challenge: They didn't just use English. They translated everything into German, Italian, and Chinese.
• The Formats: They tested the robots on:
  • Text: Just the written transcript.
  • Audio: Just the sound of the speaker.
  • Video: The full video with slides and the speaker.
  • Mix: Audio + Video together.

2. The Four "Obstacle Courses" (Tasks)

The robots had to navigate four different types of challenges, each getting harder:

• 🗣️ Recognition (The "Hearing Test"): The robot listens to a speaker and must write down exactly what they said.
  • Analogy: Like a court stenographer who must type out a fast-talking lawyer perfectly, even if there is background noise.
• 🌍 Translation (The "Universal Translator"): The robot hears a speech in English and must write it out in Chinese or German.
  • Analogy: Like a simultaneous interpreter at the UN, but they have to do it instantly and perfectly.
• ❓ Question Answering (The "Pop Quiz"): The robot watches the video and answers specific questions.
  • Analogy: Like a teacher asking, "What was the main point of the third slide?" or "What is the speaker's name?" Some questions could only be answered by listening, some by looking at the slides, and some by both.
• 📝 Summarization (The "Executive Brief"): The robot must watch a 20-minute lecture and write a 200-word summary.
  • Analogy: Like a busy CEO who needs the "TL;DR" (Too Long; Didn't Read) version of a meeting before their next appointment.

3. The "Trick Questions" (Prompt Variations)

The researchers didn't just ask the same question twice. They created two versions of the test:

• MCIF-Fix: The robot gets the exact same instruction every time (e.g., "Translate this").
• MCIF-Mix: The robot gets the same task but phrased differently (e.g., "Can you put this into German?", "Give me the German version," "Translate the audio").
• Why? To see if the robot gets confused if you change your wording, just like a human might get confused if you ask a question in a weird way.

4. The Results: "The Reality Check"

The researchers tested 23 different AI models (from big companies like Google, Microsoft, and open-source communities). Here is what they found:

• The "Short Memory" Problem: Most robots are great at short clips (like a 30-second TikTok) but fall apart when the video gets long (like a 20-minute lecture). They forget the beginning by the time they reach the end.
• The "Multitasking" Struggle: When you give a robot both audio and video at the same time, it often gets overwhelmed. It's like trying to listen to a podcast while watching a movie; the robot tends to ignore one of them or mix them up.
• The "Language Switch" Glitch: Even when told to speak Italian, many robots accidentally slipped back into English, especially when the task was hard.
• The "Hallucination" Issue: When the robots didn't know the answer, instead of saying "I don't know," they often made up facts or described the wrong slide.

The Big Takeaway

Think of MCIF as a stress test for the next generation of AI.

Currently, these AI assistants are like smartphones that are great at texting but terrible at video calls. They can read a book, but they struggle to understand a whole movie, translate a foreign language speech, and summarize it all at once without getting confused.

This paper says: "We have built the ultimate test track. Now, AI developers, go fix your robots so they can actually handle the real, messy, multi-language, long-form world we live in."

The good news? They released the test data for free, so anyone can try to build a better robot to pass the test.

Technical Summary

1. Problem Statement

Despite rapid advancements in Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs), existing evaluation benchmarks suffer from significant limitations when assessing crosslingual and multimodal instruction-following (IF) capabilities. Current benchmarks typically:

• Lack Multilingual/Crosslingual Scope: Most are restricted to English or focus on single-language pairs, failing to test models on inputs and instructions in different languages (e.g., English speech with a German instruction).
• Ignore Modality Integration: Many focus on only two modalities (e.g., text-vision or text-speech) rather than the full triad of text, speech, and video.
• Focus on Short-Form: They predominantly use short clips, neglecting the ability of models to handle long-form dependencies (e.g., hour-long scientific talks).
• Rely on Synthetic Data: Many benchmarks use synthetic speech or lack rigorous human annotation, raising concerns about data quality and bias.

There is a critical need for a unified, human-annotated benchmark that evaluates how well models can interpret instructions, integrate multiple modalities, and process long contexts across diverse languages.

2. Methodology: The MCIF Benchmark

The authors introduce MCIF, the first manually curated, crosslingual, multimodal instruction-following benchmark derived from scientific talks.

Data Construction

• Source: 21 scientific presentations from the ACL 2023 conference (plus 79 additional abstracts for summarization), ensuring the content is recent and avoids training data contamination.
• Modalities: Each sample includes Text (transcripts), Speech (audio), and Video (presentation slides and speaker).
• Languages: Fully parallel across English, German, Italian, and Chinese.
• Context Length: Includes both Short-form (automatically segmented ~16s clips) and Long-form (full talks, up to ~10 hours of total content).
• Annotation Process:
  • Transcripts: Professional linguists created high-quality US English transcripts.
  • Translations: Professional translators translated transcripts, summaries, and questions into the three target languages.
  • Question Answering (QA): 220 unique QA pairs were created by NLP experts. Questions are categorized by source:
    • General: Applicable to any talk (e.g., author affiliations).
    • Transcript-based: Requires specific details from the full talk.
    • Abstract-based: Simulates querying based only on the summary.
  • Answerability Labels: Each QA pair is labeled as NA (Not Answerable), A (Audio only), V (Video only), or AV (Audio-Video).

Task Design

MCIF covers 13 specific tasks organized into 4 macro-tasks:

1. Recognition: Automatic Speech Recognition (ASR), Audio-Video Recognition (AVR).
2. Translation: Machine Translation (MT), Speech Translation (ST), Audio-Video Translation (AVT).
3. Question Answering: Textual (TQA), Spoken (SQA), Video (VQA), Audio-Video (AVQA).
4. Summarization: Text (TSUM), Speech (SSUM), Video (VSUM), Audio-Video (AVSUM).

Prompt Variants

To test robustness, the benchmark offers two variants:

• MCIFfix: Uses a single, fixed prompt for each macro-task.
• MCIFmix: Samples prompts from a pool of 10 diverse formulations per task to evaluate generalization against prompt engineering variations.

3. Key Contributions

1. First Parallel Crosslingual Multimodal Benchmark: MCIF is the first dataset to simultaneously evaluate text, speech, and video across four typologically diverse languages in a fully parallel setting.
2. Long-Form Evaluation: It addresses the gap in long-context evaluation by including full-length scientific talks, challenging models' ability to maintain coherence over extended inputs.
3. Human-Curated Quality: Unlike many synthetic benchmarks, MCIF relies on professional human annotation for transcripts, translations, and QA pairs, ensuring high data fidelity.
4. Comprehensive Baseline Analysis: The paper evaluates 23 state-of-the-art models (7 LLMs, 5 SpeechLLMs, 5 VideoLLMs, 6 MLLMs, and 1 commercial API model) to provide a holistic view of current capabilities.

4. Experimental Results

The evaluation of 23 models reveals several critical insights:

• Task Difficulty Hierarchy:

  • Summarization is the most challenging task, with many models producing negative BERTScores (worse than random) or ignoring instructions entirely.
  • Recognition (ASR) is generally feasible for specialized models but highly sensitive to prompt variations.
  • Question Answering benefits from multimodal inputs, but models struggle with fine-grained, content-specific questions compared to general knowledge.
  • Translation is dominated by text-based LLMs; multimodal models often underperform or hallucinate.

• Modality Integration Challenges:

  • Speech + Video: Contrary to expectations, adding video to speech often degrades performance in recognition and translation tasks. Models struggle to effectively fuse these modalities, sometimes treating them as separate streams rather than a unified context.
  • Video-Only: Video-only inputs often yield the weakest results, indicating current MLLMs underutilize visual signals compared to audio/text.

• Long-Form vs. Short-Form:

  • Models generally perform significantly worse on long-form inputs.
  • Common failure modes include under-generation (stopping prematurely), hallucination, and language switching (defaulting to English despite instructions in other languages).
  • Only a few models (e.g., Gemini 2.5 Flash, Ola) showed robustness in long-form contexts.

• Prompt Robustness:

  • Most models exhibit low robustness to prompt reformulation (MCIFmix vs. MCIFfix), with significant score fluctuations in recognition and summarization tasks.

5. Significance and Future Directions

• Gap Identification: MCIF highlights that current MLLMs are not yet ready for robust, general-purpose crosslingual instruction following, particularly in integrating speech and video or handling long contexts.
• Research Roadmap: The findings suggest future work should focus on:
  • Improving multimodal fusion mechanisms to prevent modality interference.
  • Developing sequence compression or extended context modeling for long-form processing.
  • Enhancing instruction tuning with diverse prompt variations to improve robustness.
• Open Science: The benchmark, code, and model outputs are released under CC-BY 4.0 and Apache 2.0 licenses, fostering open research and reproducibility.

In conclusion, MCIF serves as a rigorous diagnostic tool that exposes the substantial gap between current multimodal systems and the ideal of a general-purpose, crosslingual, multimodal assistant.