RefereeBench: Are Video MLLMs Ready to be Multi-Sport Referees

1. Problem Statement

While Multimodal Large Language Models (MLLMs) have demonstrated strong capabilities in generic video understanding, their ability to perform specialized, rule-grounded decision-making in high-stakes environments remains unexplored. Automatic sports refereeing is a critical application area that requires more than just action recognition; it demands:

• Rule Application: Determining if a specific action violates sport-specific regulations.
• Temporal Grounding: Pinpointing the exact moment a violation occurs.
• Entity Perception: Identifying involved players and their states.
• Reasoning: Distinguishing between visually similar but legally distinct events (e.g., a legal body check vs. a foul).

Existing benchmarks focus on action spotting, commentary generation, or general sports QA, often limited to a single sport (usually soccer) or lacking the full scope of officiating logic. There is a lack of a comprehensive benchmark to evaluate whether current MLLMs can serve as reliable, multi-sport referees.

2. Methodology: RefereeBench Construction

The authors introduce RefereeBench, the first large-scale benchmark dedicated to automatic multi-sport refereeing. The construction process involved three rigorous stages:

• Data Collection & Curation:

  • Scope: Covers 11 distinct sports (Soccer, Basketball, Volleyball, Tennis, Table Tennis, Badminton, Handball, Field Hockey, Ice Hockey, Water Polo, Short Track Speed Skating).
  • Source: 925 curated real-match videos (totaling ~37 seconds on average) sourced from YouTube and official platforms.
  • Filtering: Raw footage was processed using ASR (WhisperX) and keyword matching to identify officiating incidents, followed by rigorous human review to filter out false positives and ensure continuous match context.

• Expert Annotation:

  • Annotators: ~30 certified national-level referees with strong English proficiency.
  • Process: A two-stage annotation process. First, referees extracted structured metadata (foul existence, type, rationale, penalty, timestamps, entities). Second, they generated 6,475 multiple-choice QA pairs based on this metadata.
  • Question Types: The benchmark evaluates seven question types across five dimensions:
    1. Foul Existence: Is there a violation?
    2. Classification: What is the foul type? What is the penalty type?
    3. Reasoning: Why is it a foul? What is the rule basis?
    4. Entity Perception: Who is involved?
    5. Temporal Grounding: When did the incident occur?

• Quality Control:

  • Cross-verification: Double-blind annotation with senior referee adjudication for disputes.
  • Bias Mitigation: "Blind" text-only testing was performed to ensure questions cannot be answered via linguistic shortcuts without visual evidence (model accuracy dropped to <30% in text-only mode).

3. Key Contributions

1. RefereeBench Benchmark: A novel, large-scale dataset spanning 11 sports with 6,475 expert-annotated QA pairs, covering the full spectrum of officiating logic (existence, classification, reasoning, perception, grounding).
2. High-Quality Expert Data: Unlike previous datasets, this is fully annotated by certified referees, ensuring the "ground truth" aligns with authentic officiating standards and multimodal evidence (including audio and replay views).
3. Comprehensive Evaluation: The first systematic evaluation of both proprietary (GPT-5, Gemini-3, Claude) and open-source (Qwen3-VL, VideoLLaMA3) MLLMs on the specific task of rule-grounded sports decision-making.

4. Experimental Results

The authors evaluated state-of-the-art models on RefereeBench, revealing significant gaps between current AI capabilities and human-level refereeing:

• Overall Performance:

  • Best Closed-Source: Doubao-Seed-1.8 achieved the highest accuracy at ~61%, followed closely by Gemini-3-Flash and Gemini-3-Pro (~60%).
  • Best Open-Source: Qwen3-VL reached only 47%.
  • Conclusion: Even the strongest models are far from reliable referees (60% accuracy is insufficient for high-stakes decision-making).

• Capability Breakdown:

  • Strengths: Models perform relatively well on Foul Existence (detecting if an event happened) and Entity Perception (identifying players).
  • Weaknesses: Models struggle significantly with Reasoning (applying rules) and Temporal Grounding (pinpointing the exact moment). For example, reasoning accuracy often hovers around 40-50%, and grounding accuracy is frequently below 50%.

• Key Findings:

  • Audio is Critical: Adding audio input improved performance significantly (e.g., Gemini-3-Flash jumped from 60.6% to 76.5%), proving that refereeing is inherently multimodal.
  • Over-Calling Fouls: Models exhibit a strong bias toward "over-officiating," frequently identifying fouls in normal clips, especially when prompted with suggestive wording.
  • Knowledge Gap: Text-only evaluations on professional referee exams showed models possess partial rule knowledge but fail to integrate it with visual evidence effectively.
  • Sport Variance: Performance varies wildly across sports; models are better at Soccer/Ice Hockey but struggle with Badminton/Water Polo, indicating poor cross-sport generalization.

5. Significance and Future Directions

• Trustworthy AI: The paper highlights that current MLLMs are not yet ready for real-world deployment as autonomous referees due to their inability to apply rules consistently and their susceptibility to bias.
• Research Gap: It identifies that the bottleneck is not just perception, but the integration of domain knowledge with multimodal grounding.
• Future Work: The authors suggest that future progress requires:
  • Training-based methods: Visual Chain-of-Thought (CoT) and rule-grounded supervision.
  • Better RAG: Dynamic retrieval of rulebooks that is tightly coupled with visual evidence, rather than naive text injection.
  • Agentic Pipelines: Decomposing the task into detection, candidate generation, and rule verification stages.

In summary, RefereeBench serves as a rigorous stress test, demonstrating that while MLLMs are improving, they currently lack the robustness, rule adherence, and temporal precision required to function as trustworthy sports officials.