Technical Summary: K9-Bench

Problem Statement

Multimodal Large Language Models (MLLMs) have demonstrated significant zero-shot capabilities in human-centric scenarios, yet their ability to understand animal-centric environments remains underexplored. Existing benchmarks for video understanding are predominantly focused on human actions, social cues, and conventions, offering limited insight into whether models can generalize to non-human subjects. Animal-centric scenarios present distinct challenges, including unique motion patterns, different interaction cues, and specific sources of ambiguity. While prior work has compiled species-focused datasets, they often rely on narrow tasks, manual supervision, or short clips, failing to evaluate the long-horizon, fine-grained multimodal reasoning required for everyday human-animal interactions.

Methodology

Dataset Construction: K9-Bench

The authors introduce K9-Bench, a novel benchmark designed to evaluate MLLMs on canine-centric video question answering. The dataset comprises:

• 907 real-world videos of domestic dogs engaged in free-living activities (e.g., obedience training, home routines, interactions with humans and other dogs).
• 4,744 five-way multiple-choice question-answer (QA) pairs derived from these videos.
• Five distinct task categories grounded in an expert-curated canine behavior ontology (developed with a KPA-certified trainer):
  1. Posture Analysis: Recognizing coarse poses and fine-grained cues (ear orientation, tail position, gaze).
  2. Action Sequence: Short-horizon temporal reasoning over ordered action transitions.
  3. Context Analysis: Interpreting behavioral changes in response to environmental stimuli or other agents.
  4. Cause-Effect Analysis: Linking trigger events to subsequent behavioral changes across temporal segments.
  5. Interaction Analysis: Interpreting dog-human and dog-dog interactions via non-verbal cues.

Scalable Data Generation Pipeline

To overcome the limitations of manual annotation, the authors propose a VLM/LLM-powered pipeline:

1. Video Curation: A semi-automated process mines videos from open web sources (YouTube) using a seeding strategy (query-based and curated seeds) followed by scalable retrieval via uploader channels and recommendation engines.
2. Filtering: A multi-stage filtering process removes irrelevant content. This includes text-based filtering using Gemini-2.0-Flash, followed by human-in-the-loop verification to identify artificial content, unnatural behaviors, or poor quality. A refined VLM-based rejection pipeline is then trained to automate the rejection of semantically unrelated videos with high agreement to human reviewers (84.3% accuracy).
3. QA Generation: A two-stage process generates QA pairs:
   • Correct QA Generation: VLMs (Gemini-2.5-Flash) generate questions and correct answers based on video input and a structured Knowledge Base (clustering the expert ontology).
   • Distractor Generation: To create plausible wrong answers, the pipeline first generates detailed textual video narrations. These narrations, rather than raw video frames, are used to prompt the generation of four distractors, ensuring they are semantically challenging but grounded in the video context.
4. Bias Mitigation: To prevent models from solving questions via textual shortcuts or speaker-specific cues (e.g., "the man," "the woman"), the authors implement a Deaf-Blind LLM Filtering step. An ensemble of text-only LLMs attempts to answer the questions without video access; pairs solvable without visual input are removed. Additionally, a Speaker Information Removal step uses GLM-4.5-Flash to strip speaker identities and auditory references from the text.

Experimental Setup

The benchmark evaluates a suite of frontier closed-source (Gemini-2.5 Pro, GPT-4o, Qwen3VL-235B) and open-source models (Qwen3-VL, InternVL). Evaluations include:

• MCQ Accuracy: Using cosine similarity between model responses and ground truth options.
• Subjective Evaluation: Using GPT-4o as a judge to score responses on logical consistency, factual correctness, accuracy, conciseness, and overall quality.
• Human Evaluation: A subset of 100 QA pairs was evaluated by human annotators to establish a performance baseline.

Key Results

• Limited Zero-Shot Performance: Frontier MLLMs exhibit limited zero-shot performance on canine-centric tasks. The best-performing closed-source model, Gemini-2.5 Pro, achieves an overall MCQ accuracy of 40.1%, while open-source models lag behind (e.g., InternVL-14B at 33.0%).
• Task Difficulty: Action Sequence and Posture Analysis are the most challenging categories for all models, requiring precise temporal decomposition of fine-grained cues. Cause-Effect and Interaction Analysis show relatively higher performance, likely due to the availability of high-level semantic priors.
• Impact of Multimodal Inputs: Adding audio input to video (e.g., Gemini-2.5 Pro with A+V) yields modest improvements (40.1% to 41.9% MCQ accuracy), particularly in Action Sequence analysis, where audio provides temporal landmarks.
• Thinking vs. Non-Thinking: "Thinking" model variants (Chain-of-Thought) show mixed results. While they often improve subjective scores (logical coherence), they do not consistently improve MCQ accuracy and sometimes degrade performance by over-analyzing visual inputs.
• Human vs. Model Gap: Human evaluators achieve 69.9% accuracy on the subset, significantly outperforming all evaluated models. This gap is attributed to humans' superior ability to integrate visual and audio evidence over long temporal horizons.
• Failure Modes: Analysis of errors reveals that Observable Cue Misinterpretation (48.6% of errors) is the dominant failure mode, where models overlook subtle cues like ear or tail movements.

Significance and Claims

The paper claims that K9-Bench serves as a challenging testbed for multimodal reasoning in shared human-animal environments, highlighting that current state-of-the-art MLLMs struggle with compositional reasoning over subtle, temporally extended canine interactions.

Key contributions include:

1. A Novel Benchmark: Providing a dataset specifically focused on everyday canine activities and interactions, moving beyond short clips to long-form video understanding.
2. A Scalable Pipeline: Demonstrating a VLM/LLM-driven approach to dataset construction that can be adapted to other low-data domains, addressing issues of semantic relevance and model bias.
3. Empirical Evidence: Establishing that while models can handle coarse social cues, they lack the fine-grained temporal grounding required for detailed animal behavior analysis, suggesting that generic Chain-of-Thought prompting is insufficient for these long-horizon scenarios.

The authors position K9-Bench not as a complete ethological taxonomy, but as a practical framework for evaluating and advancing AI systems intended to live and work alongside domestic animals.