Technical Summary: CuriosAI Submission to the CASTLE Challenge at EgoVis 2026

Problem Definition

The CASTLE Challenge at EgoVis 2026 presents a large-scale retrieval and verification task involving 185 multiple-choice questions derived from over 600 hours of synchronized multi-view video. The dataset comprises footage from 12 participants recorded over four days using 10 head-mounted ego-cameras and 5 fixed exo-cameras, resulting in 15 synchronized 4K streams with aligned audio.

The core difficulty lies in coupling long-form retrieval with fine-grained multimodal verification. The authors identify that unconstrained video-audio specialists frequently fail at this scale due to "confabulation," characterized by:

• Re-quoting prompt context rather than observing video.
• Selecting answers based on silent clips or test patterns.
• Asserting counts without spatial grounding.
• Displaying high confidence without evidence.

Methodology

The submission explores two distinct approaches built upon a shared multimodal preprocessing layer. This shared layer processes the offline CASTLE 2024 corpus into five read-only data lanes:

1. Person Identity: Uses YOLO11x for body detection, ArcFace for face embeddings, and OSNet for cross-camera re-identification to track 12 participants.
2. Visual Captions: Generates captions using four models (Qwen3-VL, InternVL3, Qwen3-Omni) across varying window sizes (300s to 1800s) and prompt styles (scene, narrative, action-focused, reasoning-heavy).
3. Detected Objects: Employs SAM 3 for open-vocabulary detection to handle objects outside standard ImageNet/COCO vocabularies (e.g., specific board games, holiday decorations).
4. Audio Transcripts: Utilizes WhisperX for transcription, pyannote for diarization, and WeSpeaker for speaker resolution, with a consensus rule for fixed cameras to verify speaker identity.
5. Action Timeline: Summarizes per-person actions as (time-span, verb, co-actors) tuples using Qwen3.5.

Approach A: SVA (Search–Verify–Answer)

SVA is a three-stage pipeline designed to isolate final judgment behind explicit anti-confabulation discipline.

• Search: Aggregates preprocessing data into 50 (day, hour) cells. It uses hybrid retrieval (BM25 + e5-large-v2) and a GPT-5-mini reranker to narrow candidates, followed by a GPT-5 reasoning call to select a primary 15-minute window and a tentative answer.
• Verify: Expands the search anchor into ~24 adjacent 5-minute sub-windows. A VLM (Qwen3-Omni-30B) verifies these windows under four strict anti-confabulation rules:
  1. No echo: Do not re-quote prompt context.
  2. Abstain: Refrain from answering if evidence is missing (e.g., silent clips).
  3. Localise: Must provide spatial grounding for counts.
  4. Ground: Avoid ungrounded high confidence.
• Answer: A single LLM judge (GPT-5) fuses the question, choices, ranked evidence, and a reasoning trace from a Gemini-2.5-Pro. It follows an evidence-priority hierarchy (OCR > audio quote > visual > context) and rejects ungrounded counting answers.

Approach B: TMKG (Temporal–Multimodal–Knowledge–Graph)

TMKG serves as a contrast, testing whether a knowledge-graph structure can replace the decoupled verify–judge stack.

• Graph Construction: Bundles concurrent transcripts, captions, actions, and objects into "Observation" nodes (5-minute granularity). Multi-view observations sharing time, location, and people are aggregated into "GlobalEvent" nodes. Typed edges link observations to events, people to events, and events to locations/objects.
• Retrieval: Parses questions to extract entities and intents. It uses hybrid indexes (BM25 + e5-large-v2) over graph summaries and applies graph predicates (PARTICIPATES_IN, LOCATED_AT, INVOLVES) to rerank. The PRECEDES edge handles temporal proximity queries.
• Answer: Passes the selected cell's multi-camera video frames and bundled evidence to a single grounded VLM (Qwen3-Omni-30B) to emit the final choice, confidence, and rationale.

Key Contributions

1. Dual-Approach Comparison: The paper contrasts a hierarchical, multi-stage pipeline (SVA) against a knowledge-graph-based single-stage approach (TMKG) to determine the efficacy of retrieval structure versus verification discipline.
2. Anti-Confabulation Discipline: The authors formalize a set of prompt-level constraints (no echo, abstain, localise, ground) specifically designed to mitigate hallucination in unconstrained video models.
3. Shared Preprocessing Substrate: A robust offline pipeline that integrates identity tracking, open-vocabulary detection, and multi-model captioning to support diverse retrieval strategies.

Results

The authors report the following accuracy on the official leaderboard (185 questions):

• Approach A (SVA): 0.50 accuracy (Final Submission).
• Approach B (TMKG): 0.35 accuracy.

Analysis of Failure Modes:

• The dominant failure mode for both pipelines is the Search stage narrowing to an incorrect cell, from which downstream verification cannot recover.
• SVA requires significantly more computational resources (28 LLM/VLM calls per question) compared to TMKG (1 call).
• The anti-confabulation rules in SVA were not ablated; thus, their specific contribution to the score gain is not isolated, but they are presented as a critical design component.

Significance and Claims

The paper claims that at the scale of the CASTLE 2026 challenge, disciplined clip-level verification contributes more to performance than altering the retrieval structure. The 0.15 accuracy gap between SVA and TMKG suggests that adding a layer of verification discipline (abstention, count localization, evidence grounding, and no-echo) is essential, even if it increases the number of model calls.

The authors conclude that while both pipelines are currently bounded by the same Search-stage localization failures, the natural next direction is to combine higher-precision retrieval with this established verification discipline. The paper does not claim to have solved the retrieval bottleneck but emphasizes that verification discipline is a necessary layer on top of any retrieval substrate for this task scale.