LLandMark: A Multi-Agent Framework for Landmark-Aware Multimodal Interactive Video Retrieval

This paper introduces LLandMark, a modular multi-agent framework that enhances multimodal video retrieval for complex, real-world queries by integrating specialized agents for landmark detection, LLM-driven image generation, and OCR refinement to achieve culturally grounded and explainable results, particularly for Vietnamese scenes.

The Gist

Imagine you have a massive library containing 250 gigabytes of video footage—thousands of hours of news, documentaries, and travel shows. Now, imagine you need to find a specific 5-second clip where a man is being interviewed in front of a specific cathedral in Hanoi, but you only have a vague description like "that old church with the twin towers."

Trying to find this needle in a haystack using a standard search engine is like trying to find a specific book by guessing the color of its cover. It's frustrating and often fails.

LLandMark is a new, super-smart team of digital detectives designed to solve this problem. Instead of one giant brain trying to do everything, it uses a multi-agent framework—think of it as a specialized task force where every member has a specific job.

Here is how LLandMark works, broken down into simple analogies:

1. The Team of Detectives (The Multi-Agent Framework)

When you ask a question, LLandMark doesn't just "search." It breaks the job down among four specialized agents:

• The Planner (Query Parsing Agent): This is the team leader. When you type, "Show me the video near the Turtle Tower," the Planner doesn't just take the words literally. It analyzes your intent, breaks the sentence into parts, and creates a "search map." It decides, "Okay, we need to look for visual clues of a tower, listen for the words 'Turtle Tower' in the audio, and check for text on the screen."
• The Local Expert (Landmark Knowledge Agent): This is the cultural specialist. Standard search engines often fail with local landmarks because they don't "know" what a specific Vietnamese landmark looks like. If you say "St. Joseph's Cathedral," this agent knows that's not just a building; it's a "dark gray stone building with twin square bell towers and Gothic architecture." It rewrites your query into a detailed visual description so the computer can actually see what you mean.
• The Scanners (Parallel Search): While the Planner and Expert are working, a team of scanners runs in parallel. One scans the video for objects (like "person" or "car"), one listens to the audio (transcribing speech), and one reads the text on the screen (OCR).
• The Judge (Reranking Agent): This agent takes all the clues gathered by the scanners and the experts. It weighs the evidence: "The audio said 'cathedral,' the visual scan saw 'twin towers,' and the text said 'Hanoi.' This is a match!" It then synthesizes a clear answer for you.

2. The "Magic Translator" for Text (OCR Refinement)

One of the biggest headaches in video search is reading text that appears on screen. In Vietnam, text often has diacritics (accents like á, ờ, ỹ). Standard tools often drop these accents, turning "Hà Nội" into "Ha Noi," which changes the meaning or makes it unsearchable.

LLandMark uses a Gemini AI as a "spell-checker and translator." It takes the messy, accent-less text from the video and magically fixes it, adding the correct accents and fixing typos. It's like having a native speaker sit next to the computer, correcting its reading mistakes in real-time so it understands the Vietnamese context perfectly.

3. The "Show Me, Don't Just Tell Me" Feature (Image-to-Image)

Sometimes, words aren't enough. If you ask for "Ben Thanh Market," a computer might just look for the words "Ben Thanh" or a generic picture of a market.

LLandMark has a special mode where it acts like a visual detective.

1. You say, "Show me Ben Thanh Market."
2. The system automatically goes to the internet, finds a real, high-quality photo of Ben Thanh Market, and says, "Okay, now I know exactly what that looks like."
3. It then uses that photo as a "search key" to scan through millions of video frames, looking for visual matches. It's like handing a detective a photo of the suspect and saying, "Find anyone who looks like this," rather than just describing them.

4. The Result: A Clear, Explainable Answer

In the end, LLandMark doesn't just give you a list of video links. It gives you a story. It shows you the specific frame, highlights the text it read, shows you the audio it heard, and explains why it thinks this is the right video.

Why does this matter?
Previous systems were like a student trying to memorize a dictionary but failing to understand the culture. LLandMark is like a local guide who knows the city, speaks the language, understands the history, and can point you exactly where to look. It makes searching through massive video libraries as easy as asking a knowledgeable friend for directions.

In the recent HCMAIC 2025 competition, this team of digital detectives proved they were the best in the country, ranking in the top 56 out of 680 teams by successfully finding complex, culturally specific video clips that others missed.

Technical Summary

1. Problem Statement

The paper addresses the significant challenges in retrieving evidence from large-scale, heterogeneous video corpora (specifically a 250GB dataset of broadcast and documentary videos). Key difficulties include:

• Multilingual and Cultural Nuance: Existing systems often fail to handle Vietnamese queries that rely heavily on spatial and cultural context, particularly regarding specific landmarks (e.g., "in front of St. Joseph's Cathedral").
• Noisy Text Extraction: Standard Optical Character Recognition (OCR) pipelines struggle with Vietnamese diacritics, leading to semantic errors in text-based retrieval.
• Rigid Pipelines: Previous agent-based frameworks (e.g., MAVEN) lacked adaptability in planning and landmark reasoning, while fixed pipelines failed to integrate multimodal reasoning effectively.
• Manual Dependency: Image-to-image retrieval often requires manual image input, lacking autonomous detection and semantic understanding of landmarks.

2. Methodology

The authors propose LLandMark, a modular multi-agent framework designed for adaptive, explainable, and culturally grounded video retrieval. The system operates through four integrated stages:

A. Preprocessing & Data Foundation

• Keyframe Extraction: Uses TransNetV2 to segment videos into shots, extracting representative keyframes (using percentile-based selection: 0.15, 0.5, 0.85) to reduce computational load.
• Indexing:
  • Visual: Keyframes are encoded using CLIP ConvNeXt-XXLarge and stored in Milvus for vector similarity search.
  • Text (ASR): Audio is transcribed using WhisperX (optimized for speaker diarization and accents) and indexed in Elasticsearch.
  • Text (OCR): Raw text is extracted via PaddleOCR and stored in Elasticsearch.
  • Objects: YOLOv9-e detects objects, with results stored in a JSON file for lazy loading.
  • Metadata: A unified index in MongoDB links all components.

B. The LLandMark Multi-Agent Framework

1. Query Parsing and Planning Agent:
   • Analyzes user intent (Vietnamese/English) and constructs a weighted SearchPlan.
   • Translates queries into descriptive English for CLIP alignment while retaining exact Vietnamese terms for ASR/OCR matching.
   • Detects landmark entities to trigger specialized processing.
2. Landmark Knowledge Agent:
   • Reformulates landmark names (e.g., "St. Joseph's Cathedral") into rich visual descriptions (e.g., "Twin square bell towers, dark gray stone, Gothic architecture").
   • This bridges the semantic gap in CLIP's embedding space, shifting retrieval from lexical identity to visual appearance.
3. Parallel Multimodal Search:
   • Executes the SearchPlan concurrently across semantic (Milvus), text (Elasticsearch), and object (JSON) modules.
4. Reranking and Answer Agent:
   • Fuses results using a weighted averaging formula.
   • Synthesizes a coherent, grounded natural-language answer referencing specific frames, supported by multimodal evidence (images, text snippets, detected objects).

C. Key Technical Innovations

• OCR Refinement Module: A hybrid pipeline combining PaddleOCR with Gemini 2.5 Flash (via LlamaIndex). It normalizes text to non-accent forms, then uses the LLM to reconstruct diacritics, correct spelling, and remove noise, significantly improving Vietnamese text quality.
• LLM-Assisted Image-to-Image Retrieval: An autonomous pipeline that:
  1. Detects landmarks in the query.
  2. Generates descriptive web search queries.
  3. Retrieves representative images via Google Custom Search API.
  4. Encodes these images with CLIP and performs similarity matching against video keyframes.
  5. Eliminates the need for manual image input.

3. Key Contributions

1. LLandMark Architecture: A modular multi-agent system specifically designed for query planning, landmark reasoning, and multimodal reranking, addressing the lack of cultural context in previous systems.
2. Gemini-Refined OCR: A novel post-processing pipeline that leverages LLMs to correct Vietnamese OCR errors (specifically diacritics), enhancing the reliability of text-based retrieval.
3. Autonomous Landmark Image-to-Image Retrieval: A fully automated pipeline that detects landmarks, fetches reference images, and performs visual matching, enabling precise retrieval of culturally specific scenes without manual intervention.

4. Experimental Results

The system was evaluated on the HCMAIC 2025 benchmark, which includes three tasks: Textual-KIS, Visual QA, and Temporal Reasoning (TRAKE).

• Quantitative Performance:
  • Achieved a total score of 77.40/88.00 in the qualifier round.
  • Ranked within the top 56 teams out of 680+ registered participants.
  • Demonstrated consistent performance across all three task types (KIS, QA, TRAKE) and all evaluation rounds.
• Qualitative Performance:
  • Landmark Queries: Outperformed baseline CLIP models in queries involving specific landmarks (e.g., "Ben Thanh Market," "St. Joseph's Cathedral"). While baselines returned generic results, LLandMark successfully identified the specific visual features and retrieved the correct video segments.
  • Explainability: The system provides transparent reasoning, showing how agents parse queries, refine landmarks, and fuse evidence.

5. Significance

LLandMark represents a significant advancement in culturally aware multimedia search. By integrating specialized agents for landmark reasoning and LLM-driven text/image refinement, the framework moves beyond simple keyword matching to adaptive, semantic understanding of complex, real-world queries.

Its ability to autonomously handle Vietnamese linguistic nuances (diacritics) and spatial contexts (landmarks) makes it a robust solution for large-scale video retrieval in non-English, culturally rich environments. The modular design offers a scalable foundation for future systems that integrate vision, language, and reasoning for human-centered video analysis.