💻 computer science

Knowledge is Power: Advancing Few-shot Action Recognition with Multimodal Semantics from MLLMs

This paper introduces FSAR-LLaVA, a novel end-to-end framework that leverages Multimodal Large Language Models as a knowledge base to extract enriched spatiotemporal representations and construct composite task-oriented prototypes, thereby significantly advancing few-shot action recognition through a training-free multimodal metric learning approach.

Original authors: Jiazheng Xing, Chao Xu, Hangjie Yuan, Mengmeng Wang, Jun Dan, Hangwei Qian, Yong Liu

Published 2026-03-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiazheng Xing, Chao Xu, Hangjie Yuan, Mengmeng Wang, Jun Dan, Hangwei Qian, Yong Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Blank Page" Dilemma

Imagine you are a security guard at a museum. Usually, you have a photo album with thousands of pictures of every possible thief, so you know exactly who to look for. This is how most AI video recognition works today: it needs massive amounts of data to learn.

But what if a new type of thief appears, and you only have one blurry photo of them? This is called Few-Shot Action Recognition (FSAR). It's like being asked to identify a bird species you've never seen before, based on a single snapshot.

Traditional AI struggles here. It's like trying to guess the plot of a movie by looking at a single frame; it often gets confused between similar actions (like "running" vs. "jogging").

The Old Solution: The "Translator" Trap

Previous attempts to solve this used a "translator" approach. They would:

Look at the video.
Ask an AI to write a caption (e.g., "A man is jumping").
Take that text and turn it back into a math code to compare with other videos.

The Analogy: Imagine trying to understand a foreign movie. Instead of watching it, you ask a translator to write a summary, then you read the summary, and then you try to guess the movie's genre based only on the summary. You lose all the nuance, the tone, and the visual details in the middle step. The paper calls this the "feature → caption → feature" pipeline, and it's inefficient because information gets lost in translation.

The New Solution: FSAR-LLaVA (The "Expert Intern")

The authors propose a new framework called FSAR-LLaVA. Instead of translating the video into text and back, they treat a Multimodal Large Language Model (MLLM) (like Video-LLaVA) as a super-smart Intern who has read every book in the library and watched every movie ever made.

Here is how their system works, step-by-step:

1. The Knowledge Base (The Intern)

They use a pre-trained AI (Video-LLaVA) that already knows the world. It doesn't need to be re-taught from scratch. It acts as a "knowledge base."

Analogy: Instead of hiring a rookie and showing them 10,000 photos, you hire a world-class expert who already knows what a "somersault" looks like, what "cooking" feels like, and the difference between "dancing" and "marching."

2. The Feature-Enhanced Module (The Dual-Brain)

The system doesn't just take the Intern's answer. It splits the Intern's brain into two parts:

The Visual Brain: Focuses on the movement, the shapes, and the colors.
The Textual Brain: Focuses on the meaning, the context, and the story.
The Magic: These two brains talk to each other. The Visual Brain says, "I see a person moving fast," and the Textual Brain says, "That looks like 'sprinting'." They combine their notes to create a super-accurate description.
Analogy: It's like having a detective (Visual) and a profiler (Textual) working together. The detective sees the footprints; the profiler knows the criminal's habits. Together, they solve the case better than either could alone.

3. The Prototype Construction (The "Best Guess" Strategy)

In a "few-shot" scenario, you have very few examples. The system needs to build a "prototype" (a perfect mental average) of what the action looks like.

The Problem: The examples you have (the training set) might look slightly different from the new video you are testing (the test set).
The Fix: The system creates two types of "mental averages":
- Local Prototype: Looks at specific details (like the exact way a hand moves).
- Global Prototype: Looks at the big picture (the whole scene).
The Analogy: Imagine trying to recognize a friend in a crowd.
- Local: "He has a red hat and a scar."
- Global: "He is wearing a blue jacket and walking fast."
- The system mixes these two views to make a perfect mental image of the friend, even if the lighting is different.

4. The Matching Metric (The "Smart Filter")

Finally, the system compares the new video to its mental images. But here's the trick: Not all information is useful.

The Problem: Sometimes the AI gets distracted by irrelevant details (e.g., the color of the wall, or a random bird in the background).
The Fix: The system has a "Smart Filter" that ignores the noise and only focuses on the top 10% most important clues that actually define the action.
Analogy: If you are looking for a needle in a haystack, you don't count every piece of straw. You use a magnet to find only the metal parts. This system uses a "magnet" to find only the action-relevant parts of the video.

Why is this a Big Deal?

No "Translation" Loss: It skips the middleman (writing a caption). It reads the video's "hidden thoughts" directly.
Super Efficient: It doesn't need to be retrained on thousands of videos. It uses the existing knowledge of the big AI model, so it only needs to learn a tiny bit of new stuff (very few parameters).
Works Without Labels: It can work even if you don't know the name of the action beforehand (the "Unknown" mode). You just ask, "What is happening?" and the system figures it out using its world knowledge.

The Result

The paper shows that this method beats almost every other method on standard video tests. It's like taking a student who has read the entire encyclopedia and giving them a single photo to identify a new animal, and they get it right 95% of the time, whereas other methods only get it right 70%.

In short: They stopped trying to force AI to "translate" videos into text and back. Instead, they let the AI "think" about the video directly using its massive internal knowledge, making it a much smarter and faster detective for video actions.

1. Problem Statement

Few-Shot Action Recognition (FSAR) aims to classify unseen action categories using only a limited number of labeled samples (e.g., 1-shot or 5-shot). While traditional data-driven methods rely heavily on metric learning within the visual space or require manual text annotations to guide prototype construction, they face significant challenges:

Information Degradation: Existing knowledge-driven approaches (like CapFSAR) often follow a feature → caption → feature pipeline. This intermediate step of generating natural language captions and re-encoding them causes information loss and fails to fully utilize the rich semantic knowledge of Multimodal Large Language Models (MLLMs).
Unimodal Limitations: Purely visual methods struggle to distinguish similar actions in low-shot settings due to the scarcity of support videos.
Distribution Gap: There is a significant distribution shift between meta-training and meta-testing sets, which standard prototype construction methods fail to bridge effectively without external knowledge.

The paper proposes a solution to leverage the vast world knowledge of MLLMs directly and end-to-end, avoiding intermediate captioning steps and manual text annotation.

2. Methodology: FSAR-LLaVA

The authors propose FSAR-LLaVA, the first end-to-end, knowledge-driven framework that uses MLLMs (specifically Video-LLaVA, with experiments on Qwen2/3-VL) as a frozen knowledge base. The architecture consists of four key components:

A. Multimodal Feature Extraction & Decoupling

Instead of generating captions, the model extracts intermediate hidden features directly from the multimodal decoder of the MLLM (e.g., the 31st layer of Video-LLaVA).

Decoupling: These multimodal tokens are decoupled into distinct Visual Tokens ( $T_v$ ) and Textual Tokens ( $T_t$ ) based on the token positions used during the MLLM's fusion process.
Rationale: This preserves the raw semantic and spatiotemporal information without the noise introduced by natural language generation.

B. Multimodal Feature-Enhanced Module (MFM)

A symmetric dual-branch structure that refines the decoupled features:

Visual Branch: Applies spatial self-attention, temporal self-attention, and visual-lead cross-attention (using textual tokens as keys/values) to enhance motion and appearance patterns.
Textual Branch: Applies textual-lead cross-attention (using visual tokens as keys/values) to ground high-level semantic priors in visual context.
Output: Enhanced visual features ( $F_v$ ) and textual features ( $F_t$ ).

C. Composite Task-Oriented Prototype Construction (CTPCM)

To address the distribution gap between training and testing sets, the framework constructs class prototypes using a composite strategy:

Local Prototype Construction (LPC): Focuses on token-level fusion. It calculates similarity between support and query tokens to weight the support prototype. This is particularly effective for textual branches due to the consistent word order in MLLM outputs.
Global Prototype Construction (GPC): Focuses on video-level fusion using global average pooling. This is more effective for visual branches where token positions vary significantly across frames.
Adaptive Fusion: The final prototype is a weighted combination of the original, local, and global prototypes, controlled by a learnable (or adaptive) parameter $\alpha$ .

D. Multimodal Prototype Matching Metric (MPMM)

A novel training-free metric designed to handle the dual-branch architecture:

Token-Level Distance: Calculates distances between support and query tokens in both visual and textual branches.
Top- $u$ Selection: Instead of summing all token distances (which introduces noise from irrelevant tokens), the metric adaptively selects the top $u$ largest distances (most decisive cues) from both branches to compute the final video-level distance. This suppresses noise and focuses on action-relevant information.

3. Key Contributions

First End-to-End Knowledge-Driven FSAR: Introduces FSAR-LLaVA, which bypasses the feature→caption→feature bottleneck by directly utilizing intermediate MLLM hidden states.
Novel Architectural Modules:
- MFM: Effectively decouples and enhances multimodal features via cross-attention.
- CTPCM: Dynamically refines prototypes by combining local (token-level) and global (video-level) construction to bridge the train-test distribution gap.
- MPMM: A robust matching metric that adaptively selects the most discriminative cues from multimodal tokens.
High Efficiency: The framework is training-free regarding the MLLM backbone. It requires minimal trainable parameters (under 10MB for the enhancement modules) while achieving state-of-the-art performance.
Prompt Flexibility: Demonstrates effectiveness with both "Known" prompts (using support labels) and "Unknown" prompts (generic questions like "What's the action?"), making it applicable to real-world scenarios where labels are missing.

4. Experimental Results

The method was evaluated on five benchmarks: Kinetics, HMDB51, UCF101, SSv2-Small, and SSv2-Full.

Performance: FSAR-LLaVA achieves State-of-the-Art (SOTA) results across all datasets and shot settings (1-shot and 5-shot).
- On HMDB51 (5-shot), it achieves 93.3% accuracy (vs. 88.8% for the previous SOTA Task-Adapter).
- On SSv2-Small (5-shot), it reaches 76.1% (vs. 70.2% for Task-Adapter).
Comparison: It significantly outperforms data-driven unimodal methods, multimodal methods (like CLIP-based approaches), and the previous knowledge-driven method CapFSAR.
Efficiency: Despite using a massive 7B parameter MLLM backbone, the trainable parameters are negligible (~9.3M). In "Offline" settings (pre-extracted features), it is faster and has lower FLOPs than many data-driven baselines.
Ablation Studies:
- Direct feature extraction (feature→) outperforms the caption-based pipeline.
- Feature decoupling is crucial; using only visual or only textual branches yields lower performance than the combined approach.
- The proposed MPMM metric significantly outperforms standard Hausdorff distance or Bi-MHM.

5. Significance and Impact

Paradigm Shift: The paper shifts the FSAR paradigm from "learning to compare" purely visual features to "leveraging world knowledge" via MLLMs. It proves that frozen MLLMs contain sufficient semantic and spatiotemporal knowledge to solve FSAR without extensive fine-tuning.
Practicality: By eliminating the need for manual text annotation and minimizing trainable parameters, the framework is highly suitable for industrial applications where data is scarce and labeling is expensive.
Generalization: The framework demonstrates strong generalization across different MLLM backends (Video-LLaVA, Qwen2-VL, Qwen3-VL), suggesting the approach is robust and not tied to a specific model architecture.
Future Direction: The authors highlight that while current "Unknown" prompts work well, there is potential for further improvement through more sophisticated, label-agnostic prompt engineering to better handle complex temporal reasoning.