💻 computer science

IntentVLM: Open-Vocabulary Intention Recognition through Forward-Inverse Modeling with Video-Language Models

The paper introduces IntentVLM, a novel two-stage video-language framework inspired by forward-inverse modeling that achieves state-of-the-art open-vocabulary human intention recognition by decomposing the task into goal candidate generation and structured selection, thereby significantly reducing hallucinations and matching human performance.

Original authors: Hamed Rahimi, Clemence Grislain, Adrien Jacquet Cretides, Olivier Sigaud, Mohamed Chetouani

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hamed Rahimi, Clemence Grislain, Adrien Jacquet Cretides, Olivier Sigaud, Mohamed Chetouani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a friend in a kitchen. You see them open the fridge, take out a carton of milk, and then walk toward the coffee machine.

A standard robot might just say, "Person is holding milk" or "Person is near coffee." But a truly helpful robot needs to understand why they are doing this. Is the friend making coffee? Are they just checking if the milk is there? Or are they about to pour it into a cup for a guest?

This paper introduces IntentVLM, a new kind of "brain" for robots designed to solve this exact problem. It helps robots figure out what a human is planning to do, even when the robot hasn't been taught a specific list of possible answers.

Here is how it works, using simple analogies:

The Problem: The "Multiple Choice" Trap

Most current robots are like students taking a multiple-choice test where the teacher only gives them five options to pick from. If the answer isn't on the list, the robot gets stuck or guesses wrong.

The Old Way: The robot sees the milk and coffee and has to choose from a fixed list like: "A) Make coffee, B) Make tea, C) Clean up." If the person is actually making "hot chocolate," the robot might fail because that option wasn't on the list.
The New Way (IntentVLM): The robot can understand open-ended ideas. It doesn't need a pre-written list. It can figure out "Make hot chocolate" on its own.

The Solution: A Two-Step Detective Process

The authors were inspired by how human brains work. They call this "Forward-Inverse Modeling." Think of it like a detective solving a mystery in two steps:

Step 1: The "What If?" Generator (Forward Model)
Instead of guessing the answer immediately, the robot first acts like a brainstorming session. It looks at the video and asks, "What are all the possible reasons this person is doing this?"

Analogy: Imagine a detective listing suspects. "Maybe they are making coffee. Maybe they are heating milk for tea. Maybe they are just checking the expiration date."
The robot generates a short list of these "candidate ideas" based on what it sees.

Step 2: The "Best Fit" Judge (Inverse Model)
Once the robot has a list of possibilities, it acts like a strict judge. It looks at the video again and asks, "Which of these ideas fits the evidence best?"

Analogy: The detective looks at the clues (the person grabbed a mug, not a tea cup) and says, "Okay, 'making tea' doesn't fit. 'Checking the date' doesn't fit. 'Making coffee' fits perfectly."
The robot picks the best match from its own list.

Why This is Special

It Reduces "Hallucinations": Sometimes AI makes things up (hallucinations). By forcing the robot to first list possibilities and then choose the best one, it stops the robot from just guessing wildly. It's like asking a student to show their work before giving the final answer.
It's Open-Minded: Because the robot generates its own list of ideas, it isn't limited to a small vocabulary. It can understand complex, unique intentions that a human might have.
It's Efficient: The researchers used a "smart shortcut" (called LoRA) to teach the robot. This is like giving the robot a specialized notebook to learn the new skill without having to rewrite its entire brain. This keeps the robot fast and doesn't make it forget how to do other things (like recognizing objects).

The Results

The team tested this robot brain on two different challenges:

IntentQA: A test where the robot had to answer questions about what people were trying to do in videos.
Inst-IT Bench: A test to see if the robot could still recognize objects and scenes after learning to understand intentions.

The Outcome:

The new robot got about 80% accuracy, which is a huge jump (about 30% better) compared to previous methods.
It performed as well as humans on these tests.
Crucially, learning to understand intentions didn't make the robot "forget" how to see objects or understand the scene. It kept its general knowledge intact.

In Summary

IntentVLM is a system that teaches robots to think like a detective: first, imagine all the possible reasons for an action, and then, use the evidence to pick the most logical one. This allows robots to understand human goals in a flexible, natural way, making them much better partners for everyday tasks.

1. Problem Definition

The paper addresses the challenge of open-vocabulary human intention recognition in human-robot interaction (HRI).

The Core Challenge: Unlike physical actions, intentions are latent mental states that are unobservable. Existing methods often rely on closed-vocabulary approaches (predefined sets of labels), which fail to capture the nuance, diversity, and compound motivations of human goals in unconstrained environments.
Limitations of Current Approaches:
- Closed-Set Classification: Restricts the model to a fixed label space, limiting expressiveness.
- Direct Generation: Single-pass generation by Large Language Models (LLMs) or Vision-Language Models (VLMs) often leads to hallucinations and inconsistent predictions when asked to infer intentions directly from video without constraints.
- Static Perception: Many prior works rely on static images, failing to capture the temporal dynamics and contextual evolution essential for understanding intent.
Goal: To develop a system that can infer free-form, natural language descriptions of human intentions from video streams without relying on a predefined action vocabulary, while maintaining robustness and reducing hallucinations.

2. Methodology: IntentVLM

The authors propose IntentVLM, a novel two-stage video-language framework inspired by forward-inverse modeling in cognitive science. The core hypothesis is that the brain first simulates plausible future states (forward) before reasoning backward to infer intent (inverse).

Architecture Overview

The framework utilizes two frozen Video-Language Models (VLMs) augmented with LoRA (Low-Rank Adaptation) adapters for parameter-efficient fine-tuning. The process is divided into two sequential stages:

Stage 1: Goal Candidate Generation (The "Forward" Step)

Input: Observed video sequence ( $V_O$ ) and a textual query ( $Q_O$ ).
Function: A generative module produces a diverse set of $K$ plausible goal candidates ( $G_t = \{G_1, ..., G_K\}$ ) that explain the observed behavior.
Mechanism: This acts as a "proposal" step, simulating potential outcomes or goals based on the visual context. It expands the search space to include open-vocabulary possibilities rather than restricting to a fixed list.
Implementation: A Qwen3-VL model (8B parameters) is fine-tuned to generate these candidates.

Stage 2: Structured Selection (The "Inverse" Step)

Input: The original video, the query, and the set of candidate goals generated in Stage 1.
Function: An inference module evaluates the candidates and selects the single most plausible intention.
Mechanism: This acts as an "expert" that reasons over the candidates, ranking them based on consistency with visual evidence and contextual information.
Implementation: A separate Qwen3-VL instance (2B or 4B parameters) is fine-tuned to perform the selection. The final output is the goal with the highest posterior probability, treated as the inferred intention.

Training Strategy:

Parameter-Efficient Fine-Tuning (PEFT): Only LoRA adapters are trained, keeping the base VLM weights frozen to preserve general world knowledge and prevent catastrophic forgetting.
Loss Function: Cross-entropy loss is computed between the generated tokens and the ground-truth labels (candidates for Stage 1, true intention for Stage 2).

3. Key Contributions

Open-Vocabulary Framework: Moves beyond closed-set classification to a generative-selection paradigm, allowing the model to recognize intentions not present in a predefined training vocabulary.
Cognitive-Inspired Two-Stage Design: Explicitly decomposes intention recognition into proposal (generating hypotheses) and selection (verifying hypotheses). This structure mimics human cognitive processes (forward-inverse modeling) and significantly reduces hallucinations compared to direct generation.
State-of-the-Art Performance: Achieves human-level performance on intention recognition benchmarks, outperforming existing baselines by a significant margin.
Robustness and Efficiency: Demonstrates that the modular design prevents catastrophic forgetting (preserving general scene understanding) and achieves high performance with smaller model sizes (4B) compared to larger baselines.

4. Experimental Results

The model was evaluated on two benchmarks: IntentQA (intention recognition) and Inst-IT Bench (instance-level reasoning).

IntentQA Performance:
- Accuracy: IntentVLM achieved ~80% accuracy on the test set.
- Comparison: This represents a ~30% improvement over existing baselines (e.g., HME, HQGA, VGT) and matches human performance (78.49%).
- Ablation: The 4B parameter model outperformed the 2B version. Fine-tuning significantly boosted performance over zero-shot settings.
Open-Vocabulary Capability:
- In a free-form generation setting (no candidate options provided), direct generation by standard VLMs performed poorly (<20 ROUGE score).
- IntentVLM's two-stage approach improved performance by ~5 points on average in open-vocabulary settings by constraining the generation space through the candidate selection mechanism.
Generalization (Catastrophic Forgetting):
- Training on IntentQA did not degrade the model's ability to perform instance-level reasoning on the Inst-IT Bench. The fine-tuned model remained competitive with zero-shot counterparts.
- Efficiency: The 4B parameter IntentVLM achieved performance comparable to models exceeding 7B parameters, highlighting a superior size-performance trade-off.

5. Significance and Impact

Advancing Human-Robot Interaction: By enabling robots to understand subtle, open-ended human goals (e.g., "Make coffee" vs. "Heat water for tea") without pre-programmed scripts, IntentVLM paves the way for more proactive and socially aware robotic assistants.
Bridging Cognitive Science and AI: The work successfully translates the theoretical concept of forward-inverse modeling into a practical, scalable deep learning architecture.
Reliability in Reasoning: The structured reasoning approach (generate-then-select) offers a solution to the "hallucination" problem common in generative AI, making it more suitable for safety-critical applications like caregiving or assistive robotics.
Scalability: The use of LoRA adapters allows for efficient adaptation of large foundation models to specific cognitive tasks without the computational cost of full fine-tuning.

In conclusion, IntentVLM demonstrates that decomposing intention recognition into a structured, two-stage cognitive process allows video-language models to achieve human-level understanding of complex, open-vocabulary human goals, offering a robust foundation for the next generation of intelligent robotic systems.