Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding

Molmo2 is a new family of open-weight vision-language models that achieves state-of-the-art performance in video understanding and pixel-level grounding by leveraging seven newly collected video datasets and a novel training recipe, all developed without relying on proprietary models.

The Gist

Imagine you have a super-smart robot assistant that can watch videos, look at photos, and answer questions about them. For a long time, the best versions of this robot were like secret recipes kept in a locked vault by big tech companies. You could use them, but you couldn't see how they were made, what they learned from, or how to make them better.

Molmo2 is a new family of robots that breaks down those vault doors. It is the most advanced "open-source" video brain available today, meaning anyone can download it, study it, and build upon it.

Here is the simple breakdown of what makes Molmo2 special, using some everyday analogies:

1. The "Pointing" Superpower

Most video robots are like people who can tell you what is happening in a movie ("A dog is running"), but they can't tell you exactly where or when.

• The Old Way: If you asked, "When did the cup fall off the table?", a standard robot might just say, "It fell around the middle."
• The Molmo2 Way: Molmo2 is like a laser-pointer-wielding detective. It can say, "The cup fell at this exact second (0:45), and here is the exact pixel on the screen where it landed." It can even track the cup as it rolls across the floor, keeping its finger on it the whole time.

2. Learning Without Cheating (No "Distillation")

Many open-source robots today are like students who cheat by copying the homework answers from the "smart kids" (the proprietary, secret robots). They learn by watching what the secret robots say, which limits how smart they can get because they are just mimicking.

• Molmo2's Approach: Molmo2 is like a student who went to a massive library and read millions of books and watched millions of videos themselves. They didn't copy the answers; they learned the concepts from scratch. This makes their knowledge more original and robust.

3. The "Human Narrator" Training

To teach a robot to understand video, you need to describe what's happening.

• The Problem: Typing descriptions is slow and often misses small details.
• The Molmo2 Solution: The creators hired humans to speak their descriptions of video clips. Imagine a person watching a chaotic scene and talking fast: "Okay, the raccoon is typing on the laptop, then the dog drops his pencil, and the fan starts spinning!"
  • Because humans speak faster than they type, they included way more detail.
  • Then, the robot listened to these stories and learned to match the words to the moving pictures. This resulted in a robot that understands tiny details, like "the dog is wearing a red collar," not just "there is a dog."

4. The "Long Movie" Challenge

Most video robots get tired and confused when watching a long movie (like a 30-minute clip). They tend to forget the beginning by the time they reach the end.

• Molmo2's Trick: The researchers taught Molmo2 a special way to "pack" information. Imagine trying to fit a whole novel into a small suitcase. Instead of just shoving it in, Molmo2 folds the pages efficiently so it can read the whole story without losing a single word. This allows it to handle longer videos better than other open robots.

5. The "Counting" and "Tracking" Gym

The paper introduces a new set of "gym equipment" (datasets) specifically designed to train the robot's muscles:

• Counting: Can you count how many cars pass a yellow taxi? (Molmo2 is great at this).
• Tracking: Can you follow a specific dancer in a group of 50 people moving from left to right? (Molmo2 can do this).
• Pointing: Can you click on the exact moment a ball hits the net? (Molmo2 can do this).

The Bottom Line

Think of the current AI world as a race. The big companies (like Google and OpenAI) have the fastest cars, but they are driving in a closed track where no one else can see the engine.

Molmo2 is the best car built in a public garage. It's not quite as fast as the secret super-cars yet, but it's beating every other public car by a huge margin. More importantly, because the blueprints are open, the whole world can now look under the hood, fix the engine, and help build the next generation of video understanding together.

In short: Molmo2 is a video-watching robot that can point at things, count them, and track them through time, all while being built entirely in the open so everyone can learn from it.

Technical Summary

1. Problem Statement

Despite the rapid advancement of Vision-Language Models (VLMs), the state-of-the-art (SOTA) models for video understanding remain proprietary (closed weights, data, and training recipes). Existing open-weight models suffer from two critical limitations:

• Data Dependency: They often rely on synthetic data distilled from proprietary VLMs, inheriting their biases and limiting the community's ability to improve upon them.
• Lack of Grounding: Most models lack fine-grained grounding capabilities (pointing, tracking, and counting) in the temporal domain. While image grounding is common, video grounding requires identifying when and where an event occurs across frames, a capability largely missing in open models and even limited in proprietary ones.

2. Methodology

Molmo2 addresses these gaps through a fully open ecosystem comprising novel datasets, a robust training recipe, and architectural innovations.

A. Data Collection (The Molmo2 Corpus)

The authors constructed a massive, fully open multimodal corpus without distilling from proprietary models. Key components include:

• 9 New Datasets:
  • Molmo2-Cap: 104k dense video captions (avg. 924 words/video) generated via a human-in-the-loop pipeline where annotators speak descriptions, which are transcribed and enriched with frame-level details from an early Molmo model.
  • Molmo2-AskModelAnything: 140k human-authored video QA pairs with rich, fine-grained answers.
  • Molmo2-VideoPoint & AcademicVideoPoint: 650k+ pointing queries across 280k videos, covering objects, actions, and anomalies.
  • Molmo2-VideoTrack & AcademicVideoTrack: 3.6k video clips with 15k complex tracking queries, converting bounding-box and segmentation tracks into point-based supervision.
  • Synthetic QA: Large-scale synthetic datasets (CapQA, SubtitleQA) generated using the authors' own captioning models to ensure no proprietary leakage.
  • Multi-Image Datasets: New datasets for multi-image QA and pointing to support complex reasoning across image sets.
• Data Pipeline: Utilizes a "human-and-LLM collaboration" approach. Humans provide high-quality initial data (speech-to-text, manual QA), which is then refined or expanded by open LLMs, avoiding the "closed-loop" bias of proprietary distillation.

B. Model Architecture

Molmo2 follows a standard encoder-connector-LLM design but with specific adaptations for video:

• Vision Encoder: Uses SigLIP 2 (So400m/14) for image encoding.
• Connector: Employs a pooling mechanism (2x2 for images, 3x3 for video frames) to reduce token count, followed by a shared MLP.
• LLM: Based on Qwen3 (for 4B/8B models) and OLMo 3 (for the 7B model).
• Input Handling: Supports single images, multi-image sets, and videos. Videos are sampled at 2 fps (up to 128 frames normally, 384 for long-context).
• Output Format: Uses a compact, HTML-like token format for points and tracks (e.g., <points coords="t x y...">), enabling efficient generation of spatial-temporal coordinates.

C. Training Strategy

The training pipeline consists of three stages:

1. Pre-training: Image captioning, image pointing, and NLP tasks.
2. Supervised Fine-Tuning (SFT): Joint training on the integrated multimodal mixture (images, videos, multi-image).
3. Long-Context SFT: A final stage increasing sequence length to 36,864 tokens to handle long videos.

Key Training Innovations:

• Token Weighting: Adjusts loss weights to prevent long outputs (like dense captions) from dominating short-answer tasks. Long captions are weighted at 0.1, pointing at 0.2, and short answers follow a heuristic based on token count.
• Message Trees: Encodes videos with multiple annotations (e.g., a video with several QA pairs) as a tree structure, linearized into a single sequence with custom attention masks to prevent cross-attention between distinct queries.
• Efficient Packing: An on-the-fly algorithm merges multiple short examples into a single sequence to maximize GPU utilization and reduce padding waste.
• Bi-directional Attention: Enables visual tokens to attend to one another, improving performance on grounding tasks.

3. Key Contributions

1. Fully Open SOTA VLMs: Release of Molmo2-4B, Molmo2-8B, and Molmo2-O-7B (based on OLMo), which are the strongest open-weight models for video understanding.
2. Novel Grounding Datasets: Introduction of the first large-scale, open datasets for video pointing and video tracking with complex, open-vocabulary queries.
3. Dense Video Captioning: A corpus of highly detailed video captions (avg. 924 words) that far exceeds the granularity of previous datasets.
4. Training Recipe: A reproducible, efficient training pipeline featuring message-tree encoding, token weighting, and packing, which significantly boosts throughput and performance.

4. Results

Molmo2 was evaluated across a broad spectrum of benchmarks, including video understanding, grounding, counting, and human preference.

• Video Understanding: Molmo2-8B achieves SOTA among open models on short-video benchmarks (MVBench, MotionBench) and is competitive on long-video tasks (Video-MME, LongVideoBench). It outperforms previous open models and approaches proprietary systems like Gemini 2.5 Pro.
• Grounding (Pointing & Tracking):
  • Video Pointing: Molmo2-8B achieves 38.4 F1 on video pointing, significantly outperforming Qwen3-VL-8B (1.5 F1) and surpassing proprietary models like Gemini 3 Pro (20.0 F1).
  • Video Tracking: On the MeViS and ReasonVOS benchmarks, Molmo2-8B achieves 70.8 J&F and 75.9 F1, outperforming specialized segmentation models and all other VLMs.
• Counting: Molmo2 excels at counting objects in videos, achieving 35.5 accuracy on video counting benchmarks, outperforming Qwen3-VL (29.6) and competing with Gemini 3 Pro.
• Human Preference: In a large-scale human preference study (105k ratings), Molmo2-8B ranks 5th overall (Elo 1057), outperforming other open-weight models and rivaling proprietary models like GPT-5 and Claude Sonnet 4.5.
• Image & Multi-Image: Maintains strong performance on image benchmarks (VQA v2, DocVQA) and achieves competitive results on multi-image tasks.

5. Significance

• Democratizing Video AI: Molmo2 breaks the monopoly of proprietary video models by providing a fully open, high-performance alternative. It proves that open-source models can match or exceed closed systems in complex tasks like video grounding.
• Foundation for Future Research: By releasing the data, code, and weights, the authors provide the community with the necessary foundations to build upon, rather than forcing researchers to rely on black-box distillation.
• Advancing Grounding: The work establishes a new standard for spatio-temporal grounding, demonstrating that open models can effectively track objects and point to events in videos, a critical capability for robotics, surveillance, and assistive technologies.
• Methodological Impact: The training techniques (message trees, token weighting, packing) offer valuable insights for efficiently training large multimodal models on diverse, long-context data.

In summary, Molmo2 represents a major leap forward in open-source video-language modeling, successfully bridging the gap between open and proprietary systems through rigorous data collection, innovative training strategies, and exceptional grounding capabilities.