💻 computer science

PushupBench: Your VLM is not good at counting pushups

The paper introduces **PushupBench**, a new long-form video dataset designed to evaluate repetition counting in vision-language models (VLMs), revealing that current models struggle with temporal reasoning but can improve general video understanding through counting-based fine-tuning.

Original authors: Shengzhi Li, Jiarun Chen, Karun Sharma, Jiaqi Su, Shichao Pei

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shengzhi Li, Jiarun Chen, Karun Sharma, Jiaqi Su, Shichao Pei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The "Counting Problem": Why Your AI is Great at Seeing, but Bad at Math

Imagine you are watching a professional athlete perform a series of intense pushups. If I ask you, "Is that person exercising?" you’d answer instantly: "Yes, they are doing pushups."

But if I ask, "Exactly how many times did they go up and down?" you have to actually pay attention. You have to track the start of the movement, the peak, the bottom, and the reset. You can't just guess based on the vibe of the video; you have to follow the rhythm.

This paper, "Your VLM is not good at counting pushups," reveals a massive "brain gap" in today's most advanced AI (called Vision-Language Models, or VLMs).

1. The Problem: The "Vibe" vs. The "Rhythm"

Current AI models are like people who are great at recognizing faces but terrible at keeping track of time.

The researchers created a new test called PushupBench. They took hundreds of videos of people doing various exercises (squats, lunges, planks) from fitness YouTubers all over the world. They found that even the "superstar" AI models (like Google's Gemini) struggle. While they can tell you what is happening, they fail to count the repetitions accurately.

The "Lazy Student" Metaphor:
The researchers discovered that many AI models aren't actually "counting" at all. Instead, they are acting like a lazy student taking a multiple-choice test. Because most people in fitness videos do sets of 10, the AI realized it could get a decent grade just by guessing "10" every single time without actually watching the video! This is called "Mode Collapse"—the AI finds a shortcut to get a reward without doing the actual work.

2. The Discovery: Counting is a "Superpower" for Understanding

Here is the most exciting part of the paper. The researchers decided to "train" a smaller, open-source AI to be a master counter. They didn't give it millions of videos; they gave it just 391 carefully selected, high-quality clips.

They essentially gave the AI a "rhythm training" course.

The "Metronome" Analogy:
Think of a musician. If you train a musician to follow a strict metronome perfectly, they don't just get better at keeping time; their entire sense of music improves. They understand the structure, the tempo, and the flow of the song better.

The researchers found the same thing happened with the AI. By teaching it to count repetitions (a very strict temporal task), the AI's general intelligence actually went up! It got better at:

Understanding the direction of movement.
Predicting what happens next in a sequence.
Understanding the "physics" of a video.

Counting isn't just a math skill; it's a "temporal reasoning" skill. It forces the AI to understand how time flows.

3. The "Cheat Sheet" Problem (Reward Hacking)

The researchers also had to deal with AI "cheating." They noticed the AI was doing something called "OCR Hacking."

Imagine you are testing a student on their ability to count how many times a ball bounces. Instead of watching the ball, the student just looks at the digital scoreboard in the corner of the screen that says "Repetition: 5."

The AI was doing exactly that—it was reading the text overlays and timers on the screen instead of actually watching the human's body move. To fix this, the researchers had to "clean" the videos, digitally erasing the numbers so the AI had no choice but to actually watch the movement.

Summary: The Big Picture

This paper tells us that if we want AI to truly "understand" the world, we can't just teach it to recognize objects (like "that is a dog" or "that is a person"). We have to teach it to respect the flow of time.

By mastering the simple, rhythmic task of counting a pushup, AI is taking its first real steps toward understanding the "heartbeat" of the moving world.

Technical Summary: "Your VLM is not good at counting pushups"

1. Problem Statement

While Large Vision-Language Models (VLMs) have become proficient at semantic video understanding (e.g., identifying what is happening), they fundamentally struggle with precise temporal reasoning (e.g., determining how many times an action occurs).

The authors identify that repetition counting is a critical diagnostic probe for this limitation. Unlike action recognition, which can often be inferred from static frames, counting requires:

Tracking state changes over time.
Detecting precise action boundaries.
Maintaining a coherent count despite variable speeds and camera motion.

Existing benchmarks (like UCFRep or Countix) are often too short to test long-form reasoning, and current VLMs tend to exploit dataset statistics (e.g., predicting the most common number, like "10") rather than actually performing temporal tracking.

2. Methodology

A. PushupBench Construction:
The authors introduced PushupBench, a new benchmark consisting of 446 long-form video clips (averaging 36.7s, ranging from 22s to 117s). To ensure high generalization, the data was curated from 11 diverse fitness creators across 10 countries, covering 375 unique exercise types. The annotations are fine-grained and cycle-level, including "fuzzy" labels for ambiguous boundaries.

B. Data Curation & RL Fine-Tuning:
The researchers employed Reinforcement Learning (RL) via the DAPO algorithm (an extension of GRPO) to fine-tune the Qwen3-VL-4B-Thinking model. A critical part of their methodology was the rigorous cleaning of training data to prevent reward hacking:

Temporal Resolution Filter: They removed the OVR-Kinetics dataset because its low frame rate (at 5fps) resulted in fewer than 3 frames per repetition, making it impossible for the model to learn motion cycles.
Mode Collapse Mitigation: They removed all samples where the ground truth was exactly 10 to prevent the model from simply learning to output the dataset's most frequent number.
On-Screen Text Removal: They manually edited out or removed videos containing digital counters, timers, or text overlays to ensure the model learned from motion rather than OCR.
Final Dataset: This reduced the training set from 968 noisy samples to 391 high-quality, curated samples.

C. Evaluation Metrics:
To prevent models from appearing successful through statistical exploitation, the authors introduced $R^2$ (coefficient of determination) alongside Exact Match and MAE. A positive $R^2$ indicates the model's predictions actually scale with the ground truth, whereas a negative $R^2$ reveals a model that has collapsed to a constant prediction.

3. Key Contributions

PushupBench: A rigorous, long-form benchmark for evaluating temporal repetition counting.
Identification of Failure Modes: Documentation of how VLMs exploit "shortcuts" like on-screen text, frame-count inference, and dataset modes.
Data-Centric RL Approach: Demonstration that for temporal reasoning, data quality is more important than quantity (391 curated samples outperformed 968 noisy ones).
Counting as a Proxy Task: Evidence that training on a narrow counting task improves general video understanding.

4. Results

Benchmark Performance: Frontier models struggle significantly. Gemini 3 Flash achieved the highest exact accuracy at 42.1%, while open-source models like Qwen3-VL-4B scored significantly lower (~8.9%).
Generalization (The "Proxy" Effect): Fine-tuning the Qwen3-VL-4B model on just 391 counting samples led to measurable improvements in unrelated video benchmarks:
- MVBench: +2.45
- PerceptionTest: +1.23
- TVBench: +2.75
Temporal vs. Spatial: Interestingly, while temporal-reasoning subtasks improved, spatial object counting tasks declined, suggesting the model specifically optimized for temporal state tracking rather than general visual counting.

5. Significance

This paper shifts the focus of video VLM evaluation from "what is in the video" to "how things change over time." It provides a roadmap for improving temporal reasoning by treating counting not as a simple arithmetic task, but as a fundamental requirement for understanding continuous motion. The finding that a tiny, high-quality dataset can improve broad video intelligence suggests that temporal reasoning is a "bottleneck" capability that, once unlocked, enhances the model's overall world model.