💻 computer science

Latent Visual Diffusion Reasoning with Monte Carlo Tree Search

This paper proposes Latent Visual Diffusion Reasoning (LVDR), a novel framework that integrates keypoint-guided Monte Carlo Tree Search to enhance action quality assessment by generating both accurate skill evaluations and interpretable, step-by-step visual reasoning trajectories.

Original authors: Xirui Teng, Nan Xi, Junsong Yuan

Published 2026-06-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xirui Teng, Nan Xi, Junsong Yuan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a gymnast perform a complex routine or a surgeon performing a delicate operation. You want to know: "How good was that?"

Current computer programs can look at the video and give you a score (like an 8.5 out of 10). But they are like black boxes: they give you the grade, but they won't tell you why. They don't say, "The score is low because the gymnast's knee bent too early," or "The surgeon's hand shook during the stitching." They just spit out a number, leaving humans in the dark.

This paper introduces a new system called LVDR (Latent Visual Diffusion Reasoning) that acts like a super-observant coach who doesn't just give a score, but walks you through their thought process step-by-step.

Here is how it works, using simple analogies:

1. The "Denoising" Coach (The Diffusion Part)

Imagine you are trying to solve a mystery, but you only have a blurry, static-filled photo of the crime scene.

The Problem: At the very beginning of a video (the first second), the computer doesn't know much. It's like looking at that blurry photo. It has a "noisy" guess about what happened.
The Solution: The LVDR system uses a process called Diffusion. Think of this like a detective slowly cleaning a dirty window. As the video plays, the system "cleans" its initial blurry guess frame by frame.
The Result: By the end of the video, the "window" is crystal clear. The system has refined its understanding from a vague guess into a precise, coherent story of what happened. This allows it to understand the flow of the action, not just freeze-frames.

2. The "Keypoint Detective" (The MCTS Part)

Now that the system has a clear understanding, how does it explain why it gave that score? This is where Monte Carlo Tree Search (MCTS) comes in.

Imagine a human referee watching a soccer game. They don't stare at the whole field at once. They have a strategy:

First, they look at the player's feet to see if they tripped.
Then, they look at the hips to see if the balance was off.
Then, they check the arms.

The LVDR system does the same thing, but it's a digital detective that plays out thousands of "what-if" scenarios in its head very quickly.

It asks: "If I focus on the elbow, does the score change? What if I focus on the knee?"
It builds a decision tree (like a choose-your-own-adventure book) to find the most important body parts (keypoints) that actually mattered for the final score.
The Output: It highlights these specific body parts on the video with different colors. If the knee is bright red, it means the system paid the most attention to the knee to make its decision.

3. Putting It Together: The "Transparent" Score

When you use LVDR, you get two things:

The Score: Just like the old black-box models, it gives a number (e.g., "8.5").
The Reasoning Trail: It shows you a visual map of its thinking. You can see a "trail" of attention moving across the body parts, just like a human expert would scan the athlete.

Where Did They Test It?

The authors tested this "super-coach" on two very different worlds:

Sports: Basketball, soccer, climbing, and fitness exercises (like squats).
Surgery: Robot-assisted surgeries and cataract operations.

The Results

Accuracy: The system gave scores just as accurate (or better) than the best existing computer programs.
Trust: Unlike the old "black box" models, LVDR showed its work. When the researchers asked human experts to check the system's "reasoning trail," the experts agreed with the system 86% of the time.
Proof: When the researchers tried to "blind" the system by hiding the body parts the system said were important, the system's score accuracy dropped significantly. This proved that the system was actually looking at the right things, not just guessing.

In short: This paper teaches computers how to not just grade a performance, but to explain their grading by simulating a step-by-step, human-like thought process that highlights exactly which body movements mattered most.

Technical Summary: Latent Visual Diffusion Reasoning with Monte Carlo Tree Search

Problem Statement

Fine-grained skill assessment (e.g., in sports and surgery) presents a significant challenge in computer vision. Unlike standard action recognition, which identifies what action is occurring, skill assessment requires evaluating how well an action is executed. This task demands not only the recognition of visual patterns but also step-by-step visual reasoning that links low-level motion dynamics to high-level evaluative outcomes.

While recent Action Quality Assessment (AQA) models have achieved strong predictive accuracy using deep neural networks, they largely function as "black boxes." These models lack the ability to explicitly reveal the intermediate reasoning processes underlying their judgments. This opacity limits interpretability and hinders the ability of human experts to validate model outputs or derive actionable insights, which is a critical barrier for deployment in high-stakes domains like surgical education and elite athlete training where transparency and trust are indispensable.

Methodology: Latent Visual Diffusion Reasoning (LVDR)

The authors propose Latent Visual Diffusion Reasoning (LVDR), a framework designed to model and visualize the latent visual reasoning process. The framework integrates two primary components: a diffusion-based generative model for latent reasoning and a keypoint-guided Monte Carlo Tree Search (MCTS) for explicit visualization.

1. Latent Visual Diffusion Reasoning

The core hypothesis is that visual reasoning can be formulated as a progressive denoising process within a latent space.

Latent Space Construction: The model learns a target semantic distribution derived from ground-truth video semantics (e.g., expert commentary or labels).
Diffusion Process: Based on Denoising Diffusion Implicit Models (DDIM), the framework treats the beginning of a video as a noisy initialization of the reasoning process. As the video progresses, the diffusion model iteratively denoises latent embeddings of the frames.
Mechanism:
- Input: For each frame $I_i$ , 3D keypoints $K_i$ are extracted.
- State Learning: A Reasoning State Learner ( $\theta_R$ ) generates an initial visual reasoning embedding $P_i$ from the frame and keypoints.
- Denoising: A diffusion transformer ( $\theta_D$ ) refines the embedding $\hat{P}_i$ based on the previous state $\hat{P}_{i-1}$ and the initial embedding $P_i$ . This models the evolution of reasoning from a non-informative prior to a coherent target semantic state.
- Prediction: The final latent embedding is passed through a fully connected layer to predict the performance score $\hat{s}$ .
Training: The model is trained using a combined loss function comprising the diffusion denoising loss ( $\mathcal{L}_{DM}$ ) and a score regression loss ( $\mathcal{L}_{score}$ ).

2. Keypoint-Guided Monte Carlo Tree Search (MCTS)

To transform the implicit latent reasoning trajectory into an explicitly interpretable form, LVDR employs MCTS.

Motivation: Inspired by human referees who use sequential visual strategies (e.g., focusing on upper-body kinematics before lower-limb movements), the framework models visual processing as a combinatorial search space.
Tree Structure:
- Nodes: Represent states defined by specific keypoints. In sports, nodes are based on major joints (elbow, shoulder, hip, knee); in surgery, they are based on instrument parts (tip, body, tail).
- Features: Each node is represented by the joint's angle, speed, and 3D position.
Search Process: The MCTS operates over $M$ $M$ iterations per frame, consisting of four phases:
1. Selection: Traversing the tree using the Upper Confidence Bound for Trees (UCT) criterion to balance exploration and exploitation.
2. Expansion: Adding child nodes to explore unvisited paths.
3. Simulation: Estimating the expected reward of a node by leveraging the learned diffusion model $\theta_D$ to simulate interactions until a terminal state.
4. Backpropagation: Updating value estimates along the path from the leaf to the root based on simulation outcomes.
Output: The path with the highest expected reward ( $v^*$ ) is selected, revealing the sequence of critical keypoints (visual groundings) that contributed most to the final judgment.

Key Contributions

LVDR Framework: Introduction of a novel framework that models step-by-step visual reasoning as a progressive diffusion process within a latent reasoning space, enabling coherent and interpretable assessment.
Keypoint-Guided MCTS: Proposal of a mechanism to uncover and visualize latent visual reasoning sequences, identifying the most influential reasoning steps and critical keypoints.
Empirical Validation: Demonstration that LVDR achieves competitive or superior quantitative performance across multiple sports and surgical datasets while providing unprecedented transparency in its decision-making process.

Experimental Results

The authors evaluated LVDR on four diverse datasets: EgoExo4D (sports), JIGSAWS (robot-assisted surgery), FitnessAQA (fitness skills), and Cataract-101 (cataract surgery).

Quantitative Performance:
- LVDR achieved state-of-the-art results on the EgoExo4D dataset, outperforming domain-specific approaches (e.g., MAGR) and large vision-language models (e.g., GPT-4o, Gemini 2.5) in Spearman's Rank Correlation ( $\rho$ ) and relative $L_2$ distance.
- Similar superior performance was observed across JIGSAWS, FitnessAQA, and Cataract-101, with the model consistently surpassing baselines in score prediction accuracy.
- Ablation studies confirmed that removing the diffusion module leads to consistent performance degradation, validating its necessity for modeling progressive reasoning.
Interpretability and Validation:
- User Study: A domain expert evaluated the keypoint reasoning processes generated by MCTS on 50 sports videos, resulting in an 86% correctness rate, indicating strong alignment with human expert judgment.
- Keypoint Masking: Experiments showed that masking keypoints selected by MCTS caused a substantial drop in model performance, whereas masking unselected keypoints had negligible impact. This confirms that MCTS effectively identifies the critical motion cues for assessment.
- Visualization: The framework successfully visualized reasoning trajectories where the model's state progressively converged toward the ground truth, and keypoint attention patterns (e.g., focus on knee joints during squats) aligned with domain-specific biomechanical requirements.

Significance

The paper claims that LVDR bridges the gap between high predictive performance and human-understandable reasoning in fine-grained skill assessment. By explicitly modeling the latent reasoning process and visualizing the critical spatiotemporal patterns, the framework addresses the "black box" limitation of current AQA systems. This transparency is presented as a crucial step toward the real-world adoption of AI-based assessment systems in high-stakes fields where accountability and trust are required. The authors note that while the current work focuses on single-person scenarios, future directions include extending the framework to multi-person scenarios and live, real-time applications.