💻 computer science

TreeSoc: Tree-Structured Dynamic Reasoning and Tool Synergy for Soccer Video Understanding

TreeSoc is a novel framework that enhances soccer video understanding by reformulating question answering as a hierarchical search problem using dynamic depth-first search and adaptive tool integration, achieving state-of-the-art performance on SoccerBench and demonstrating strong cross-domain generalization.

Original authors: Thanh-Nhan Vo, Thanh-Khoi Nguyen, Trong-Thuan Nguyen, Trung-Hoang Le, Minh-Triet Tran

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Thanh-Nhan Vo, Thanh-Khoi Nguyen, Trong-Thuan Nguyen, Trung-Hoang Le, Minh-Triet Tran

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to understand a chaotic, fast-paced soccer match just by glancing at a single snapshot. You might guess the score, but you'd likely miss why a player got a yellow card, who that specific striker is, or the intricate play that led to the goal. For a long time, computer vision models tried to do exactly this: they looked at a video and tried to spit out an answer in one giant, instant leap. The paper argues that this "one-shot" approach is like trying to solve a massive jigsaw puzzle by guessing the whole picture before looking at a single piece. It often leads to mistakes that can't be fixed because the model never checks its work.

Enter TreeSoc, a new way of thinking about soccer video questions. Instead of a single leap, TreeSoc acts like a super-organized detective who refuses to guess until they've gathered all the clues.

The Detective's Notebook: A Tree of Questions

The paper suggests that the best way to understand a complex soccer video is to break the big question down into a "tree" of smaller, manageable questions. Imagine you ask, "Why did the referee stop the game?" A standard model might just guess "foul." TreeSoc, however, builds a branching path:

First, where is the ball?
Next, who touched it?
Then, did the referee see it?
Finally, what does the rulebook say about that action?

This process uses a "depth-first search," which is a fancy way of saying the detective follows one branch of the tree all the way down to the bottom before jumping to the next branch. If the detective finds a clue that changes the story (like realizing the player wasn't actually fouled), they can update their plan by modifying, skipping, or retrying specific steps in the queue. This "adaptive replanning" allows the system to refine its path based on new evidence, though it does not discard the underlying tree structure.

The Toolbox: Calling in the Experts

TreeSoc doesn't try to be an expert at everything itself. Instead, it acts as a manager who knows exactly which specialist to call for each job.

Need to count players? It calls a player detector (like YOLO26).
Need to read the scoreboard? It uses OCR (optical character recognition).
Need to know a player's name or team history? It digs into external databases like SoccerWiki.
Need to understand a specific foul? It uses a foul recognition tool.

The paper explicitly argues against the idea that a single, giant "monolithic" model can do all this perfectly on its own. They found that relying on just one big brain often leads to "hallucinations"—making up facts that sound good but aren't true. By forcing the system to use specific tools for specific jobs, TreeSoc aims to keep the facts straight, though the system is only as reliable as the perception tools it uses.

The Scoreboard: How Well Did It Do?

The authors tested this detective on a rigorous set of soccer challenges called SoccerBench. The results were quite promising:

On text-based questions (like "Who is the coach?"), it got 85.2% right.
On image-based questions (like "What number is on the jersey?"), it hit 87.4%.
On video-based questions (like "What happened in the last 10 seconds?"), it scored 82.2%.

These numbers suggest that TreeSoc is currently better at these tasks than many other open-source models and even some expensive commercial AI tools.

But here is the really cool part: The paper shows that this detective isn't just good at soccer. When they tested TreeSoc on a completely different dataset of everyday videos called NExT-QA (which has nothing to do with soccer), it still scored 74.16%. This suggests that the method of breaking problems down into a tree and using tools is a powerful trick that works even outside the soccer field.

Where It Still Trips Up

The paper is honest about where the detective fails. Sometimes, TreeSoc gets distracted by a flashy but unimportant action (like a goalkeeper saving a ball) and misses the main event (like a substitution). Crucially, the system does not always "catch" its own mistakes; if the very first clue it gathers is wrong (like misidentifying a player), that error can propagate through the rest of the tree, leading to a wrong final answer. The authors note that the system is only as good as the tools it uses; if the player detector is confused, the whole reasoning chain can wobble.

In short, TreeSoc proposes that understanding complex videos isn't about having a bigger brain, but about having a better strategy: ask small questions, use the right tools, and be willing to adjust your plan when the evidence changes. It's a shift from "guessing the answer" to "solving the mystery," even if the solution isn't always perfect.

Technical Summary: TreeSoc

Problem Statement

Automated understanding of complex soccer scenarios from video remains a significant challenge for contemporary Vision-Language Models (VLMs). Current models often suffer from shallow cross-modal alignment and fundamental limitations in multi-step reasoning and coordinated tool integration. In the soccer domain, high-level Visual Question Answering (VQA) requires reasoning over complex spatio-temporal events, such as tactical buildups, player interactions, and causal factors behind refereeing decisions. Existing approaches typically rely on single-pass inference and implicit visual-textual alignment, which limits their ability to construct explicit reasoning trajectories, coordinate specialized perception tools, or revise intermediate decisions. Consequently, early visual misinterpretations can propagate directly to the final answer, particularly when relevant evidence is distributed across multiple frames or dependent on external knowledge.

Methodology: TreeSoc Framework

The authors propose TreeSoc, a structured reasoning framework that reformulates soccer video question answering as a hierarchical search problem rather than a single-pass prediction. The system employs a Multimodal Large Language Model (MLLM) as a central coordinator to recursively decompose complex queries into manageable subtasks within a dynamic reasoning tree.

Core Mechanisms

Dynamic Depth-First Search (DFS): TreeSoc utilizes a DFS agent to traverse a hierarchical reasoning tree. Instead of a fixed pipeline, the system incrementally constructs a query-specific tree where nodes represent sub-queries and edges represent dependencies. The search is bounded by a maximum depth ( $D_{max}$ ) and branching width ( $W_{max}$ ).
Recursive Decomposition and Planning: At each node, the coordinator determines whether a query can be answered directly or must be decomposed into an ordered subtask queue ( $Q_n$ ). This ordering captures dependencies (e.g., localizing an action before identifying a player).
Adaptive Tool Routing: For atomic subtasks, the system dynamically routes requests to specialized domain-expert modules or external databases.
- Perception Tools: Includes YOLO26 for object detection, PRTReID for player re-identification, face recognition modules, and UniSoccer for action classification.
- Retrieval Sources: Queries external knowledge bases like SoccerWiki and SoccerReplay1988 for factual data (lineups, player history, match stats).
Self-Correcting Feedback Loop: A critical component is the "Reflection" step. After executing a subtask, the system evaluates the intermediate result ( $r_{n,i}$ ) against the accumulated evidence ( $S_t$ ). Based on this reflection, the execution queue is updated to Keep, Modify, Insert, Skip, Retry, or Stop specific subtasks. This allows the system to replan dynamically, mitigating error propagation by verifying intermediate states before aggregating the final answer.
Evidence Accumulation: The system maintains a context state $S_t$ that grows with each resolved subtask. Later operations utilize concrete outputs (e.g., recognized player identities, localized timestamps) rather than relying on assumptions made during initial decomposition.

Key Contributions

The paper outlines four primary contributions:

Tree-Structured Reasoning Formulation: A novel approach for soccer VQA that explicitly decomposes complex questions into hierarchical subtasks, moving away from monolithic end-to-end prediction.
LLM-Driven Coordination: A mechanism that dynamically routes subtasks to specialized perception tools and external knowledge sources, enabling the integration of diverse analytical skills.
Adaptive Replanning: The incorporation of DFS-based execution allows the system to revise its reasoning path based on intermediate results, reducing the impact of error propagation.
Empirical Validation: Demonstration of the framework's effectiveness on both domain-specific (SoccerBench) and general video QA (NExT-QA) benchmarks.

Experimental Results

The authors evaluated TreeSoc on the SoccerBench test set and the NExT-QA dataset.

SoccerBench Performance: TreeSoc achieved state-of-the-art results across all modalities:
- TextQA: 85.2%
- ImageQA: 87.4%
- VideoQA: 82.2%
- Notably, it outperformed commercial APIs (e.g., GPT-4o, Claude 3.7 Sonnet) and open-source models (e.g., Qwen2.5-VL 72B, LLaVA-Video). It achieved a perfect 100% on commentary generation (Q11) and 100% on jersey number recognition (Q5), significantly surpassing the previous agentic baseline, SoccerAgent.
Cross-Domain Generalization: On the NExT-QA dataset (in-the-wild videos), TreeSoc attained 74.16% accuracy, outperforming several strong baselines including SeViLA (73.80%) and TS-LLaVA.
Fine-Grained Analysis: Ablation studies on NExT-QA showed strong performance in causal reasoning (75.64% on "How" tasks) and descriptive reasoning (90.87% on "Location" tasks), suggesting the tree-structured approach effectively isolates causal conditions and prevents logical hallucinations.

Significance and Claims

The paper claims that structured, tool-augmented tree reasoning is an effective paradigm for robust video understanding. By reformulating VQA as a state-space search, TreeSoc addresses the limitations of single-pass models in handling complex, multi-step soccer scenarios. The authors argue that the ability to adaptively route tools and replan based on intermediate evidence allows the system to ground predictions in domain-specific concepts and external knowledge, thereby reducing visual-temporal noise.

While the results are strong, the paper maintains a modest tone regarding limitations. It acknowledges that TreeSoc's performance is dependent on the reliability of its underlying perception tools and retrieval modules. Furthermore, the tree-search process introduces additional computational overhead compared to single-pass VLM inference. The authors conclude that future work should focus on more efficient search strategies, uncertainty-aware replanning, and broader adaptation to general video understanding beyond the soccer domain.