Towards Comprehensive Basketball Understanding

Watching a professional basketball game is a rich, layered experience. To truly understand what is happening, a viewer must do more than simply see a ball move across a court. One must recognize the specific players involved, track the precise moment a pass is made, locate exactly where a shot was taken, and connect these visual moments to a vast library of facts about the teams, the players' careers, and the rules of the sport. For decades, computers have struggled to perform this kind of deep, integrated understanding. While modern artificial intelligence has become quite good at identifying objects in photos or answering simple questions about images, it often fails when asked to combine these skills. It might recognize a player's face but fail to recall their career statistics, or it might see a shot being taken but be unable to say exactly where on the court it happened. This gap between seeing and understanding is the central challenge that a new study from researchers at Peking University and Shanghai Jiao Tong University aims to bridge.

The researchers approached this problem by first building a massive, rigorous testing ground called BasketballBench. They gathered data from the 2025–2026 NBA season, collecting over 2,500 video clips of game possessions, official records for 530 active players, and thousands of structured game details. From this material, they crafted nearly 8,000 specific questions that required different types of thinking. Some questions were simple, asking a computer to read the score from a broadcast graphic or identify a player from a photo. Others were far more complex, demanding that the computer watch a video, figure out who made a specific play, locate where it happened in time and space, and then retrieve a specific fact about that player's career from a database. The goal was to see if current artificial intelligence systems could handle these tasks individually and, more importantly, whether they could combine these abilities to solve a single, multi-step problem.

When the researchers tested the most advanced general-purpose artificial intelligence models available, the results were revealing. These powerful systems performed well on straightforward tasks. They could read the score from a screen or identify a jersey number with high accuracy. However, their performance dropped sharply when the questions required them to link what they saw in the video to what they knew about the game. For instance, when asked to identify the player who made a shot and then find that player's career statistics, the models often stumbled. They could see the action, but they struggled to bind that action to the correct person and then pull the right information. The study showed that these general models are not naturally equipped to weave together visual perception, spatial reasoning, and factual knowledge into a single, coherent answer.

To address this limitation, the team developed a new system called BasketballSkills. Instead of relying on a single, massive model to do everything at once, they built a framework that acts like a specialized team of experts. This system uses a central controller that reads a question and then selects a specific set of tools to solve it. These tools are specialized for basketball: one tool tracks players and the ball, another identifies the type of play happening, a third reads the jersey numbers, and a fourth searches a structured database of player facts. The controller does not guess; it follows a clear, step-by-step procedure. If a question asks about a player's shooting percentage after a specific play, the system first tracks the video to find the player, then identifies their jersey number, and finally queries the database for their statistics. It validates each step before moving to the next, ensuring that the evidence gathered is accurate before forming a final answer.

The results of this new approach were striking. In tests across the ten different types of questions, the BasketballSkills system outperformed the best commercial artificial intelligence models on eight of them. It excelled particularly on the complex tasks that required combining multiple abilities, such as identifying a player from a video clip and then retrieving their career data. While the general models often failed to connect the visual evidence to the factual knowledge, the specialized system successfully composed these different capabilities to reach the correct conclusion. The study suggests that for complex, real-world domains like professional sports, the path forward may not be to make general models bigger, but to build systems that can explicitly organize and combine specialized skills. By breaking down a difficult problem into a sequence of reliable, domain-specific steps, the researchers demonstrated that computers can be taught to understand the game of basketball with a depth and accuracy that was previously out of reach.