UniTraffic-Agent: Unified Traffic Video Reasoning for AI City Challenge 2026 Track 3 with Two Out-of-Domain Evaluations

Imagine you are a detective trying to solve a mystery, but instead of a crime scene, your evidence is a chaotic, moving video of a busy city street. This is the world of traffic video understanding, a field where computers try to make sense of what's happening on our roads. For a long time, computers were great at spotting a single object, like a red car or a stop sign, in a still photo. But real life isn't a photo; it's a movie. To understand a traffic accident or a near-miss, a computer needs to do more than just "see"; it needs to watch, think, and connect the dots over time. It has to figure out who did what, why they did it, and when it happened. This is tricky because traffic videos are messy: the camera might be a weird fish-eye lens, the action might happen in a split second, and the same clip might need to answer many different questions. The goal of this research is to build a computer system that acts like a super-smart, patient detective who never misses a detail, no matter how the camera is set up.

Enter UniTraffic-Agent, a new digital detective created by a team of researchers to tackle the "Traffic Anomaly Reasoning" challenge. Think of this agent as a highly organized investigator who doesn't just glance at a video and guess. Instead, it follows a strict four-step routine: Observe, Reason, Act, and Verify.

First, in the Observe phase, the agent is smart about how it looks at the video. Instead of trying to process every single frame (which would be like reading every word of a book when you only need the plot summary), it picks the most important moments. If a question asks about something happening at a specific time, the agent zooms in on the seconds right before and after that moment, while also grabbing a few frames from the beginning and end to keep the big picture in mind. It's like a detective who knows exactly which seconds of a security tape to rewind to catch the crucial clue.

Next comes Reason. Here is where the agent shines. Often, a single video clip has many questions attached to it, like "Who ran the light?" and "What was the weather?" and "How long did the car wait?" Older systems might try to answer each question one by one, which can lead to contradictions—like saying the car was red in one answer and blue in another. UniTraffic-Agent does something different: it watches the whole clip once, builds a single, shared story about what happened, and then uses that same story to answer all the questions at once. This ensures the detective's story is consistent from start to finish.

Then, the Act phase kicks in. Different tasks require different formats. One task might need a simple "Yes" or "No," while another needs a detailed JSON file with 13 different fields describing a traffic violation. The agent uses special "adapters"—think of them as translators or shape-shifters—to take its shared story and reshape it into the exact format required for each specific job.

Finally, the Verify step acts as a quality control inspector. It double-checks the answers, makes sure the names and times match up, and if something looks weird, it goes back to the cached video frames to try again. This "retry" mechanism helps the agent fix its own mistakes without needing human help.

The researchers tested this agent on three very different types of traffic videos: standard surveillance footage, distorted fish-eye lens videos, and dashcam videos from cars. The results were impressive. On the fish-eye task, the agent ranked 2nd out of all competitors, and on the pedestrian intention task, it ranked 4th. Even on the main, most difficult task, it placed 16th. The team found that while the agent was excellent at spotting the actors and the general flow of events, it sometimes struggled with the very long, detailed descriptions or with interpreting the warped geometry of fish-eye lenses. However, the study suggests that by combining smart observation with a unified reasoning process, we can build traffic AI that is much more reliable and consistent than before. The code for this "detective" is now open for others to learn from and improve upon.