Offline Dynamic AOI Generation and Runtime Gaze Mapping for 360° Video in Virtual Reality: A Systems and Feasibility Study
This paper presents and validates an offline-to-runtime pipeline for 360° VR eye-tracking that utilizes open-vocabulary detectors to generate dynamic areas of interest and performs frame-synchronized gaze mapping, benchmarking models like Grounding DINO and YOLO-World to balance detection coverage with processing efficiency while identifying key technical challenges for future refinement.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine trying to understand what a person is looking at while they wear a headset that shows them a full, 360-degree world. In a flat movie, you can easily draw a box around a person or a car and say, "They are looking at this." But in a spherical video, the world wraps around the viewer. Objects can stretch, shrink, or split in half as they move across the edges of the image. To study how people pay attention in these immersive environments, researchers need a way to automatically identify these moving targets and track the viewer's eyes as they follow them. This is the core challenge of eye-tracking in virtual reality: creating a reliable map of what matters in a scene that is constantly changing and wrapping around the user.
A team of researchers at the University of Cádiz in Spain has built a new system to solve this problem. They created a two-part process that separates the hard work of identifying objects from the actual moment a person wears the headset. First, they take a 360-degree video and use powerful computer programs to scan it, looking for specific things like people, animals, or objects based on simple text descriptions. These programs draw invisible boxes around the items they find. The team then turns these boxes into a special digital map that the headset can read. Later, when a participant puts on the headset to watch the video, the system uses this map to instantly know which object the person is looking at, frame by frame, without needing to do the heavy computer work while the video is playing.
To test if this system works, the researchers tried it with three different types of videos: a room with people and laptops, a forest scene, and a wildlife clip featuring lions and gazelles. They also tested three different computer programs to see which one was best at finding the objects. One program, called Grounding DINO, was the most thorough, finding the most objects even in difficult scenes, but it took a long time to process the video. Another program, YOLO-World, was incredibly fast—about nine times faster than the first one—but it missed many objects, especially the small animals in the wildlife clip. A third option, OWLv2, fell somewhere in the middle. The researchers found that no single program was perfect for every situation; the best choice depended entirely on what was in the video.
When they tested the full system with eight volunteers, the technology worked smoothly. The headset successfully tracked where the participants were looking and matched those glances to the objects the computer had identified. The system recorded every time a person focused on something, creating a detailed log of their attention. However, the study also revealed where the system struggles. In the wildlife video, the system often failed to notice the animals. This happened because the animals were small, moved quickly, and sometimes appeared in parts of the video where the image is naturally distorted. The computer programs had trouble finding these tiny, shifting targets, meaning the system missed a lot of the time the participants were actually looking at the animals.
The researchers concluded that their method is a solid foundation for future studies. It proves that you can automatically create these attention maps and use them in real-time virtual reality experiments. The separation between the offline preparation and the live experiment makes the process reliable and repeatable. However, they also made it clear that the technology is not yet ready for every type of video. While it works well for clear, structured scenes like a room with people, it still needs improvement to handle small, fast-moving objects in complex natural environments. The study does not claim to have solved the problem of eye-tracking in 360-degree video completely, but it has provided a working blueprint and a clear list of the specific hurdles that still need to be overcome.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.