💻 computer science

Multi-Camera Trajectory Forecasting with Trajectory Tensors

This paper introduces a multi-camera trajectory forecasting framework that utilizes novel "trajectory tensors" and a "Which-When-Where" approach to predict object movements across multiple camera views, demonstrating superior performance over existing single-camera methods on a large-scale, multi-view dataset.

Original authors: Olly Styles, Tanaya Guha, Victor Sanchez

Published 2026-08-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Olly Styles, Tanaya Guha, Victor Sanchez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a bustling city square through a single security camera. You see a person walking briskly toward the edge of your screen. In the world of computer vision, this is a classic puzzle: "Where will they go next?" Scientists have spent years teaching computers to guess this, but they usually only look at one camera at a time. It's like trying to predict a friend's path through a maze while only seeing them through a tiny peephole; once they step out of view, you're guessing in the dark. This limitation makes it hard to track people for long periods or across large areas. To solve this, researchers are now trying to connect the dots between many cameras, creating a "super-vision" that sees the whole picture at once. The goal is to build a system that doesn't just watch what is happening right now, but predicts where a person will pop up next, in which room, and at what time, even if they are moving between different camera zones.

This paper introduces a new way to solve that puzzle, called Multi-Camera Trajectory Forecasting (MCTF). The authors, Olly Styles, Tanaya Guha, and Victor Sanchez, argue that the old way of thinking about movement—using simple X and Y coordinates like a GPS pin—is too rigid for a world with many cameras. They propose a new, more flexible tool called "trajectory tensors." Think of a coordinate as a single dot on a map, while a trajectory tensor is like a glowing, fuzzy cloud of light that covers a whole grid. This cloud can show exactly where a person is, how big they are, and even how likely they are to be in a specific spot, all while handling the fact that they might disappear from one camera and reappear in another.

The researchers tested their idea on a massive dataset they created, the Warwick-NTU Multi-camera Forecasting (WNMF) dataset. This isn't just a few seconds of video; it's 600 hours of footage captured by 15 different cameras in a university building, tracking people as they move through the halls. They set up a challenge: using just 2 seconds of a person's past movement, could the computer predict where they would be for the next 12 seconds? They broke this down into three questions: "Which camera will they appear in?" (the Which), "When will they get there?" (the When), and "Exactly where in that camera's view will they be?" (the Where).

The results suggest that their new "tensor" approach is superior to the old methods. While traditional models that use simple coordinates struggled to handle the complexity of multiple cameras, the trajectory tensor models—especially one using a 3D-CNN architecture—performed the best. They were better at guessing the right camera, the right time, and the right spot. The authors found that by treating the camera network as a single, unified grid rather than a collection of separate peepholes, the computer could learn patterns much more effectively. For instance, they showed that watching a person in multiple cameras at once (multi-view) gave the model a significant boost in accuracy compared to just looking at one camera.

However, the paper is careful not to claim this is a perfect, solved problem. The authors note that while their models outperform existing baselines, the task is still incredibly difficult, especially when predicting the exact location (Where) rather than just the general area. They also point out that their system currently works best when the camera network stays the same between training and testing; if you suddenly add or remove cameras, the model might get confused. Furthermore, while they successfully demonstrated this on a dataset of 119 multi-person scenarios, they caution that these results should be treated carefully due to the small sample size compared to the full dataset.

In essence, this paper suggests that to truly predict human movement in a complex, multi-camera world, we need to stop thinking in single dots and start thinking in glowing, multi-dimensional clouds. By using these "trajectory tensors," computers can anticipate where a person will be next with greater confidence, potentially helping surveillance systems become smarter, more efficient, and capable of tracking people over longer distances without losing them in the gaps between cameras.

Technical Summary: Multi-Camera Trajectory Forecasting with Trajectory Tensors

Problem Definition
The paper addresses the challenge of Multi-Camera Trajectory Forecasting (MCTF), a task defined as predicting the future trajectory of a moving object across a network of non-overlapping or partially overlapping cameras. While existing literature focuses heavily on Single-Camera Trajectory Forecasting (SCTF), the authors identify critical limitations in the single-camera paradigm:

Limited Field-of-View: SCTF is restricted to short-term predictions (typically 1–5 seconds) because objects quickly exit the camera's view.
Inability to Anticipate Entry: Single-camera models cannot predict when or where a new object will enter the scene from an adjacent camera.
Computational Inefficiency: Monitoring large camera networks with full-resolution video for detection and tracking is computationally expensive, often leading to missed detections if resolution or frame rates are reduced.

The MCTF problem is formulated as a proactive system that, given an object's tracklet (2 seconds of history) in one or more views, predicts its future location across the entire network for up to 12 seconds. This is broken down into three hierarchical tasks:

Which: In which camera(s) will the object appear next?
When: At which timesteps will the object appear in those cameras?
Where: Precisely where (spatial coordinates) will the object appear within those camera views?

Methodology
The authors propose a deep encoder-decoder framework centered on a novel data representation called Trajectory Tensors.

Trajectory Tensors: Unlike traditional coordinate-based representations (sequences of $(x, y)$ coordinates) which struggle with occlusions, multi-view generalization, and uncertainty modeling, trajectory tensors encode object presence as heatmaps.
- Each camera view is divided into a $w \times h$ grid.
- Object locations are represented as binary values (presence/absence) in these grid cells, smoothed via a Gaussian kernel.
- Heatmaps from $k$ cameras over $t$ timesteps are stacked to form a tensor of dimension $k \times t \times w \times h$ .
- This representation naturally handles "null trajectories" (when an object is not visible in a specific camera) and models multi-modal futures (uncertainty) within a single structure.
Model Architectures: The paper evaluates several encoder-decoder models:
- Coordinate-Based Baselines: Adapted SCTF models (GRU, LSTM, 1D-CNN) applied to coordinate data. These require separate models for each camera view.
- Trajectory Tensor Models: Unified models processing the tensor data directly. These include:
  - 3D-CNN: Uses 3D convolutions for simultaneous spatio-temporal feature extraction.
  - 2D-1D-CNN: Separates spatial (2D) and temporal (1D) feature extraction to reduce parameters.
  - CNN-GRU: A hybrid approach using a CNN auto-encoder to compress spatial features into a vector, which is then processed by a GRU for temporal prediction.
Evaluation Metrics: Standard displacement errors (ADE/FDE) are adapted for the multi-camera setting. The authors introduce Average Precision (AP) for all tasks and propose Soft-Intersection-over-Union (SIOU) metrics for the "When" and "Where" tasks to better handle the probabilistic nature of the predictions.

Key Contributions

Problem Formulation: The introduction of the MCTF framework, explicitly defining the "Which-When-Where" hierarchy for predicting object trajectories across a camera network.
Trajectory Tensors: A new data representation technique that encodes multi-camera trajectories and their associated uncertainties, overcoming the limitations of coordinate-based methods regarding occlusion and multi-view generalization.
Unified Modeling: The demonstration that a single unified model can process data from all cameras simultaneously, contrasting with the multi-model approach required by coordinate-based methods.
Dataset: The release of the Warwick-NTU Multi-camera Forecasting (WNMF) dataset, collected from 15 cameras on the Nanyang Technological University campus, containing 2.0K cleaned cross-camera matches with multi-viewpoint departures.

Results
Experiments were conducted on the WNMF dataset using 5-fold cross-validation.

Performance: Trajectory tensor-based models consistently outperformed both the coordinate-based baselines and hand-crafted feature baselines across the "Which," "When," and "Where" tasks.
Best Model: The 3D-CNN architecture generally achieved the highest performance, particularly in the "Which" and "Where" tasks.
Multi-View Advantage: Ablation studies confirmed that utilizing trajectory tensors with inputs from multiple camera views significantly improves performance compared to single-view inputs.
Multi-Target Extension: The framework was extended to handle multiple targets simultaneously by stacking trajectories along the batch dimension, showing that the unified tensor approach remains effective even with multiple objects.

Significance and Claims
The paper claims that the MCTF framework and trajectory tensor representation offer a significant advancement over existing SCTF methods for surveillance applications. The authors assert that their approach:

Enables long-term forecasting by removing the single-camera viewpoint constraint.
Facilitates intelligent camera monitoring, allowing systems to focus resources on specific cameras where objects are predicted to appear, rather than monitoring all feeds continuously.
Provides robustness against camera failures by predicting object locations across redundant views.
Offers a more intuitive and elegant way to model uncertainty and multi-modal futures compared to coordinate-based approaches.

The authors conclude that while their models show promising results on the WNMF dataset, the field requires more datasets to fully evaluate the approach in diverse settings. They envision their work complementing existing person re-identification and multi-camera tracking systems.

More like this