TrajTok: Adaptive Spatial Tokenization for Trajectory Representation Learning

Technical Summary: TrajTok: Adaptive Spatial Tokenization for Trajectory Representation Learning

Problem Statement
Learning generalizable representations from raw GPS trajectories is challenging due to the data's continuous, noisy, and irregularly sampled nature. Existing foundation models and geographic representation learning paradigms often rely on fixed-size grid discretization. This approach presents a fundamental tradeoff: fine-resolution grids create cells with insufficient GPS visits to learn robust embeddings, while coarse-resolution grids suffer from internal fragmentation, merging heterogeneous local movement patterns into single tokens. Furthermore, traditional methods struggle to simultaneously capture geometric route structures and kinematic motion dynamics without relying on rigid map-matching preprocessing.

Methodology
The authors propose TrajTok, a trajectory encoder framework that converts raw GPS traces into discrete tokens via adaptive spatial tokenization and learns representations through a factorized transformer architecture.

1. Adaptive Spatial Tokenization:
   Instead of fixed grids, TrajTok constructs a multi-resolution hexagonal vocabulary using the H3 system. The vocabulary is built hierarchically: cells in high-density regions are recursively split into finer resolutions until a capacity threshold is met or a maximum resolution is reached, while sparse regions remain at coarser resolutions. This allocates finer spatial units to dense areas while maintaining a compact vocabulary. Raw GPS points are mapped to the highest-resolution vocabulary cell containing them, producing a discrete token sequence with associated cell IDs, resolutions, coordinates, and timestamps.

2. Factorized Transformer Encoder:
   The architecture employs a dual-stream design to separate geometric and kinematic learning before fusing them:

   • Geometric Channel: Encodes spatial location via learned cell-identity vectors.
   • Kinematic Channel: Encodes motion features (normalized speed and heading) continuously, using sine/cosine transformations for heading to respect circular structure.
   • Spatiotemporal Positional Encoding: The geometric channel utilizes Spatiotemporal Rotary Position Embeddings (ST-RoPE) over latitude, longitude, and time. The kinematic channel uses temporal-only RoPE.
   • Fusion: Early layers process each stream independently via self-attention. Subsequent fusion layers apply cross-attention, allowing the geometric and kinematic streams to attend to one another, creating a unified trajectory representation.

3. Pretraining Objective:
   TrajTok is pretrained using a co-masked masked token modeling objective. Given a trajectory with shared masked positions across both channels:

   • The geometric channel predicts the masked cell IDs.
   • The kinematic channel reconstructs normalized speed and heading.
   • The loss is a linear combination of geometric prediction loss and kinematic regression loss.
   • The authors introduce run-aware span masking, which rejects candidate mask spans lying entirely within a single repeated-cell run to prevent the model from learning trivial shortcuts (copying endpoints).

Key Contributions

• Density-Adaptive Tokenization: A method to partition continuous trajectories into multi-resolution spatial tokens, allocating finer cells to dense regions while preserving a compact vocabulary, addressing the tradeoff inherent in fixed-grid approaches.
• Factorized Architecture: A transformer encoder that separates geometric (location) and kinematic (motion) signal processing in early layers, fusing them via cross-attention to capture both route geometry and motion dynamics.
• Pretraining Recipe: A simple yet effective masked token modeling objective tailored for raw GPS streams, incorporating co-masking and run-aware span masking to encourage the learning of transferable structure.
• Generalizability: Demonstration that a single frozen pretrained encoder, paired with lightweight task adapters, achieves state-of-the-art or competitive performance across diverse downstream tasks, including both geometry-dominated (similarity search, classification) and kinematics-dominated (ETA, travel-time regression) scenarios.

Experimental Results
Evaluated on the Porto taxi dataset, a single frozen TrajTok encoder with task-specific adapters achieved the following:

• Trajectory Similarity Search: Achieved an HR@1 of 0.435, outperforming strong baselines like NeuTraj (0.308) and SIMformer (0.300).
• Call-Type Classification: Achieved a macro-F1 of 0.773, matching or slightly exceeding specialized methods like START and JGRM, despite TrajTok operating solely on raw GPS without map-matching.
• Estimated Time of Arrival (ETA): Achieved an MAE of 42.27 seconds on prefix-based prediction, substantially outperforming prior neural travel-time systems (e.g., MetaER-TTE at 93.75s).
• Full-Trajectory Travel-Time Regression: Achieved an MAE of 38.41 seconds, significantly outperforming baselines like TIGR (86.86s).

Ablation studies confirmed that density-adaptive tokenization, cross-attention fusion, and the specific pretraining recipe are all critical drivers of performance. Notably, the authors observed that extending pretraining beyond a certain point (increasing steps from 60k to 120k) reduced validation loss but degraded downstream retrieval performance, suggesting that pretraining loss alone is not a sufficient proxy for transferability.

Significance and Claims
The paper claims that TrajTok demonstrates that learned multi-resolution spatial tokenization, combined with a simple masked-token pretraining objective, is a promising framework for general-purpose trajectory foundation models. The key finding is that a single frozen encoder can effectively transfer to both geometry-dominated tasks (where the "where" matters) and kinematics-dominated tasks (where the "how" matters). This indicates that the learned representation captures transferable trajectory structure rather than task-specific shortcuts. The authors position this as a step toward foundation models for trajectory intelligence that operate directly on raw GPS data, avoiding the preprocessing assumptions and computational costs of map-matching pipelines.

Limitations
The authors note that all experiments were conducted on the Porto taxi dataset. While they verified transfer across four downstream applications, they acknowledge the need for future work to test transfer learning to other datasets with minimal fine-tuning, scaling to multiple cities, and potentially enriching tokens with lightweight geographic priors without reverting to full map-matching.