TransactionGPT

1. Problem Definition

The paper addresses the challenge of building foundation models for consumer payment transaction data, which is characterized as Multi-Modal-Temporal-Tabular (MMTT). Unlike standard time series or natural language data, transaction data presents unique complexities:

• Heterogeneity: Each transaction contains diverse fields: numerical metadata (amount, time), high-cardinality categorical entities (merchant IDs, MCCs), and task-specific downstream features (hundreds of numerical/categorical values).
• Structural Mismatch: Existing models fail to handle this specific structure:
  • LLMs are computationally inefficient and semantically mismatched for numerical/tabular data.
  • Tabular Foundation Models typically handle small-scale data or require semantically rich column names, lacking the temporal sequence modeling needed for payment trajectories.
  • Time Series Models assume scalar values per timestep, whereas transactions contain high-dimensional, multi-modal vectors.
• Goal: To develop a foundation model capable of understanding and generating consumer transaction trajectories while supporting diverse downstream tasks (anomaly detection, classification, representation learning) with strict efficiency and latency constraints.

2. Methodology: TransactionGPT Architecture

The authors propose TransactionGPT, a specialized Transformer-based foundation model trained on billions of real-world transactions. The core innovation is a 3D-Transformer architecture that hierarchically encodes three distinct data modalities: Metadata, Features, and Temporal sequences.

A. Model Evolution (1D $\to$ 3D)

The architecture evolved through three stages to address specific data challenges:

1. TGPT-1D: Uses a single Temporal Transformer ($TF(S)$) to model sequence dependencies, with a simple MLP for intra-transaction field interactions.
2. TGPT-2D: Introduces a second Transformer ($TF(tr)$) to encode interactions between metadata fields within a single transaction, replacing the MLP.
3. TGPT-3D: Adds a third Transformer ($TF(F)$) dedicated to downstream features. This is crucial because:
   • Feature sets ($F$) are much larger than metadata ($M$).
   • Metadata often requires large embeddings (high cardinality), while features require small embeddings. A shared embedding size creates a trade-off between overfitting and under-representation.

B. Key Innovation: Virtual Token Layer (VTL) & Modality Fusion

The primary challenge in the 3D architecture is fusing embeddings of vastly different sizes (high-dimensional metadata vs. low-dimensional features) without creating a computational bottleneck. The authors introduce a Virtual Token Layer (VTL):

• Mechanism: VTL transforms a set of embeddings into an arbitrary number of "virtual tokens" using a dual-channel design:
  • Linear Path: A softmax-based linear combination to select and aggregate informative embeddings (preserving information flow).
  • Nonlinear Path: An MLP to rescale embeddings to desired dimensions.
• Application:
  1. Feature Tokenization: Compresses the large output of the Feature Transformer into $v_f$ virtual feature tokens matching the metadata embedding dimension.
  2. Transaction Tokenization: Fuses metadata and virtual feature tokens, then compresses them into $v_t$ virtual transaction tokens.
• Benefit: This allows the Temporal Transformer to process sequences of virtual tokens rather than raw high-dimensional vectors, significantly reducing computational complexity ($O(v_t w \cdot (d_{tr}/v_t)^2)$) while maintaining information bandwidth.

C. Training Objectives

TGPT is trained using Joint Optimization:

• Self-Supervised (Generative): Predicts future transaction properties (time gap, amount, merchant, MCC) based on historical sequences.
• Supervised (Predictive): Fine-tuned for downstream classification tasks (e.g., anomaly detection).
• Loss Function: Combines MSE for numerical fields, Cross-Entropy for categorical fields, and regularization terms (weight decay, VTL orthogonality).

3. Key Contributions

1. TransactionGPT Model: The first foundation model specifically designed for MMTT payment data, trained on a billion-scale dataset from a major payment network.
2. 3D-Transformer Architecture: A novel design that decouples the encoding of metadata, features, and temporal sequences, allowing optimal embedding sizes for each modality.
3. Virtual Token Mechanism: A scalable fusion technique that resolves the dimensionality mismatch between modalities, enabling efficient multi-modal integration without information bottlenecks.
4. Empirical Validation: Comprehensive evaluation across three distinct tasks (anomaly detection, trajectory generation, and merchant category prediction) demonstrating superiority over production baselines and LLMs.

4. Experimental Results

The model was evaluated on three datasets: T-JGC (Anomaly Detection), T-RES (Dining Trajectory), and T-MCC (MCC Prediction).

• Anomaly Detection (T-JGC):
  • TGPT-3D-FMT (the full 3D model with VTL) achieved a 22% relative improvement over the existing production model on a critical business metric.
  • It significantly outperformed 1D and 2D variants, proving that proper feature fusion is critical for classification.
• Transaction Generation (T-RES):
  • TGPT-2D achieved 45.6% Recall@50 for predicting the next restaurant in a trajectory (out of 500K options), outperforming the SASRec baseline (38.5%).
  • Qualitative analysis (UMAP) showed that TGPT embeddings successfully captured geographic proximity and merchant similarities (e.g., clustering by city or airport).
• MCC Prediction vs. LLMs (T-MCC):
  • TGPT-2D outperformed fine-tuned LLMs (e.g., Llama2-7B) with 92% fewer parameters (56M vs. 7B).
  • Inference Speed: TGPT was 300x faster (0.27ms vs. 84.9ms) on a single A100 GPU, demonstrating superior cost-efficiency for tabular tasks.
• Scalability: The model adheres to data scaling laws; performance improved consistently with larger training datasets, with the 3D architecture showing the most significant gains at scale.

5. Significance and Impact

• Bridging the Gap: This work establishes a new paradigm for foundation models in the financial sector, moving beyond generic LLMs or simple sequential models to handle the specific structural complexity of payment data.
• Practical Deployment: The architecture is designed for production viability, meeting strict latency (SLA) and memory constraints while delivering state-of-the-art accuracy.
• Efficiency over LLMs: The paper demonstrates that for structured, multi-modal tabular data, specialized architectures (TGPT) are far more efficient and effective than adapting large language models, challenging the "bigger is better" trend in this specific domain.
• Open Guidelines: The authors provide practical guidelines on normalization, positional encoding, and information bandwidth management, offering a roadmap for practitioners building similar foundation models.

In conclusion, TransactionGPT represents a significant advancement in financial AI, proving that tailored foundation models can effectively learn complex consumer behaviors from massive, heterogeneous transaction datasets, leading to tangible improvements in fraud detection and user experience.