PRAGMA: Revolut Foundation Model

The paper introduces PRAGMA, a family of Transformer-based foundation models pre-trained on large-scale, heterogeneous banking event sequences via masked modeling, which achieve superior performance across diverse financial tasks like credit scoring and fraud detection by learning general-purpose representations from raw data.

The Gist

Imagine a massive, chaotic library where every book is a person's entire financial life. These books aren't written in neat paragraphs; they are a jumbled mix of receipts, calendar notes, text messages, bank balances, and shopping lists, all written in different languages and formats. Some pages are missing, some are written in code, and the order changes every day.

For years, banks tried to understand these books by hiring a team of specialists for every single question. One team read the books to guess if you'd pay back a loan (Credit Scoring). Another team read them to spot a thief (Fraud Detection). A third team tried to guess what you'd buy next (Product Recommendations). Each team had to manually re-read the same messy pages, highlight different parts, and build their own unique filing system. It was slow, expensive, and they couldn't easily share what they learned with each other.

Enter PRAGMA: The "Financial Super-Reader."

The Revolut team built a new kind of AI called PRAGMA. Think of it not as a specialist, but as a genius librarian who has read every financial book in the world.

Here is how it works, broken down into simple concepts:

1. The Translator (Tokenization)

The problem with financial data is that it's messy. A bank transfer isn't just a sentence; it's a mix of numbers, dates, codes, and text.

• The Old Way: Imagine trying to read a receipt by turning every number into a string of letters (e.g., "$100" becomes "dollar, one, zero, zero"). You lose the meaning of the number's size.
• The PRAGMA Way: PRAGMA has a special translator. It looks at a transaction and says, "Ah, this is a Key (what happened?), a Value (how much?), and a Time (when?)." It treats "Amount: $100" not as text, but as a structured concept. It understands that "Metal Plan" is a specific status, not just random words. This allows the AI to "read" the financial language fluently without getting confused by the formatting.

2. The Two-Brain System (Architecture)

PRAGMA is designed with two distinct "brains" that work together:

• Brain A (The Event Encoder): This brain reads the story. It looks at the sequence of events: "You bought coffee, then you transferred money, then you checked your balance." It understands the flow and timing of your life.
• Brain B (The Profile Encoder): This brain reads the context. It looks at static facts that don't change often: "You are 25, you live in the UK, and you have a premium metal card."
• The History Encoder: This is the conductor. It takes the story from Brain A and the context from Brain B and blends them together to create a single, deep understanding of who you are at this specific moment.

3. The Training Method (Masked Modelling)

How do you teach a computer to understand money without giving it the answers?
Imagine a game of "Financial Mad Libs."

• The AI is shown a user's history, but it has to guess the missing pieces.
• "The user transferred money to [BLANK] on [BLANK] for [BLANK]."
• The AI has to use the surrounding clues to fill in the blanks.
• By playing this game billions of times on billions of fake (but realistic) financial histories, the AI learns the hidden patterns of human behavior. It learns that people who buy a lot of coffee on weekends often have a certain spending pattern, or that a sudden drop in balance followed by a specific app click might signal fraud.

4. The "One-Size-Fits-All" Magic (Downstream Tasks)

Once PRAGMA has learned the language of money, it becomes a universal adapter.

• The Old Way: To build a fraud detector, you had to build a new model from scratch.
• The PRAGMA Way: You take the "Super-Reader," freeze its brain, and just attach a tiny, simple "decision head" on top.
  • Need to predict Credit Score? Attach a "Loan" head.
  • Need to predict Fraud? Attach a "Thief" head.
  • Need to guess Lifetime Value? Attach a "Profit" head.

Because the heavy lifting (understanding the data) was already done during training, these new heads learn incredibly fast and perform better than the old specialist teams.

5. The Results: Why It Matters

The paper tested PRAGMA on six different banking tasks. The results were like finding a Swiss Army Knife that works better than six separate tools:

• Credit Scoring: It became 130% better at spotting risky borrowers than the old methods.
• Fraud Detection: It caught 65% more fraudsters while missing fewer legitimate transactions.
• Efficiency: Instead of training six different massive models, the bank now trains one giant model and uses it for everything.

The One Weakness

The paper admits one limitation: Anti-Money Laundering (AML).
PRAGMA is great at reading one person's story. But money laundering often involves a network of people passing money back and forth. It's like trying to solve a conspiracy by reading only one suspect's diary; you miss the connections between the suspects. For this specific task, PRAGMA currently struggles because it doesn't "see" the whole network, just the individual.

The Big Picture

PRAGMA is a shift from "hiring a specialist for every job" to "hiring one genius who can do them all." It takes the messy, chaotic reality of how we use money and turns it into a clear, understandable signal. It proves that even in the rigid world of finance, a foundation model can learn the "grammar" of human behavior and apply it to solve almost any problem.

Technical Summary

1. Problem Statement

Modern financial institutions generate vast, heterogeneous data streams comprising transaction logs, app navigation, communications, and static user profiles. While foundation models have revolutionized NLP and Computer Vision, applying them to multi-source banking user histories remains underexplored due to three specific challenges:

• Heterogeneity: Banking records are variable-length, mixed-type (categorical, numerical, free-text) structures, unlike uniform text tokens.
• Irregularity: Histories are long-tailed in length and irregular in time, with strong daily/weekly cycles, making standard text serialization inefficient (inflating sequence lengths and losing numerical magnitude).
• Contextual Constraints: Practical deployment requires handling static profile states (e.g., account tenure, plan type) alongside dynamic events, while adhering to strict privacy and regulatory constraints.

Current solutions often rely on task-specific pipelines with extensive manual feature engineering or single-source models that fail to generalize across diverse banking tasks (e.g., credit scoring vs. fraud detection).

2. Methodology

The authors propose PRAGMA, a family of encoder-style foundation models designed specifically for multi-source banking event sequences.

A. Data Representation & Tokenization

Instead of treating data as raw text, PRAGMA uses a Key–Value–Time tokenization scheme:

• Semantic Types (Keys): Encoded as single tokens (e.g., Amount, Currency).
• Values:
  • Numerical: Mapped to learned percentile buckets to preserve magnitude and ordering.
  • Categorical: Mapped to single tokens (e.g., MCC codes).
  • Textual: Tokenized via BPE subword tokenizers.
• Temporal Coordinates: Encoded in two ways:
  1. Log-seconds: Time elapsed since the last event (using soft logarithmic transformation to handle long-tail gaps).
  2. Calendar Features: Cyclical components (hour, day of week, month) embedded via periodic functions.
• Profile State: Static attributes (e.g., Plan: metal) are treated as a special "event" with a timestamp, augmented with "lifelong events" (e.g., first_topup) to preserve historical milestones.

B. Model Architecture

PRAGMA is an encoder-only, bidirectional Transformer with a three-branch design:

1. Profile State Encoder: Processes static user attributes and lifelong event timestamps.
2. Event Encoder: Independently encodes each event in the history, adding calendar features.
3. History Encoder: Fuses the outputs of the Profile and Event encoders. It uses Rotary Positional Embeddings (RoPE) to encode the temporal distance to the last event.

• Scale: The model family includes three variants: PRAGMA-S (10M), PRAGMA-M (100M), and PRAGMA-L (1B) parameters.

C. Pre-training Strategy

• Objective: Masked Language Modeling (MLM). The model reconstructs masked tokens, events, or semantic keys.
• Masking Strategy: A combination of token-level (15%), event-level (10%), and key-level (10%) masking. It also uses [UNK] tokens as a form of input dropout to prevent over-reliance on the [MASK] token.
• Efficiency: To handle long, variable-length histories, the authors employ sequence packing (using variable-length attention kernels) and dynamic batching (grouping users by event count to minimize padding).

D. Downstream Adaptation

PRAGMA supports two adaptation modes:

1. Embedding Probe: Freezing the backbone and training a lightweight linear head on extracted embeddings.
2. LoRA Fine-tuning: Updating only 2–4% of parameters (attention and MLP projections) via Low-Rank Adaptation, allowing fast specialization without catastrophic forgetting.

3. Key Contributions

1. First Multi-Source Banking Foundation Model: PRAGMA is the largest published encoder backbone (up to 1B parameters) for consumer banking event sequences, capable of handling heterogeneous, multi-source data with explicit profile state.
2. Novel Tokenization & Architecture: Introduces a key-value-time tokenization scheme and a dual-branch encoder architecture that effectively separates static profile context from dynamic event sequences.
3. Efficient Training Recipe: Demonstrates that sequence packing and dynamic batching enable efficient training on 207 billion tokens (24 billion events) from 26 million users.
4. Validation of Transfer Learning: Proves that a single pre-trained backbone can outperform multiple task-specific models across diverse domains, reducing the need for hand-crafted features.

4. Results

The model was evaluated on six downstream tasks: Credit Scoring, Fraud Detection, Communication Engagement, Product Recommendation, Recurrent Transaction Detection, and Lifetime Value (LTV) prediction.

• Superior Performance: PRAGMA consistently outperformed strong task-specific baselines.
  • Credit Scoring: +130.2% relative improvement in PR-AUC.
  • Communication Engagement: +79.4% relative improvement in PR-AUC.
  • External Fraud: +64.7% relative improvement in Recall.
• Scaling Laws: Performance gains were highly task-dependent. The 1B parameter model showed massive gains in complex tasks like Credit Scoring (+35.2% PR-AUC over the 10M model), while smaller models (10M) were already competitive for LTV and Recurrent Transactions.
• Adaptation Efficiency: LoRA fine-tuning consistently matched or exceeded the performance of training models from scratch, often achieving results with only 1/8th of the wall-clock training time.
• Ablation Studies:
  • Profile State: Removing the profile encoder significantly hurt Credit Scoring (+31.8% drop in PR-AUC) but had minimal impact on LTV, confirming the value of static signals for risk tasks.
  • Text Encoders: Integrating a pre-trained text encoder (Nemotron) improved text-heavy tasks (Credit Scoring) but added latency and offered no benefit for text-sparse tasks.
  • Limitations: The model underperformed on Anti-Money Laundering (AML) (−47.1% F0.5), as AML requires cross-record relational analysis (network effects) which the isolated record-level PRAGMA cannot capture.

5. Significance

PRAGMA represents a paradigm shift in financial AI, moving away from fragmented, task-specific feature engineering toward general-purpose representation learning.

• Operational Impact: It allows banks to consolidate multiple high-maintenance models into a single shared system, reducing engineering overhead and improving consistency.
• Data Efficiency: By leveraging self-supervised pre-training on raw event streams, it extracts signals that are difficult to capture with traditional supervised learning, particularly for rare events (e.g., fraud, default).
• Future Direction: The paper highlights that while foundation models excel at individual record analysis, the next frontier for financial AI lies in extending these architectures to capture cross-record relational structures for tasks like AML.

In summary, PRAGMA demonstrates that heterogeneous financial event sequences, despite their irregularity and complexity, admit transferable representations similar to text and vision, offering a scalable foundation for next-generation financial applications.