Technical Summary: SeqLLM

Problem Statement

Large-scale payment platforms, such as WeChat Pay, face a critical challenge in merchant risk control: screening tens of millions of merchants daily where errors are costly in both directions. False positives harm legitimate businesses, while false negatives allow illicit activities to persist. The most difficult cases cannot be resolved by analyzing textual profiles (e.g., merchant descriptions, complaints) or behavioral sequences (e.g., transaction histories) in isolation. These cases require joint reasoning over heterogeneous textual evidence and long temporal event streams.

While Large Language Models (LLMs) excel at textual reasoning, they cannot natively process behavioral sequences. Conversely, dedicated behavior models capture temporal patterns but lack language reasoning and explanation capabilities. Existing attempts to bridge this gap by injecting behavior tokens into LLMs often rely on Continual Pre-Training (CPT). However, CPT at the scale required for industrial applications (tens of millions of sequences) leads to catastrophic forgetting, severely degrading the model's general language and reasoning abilities. Current "adapt-then-repair" pipelines (using mixed-domain data and distillation) fail to fully recover these lost capabilities, especially as the behavioral corpus grows.

Methodology: SeqLLM

The authors propose SeqLLM, a framework designed to inject native behavioral-sequence modeling into a pretrained LLM while preserving its language competence. The framework consists of three core components:

1. Compact Discrete Behavior Vocabulary

Instead of serializing events into natural language (which creates long contexts and overloads word tokens), SeqLLM encodes each event as a sequence of field-level tokens.

• Structure: Each event is split into fields (e.g., time, amount, channel, status), and each field value is mapped to a dedicated token (e.g., <Amount:200-500 CNY>).
• Benefits: This approach compresses events (from dozens of tokens to ~9 on average), avoids collisions with the original word vocabulary, and supports compositional generalization (novel combinations of fields remain expressible).

2. Lightweight Behavior Projector with Two-Stage Alignment

To ensure the LLM understands these new tokens, a lightweight projector aligns them with the LLM's semantic space using a translation-then-reasoning curriculum:

• Text-Grounded Initialization: Behavior tokens are initialized using the mean-pooled embeddings of their readable text descriptions, rescaled to match the backbone's embedding distribution.
• Stage 1 (Translation): The model is trained to translate behavior tokens back into natural language descriptions, anchoring the tokens in the LLM's semantic space.
• Stage 2 (Reasoning): The model is trained on multi-event reasoning queries (e.g., identifying patterns across transactions) to learn how to interpret and reason over sequences of these tokens.
• Mechanism: A shared MLP projector with a residual connection maps all behavior tokens, imposing a consistent alignment constraint that allows the backbone to compose tokens across fields and events.

3. Prefix-Guided Capability Injection

To learn sequence modeling without eroding language ability, SeqLLM replaces standard CPT with Prefix-Guided Supervised Fine-Tuning (SFT).

• Mechanism: Instead of applying next-token loss across the entire behavior stream (which updates the backbone broadly), the model is trained on instruction-conditioned generation. A natural language instruction (task prefix) is provided, and the model predicts the future events (the suffix) based on the past events (the prefix).
• Loss Application: The loss is computed only on the response tokens (the predicted future events).
• Advantage: This confines the sequence learning to specific task contexts activated by the prefix, preventing the broad interference with pretrained language knowledge that causes catastrophic forgetting. Crucially, this method does not require a separate "capability recovery" stage or massive amounts of auxiliary language data.

Key Contributions

1. Formulation of Joint Text-Behavior Modeling: The paper defines a regime for high-stakes decisions where risk emerges only from the interaction between textual evidence and long event streams, a scenario systematically missed by unimodal pipelines.
2. The SeqLLM Framework: A general framework comprising a compact field-level vocabulary, a text-grounded residual projector with a translation-to-reasoning curriculum, and prefix-guided capability injection.
3. Prefix-Guided SFT as a Scalable Alternative: The authors establish that prefix-guided SFT can inject sequence modeling at scale while preserving language ability, eliminating the need for the "adapt-then-repair" pipeline used in prior work.
4. Industrial and Public Scale Impact: The framework is fully deployed in production and validated on public benchmarks, demonstrating superior performance and efficiency compared to state-of-the-art baselines.

Results

Industrial Deployment at WeChat Pay

• Merchant Risk Screening: SeqLLM is deployed in a 0.6B–8B cascade. In a three-month shadow evaluation, it improved screening precision from 92.0% to 97.5% compared to a DeepSeek-based LLM baseline (text-only). Post-launch, the appeal rate dropped from 12% to ~2%, with zero exonerations observed.
• Fraud Detection: When used to initialize behavior embeddings in a downstream fraud detector, SeqLLM's pretrained embeddings improved Precision@Top-0.01% by 26.8 percentage points, Precision@Top-0.1% by 7.6 percentage points, Recall@Top-0.1% by 12.9 percentage points, and Recall@Top-1% by 33.1 percentage points, representing the largest improvement in the model's history.
• Language Retention: Unlike CPT, which dropped C-Eval scores from 0.78 to 0.27, SeqLLM maintained a C-Eval score of 0.783 while achieving comparable sequence modeling performance (HR@10 ~0.80).

Public Benchmarks

• Recommendation (MovieLens & Amazon): SeqLLM outperformed the User-LLM baseline by up to 32% relative Recall@5 while retaining significantly stronger language abilities (MMLU, C-Eval, AGIEval).
• RecIF Benchmark: SeqLLM+RL improved Pass@32 by 14.2% over the full OneRec-8B pipeline while using 4.8x fewer GPU-days and requiring no separate distillation stage.
• Semantic Understanding: On the constructed RecProbe suite, SeqLLM significantly outperformed OneRec-8B in video-topic matching and interest ranking, demonstrating superior semantic grounding of behavior tokens.

Significance

The paper claims that SeqLLM establishes a scalable route to unified language and behavioral-sequence models. By decoupling sequence learning from the backbone's general language updates via prefix-guided SFT, it solves the trade-off between acquiring new capabilities and retaining existing ones. This enables the deployment of LLMs in high-stakes, multimodal domains (like financial risk control) where reliability, explainability, and the integration of diverse data modalities are critical, without the prohibitive cost of catastrophic forgetting or complex recovery pipelines.