A Unified Framework for Modeling Heterogeneous Financial Data via Dual-Granularity Prompting

1. Problem Statement

Industrial credit risk assessment faces significant challenges due to the nature of financial data and the limitations of current modeling approaches:

• Data Heterogeneity & Complexity: User data is multi-source, noisy, high-dimensional, sparse, and discrete (e.g., credit reports, transaction logs, product usage). It contains substantial missing values.
• Dynamic Evolution: Creditworthiness is not static; it evolves over time. Traditional models struggle to capture both short-term behavioral shifts and long-term profile evolution.
• Limitations of Current State-of-the-Art:
  • XGBoost (Industry Standard): While stable and interpretable, it relies heavily on manual feature engineering, fails to capture temporal dependencies in sequential data, and cannot model risk across multiple future time horizons effectively.
  • Deep Learning: Existing deep learning models (RNNs, Transformers) often struggle with the sparsity and heterogeneity of financial data. Most focus on point-in-time predictions rather than multi-scale forecasting.
  • LLMs: Large Language Models (e.g., GPT-4, DeepSeek) show promise but fail to outperform specialized models in zero-shot settings for specific credit tasks and lack the efficiency for real-time industrial deployment.

2. Methodology: FinLangNet

The authors propose FinLangNet, a novel framework that reformulates credit scoring as a multi-scale sequential learning problem. It utilizes a dual-module architecture to process heterogeneous data:

A. Non-Sequential Module (Static Features)

• Goal: Capture complex static interactions in user profiles (e.g., age, income, occupation).
• Architecture: Based on DeepFM.
  • Factorization Machine (FM) Component: Models low-order (second-order) feature interactions.
  • Deep Neural Network (DNN) Component: Models high-order non-linear correlations via an MLP.
• Output: A static interaction embedding ($O_m$) representing the baseline risk level.

B. Sequential Module (Sequence Representation Generator - SRG)

• Goal: Model temporal dynamics from multi-source, sparse, and noisy financial sequences.
• Discretization: Continuous financial signals are transformed into discrete tokens via a tokenizer to mitigate sparsity and noise.
• Dual-Prompt Mechanism (Key Innovation): Inspired by the [CLS] token in Transformers, this mechanism introduces learnable prompts at two granularities to guide representation learning:
  1. Feature-Level Prompt ($e_{\phi_c}$): Prepend to each channel sequence to capture channel-specific global patterns and fine-grained temporal nuances.
  2. User-Level Prompt ($P_s$): Prepend to the aggregated sequence (formed by weighted attention across all channels) to capture holistic user risk profiles.
• Architecture: The augmented sequences are processed by a Transformer encoder. The final state of the user-level prompt serves as the comprehensive sequential representation ($O_{ns}$).

C. Multi-Scale Prediction & Optimization

• Fusion: Static ($O_m$) and sequential ($O_{ns}$) embeddings are concatenated into a unified representation.
• Multi-Task Setup: The model predicts risk probabilities across $L$ different future time horizons (e.g., 30, 60, 90 days) simultaneously.
• Loss Function: A Dynamically Weighted Hybrid Loss is used to handle class imbalance and varying sample difficulty:
  • Weighted Logarithmic Loss (WLL): Penalizes minority classes (defaults) more heavily.
  • Dynamic Hard Example Mining: Up-weights samples where the model has high prediction uncertainty (based on gradient norms).

3. Key Contributions

1. Reformulation of Credit Scoring: Shifts the paradigm from static binary classification to dynamic multi-scale forecasting, enabling the generation of future behavioral distributions across different time horizons.
2. Dual-Granularity Prompting: Introduces a novel mechanism within a Transformer-based sequential module that operates at both feature-level (fine-grained) and user-level (holistic) granularities, effectively addressing data heterogeneity.
3. Hybrid Architecture: Combines DeepFM for static tabular data with a specialized Sequential Representation Generator (SRG) for temporal data, creating a unified framework for heterogeneous inputs.
4. Industrial Deployment: Successfully deployed in a real-world financial platform, bridging the gap between deep learning performance and industrial interpretability requirements.

4. Experimental Results

A. Industrial Credit Risk Assessment (Real-World Data)

• Dataset: 7 million+ users from a real-world financial platform.
• Baselines: Compared against XGBoost (production baseline), deep sequential models (Transformer, Mamba, TimesNet), and LLMs (DeepSeek-V3.2, GPT-4.1).
• Performance:
  • KS Statistic: Achieved a 6.3 percentage point (pp) absolute improvement over the XGBoost baseline.
  • Bad Debt Rate: Reduced the bad debt rate by 9.9% relative to the baseline.
  • Multi-Horizon: Consistently outperformed baselines across all 6 time-horizon tasks ($y_1$ to $y_6$). Notably, the fusion of FinLangNet + XGBoost achieved the best overall results, though FinLangNet alone outperformed XGBoost significantly on long-term predictions where XGBoost failed.

B. Generalization (UEA Time Series Benchmarks)

• Datasets: 5 multivariate time series classification tasks from the UEA archive (medical, motion, etc.).
• Results: The SRG module alone achieved state-of-the-art or competitive performance, outperforming specialized time-series models (e.g., TStamp, SVP-T, ShapeNet) and demonstrating the framework's versatility beyond finance.

C. Ablation Studies

• Dual-Prompt Necessity: Removing either the feature-level or user-level prompt significantly degraded performance. The combination yielded the highest KS (34.08%).
• Sequential vs. Static: Removing the sequential module caused a catastrophic drop in KS (>20%), proving that temporal dynamics are the dominant predictor of credit risk.
• Robustness: The model maintained high stability even when input sequence lengths were truncated to 25% of the original, indicating strong capability in handling cold-start scenarios.

5. Significance and Impact

• Industrial Viability: The paper demonstrates that deep learning can outperform the industry-standard XGBoost in credit risk without sacrificing scalability. The system runs on GPU clusters with sub-100ms latency, processing 100 QPS.
• Regulatory Compliance: To address the "black-box" nature of deep learning, the authors deployed a fusion strategy where FinLangNet generates a scalar "language-risk subscore" that acts as a dense feature for an interpretable XGBoost model. This satisfies regulatory requirements while leveraging deep learning's predictive power.
• Financial Impact: The deployment resulted in a 12.3% reduction in expected losses and a 6.3 pp gain in KS, validating the commercial value of the approach for large-scale financial institutions.
• Methodological Advancement: The "Dual-Granularity Prompting" mechanism offers a new blueprint for modeling heterogeneous sequential data in domains beyond finance, such as healthcare and IoT.