Technical Summary: UPLOTS – A Unified Pretrained Language Model for Constrained Time-series Generation

1. Problem Statement

Current time-series generation (TSG) methods suffer from significant fragmentation. Existing approaches, including Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), and Diffusion models, typically adhere to a "one model per dataset" paradigm. This necessitates training, tuning, and deploying distinct models for every specific dataset or task. This fragmentation leads to:

• Poor Scalability: Linear or superlinear increases in computational and human effort as the number of diverse datasets grows.
• Limited Generalization: Models fail to leverage shared temporal structures across different domains.
• Inefficient Knowledge Assimilation: The inability to internalize extensive knowledge from varied datasets restricts performance in constrained generation tasks.

While recent studies have explored prompt-driven or prototype-driven transfer, they often rely on domain-specific backbones or exemplar series. There remains a gap for a single, unified generator capable of constraint-aware synthesis across unseen domains without retraining.

2. Methodology: The UPLOTS Framework

The authors propose UPLOTS (Unified Pretrained Language Model for constrained Time-Series Generation), a framework that replaces task-specific models with a single pre-trained Large Language Model (LLM) backbone guided by learned constraint prompts.

2.1 Core Architecture

UPLOTS integrates a pre-trained LLM (specifically GPT-2 or LLaMA 3.2-1B) with a time-series generation pipeline. The framework consists of three primary components:

1. Time-series Prompt Embedding Module (TPEM):

   • Function: This module serves as the bridge between natural language constraints and time-series data. It encodes dataset-specific constraints (e.g., "Morning Peak Patterns") as lightweight natural language prompts.
   • Mechanism: Time-series inputs are projected into the model's hidden dimension via a Convolutional MLP. Simultaneously, textual prompts are tokenized and embedded using the LLM's frozen word-embedding layer.
   • Fusion: The prompt embeddings and time-series embeddings are concatenated along the temporal dimension and fed into a truncated pre-trained Transformer architecture. The output is mapped back to the original feature space via LayerNorm (LN) and linear projection layers.
   • Adaptability: Only the LayerNorm and positional embeddings are trainable; the core LLM parameters remain frozen, ensuring parameter efficiency.

2. Dynamic Weighted Training Strategy (DWTS):
   To enable effective joint learning across heterogeneous datasets, UPLOTS employs a two-pronged strategy:

   • Curriculum-Aware Loss Modulator (CALM): During the warm-up phase, this mechanism down-weights "harder" datasets (those with higher recent loss) to prevent early training instability. It computes running means of losses for each dataset and assigns static curriculum weights accordingly.
   • Rolling-Loss-Based Dynamic Sampling (RLDS): This component dynamically adjusts batch sampling probabilities based on the recent loss of each dataset. Batches from underperforming (higher-loss) datasets are drawn more frequently, ensuring the model focuses on difficult domains during training.

3. Generation Backbone:
   The framework utilizes a Diffusion-TS model as the underlying generative engine. The output of the TPEM-enhanced LLM serves as the input for the diffusion process, which proceeds via forward noising and reverse denoising steps. The training objective minimizes a joint time/frequency loss to preserve both pointwise accuracy and spectral content.

2.2 Workflow

• Training: Heterogeneous time-series datasets are jointly trained within the single LLM. Dataset-specific constraints are encoded as prompts, allowing the model to learn domain-invariant patterns and dataset-specific dynamics simultaneously.
• Inference: No retraining is required. To generate a specific target sequence (e.g., "Evening Peak" for an Energy dataset), the user simply provides the corresponding natural language prompt. The model conditions on this prompt to synthesize the desired time-series segment on demand.

3. Key Contributions

The paper outlines three primary contributions:

1. Unified LLM Framework: A novel architecture that breaks the traditional one-to-one paradigm by enabling a single pre-trained model to train across all datasets and infer on test sets with diverse objectives.
2. Prompt-Driven On-Demand Generation: The framework leverages prompt learning to enable task-specific generation without retraining. For instance, the model can generate morning peak segments for an Energy dataset using textual prompts, reusing diverse prompts to generate different targets.
3. Empirical Superiority and Generalizability: The framework demonstrates significant advantages over state-of-the-art (SOTA) baselines and basic model variants across four real-world datasets and 14 prompt configurations spanning four constraint families (temporal peaks, calendar, load level, and volatility).

4. Experimental Results

The authors evaluated UPLOTS on four real-world benchmarks: ETTh (Electricity Transformer Temperature), Energy, PEMS04, and PEMS08.

4.1 Evaluation Metrics

Performance was measured using:

• Context-FID (C-FID): Assesses distributional fidelity and temporal feature dependencies.
• Discriminative Score: Evaluates the ability of a classifier to distinguish real from generated data.
• Predictive Score: Measures downstream utility via the Train-on-Synthetic, Test-on-Real (TSTR) paradigm.

4.2 Main Findings

• Performance vs. Baselines: UPLOTS variants (using GPT-2 and LLaMA 3.2) significantly outperformed single-dataset models (TimeGAN, TimeVAE, DiffWave, DiffTime) and multi-dataset baselines (TimeDP).
  • On the ETTh and Energy datasets, UPLOTS improved Context-FID by an average of 12.3× and Discriminative Score by 9.3× compared to a prompt-free "Empty" baseline.
  • The full UPLOTS model achieved the best results across almost all metrics and datasets, with the LLaMA 3.2 backbone yielding the highest performance.
• Constraint Compliance: The model successfully generalized beyond the specific "morning/evening peak" settings used in initial training. It maintained high fidelity across 14 different prompt configurations, including calendar constraints (weekday/weekend), load levels (high/low), and volatility patterns.
• Ablation Studies:
  • Removing the TPEM module caused a drastic performance drop (e.g., C-FID increased from 0.0243 to 0.1682 on ETTh_MP), confirming the necessity of the prompt embedding mechanism.
  • Removing CALM and RLDS resulted in modest but consistent performance decreases, validating their role in balancing multi-dataset training.
• Downstream Utility: In data augmentation experiments for forecasting tasks (using PhaseFormer and SparseTSF), UPLOTS-generated data consistently improved forecasting accuracy, particularly in low-data regimes (1% to 10% real data ratios).

4.3 Hyperparameter Sensitivity

• Hidden Dimension: Performance peaked at a hidden dimension of 64; smaller dimensions (16) led to underfitting, while larger dimensions offered no further benefit.
• Prompt Length: The model was largely insensitive to prompt length (ranging from single characters to full sentences), though semantic richness was found to be more critical than length.

5. Significance and Claims

The paper positions UPLOTS as a scalable, adaptive solution for real-world time-series applications. Its significance lies in:

• Paradigm Shift: Moving from fragmented, dataset-specific models to a unified, pre-trained backbone.
• Efficiency: Achieving cross-domain generalization and precise constraint control without the computational overhead of training separate models or the high latency of pure diffusion pipelines.
• Flexibility: Enabling "on-demand" generation of customized time-series segments via natural language prompts, making the system highly adaptable to changing requirements.

The authors conclude that UPLOTS successfully internalizes diverse temporal structures and offers a robust framework for constrained time-series generation, with future work planned to explore broader temporal domains and zero-shot distribution shifts.