Technical Summary: Life Sequence Transformer

Problem Statement

Longitudinal administrative records contain rich information on individuals' trajectories across labour markets, health, education, and other socio-economic domains. However, these data are rarely treated as sequential structures suitable for generative modelling. While Transformer-based generative models have successfully simulated dynamical systems in physics, chemistry, and health (e.g., predicting disease outcomes), their application to socio-economic life histories remains underexplored. Existing models like Life2Vec (Savcisens et al., 2024) excel at predictive tasks (e.g., mortality prediction) but lack generative capabilities to simulate future sequences. Other models, such as CAREER (Vafa et al., 2024) and TransformEHR (Yang et al., 2023), either lack fine-grained temporal resolution or require predefined event timings, preventing the simulation of flexible, counterfactual life paths where the model determines when and how long events occur.

Methodology

Data Source and Preprocessing

The model is trained on the Work History Italian Panel (WHIP), a 1:15 sample of individuals from the Italian social security administration (INPS). The dataset covers 1990–2015 and includes over 4 million individuals. The authors apply strict selection criteria to retain approximately 1.65 million life trajectories, ensuring complete information up to 2015, sufficient observation periods, and plausible labour force attachment.

Novel Encoding Scheme

A core contribution is a novel encoding method that transforms tabular administrative data into a structured sequence of socio-economic states. Unlike previous approaches that treat time as a fixed grid or auxiliary attribute, this method imposes an explicit calendar-like structure:

• Temporal Structure: The history is divided into yearly segments, further subdivided into 12 monthly intervals.
• Event Representation: Each labour event is encoded as a discrete unit containing:
  • Start Month: <MONTH_X> (1–12).
  • Event Type: <TYPE_X> (e.g., employment, unemployment).
  • Modifiers: Optional tokens for sector (<ATECO_X>), job title (<WRK_TITLE_X>), and intensity.
  • Duration: <DUR_X> (in months), allowing for variable event lengths within a year.
  • Income: Discretized into 100 quantiles.
• Sequence Markers: Special tokens denote the beginning of life (<BOL>), end of year (<EOY>), and end of life (<EOL>).
• Static Attributes: Demographic background (sex, birth year, region) is encoded as tokens at the sequence start.
• Positional Embeddings: The model utilizes absolute position (year), relative position (age), and standard token position embeddings.

Model Architecture

The authors propose a decoder-only Transformer trained autoregressively to predict the next token in the sequence. Key architectural modifications include:

1. Performer Attention: Replaces standard attention to reduce trainable parameters and improve computational efficiency via linear complexity.
2. Rotary Position Embeddings (RoPE): Adapted to work with Performer attention to better capture temporal patterns.
3. RMS Normalisation: Applied before attention and feed-forward layers to stabilize training.
4. Time Embeddings: Uses a modified Time2Vec approach. Crucially, the authors adapt the linear and periodic components of Time2Vec: they allocate more linear capacity to the "year" embedding (short 25-year range) and more periodic capacity to the "age" embedding (70-year range) to prevent the model from losing coherence when extrapolating beyond the training period.

The model consists of 10 decoder layers, 8 attention heads (6 local, 2 global), and approximately 10.1 million parameters.

Key Contributions

1. Novel Encoding: A method to represent overlapping socio-economic events as a sequence of concept tokens with flexible temporal resolution (start month and duration), enabling the modelling of life histories that extend beyond instantaneous diagnoses.
2. Generative Adaptation: Adapting generative modelling techniques to simulate plausible alternative life trajectories conditioned on past histories, specifically enabling the generation of counterfactual "what-if" scenarios.
3. Scalable Training: Demonstrating the feasibility of training a Transformer on 1.6 million individual histories from administrative data to reproduce realistic labour market patterns.

Empirical Results

Next-Token Prediction

The model achieves macro-averaged accuracy of 0.7039 and an F1 score of 0.713 on the test set when conditioned on 10 years of past history. Performance improves with longer historical context, consistent with the stability of life patterns over time.

Causal Benchmarks

The authors evaluate the model's ability to reproduce established causal relationships from labour economics:

• Unemployment Benefits (Mobility Allowance): The model captures the general positive relationship between age and benefit duration. However, it fails to reproduce the sharp discontinuity at the policy cutoff (age 40) due to data scarcity (<1% of sequences). In wider bandwidths (±144 months), the model's Average Treatment Effect (ATE) aligns closely with empirical estimates (difference < 1 month).
• Motherhood Penalty: The model successfully replicates the long-term career setbacks and reduced earnings for women after childbirth. It captures the key features of income dynamics, though it tends to understate the immediate short-run penalty while generating a more persistent effect in the medium-to-long run compared to empirical data.
• Retirement Timing: The model reproduces the effect of birth month on retirement age (January-born individuals retiring later than December-born). The accuracy of this effect is sensitive to the conditioning horizon; conditioning four years before retirement yields results closer to empirical benchmarks than conditioning one year prior.

Out-of-Sample Generation

When generating trajectories beyond the training period (post-2015), the model maintains structural coherence for approximately 7–8 years on average. While 5% of sequences remain structurally valid after 20 years, the probability of structural failure increases exponentially with the generation horizon.

Significance and Claims

The paper claims to demonstrate the feasibility of generative modelling for socio-economic trajectories. By training on large-scale administrative data, the model can generate coherent, synthetic life paths that reflect known causal relationships without the need for external control groups.

The authors position this work as a tool for policy-oriented research, particularly for counterfactual generation. The ability to simulate "what-if" scenarios (e.g., how a career path might differ without a specific event) offers a scalable framework for constructing counterfactuals. However, the authors remain modest, noting that the model smooths over rare, rule-driven discontinuities and that the definition of counterfactuals (e.g., treatment timing, control group selection) significantly influences the results. They emphasize that while the model aligns with empirical findings, it carries ethical risks regarding structural biases and requires human oversight for applications in automated decision-making.