💬 NLP

Privately Fine-Tuned LLMs Preserve Temporal Dynamics in Tabular Data

The paper introduces PATH, a novel framework that leverages privately fine-tuned large language models to synthesize differentially private longitudinal tabular data, effectively preserving temporal dynamics and long-range dependencies that traditional marginal-based methods fail to capture.

Original authors: Lucas Rosenblatt, Peihan Liu, Ryan McKenna, Natalia Ponomareva

Published 2026-02-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lucas Rosenblatt, Peihan Liu, Ryan McKenna, Natalia Ponomareva

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to create a fake set of medical records that looks just like real ones, but without revealing any actual patient's secrets. This is a common challenge in data science: how do you share useful data without breaking privacy laws?

For a long time, the standard way to do this was to treat every single row in a spreadsheet as a separate person. If a patient had 50 doctor visits, the system would treat those 50 rows as 50 different people. The paper calls this the "flattening" approach.

The Problem: The "Scrambled Puzzle" Effect
The authors argue that this "flattening" method is like taking a beautiful, complex story and chopping it up into individual sentences, then shuffling them into a giant pile. You might keep the vocabulary (the words used) correct, but you lose the plot.

In real life, a patient's health is a story with a beginning, middle, and end. Their heart rate today depends on what happened yesterday. When you chop these stories into isolated rows, the computer loses the ability to see the timeline. It might generate a fake record where a patient has a heart attack, then instantly recovers to perfect health the next second, then has a heart attack again. Locally, the numbers look okay, but the story makes no sense.

The Solution: PATH (Private Autoregressive Trajectory Histories)
The authors introduce a new method called PATH. Instead of treating rows as separate people, PATH treats an entire patient's history as one single story.

Think of it like this:

Old Method (Flattening): You give the AI a bag of 1,000 loose Lego bricks and ask it to build a castle. It might build a castle, but the walls might fall over because it doesn't know which bricks connect to which.
PATH: You give the AI a set of instructions to build a castle brick by brick, where each new brick must fit perfectly with the one before it.

PATH uses a special type of AI (a Large Language Model) that is really good at reading and writing stories. It learns to predict the next row of a patient's data based on the rows that came before it, just like a writer predicting the next sentence in a novel.

How They Keep It Private
To ensure no real patient's data is leaked, they use a technique called "Differential Privacy." Imagine the AI is trying to learn a secret recipe. Instead of letting it taste the exact ingredients, they give it a slightly "noisy" version of the recipe. It learns the general flavor and structure without ever memorizing the exact amount of salt in a specific dish.

In PATH, the "secret" being protected is the entire story of one person, not just a single sentence. This is a crucial shift. They protect the whole timeline, ensuring that even if someone tries to reverse-engineer the data, they can't figure out what happened to a specific patient.

What They Found
The researchers tested PATH on real-world data (like hospital records and city service requests) and compared it to the old "flattening" methods.

Better Stories: PATH created fake data that looked much more like real life. The fake patients had realistic timelines where their health conditions evolved naturally, rather than jumping randomly.
Less "Hallucination": The old methods often created impossible scenarios (like a patient having two different heart rates at the exact same time). PATH avoided these errors.
Privacy vs. Quality: Even when they made the privacy protection very strict (adding more "noise"), PATH still managed to create useful data. The old methods fell apart completely when privacy got too strict because they were trying to solve a puzzle with too many missing pieces.

The Bottom Line
This paper shows that when dealing with time-based data (like health records or city complaints), we shouldn't just look at the data as a spreadsheet of isolated facts. We need to treat it as a collection of stories. By using AI that understands how stories flow from one moment to the next, and by protecting the whole story rather than just the individual sentences, we can create high-quality, fake data that is safe to share and incredibly useful for research.

Technical Summary: PRIVATELY FINE-TUNED LLMS PRESERVE TEMPORAL DYNAMICS IN TABULAR DATA

Problem Formulation

Current research in differentially private synthetic tabular data (DPSD) predominantly focuses on independent and identically distributed (i.i.d.) rows, where a single row corresponds to a unique individual. This paradigm fails to address the complexity of longitudinal datasets, such as electronic health records (EHR) or service request logs, where a single user contributes an entire sequence of events (a sub-table) over time. In these settings, the unit of ownership is the full table (trajectory), not the individual row.

Existing approaches attempt to adapt to this by "flattening" user trajectories into high-dimensional fixed-width vectors (concatenating all time steps into a single row) to apply standard marginal-based mechanisms (e.g., AIM, Private-PGM). The authors argue this strategy is fundamentally insufficient. Flattening explodes the dimensionality of the domain, introduces artificial sparsity through padding, and forces privacy mechanisms to rely on low-order local marginals to stay within budget. Consequently, these methods fail to capture long-range temporal dependencies, often generating trajectories that are locally plausible but globally incoherent (e.g., mixing distinct user profiles). Furthermore, applying group privacy to flattened data is overly restrictive, as it protects groups of rows that may belong to different users.

Methodology: The PATH Framework

To address these limitations, the authors introduce PATH (Private Autoregressive Trajectory Histories), a generative framework that treats the full user table as the unit of synthesis and leverages the autoregressive capabilities of Large Language Models (LLMs).

Core Components

Privacy Unit Definition: The framework explicitly defines the privacy unit as the full user table $D^{(i)}$ . Two datasets are considered neighboring if one can be obtained by adding or removing exactly one user's entire trajectory. This ensures the privacy guarantee protects the complete temporal history of an individual.
Serialization: User tables are deterministically serialized into token sequences using a schema definition followed by row-by-row encoding (e.g., [Row 1]: charttime is ..., heartrate is ...). Row-delimiting tokens encourage the model to distinguish temporal steps.
Training Strategy:
- DP Fine-tuning: The authors fine-tune pre-trained LLMs (specifically the Gemma 3 family, 1B and 4B parameters) using Differentially Private Stochastic Gradient Descent (DP-SGD).
- Dynamic Windowing: To ensure robust learning of transitions at any stage of a trajectory, the training process employs a dynamic windowing strategy. For each user, a split point is sampled to create a context (history) and a target (continuation). This allows the model to learn from the full diversity of the user's history while maintaining a sensitivity of 1 per user per step.
- Density-based Sampling: Split points are sampled based on the empirical distribution of table lengths to prevent bias toward short contexts.
- Architecture: Low-Rank Adaptation (LoRA) is used to update a small fraction of parameters, reducing the dimensionality of gradient updates that require noise injection.
Inference and Selection:
- Autoregressive Generation: Synthetic tables are generated row-by-row, conditioning each new row on the schema and previously generated rows.
- Robust Parsing: A multi-step parsing strategy handles malformed outputs, falling back to schema-matching or partial infilling.
- Private Selection: To ensure high utility, a private selection mechanism filters an over-generated pool of candidate tables. This step uses a differentially private voting mechanism based on semantic embeddings (Gecko) to select the most representative tables without consuming additional privacy budget beyond the allocated selection budget.

Key Contributions

Privacy Unit as Full Table: The paper is the first to formalize the full user table as the privacy unit for differentially private data synthesis and demonstrates a practical application for preserving temporal trajectories.
Novel Framework for Temporal Trends: PATH introduces a methodology for fine-tuning LLMs to generate structured, multi-row tabular data, utilizing a two-stage process of autoregressive row generation and private selection to preserve temporal dependencies.
Novel Metrics: The authors propose Table-wise Distance to Closest Record (TDCR), a metric adapted from Distance to Closest Record (DCR) that utilizes Dynamic Time Warping (DTW) to evaluate the distributional fidelity of generated user trajectories against real data, penalizing failures to replicate temporal shape.
Extensive Empirical Evaluation: The framework is evaluated on synthetic HMM data, MIMIC-IV vital signs, and NYC 311 service requests, demonstrating superior performance over state-of-the-art marginal baselines.

Experimental Results

The authors conducted extensive evaluations across three datasets:

MIMIC-IV (Vital Signs): PATH (Gemma 4B) significantly outperformed marginal-based baselines (AIM, Direct Mechanism). At $\epsilon=2.0$ , PATH reduced the TDCR by over 50% (0.289 vs. 0.736 for AIM) and reduced state transition errors by nearly 50%. Crucially, PATH maintained high marginal fidelity while capturing temporal dynamics that flattened methods missed.
Synthetic HMM Data: On data with known ground-truth transition probabilities, PATH demonstrated superior ability to capture the underlying latent process. While marginal methods achieved lower univariate marginal divergence, they produced trajectories with prohibitively high HMM log-likelihoods (indicating invalid sequences). PATH's generated sequences closely matched the true HMM likelihood distribution.
NYC 311 (Spatiotemporal): PATH successfully preserved complex geospatial distributions, diurnal rhythms, and complaint type transitions. Non-private few-shot prompting (Gemini 2.5 Flash-Lite) failed to generalize the full distribution, often resulting in mode collapse, whereas private fine-tuning captured population variety.

Key Findings:

Flattening is Insufficient: Flattening longitudinal data leads to a "catastrophic" loss of utility due to dimensionality explosion and the inability of marginal mechanisms to capture global dependencies.
LLMs vs. Marginals: While marginal methods excel at aggregate statistics in simple, well-behaved datasets, LLMs dominate in capturing temporal dynamics and state transitions, offering a "best of both worlds" solution for complex longitudinal data.
Fine-tuning vs. Prompting: Private fine-tuning is more effective than in-context learning (few-shot prompting) for distribution matching. Prompting often leads to mode collapse and fails to capture the full support of the data manifold.
Model Scaling: Larger models (Gemma 4B) exhibit greater robustness under strict privacy budgets ( $\epsilon < 2.0$ ) compared to smaller variants, likely due to stronger pre-trained semantic priors that stabilize learning when DP noise is high.

Significance and Claims

The paper claims that PATH redefines the unit of differential privacy from the individual row to the entire user table, addressing a fundamental gap in the synthesis of sensitive temporal data. The authors assert that their work demonstrates that LLMs, when privately fine-tuned, can serve as effective synthesizers for sequential tabular domains, preserving intricate temporal dynamics that traditional marginal-based mechanisms fail to capture.

The significance lies in the operationalization of a multi-table privacy unit, which is essential for applications like EHR analysis where a patient's history is a coherent trajectory rather than a collection of independent records. The results suggest that for longitudinal data, the "unit of synthesis" matters: treating a user's trajectory as a single autoregressive sequence is a more efficient and effective representation than flattening user histories. The paper concludes that private fine-tuning offers a robust solution for data sharing in sequential domains, enabling the release of high-fidelity synthetic data that respects both privacy and temporal coherence.