LiFT: Does Instruction Fine-Tuning Improve In-Context Learning for Longitudinal Modelling by Large Language Models?

1. Problem Statement

Longitudinal NLP tasks involve reasoning over temporally ordered text sequences to detect persistence, state transitions, and changes in human behavior or opinions (e.g., mental health monitoring, stance change detection, timeline summarization).

Key Challenges:

• Temporal Dependencies: Unlike static NLP tasks, longitudinal tasks require modeling how observations evolve over time.
• Context Limits: Histories often span long time horizons, exceeding the context windows of standard Large Language Models (LLMs).
• In-Context Learning (ICL) Limitations: Current LLMs struggle with zero-shot or few-shot prompting on longitudinal tasks. Standard approaches often concatenate historical observations into a single prompt without explicitly modeling temporal structure, leading to poor performance on rare change events and evolving interactions.
• Prompt Brittleness: Improving prompt design alone (e.g., better examples, chain-of-thought) does not sufficiently address the structural difficulties of integrating temporally distributed information.

2. Methodology: The LiFT Framework

The authors propose LiFT (Longitudinal Instruction Fine-Tuning), a framework designed to unify diverse longitudinal modeling tasks under a shared instruction schema. LiFT combines curriculum learning, temporal conditioning, and parameter-efficient fine-tuning.

Core Components:

1. Longitudinal Prompting Framework:

   • Data is grouped by sequence keys (e.g., user ID) and sorted chronologically.
   • Rolling History: Training instances consist of a rolling historical context paired with the current observation. To fit context limits, the oldest history is iteratively truncated.
   • Relative Time Indices: History lines are prefixed with relative time indices (e.g., $t-3, t-2, t-1$) to reinforce temporal ordering.

2. Curriculum-Based Instruction Tuning:

   • Training proceeds in three stages of increasing temporal difficulty:
     • Stage 1 (AnnoMI): Structured motivational interviewing dialogues with explicit behavioral intent labels (dense supervision).
     • Stage 2 (LRS): Longitudinal rumor stance detection with evolving user stances and noisier social media language.
     • Stage 3 (TalkLife): Mental health timelines where change signals are sparse, implicit, and require inference over subtle cues.
   • The number of in-context demonstrations ($k$-shots) increases progressively from 1 to 3 across stages.

3. Temporal–Label Conditioning Module:

   • Token embeddings are augmented with four signals: absolute position ($p_t$), relative position-to-end ($r_t$), global label ID ($z_t$), and absolute timestep ID ($\tau_t$).
   • These are projected and added to the LM input space, forcing the model to attend to temporal structure explicitly.

4. Multi-Objective Loss Function:
   The model is trained using a weighted sum of four objectives:

   • $L_{CE}$: Standard next-token cross-entropy (excluding history spans) to stabilize instruction following.
   • $L_{focalLM}$: Focal language modeling loss applied only to output tokens, focusing learning on difficult predictions.
   • $L_{focalCls}$: Focal classification loss over a shared global label space to handle class imbalance.
   • $L_{histCls}$ (Auxiliary): A history-only classification loss computed from the pooled representation of the <HIST> tokens. This explicitly encourages the model to extract predictive signals from the historical context.

5. Training Strategy:

   • LoRA (Low-Rank Adaptation): Used for parameter-efficient fine-tuning.
   • Progressive Rank Scaling: The LoRA rank increases across curriculum stages (e.g., $r=4 \to 8 \to 16$). Previous adapter weights are loaded and resized, freezing previously learned ranks while updating new ones.

3. Key Contributions

• LiFT Framework: A novel instruction fine-tuning framework that unifies longitudinal tasks via a shared schema and curriculum learning.
• Temporal Conditioning: Introduction of a specific module to inject timestep and label signals, alongside an auxiliary loss to force the model to utilize historical context.
• Comprehensive Evaluation: Evaluation across five datasets (AnnoMI, LRS, TalkLife, Reddit, CMV) and four model scales (OLMo 1B/7B, LLaMA-8B, Qwen-14B).
• Mechanistic Interpretability: Probing, attention analysis, and activation patching experiments demonstrating why LiFT works (increased reliance on temporally structured history).

4. Experimental Results

The study compares base models using standard ICL against LiFT-trained models under identical prompting conditions (0, 1, and 3-shot).

• Performance Gains: LiFT consistently outperforms base models across all datasets and model sizes.
  • Minority Class Improvement: Significant gains were observed on minority "change" events (e.g., on TalkLife, OLMo-7B improved from ~0.05 to ~0.12 Macro-F1).
  • Out-of-Distribution (OOD) Generalization: Models trained on AnnoMI, LRS, and TalkLife successfully transferred improvements to held-out datasets (Reddit, CMV).
  • Few-Shot Synergy: LiFT models benefit more from additional in-context examples (1-shot, 3-shot) than base models, which often plateau or degrade.
• Ablation Studies:
  • Removing Temporal Conditioning caused the largest performance drops, confirming the necessity of explicit temporal signals.
  • Removing Curriculum Learning reduced performance, highlighting the benefit of staged difficulty.
  • Removing the History-Aware Loss had mixed but generally negative effects, proving the value of auxiliary supervision on historical context.
• Mechanistic Analysis:
  • Probing: LiFT models showed higher linear decodability of label information in history representations compared to base models.
  • Attention: LiFT shifted attention from instruction scaffolding toward evidence-bearing regions (history and current tokens), with a stronger focus on recent history.
  • Activation Patching: Corrupting history activations in LiFT models caused a significantly larger drop in performance than in base models, providing causal evidence that LiFT relies heavily on temporally structured history.

5. Significance and Conclusion

• Paradigm Shift: The paper demonstrates that for longitudinal tasks, standard ICL is insufficient. Explicit instruction fine-tuning that models temporal dependencies is necessary to unlock LLM capabilities in this domain.
• Generalizability: The framework is model-agnostic and domain-agnostic, working effectively across different architectures (OLMo, LLaMA, Qwen) and diverse tasks (mental health, stance detection, persuasion).
• Interpretability: The work moves beyond "black box" performance gains, providing mechanistic evidence that LiFT successfully teaches models to treat history as a causal, predictive signal rather than just context.
• Ethical Considerations: The authors highlight the sensitivity of mental health data used (TalkLife), emphasizing that these models are for research only and require strict ethical oversight before any potential clinical application.

In summary, LiFT establishes that combining curriculum learning, temporal conditioning, and history-aware auxiliary objectives significantly enhances the ability of LLMs to reason over time, offering a robust solution for longitudinal NLP tasks where current prompting strategies fail.