🤖 AI

LENS: LLM-Enabled Narrative Synthesis for Mental Health by Aligning Multimodal Sensing with Language Models

The paper introduces LENS, a framework that aligns multimodal sensor data with large language models via a patch-level encoder and a large-scale sensor-text dataset to generate clinically meaningful mental health narratives, thereby enabling LLMs to reason over raw behavioral signals for improved clinical decision-making.

Original authors: Wenxuan Xu, Arvind Pillai, Subigya Nepal, Amanda C Collins, Daniel M Mackin, Michael V Heinz, Tess Z Griffin, Nicholas C Jacobson, Andrew Campbell

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenxuan Xu, Arvind Pillai, Subigya Nepal, Amanda C Collins, Daniel M Mackin, Michael V Heinz, Tess Z Griffin, Nicholas C Jacobson, Andrew Campbell

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a smartwatch and a smartphone that are constantly watching your daily life. They record your heart rate, how many steps you take, how much you sleep, where you go, and even how often you unlock your phone. This is a lot of raw numbers, like a massive, chaotic spreadsheet that no one wants to read.

Now, imagine a doctor who needs to understand your mental health but doesn't have time to stare at that spreadsheet. They need a story—a clear, human-readable summary of how you've been feeling.

This is the problem the paper LENS tries to solve.

The Problem: The Language Barrier

Currently, computers are great at two things separately:

Reading Numbers: They can analyze your heart rate and sleep data perfectly.
Writing Stories: Large Language Models (LLMs) like the ones powering chatbots are amazing at writing natural, fluent text.

But they struggle to do both at once. If you try to feed a computer a 24-hour stream of heart rate numbers directly into a chatbot, it gets confused. It's like trying to explain a symphony by reading a list of every single note frequency to someone who only speaks English. The computer can't "hear" the music in the numbers. Also, there aren't many examples in the world where someone has paired raw sensor data with a perfect written story about mental health, so the computers haven't learned how to do it yet.

The Solution: LENS (The Translator)

The researchers built a system called LENS (LLM-Enabled Narrative Synthesis). Think of LENS as a specialized translator that turns the "language of numbers" into the "language of stories."

Here is how it works, using a simple analogy:

1. Building the Dictionary (The Dataset)

First, the team needed a dictionary to teach the computer how to translate. They couldn't just make up stories; they needed real data.

The Source: They worked with 258 people who have depression or anxiety. These people wore smartwatches and filled out digital surveys (called EMAs) several times a day, rating things like "How interested were you in doing things?" on a scale of 0 to 100.
The Translation: The researchers took these survey answers and used AI to turn the numbers (e.g., "75 out of 100") into natural sentences (e.g., "The user often felt a lack of interest in activities").
The Result: They created over 100,000 pairs of "Sensor Data + Story." This is the textbook the computer uses to learn.

2. The New Translator (The Architecture)

Most computers try to turn numbers into words by just listing them out (e.g., "Heart rate: 80, 82, 79..."). This is clunky and takes up too much space.

LENS's Trick: Instead of listing every number, LENS uses a special "patch encoder." Imagine taking a long movie and cutting it into small, manageable 8-second clips. LENS looks at these small clips of sensor data and converts them into a compact "summary token" that the language model can understand instantly.
The Magic: It then feeds these compact tokens directly into the language model, allowing the AI to "read" the sensor data as if it were part of the story itself.

3. The Test (Did it work?)

The researchers tested LENS against other methods:

Method A (The Text-Only): Trying to feed the raw numbers as text.
Method B (The Visual): Turning the numbers into charts and asking an AI to look at the pictures.
LENS: The new translator.

The Results:

Better Stories: LENS wrote summaries that were much more accurate and fluent than the other methods. It didn't miss important details or make up fake symptoms (hallucinations).
Expert Approval: They showed the stories to 13 mental health professionals (therapists and psychologists). The experts rated LENS's stories as significantly more useful, accurate, and comprehensive than the text-only method.
Efficiency: LENS was also much faster and lighter. It didn't need to process huge images or massive lists of numbers; it used a "smart summary" approach that saved a lot of computing power.

What LENS Is (and Isn't)

The paper is very specific about what this tool is designed to do:

It is a Pre-Consultation Tool: LENS is meant to help a doctor before they meet a patient. It gives the doctor a quick, readable snapshot of the patient's recent state based on their daily data.
It is NOT a Diagnostic Tool: The paper explicitly states that LENS is not meant to diagnose someone with depression or anxiety on its own. It is an assistant to help human doctors make better-informed decisions.
It is NOT for the General Public: The study focused specifically on people already diagnosed with depression or anxiety. It hasn't been tested on the general population yet.

The Bottom Line

LENS is like a smart interpreter that sits between your wearable device and your doctor. It takes the cold, hard data of your daily life and translates it into a warm, clear narrative that a human can understand, helping doctors see the full picture of a patient's mental health without getting lost in the numbers.

Technical Summary: LENS (LLM-Enabled Narrative Synthesis for Mental Health)

Problem Statement

Multimodal health sensing offers rich behavioral signals (e.g., heart rate, sleep patterns, mobility) for assessing mental health conditions like depression and anxiety. However, translating these numerical time-series measurements into natural language remains a significant challenge. Current Large Language Models (LLMs) face two primary limitations:

Inability to Natively Ingest Long Time-Series: LLMs struggle with long-duration sensor streams due to context length constraints and tokenizer designs that prevent direct ingestion of raw numerical sequences.
Scarcity of Paired Data: There is a lack of large-scale datasets that pair raw sensor streams with clinically grounded text descriptions. Existing methods often rely on synthetic data or convert time-series into images, which introduces biases and indirect representations.

Consequently, existing approaches rarely operate directly on raw sensor streams to generate clinically meaningful narratives, limiting the scalability of LLMs as interfaces for health sensing.

Methodology

The authors introduce LENS, a framework designed to align multimodal sensing data with language models to generate clinically grounded mental health narratives. The approach consists of two main components:

1. Dataset Construction Pipeline

To address the data scarcity, the authors constructed a large-scale dataset of over 100,000 sensor–text QA pairs from 258 participants diagnosed with Major Depressive Disorder (MDD).

Source Data: The study utilized 90-day longitudinal data collected via Garmin wearables and Android smartphones, capturing passive sensing (GPS, steps, heart rate, stress, phone usage) and Ecological Momentary Assessments (EMA).
EMA Alignment: Participants completed 14 EMA items (adapted from PHQ-9 and GAD-4) three times daily, rating symptoms on a 0–100 scale. These responses were temporally aligned with the preceding four hours of sensor data.
Narrative Synthesis:
- Template Generation: Numeric EMA responses were converted into raw narrative templates using frequency phrases (e.g., "not at all," "constantly").
- LLM Refinement: A locally deployed LLM (Qwen2.5-14B) refined these templates to enhance fluency and lexical diversity while enforcing factual accuracy and stigma-free language.
- Quality Control: A multi-agent "LLM-as-a-Judge" system (using Mistral-7B, Llama-3.1-8B, and Qwen2.5-7B) evaluated the narratives against the original templates across five dimensions (factual alignment, coverage, severity fidelity, fluency, hallucination risk). Only narratives exceeding a quality threshold were accepted.
Output: The pipeline produced Item-level QA pairs (single symptom) and Summary-level QA pairs (full symptom overview).

2. Model Architecture and Training

LENS employs a patch-based encoder to enable native time-series integration into an LLM.

Time-Series Encoder: Raw scalar sensor values undergo reversible value normalization. The sequence is divided into non-overlapping patches (width $w=8$ ). Each patch is augmented with learnable positional embeddings and projected into the LLM's hidden space via a Multi-Layer Perceptron (MLP). This preserves absolute numerical semantics while capturing localized temporal patterns.
Architecture: The model interleaves text embeddings (from instructions and metadata) with patch embeddings from the time-series encoder. Special tokens (<ts>, </ts>) mark the positions of sensor streams within the prompt.
Training Strategy:
- Stage 1 (Encoder Alignment): The encoder is aligned with textual descriptions using a ChatTS dataset to capture trends and correlations, interleaved with narrative samples to prepare for downstream generation.
- Stage 2 (Supervised Fine-Tuning): The model is fine-tuned on the constructed EMA-derived QA datasets. The training mix includes Item-level QA, Summary-level QA, Instruction-Following (IF) data, and an alignment-random subset to handle variable sequence lengths.

Key Contributions

LENS Data Synthesis Pipeline: A scalable pipeline that transforms EMA responses into high-quality, clinically grounded sensor–text pairs, addressing the shortage of such resources with over 100,000 generated samples.
Native Time-Series Integration: A training strategy utilizing a patch-level encoder that projects raw sensor signals directly into the LLM's representation space, avoiding the need for text serialization or image conversion.
Comprehensive Evaluation: Evaluation on a clinical dataset of 258 participants (50,957 samples) and a user study with 13 mental health professionals, demonstrating performance on both standard NLP metrics and clinical symptom-severity accuracy.

Results

Performance vs. Baselines: LENS outperformed strong baselines, including a text-only serialization approach (TS-Text) and a vision-based approach converting signals to plots (TS-Image).
- Summary-Level: LENS achieved a ROUGE-L of 0.409 and BERTScore of 0.775, surpassing TS-Image (0.373 ROUGE-L) and significantly outperforming TS-Text (0.151 ROUGE-L).
- Item-Level QA: LENS achieved a ROUGE-L of 0.603, more than four times higher than the baselines, indicating superior precision in retrieving specific symptom intensities.
- Clinical Alignment: LENS demonstrated higher symptom coverage (0.801) and presence alignment (0.601) compared to baselines, minimizing hallucinations.
User Study: In a study with 13 mental health experts, LENS narratives were rated significantly higher than TS-Text across all four dimensions: comprehensiveness, accuracy, clinical utility, and language cohesion.
Parity with Larger Models: LENS (14B parameters) matched the performance of the 32B-parameter TS-Image model in expert ratings, despite the latter's larger size and reliance on visual plot engineering.
Efficiency: LENS required approximately 930 tokens per sample, a 94% reduction compared to text-only serialization and a 4x reduction compared to vision-based models.
Ablation: Experiments confirmed that explicit time-series encoding is essential; a text-only fine-tuned baseline performed significantly worse. Sensor mismatch experiments showed a 20.1% drop in presence alignment when sensor streams were shuffled, proving the model grounds content in actual signals rather than stylistic patterns.

Significance

The paper positions LENS as a step toward ecologically valid mental health monitoring. By transforming raw sensor data into interpretable clinical insights, LENS advances the capability of LLMs to reason over raw behavioral signals. The authors frame the system not as a diagnostic tool, but as a pre-consultation monitoring tool that provides clinicians with naturalistic behavioral summaries, enabling more informed and time-efficient consultations. The work demonstrates a scalable path for integrating passive sensing with language models, potentially supporting downstream clinical decision-making without the overhead of manual data interpretation or complex visual engineering.

The authors acknowledge limitations, noting the current focus on depression and anxiety within a specific clinical cohort, the 4-hour window analysis (which may miss long-term trends), and the reliance on specific model families. They emphasize that generated narratives must augment, not replace, clinical judgment.