Quantifying the human visual exposome with vision language models

Technical Summary: Quantifying the Human Visual Exposome with Vision Language Models

Problem Statement
The environmental exposome is a recognized determinant of mental health, yet its immediate visual component—the "visual exposome"—remains largely unquantified. Current epidemiological methods rely on coarse geospatial proxies (e.g., satellite-derived greenness indices) or biased self-reports. These approaches fail to capture the semantic richness, spatial nuance, and first-person perspective of daily life. Specifically, they cannot determine what an individual actually sees in their immediate field of view (e.g., the presence of animals, crowd density, or specific stress-inducing cues). While participant-generated photography offers a solution for capturing first-person context, its application in quantitative research has been limited by the reliance on time-consuming, subjective, and non-scalable manual expert coding.

Methodology
The study employed a multimodal approach combining ecological momentary assessment (EMA) with Vision-Language Models (VLMs) and large-scale AI-driven literature mining.

1. Data Collection: The researchers conducted a 7-day EMA study with 106 adults using the esmira smartphone application. Participants received seven random daily alarms to report momentary positive and negative affect, perceived greenness, and to capture a photograph of their current environment from a first-person perspective. This yielded 2,674 participant-generated photographs paired with self-reports. Participants also completed baseline sociodemographic questionnaires and the Perceived Stress Scale (PSS) for chronic stress.
2. VLM Validation (Greenness Benchmark): To validate the framework, the team used the open-weight LLaMA 4 VLM (Maverick-17B-128E-Instruct-FP8) to generate quantitative ratings for five greenness-related features: greenness, nature score, plant presence, natural light exposure, and inside/outside classification. Each image was rated five times per feature to ensure reliability.
3. Literature Mining for Feature Discovery: To expand beyond greenness, a semi-autonomous pipeline using the GPT-OSS-120B model mined over 7 million open-access publications from the Europe PMC database. This process identified 997 unique contextual features with empirical support for associations with affect and stress.
4. Large-Scale Application: The identified 997 features were used as prompts for the VLM to rate the participant photographs.
5. Statistical Analysis: Multilevel linear models (using R and lme4) analyzed the correspondence between VLM-derived ratings and self-reports, as well as associations with state-level (momentary) and trait-level (chronic) mental health variables. Cross-model consistency was verified using the Qwen3 VL architecture.

Key Results

• Validation against Self-Report: VLM-derived ratings of greenness and related features showed significant associations with participants' subjective self-ratings of greenness.
• Association with Mental Health:
  • State Level: Greater VLM-derived greenness was significantly associated with higher momentary positive affect.
  • Trait Level: Higher average greenness and nature scores were associated with higher average positive affect, lower average negative affect, and lower perceived chronic stress.
  • Inside/Outside: Being outdoors was significantly linked to higher positive affective states.
• Cross-Model Consistency: Ratings generated by LLaMA 4 and Qwen3 VL were highly correlated (Spearman's r = 0.83–0.89 for greenness, nature, and plants), confirming the robustness of the approach across different model architectures.
• Scalability and Generalizability: When applied to the 997 literature-derived contextual features, up to 33% of the VLM-extracted context ratings showed significant correlations with momentary affect and stress (at p < 0.05), and up to 12% showed expected patterns for trait-level measures. This exceeded the 5% rate expected by chance, demonstrating the framework's ability to detect meaningful associations across a broad conceptual space.

Key Contributions

• Definition and Quantification: The study establishes the "visual exposome" as a measurable construct, moving beyond location-based proxies to direct, first-person visual quantification.
• Scalable Methodology: It introduces a pipeline that replaces labor-intensive manual coding with automated VLM analysis, enabling high-throughput decoding of visual context in large-scale longitudinal studies.
• Hypothesis-Generating Framework: By mining millions of publications to identify nearly 1,000 potential environmental determinants, the study shifts the paradigm from testing single hypotheses to screening thousands of visual determinants simultaneously.

Significance and Claims
The authors claim this work establishes a foundational technology for "visual exposomics." They argue that by coupling EMA with VLMs, the measurement gap between macro-level environmental data and micro-level individual perception is resolved. The study posits that this approach provides a higher-resolution phenotype of environmental exposure that avoids the biases of self-report and the spatial coarseness of satellite metrics.

The paper concludes that the human visual experience can be objectively quantified and mapped to affect and stress. It suggests this framework opens new avenues for precision public health, enabling the systematic decoding of how the visible world shapes health outcomes. The authors maintain a modest stance regarding causality, noting that the micro-longitudinal design captures associations rather than causal effects, and that future work is required to disentangle bidirectional influences (e.g., whether positive affect drives individuals to seek specific visual environments). The study emphasizes that this methodology parallels technological shifts in wearable biosensors and AI-driven healthcare, offering a scalable tool for integrating detailed contextual information into personalized health models.