Technical Summary: TEMPVIZ – On the Evaluation of Temporal Knowledge in Text-to-Image Models

Problem Statement

Time fundamentally alters the visual appearance of entities in the real world, including objects, places, animals, and landscapes. While substantial research exists on temporal knowledge and reasoning in Natural Language Processing (NLP), the landscape of resources and evaluation methods for temporal knowledge in Text-to-Image (T2I) models remains largely unexplored. Users expect T2I models to generate contextually relevant images based on temporal instructions (e.g., a landscape in winter vs. summer, or a building before vs. after a specific historical event). However, it is currently unclear how well existing T2I models can interpret and apply these temporal cues, nor is there a reliable method to automatically evaluate such capabilities.

Methodology

To address this gap, the authors introduce TEMPVIZ, a novel dataset and evaluation framework designed to holistically assess temporal knowledge in image generation.

1. The TEMPVIZ Dataset

The dataset consists of 7,940 prompts spanning five distinct subject categories, chosen to reflect common use cases where temporal effects are visually observable:

• Landscapes: Tests seasonal and daily variation (e.g., "a lake in January in Europe").
• Animals: Tests biological aging across life stages (e.g., "an Alpaca that is one month old"), utilizing lifespan data from HAGR.
• Buildings: Tests historical existence and destruction (e.g., "the Berlin Wall on 10/23/1983"), focusing on structures with known inception and demolition dates.
• Maps: Tests geopolitical border changes over time (e.g., "a map of Europe in 1938").
• Artworks: Tests adherence to specific art historical periods and styles (e.g., "an artwork in the style of Impressionism").

For each prompt, the authors generated four paraphrased variations. The dataset includes >600 reference images (sourced from Wikidata, Google Arts & Culture, and OpenHistoricalMap) and "expected values" (textual descriptors of visual attributes) to support annotation. Notably, the dataset excludes human subjects to avoid ethical complications regarding demographic bias and stereotype reinforcement.

2. Evaluation Protocol

The study employs a dual-evaluation approach:

• Human Evaluation: Two annotators assessed a stratified sample of 500 generated images across five T2I models (Stable Diffusion v1.5, SDXL-Turbo, SDXL-Base, SD 3.5, and FLUX.1-dev). Annotators answered three binary questions:
  1. Is the image free from visual generation errors?
  2. Is the correct subject present?
  3. Did the model apply the required temporal knowledge correctly?
• Automatic Evaluation: The authors tested several established automated methods against human judgments to determine their reliability in detecting temporal cues. These included:
  • CLIPScore: Measuring text-image embedding similarity.
  • Captioning: Generating image captions and comparing them to the prompt.
  • Decompositional VQA: Using an LLM to break prompts into fine-grained questions answered by a Vision-Language Model (VLM).
  • Direct VLM Judging: Prompting VLMs (Qwen2.5-VL, GPT-4o mini, GPT-5) to directly judge if the image matches the prompt's temporal constraints.

Key Results

Human Evaluation Findings

• Low Temporal Competence: Across all five models and categories, temporal competence is generally weak. No model exceeded 75% accuracy in correctly applying temporal cues for any single category.
• Category Variance: The Maps category proved most challenging, with even the best-performing model (FLUX) achieving only 50% error-free generation. Artworks showed the highest error-free rates, though this may be due to annotators interpreting generation errors as stylistic variations.
• Subject vs. Temporal Performance: While models generally performed well on subject presence (Q2), this did not correlate with temporal accuracy (Q3). For instance, FLUX achieved high scores for image quality and subject presence but only 38% temporal accuracy, whereas SD v1.5 achieved the highest average temporal accuracy (46%) despite lower overall coherence.
• Conclusion: General image quality is not an indicator of temporal knowledge encoding. Current T2I models struggle to integrate temporal cues consistently, even in categories that appear simpler.

Automatic Evaluation Findings

• Ineffectiveness of Traditional Metrics: CLIPScore showed no significant correlation with human judgments regarding temporal correctness. Caption-based evaluation showed some correlation with subject presence but yielded negative correlations for temporal cues, indicating these methods can be misleading when correctness depends on time rather than object identity.
• Limitations of VLM Judges: Even advanced VLMs (including GPT-5) failed to provide reliable assessments. The best-performing configuration (GPT-5 with 3-shot prompting) achieved a macro F1 score of only 72%, well below the threshold for reliable automated evaluation.
• Misalignment with Human Difficulty: Automated judges did not consistently align with human-perceived task difficulty. For example, GPT-5 frequently misjudged the "Animals" category (aging) despite humans finding it less challenging than "Buildings" or "Maps."

Significance and Contributions

The paper makes three primary contributions:

1. TEMPVIZ Dataset: The first dataset specifically designed to analyze temporal knowledge in T2I models, comprising 7.9k prompts across five diverse categories with reference images and expected values.
2. Empirical Benchmarking: A comprehensive human evaluation of five state-of-the-art T2I models, revealing that even the strongest models fail to consistently apply temporal knowledge, with accuracy rarely exceeding 75%.
3. Evaluation Gap Analysis: A systematic assessment of automated evaluation methods, demonstrating that current approaches (CLIPScore, captioning, VQA, and direct VLM judging) are insufficient for reliably judging temporal cues in generated images.

The authors conclude that there is a pressing need for future research focused on improving both the generation of temporal cues in T2I models and the development of evaluation metrics capable of assessing these fine-grained visual-temporal relationships. The paper positions TEMPVIZ as a foundational resource to motivate this direction, rather than claiming to have solved the underlying generation or evaluation challenges.