Daily and Weekly Periodicity in Large Language Model Performance and Its Implications for Research

This paper challenges the assumption of time invariance in large language model performance by demonstrating through a longitudinal study of GPT-4o that its output quality exhibits significant daily and weekly periodic variability, accounting for approximately 20% of total variance.

The Gist

Imagine you have a super-smart robot tutor named "GPT-4o." You ask it the same difficult physics question every day, hoping it gives you the same perfect answer every time. You assume that because the robot's brain (the code) hasn't changed, its performance should be rock-steady, like a lighthouse beam that never flickers.

This paper is like a detective story that proves that assumption wrong. The researchers discovered that this robot tutor actually has a "mood swing" that follows a strict schedule, changing its performance based on the time of day and the day of the week.

Here is the breakdown of their findings using simple analogies:

1. The "Time-Invariant" Myth

The Assumption: Scientists often treat AI like a calculator. If you type 2 + 2 at 9:00 AM or 9:00 PM, you expect 4 both times. They assume the AI's "average quality" is time-invariant (it doesn't matter when you ask).

The Reality: The researchers treated the AI like a human employee. They asked it the same physics puzzle 6,930 times over three months, checking it every three hours. They found that the AI's performance wasn't a flat line; it was a wavy line.

2. The "Server Traffic Jam" Analogy

Why does the robot's performance change? Think of the AI not as a single computer, but as a giant, busy highway system connecting millions of users.

• The Rush Hour Effect: Just like a highway gets clogged during morning and evening commutes, the servers hosting the AI get flooded with requests during work hours and weekdays.
• The "Fast Lane" vs. "Slow Lane": When the highway is jammed, the service provider (OpenAI) has to manage the traffic. They might use shortcuts, like compressing the data or simplifying the route to keep things moving fast.
• The Cost of Speed: These shortcuts make the AI faster but dumber. It's like a chef who, when the restaurant is too busy, starts using pre-made sauces instead of cooking from scratch. The food comes out quicker, but it tastes slightly worse.
• The Result: The AI performs better late at night or on weekends when the "traffic" is light, and slightly worse during the weekday rush.

3. The "Tide" and the "Moon" (Daily & Weekly Rhythms)

The researchers used a mathematical tool called Fourier Analysis (think of it as a "sound analyzer" for time) to find the pattern in the wavy data.

• The Daily Tide: The AI's performance goes up and down every 24 hours.
• The Weekly Moon: This daily rhythm changes depending on whether it's a Tuesday or a Saturday.
• The Interaction: It's not just "Day + Week." It's more like the tide changing based on the moon. The "weekday rush" makes the daily dip deeper, while the "weekend calm" makes the daily peak higher.

The study found that these time-based rhythms account for 20% of all the variation in the AI's answers. That is a huge chunk! It means if you test the AI on a Tuesday morning, you might get a "B" grade, but if you test it on a Sunday night, you might get an "A," even though the question and the AI's code are identical.

4. Why This Matters for Science

Imagine a scientist trying to measure the height of a plant. If they only measure the plant at 3:00 PM every day, they might think it's shorter than it actually is because plants droop in the afternoon heat.

• The Danger: If researchers only test AI during "rush hour" (bad performance times), they might conclude the AI is dumber than it really is. If they only test during "quiet hours," they might think it's a genius.
• The Reproducibility Crisis: If Scientist A tests the AI on a Monday and Scientist B tests it on a Friday, they will get different results. They might argue about who is right, when the real culprit is just when they asked the question.

5. The Takeaway: How to Fix It

The paper suggests that to get a fair test of an AI, we can't just ask it once. We need to treat it like a weather forecast:

• Sample the Whole Week: Don't just test on Monday. Test on Monday, Wednesday, and Sunday.
• Sample All Day: Don't just test at noon. Test at 6 AM, 2 PM, and 10 PM.
• Take the Average: Only by averaging out these "mood swings" can we find the AI's true, stable intelligence.

In a nutshell: The AI isn't a static machine; it's a dynamic system affected by the human world's busy schedule. If we want to trust AI research, we have to stop assuming the AI is the same at 9 AM as it is at 9 PM. We have to account for the "traffic jams" in its digital brain.

Technical Summary

1. Problem Statement

The research addresses a critical, often overlooked assumption in Large Language Model (LLM) research: time invariance.

• The Assumption: Most studies treating LLMs as research objects or tools assume that under fixed conditions (identical model snapshot, hyperparameters, and prompts), the model's average performance is stable over time.
• The Risk: If performance fluctuates systematically based on when a query is made, research findings regarding reliability, validity, and reproducibility are compromised.
• The Hypothesis: The authors hypothesize that LLM performance exhibits periodic variability driven by server load management strategies (e.g., load shedding, model compression) that respond to human usage patterns (daily and weekly cycles). This would manifest as a 24-hour rhythm modulated by a 7-day cycle.

2. Methodology

The study employed a longitudinal time-series design with rigorous spectral analysis.

• Model & Task:
  • Model: A specific snapshot of GPT-4o (gpt-4o-2024-08-06).
  • Task: A multiple-choice physics problem from the German Physics Olympiad (intermediate difficulty).
  • Scoring: An option-wise scoring scheme (0 to 1.0 in 0.25 increments) based on the correctness of selecting or rejecting specific answer choices.
• Data Collection Protocol:
  • Duration: Approximately 3 months (August 5, 2025 – October 31, 2025).
  • Frequency: Queries were issued every 3 hours.
  • Replication: 10 queries were sent at each time point to average out stochastic noise inherent in autoregressive generation.
  • Conditions: Fixed temperature ($T=1$), identical system/user prompts, and API access.
  • Total Data: $N = 6,930$ valid responses.
• Analytical Techniques:
  • Descriptive Statistics: Ordinary Least Squares (OLS) regression with Heteroskedasticity- and Autocorrelation-Consistent (HAC) standard errors to check for linear drift.
  • Spectral Analysis: Fast Fourier Transform (FFT) combined with Welch's method (using Hann windowing) to identify dominant periodic components in the time series.
  • Significance Testing: A non-parametric permutation test (1,000 shuffles) to establish a 95% significance threshold for spectral peaks.
  • Variance Decomposition: Calculating the proportion of total variance explained by significant periodic components.

3. Key Results

The study found substantial evidence rejecting the assumption of time invariance.

• No Linear Drift: There was no systematic long-term trend in performance ($p = 0.303$), confirming that fluctuations were cyclical rather than a degradation or improvement over time.
• Periodic Variability Detected: Fourier analysis revealed statistically significant peaks in the power spectrum that exceeded the permutation threshold.
  • Weekly Component: Peaks at ~5.5 days and ~7.3 days (likely a single weekly component affected by spectral leakage).
  • Daily Modulation (Sidebands): Instead of a single 24-hour peak, the analysis showed sidebands at ~21.0 hours and ~30.9 hours. This confirms the hypothesis that the daily rhythm is modulated by the weekly cycle (multiplicative interaction), rather than being a simple additive sum.
  • Sub-daily Harmonics: Peaks at ~9.6h and ~8.6h were observed, likely representing harmonics of the daily rhythm modulated by the weekly cycle.
• Magnitude of Effect:
  • Periodic components accounted for ~20.3% of the total variance in performance.
  • The peak-to-peak variation attributable to these cycles was 0.139 on a 0–1 scale, representing a ~14% fluctuation in the full performance range.
• Interaction Pattern: Heatmap analysis (Fig. 2) showed that the performance pattern across hours of the day varied significantly depending on the day of the week (e.g., weekday vs. weekend profiles differ), supporting the multiplicative modulation model.

4. Key Contributions

1. Empirical Validation of Temporal Instability: The study provides the first rigorous, longitudinal evidence that LLM performance is not time-invariant, even when model snapshots and prompts are fixed.
2. Identification of Mechanism: It links performance variability to server load management. The observed 24h/7d interaction suggests that providers adjust inference strategies (e.g., quantization, prompt pruning) based on global demand cycles, inadvertently altering output quality.
3. Methodological Framework: The paper establishes a protocol for detecting such variability using Fourier analysis and permutation testing, offering a template for future reproducibility checks in AI research.
4. Quantification of Uncertainty: It quantifies that temporal sampling choices can introduce ~14% systematic error, a magnitude that rivals or exceeds many other sources of noise in LLM evaluation.

5. Significance and Implications

The findings have profound implications for the reliability of AI research:

• Reproducibility Crisis: Studies collecting data in narrow time windows (e.g., only weekdays or only nights) may yield biased estimates that do not reflect the model's "true" average capability. Replicating a study at a different time of year or week could yield significantly different results.
• Research Tool Validity: When LLMs are used as tools (e.g., for qualitative coding, data extraction), temporal variability can introduce systematic biases into research outputs. If data collection is confined to a specific time window, the results may reflect the model's temporal state rather than the data's properties.
• Best Practices for Future Research:
  • Sampling Duration: Data collection should span at least one full week (or multiples thereof) to capture the longest periodicity.
  • Sampling Resolution: Measurements should be evenly spaced (ideally hourly) to capture higher-frequency structures.
  • Replication: Multiple repetitions per time point are necessary to mitigate stochastic noise, though the study notes that ~80% of variance remains unexplained by time, suggesting large sample sizes are still required.
  • Reporting: Researchers must report variability measures and propagate temporal uncertainty in downstream analyses.
• Systemic vs. Local: The authors suggest that locally hosted models (without shared server load) might offer greater temporal stability, potentially becoming the preferred standard for high-stakes reproducibility studies.

In conclusion, the paper argues that time is a critical experimental variable in LLM research that cannot be ignored. Ignoring daily and weekly periodicity threatens the validity of conclusions drawn from LLM benchmarks and AI-assisted research.