The Boiling Frog Threshold: Criticality and Blindness in World Model-Based Anomaly Detection Under Gradual Drift

This paper investigates world model-based anomaly detection under gradual observation drift, revealing a universal sharp detection threshold that depends on the interaction between detector sensitivity, noise floor, and environment-specific dynamics, while identifying critical failure modes such as the undetectability of sinusoidal drift and agent collapse prior to detection.

The Gist

Imagine you are driving a self-driving car. You have a "co-pilot" inside the computer (the World Model) that constantly predicts what the road ahead will look like. If the co-pilot says, "I expect a tree here," but the camera sees a wall, the car knows something is wrong.

This paper asks a scary question: What happens if the car's sensors don't break all at once, but slowly get foggy over time? Like a camera lens slowly gathering dust, or a GPS slowly drifting off course.

The researchers call this the "Boiling Frog Threshold." Just like a frog in slowly heating water might not jump out until it's too late, does the AI "wake up" to the problem, or does it keep driving until it crashes?

Here is the breakdown of their findings, using simple analogies:

1. The "Boiling Frog" Has a Hard Stop

The researchers found that the AI doesn't just slowly get confused. Instead, it behaves like a light switch.

• Below a certain point: The AI thinks, "Oh, the view is a little blurry, but that's just normal fog." It ignores the problem.
• Above a certain point: The AI suddenly screams, "ERROR! SOMETHING IS WRONG!" and wakes up.

This "switch" exists for every type of detector they tested. It's not a matter of if the AI wakes up, but at what exact temperature the water gets hot enough to trigger the alarm.

2. The "Sine Wave" Blindness (The Invisible Drift)

This is the most surprising finding. The researchers tried to trick the AI with two types of drift:

• Linear Drift: The sensor slowly gets worse and worse (like a camera lens getting dirtier every second). The AI eventually notices this.
• Sinusoidal Drift: The sensor wobbles back and forth perfectly (like a camera shaking left, then right, then left, then right, perfectly balancing out).

The Result: The AI is completely blind to the wiggling. Even if the camera is shaking violently, as long as it shakes back and forth equally, the AI's co-pilot thinks, "Ah, that's just normal vibration." It absorbs the shaking as part of the "background noise."

• Analogy: Imagine you are trying to hear a whisper in a noisy room. If someone whispers a steady "Hello," you hear it. But if they whisper "Hello" while the room noise goes "Up-Down-Up-Down" perfectly in sync with their voice, your brain cancels it out. The AI does the same thing; it "dreams through" the wiggles.

3. The "Crash Before Awakening" (The Fragile Robot)

In some environments, the AI is like a tightrope walker.

• The Scenario: The researchers tested a robot that balances on one leg (Hopper).
• The Problem: When the sensors started drifting, the robot fell over before the alarm could even ring.
• The Analogy: Imagine a tightrope walker who is already wobbling. If a gust of wind hits them, they fall instantly. If you have a security guard who takes 5 seconds to realize the wind is blowing, the walker is already on the ground.
• The Lesson: In fragile systems, the danger can be so immediate that the internal "alarm system" is too slow to save the day. The robot dies before it realizes it's sick.

4. It's Not Just About "How Smart" the AI Is

You might think, "If we make the AI's brain bigger or smarter, it will catch the drift earlier."

• The Finding: No. Whether the AI has a small brain or a giant super-brain, the "switch point" stays the same.
• Why? Because the AI compares the current view to what it expects. If the whole world gets a little blurrier, the AI's expectation also gets blurrier. It's like wearing glasses that get slightly foggy; you don't notice the fog because your vision of the world is foggy too. The AI only notices when the fog gets too thick compared to its own blurry expectations.

5. The "Three-Way Dance"

The paper concludes that the point where the AI wakes up isn't determined by just one thing. It's a dance between three partners:

1. The Noise Floor: How "messy" the world usually is (is it a calm lake or a stormy sea?).
2. The Detector: How sensitive the alarm is set to be (is it a sensitive smoke detector or a loud fire siren?).
3. The Environment: How the specific robot reacts to the mess (does a wobble make a car spin, or just a robot fall?).

Why Should You Care?

If you are building AI for real life (like self-driving cars or medical robots), this paper warns you:

• Watch out for "wiggling" attacks: Hackers could slowly wiggle your sensors back and forth to hide their tracks, and your AI won't see it.
• Don't trust the "average" error: Just because your AI is usually accurate doesn't mean it will catch a slow drift.
• Fragile things need external eyes: If your robot is unstable, you can't rely on its internal alarm. You need an outside observer to watch it, because it might crash before it realizes it's in trouble.

In short: AI has a "Boiling Frog" limit. It ignores slow, steady changes until they are too big, it completely ignores perfectly balanced wiggles, and sometimes it crashes before it even knows it's in danger.

Technical Summary

1. Problem Statement

Reinforcement Learning (RL) agents increasingly rely on world models (internal models of environment dynamics) for planning. While these models can detect abrupt environmental changes via prediction errors, real-world sensor degradation (e.g., camera fogging, LiDAR drift) is often gradual.

The core research question is: At what rate of gradual observation drift does an RL agent "wake up" (detect the anomaly), and what determines this boundary? The authors investigate whether agents can detect slow perceptual corruption and identify the factors that define the transition from "oblivion" (drift absorbed as noise) to "awareness" (detection).

2. Methodology

The study employs a systematic ablation approach across four dimensions:

• Environments: Four MuJoCo-v5 locomotion tasks with varying dynamics and fragility:
  • HalfCheetah, Ant: Stable, multi-limb dynamics.
  • Hopper: Fragile, single-leg dynamics.
  • Walker2d: Moderate stability.
• World Models: A simple 3-layer MLP trained to predict the next state ($s_{t+1}$) given current state ($s_t$) and action ($a_t$). Three model capacities were tested (Small: 128 hidden units, Medium: 512, Large: 1024).
• Drift Injection:
  • Linear Drift: Monotonically increasing corruption ($g = \epsilon \cdot t$).
  • Sinusoidal Drift: Periodic, zero-mean corruption ($g = \epsilon \cdot \sin(\dots)$).
  • Drift is applied to velocity-related observation dimensions starting at step 300.
• Detector Families: Three distinct anomaly detection mechanisms were tested to rule out detector-specific artifacts:
  1. Doubt Index (DI): Z-score based on Exponential Moving Average (EMA) of prediction errors.
  2. Variance Detector: Monitors the variance (2nd moment) of errors in a sliding window.
  3. Percentile Detector: Flags errors exceeding a specific percentile of the baseline distribution (no temporal smoothing).
• Protocol: 10 seeds $\times$ 8 episodes per condition. Detection rates were measured across 16 drift intensities ($\epsilon$).

3. Key Contributions & Findings

A. The Existence of a Sharp Detection Threshold ($\epsilon^*$)

• Universal Sigmoid Shape: Across all environments, detector families, and model capacities, the detection rate follows a sharp sigmoid transition from ~0% to ~100% as drift intensity increases.
• Position Variability: While the shape of the threshold is universal, the position ($\epsilon^*$) varies. It is determined by a three-way interaction:
  1. Noise Floor Structure: The shape of the prediction error (PE) distribution (e.g., tail heaviness), not just the baseline Mean Squared Error (MSE).
  2. Detector Sensitivity: Hyperparameters (e.g., Z-score threshold, window size).
  3. Environment Dynamics: How the environment's prediction error responds to drift ($\partial PE / \partial \epsilon$).
• Power Law Relationship: Within a single environment, $\epsilon^*$ follows a power law with detector parameters ($R^2 = 0.89–0.97$). However, a global model fails to predict $\epsilon^*$ across environments ($R^2 = 0.45$), proving that environment-specific dynamics are the missing variable.

B. Sinusoidal Blindness (A World Model Property)

• Total Blindness: All detector families are completely blind to sinusoidal (periodic) drift, even at high intensities where the instantaneous perturbation exceeds the noise floor.
• Mechanism: This is not a detector artifact (proven by the lack of temporal smoothing in the Percentile detector). Instead, it is a property of the world model. The model treats zero-mean periodic oscillations as "normal variation" to optimize model evidence (minimize free energy). The prediction error signal itself contains no cumulative drift information for periodic perturbations.

C. Collapse Before Awareness (CBA)

• The Phenomenon: In fragile environments (specifically Hopper), the agent's policy physically collapses (the robot falls) before any detector can accumulate sufficient evidence to trigger.
• Implication: There exists a "lethal but invisible" regime where drift is strong enough to cause catastrophic failure but too subtle or rapid for internal monitors to react in time. This creates a fundamental unmonitorable failure mode for safety-critical agents.

D. Capacity Independence

• Increasing model capacity (from 128 to 1024 units) reduces absolute baseline noise (MSE) but does not shift the detection threshold ($\epsilon^*$).
• Reason: Detectors use relative metrics (e.g., Z-scores). As model accuracy improves, both the signal (drift) and noise (baseline variance) scale down proportionally, leaving the signal-to-noise ratio (and thus the threshold) invariant.

4. Significance and Implications

• Reframing Self-Monitoring: The paper moves the understanding of detection boundaries from a simple "emergent property of the model" to a structured interaction between the noise floor, the detector, and the environment.
• Predictive Processing Alignment: The findings align with the Free Energy Principle. The world model "dreams through" periodic perturbations (absorbing them as noise) to maintain model evidence, explaining the sinusoidal blindness.
• Practical Guidelines for Deployment:
  1. Adversarial Vulnerability: Adversaries can exploit sinusoidal/zero-mean drift patterns to bypass PE-based monitors.
  2. Fragility Risk: For fragile agents, internal monitoring is insufficient; external supervision is required to catch "Collapse Before Awareness" scenarios.
  3. Metric Selection: Baseline MSE is a poor predictor of detection capability. Practitioners must characterize the environment's specific response to drift ($\partial PE / \partial \epsilon$) and the tail structure of the error distribution.
• Limitations of Classical Methods: Traditional change-point detection methods (e.g., CUSUM) fail in this context because they assume stationary noise, whereas world model prediction errors are inherently non-stationary.

Conclusion

The paper establishes that while world models possess a universal ability to detect gradual drift via a sharp sigmoid threshold, this ability is bounded by the specific dynamics of the environment and the nature of the drift. Crucially, periodic drift is fundamentally undetectable by these systems, and fragile agents face a "blind spot" where failure occurs before detection, necessitating new approaches for safety in non-stationary RL environments.