Can Multimodal LLMs Perform Time Series Anomaly Detection?

1. Problem Statement

Time Series Anomaly Detection (TSAD) is critical for web-scale systems (e.g., AIOps, system monitoring). While Large Language Models (LLMs) have shown promise in TSAD by treating numerical data as text, existing approaches often fail to capture the human-like process of detecting anomalies, which typically involves visualization combined with textual reasoning.

Current research gaps include:

• Oversimplification: Treating point-wise anomalies as special cases of range-wise ones or aggregating points to approximate ranges.
• Limited Scope: Ignoring complex real-world scenarios like multi-granular anomalies (point, range, and variate-wise) and irregular time series (missing data, non-uniform sampling).
• Modality Limitations: Lack of investigation into whether Multimodal LLMs (MLLMs), which process both text and images, can outperform text-only LLMs or traditional methods in TSAD.

2. Methodology

A. VisualTimeAnomaly Benchmark

The authors introduce VisualTimeAnomaly, a comprehensive benchmark designed to evaluate the zero-shot capabilities of MLLMs.

• Data Construction: Numerical time series data is converted into Time Series Images (TSIs).
• Anomaly Granularity: The benchmark covers:
  • Point-wise: Global and contextual anomalies.
  • Range-wise: Seasonal, trend, and shapelet anomalies.
  • Variate-wise: Anomalies in specific variables within multivariate series.
  • Irregularity: Simulated by randomly dropping data points (up to 25% missing) to test robustness.
• Models Evaluated: Proprietary (GPT-4o, Gemini-1.5) and Open-source (LLaVA-NeXT, Qwen2-VL) MLLMs, tested in Text, Vision, and Hybrid (Text+Vision) modalities.

B. TSAD-Agents Framework

Based on empirical findings, the authors propose TSAD-Agents, a multi-agent system designed to automate TSAD without prior assumptions about anomaly types. The framework consists of four collaborative agents:

1. Scanning Agent: Uses a Chain-of-Scanning (CoS) technique to hierarchically reason about the data:
   • Step 1: Is the series regular or irregular?
   • Step 2: Is the anomaly point-wise or range-wise?
2. Planning Agent: Generates a tailored execution plan based on the scan results. It dynamically selects the best modality and tool:
   • Point/Irregular-Point: Uses Traditional Methods (Text modality).
   • Range/Irregular-Range: Uses MLLMs (Image modality).
3. Detection Agent: Executes the plan by calling tools from a pool (e.g., iForest, OCSVM, or MLLMs) to generate initial predictions.
4. Checking Agent: Performs self-reflection. It visualizes the prediction on the time series image, evaluates if the prediction is too broad/narrow, and refines the output to reduce false positives/negatives.

• Shared Dynamic Memory: Ensures all agents share context (e.g., detected anomaly type) for coherent decision-making.

3. Key Findings & Results

Key Insights from VisualTimeAnomaly

1. Complementarity of Methods:
   • MLLMs excel at coarse-grained anomalies (Range-wise and Variate-wise) where F1 scores reach ~63%.
   • Traditional Methods (e.g., iForest) excel at fine-grained anomalies (Point-wise) where MLLMs struggle (F1 < 5%) due to limited numerical reasoning.
2. Resilience to Irregularity:
   • MLLMs are highly robust to irregular time series (up to 25% missing data), maintaining stable performance.
   • Traditional methods degrade significantly (e.g., ~47% F1 drop) because they require imputation, which distorts structural features. MLLMs treat missing data as blank regions in images, preserving the overall trend.
3. Modality Shift:
   • Text Input: Focuses on quantitative variations (good for point anomalies) but suffers from high hallucination rates and context length limits.
   • Image Input: Focuses on qualitative patterns (good for range/variate anomalies) and significantly reduces hallucinations (e.g., LLaVA-72B hallucinations dropped from 98.3 to 21.7).
   • Hybrid (Text+Image): Best for point anomalies; Image alone is superior for range/variate anomalies.

Performance of TSAD-Agents

• State-of-the-Art: TSAD-Agents outperforms all baselines (including classical methods like iForest and recent LLM-based approaches like LLMAD and TAMA) across all anomaly types.
• Adaptive Success: By dynamically switching between traditional tools for point anomalies and MLLMs for range anomalies, the framework achieves an F1 score of 65.4% on point anomalies and 38.6% on range anomalies, significantly beating single-method approaches.
• Ablation Study: Removing any component (Scanning, Planning, Checking, or CoS) leads to significant performance drops, confirming the necessity of the multi-agent synergy.

4. Key Contributions

1. VisualTimeAnomaly Benchmark: The first benchmark to systematically evaluate MLLMs on multi-granular (point, range, variate) and irregular time series anomalies.
2. Empirical Insights: Discovery that MLLMs and traditional methods are complementary, and that visualizing time series shifts the model's focus from quantitative to qualitative patterns while reducing hallucinations.
3. TSAD-Agents Framework: The first multi-agent system for multimodal TSAD that leverages reasoning, planning, tool-use, and self-reflection to achieve automatic, high-accuracy detection without manual hyperparameter tuning or anomaly type assumptions.

5. Significance

This work bridges the gap between human intuition (visualizing data) and machine learning. It demonstrates that Multimodal LLMs are not just a replacement for traditional methods but a powerful complement. By combining the precision of classical algorithms for fine-grained tasks with the pattern recognition and robustness of MLLMs for coarse-grained and irregular tasks, the proposed framework offers a robust, automated solution for real-world AIOps and system monitoring where data is often noisy, irregular, and multi-faceted.