VTBench: A Multimodal Framework for Time-Series Classification with Chart-Based Representations

1. Problem Statement

Time-series classification (TSC) has seen significant advancements through deep learning, yet most state-of-the-art models rely exclusively on raw numerical inputs. While alternative representations exist (e.g., Gramian Angular Fields, Recurrence Plots), they often require heavy preprocessing and lack human interpretability. Conversely, chart-based visualizations (line, bar, scatter, area plots) offer intuitive, human-aligned representations that leverage powerful computer vision backbones.

However, the effectiveness of chart-based representations remains underexplored due to:

• A lack of systematic evaluation across different chart types and visual encoding choices.
• Uncertainty regarding whether charts should replace raw data or complement it.
• Unknown impacts of data characteristics (sequence length, dataset size) on visual encoding utility.

2. Methodology: The VTBench Framework

The authors introduce VTBench, a modular, extensible framework designed to systematically evaluate chart-based and multimodal TSC.

A. Visual Encoding Pipeline

VTBench converts univariate time-series instances into four standard chart types using matplotlib:

1. Line Charts: Emphasize continuous trajectories.
2. Area Charts: Filled line plots emphasizing magnitude.
3. Bar Charts: Discretize the signal into bins.
4. Scatter Plots: Highlight local outliers and sparsity.

Visual Configurations: The framework systematically varies:

• Color Mode: RGB vs. Grayscale.
• Annotations: Presence vs. absence of axes, titles, and tick marks.
• Resolution: 64×64, 128×128, and 256×256 pixels.

B. Architecture

VTBench employs a multimodal fusion architecture consisting of:

1. Vision Encoders: Dedicated Convolutional Neural Networks (CNNs) process chart images. Two variants are tested:
   • ShallowCNN: 3-layer lightweight encoder (~286K params).
   • DeepCNN: 5-layer VGG-style encoder (~1.2M params).
2. Numerical Encoder: Processes raw time-series data. Options include:
   • Fully Connected Network (FCN).
   • Transformer Encoder.
   • OS-CNN: A state-of-the-art (SOTA) convolutional TSC model used as a strong baseline.
3. Fusion Strategies:
   • Concatenation: Simple merging of feature vectors.
   • Dynamic Weighted Fusion: An attention-based mechanism that learns to assign importance weights to different modalities (charts and raw data) to handle redundancy.
4. Classification Head: A fully connected layer with ReLU and dropout for final prediction.

C. Configurations

The framework supports three experimental modes:

• Single-Chart: Unimodal classification using only one chart type.
• Multi-Chart: Fusing multiple chart types (visual-only).
• Multimodal: Fusing chart views with the raw numerical sequence.

3. Key Contributions

1. First Systematic Benchmark: VTBench is the first framework to evaluate chart-based TSC across 31 diverse UCR datasets, covering four chart types, various visual settings, and fusion strategies.
2. Modular Extensibility: The architecture allows for "plug-and-play" experimentation with new chart types, encoders (e.g., Vision Transformers), and modalities.
3. Empirical Guidelines: The study distills practical rules for when and how to use visual representations, moving beyond anecdotal evidence to data-driven insights.

4. Experimental Results & Findings

Experiments were conducted on 31 UCR datasets (binary and multiclass) against SOTA baselines (HC2, InceptionTime, TimesNet, GPT4TS, SoftShape).

A. Performance Overview

• Overall Accuracy: The best VTBench configuration achieved 78.0% average accuracy, which is competitive but generally lower than specialized SOTA models (e.g., SoftShape at 93.3%, HC2 at 86.7%).
• Contextual Strength: VTBench excels in specific settings, particularly on smaller datasets and binary classification tasks, where chart-only models can be highly effective.

B. Key Findings

1. Chart Type Effectiveness:

   • Line and Area charts generally outperform Bar and Scatter plots, as they preserve the continuous temporal structure of the data.
   • Bar and Scatter plots disrupt temporal continuity, often leading to lower accuracy unless the data is inherently discrete.
   • Visual Design: Removing axis labels (reducing clutter) often improves performance, while color encoding helps distinguish features in complex tasks.

2. Complementarity vs. Redundancy:

   • Multimodal Fusion: Combining charts with raw numerical data generally yields the best results, as charts capture shape/pattern motifs while raw data preserves precise timing.
   • Redundancy Risk: In motion-oriented datasets (e.g., human activity, robotics) and multiclass tasks, adding charts can introduce noise or redundancy, degrading performance compared to raw-data-only models.
   • Binary vs. Multiclass: Multimodal fusion is more beneficial for binary tasks; multiclass tasks are more sensitive to redundant visual cues.

3. Data Characteristics:

   • Dataset Size: On small datasets, chart-only models are surprisingly competitive due to the inductive bias of CNNs. However, multimodal models risk overfitting on small data.
   • Sequence Length: For short sequences (<200 timesteps), 128×128 resolution is optimal. For very long sequences, high-resolution charts suffer from "overplotting," compressing critical temporal variations.

4. Fusion Strategy:

   • Dynamic Weighted Fusion (attention-based) outperformed simple concatenation, allowing the model to down-weight uninformative modalities (e.g., ignoring a noisy chart in favor of raw data).

5. Significance and Impact

• Interpretability: VTBench bridges the gap between "black-box" numerical models and human-readable visual analysis, making TSC more interpretable for domains like healthcare and finance.
• Leveraging Vision Ecosystem: By converting time series to images, the framework enables the reuse of pre-trained computer vision models (e.g., ResNet, ViT), accelerating development and transfer learning.
• Design Principles: The paper provides a "rulebook" for practitioners:
  • Use Line/Area charts for continuous signals.
  • Use Multimodal fusion (Raw + Chart) for robustness, but employ attention mechanisms to prevent redundancy.
  • Avoid complex visual encodings for motion-based or multiclass problems unless the visual cues are distinct.
• Foundation for Future Work: VTBench establishes a unified foundation for exploring adaptive chart generation, domain-specific visual encodings, and automated modality selection.

In conclusion, VTBench demonstrates that while chart-based representations may not universally outperform specialized numerical models, they offer a powerful, interpretable, and complementary approach that significantly enhances TSC when applied with the correct visual encoding and fusion strategy.