💻 computer science

VEIL: How Visual Encoding Hijacking Induces Bias In Vision Models

The paper VEIL demonstrates that chart-based time-series classification models often rely on encoding-specific visual cues rather than underlying temporal patterns, revealing that visualization design choices fundamentally shape learned representations and should be treated as a measurement problem.

Original authors: Suranjana Sooraj, Xuyang Chen, Madhumitha Venkatesan, Dongyu Liu

Published 2026-07-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Suranjana Sooraj, Xuyang Chen, Madhumitha Venkatesan, Dongyu Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to recognize different types of music by showing it pictures of the sound waves. You could draw the waves as a line, fill the space under the line with color (area), turn the waves into vertical bars, or scatter dots across the page.

The paper "VEIL" asks a simple but tricky question: Is the robot actually learning the music, or is it just learning to recognize the style of the drawing?

Here is the breakdown of their findings using everyday analogies:

1. The Problem: "Visual Encoding Hijacking"

Think of the robot like a student taking a test.

The Ideal: The student reads the story (the time-series data) and understands the plot.
The Reality (Hijacking): The student realizes that if the story is drawn with red lines, the answer is "A," but if it's drawn with blue bars, the answer is "B." The student isn't reading the story; they are just memorizing the font style.

The authors call this "Visual Encoding Hijacking." The robot gets so good at recognizing the shape of the chart (like the thickness of a line or the density of dots) that it ignores the actual data underneath. It's like a dog learning to sit only when you hold a treat in your left hand, not because it understands the command "sit."

2. The Experiment: The "Translation" Test

To prove this, the researchers acted like language teachers.

They taught the robot using Line charts.
Then, they tested it on Bar charts without retraining it.
The Result: In many cases, the robot failed miserably. It was like teaching someone to speak French, then handing them a German dictionary and expecting them to understand. The robot had learned "Line-ese," not the universal language of the data.

However, for some simpler datasets, the robot could translate between styles. This suggests that for easy problems, the robot learns the real signal. For complex problems, it relies entirely on the visual "tricks" of the specific chart type.

3. The Detective Work: "Where are you looking?"

The researchers used a special tool (called Grad-CAM) that acts like a heat map on the robot's eyes. It shows exactly which part of the image the robot is staring at to make a decision.

Good behavior: The robot looks at the shape of the wave (the actual data).
Bad behavior (Hijacking): The robot looks at the edges of the bars, the grid lines, or the density of the dots. It's ignoring the story and focusing on the frame of the picture.

4. The "Magic Glasses" (HINT)

The researchers tried to fix this by putting "magic glasses" on the robot. They covered up the parts of the image the robot was obsessing over (like the bar edges) and forced it to look elsewhere.

Did it work? Sometimes, yes! For some difficult datasets, forcing the robot to look at the "real" data improved its score significantly (sometimes by huge margins).
Did it always work? No. For some datasets, covering up the "tricks" actually made the robot worse. This means that for some problems, those visual tricks were actually helpful clues, and removing them confused the robot.

5. The Big Takeaway

The main lesson of this paper is that how you draw the data matters just as much as the data itself.

It's not just a tool: Choosing between a line chart or a bar chart isn't just an aesthetic choice; it changes what the robot learns.
High scores aren't enough: Just because a robot gets 99% accuracy doesn't mean it understands the data. It might just be a master of recognizing chart styles.
The "Hijack": If you change the chart style, the robot's "brain" (its internal representation) completely changes. It's not a universal learner; it's a style-specific learner.

In short: If you want a robot to truly understand time-series data, you can't just throw any chart at it. You have to be careful that the robot isn't just memorizing the font, the colors, or the grid lines, but is actually reading the story the data is telling.

Technical Summary: VEIL – Visual Encoding Hijacking in Vision Models

Problem Statement

The paper addresses a critical gap in Time-Series Classification (TSC) where numerical sequences are rendered as chart images (e.g., line, bar, scatter plots) to leverage powerful Convolutional Neural Network (CNN) backbones. While this approach has achieved competitive performance, it remains unclear whether models learn the underlying temporal dynamics of the data or rely on visual encoding-specific cues introduced by the chart design (e.g., stroke continuity, edge boundaries, point density).

The authors term this phenomenon Visual Encoding Hijacking: a behavior where learned representations align more closely with the rendering style than the temporal class structure. This creates an "illusion of learning" where models exploit superficial visual artifacts rather than genuine signal properties. The core research questions are:

Do different visual encodings produce consistent or divergent representations of the same time series?
Are features learned from one encoding transferable to others, or are they encoding-specific?
What visual evidence do CNN-based TSC models rely on, and can attention guidance mitigate encoding-sensitive behavior?

Methodology: The VEIL Framework

The authors introduce VEIL (Visual Encoding Illusion in Learning), a diagnostic framework designed to systematically evaluate how chart encodings influence learned representations. The study evaluates four standard encodings (line, area, bar, scatter) across 31 datasets from the UCR benchmark.

The methodology employs a multi-faceted analysis pipeline:

Experimental Setup: All models use an identical CNN backbone and training protocol (Adam optimizer, early stopping) on standard UCR splits to ensure performance differences arise solely from encoding choices, not optimization.
Representation Similarity (CKA): Centered Kernel Alignment is used to measure the alignment between feature spaces learned from different chart types.
Cross-Chart Transferability: Linear probing (logistic regression) is performed on features extracted from one encoding type to classify data from another. This measures the lower bound of shared task-relevant information.
Attribution Analysis (Grad-CAM): Gradient-weighted Class Activation Mapping is applied to visualize model focus, distinguishing between reliance on signal structure, encoding-specific features, or irrelevant elements (e.g., axes).
Geometric and Separability Analysis: PCA and UMAP are used to visualize feature geometry, intrinsic dimensionality, and class separation patterns across encodings.
Encoding Sensitivity Index (ESI): Scalar metrics ( $ES_{CKA}$ and $ESI_{Probe}$ ) are derived to quantify the degree of encoding dependence at the dataset level.
Perturbation Stress Tests: Controlled perturbations (Gaussian blur, bar merging, alpha fading) are applied to test model sensitivity to specific visual elements.
HINT Attention Guidance: To test mitigation, the authors apply HINT (Human Importance-aware Network Tuning), masking salient Grad-CAM regions during training to force the model to attend to other temporal regions.

Key Results

1. Representational Divergence and Encoding Sensitivity

The study finds that encoding effects are bimodal and heavily dependent on dataset complexity:

Simple Datasets: Binary-class datasets (e.g., Wine) show high CKA scores ( $\approx 0.89–0.98$ ) and high cross-chart transfer, indicating encoding-invariant representations.
Complex Datasets: Multi-class datasets (e.g., FordB, Yoga) exhibit substantial divergence. CKA scores drop significantly ( $\approx 0.44–0.69$ ), and cross-chart transfer is weak. Models trained on these datasets appear to specialize in the visual patterns of their specific training encoding.
Correlates of Sensitivity: Encoding sensitivity is more strongly associated with class count than sequence length. Multi-class datasets show elevated sensitivity even with shorter sequences, while binary datasets often remain robust despite varying lengths.

2. Model Behavior and Perturbations

Inductive Bias: No single encoding consistently dominates across all datasets. Instead, chart types act as distinct inductive priors.
Perturbation Impact: Targeted perturbations cause uneven accuracy drops. Bar charts remain comparatively robust to fine-grained disruptions, suggesting a reliance on coarse structural patterns, whereas other encodings are more sensitive to specific visual cues.
Attribution Shifts: Grad-CAM analysis reveals that models often focus on encoding-specific features (e.g., bar edges, scatter density) rather than pure temporal trends, though this does not definitively prove these cues are "shortcuts" rather than legitimate signal reflections.

3. Effectiveness of HINT Attention Guidance

The application of HINT-based attention guidance yields mixed results, confirming it is not a universal fix:

Success Cases: On encoding-divergent datasets (low CKA, high ESI), HINT significantly improved performance (e.g., ToeSegmentation2 +38.46%, Ham +23.81%).
Failure Cases: On encoding-invariant datasets or where representations were already effective, HINT either provided negligible benefit or degraded performance (e.g., Computers -36.00%, SonyAIBORobotSurface1 -14.14%).
Conclusion: Attention guidance reduces encoding-sensitive behavior only when supported by convergent evidence across multiple diagnostics (CKA, transfer, UMAP). It should be treated as a diagnostic intervention rather than a reliable mitigation strategy.

Contributions and Significance

The paper makes three primary contributions:

Formalization of Visual Encoding Hijacking: The authors define and distinguish "hijacking" (encoding-dependent behavior undermining alignment) from legitimate encoding-specific evidence that faithfully reflects signal properties.
The VEIL Diagnostic Framework: A systematic approach combining representation similarity, transferability, and attribution to evaluate encoding effects, moving beyond simple accuracy metrics.
Empirical Evidence of Inductive Bias: A study across 31 datasets demonstrating that encoding choice is a critical inductive bias in image-based TSC, causing substantial variation in representation alignment and model attention.

Significance:
The paper reframes chart-based TSC not merely as a modeling decision but as a representation and measurement problem. It extends the concept of graphical perception from human readers to vision models, showing that visualization design choices fundamentally shape what models learn. The findings suggest that high accuracy in TSC does not necessarily imply semantic understanding of temporal dynamics; models may be exploiting rendering artifacts. Consequently, the authors advocate for encoding-aware evaluation protocols to ensure model reasoning aligns with human-interpretable structures.