💻 computer science

EVA: Bridging Performance and Human Alignment in Hard-Attention Vision Models for Image Classification

The paper introduces EVA, a neuroscience-inspired hard-attention framework that explicitly balances classification accuracy with human-like scanpath alignment through variance control and adaptive gating, achieving superior interpretability on datasets like CIFAR-10 and COCO-Search18 without requiring gaze supervision.

Original authors: Pengcheng Pan, Yonekura Shogo, Kuniyoshi Yasuo

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pengcheng Pan, Yonekura Shogo, Kuniyoshi Yasuo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery in a dark room. You have a flashlight, but the battery is weak, so you can only shine it on one small spot at a time. You have to decide where to look next based on what you just saw.

This is exactly how human vision works. We don't see the whole world in high definition all at once. Instead, our eyes dart around (saccades), focusing on interesting details (fixations) to build a picture in our brains.

Now, imagine building a robot to do the same thing. For a long time, scientists tried to make robots that were just super accurate at identifying objects (like "That's a dog!"). But they did it by looking at the entire picture at once, like a super-powered camera.

The problem? When these robots get really good at guessing the right answer, they stop looking like humans. They might guess "Dog" correctly, but they might be staring at the dog's tail or the background grass, not the dog's face. They are "black boxes"—they get the right answer, but we don't know how or why they looked where they did.

Enter EVA: The "Smart Detective"

The paper introduces a new AI model called EVA. Think of EVA not as a camera, but as a detective with a flashlight.

EVA is designed to solve two problems at once:

Be accurate: It needs to guess the right object.
Be human-like: It needs to look at the object the way a human would (e.g., looking at the dog's face, not the grass).

Usually, there is a trade-off. If you make the detective smarter (better at guessing), they stop looking around and just guess based on a tiny clue. If you make them look around more like a human, they might make more mistakes. The authors call this the "Alignment Tax"—the price you pay for being human-like.

How EVA Works (The Secret Sauce)

EVA is built with three special "brain parts" inspired by how our nervous system works, but simplified for a computer:

The Fovea and Periphery (The Flashlight and the Blur):
Just like your eyes, EVA has a sharp center (fovea) and a blurry edge (periphery). It only sees a tiny, sharp patch of the image at a time, plus a blurry view of the surroundings. This forces it to move its "flashlight" to see the whole picture.
The "Uncertainty Meter" (Variance Control):
Imagine you are looking for your keys. If you are very sure you saw them on the table, you look closely and steadily. If you are unsure ("Did I leave them in the car?"), you start looking around wildly, checking every corner.
EVA has a built-in "uncertainty meter." When it's confused, it makes its "flashlight" jump around more (exploring). When it's confident, it steadies its gaze. This keeps it looking like a curious human rather than a rigid machine.
The "Gatekeeper" (Adaptive Gating):
Imagine you are talking to a friend who is gathering clues. Sometimes, your friend brings you a new clue that changes your mind completely. Sometimes, they bring a clue that doesn't matter.
EVA has a "gatekeeper" that decides how much weight to give new information. If the new glimpse is important, the gate opens wide. If it's just noise, the gate stays closed. This helps EVA balance between learning new things and sticking to what it already knows.

The Results: Why It Matters

The researchers tested EVA on pictures of cats, dogs, cars, and more. Here is what they found:

It gets the job done: EVA is just as good at guessing the right object as the "super-cameras" that look at the whole image.
It looks like us: When you watch EVA's "eyes" move across a picture, it traces a path very similar to how a human would look. It looks at the face of the dog, the wheels of the car, etc.
No cheating: Usually, to make a robot look like a human, you have to show it thousands of videos of humans looking at things (gaze supervision). EVA learned to look like a human without ever seeing a human eye movement. It figured it out just by trying to get the right answer.

The Big Picture

Why do we care?

In the future, we might use AI to help doctors diagnose diseases or self-driving cars to navigate traffic. We need to trust these AI systems.

If an AI says, "There is a tumor here," but we can't see why it thinks that, we can't trust it. But if the AI is like EVA, we can watch its "flashlight" move. We can see, "Oh, it looked at the suspicious spot, then zoomed in, then looked at the surrounding tissue. That makes sense!"

EVA bridges the gap. It proves that we don't have to choose between a smart AI and a human-like AI. By designing the AI to "look" like a human, we get a system that is not only accurate but also interpretable, trustworthy, and safe.

1. Problem Statement

The paper addresses a critical tension in computer vision known as the "alignment tax."

The Conflict: Optimizing vision models purely for classification accuracy often leads to a degradation in "human-likeness." As models become more accurate (often using strong, parallel feature extractors like CNNs or ViTs), their internal decision processes become less interpretable. Specifically, scanpaths (the sequence of eye fixations or "glimpses" a model takes) become less similar to human visual behavior.
The Limitation of Current Methods:
- Post-hoc explanations (e.g., Grad-CAM, saliency maps) are often brittle and do not reflect the actual reasoning process.
- Hard-attention models (which select discrete glimpses) often suffer from "policy collapse" when paired with strong feature extractors; the model becomes so accurate that it stops needing to sample informative sequential evidence, leading to non-human-like, random, or static fixation patterns.
- Gaze-supervised models (trained on human eye-tracking data) can mimic human scanpaths but often fail to maintain high classification accuracy or require expensive gaze supervision.
The Goal: To design a hard-attention model that achieves high classification accuracy without gaze supervision, while maintaining scanpaths that align with human visual behavior, treating the performance-human-likeness trade-off as a tunable design variable rather than an inevitable cost.

2. Methodology: The EVA Architecture

The authors introduce EVA (Explicit Vision Alignment), a neuroscience-inspired hard-attention testbed. EVA is trained solely on standard classification labels (no gaze supervision) but incorporates specific architectural mechanisms to regulate how evidence is acquired over time.

Core Components

Minimal Fovea-Periphery Representation:
- Unlike complex multi-scale designs, EVA uses a simple two-level view: a high-resolution foveal crop ( $x^f_t$ ) and a low-resolution peripheral context ( $x^p_t$ ).
- A CNN-based feature extractor processes the foveal input to ensure high classification accuracy.
- A lightweight encoder processes the peripheral view and current fixation location.
Two-Level Recurrent Backbone:
- Lower RNN ( $h^1_t$ ): Controls the sampling policy. It integrates the glimpse representation and predicts the next fixation location.
- Upper RNN ( $h^2_t$ ): Integrates sequential evidence for the final classification decision.
- This separation allows the model to decouple where to look (sampling control) from what the image is (evidence integration).
Variance Control via Prediction Error (Neuromodulation):
- Inspired by biological uncertainty modulation, EVA adapts the variance ( $\sigma_t$ ) of the fixation sampling distribution based on prediction error.
- It computes long-term and short-term moving averages of the prediction error. The discrepancy between these averages serves as an uncertainty signal.
- Mechanism: High uncertainty $\rightarrow$ High variance (encourages exploration); Low uncertainty $\rightarrow$ Low variance (stabilizes sampling). This prevents the model from collapsing into a static policy.
Adaptive Gating:
- An adaptive gate ( $\beta_t$ ) regulates the flow of information from the lower sampling state ( $h^1_t$ ) to the upper classification state ( $h^2_t$ ).
- The gate is modulated by bottom-up signals, top-down feedback, and the current sampling variance.
- Function: It controls how much newly acquired evidence updates the classifier's belief, effectively balancing the trade-off between aggressive accuracy-seeking and human-like sequential exploration.

Training Objective

Loss Function: A combination of Cross-Entropy (classification loss) and REINFORCE (policy gradient loss for fixation selection).
No Gaze Supervision: The model learns the attention policy purely from the classification reward (correct/incorrect).
Uncertainty Proxy: During training, the prediction error is derived from ground-truth labels. At inference, a self-error signal (change in predictions between steps) is used.

3. Key Contributions

Formalization of the Alignment Tax: The paper empirically demonstrates that improving representation strength (e.g., stronger CNNs) systematically degrades scanpath human-likeness in hard-attention models.
EVA Framework: Introduction of a mechanistic testbed that makes the performance-human-likeness trade-off explicit and adjustable through architectural constraints (variance control and gating) without changing the training objective.
Gaze-Supervision-Free Alignment: EVA achieves human-aligned scanpaths on standard classification tasks without ever seeing human gaze data during training.
Cross-Task Generalization: The learned attention policy transfers effectively to new tasks (e.g., search tasks) and datasets without fine-tuning.

4. Experimental Results

Datasets and Benchmarks

CIFAR-10: Used for rigorous mechanistic ablation with dense human gaze annotations (Gaze-CIFAR-10).
ImageNet-100: Used to test scalability on high-resolution natural images.
COCO-Search18: Used for cross-task evaluation (trained on COCO classification, tested on COCO search behavior) without gaze supervision.

Key Findings

Accuracy vs. Alignment:
- On CIFAR-10, EVA achieves 79.77% accuracy, outperforming other hard-attention baselines (RAM, DRAM, MRAM) which hover around 60-65%.
- Crucially, EVA achieves a Center-Debiased Gaze Consistency Score (GCS) of 0.064, significantly outperforming baselines (which are often negative or near zero) and approaching the performance of gaze-supervised models like Gazeformer, despite having no gaze supervision.
Ablation Studies:
- CNN Feature Extractor: Essential for accuracy. Removing it drops accuracy to ~62%.
- Variance Control & Gating: Essential for human-likeness. Removing these improves accuracy slightly but causes a massive drop in GCS (scanpath alignment).
- Synergy: The combination allows EVA to maintain high accuracy while recovering human-like trajectories.
Scalability: On ImageNet-100, EVA-Mobile (using a pretrained backbone) achieves 71.92% Top-1 accuracy, demonstrating the framework scales to complex, high-resolution data.
Cross-Task Transfer: When trained on COCO classification and evaluated on COCO-Search18 (a search task), EVA produces human-like search scanpaths without any search-specific training or gaze labels. This suggests the learned policy is a generalizable strategy for evidence acquisition.
Mechanistic Interpretability: PCA analysis of hidden states shows that the lower layer controls sampling dynamics, while the upper layer integrates category-level evidence. Qualitative analysis shows EVA revises its hypotheses (e.g., from "ship" to "airplane") as new evidence is gathered, mirroring human reasoning.

5. Significance and Conclusion

Trustworthy AI: EVA provides a "process-level" interpretability. Users can inspect the sequence of glimpses the model took to reach a decision, verifying that the model looked at relevant object parts (e.g., wheels, faces) rather than relying on spurious correlations.
Design Principle: The paper establishes that mechanism-aware design (incorporating variance control and adaptive gating) can resolve the alignment tax. It proves that human-like behavior is not a byproduct of gaze supervision but can emerge from task-driven objectives if the architecture supports sequential evidence acquisition.
Future Impact: EVA offers a blueprint for building active vision systems that are both high-performing and transparent, suitable for high-stakes applications where understanding why a model made a decision is as important as the decision itself.

In summary, EVA bridges the gap between "black box" accuracy and "white box" interpretability by introducing neuro-inspired control mechanisms that regulate how a model explores an image, ensuring that the path to the answer remains human-readable and aligned with human visual behavior.