Technical Summary: Rethinking Global Average Pooling

Problem Statement

Modern image classifiers predominantly rely on Global Average Pooling (GAP) followed by a linear classification head. While this architecture ensures translation invariance and reduces parameters, it collapses a spatial feature grid into a single image-level vector. This aggregation obscures the underlying spatial distribution of class evidence. Consequently, standard evaluation metrics (e.g., Top-1 accuracy) treat the model as a black box that outputs a single decision, failing to distinguish whether a correct prediction arises from the target object, surrounding context, or a mixture of both. Furthermore, when image-level predictions fail, it is unclear whether the target class evidence is entirely absent from the feature representation or merely diluted by the averaging process.

Methodology

The paper proposes a reframing of GAP-based classifiers through the lens of Multiple-Instance Learning (MIL). In this formulation, an image is treated as a "bag" of spatial instances (the feature vectors in the final grid), where supervision is provided only for the bag as a whole.

Core Mechanism: Dense Readout
The authors demonstrate that for a linear classification head $f(h) = \Theta^\top h + b$, the image-level logit $z$ is mathematically equivalent to the average of local logits computed pointwise on the spatial feature grid:
$$z = f(\bar{h}) = \frac{1}{N} \sum_{i=1}^{N} (\Theta^\top h_i + b)$$
By applying the trained classification head to each spatial feature vector before the global pooling step, the authors generate a dense readout (a $K \times H \times W$ class-score map). This process exposes the spatial class evidence that is otherwise averaged out.

Experimental Protocol
The study evaluates this dense readout across three distinct settings:

1. ImageNet-1K: Using ILSVRC bounding box annotations to project ground truth onto the spatial grid. The authors analyze foreground localization, background activation, and the recovery of "ReaL" labels (valid labels beyond the single ground truth) within the grid.
2. ImageNet-A: A natural adversarial benchmark where image-level accuracy collapses. The authors measure whether the target class remains detectable in the spatial grid despite the failure of the global prediction.
3. MS-COCO: Evaluating frozen self-supervised backbones (CLIP, DINO, MAE, BEiT) where only a linear head is trained on pooled embeddings. The head is then applied pointwise to the feature grid to test if spatial evidence emerges without end-to-end supervision.
4. Synthetic Multi-Object Dataset: A controlled experiment where images contain 3–5 objects, but training uses only a single image-level label. This isolates whether GAP-based models can learn to associate specific spatial instances with the label despite incomplete supervision.

Key Results

1. Recovery of Spatial Evidence
Across all architectures (ConvNeXt, Swin, MaxViT, ResNet), the dense readout consistently recovers the ground-truth class within foreground regions, even when the image-level prediction is incorrect.

• Foreground Detection: On ImageNet, foreground detection rates exceed 90% across architectures, significantly higher than Top-1 accuracy.
• ReaL Recovery: The set of classes predicted anywhere on the grid recovers valid "ReaL" labels at a rate 1.2 to 7.1 percentage points higher than the image-level Top-1 accuracy, indicating that valid evidence is often present but suppressed by averaging.

2. Architecture-Specific Dynamics

• Transformers (ViTs): Show high background activation (predicting the target class in background regions) on unmodified images (e.g., 90% for Swin/MaxViT). However, when the foreground object is occluded, this background activation drops sharply (36–38%). This suggests ViTs distribute object evidence globally via self-attention, making background tokens dependent on the presence of the foreground object.
• CNNs: Exhibit lower baseline background activation and a less sharp drop upon occlusion. However, for specific classes, occluding the foreground paradoxically increases background predictions, suggesting complex interactions where the presence of the object suppresses context-based shortcuts.

3. Robustness to Adversarial Shifts
On ImageNet-A, where image-level accuracy drops to as low as 3.8% (EfficientNet), spatial detection rates remain substantially higher (e.g., 30.6% to 56.9%). This indicates that localized class evidence persists in the feature grid even when global aggregation fails to produce a correct image-level decision.

4. Learning from Incomplete Supervision
In the synthetic multi-object experiment, models trained from random initialization with a single label per image (containing 3–5 objects) achieved ~26% image-level accuracy (matching the expected random guess rate for $n \in \{3,4,5\}$ objects). However, spatial localization accuracy was significantly higher (94% for ResNet-18, 63% for ViT-S/32), demonstrating that GAP-based classifiers can learn to assign correct class scores to specific spatial instances even under single-label supervision.

Significance and Claims

The paper argues that the limitations of standard image classifiers are not necessarily failures of representation learning, but rather failures of the aggregation rule (mean pooling).

• Diagnostic Tool: The dense readout serves as a post-hoc diagnostic that reveals spatial class evidence without requiring retraining, additional annotations, or architectural changes. It allows researchers to distinguish between object-based predictions and context-based shortcuts.
• MIL Interpretation: By viewing GAP-based classifiers as mean-aggregation MIL models, the paper reframes common failure modes (e.g., sensitivity to background context, failure on adversarial examples) as inherent limitations of mean aggregation when evidence is sparse or spatially competing.
• Architectural Implications: The results suggest that alternative aggregation rules (e.g., max pooling, attention-based MIL) could mitigate these issues. The paper posits that the "bottleneck" imposed by GAP obscures the true capabilities of the learned features.

The authors conclude that standard classifiers inherently retain spatial signals that are currently inaccessible to standard evaluation protocols, and that re-examining the aggregation step is crucial for understanding and improving model robustness and interpretability.