Directional Confusions Reveal Divergent Inductive Biases Through Rate-Distortion Geometry in Human and Machine Vision

1. Problem Statement

While modern Artificial Neural Networks (ANNs) and humans often achieve similar categorical classification accuracy, they frequently make systematically different types of errors. Traditional evaluation metrics (e.g., accuracy, top-k error) treat confusion as unstructured and symmetric, focusing on how often a system errs rather than who is mistaken for whom and in which direction.

The paper addresses the following core questions:

• Do humans and ANNs exhibit fundamentally different directional confusion structures (asymmetries) when matched for task, perturbation, and accuracy?
• How does the organization of these asymmetries (e.g., broad vs. sparse, weak vs. strong) relate to the system's generalization geometry?
• Can Rate-Distortion (RD) theory provide a compact, interpretable signature of these inductive biases that is invisible to accuracy alone?

2. Methodology and Framework

2.1 Data and Systems

• Stimuli: The study uses the Generalization (GEN) repository, featuring natural images from ImageNet ($K=16$ categories) subjected to 12 distinct perturbation families (e.g., color/grayscale, contrast, phase noise, rotation, Eidolon variants).
• Subjects:
  • Humans: $\sim$83,000 psychophysics trials.
  • Models: Baseline CNNs (GoogLeNet, ResNet-152, VGG-19) and robustness-trained variants (Distortion-trained, Specialized single-distortion, All-noise/multi-corruption).
• Matching: Analyses are performed on "blocks" defined by unique combinations of experiment, condition, and model instance to ensure fair comparison.

2.2 Rate-Distortion (RD) Framework

The authors model each system as a noisy communication channel defined by its stimulus-response confusion matrix. They apply an information-theoretic RD framework to quantify the trade-off between mutual information (preserved signal) and distortion (categorical error).

• Inference: Using Maximum-A-Posteriori (MAP) estimation, they infer a latent distortion matrix $\rho$ from the empirical confusion matrix.
• RD Frontier Tracing: By varying an inverse-temperature parameter $\lambda$, they trace the optimal RD frontier using Blahut-Arimoto updates.
• Geometric Signatures: Three metrics summarize the RD frontier:
  1. Slope ($\beta$): The marginal information cost of reducing error.
  2. Curvature ($\kappa$): The non-uniformity of the trade-off across operating points.
  3. Efficiency (AUC): The area under the RD curve, summarizing overall information-error efficiency.

2.3 Asymmetry Metrics

To capture directional bias, the authors decompose confusion asymmetry into two dimensions:

1. Breadth ($f_{pairs}$): The fraction of class pairs exhibiting significant directional asymmetry.
2. Strength ($\bar{\Delta}$): The mean magnitude of deviation among those asymmetric pairs.

• Global Metric: Normalized Frobenius norm of the off-diagonal asymmetry ($A_{off}^F$).

2.4 Mechanistic Simulations

To validate causality, the authors simulate systems with tunable distortion matrices containing two types of antisymmetry:

• Broad-Weak: Dense, weak biases distributed across many class pairs.
• Sink-Like: Concentrated, strong biases funneling probability mass into a few dominant "sink" classes.
  These simulations test whether these structural differences produce divergent RD geometries independent of overall accuracy.

3. Key Results

3.1 Dissociation of Breadth vs. Strength

• Humans: Exhibit broad but weak asymmetries. A high fraction of class pairs show small directional biases (distributed structure).
• ANNs: Exhibit sparse but strong asymmetries. Fewer pairs are asymmetric, but the magnitude of error in those pairs is significantly larger (concentrated "sink-like" collapses).
• Accuracy Independence: This structural dissociation persists even when models and humans are matched for accuracy. Groups with significantly higher global asymmetry than humans (e.g., baseline CNNs) show no significant difference in accuracy, while groups matching human asymmetry levels are often significantly more accurate. This confirms a double dissociation between asymmetry structure and performance.

3.2 Link to RD Geometry

• Efficiency (AUC): The organization of asymmetry predicts RD efficiency beyond global magnitude.
  • In Distortion-trained models, higher asymmetry correlates with lower efficiency, but this is largely driven by accuracy.
  • Crucially, when controlling for accuracy, Specialized models show a significant negative coupling between asymmetry and slope ($\beta$), while humans show no reliable coupling.
• Curvature ($\kappa$): Apparent correlations between asymmetry and curvature in robustness-trained models disappear under accuracy control, suggesting curvature primarily tracks performance rather than directional structure.
• Key Finding: The breadth-strength profile is the critical predictor. Broad-weak structures (human-like) and sink-like structures (ANN-like) occupy different regions of the RD space even with identical accuracy.

3.3 Mechanistic Simulation Insights

• Opposite Effects: Increasing antisymmetry strength has opposite effects on RD geometry depending on the organization:
  • Broad-Weak: Expands the RD frontier (increases efficiency).
  • Sink-Like: Collapses the RD frontier (decreases efficiency).
• Accuracy Control: These effects persist even after regressing out the linear effect of accuracy, proving that the organization of errors, not just the amount, dictates generalization geometry.
• Recovery: The simulation confirms that the "breadth-strength" decomposition is necessary to explain residual variations in RD efficiency, whereas a single global magnitude metric is insufficient.

3.4 Robustness Training Limits

Robustness training (e.g., training on noise mixtures) successfully reduces the global magnitude of asymmetry toward the human range. However, it fails to recover the human-like breadth-strength profile. Models remain prone to sparse, strong collapses rather than developing broad, graded similarities.

4. Significance and Contributions

1. New Diagnostic Signature: The paper establishes directional confusion structure (specifically the breadth-strength decomposition) as a compact, interpretable signature of inductive bias that is invisible to standard accuracy metrics.
2. Theoretical Unification: It bridges cognitive science (asymmetric similarity judgments) and machine learning (generalization geometry) using Rate-Distortion theory. It demonstrates that human-like representations rely on graded similarity manifolds, while ANNs rely on collapsed manifolds with dominant attractor states.
3. Limitations of Scalar Metrics: The study proves that matching accuracy (or even global asymmetry magnitude) is insufficient for achieving human-like robustness. Systems can fail for qualitatively different reasons despite identical performance scores.
4. Practical Implications: The authors propose that future robustness training should target the redistribution of error structure. Instead of just minimizing error, training objectives could penalize "sink-like" collapses and encourage "broad-weak" distributions to align models with human inductive biases.
5. Generalizability: The findings hold across diverse architectures, including modern Vision Transformers and big transfer models, suggesting this is a fundamental property of artificial vs. biological vision under distribution shift.

In summary, the paper argues that how a system confuses categories is more informative about its underlying representational geometry and inductive biases than how often it is wrong. By analyzing the geometry of these confusions through an RD lens, researchers can better diagnose and potentially correct the structural misalignments between human and machine vision.