Not Too Generative, Not Too Discriminative: The Human Alignment Sweet Spot

Technical Summary: "Not Too Generative, Not Too Discriminative: The Human Alignment Sweet Spot"

Problem Statement
A central debate in computational vision concerns whether human-like visual representations are best explained by discriminative learning (mapping inputs to labels) or generative learning (inferring inputs from an internal world model). Existing comparisons between these paradigms are confounded by simultaneous differences in architecture, model scale, and training data. Consequently, it remains unclear whether observed differences in human alignment stem from the learning objective itself or from correlated model properties. The paper posits that the dichotomy between generative and discriminative objectives may be the wrong axis for understanding human-aligned vision, suggesting instead that the optimal balance lies somewhere in between.

Methodology
To isolate the effect of the learning objective, the authors employ Joint Energy-Based Models (JEMs). JEMs define a single energy function over input-label pairs, $E_\theta(x, y)$, allowing a single network to simultaneously support discriminative prediction ($p(y|x)$) and generative density modeling ($p(x)$).

• Interpolation: The authors introduce a mixing coefficient $\alpha \in [0, 1]$ to control the training objective. The loss function is an interpolation between the discriminative log-likelihood ($L_D$) and the generative log-likelihood ($L_G$):
  $$\mathcal{L} = \alpha \cdot \frac{\|\nabla_\theta L_D\|}{\|\nabla_\theta L_G\|} \cdot L_G + (1 - \alpha) \cdot L_D$$
  Here, $\alpha=0$ corresponds to purely discriminative training, and $\alpha=1$ to purely generative training. The gradient norm ratio ensures both terms contribute at a comparable scale.
• Architecture and Training: The experiments utilize a Wide ResNet architecture. For ImageNet-scale experiments, training occurs in the latent space of a pretrained AutoencoderKL to make hybrid training computationally feasible. The models are trained across the full continuum of $\alpha$ (11 models from 0.0 to 1.0 in steps of 0.1).
• Evaluation: The resulting models are evaluated across six distinct human-alignment benchmarks spanning low-level perception, mid-level perception, uncertainty, robustness, cue conflict, and diagnostic feature attribution. These include:
  1. Perceptual Similarity (BAPPS): 2AFC and JND tasks.
  2. Gloss Perception: Correlation with human judgments on synthetic surfaces.
  3. Response Uncertainty (CIFAR-10H): Cross-entropy between model and human response distributions.
  4. Robustness (OOD): Error consistency with humans under distribution shifts.
  5. Shape–Texture Cue Conflict: Bias toward shape vs. texture in conflicting images.
  6. Diagnostic Features (ClickMe): Alignment of model attribution maps with human visual attention.

Key Results
Across all six benchmarks, human alignment is consistently maximized at intermediate values of $\alpha$, rather than at the extremes ($\alpha=0$ or $\alpha=1$).

• Perceptual Similarity: Hybrid models ($\alpha \approx 0.6$) achieve the highest 2AFC scores while matching discriminative baselines on JND tasks, outperforming both purely generative and purely discriminative models.
• Gloss Perception: While prior work suggested generative models (PixelVAE) outperformed discriminative ones (ResNet), the controlled JEM experiments show that the purely discriminative JEM already outperforms the purely generative JEM. The optimal alignment occurs at $\alpha \approx 0.5$, surpassing both the PixelVAE and ResNet baselines.
• Response Uncertainty: Hybrid models ($\alpha \approx 0.8$) best reproduce the full distribution of human responses, balancing categorical structure with sensitivity to ambiguity.
• Shape–Texture Bias: Shape bias increases monotonically with $\alpha$. Hybrid models achieve a shape bias near 0.7, significantly higher than standard discriminative baselines (e.g., ResNet-50) and approaching the levels of large generative models.
• Diagnostic Features: Alignment with human saliency maps peaks at $\alpha \approx 0.3$. Purely generative models lose selectivity, while purely discriminative models produce diffuse attribution maps.
• Test-Time Refinement: The authors demonstrate that applying Stochastic Gradient Langevin Dynamics (SGLD) to refine inputs at test time amplifies shape-consistent responses in hybrid and generative models without retraining, revealing the shape-favoring energy landscape induced by the generative objective.

Significance and Claims
The paper claims to reframe the generative–discriminative debate. The results suggest that the dichotomy is not the correct framework for understanding human-aligned vision. Instead, human-like representations emerge from balancing both objectives:

• Discriminative learning provides necessary categorical structure.
• Generative learning provides sensitivity to the input distribution and global coherence (e.g., shape).

The authors argue that human alignment is not a single scalar property but a family of constraints where the optimal balance of generative world-modeling versus discriminative labeling varies by task (e.g., diagnostic features favor a more discriminative regime, while response uncertainty favors a more generative one). The study concludes that human-aligned AI requires moving beyond the "which objective is best?" question toward understanding "how should these objectives be balanced?"

Limitations
The authors acknowledge that energy-based training remains computationally demanding and requires careful stabilization. The optimal $\alpha$ varies across benchmarks, indicating that human alignment is multidimensional. Furthermore, while hybrid objectives improve alignment with human perception and uncertainty, they do not inherently guarantee fairness, safety, or robustness, as they may also reproduce human errors and biases.