Context Sensitivity Improves Human-Machine Visual Alignment

1. Problem Statement

Modern machine learning models, particularly Vision Transformers (ViTs) and other foundation models, represent inputs as fixed points in a high-dimensional embedding space. While effective for generalization tasks (e.g., ImageNet classification), this paradigm assumes a one-to-one mapping between entities and representations, ignoring the role of context.

In contrast, human perception is inherently context-sensitive. Humans dynamically adjust their representation of objects based on surrounding cues, goals, and sequential dependencies. This gap leads to a misalignment between machine representations and human similarity judgments. Current models are increasingly used as "autoraters" (proxies for human data), making it critical to develop representations that adapt to context to better reflect human cognition.

2. Methodology

The authors propose a Context-Sensitive Similarity (CSS) framework that modifies standard neural network embeddings to account for a contextual reference image.

A. Task Formulation

The study utilizes the ImageNet-HSJ dataset (Roads & Love, 2021), originally an 8-choose-2 similarity task. The authors restructured this data into a Triplet-with-Context task:

• Input: A context image ($c$) and a triplet of images $\{p, q, r\}$.
• Goal: Predict which image in the triplet is the "odd-one-out" relative to the context.
• Mechanism: The model must determine that two items are similar given the context, leaving the third as the outlier.

B. Model Architecture

The proposed model (Fig. 2 in the paper) consists of three main components:

1. Feature Extraction: A Foundation Model (FM) extracts initial embeddings for the context and triplet images. The authors tested three FMs: ViT (supervised), SigLIP (contrastive), and DINOv2 (self-supervised), including their "human-aligned" variants.
2. Context-Insensitive Transform (CIT): An affine transformation ($Wx + b$) followed by normalization is applied to the triplet embeddings. This aligns the base representation with human judgments before context is applied.
3. Context-Sensitive Similarity (CSS):
   • A neural network (single linear layer) maps the context embedding ($c$) to a low-rank matrix $B_c \in \mathbb{R}^{r \times d}$.
   • This generates a context-dependent symmetric positive semidefinite matrix $A_c = B_c^\top B_c$.
   • The similarity between two triplet images $i$ and $j$ conditioned on context $c$ is computed as:
     $$s_{i,j|c} = \tilde{x}_i^\top A_c \tilde{x}_j$$
   • This effectively re-weights the embedding space dynamically based on the context, rather than using a static dot product.

C. Training Objective

The model is trained via Maximum Likelihood Estimation to predict the human "odd-one-out" choice. The loss function includes:

• Cross-entropy loss for the odd-one-out prediction.
• Identity Regularization: Two terms ($\lambda_1$ and $\lambda_2$) penalize deviations from the identity matrix for the transform $W$ and the context matrix $A_c$. This ensures the model retains the pre-trained representational structure while learning context-specific modulations.

3. Key Contributions

1. Context-Aware Representation: Introduction of a method to compute similarity that dynamically shifts based on a reference image, moving beyond fixed embedding spaces.
2. Empirical Validation: Demonstration that context sensitivity significantly improves the alignment between machine predictions and human judgments on a large-scale behavioral dataset.
3. Generalizability: The approach is shown to be effective across different pretraining objectives (supervised, contrastive, self-supervised) and works in conjunction with existing "human-aligned" fine-tuning methods (e.g., AligNet).
4. Interpretability: The method provides a mechanism to visualize how context reshapes the representational space (e.g., via PCA and Representational Similarity Matrices).

4. Experimental Results

Quantitative Performance

The authors evaluated the models on a held-out test set using Odd-One-Out Accuracy.

• Baseline: Standard Foundation Models (FM).
• Improvement 1: Adding the Context-Insensitive Transform (CIT) improved accuracy over the baseline.
• Improvement 2: Adding the Context-Sensitive Similarity (CSS) yielded the largest gains.
• Key Statistic: The context-sensitive model achieved up to a 15% absolute improvement in accuracy over the context-insensitive model.
  • Example (DINOv2 Original): FM (53.6%) $\to$ CI (59.0%) $\to$ CS (65.7%).
  • Example (SigLIP Original): FM (47.1%) $\to$ CI (58.2%) $\to$ CS (68.2%).
• Consistency: These gains were consistent across both original and human-aligned versions of the models. Bootstrap analyses confirmed all improvements were statistically significant ($p < 0.05$).

Qualitative Analysis

• Representational Similarity Matrices (RSMs): The context-sensitive model successfully modulated similarities. In a specific example (Context: Deer; Triplet: Dog, Canoe, Horse-cart), the context-insensitive model grouped the Canoe and Horse-cart (both vehicles) as similar. The context-sensitive model, recognizing the "animal" theme of the deer, upweighted the similarity between the Dog and Horse-cart, correctly identifying the Canoe as the odd-one-out.
• PCA Visualizations: The PCA plots showed that the context-sensitive model dynamically shifted the embedding of the "Horse-cart" from the "Vehicle" cluster toward the "Animal" cluster when the context was an animal, mirroring human cognitive flexibility.

5. Significance and Discussion

• Bridging the Gap: The work demonstrates that incorporating cognitive principles (context sensitivity) into deep learning architectures significantly improves human-machine alignment.
• Beyond Static Embeddings: It challenges the "Platonic" view of fixed representations, showing that dynamic, context-dependent re-weighting is necessary for tasks requiring human-like reasoning.
• Practical Implications: As AI models are increasingly used to generate synthetic data or act as proxies for human feedback (e.g., in RLHF), improving their ability to simulate human context-dependent judgment is crucial for safety and reliability.
• Limitations & Future Work: The current study relies on a dataset with single-participant judgments per trial (introducing noise) and focuses on symmetric comparisons. Future work aims to incorporate sequential dependencies, internal goals, and asymmetric comparisons (Tversky's features).

In conclusion, this paper provides a robust technical framework for context-sensitive visual alignment, proving that even small, cognitively inspired architectural changes can yield substantial improvements in modeling human perception.