Technical Summary: Ethology of Latent Spaces

Problem Statement

This study challenges the prevailing assumption that latent spaces in Vision-Language Models (VLMs), specifically CLIP-type architectures, are neutral, homogeneous, or ontologically indeterminate. While these models operate on mathematical vector spaces where metric similarity is equated with semantic similarity, the paper argues that this equivalence is locally unstable and heavily dependent on training distributions. The central problem is the "blind spot" regarding the internal structuring modalities of these models: users often assume control over generation and interpretation, unaware that the latent space completes visual reality based on learned, non-intentional distributions. The paper posits that these spaces are not empty containers of meaning but are structured by specific "algorithmic sensitivities" and "discursive formations" inherited from their training data and architectural designs, leading to emergent political and cultural biases that are invisible to human agents.

Methodology

The research employs an "ethological approach," observing the behaviors of algorithmic processes rather than deducing their internal mechanisms, drawing on the framework of Machine Behaviour (Rahwan et al., 2019). The study utilizes a comparative analysis of three distinct VLMs applied to a corpus of 301 artworks spanning the 15th to the 20th centuries (including non-Western productions):

1. OpenAI CLIP Large (ViT-L/14): Trained on WIT, a curated proprietary dataset (~400M pairs).
2. OpenCLIP ViT-L/14: Trained on LAION-2B, a minimally filtered, web-scraped open dataset (~5.85B pairs).
3. SigLIP Large: Trained on WebLI, a proprietary Google dataset utilizing a pairwise learning architecture rather than contrastive learning.

Technical Protocol:

• Bipolar Semantic Axes: Inspired by vector analogies (Mikolov et al., 2013), the authors construct semantic axes by computing the vector difference between opposing text embeddings (e.g., "apolitical neutral" vs. "political engaged").
• Projection and Scoring: Artworks are projected onto these axes to measure their relative proximity to the poles, creating a gradient of salience rather than a binary classification.
• Dimensionality Reduction: T-SNE is used to visualize local neighborhoods and relative tensions within the high-dimensional latent space, acknowledging that global geometry is distorted but local asymmetries are preserved.
• Multi-Level Analysis: The analysis operates on three levels:
  1. Pixel Level: Luminance and chromatic fields.
  2. Object Level: Recognition of bodies, faces, and architecture.
  3. Semantic Level: Attribution of abstract concepts (e.g., political engagement, colonialism).

Key Contributions

The paper introduces three operational concepts to describe the behavior of VLMs:

1. Computational Latent Politicization: The process by which political categories emerge within the latent space not through intentional encoding, but as a structural effect of uneven discursive distributions in training data. The dataset functions as a "quasi-archive" (Foucault) that structures what becomes salient.
2. Emergent Bias: A third category of bias distinct from statistical bias (deviation from a reference distribution) and normative bias (deviation from an ethical norm). Emergent bias is irreducible to input distributions; it arises from the vectorial entanglements during the formation of the latent space and is only diagnosable through contrastive inter-model analysis.
3. Algorithmic Scopic Regimes: A typology characterizing distinct modes of structuring the visible:
   • Entropic (LAION): Characterized by accidental, stochastic politicization inherited from uncurated web data.
   • Institutional (OpenAI): A controlled regime reproducing Western museographic hierarchies and critical discourses.
   • Semiotic (SigLIP): A regime projecting contemporary critical vocabulary onto visual forms with high internal coherence but potential reductive systematicity.

Key Results

The comparative analysis reveals significant divergences in how models attribute cultural and political categories, demonstrating that "metric similarity" does not guarantee "semantic similarity" across models.

• Political Engagement:
  • SigLIP classified 59.4% of the artworks as politically engaged. Notably, African masks received the highest scores, suggesting an association between non-Western art and political intent.
  • OpenCLIP/LAION classified only 4% as politically engaged.
  • OpenAI CLIP classified 33.3%, associating politics primarily with avant-garde movements and institutional discourse.
• Aesthetic/Colonial Axis: The divergence was most extreme here, with a 72.6 percentage point gap between models. SigLIP classified only 9.2% of images toward the "colonial" pole, whereas OpenAI CLIP and OpenCLIP/LAION classified 69.6% and 81.8% respectively.
• Object Recognition (Body vs. Landscape):
  • SigLIP recognized only 6.9% of images as representing a human body, heavily favoring academic/naturalistic nudes (e.g., Degas) and invisibilizing modernist, deformed, or stylized bodies (e.g., Bacon, Schiele) and African masks.
  • OpenAI CLIP and OpenCLIP recognized 59.7% and 52.1% respectively, integrating transgressive and modernist representations.
• Luminance Sensitivity:
  • OpenAI CLIP classified 97.7% of artworks as "bright," confusing the concept of light (e.g., the word "neon" in captions) with physical luminance.
  • SigLIP correctly identified chiaroscuro masters (Rembrandt, Vermeer) as the darkest images, demonstrating a sensitivity closer to pixel-level ratios.

Significance and Claims

The paper argues that latent spaces are not neutral tools but active devices that structure the visible and the sayable. Its significance lies in:

• Critique of Interpretive Conditions: It establishes that digital art history cannot be constituted without interrogating the specific logic of the models employed. The "interpretative conditions of possibility" are inscribed within the representational spaces of the models themselves.
• Reframing Bias: By identifying "emergent bias," the study moves beyond simple dataset auditing. It demonstrates that biases can arise from the interaction of architecture, optimization, and data, producing political effects (ideologemes) that are not traceable to explicit intent or simple statistical underrepresentation.
• Methodological Shift: It calls for an ethological methodology that integrates the analysis of learning architectures into any delegation of cultural interpretation to algorithmic agents. The paper asserts that "looking at data with multimodal models means looking at multimodal models with data," necessitating a reciprocal critique of the conceptual space inherent to the tools.

The study concludes that while human-centered art history has long grappled with biases, a digital perspective must critically interrogate the foundational processes of model production, acknowledging that the visual culture of art is now mediated by algorithmic regimes of visibility that escape both human consciousness and the systems themselves.