💬 NLP

What Makes Two Language Models Think Alike?

This paper introduces a novel method for mapping neural activity to interpretable linguistic features to reveal that similarities between 43 diverse language models are primarily driven by their release date and model family rather than by scale or architectural class.

Original authors: Louis Jalouzot, Christophe Pallier, Emmanuel Chemla, Yair Lakretz

Published 2026-06-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Louis Jalouzot, Christophe Pallier, Emmanuel Chemla, Yair Lakretz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a room full of different chefs. Some are trained in French cuisine, others in Japanese, and some are self-taught. They all have the same job: predict the next ingredient in a recipe.

The big question the researchers asked is: Do these chefs actually think about ingredients in the same way?

For example, when a chef sees a sentence like "The cat chased the mouse," do they focus on the fact that it's a story about animals (syntax), or do they just focus on the specific words "cat" and "mouse" (vocabulary)?

The Old Way: Measuring "Shape"

Previously, scientists tried to compare these chefs (AI models) by looking at the "shape" of their internal thoughts. Imagine taking a photo of how each chef arranges their ingredients on a counter. If the photos look geometrically similar, the scientists assumed the chefs were thinking alike.

But this had a problem: It was like a black box. It could tell you that two chefs arranged their ingredients similarly, but it couldn't tell you why. Maybe they both grouped things by color, or maybe they both grouped things by weight. The old method couldn't explain the "why."

The New Way: The "Linguistic Signature"

This paper introduces a new, simpler way to look inside the chefs' minds. Instead of just looking at the shape of the arrangement, they ask: "How much does each specific rule matter to this chef?"

They created a tool called a Metric-Learning Encoding Model (MLEM). Think of this as a translator that converts the chef's messy, complex internal thoughts into a simple report card of Linguistic Features.

This report card lists things like:

Tense: Is the sentence about the past or future?
Subject: Is the sentence about one person or many?
Sentence Structure: Is it a simple sentence or a complex one with a "relative clause" (like "The man who is wearing a hat...")?

The tool measures how much the chef's internal thoughts change when these rules change. If the chef's thoughts jump wildly when you switch from past to future tense, that feature has a high score on their report card. This report card is called a "Linguistic Signature."

The Experiment: 43 Chefs, 10 Families

The researchers tested 43 different AI models (the "chefs"). These models came from different "families" (like the Llama family, the GPT family, or the Mamba family) and had different sizes (from tiny to huge) and different architectures (different ways of building the brain).

They compared the Linguistic Signatures of every model against every other model.

The Big Discovery: It's About "Family" and "Era," Not Size

The results were surprising. The researchers found that what makes two models think alike is not how big they are, but who their parents are and when they were born.

Family Matters Most: Models from the same family (e.g., all the "Llama" models) had very similar signatures, even if one was tiny and the other was huge. They were raised with the same "training recipes" and architectural blueprints, so they learned to prioritize the same linguistic rules.
Release Date Matters: The date a model was released was the strongest predictor of similarity. This is like saying, "Chefs trained in 2024 think more alike than chefs trained in 2020." This is because the whole industry changes its techniques together over time (newer activation functions, better training data, etc.).
Size Doesn't Dictate Style: Simply making a model bigger (adding more parameters) didn't make it suddenly start thinking like a completely different type of model. A small model and a giant model from the same family still shared the same "linguistic priorities."

The "Loop" of Thought

The researchers also watched how these signatures changed as the information moved through the layers of the model (like a message passing through a relay race).

They found that models from the same family followed the same path. They started with a certain focus, moved to a peak of complexity in the middle layers, and then circled back to a simpler focus at the end. It's like a loop. This suggests that even though the models are different sizes, they all solve the problem of language in the same "dance steps."

The Conclusion

In short, this paper argues that we shouldn't just look at how "big" or "complex" an AI is to understand how it thinks. Instead, we should look at its Linguistic Signature—a profile of what rules it cares about most.

The study shows that architecture and scale are less important than the "family" and the "generation" of the model. Two models can be built differently and be different sizes, but if they come from the same family and era, they will likely prioritize the same linguistic features, making them "think alike."

Technical Summary: What Makes Two Language Models Think Alike?

Problem Statement

Traditional methods for comparing Large Language Models (LLMs) rely on geometric similarity metrics (e.g., Representational Similarity Analysis, Canonical Correlation Analysis, or linear regression) to determine if two models share similar representational spaces. While these methods can confirm that representations align, they function as "black boxes" that fail to explain why they align. Specifically, they cannot distinguish whether similarity arises from shared explicit linguistic strategies (e.g., encoding the same syntactic structures) or superficial lexical biases.

This paper addresses the question of whether language models converge at the algorithmic level of Marr's hierarchy (the level of representations and operations) despite divergent implementational choices (architecture and training). It seeks to identify which specific linguistic features drive the similarities and differences between models, moving beyond raw geometry to feature-based interpretability.

Methodology: Linguistic Signature Similarity

The authors propose a novel approach using Metric-Learning Encoding Models (MLEMs) to map neural activity onto interpretable linguistic features.

Metric-Learning Encoding Models (MLEMs):
- Instead of predicting features from neural activity (decoding) or fitting univariate encoding models to individual neurons, MLEMs learn a distance function over a set of interpretable stimulus features (e.g., tense, subject number, relative clause type).
- The model learns a symmetric positive definite matrix $W$ that weights individual features and their interactions.
- The objective is to maximize the Spearman correlation between the pairwise distances of neural embeddings ( $D_N$ ) and the pairwise distances of feature vectors ( $D_F$ ) weighted by $W$ . This preserves the relative topology of stimuli rather than absolute scales.
Linguistic Signatures (Feature Importance Profiles):
- For each layer of a language model, the authors compute Feature Importances (FIs). These are defined as the average decrease in the MLEM's Spearman correlation score when a specific feature's distance matrix is randomly permuted.
- This FI vector constitutes the layer's linguistic signature, quantifying how much each linguistic feature drives the organization of neural distances in that layer.
- A model's overall signature is defined as the profile of these layer-wise FIs.
Similarity Measurement:
- Layer-Level: Signatures of two specific layers are compared using Euclidean distance (L2).
- Model-Level: To compare entire models with varying numbers of layers, the authors use Multi-dimensional Dynamic Time Warping (DTW). This aligns feature-importance profiles along the depth axis, allowing for the comparison of models with different layer counts or shifted feature emergence points.
- Predictive Modeling: A second MLEM is trained on the resulting model-level similarity matrix to determine which model properties (e.g., architecture, scale, release date) best predict linguistic signature similarity.

Experimental Setup

Models: 43 autoregressive language models from 10 families, covering 100M to 14B parameters.
Architectures: Decoder Transformers (e.g., GPT-2, Llama-3.2, OLMo-2), State-Space Models (Mamba, Mamba-2), and Recurrent Neural Networks (RWKV-7).
Dataset: A synthetic dataset of 7.7k sentences generated via Context-Free Grammars (Relative Clause dataset), balanced to mitigate correlations between 12 linguistic features (including syntactic properties, attachment sites, and lexical frequencies).
Evaluation: Hidden states of the last token were extracted for all models.

Key Results

1. Layer-wise Dynamics

Encoding Performance: MLEM performance (Spearman correlation) follows a consistent profile across models: low in shallow and deep layers, peaking in middle layers (0.5–0.7). This aligns with prior findings that syntactic information is primarily processed in middle layers.
Feature Evolution: Syntactic features (Relative Clause type, Attachment site) show peak importance in middle layers.
Family Trajectories: Models within the same family trace remarkably similar trajectories in a shared PCA space of linguistic signatures, even when they differ significantly in parameter count and depth. Some families exhibit "loop-like" trajectories where final layers return close to initial layer signatures, potentially reflecting constraints imposed by the next-token prediction objective.

2. Drivers of Model Similarity

Primary Drivers: The similarity between model linguistic signatures is driven most strongly by release date and model family.
- Release Date acts as a proxy for general LLM development, summarizing correlated changes in architecture, normalization, positional encoding, and training data.
- Model Family indicates that linguistic signatures are shaped by specific architectural lineages rather than just scale.
Secondary/Minor Drivers: Parameter count, training token count, and depth make smaller contributions.
Architecture Class: Once family and release date are accounted for, the broad architecture class (Transformer vs. SSM vs. RNN) contributes little to explaining similarity.

3. Convergence vs. Divergence

The results support a partial version of the Platonic Representation Hypothesis. While different architectures can share representational profiles, they do not collapse to a single endpoint as scale increases. Instead, convergence is structured by model family and generation.
Feature-level analysis confirms that similar signatures correspond to actual neural space organization regarding the leading features (validated via Multi-Dimensional Scaling), whereas dissimilar signatures show distinct clustering patterns.

Significance and Claims

The paper claims to provide a method to link theoretically motivated symbolic descriptions directly to neural representations. By shifting from raw geometric comparisons to feature-based similarity, the approach:

Explains Alignment: It moves beyond stating that models are similar to explaining why (i.e., they prioritize the same linguistic variables).
Challenges Scale/Architecture Primacy: It suggests that the "way" language models solve language processing is shaped more by model family and generation (release date) than by scale or specific architectural classes.
Generalizability: The authors posit that this feature-based comparison framework can be extended to other domains (speech, vision) and other neural systems (biological brains) whenever stimuli can be paired with explicit theoretical features.

The work does not claim to solve the interpretability problem entirely but offers a specific, interpretable metric for comparing the "linguistic priorities" of different neural systems.