Modelling Language using Large Language Models

This paper argues that large language models serve as valuable scientific models of public languages as external social entities, defending their utility against claims of lacking linguistic insight and proposing a framework to interpret their internal mechanisms as model construals.

The Gist

Here is an explanation of the paper, broken down into simple concepts with some creative analogies.

The Big Question: Are AI Chatbots Just Parrots or Real Scientists?

Imagine you have a super-smart parrot. This parrot has read every book, tweet, and article on the internet. If you ask it a question, it answers perfectly. It knows grammar, facts, and how to tell a joke.

Now, a famous linguist (like Noam Chomsky) might say: "That's just a parrot! It's mimicking sounds. It doesn't actually understand the rules of language or how our brains work. It's just a fancy tape recorder."

But the author of this paper, Jumbly Grindrod, says: "Wait a minute! We are looking at this the wrong way. We shouldn't be asking if the parrot understands the human brain. We should be asking if the parrot is a perfect map of human language as a social thing."

Here is the breakdown of the argument:

---

1. Two Different Kinds of "Language"

To understand the paper, we need to split "language" into two buckets:

• Bucket A: The Internal Brain (I-Language). This is the secret, biological software inside your head. It's the invisible rules that make you know that "The cat sat on the mat" is good, but "Mat the on sat cat the" is nonsense. Chomsky thinks this is the only thing linguists should study.
• Bucket B: The Public Social Thing (E-Language). This is language as it exists out in the world. It's the messy, shared agreement between millions of people. It's the slang, the dialects, the grammar rules we all agree to use so we can talk to each other. It's like a giant, invisible web connecting everyone who speaks English.

The Paper's Argument:
Most people think AI (Large Language Models or LLMs) fails because it doesn't perfectly mimic Bucket A (the human brain). The author says, "Who cares? Let's stop trying to make AI a psychologist and start using it as a sociologist."

The author argues that AI is actually a fantastic tool for studying Bucket B (the public language).

---

2. The "Map vs. Territory" Analogy

The author says AI shouldn't be thought of as a "Theory" (a set of rules written in a textbook). Instead, it should be thought of as a Scientific Model.

• The Theory: A textbook that says, "Birds fly because of lift and drag."
• The Model: A wind tunnel with a plastic bird in it. You can't explain why the bird flies just by looking at the plastic, but you can watch the plastic bird fly and learn a lot about how real birds behave.

The Analogy:
Think of an LLM as a giant, digital simulation of a city.

• It's not a real city (it's not made of brick and mortar).
• It's not the people living there (it's not a human brain).
• But it is a perfect simulation of how traffic flows, where people shop, and how the streets connect.

If you want to study how traffic jams happen in a real city, you don't need to be a traffic cop inside every car. You can build a computer simulation (a model) that mimics the traffic patterns. Even if the simulation isn't "real," it teaches you valuable things about the real world.

The author says: LLMs are the traffic simulation for language. They aren't trying to be the human brain; they are trying to be a proxy for the English language itself.

---

3. The "Black Box" Problem (and How We Solve It)

The Objection:
Critics say, "But these AI models are 'Black Boxes.' We put text in, and text comes out, but we have no idea what's happening inside the machine. It's like a magic trick. If we can't see the gears, how can it be a scientific model?"

The Author's Reply:
Imagine you are trying to understand how a giant, complex clock works. You can't see the gears because they are covered by a metal case.

• Old way: You just guess how it works.
• New way (The Author's way): You use a special X-ray camera (called Explainable AI or XAI).

The paper points out that scientists are now developing "X-ray cameras" for AI. They can peek inside and see that when the AI processes a sentence, specific parts of its "brain" light up exactly when it's thinking about grammar, and other parts light up when it's thinking about meaning.

Even though we can't control every single gear, we can see enough to say, "Okay, this part of the model represents the rule of 'Subject-Verb Agreement'." This allows us to treat the AI as a valid scientific model.

---

4. The "Stuck in the Past" Objection

The Objection:
Critics say, "The AI is just a mirror. It only knows what it read in its training data. It's just a compressed version of the internet. It's not modeling 'Language'; it's modeling 'The Internet'."

The Author's Reply:
Imagine you are teaching a student to write a story.

• The Bad Student: Memorizes the book word-for-word. If you ask them to write a new story, they can't. They just recite the old one. This is overfitting (memorizing the data).
• The Good Student: Reads the book, understands the patterns of storytelling, and then writes a new story they've never seen before.

The author argues that modern AI is the Good Student.

• It is trained to predict the next word, but it is also tested on whether it understands logic, grammar, and nuance (using things like the GLUE benchmark).
• If the AI were just memorizing the internet, it would fail these tests.
• Because it passes these tests, it proves it has learned the underlying rules (the conventions) of the language, not just the specific words in the book.

It's like a chef who tastes a million recipes. They don't just memorize the recipes; they learn the principles of cooking (heat, salt, texture). Now, they can cook a dish they've never seen before. The AI has learned the "principles" of English.

---

The Bottom Line

The paper is an invitation to change our perspective.

• Don't ask: "Does this AI think like a human?" (The answer is probably no).
• Do ask: "Does this AI act like a perfect map of how humans use language together?" (The answer is yes).

By treating AI as a scientific model of public language, we can use it to discover new things about how our language works, how it changes, and how we communicate, without needing to solve the mystery of the human brain first. It turns the AI from a "parrot" into a powerful laboratory for language.

Technical Summary

Based on the paper "Modelling Language using Large Language Models" by Jumbly Grindrod, here is a detailed technical summary covering the problem, methodology, key contributions, results, and significance.

1. The Problem

The paper addresses a significant divide in the field of linguistics and philosophy of language regarding the utility of Large Language Models (LLMs).

• The Divide: There is a conflict between computational linguists who argue LLMs offer linguistic insights (Baroni, Piantadosi) and generative linguists/philosophers (Chomsky, Dupre) who argue they cannot.
• The Core Objection: Critics argue that LLMs are merely statistical predictors of next words based on training data, lacking the explanatory power of scientific theories. They claim LLMs model performance (surface patterns) rather than competence (internal cognitive rules), and that they are trained on data volumes and methods radically different from human language acquisition.
• The Ontological Conflict: The debate often hinges on the definition of "language." Generative linguistics focuses on I-language (Internal language: a cognitive/biological faculty), whereas critics argue LLMs cannot model this. The paper posits that the rejection of LLMs stems from a narrow focus on I-language, ignoring E-language (External language: language as a public, social entity).

2. Methodology

The author employs a philosophical and methodological framework rooted in the philosophy of science, specifically the theory of scientific modeling.

• Ontological Pluralism: The paper adopts a pluralist stance (following Santana and Stainton), arguing that language can be studied as a mental object (I-language), an abstract object, or a social object (E-language). The author focuses on E-language as a set of linguistic conventions adopted by a community.
• Model Construal Framework: Utilizing Michael Weisberg's (2007, 2013) framework, the author distinguishes between a scientific theory (which explains why phenomena occur) and a scientific model (a structure serving as a proxy for a real-world phenomenon).
  • The paper argues LLMs should not be viewed as theories of language but as models of E-languages.
  • It applies Weisberg's concept of construal, which involves:
    1. Assignment: Mapping model parts to target phenomenon parts.
    2. Scope: Defining what is captured and what is ignored.
    3. Fidelity Criteria: Determining dynamical (output similarity) and representational (internal structural similarity) accuracy.
• Addressing Opacity: The methodology addresses the "black box" nature of LLMs by integrating recent advances in Explainable AI (XAI) and probing classifiers. The author argues that while LLMs are opaque, link uncertainty can be reduced by correlating internal model features (e.g., attention heads) with known linguistic phenomena.

3. Key Contributions

The paper makes three primary theoretical contributions:

1. Reframing LLMs as Models of E-Language: It argues that LLMs are valuable not as theories of human cognition (I-language), but as scientific models of public, social languages (E-language). Since E-languages are defined by conventions and usage patterns across a community, LLMs trained on massive corpora are uniquely suited to capture these external regularities.
2. The "Model Construal" Strategy: It proposes a specific methodology for treating LLMs as scientific models. Instead of demanding LLMs explain human cognition, researchers should develop a construal that maps LLM internal structures (like self-attention mechanisms) to linguistic conventions (syntax, semantics) of the target language.
3. Refutation of the "Corpus Model" Objection: The paper counters the argument that LLMs are merely "models of their training data" (or compressed corpora). It argues that through regularization, validation sets, and evaluation on generalization benchmarks (like GLUE), LLMs are forced to learn underlying rules and conventions rather than simply memorizing data.

4. Results and Arguments

The author synthesizes existing computational linguistics research to support the modeling hypothesis:

• Predictive Success: Citing studies (Manning et al., Nair et al., Schrimpf et al.), the paper notes that LLMs successfully predict human judgments of semantic relatedness, syntactic dependencies, and neural responses to text.
• Internal Representation of Linguistics: Research using probing classifiers and attention visualization has shown that specific layers and attention heads in transformers encode syntactic dependencies and parts-of-speech categories. This provides the necessary "assignment" for a model construal.
• Iterative Refinement: The paper demonstrates that the construction of LLMs (e.g., the transition from BERT to RoBERTa) is driven by performance on linguistic comprehension tasks (GLUE/SuperGLUE), not just next-word prediction. This iterative process effectively refines the model's ability to capture E-language features.
• Generalization vs. Memorization: By analyzing training techniques (weight decay, validation sets), the paper concludes that LLMs generalize from patterns rather than overfitting to the training corpus, thereby capturing the "ultimate corpus" of a language's conventions.

5. Significance

• New Branch of Linguistic Inquiry: The paper opens a new avenue for linguistic research that treats language as an external, social phenomenon. It allows linguists to use LLMs to investigate the structure of public languages without needing to solve the harder problem of modeling human cognition.
• Bridging Disciplines: It provides a philosophical justification for the work already being done in computational linguistics, validating the use of LLMs as scientific tools rather than just engineering artifacts.
• Methodological Clarity: By defining the "construal" process, the paper offers a roadmap for how to rigorously interpret LLMs. It suggests that the "black box" problem is not a barrier to scientific modeling, provided that researchers use XAI techniques to establish the link between model internals and linguistic targets.
• Re-evaluating Chomskyan Critiques: It effectively neutralizes the strongest arguments against LLMs (which rely on the I-language framework) by shifting the ontological target to E-language, where LLMs are arguably the most powerful tools available.

In summary, Grindrod argues that LLMs are not failed theories of human mind, but highly successful scientific models of public language. By developing a rigorous construal that links model internals to linguistic conventions, the scientific community can leverage LLMs to gain unprecedented insights into the nature of language as a social entity.