💬 NLP

Measuring Form and Function in Language Models

This paper introduces the Contextual Alternative Choice (CAC) metric to quantitatively evaluate language models on the formal syntactic and functional discourse properties of English determiners, revealing that while some very large models can match human children's performance on both benchmarks, no current model trained on comparable data achieves this simultaneously.

Original authors: Héctor Javier Vázquez Martínez, Charles Yang

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Héctor Javier Vázquez Martínez, Charles Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to speak like a human. You have two ways to check if it's working:

The Grammar Test: Does it say "The cat" instead of "Cat the"? (Correct form)
The Conversation Test: Does it know when to say "The cat" versus "A cat" based on what the other person just said? (Correct function)

This paper is about building a better test for Language Models (LLMs) by looking at how real children learn to talk. The authors, researchers from the University of Pennsylvania, argue that we've been testing AI too much like a grammar quiz and not enough like a real conversation.

Here is the breakdown of their work using simple analogies:

1. The Problem: The "Multiple Choice" Trap

Most AI tests today are like multiple-choice exams. You show the AI two sentences:

"The dog barked."
"Dog the barked."
And ask, "Which one is right?"

The AI usually gets this right. But the authors say this is like testing a child by asking them to point to a picture of a dog versus a cat. It doesn't prove they understand how to use the word in a real story. Real children don't take multiple-choice tests; they just talk. To truly know if an AI is "learning" like a human, we need to see how it handles the messy, back-and-forth flow of real conversation.

2. The New Tool: "Contextual Alternative Choice" (CAC)

The authors invented a new way to test AI called CAC. Think of it like a "fill-in-the-blank" game, but with a twist.

The Setup: They take a real recording of a mother talking to her 2-year-old child.
The Game: They hide the word the child used (usually a small word like "the" or "a") and ask the AI: "Given what the mom just said, which word should go here?"
The Goal: They don't just want the AI to pick the right word. They want to see if the AI's pattern of choices matches the child's pattern.

3. The Two Benchmarks (The Scorecard)

The authors created two specific scorecards to see if the AI is acting like a human child.

Scorecard A: The "Mix-and-Match" Test (Formal Productivity)

The Concept: A smart child knows that "the" and "a" can go with almost any noun. They aren't just memorizing "the dog" and "a cat." They know the rule.
The Test: The researchers look at how many different nouns the AI uses with "the" and "a." If the AI only uses "the" with specific words it memorized, it fails. If it mixes them up freely with many different words, it passes.
The Result: Many AI models passed this. They learned the "mix-and-match" rule.

Scorecard B: The "Conversation Flow" Test (Discourse Function)

The Concept: This is the hard part. In a real conversation, if a mom says, "Look at the dog," the child usually replies, "The dog is big." But if the mom says, "I see a dog," the child might say, "A dog is running."
- Sometimes you keep the same word ("the" -> "the").
- Sometimes you switch ("a" -> "the").
- The switch depends entirely on the context of the story.
The Test: The researchers measured how often the AI switched words compared to how often real humans switch words.
The Result: This is where most AIs failed. Even the big, smart models tended to switch words randomly, like a coin flip. They didn't seem to understand the story behind the words. They knew the grammar, but they didn't know the social rules of conversation.

4. The Big Surprise

The authors tested 45 different AI models.

The "Small" Models: The models trained on about the same amount of data as a 2-year-old child (10–20 million words) generally failed both tests. They hadn't learned enough.
The "Big" Models: The massive models trained on billions of words (like the size of the entire internet) did much better.
- Two models passed both tests. They sounded like humans in both grammar and conversation flow.
- One model passed the conversation test but failed the grammar test.
- Many others passed the grammar test but failed the conversation test.

5. The Takeaway

The paper concludes that size matters, but it's not everything.

Just because an AI can get a grammar quiz right doesn't mean it understands language the way a human does. It might just be memorizing patterns. To truly be a "cognitive model" of human learning, an AI needs to pass the "Conversation Flow" test, showing it understands why we choose certain words based on the story we are telling.

Currently, only the very largest models are starting to show this human-like understanding, but even they are a mixed bag. The authors suggest that if we want to build better AI, we should stop treating them like students taking a test and start treating them like toddlers in a conversation.

Technical Summary: Measuring Form and Function in Language Models

Problem Statement
Current evaluations of Large Language Models (LLMs) predominantly rely on classification tasks (e.g., grammaticality judgments on minimal pairs) which often fail to capture the functional and discourse-level aspects of linguistic knowledge. Furthermore, these classification-based benchmarks lack a rigorous ground truth when comparing LLMs to child language acquisition, as young children cannot provide reliable metalinguistic judgments. While the "BabyLM" initiative seeks to build data-efficient models comparable to children, there is a methodological gap in how to rigorously compare the external behavior of LLMs against the naturalistic production data of children. Specifically, it remains unclear whether LLMs possess a productive grammatical system or merely memorize input chunks, and whether they utilize language for appropriate communicative functions beyond sentence well-formedness.

Methodology
The authors propose a framework that adapts quantitative metrics from child language research to evaluate LLMs, focusing on English determiners (the vs. a/an), a domain acquired early and accurately by children. The methodology consists of three core components:

Contextual Alternative Choice (CAC) Prompting:
Instead of asking models to classify sentences, the authors use a prompting scheme derived from the psycholinguistic "cloze" task. For every determiner-noun (D×N) site in a child's utterance within the Manchester Corpus, the model is presented with the full preceding discourse context and asked to predict the determiner. This elicits a probabilistic distribution over the choices (the vs. a) rather than a single binary label, allowing for the analysis of the model's internal expectations in identical contexts to those used by children.
Formal Productivity Benchmark (Overlap Test):
To distinguish between genuine grammatical productivity and rote memorization, the paper employs a statistical test based on Yang (2013). This test calculates the expected "overlap" (the proportion of noun types appearing with both the and a) for a corpus of a given size ( $S$ ) and vocabulary ( $N$ ), assuming a productive grammar where determiners combine independently with nouns. The test incorporates Zipf's Law for noun frequencies and an aggregate "bias" ( $b$ ) reflecting the tendency of specific nouns to favor one determiner. If a model's empirical overlap does not significantly differ from the expected overlap derived from these statistical properties, it is considered to exhibit formal productivity.
Discourse-Functional Benchmark (Transitional Probability of Reference - TPR):
To assess functional knowledge, the authors measure the TPR, defined as the probability that a speaker changes the determiner when the topic noun remains constant across conversational turns. This metric captures the appropriate use of definiteness and specificity in discourse. The model's TPR is calculated by comparing its probabilistic determiner choices against the caretaker's preceding utterance in the dialogue. This is compared against a baseline established from adult-to-adult conversations and the empirical TPR of children.

Key Contributions

Methodological Innovation: The introduction of CAC prompting allows for the direct comparison of LLMs and children using naturalistic production data, bypassing the need for unreliable metalinguistic judgments from children.
Dual-Benchmark Evaluation: The paper establishes two independent benchmarks—formal productivity (syntax) and discourse-functional appropriateness (pragmatics)—that do not require access to the model's training data, making them applicable to any language sample.
Empirical Analysis: The study evaluates 45 language models (including BabyLM baselines, winners, and reference models) across these benchmarks, providing a granular view of model performance relative to 2-3 year-old children.

Results
The evaluation reveals a significant dissociation between different facets of linguistic competence in LLMs:

Formal Productivity: Many models, including those trained on data amounts comparable to that available to 2-3 year olds (approx. 10–20 million words), pass the formal overlap test. This suggests they have learned to combine determiners and nouns productively rather than simply memorizing collocations.
Discourse Function: None of the models trained on data comparable to that of young children pass the TPR test. Their TPR values are significantly higher than human baselines, indicating they choose determiners stochastically without regard for referential restrictions or discourse context.
Large Models: Only two models trained on significantly larger datasets (100M and 30B words) passed both benchmarks. One model trained on 156B words passed the TPR test but failed the overlap test.
Accuracy vs. Competence: Several models achieved high accuracy in maximum likelihood (MLE) determiner choices (matching children ~89% of the time) yet failed the TPR and overlap tests. This confirms that high classification accuracy does not equate to adult-like linguistic competence or productive grammar.

Significance and Claims
The paper argues that current LLMs trained on developmentally plausible data do not yet exhibit the full spectrum of linguistic knowledge demonstrated by human children by age 2-3. Specifically, while models can learn formal syntactic productivity, they struggle with the functional, discourse-level constraints of language use. The authors claim that optimizing for one aspect of competence (e.g., prediction accuracy or formal syntax) does not guarantee the acquisition of others (e.g., discourse function).

The work serves as a methodological contribution, demonstrating that child language research provides necessary frameworks for evaluating LLMs as cognitive models. The authors suggest that early acquisition milestones, such as determiner usage, serve as critical checkpoints for the field, highlighting that LLMs may require different learning trajectories or data efficiencies to truly model human language acquisition. The paper concludes modestly, noting that while some very large models meet the benchmarks, the causes of success or failure remain unclear, and further investigation is needed into how instruction tuning and reinforcement learning from human feedback (RLHF) might interact with these specific linguistic criteria.

1. The Problem: The "Multiple Choice" Trap

2. The New Tool: "Contextual Alternative Choice" (CAC)

3. The Two Benchmarks (The Scorecard)

4. The Big Surprise

5. The Takeaway

More like this