How Human-Like Are Large Language Models? A Register-Aware Linguistic Evaluation Framework

Technical Summary: How Human-Like Are Large Language Models? A Register-Aware Linguistic Evaluation Framework

Problem Statement
While current Large Language Model (LLM) research heavily prioritizes factual correctness and task-performance benchmarks (e.g., MMLU, domain-specific QA), the fundamental question of how closely LLM-generated text resembles human language on a linguistic level remains under-explored. From a corpus-linguistic perspective, human language production is inherently context-dependent; distinct communicative contexts (registers) give rise to specific frequencies and co-occurrence patterns of linguistic features. A text may be factually correct yet linguistically "unnatural" if it fails to adhere to the patterns of its target register. Existing detection methods often rely on model embeddings (which are not human-interpretable) or fail to properly measure register-alignment, particularly when models are conditioned on explicit register examples rather than inferring register from situational context.

Methodology
The authors propose a register-aware evaluation framework that assesses human-likeness as a two-sample problem between the linguistic feature distribution of a human reference corpus and a corresponding LLM-generated corpus.

1. Data Generation & Implicit Register Conditioning:

   • Five distinct English registers were selected: Spoken conversation (BNC2014Spoken), Academic writing (S2ORC_ACL), Instructive online text (wikiHow), Creative writing (WritingPrompts), and News reporting (XSum).
   • Human reference corpora were collected and metadata was used to construct natural language prompts.
   • Crucially, the framework employs implicit-register conditioning: prompts describe the situational context without including register-specific text examples, forcing the model to infer the target style.
   • Seven open-source, instruction-tuned models (ranging from 8B to 70B parameters, including Llama 3, Qwen 3, Gemma 3, and Apertus) were evaluated in zero-shot and few-shot settings.

2. Feature Extraction:

   • The framework utilizes the 67 lexico-grammatical and discourse features introduced by Biber (1988), a standard in corpus linguistics for analyzing variation across registers.
   • Texts were standardized to a length of 400 lexical tokens to ensure comparability.

3. Distance Metric (MMD):

   • The discrepancy between the human and synthetic corpora is quantified using the Maximum Mean Discrepancy (MMD) in a Reproducing Kernel Hilbert Space (RKHS) with an RBF kernel.
   • MMD measures the distance between the mean embeddings of two distributions. A value of zero implies the distributions are identical.
   • To establish a baseline, the authors calculate the MMD between two disjoint subsamples of the human corpus ($MMD_{human-human}$) to generate a 95% Confidence Interval (CI). If an LLM's distance exceeds the upper bound of this CI, it indicates a statistically significant deviation from human language patterns.

Key Contributions

• A Register-Aware Framework: A method for assessing human-likeness based on the distributional difference of linguistic features using MMD, moving beyond task-specific metrics.
• Open-Source Implementation: The release of code, generation settings, and results using Biber's 67 features.
• Comprehensive Benchmark: A comparative analysis of seven models across five registers, including an ablation study on prompt stability and few-shot generation effects.

Results

• Universal Deviation: Across all tested setups, every LLM produced a distribution of linguistic features that deviated significantly from the human baseline (exceeding the human-human CI).
• Register Dependency: The ranking of models by "human-likeness" is not consistent across registers. For instance, Qwen models performed best on Spoken conversation, Creative writing, and News, while Apertus 70B was closest to human baselines for Academic writing and Instructive text.
• Model Size vs. Performance: Human-likeness is not dictated by model size. Smaller models (e.g., Qwen 8B) often performed as well as or better than larger counterparts (e.g., Llama 70B) depending on the register.
• Specific Linguistic Biases:
  • Models generally overused Present Participle Clauses and Nominalizations compared to humans.
  • Models tended to produce longer words and used Attributive Adjectives more frequently.
  • Models showed reduced variability in linguistic features compared to humans in four out of five registers.
• Prompt Stability: While absolute distance values varied with prompt phrasing, the relative ranking of models remained consistent (correlation $\ge$ 0.985).
• Few-Shot Effects: Providing few-shot examples generally reduced the distance to the human distribution, though the effect was minor and occasionally detrimental (e.g., for Llama 8B on Spoken conversation).

Significance and Claims
The paper claims that human-likeness is a register-dependent property that must be evaluated with register sensitivity. Aggregating results across registers obscures systematic differences; a model ranked as "most human-like" in one context may not be in another.

The authors argue that their framework provides a transparent, interpretable metric (MMD on human-interpretable features) that can be optimized in future work to improve LLM language production. By minimizing the MMD between generated texts and representative register samples, models could be encouraged to produce texts that are linguistically appropriate for their communicative purpose and avoid "mode collapsing" into narrow distributions of features. The study concludes that current LLMs, regardless of size or architecture, systematically fail to replicate the full variability and specific co-occurrence patterns of human language within specific registers.