Language Re-generation: An investigation into information locality effects on reconstruction

Technical Summary: Language Re-generation

Problem Statement
This paper investigates the relationship between information locality—the tendency for syntactically and semantically related words to appear close together—and the ability of Large Language Models (LLMs) to reconstruct natural language from structurally degraded inputs. While prior research has established that LLMs exhibit inductive biases toward natural language structures and struggle to learn "impossible languages" (those violating cognitive constraints like locality), it remains unclear whether LLMs can actively recover natural language from such impossible inputs. Specifically, the authors ask: (1) What do patterns of recovery success and failure reveal about LLMs' inductive biases toward information locality? and (2) Does the difficulty of recovery scale with the severity of locality violations?

Methodology
The authors frame the problem as a translation task: translating "impossible" (perturbed) text back into "possible" (natural) English.

• Data Generation: Using subsets of the BabyLM corpus (bnc_spoken and Gutenberg), the authors applied three perturbation functions to create "impossible" language datasets. These perturbations vary in their disruption of information locality:
  1. LOCALSHUFFLE: Randomly shuffles tokens within a small local window (size 3), disrupting sequential order but maintaining bounded distances.
  2. PARTIALREVERSE: Selects a random point in the sentence, marks it with an 'R' token, and reverses all subsequent tokens. This preserves the initial segment while disrupting the rest.
  3. FULLSHUFFLE: Randomly shuffles tokens globally across the entire sentence, completely destroying dependency structure and locality.
• Model Architecture & Training: The study utilizes GPT-2 models pre-trained from scratch on these impossible languages (based on Kallini et al., 2024). This ensures the models have no prior exposure to natural English, preventing them from leveraging pre-existing knowledge to solve the task. The models are then fine-tuned to reconstruct the original English text from the perturbed inputs.
• Training Strategy: The models are trained using a causal language modeling paradigm with a masked label strategy. The input prompt ("Fix this text: [perturbed]") is masked (loss = -100), while the target output ("Corrected: [original]") is used for loss calculation. This forces the model to learn the reconstruction mapping rather than memorizing the instruction.
• Evaluation Metrics:
  • Surface Metrics: Exact Match (EM) and BLEU score to measure lexical and surface-level reconstruction.
  • Structural Metrics: Average Dependency Length (ADL) and Dependency Triple F1 to measure the recovery of syntactic relationships.
  • Perplexity: To assess how closely the generated text aligns with natural language distributions.

Key Results
The experiments, conducted across different sentence lengths (short vs. long) and training sizes, yielded three primary findings:

1. Recovery Difficulty Scales with Locality Violation: Performance is systematically ordered by the severity of the perturbation. PARTIALREVERSE is the easiest to recover, followed by LOCALSHUFFLE, with FULLSHUFFLE being the most difficult. This confirms that the degree of locality disruption directly correlates with reconstruction difficulty.
2. Dissociation of Surface and Structural Recovery: Structural recovery (measured by Dependency Triple F1) dissociates from surface recovery (Exact Match). For instance, under LOCALSHUFFLE, models recover substantial syntactic structure (high Triple F1) even when the exact word order is not perfectly restored (lower EM). Conversely, under global shuffling, models may generate fluent text that fails to reconstruct the original meaning.
3. Architectural Bias Toward Shorter Dependencies: A critical finding is that recovered structures consistently exhibit shorter dependency lengths than the original natural text. Even when models successfully reconstruct the text, they exhibit a bias toward placing related words closer together than in the source, mirroring the Dependency Length Minimization (DLM) principle observed in human language but pushing it further.
4. Sentence Length Interactions: The effect of sentence length is non-uniform. For perturbations that preserve some local structure (PARTIALREVERSE, LOCALSHUFFLE), longer sentences facilitate recovery by providing richer contextual cues. However, for FULLSHUFFLE, longer sentences lead to a complete collapse in performance, as the search space for reordering expands exponentially without any preserved structural anchors.

Significance and Claims
The paper claims to provide a quantitative signature of an architectural bias in LLMs that learnability experiments alone do not reveal. By demonstrating that LLMs can recover natural language from impossible inputs but systematically overshoot toward even shorter dependency lengths, the authors suggest that LLMs are sensitive to information-locality constraints but are not fully bounded by them.

The study bridges the gap between "learnability" (whether a model can acquire a language) and "reconstruction" (whether a model can repair a corrupted language). It posits that information locality is the shared constraint governing both acquisition and recovery. The findings indicate that while LLMs possess inductive biases aligned with human linguistic feasibility (specifically the preference for local dependencies), these biases are architectural in nature and can lead to systematic deviations from natural language statistics when under pressure to reconstruct highly disrupted inputs.

The authors conclude that information locality is the primary driver of the boundary between possible and impossible languages for LLMs, and that the ability to recover structure from noise is a direct function of how severely that locality is violated.