Technical Summary: The Text Uncanny Valley

Problem Statement

Existing benchmarks for Large Language Models (LLMs) predominantly evaluate performance on syntactically clean inputs, creating a significant gap in understanding how models handle imperfect, uncurated text. Real-world scenarios, such as documents processed via Optical Character Recognition (OCR) or content copied from web sources, frequently contain formatting artifacts, including spurious spaces inserted within words (e.g., int er nation al). While prior work has examined LLM robustness to noise, typos, and character-level perturbations, these studies generally report monotonic degradation: as noise increases, performance decreases.

This paper addresses the specific gap of word-boundary corruption on tasks requiring exact span-level matching between two document versions. The authors investigate whether the relationship between word fragmentation and model performance is strictly monotonic or if more complex dynamics exist, particularly in tasks demanding precise lexical alignment.

Methodology

The authors propose a controlled perturbation framework to study the effects of intra-word fragmentation.

• Perturbation Design: The method involves inserting whitespace characters within contiguous non-space character sequences (words). A tunable parameter, word_fragmentation_rate ($p \in [0, 1]$), controls the probability of inserting a space at any intra-word slot. The rate sweeps from 0.0 (clean text) to 1.0 (every character separated by a space) across eleven discrete levels.
• Datasets: Experiments are conducted across three distinct domains:
  1. Legal Documents: Expert-annotated commercial contracts (CUAD-QA).
  2. GitHub Pull Requests: Code diffs containing dense, non-spaced identifiers.
  3. Mathematical Reasoning: GSM8K problems concatenated into long documents.
• Evaluation Tasks: Two tasks requiring exact line-level alignment between an original and a modified document:
  1. Absence Detection: Identifying lines deliberately omitted from the text.
  2. NIAH-style Insertion Detection: Identifying lines inserted into the text (adapting the "Needle in a Haystack" paradigm to span matching).
• Models: Eight model variants were evaluated, including DeepSeek-V3, Grok-4.1-Fast, Qwen3-Plus, Gemini 3.0 Flash, GPT-5.2, and Claude 4.5 Haiku. Performance is measured using Micro-F1 based on case-insensitive exact matching.

Key Findings: The Text Uncanny Valley

Contrary to the expectation of monotonic degradation, the study reveals a non-monotonic U-shaped performance curve for three specific model families (Gemini 3.0 Flash, GPT-5.2, and Claude 4.5 Haiku) across both tasks and most domains.

• The Curve: As the word_fragmentation_rate increases from 0, Micro-F1 performance degrades, reaching a minimum at moderate fragmentation rates (the "valley"), and then partially recovers at extreme fragmentation rates (near 1.0).
• The Phenomenon: The authors term this the Text Uncanny Valley. It represents a state where text is neither fully natural nor uniformly fragmented, leaving the model in a disordered transitional state where performance is worst.
• Domain Variations: The U-shape is most pronounced in absence detection on legal and mathematical domains. On GitHub PRs, models sometimes show an initial performance rise at low fragmentation rates before declining, suggesting that breaking dense identifier runs can initially aid parsing before heavy fragmentation overwhelms the benefit.
• Model Dependence: Not all models exhibit this behavior. Grok-4.1-Fast showed largely monotonic decline. Qwen3-Plus and DeepSeek-V3 exhibited irregular patterns or API failures at high fragmentation rates.

Hypothesis and Explanatory Evidence

The authors propose a Two-Regime Hypothesis to explain the U-shape:

1. Word-Level Regime: At low fragmentation, subword tokenizers produce familiar multi-character tokens, and the model processes text via lexical units.
2. Character-Level Regime: At extreme fragmentation ($p \approx 1.0$), tokenization collapses to single characters, forcing the model to rely on character-level pattern matching to recover semantics.
3. The Valley: The performance minimum occurs in the transition zone where fragmentation is irregular. Words are inconsistently shattered; the model cannot commit to either a stable word-level strategy or a consistent character-level strategy, leading to a "regime conflict."

Supporting Experiments:

• In-Context Learning (ICL): Providing few-shot examples of fragmented text did not rescue valley-bottom performance, suggesting the bottleneck is not a lack of task strategy but a lower-level tokenization issue.
• Fixed-Slot Fragmentation: When fragmentation was made regular (inserting spaces only at the first slot of words), the U-shape was substantially reduced or eliminated. This confirms that the irregularity of the fragmentation, rather than the magnitude, drives the valley.
• Inter-word vs. Intra-word: Randomly replacing spaces between words with extra whitespace did not induce the valley, confirming that the disruption of contiguous non-space sequences (subword tokenization) is the causal factor.
• Math Reasoning Control: In a direct math reasoning task (GSM8K) without cross-document alignment requirements, the U-shape persisted for Gemini 3.0 Flash but disappeared for GPT-5.2 and Claude 4.5 Haiku, suggesting the effect is attenuated when tasks rely less on exact lexical alignment.
• Tokenization Entropy: Analysis of token-ID distribution entropy showed that entropy peaks at a lower fragmentation rate than the F1 minimum. This supports the hypothesis that the valley is not the point of maximum token confusion, but rather the unstable transition between two processing regimes.

Significance and Implications

The paper claims to reveal a failure mode invisible to clean-text benchmarks that is directly relevant to deployment scenarios involving noisy or uncurated text.

• Moderate Corruption is Most Dangerous: The findings suggest that moderate word-boundary corruption (typical of OCR artifacts) is more damaging to LLM performance than extreme corruption. Systems may be robust to clean text or fully broken text but fail catastrophically in the "uncanny" middle ground.
• Benchmarking Gaps: Current benchmarks that rely on clean inputs or discrete noise levels may miss this non-monotonic vulnerability.
• Practical Relevance: This phenomenon poses risks for document processing pipelines, OCR systems, and LLM-as-a-judge evaluators where input text quality varies. The authors argue for robustness benchmarks that explicitly cover the ambiguous middle regime between clean and uniformly corrupted text.

The study concludes that the Text Uncanny Valley marks an unstable transition between word-level and character-level processing modes, a failure mode that is particularly acute for tasks requiring exact lexical alignment over long contexts.