💬 NLP

Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D

This paper demonstrates that frontier language models struggle to copy text due to limitations in standard 1D positional encodings, and proposes 2D-RoPE, a method that organizes text into a 2D grid to enable superior copying performance and generalization across both synthetic and large-scale pretraining tasks.

Original authors: Haodong Wen, Yiran Zhang, Yingfa Chen, Kaifeng Lyu

Published 2026-07-20

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Haodong Wen, Yiran Zhang, Yingfa Chen, Kaifeng Lyu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a super-smart robot how to read and write. You give it a massive library of books, and it learns to predict the next word in a sentence with incredible accuracy. This robot is a Large Language Model (LLM), and it's the brain behind many of the AI tools we use today. But here's the catch: these robots don't "read" like humans do. They see text as a long, single line of beads on a string. To understand where they are in that line, they use something called "positional encoding." Think of this like a set of invisible labels on every bead, telling the robot, "You are the 5th bead," or "You are the 100th bead."

For a long time, scientists thought these robots were getting so smart they could solve almost any puzzle. But there was a weird, embarrassing glitch they couldn't fix. If you asked the robot to simply copy a string of numbers or letters exactly as it saw them, it would sometimes fail, even if the string was short enough to fit in its memory. It's like asking a human to copy a sentence from a book, and they accidentally skip a word or swap two letters because they got confused about where they were in the line. The big question was: Why does a machine that can write poetry and solve math problems struggle to just copy a list?

This paper investigates that exact mystery. The authors, a team of researchers from Tsinghua University, discovered that the robot's "labels" (the positional encodings) were actually the problem. The standard way of labeling text makes it hard for the robot to find the exact spot it needs to copy from, especially when the text has repeating patterns. Instead of looking at the specific position, the robot gets distracted by similar-looking chunks of text nearby. To fix this, the researchers tried a new idea: what if we stop treating text like a single long line and instead arrange it like a grid, like a spreadsheet with rows and columns? They created a new system called 2D-RoPE.

When they tested this new system, the results were surprising. In their experiments, models using the 2D grid could copy strings perfectly, even when the strings were hundreds of times longer than anything they had seen during training. In contrast, the standard models kept failing. The researchers showed that by organizing text into rows and columns, the task of copying becomes much simpler for the robot: it just needs to look at the same column in the row above. They also built a smarter version called Auto-2D-RoPE that can figure out the grid structure on its own, even if the text doesn't have clear line breaks.

The paper suggests that this simple change in how we organize text data can make AI much better at basic tasks like copying and formatting, which are crucial for things like turning raw data into code. While the researchers proved mathematically that this new method works for simple copying tasks and showed it works in large-scale tests, they note that it's still an area of active study. They hope this discovery encourages others to rethink how AI "sees" text, proving that sometimes, looking at the world in two dimensions is better than looking at it in a straight line.

Technical Summary: Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D

Problem Statement

Despite achieving state-of-the-art performance on complex reasoning tasks, frontier Large Language Models (LLMs) exhibit a fundamental failure in performing exact string copying. The authors demonstrate that even models with context windows significantly larger than the input strings fail to reproduce input sequences accurately, particularly when the inputs contain repeated substrings or when the sequence length exceeds the training distribution.

The paper identifies two primary failure modes:

Length Generalization Failure: Copying accuracy degrades as input length increases, often falling below 50% for lengths well within the model's context window.
Repetition Sensitivity: Models struggle significantly with inputs containing repeated patterns (e.g., periodic signals or recursive structures), suggesting they rely on local context matching rather than precise positional retrieval.

The authors hypothesize that this failure stems from the inductive bias of standard positional encodings (like RoPE). Instead of retrieving the $k$ -th token based on its absolute or relative index, models appear to use a "context-based" shortcut, searching for similar local substrings in the input. When repeated substrings exist, this mechanism becomes ambiguous, leading to errors.

Methodology

1. Diagnostic Analysis

The authors first establish the existence of the problem through extensive benchmarks:

Binary Copy: Models are tasked with copying binary strings generated via uniform, imbalanced, or recursive-flip distributions.
Python List Conversion: Models must convert comma-separated physical sensor data into Python lists.
Mechanistic Interpretability: Attention map analysis reveals that standard RoPE models do not learn a clean, position-based alignment. Instead, they exhibit a mixture of:
- Horizontal patterns: Attending to fixed delimiters (e.g., <\n>, <I:>), a position-based shortcut that fails on unseen lengths.
- Diagonal patterns: Induction-head behavior that matches local contexts, which fails when substrings repeat.

2. Proposed Solution: 2D-RoPE

Motivated by the observation that copying an input string $x_1, \dots, x_n$ to an output $y_1, \dots, y_n$ is structurally a 2D operation (two rows), the authors propose 2D-RoPE.

Concept: Text is organized into a 2D grid rather than a 1D sequence. The newline token <\n> acts as a row separator.
Position IDs: Each token is assigned a pair $(r_i, c_i)$ , where $r_i$ is the row index and $c_i$ is the column index within that row.
Mechanism: RoPE attention is applied separately to row and column coordinates. In a copy task, the source token $x_k$ and target token $y_k$ share the same column index. Thus, copying reduces to retrieving a token from the same column in the previous row, a fixed-offset operation that is geometrically simpler for a single attention layer to learn than a length-dependent 1D offset.

3. Adaptive Variant: Auto-2D-RoPE

Recognizing that natural text may not always contain explicit line breaks to define rows, the authors introduce Auto-2D-RoPE.

Instead of relying on a fixed separator token, this variant learns a data-dependent transformation.
At each layer, the model predicts update coefficients based on token hidden states to determine how the 2D coordinates evolve (incrementing column vs. resetting column and incrementing row).
This allows the model to infer 2D structure adaptively, maintaining length generalization even without explicit newline separators.

Key Contributions

Empirical Evidence of Copy Failure: The paper provides rigorous benchmarks showing that frontier LLMs (including GPT-5.5, Gemini 3.1 Pro, and DeepSeek V4 Pro) fail at exact copying tasks, particularly on long or repetitive sequences, contradicting the assumption that basic algorithmic capabilities are mastered by large models.
Theoretical and Mechanistic Diagnosis:
- Expressivity Theory: The authors prove that one-layer Transformers with shift-invariant positional encodings cannot solve binary copy for short sequences ( $n \le 5$ ), while two-layer RoPE models can. This suggests the failure is not due to expressivity limits but rather the inductive bias of training.
- Attention Analysis: They demonstrate that trained models rely on a fragile mixture of position-based and context-based heuristics rather than a robust algorithm.
2D-RoPE Architecture: The introduction of 2D-RoPE, which reformulates text as a 2D grid, providing a strong inductive bias for position-based retrieval.
Theoretical Guarantees:
- Expressivity: Proven that a single-layer 2D-RoPE Transformer can represent the binary copy task with bounded parameters.
- Generalization: Under $\ell_\infty$ regularization constraints (mimicking AdamW optimization), the global minima of 2D-RoPE trained on length $L$ provably generalize to lengths polynomial in $L$ .
Large-Scale Validation: The method is validated on pretraining from scratch (350M to 1.4B parameters) on the DCLM dataset. 2D-RoPE models achieve near-perfect length generalization on copy tasks while maintaining comparable performance on common-sense reasoning benchmarks.

Results

Synthetic Experiments:
- 1-Layer Models: 2D-RoPE achieves perfect copying with up to 1000 $\times$ length generalization (training on length 100, testing on length 100,000), whereas standard encodings fail immediately.
- 12-Layer Models: 2D-RoPE maintains perfect performance up to 100 $\times$ length generalization.
Large-Scale LLM Pretraining:
- Models pre-trained on DCLM and fine-tuned on copy tasks show that 2D-RoPE significantly outperforms standard RoPE and Hybrid-RoPE (H-RoPE) on both in-domain (Imbalanced) and out-of-domain (Recursive-Flip) copy tasks.
- For example, a 1.4B parameter 2D-RoPE model achieves 100% accuracy on 8K-length recursive-flip strings, while the standard RoPE baseline drops to 0%.
- Common-sense reasoning performance remains comparable to RoPE baselines, indicating that learning the copy task does not degrade general language modeling capabilities.
Auto-2D-RoPE: Successfully maintains length generalization on copy tasks even when the newline separator is replaced with a generic character, demonstrating the ability to learn 2D structure without explicit formatting cues.

Significance and Claims

The paper claims that the representation of positional information fundamentally shapes the algorithmic behaviors of language models. The inability of current LLMs to copy is not a lack of intelligence but a consequence of the inductive bias introduced by 1D positional encodings, which favor context matching over precise positional retrieval.

By viewing text in 2D, the authors argue that the copying task becomes geometrically simple (fixed-column retrieval), allowing models to learn robust, length-generalizing algorithms. The work suggests that future research should explore 2D positional encodings to better capture the latent structure of text and improve the reliability of algorithmic reasoning in LLMs. The authors remain modest, noting that the theoretical understanding of why multi-layer RoPE transformers fail to copy and the scaling of Auto-2D-RoPE to more practical regimes remain open directions for future study.