💬 NLP

When 2D Tasks Meet 1D Serialization: On Serialization Friction in Structured Tasks

This paper demonstrates that representing 2D structured tasks as 1D token sequences introduces "serialization friction" that hinders performance, whereas preserving the native 2D layout through a vision-augmented pathway significantly improves accuracy and reduces spatially structured errors in tasks like matrix transpose, Conway's Game of Life, and LU decomposition.

Original authors: Chung-Hsiang Lo, Lu Li, Diji Yang, Tianyu Zhang, Yunkai Zhang, Yoshua Bengio, Yi Zhang

Published 2026-05-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chung-Hsiang Lo, Lu Li, Diji Yang, Tianyu Zhang, Yunkai Zhang, Yoshua Bengio, Yi Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart student how to solve a puzzle. The puzzle is a grid of numbers, like a spreadsheet or a chessboard.

This paper asks a simple question: Does it matter how you show the puzzle to the student?

The Two Ways to Show the Puzzle

The "List" Method (1D Serialization): You take the grid and flatten it into a long, single line of text, like reading a book. You have to use commas and brackets to tell the student where one row ends and the next begins.
- Analogy: Imagine trying to explain a map to someone by reading them a list of street names and turns in a single sentence: "Go north, turn right at the bakery, go two blocks, turn left..." It's accurate, but you lose the big picture of how the streets connect.
The "Picture" Method (2D Layout): You show the student the actual grid, with rows and columns clearly visible, just like a real spreadsheet or a game board.
- Analogy: You hand them the actual map. They can instantly see that the bakery is right next to the park. The spatial relationships are right there in front of them.

The Problem: "Serialization Friction"

The authors call the difficulty of the "List" method "Serialization Friction."

When you flatten a 2D grid into a 1D list, you force the student's brain to do extra work. They have to remember, "Okay, I'm at the 5th number, which means I'm in the second row," or "This number is next to that one in the list, but they are actually far apart in the grid."

The paper argues that this extra mental gymnastics creates a "friction" that makes the task harder, even if the student is very smart.

The Experiments: Three Puzzles

To test this, the researchers gave the student three different types of grid puzzles:

Matrix Transpose (The Flip): Imagine a grid of numbers. The task is to flip it over its diagonal (turning rows into columns).
- Result: When shown as a picture, the student got it right almost every time, even for huge grids. When shown as a list, the student started making mistakes as the grid got bigger, eventually getting almost everything wrong.
Conway's Game of Life (The Neighbor Game): Imagine a grid of living and dead cells. The rule is: a cell lives or dies based on its immediate neighbors.
- Result: Again, the "Picture" student was perfect. The "List" student struggled to keep track of which numbers were neighbors, and their accuracy dropped as the board got bigger.
LU Decomposition (The Math Chain): This is a complex math problem where you have to break a grid down into two smaller grids through a series of steps. Each step depends on the previous one.
- Result: The "Picture" student could follow the chain of logic much better. The "List" student got lost in the middle of the process, especially when the grid was large or when they had to mix different sizes of grids.

The "Aha!" Moment

The researchers also did a special test to make sure it wasn't just about "pictures vs. words." They took the "List" method and turned it into a picture, but they scrambled the layout so it looked like a jumbled mess of text on a page.

The Result: The student did even worse with the "scrambled picture" than with the "clean list."
The Lesson: It's not just that pictures are "cool." It's that preserving the natural shape of the data (keeping rows and columns aligned) helps the brain solve the problem. When you break that shape, the brain has to work harder to rebuild the structure in its head.

What They Found

The Gap Grows: The difference between the "Picture" student and the "List" student gets bigger as the puzzles get larger.
The Mistakes Have a Pattern: When the "List" student failed, they didn't just make random mistakes. They tended to make errors in specific spots (like the bottom-right corner of the grid), suggesting they lost track of the spatial layout as the list got longer.

The Bottom Line

The paper concludes that for tasks where the shape of the data is part of the solution (like grids, maps, or tables), forcing that data into a long, flat list creates unnecessary friction. Keeping the data in its natural 2D layout allows the model to perform much better and more reliably.

In short: If you want someone to solve a grid puzzle, show them a grid. Don't make them read a long list of instructions to figure out where the pieces go.

1. Problem Statement

Large Language Models (LLMs) traditionally process inputs as 1D token sequences. While effective for natural language prose, this linearization introduces "serialization friction" when applied to tasks with explicit 2D structural dependencies (e.g., row-column alignment, local neighborhood interactions).

The Core Issue: When 2D data (like matrices or grids) is flattened into a 1D string, the model must infer spatial relationships (e.g., which element is "above" or "to the right" of another) solely from token order and delimiters.
Hypothesis: The paper posits that this representational burden, rather than just the computational complexity of the task, degrades model performance. Preserving the native 2D layout should alleviate this friction.

2. Methodology

The authors constructed a controlled diagnostic testbed to compare two input pathways using the same underlying language backbone (GLM-4-9B):

A. Experimental Setup

Text-Only Pathway (1D): Inputs are serialized into text strings (e.g., [[1, 2], [3, 4]]). The model is GLM-4-9B-0414.
Vision-Augmented Pathway (2D): The same underlying data is rendered as images preserving the native 2D layout. The model is Glyph (a visual-text compression method that connects GLM with a vision encoder).
Controlled Variables: Both pathways use identical task definitions, prompts, training/test splits, and evaluation criteria. The only variable is the input representation.

B. Task Suite

Three synthetic tasks were selected, each relying on different aspects of 2D structure:

Matrix Transpose: Tests row-column correspondence. The transformation is a direct rearrangement based on spatial alignment.
Conway's Game of Life: Tests local neighborhood interaction. The next state of a cell depends on its 8 surrounding neighbors in a grid.
LU Decomposition: Tests multi-step algorithmic reasoning. The task requires maintaining row-column relationships across a sequence of dependent elimination steps to factorize a matrix.

C. Training Paradigms

Matrix Transpose & Game of Life: Supervised Fine-Tuning (SFT) with LoRA.
LU Decomposition: Reinforcement Learning with Verifiable Reward (RLVR), as the task involves a sequence of steps where intermediate correctness is hard to define, but the final factorization is verifiable.

D. Additional Probes

Within-Visual Probe: A "disruptive-Glyph" condition was tested where the 2D layout was intentionally scrambled (presenting text as an image but without structural alignment) to isolate the effect of layout from the modality itself.
Error Analysis: Heatmaps were generated to visualize the spatial distribution of errors.

3. Key Contributions

Conceptual Framework: Introduced the term "Serialization Friction" to describe the representational burden incurred when flattening 2D structures into 1D sequences.
Diagnostic Testbed: Created a rigorous comparison framework using three distinct 2D tasks to isolate the impact of input layout on reasoning.
Empirical Evidence: Demonstrated that preserving task-relevant 2D layout consistently outperforms 1D serialization, particularly as task dimensions increase.
Error Characterization: Showed that errors in 1D serialization are not random noise but are spatially structured, often concentrating in specific regions (e.g., lower-right corners of matrices) as the problem size grows.

4. Key Results

A. Performance Gap

Across all three tasks, the 2D visual pathway (Glyph) consistently outperformed the 1D textual pathway (GLM).

Matrix Transpose:
- At $12 \times 12$ , 1D accuracy was ~92.7%, dropping to 0.8% at $20 \times 20$ .
- The 2D pathway remained above 90% even at $20 \times 20$ .
- In mixed-size training (generalization), the gap widened significantly; 1D serialization failed to generalize to unseen sizes, while 2D maintained high accuracy.
Conway's Game of Life:
- 2D layout maintained near-perfect accuracy (>99.7%) across grid sizes ( $4\times4$ to $8\times8$ ).
- 1D serialization accuracy dropped steadily from 86.5% ( $4\times4$ ) to 46.6% ( $8\times8$ ).
LU Decomposition:
- In mixed-training settings, 1D serialization accuracy hovered near 0%, whereas 2D layout maintained substantial accuracy (e.g., 69.2% at $5\times5$ ).
- This suggests that 2D layout is critical for maintaining state consistency across multi-step dependent operations.

B. Error Patterns

Spatial Structuring: Errors in the 1D pathway were not uniform.
- In Matrix Transpose, errors concentrated in the lower-right of the matrix as dimensions increased, suggesting the model loses track of long-range row-column dependencies.
- In LU Decomposition, errors clustered in later rows/columns, indicating a failure to maintain the chain of elimination steps.
Visual Probe: The "disruptive-Glyph" (scrambled 2D layout) performed worse than the structured 2D layout but better than pure text, confirming that the structural alignment itself, not just the visual modality, drives the performance gain.

5. Significance and Implications

Beyond Text-Only Paradigms: The study challenges the assumption that 1D tokenization is sufficient for all reasoning tasks. It suggests that for tasks with inherent spatial structure, layout is an inductive bias that aids reasoning.
Multimodal Utility: While recent work suggests multimodal models sometimes struggle with visualized text, this paper shows that for structured data, rendering the data in its native 2D form is superior to text serialization.
Future Directions: The findings suggest that preserving task-relevant 2D layouts is a promising direction for improving LLMs on structured tasks (e.g., tables, graphs, code, scientific diagrams). It implies that future architectures or training strategies should consider structural isomorphism (matching input representation to the problem domain's structure) rather than forcing all data into a linear sequence.

In conclusion, the paper provides strong evidence that serialization friction is a significant bottleneck for LLMs on 2D structured tasks and that leveraging vision-augmented pathways to preserve native 2D layouts can dramatically improve reasoning capabilities and generalization.