🤖 machine learning

Borrowed Geometry: Computational Reuse of Frozen Text-Pretrained Transformer Weights Across Modalities

This paper demonstrates that frozen, text-only pretrained Gemma 4 31B weights can be effectively transferred across modalities to achieve state-of-the-art performance in robotic manipulation, decision-making, and associative recall tasks using only a thin trainable interface, proving that the pretraining load-bearing capacity of the model is intrinsic to its weights rather than its specific modality or training data.

Original authors: Abay Bektursun

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Abay Bektursun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a master chef who has spent their entire life perfecting the art of cooking text. They can chop words, mix sentences, and bake paragraphs better than anyone else. This chef is a massive, frozen computer brain (a "frozen" AI model called Gemma 4) that has never seen a robot, a video game, or a physical object. It has only ever processed language.

Now, imagine you want to teach a robot to walk or a computer to play a game, but you don't want to train a new chef from scratch. Instead, you ask: "Can we borrow this text chef's brain to solve non-text problems?"

This paper says yes, but with a very specific twist.

The Core Idea: Borrowing Geometry

The authors treat the frozen text-chef's brain not as a tool for writing, but as a pre-built, high-tech kitchen. They don't change the chef's brain (the weights are "frozen"). Instead, they build a tiny, cheap "adapter" (a simple interface) to translate robot movements or game states into the language the chef understands, and then translate the chef's output back into actions.

Think of it like this: The chef knows how to organize ingredients perfectly. You don't need to teach them how to build a car; you just need to hand them the car parts in a format they recognize, and they will instinctively know how to assemble them because their brain is already wired for complex organization.

The Big Wins (What They Actually Did)

1. The Robot That Learned to Dance (OGBench)

The Test: They tried to get a simulated robot arm to pick up and move objects (a task it had never seen before).
The Result: Using the frozen text brain, the robot performed better than the current best specialized robot software.
The Catch: It wasn't perfect at every task. On one specific "cube" stacking task, the text-brain robot was actually worse than the specialized software. However, when they compared the text-brain robot to a randomly initialized brain (a brain with the same shape but no training), the text-brain was 59 points better.
The Lesson: The "frozen" text training provided a massive head start. The brain wasn't just a random shape; it had learned useful "muscle memory" for organizing sequences, even though it only learned that from reading books.

2. The Walking Robot (D4RL Walker2d)

The Test: Can the text brain help a robot learn to walk?
The Result: Yes. The text-brain robot walked just as well as the current state-of-the-art "Decision Transformer" (a model built specifically for walking).
The Bonus: The text-brain solution used less than half the trainable parameters of the specialized walker model. It was a more efficient way to get the same result.

3. The "Magic" Memory Test (Associative Recall)

The Test: A simple memory game where the computer must remember a pattern and repeat it later.
The Result: The frozen text brain, with a tiny adapter, solved this perfectly. A brand-new model of the exact same size, trained from scratch on this specific game, failed completely.
The Lesson: The text brain didn't just learn "words"; it learned a fundamental way of handling patterns and memory that applies even to non-language tasks.

The "Why": Finding the Specific "Atoms" of Thought

The most fascinating part of the paper is how they looked inside the brain to see what was happening. They didn't just say "it works"; they found the specific gears turning.

The Experiment: They looked at 192 tiny "attention heads" (think of these as individual neurons or specialized workers inside the brain).
The Discovery: They found four specific workers (heads) that were doing double duty.
- In the text world, these workers were experts at copying words or matching patterns (e.g., noticing that "cat" appears twice in a sentence).
- In the non-text world (like the robot tasks or the memory game), those exact same four workers were the most critical for solving the problem.
The Analogy: Imagine a worker who was hired to sort red and blue marbles. You then ask them to sort red and blue cars. They fail at the cars, but you find that the specific worker who was best at sorting marbles is also the one who instinctively knows how to sort the cars. The brain didn't need to be retrained; it just needed the right "translator" to show it the cars.

The Limits (Where It Breaks)

The paper is very honest about where this "borrowing" fails. The text brain is great at sequences (A leads to B leads to C) and pattern matching, but it struggles with things that require:

2D Spatial Thinking: Like playing "Game of Life" (a grid-based simulation) or navigating a maze. The brain is wired for a line of text, not a 2D map.
Continuous Precision: Like controlling a robot arm with extreme smoothness or doing complex math with decimals. It's good at discrete steps, not fluid motion.
Stacking Logic: Like balancing parentheses in a complex code structure (Dyck-2). The text brain actually got worse at this than a simple, untrained model, suggesting that learning to read English might have accidentally "blocked" the brain from learning this specific type of logic.

Summary

This paper proves that a brain trained exclusively on text contains a hidden, reusable "operating system" for organizing information. You can take this frozen brain, plug it into a robot or a game, and it will often outperform models built from scratch, simply because it has already learned the deep, structural rules of how things fit together.

It's not magic; it's computational exaptation. Just as feathers evolved for warmth but were later "borrowed" for flight, the brain's ability to copy and match text patterns was "borrowed" to solve robot and memory problems it was never designed to see.

1. Problem Statement

The paper addresses a fundamental question in deep learning: Are the weights of a frontier-scale text-pretrained transformer a general computational substrate that can be "borrowed" for tasks in modalities it has never seen?

Current industry practice treats pretrained Large Language Models (LLMs) as static artifacts for text generation or fine-tuning for text-adjacent tasks. This research hypothesizes that the "geometry" (weight matrices and attention patterns) learned during pretraining encodes low-order, symmetric, and hierarchical computational primitives (e.g., token copying, induction) that are universal to natural-world functions. The goal is to test if these frozen weights, when paired with a minimal trainable interface, can solve non-text tasks (robotics, continuous control, algorithmic reasoning) without modifying the core weights.

2. Methodology

Core Setup

Substrate: The study uses Gemma 4 31B (a hypothetical 2026 release), specifically freezing layers L24–L29 (a 6-layer slice containing ~2.93B parameters). This slice was pretrained exclusively on text tokens (web, code, math).
Interface: A "thin" trainable interface maps non-text inputs (state-action vectors, bit grids, kinematics) into the frozen model's 5376-dimensional hidden space. The interface includes linear encoders, input-statistics matchers, and decoders.
Freezing: The Gemma weights are completely frozen (zero gradients); only the interface parameters are trained.

Experimental Protocol

The paper employs a rigorous set of controls to isolate the contribution of pretraining from architecture and capacity:

Cross-Modality Transfer: Testing on OGBench (robotic manipulation), D4RL (offline RL), and supervised algorithmic tasks (copy, associative recall, cellular automata).
Architecture-Only Controls:
- FrozenRandom-GPT2: A frozen 6-layer transformer with correct $1/\sqrt{d_k}$ scaling but random initialization.
- Random Gemma Slice: The same Gemma architecture with random weights.
- Matched-Capacity Trained Transformer: A transformer trained from scratch with the same parameter count as the frozen slice.
Dual-Measurement Mechanism: A protocol to identify specific attention heads by correlating:
- Text-Activation Probing: Measuring attention to token copying/induction in English sentences.
- Task Ablation: Zeroing out specific heads and measuring performance drop on non-language tasks.

3. Key Contributions

Empirical Validation of "Borrowed Geometry": Demonstrates that frozen text weights can solve complex non-text tasks (robotics, RL) where random-initialized architectures of the same size fail completely.
Mechanistic Identification of Exapted Heads: Identifies specific attention heads (e.g., L26.28, L27.28) that are critical for both text token-copying and non-language algorithmic tasks (binary copy, associative recall), providing evidence of "computational exaptation."
Substrate Isolation: Proves that the performance gain comes from the pretrained weights, not just the architecture or the trainable interface, by showing that random-initialized versions of the same architecture fail to learn these tasks.
Compression and Efficiency: Shows that the frozen substrate can be compressed (e.g., to a 5-layer slice) while maintaining or improving performance compared to larger baselines, achieving Decision Transformer parity with significantly fewer trainable parameters.

4. Key Results

A. Cross-Modality Transfer Success

OGBench (Robotic Manipulation):
- Scene-Play-Task1: The frozen Gemma + IQL heads achieved 97.33% success, beating the published SOTA (GCIQL) by +4.33 points.
- Cube-Double-Play-Task1: The frozen model achieved 60.22% success, whereas the random-initialized Gemma slice failed entirely (0.89%). This represents a +59 point isolation gain, proving pretraining is load-bearing.
D4RL (Offline RL - Walker2d):
- The frozen Gemma slice (L24–L29) achieved a normalized score of 76.2 ± 0.8, matching the Decision Transformer (DT) baseline (74.0) while using only 0.43× the trainable parameters.
- Compression: Dropping layer L24 (using only L25–L29) improved performance to 77.84, demonstrating that the "edge" of the slice is not strictly necessary for this task.

B. Supervised Algorithmic Tasks

Associative Recall (AR): The frozen slice with a 113K-parameter linear interface achieved a per-bit error of 0.0505. A matched-capacity (6.36M) transformer trained from scratch failed completely (error 0.4395), an 8.7× advantage for the frozen model.
Cellular Automata (Rule 90): The frozen model learned the task, while a frozen random transformer with correct scaling stayed at random chance loss for 50k steps.

C. Mechanistic Findings (The "Named Heads")

Using the dual-measurement protocol, the paper identified specific heads that perform the same computation in text and non-text domains:

Head L26.28: Scores 3.7× the slice mean for English token-copying and is the #2 most critical head for binary copy ablation ( $\Delta = +0.221$ ).
Head L27.28: Scores 2.6× for text copying and is the #1 critical head for both binary copy and binary addition.
Head L27.2 & L27.3: Critical for Associative Recall and Cellular Automata, respectively, and also show high text-matching scores.
Conclusion: These heads are "exapted"—shaped by text pretraining but recruited for analogous non-language computations.

D. Limitations and Boundaries

The borrowed geometry is not universal. It fails on:

Dyck-2 (Balanced Brackets): Frozen models plateau at high error rates (~0.03), while from-scratch transformers solve it easily. This suggests pretraining creates an "anti-prior" against explicit stack structures.
2D Spatial Tasks (Game of Life): The 1D sequential primitives do not compose into 2D spatial neighborhood computations.
Continuous Control Precision: Fails on HalfCheetah and Hopper (requiring fine-grained continuous control) and smooth regression tasks (NARMA).

5. Significance

Redefining Pretraining Value: The paper argues that the value of massive pretraining is not just "knowledge" but the discovery of universal computational primitives (like token copying) that are reusable across modalities.
Efficiency: It suggests that for specific tasks, one does not need to train a massive model from scratch; a frozen, compressed slice of a text model can serve as a powerful, general-purpose compute engine.
Mechanistic Interpretability: It bridges the gap between mechanistic interpretability (understanding how models work) and transfer learning. By naming specific heads that transfer, it moves beyond aggregate "frozen models help" claims to specific, verifiable circuit reuse.
Evolutionary Analogy: The concept of Exaptation (feathers evolving for warmth, then used for flight) is applied to neural networks, suggesting that features selected for one domain (language) can be repurposed for another (robotics) without redesign.

6. Conclusion

The paper provides strong empirical evidence that frozen text-pretrained weights act as a general computational substrate. While not a panacea for all tasks (failing on explicit stack or 2D spatial reasoning), the "borrowed geometry" successfully transfers token-matching and pattern-completion primitives to robotics and algorithmic tasks, outperforming both random architectures and matched-capacity from-scratch models. The identification of specific, cross-modal attention heads offers a new lens for understanding the universality of deep learning representations.