🤖 machine learning

Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks

This paper demonstrates that while standard transformers are rarely optimal for specific tasks, optimizing their non-linearities reveals that highly task-specific architectures can significantly improve performance on algorithmic tasks, whereas more universally compatible designs offer modest gains across language and code domains.

Original authors: Damien Teney, Liangze Jiang, Hemanth Saratchandran, Simon Lucey

Published 2026-07-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Damien Teney, Liangze Jiang, Hemanth Saratchandran, Simon Lucey

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to think. For a long time, scientists have been using the same basic blueprint for these robots, called a "Transformer." Think of a Transformer like a universal Swiss Army knife: it has a specific set of tools (mathematical functions) that work surprisingly well for almost everything, from writing poetry to translating languages. Because this one design works so broadly, many people started to believe it might be the "perfect" tool for every job, and that the only way to get better results was to just make the robot bigger and feed it more data.

However, there is a catch. Just because a Swiss Army knife can open a bottle and cut a rope doesn't mean it's the best tool for either job. A dedicated corkscrew opens bottles faster, and a sharp chef's knife cuts rope more cleanly. In the world of artificial intelligence, these "best tools" are called inductive biases. You can think of an inductive bias as a robot's natural instinct or a built-in shortcut that helps it learn a specific type of pattern quickly. The big question scientists are asking is: Is the Transformer's "Swiss Army knife" instinct actually the best one for every single task, or are we forcing the robot to do things it wasn't naturally designed to do?

This paper, titled "Can Transformers Really Do It All?", sets out to find the answer. The researchers didn't just guess; they built a clever experiment to see if they could design a "custom tool" for specific jobs. They took the standard Transformer and replaced its core "thinking" parts (the non-linear functions that decide how to process information) with blank slates. Then, they let the computer learn the perfect shape for these parts specifically for one task, like a video game or a math problem, using a special training method that prevents the robot from just memorizing the answers. Once they found the perfect custom shape, they froze it and tested how well that new design worked on other tasks.

The results were a mix of "wow" and "whoa." When the researchers tested these custom designs on algorithmic tasks—like teaching a robot to do math, remember a list of numbers, or check if parentheses are balanced—the results were dramatic. The custom robots learned these tasks 2 to 3 times faster, were much more stable (they didn't fail randomly), and could generalize to longer sequences that they hadn't seen before. However, there was a huge trade-off: these custom designs were incredibly picky. A robot optimized to do math was terrible at remembering lists, and vice versa. It turned out that for these logical, rule-based tasks, the standard Transformer is actually quite clumsy, and the "perfect" tool for one math problem is useless for another.

The story was different when they looked at language and code. When they tried to optimize the robots for writing stories, Shakespeare plays, or computer code, the custom designs did perform slightly better than the standard ones, but the improvement was small and consistent. More importantly, these custom designs for language were much more flexible; a robot optimized for English could still handle computer code reasonably well. This suggests that for language, the standard Transformer is already very close to being a "local optimum"—a pretty good spot that is hard to beat without making huge changes.

Ultimately, the paper suggests that while the Transformer is a fantastic general-purpose tool, it isn't the perfect solution for everything. For strict logical tasks like math and algorithms, the standard design is far from optimal, and we might need to build very different, specialized robots for those jobs. For language, the standard design is already quite good, and while we can tweak it to be slightly better, we aren't missing out on a massive hidden potential. The takeaway is that there is no single "magic bullet" architecture that is perfect for every human skill; sometimes, to get the best results, we need to stop trying to force one tool to do it all and start designing the right tool for the specific job.

Technical Summary: Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks

Problem Statement

While Transformers have become the dominant architecture across diverse modalities (vision, language, speech), their universality is questioned by their inconsistent performance. They excel at language modeling but struggle with elementary algorithmic tasks like arithmetic or systematic generalization. This raises two critical questions:

Are Transformers uniquely endowed with generic inductive biases suitable for all data, or are they suboptimal for specific tasks?
Are the inductive biases required for different domains (e.g., natural language vs. algorithmic reasoning) compatible, or do they necessitate distinct architectural designs?

Current reliance on scaling models and data suggests a need to understand if architectural improvements, rather than just scale, are required to push AI beyond current limitations.

Methodology

The authors propose a method to optimize Transformer architectures for specific datasets to identify optimal task-specific inductive biases. The core innovation involves replacing standard non-linearities with learnable, parametrized functions optimized on held-out data.

Core Mechanism

Replaceable Components: The method targets the primary non-linearities in Transformers:
- MLP Layers: Replaces the GeLU activation with a 1D linear spline ( $\phi_{\theta_{MLP}}$ ) parametrized by learnable keypoints.
- Attention Layers: Replaces the standard softmax with a learnable non-linearity ( $\phi'_{\theta_A}$ ) applied to the query-key similarity, effectively learning a custom kernel function.
Two-Stage Optimization:
1. Stage I (Architecture Optimization): The model is trained on a dataset $D$ using a two-loss mechanism. A fraction of the data (e.g., 20%) is held out specifically to optimize the spline parameters ( $\theta_A, \theta_{MLP}$ ), while the remaining data optimizes the model weights. To prevent co-adaptation (where splines overfit specific weights or random seeds), the authors employ multi-model training: $M$ models with different random seeds share the same optimized non-linearities.
2. Stage II (Evaluation): The optimized non-linearities are frozen. New models are trained from scratch on the target dataset (either $D$ or a different dataset $D'$ ) using standard training procedures. This isolates the inductive bias of the architecture from the specific weights learned during Stage I.

Rationale for Splines

Linear splines are chosen because they offer an unbiased, tractable parametrization for functions $\mathbb{R} \to \mathbb{R}$ . Unlike small MLPs or smooth priors used in prior work, splines can represent sharp transitions, step functions, or complex waveforms without enforcing smoothness or monotonicity constraints.

Key Contributions

Optimization Framework: A method to optimize Transformer non-linearities for any dataset, treating the resulting architecture as a set of pre-tuned hyperparameters.
Algorithmic Task Analysis: Application of the method to algorithmic reasoning tasks (e.g., modular addition, Dyck languages, needle-in-a-haystack).
Language Modeling Analysis: Application to natural language and code datasets (e.g., TinyStories, Shakespeare, CodeSearchNet) at various tokenization levels.
Cross-Task Compatibility Study: A systematic evaluation of whether architectures optimized for one task transfer effectively to others, measuring the compatibility of inductive biases.

Key Findings

1. Algorithmic Reasoning Tasks

Dramatic Improvements: Optimized architectures significantly outperform standard Transformers on algorithmic tasks. Improvements include:
- Faster Convergence: Training speed increases by 2–3 $\times$ on tasks like ADD and MANO.
- Reduced Variance: The high variance in training speed and accuracy across random seeds observed in baselines is eliminated.
- Better Generalization: Models achieve perfect test accuracy where baselines fail, even when fitting training data perfectly.
- Length Generalization: On the COPY task, optimizing an Alibi-based architecture with learned non-linearities extends generalization to longer sequences beyond the training distribution.
Task Specificity: The optimized architectures are highly task-specific. An architecture optimized for "Add" does not necessarily help "Parentheses" and may even perform worse than a standard Transformer on unrelated tasks. This suggests algorithmic tasks require inductive biases fundamentally different from standard Transformers.

2. Language Modeling Tasks

Modest Improvements: On natural language and code datasets, optimized architectures yield consistent but smaller improvements compared to baselines.
Component Specificity: Optimizing MLP non-linearities (GeLUs) provides gains, whereas optimizing attention non-linearities (Softmax) yields negligible or negative results, suggesting Softmax is already near a local optimum for language.
Cross-Dataset Transfer: Unlike algorithmic tasks, optimized architectures for language modeling transfer well across different datasets (e.g., from TinyStories to Shakespeare) and tokenization schemes (character vs. subword).
Code vs. Language: Gains are slightly larger for code datasets than for natural language, suggesting code modeling shares more structural similarities with algorithmic tasks.

3. Implications for Inductive Biases

Suboptimality of Standard Transformers: Standard Transformers are rarely a local optimum in the architecture space for specific tasks. Simple alternatives (e.g., learned splines) can significantly outperform them.
Trade-off: While task-specific designs offer superior performance on narrow tasks, they sacrifice universality. Standard Transformers appear to be a "good enough" compromise that supports a wide range of capabilities (fluency, reasoning) without being optimal for any single one.
No Free Lunch: The results align with the No-Free-Lunch theorem, indicating that different domains benefit from different inductive biases.

Significance and Claims

The paper argues that the current dominance of Transformers is likely due to their ability to handle a broad spectrum of tasks reasonably well, rather than being the optimal solution for any specific task. The authors claim:

Architectural Limits: The deficiencies of Transformers in systematic generalization and algorithmic reasoning are partly due to misaligned inductive biases, not just a lack of scale.
Methodological Utility: The proposed method serves as a diagnostic tool to probe the limits of current architectures and identify where specific inductive biases are missing.
Future Directions: The findings suggest that future AI systems aiming for both fluency and robust reasoning may require architectures that can better support multiple, potentially conflicting, inductive biases simultaneously, or that domain-specific models may be necessary for tasks requiring high precision (like algorithmic reasoning).

The authors remain modest, noting that their experiments are small-scale relative to state-of-the-art LLMs and that the optimized non-linearities can be computationally expensive (though polynomial approximations can mitigate this). They do not claim their specific spline designs are the final solution, but rather evidence that the search space for better architectures is vast and under-explored.