🤖 machine learning

First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers

This paper systematically evaluates eight local adaptation properties across nine transformer models and finds that while individual task perturbations remain first-order predictable within a shared validity window, pairwise task composition is highly fragile due to order sensitivity, rotating subspaces, and a lack of universal additivity or steering correspondence.

Original authors: Irina Piontkovskaia, Sergey Nikolenko

Published 2026-07-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Irina Piontkovskaia, Sergey Nikolenko

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master chef who has just perfected a massive, complex stew. It's delicious, but you want to tweak it slightly: maybe add a pinch of salt for a bit more savoriness, or a dash of pepper for a kick. In the world of artificial intelligence, this "stew" is a giant computer brain (a Transformer model) that has already learned to speak, write, and reason. The "pinch of salt" is a tiny adjustment to its internal settings, known as fine-tuning. Scientists have long believed that if you make these adjustments small enough, the changes are predictable, like walking in a straight line on a flat field. They thought that if you wanted to mix two different flavors (like making the AI good at both math and poetry), you could just add the "math adjustments" and the "poetry adjustments" together, and the result would be a perfect blend. This idea, called "local linearity," is the foundation for many modern tools used to customize AI without retraining it from scratch.

However, just like in cooking, the kitchen might not be as flat as it seems. This new research takes a deep, microscopic look at what happens when we make these tiny tweaks to AI brains. The scientists wanted to know: How small does a change need to be to stay predictable? Does adding two changes together always work? And does it matter if you add the salt before the pepper, or the pepper before the salt? They tested this on nine different AI models, ranging from small ones to massive ones with billions of parameters, using a rigorous, pre-planned set of experiments to ensure they weren't just guessing.

The story the paper tells is one of a "one-way street" that suddenly turns into a "traffic jam" when you try to merge lanes. The researchers found that if you push an AI in just one direction (like making it better at a single task), the results are surprisingly predictable and stable. You can walk quite far—up to a scale of $10^{-2}$ in their measurements—before the AI starts to get confused or the math stops working. It's like walking down a long, straight hallway where every step feels exactly the same. This is great news for people who want to do simple, single-task tweaks or use random guessing to find improvements, because the "hallway" is wide and safe.

But the moment you try to do two things at once, or do them in a specific order, the floor gets slippery. The paper reveals that the "additivity" rule (where $A + B = B + A$ ) is fragile. If you try to combine two different task updates, the order in which you apply them matters a lot, and this problem can start happening much sooner than the safe zone for single tasks. In fact, for over a third of the measured model and task combinations, the order starts to matter at a scale strictly inside that safe window. The onset of this sensitivity varies wildly, spanning three orders of magnitude across different pairs, meaning some combinations break down almost immediately while others hold up longer. The scientists discovered that this chaos is caused by something called a "Lie bracket," which is a fancy math way of describing how the curvature of the AI's learning landscape twists when you turn left and then right, versus right and then left.

The team also tested some popular shortcuts that other researchers had been using. They checked if the AI's learning space was "low-dimensional" (like a flat sheet of paper) and found that, contrary to some previous beliefs, it wasn't a simple flat sheet at all; it was more like a crumpled ball of paper that kept changing shape as you moved. They also tested if they could predict the result of a new tweak based on a small sample of data, and found that while the math worked perfectly when using the exact same data for the prediction and the test, it failed completely when they tried to use different data. This means that while we can predict what happens in a controlled, same-data environment, we can't yet reliably predict how a tweak will perform on new, unseen data just by looking at a small sample.

In the end, the paper doesn't say that fine-tuning is broken, but it does say that the "rules of the road" are more complicated than we thought. There is a safe zone for single, straight-line adjustments, but as soon as you try to combine tasks or change the order of operations, you enter a zone where the AI's internal geometry twists and turns in unpredictable ways. The scientists conclude that we need to be much more careful: we can't just assume that adding two fixes together will work, or that the order doesn't matter. Instead, we need to check the specific "curvature" of our specific AI and our specific tasks before we start mixing and matching, because the safe distance for one task might be a disaster for a pair of tasks.

Technical Summary: First-Order Predictable but Pairwise Fragile

Problem Statement
The paper addresses a fundamental gap in understanding the local geometry of trained Transformer models. While various model-editing, merging, steering, and lightweight adaptation methods (e.g., Task Arithmetic, LoRA, activation steering, random search) operate under the assumption that the parameter space around a trained checkpoint is locally linear, additive, or low-dimensional, these assumptions are rarely tested quantitatively across scales and models. Specifically, it is unclear:

Over what displacement scale do these local approximations hold?
Are these scales universal across different model architectures and sizes?
Do single-update properties (e.g., linearity) imply pairwise properties (e.g., additivity or order independence)?

The authors posit that treating "the linear regime" as a single scalar radius is a conflation of distinct geometric phenomena: self-curvature, symmetric cross-interactions, and antisymmetric order effects.

Methodology
The authors conduct a systematic, prospectively registered empirical study ("The Atlas") measuring eight distinct local properties around a single multitask operating point ( $\theta^\star$ ).

Operating Point: A LoRA adapter (rank 16) trained for 300 steps on a uniform mixture of five tasks (SST-2, BoolQ, RTE, ARC-Easy, HellaSwag) across nine models ranging from 82M to 7B parameters (spanning seven architecture families).
Coordinate System: All measurements are performed in the fixed Euclidean coordinates of the LoRA factors (the optimizer's view), acknowledging that this geometry is gauge-dependent but consistent for the study.
The Eight Diagnostics:
1. P1 (Local Linearity): Antisymmetry ratio of probe loss along the gradient.
2. P2 (Low-Dimensionality): Rank of the mixture-gradient covariance.
3. P3 (Tangent-Space Stability): Rotation of the task-gradient subspace over single-task steps.
4. P4 (Additivity): Activation-space composition error of task vectors.
5. P5 (Parameter-Activation Correspondence): Cosine similarity between a weight update's activation shadow and a contrastive activation vector.
6. P6 (Local Searchability): Correlation between realized gain and first-order gradient projection for random perturbations.
7. P7 (Directional Curvature): Ratio of curvature along the gradient vs. a random direction.
8. P8 (Non-Commutativity): The defect (Lie bracket) arising from the order of two sequential gradient steps.

The study employs a strict development/test split, with thresholds and property lists registered prior to data collection. Controls include measurements at the raw pretrained checkpoint and per-task probes.

Key Results

1. Single-Update Predictability (Robust)

First-Order Linearity (P1): Along the task gradient, the probe loss remains first-order predictable (antisymmetry ratio < 0.10) up to a displacement scale of at least $10^{-2}$ on every model median. This window is significantly wider than the scale where pairwise interactions fail.
Searchability (P6): Within this window, random perturbations' effects are highly correlated with their projection onto the gradient ( $r \in [0.70, 1.00]$ ). This validates that local random search works because the loss landscape is locally linear enough to rank proposals.
Curvature (P7): The curvature along the gradient is orders of magnitude larger than along random directions, confirming strong anisotropy.

2. Pairwise Fragility (Failing)

Order Sensitivity (P8): The most critical finding is that pairwise interactions are fragile. The order of two sequential updates matters, and this dependence sets in at scales ( $\eta^\dagger$ $η^{†}$ ) that often fall strictly inside the single-update predictability window.
- The onset scale varies by three orders of magnitude across model-task pairs.
- The defect follows the Lie bracket law: $c(\eta) = \eta\kappa + O(\eta^2)$ , where $\kappa = \|H_B g_A - H_A g_B\| / \|g_A + g_B\|$ .
- When computed on the same minibatches, the measured defect matches the theoretical prediction with a median ratio of 1.002.
Additivity Failure (P4): Activation additivity fails at full task-vector scale ( $\alpha=1$ ) on several models, including both held-out 7B models. Additivity only holds for small fractions ( $\alpha \lesssim 0.3$ ).
Subspace Instability (P3): The task-gradient subspace rotates significantly within 16–80 steps, invalidating the assumption of a static task plane.
Invalidated Claims:
- P2 (Low-Rank Covariance): The mixture-gradient covariance shows no low-rank plateau up to $m=256$ samples; the previously observed rank-32 behavior was a sample-count artifact.
- P5 (Global Correspondence): No model median passes the threshold for a global mean-vector correspondence between weight updates and activation steering vectors.

3. Scale Discrepancy
The paper demonstrates that there is no universal radius for adaptation. The "one-direction" window (where single updates are predictable) is roughly 1–2 orders of magnitude larger than the "two-direction" window (where pairwise order effects become significant). For over a third of tested cells, pairwise order sensitivity breaks down at displacements strictly inside the single-update linear regime.

Significance and Claims
The paper claims to provide the first quantitative, cross-model atlas of local adaptation properties, moving beyond qualitative assertions to measured falsifiable statements.

Operation-Specific Validity: The authors argue that "the linear regime" is not a single property. Self-curvature, symmetric composition, and antisymmetric order effects are governed by different tensors with different scales. A method relying on additivity (e.g., model merging) cannot assume the same validity scale as a method relying on first-order search.
The Lie Bracket as a Diagnostic: The paper establishes that the Lie bracket of two task gradients ( $H_B g_A - H_A g_B$ ) provides a precise, coordinate-specific consistency fact for update-order sensitivity. It can be estimated via two Hessian-vector products (HVPs) to predict the onset of non-commutativity ( $\eta^\dagger \approx 0.10/\kappa$ ) with high accuracy when estimators are matched.
Limits of Transfer: While the matched-estimator Lie bracket law holds, the paper explicitly reports that cross-fitted predictions fail. Estimating the bracket on one data sample does not predict the endpoint or task effects of updates built on different samples. This suggests that while the local geometry is consistent within a specific batch, it is not stable across data distributions in a way that allows for generalizable forecasting.
Practical Guidance: Practitioners should not assume a universal step size or radius. Instead, they must match the diagnostic to their specific operation:
- For single-direction search, linearity holds up to $\sim 10^{-2}$ .
- For sequential updates, order sensitivity is pair-specific and can occur at much smaller scales; checking the Lie bracket is necessary.
- Assumptions of low-rank mixture covariance or static task subspaces should be discarded based on these results.

The paper concludes that while a shared first-order window makes local random search predictable, pairwise task interactions are often fragile at much smaller, pair-specific scales, and the local calculus of adaptation is operation-specific rather than universal.

Technical Summary: First-Order Predictable but Pairwise Fragile

More like this