⚡ electrical engineering

LF ${}^{2}$ AR: Accounting for Layerwise Dynamics to Improve Multimodal Adaptation of Language Models

The paper introduces LF ${}^{2}$ AR, a novel architecture that improves multimodal adaptation of language models by leveraging layerwise abstraction-refinement dynamics through late modality fusion, late fission with attention residuals, and selective access mechanisms, resulting in enhanced cross-modal alignment, reasoning performance, and efficient early-exit decoding.

Original authors: Santiago Cuervo, Adel Moumen, Yanis Labrak, Sameer Khurana, Antoine Laurent, Mickael Rouvier, Phil Woodland, Ricard Marxer

Published 2026-08-11

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Santiago Cuervo, Adel Moumen, Yanis Labrak, Sameer Khurana, Antoine Laurent, Mickael Rouvier, Phil Woodland, Ricard Marxer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant librarian who has spent their entire life reading millions of books. They know the rules of grammar, the structure of stories, and the meaning of words better than anyone else. Now, imagine you want this librarian to do more than just read text; you want them to "read" a picture of a sentence or "listen" to a spoken story and then write a reply. The problem is that text is like a neatly organized library where every book has a clear title, but pictures and sounds are like a chaotic pile of puzzle pieces. A single word in a book might take up a whole page of pixels or a hundred milliseconds of sound waves. If you just hand the librarian a pile of puzzle pieces, they get confused because their brain is wired to process neat, abstract words, not raw, messy details. This is the challenge scientists face when trying to teach text-trained AI models to understand and generate images and speech. They need a way to translate that messy, fine-grained sensory data into the librarian's clean, abstract language, and then translate the librarian's abstract thoughts back into messy sensory details without losing the magic of what they learned from reading.

This paper, titled LF²AR, proposes a clever new way to build these AI translators by looking at how the "brain" of a language model actually works. The authors noticed that these models don't just process information in a straight line; they have a two-step dance. First, they take small, messy details and combine them into big, abstract ideas (like turning individual letters into a word). Then, later in the process, they take those big ideas and refine them back into specific details to predict what comes next. The researchers suggest that when dealing with images or speech, we need to add a special "pre-processing" step before the main brain to clean up the messy input, and a special "post-processing" step after the brain to handle the messy output. They call this Late Fusion (cleaning up the input) and Late Fission (handling the output).

But here's the really cool part: they realized that sometimes the model needs to peek at the "big ideas" and sometimes it needs to focus on the "tiny details," depending on what it's doing at that exact moment. To solve this, they invented a mechanism called Attention Residuals. Think of it like a smart remote control that lets the output layer instantly switch between looking at the high-level summary and the low-level details, rather than forcing the model to carry every tiny detail through the whole brain.

The team tested this idea on two specific tasks: turning text into images (where the text is rendered as a picture) and turning text into speech. They found that their new architecture, LF²AR, worked much better than the standard methods. It helped the model understand the connection between words and sounds/images more clearly, kept the model's ability to reason in text intact, and even made the model faster. By using their "smart remote," the model learned to skip unnecessary deep layers when it didn't need them, making it generate speech 1.9 times faster without losing accuracy. The results suggest that by respecting how these models naturally organize information—first abstracting, then refining—we can make them much better at handling the messy, real-world world of sights and sounds.

Technical Summary: LF²AR – Accounting for Layerwise Dynamics to Improve Multimodal Adaptation of Language Models

Problem Statement

Text-pretrained Large Language Models (LLMs) possess rich world knowledge, yet adapting them to process and generate perceptual modalities (such as audio and images) while effectively leveraging that knowledge remains challenging. The core difficulty lies in a mismatch in representational granularity. Text tokens are semantically dense, whereas perceptual signals (pixels, audio frames) are finer-grained, lower-level, and sparsely distributed across spatial or temporal dimensions.

Existing approaches often rely on complex multi-stage pipelines or "early fusion" architectures (lightweight linear mappings) that treat modalities monolithically. However, these methods frequently result in imperfect transfer, where models underperform on non-textual inputs or suffer from catastrophic forgetting of text-only capabilities. The authors posit that these failures stem from a failure to account for the internal layerwise abstraction-refinement dynamics inherent in transformer LMs.

Methodology: LF²AR

The paper proposes LF²AR (Late Fusion and Late Fission with Attention Residuals), an architecture designed to align multimodal processing with the natural computational flow of text-pretrained transformers. The approach is grounded in the observation that transformer layers exhibit a two-phase dynamic:

Abstraction Phase: Early layers compose lower-level features into abstract, compositional representations.
Refinement Phase: Later layers refine these abstractions into lower-level features predictive of fine-grained structure.

Since perceptual modalities are finer-grained than text, the authors hypothesize that effective adaptation requires additional processing stages to match these dynamics:

1. Late Fusion (Input-side)

To handle the fine-grained nature of perceptual inputs, LF²AR adds composition layers before the pretrained backbone. These layers perform fine-to-coarse processing, transforming raw perceptual signals (e.g., image patches or speech tokens) into abstract representations that align with the early layers of the text-pretrained model. This ensures the backbone receives inputs at an appropriate level of abstraction.

2. Late Fission (Output-side)

To generate perceptual outputs, LF²AR adds refinement layers after the backbone. These layers perform coarse-to-fine processing, translating the backbone's high-level semantic predictions back into the specific, low-level details required by the target modality.

3. Attention Residuals

A key innovation is the use of attention residuals within the fission mechanism. Instead of forcing low-level details to propagate through the entire backbone (which may interfere with high-level feature formation), the fission module is given dynamic, input-dependent access to the residual stream states of all backbone layers.

The module constructs a summary of layer representations.
A selector network generates attention weights to form a weighted residual summary ( $\bar{c}_i$ ).
This allows the fission module to selectively access high-level semantic structure (for anticipating the next word) or low-level local detail (for predicting specific phonemes or pixels) as needed.

Experimental Setup

The authors evaluated LF²AR on two modalities where semantic correspondence to text can be tightly controlled:

Text-as-Images: Pixel language modeling where text is rendered as binary image patches.
Speech: Vector-quantized self-supervised speech representations.

Experiments were conducted on SmolLM backbones ranging from 135M to 1.7B parameters. The study utilized extensive ablations, comparing LF²AR against:

Base: Early fusion/fission models (standard lightweight adaptation).
Ablated Variants: Models with only late fusion, only late fission, or static (non-dynamic) residuals.

Evaluation included layerwise representation analysis (intrinsic dimensionality, cross-modal similarity, BERT alignment, perplexity probes) and downstream benchmarks (MMLU, HellaSwag, StoryCloze) in various cross-modal configurations (e.g., Image $\to$ Text, Speech $\to$ Text).

Key Results

Representation Dynamics

Abstraction & Alignment: LF²AR consistently achieved higher dataset intrinsic dimensionality and word-level cross-modal similarity compared to the Base model. Late fusion was found to induce more text-like early-layer features, leading to better semantic intermediate representations.
Preservation of Text Structure: Late fission prevented the deepest layers of the backbone from becoming overly specialized for fine-grained prediction, thereby preserving the pretrained model's ability to predict the next text token.
Semanticity: LF²AR image and speech representations were as semantic and predictive as text representations, a property often lost in standard adaptations.

Downstream Performance

Cross-Modal Gains: LF²AR significantly outperformed Base models, particularly in cross-modal reasoning tasks (e.g., answering text questions based on image or speech input).
Scaling Efficiency: The architectural improvements allowed LF²AR to outperform larger baseline models. For instance, a 150M LF²AR model was competitive with or stronger than a 360M Base model, and a 400M LF²AR consistently outperformed a 1.7B Base in cross-modal transfer.
Efficiency: The attention residuals induced a sparse, interpretable use of backbone depth. The model learned to consult high-level textual features only at specific points (correlated with word boundaries).

Early-Exit Decoding

Leveraging the sparsity of attention residuals, the authors implemented a dynamic depth allocation mechanism. By training a lightweight router to decide whether a token requires deep backbone processing or can be handled by a prefix-only approximation, they achieved a 1.9× generation speedup (throughput increase) with negligible performance loss.

Significance and Claims

The paper claims that effective multimodal adaptation requires matching the layerwise organization learned during text pretraining rather than treating the backbone as a static feature extractor.

Architectural Insight: The study demonstrates that adding modality-specific layers at the input (for abstraction) and output (for refinement) is superior to monolithic early fusion.
Dynamic Access: The use of attention residuals allows the model to dynamically switch between coarse semantic guidance and fine perceptual detail, resolving the tension between preserving high-level text knowledge and generating low-level perceptual signals.
Efficiency: The proposed architecture not only improves accuracy but also enables compute-efficient inference through variable-depth allocation, suggesting that multimodal capabilities can emerge at smaller parameter scales than previously thought possible.

The authors conclude that these design choices consistently improve compositionality, cross-modal alignment, and the preservation of text-like predictive structure, offering a principled path forward for adapting LLMs to finer-grained modalities.