💬 NLP

Improving LLM Predictions via Inter-Layer Structural Encoders

This paper introduces Inter-Layer Structural Encoders (ILSE), a novel approach utilizing Cayley-Encoder to aggregate information from intermediate LLM layers, which significantly outperforms standard final-layer predictions across diverse tasks and enables smaller models to compete with larger ones.

Original authors: Tom Ulanovski (Tel Aviv University), Eyal Blyachman (Tel Aviv University), Maya Bechler-Speicher (Meta)

Published 2026-03-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tom Ulanovski (Tel Aviv University), Eyal Blyachman (Tel Aviv University), Maya Bechler-Speicher (Meta)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, incredibly smart library (the Large Language Model, or LLM) that knows almost everything about language. When you ask it a question, it doesn't just give you an answer from one specific book. Instead, it reads through thousands of books, layer by layer, gathering clues.

The Old Way: The "Final Page" Problem
Traditionally, when people used these AI libraries, they only looked at the very last page of the final book to make a decision. They assumed that the most important information was always at the very end.

But the authors of this paper realized something: That's a waste!

The early pages (lower layers) are great at understanding simple grammar and spelling.
The middle pages are good at understanding sentence structure.
The later pages are great at understanding deep meaning and context.

By only looking at the last page, the AI is ignoring a treasure trove of useful clues hidden in the middle. It's like trying to solve a mystery by only reading the final paragraph of a novel, ignoring all the detective work done in chapters 1 through 10.

The New Solution: ILSE (The "Team Huddle")
The authors created a new method called ILSE (Inter-Layer Structural Encoders). Think of ILSE as a super-efficient team huddle.

Instead of ignoring the other chapters, ILSE gathers the "notes" from every single layer of the library and forces them to talk to each other to come up with one perfect answer.

The Secret Weapon: The Cayley Graph (The "Magic Roundabout")
To make this team huddle work without it getting messy or chaotic, they used a special mathematical structure called a Cayley Graph.

Here is a simple analogy:
Imagine you have a group of people (the layers) standing in a circle.

The Old Way: Everyone only talks to the person next to them. If you are at the far end of the circle, it takes a long time for your message to reach the other side.
The ILSE Way: They use a "Magic Roundabout" (the Cayley Graph). This is a special arrangement where everyone is connected in a way that allows information to zip from one person to anyone else almost instantly, without getting stuck in traffic jams.

This ensures that the "grammar notes" from the beginning of the book mix perfectly with the "meaning notes" from the end, creating a much smarter final decision.

Why is this a big deal?
The paper tested this on 13 different tasks (like figuring out if a tweet is angry or happy, or if two sentences mean the same thing) using 9 different AI models of various sizes.

It's a massive upgrade: ILSE made the AI significantly smarter. In some cases, it improved accuracy by 44%. That's like turning a B-student into an A+ student just by teaching them how to organize their notes better.
It's cheap: You don't need to rebuild the whole library or hire new staff. You just add a small, efficient "manager" (the ILSE encoder) on top of the existing AI. It adds almost no extra cost or memory.
It works with small models: Usually, you need a giant, expensive AI to get good results. But ILSE is so good at organizing information that it can make a tiny AI (with only 14 million parameters) perform as well as a huge AI (with 2.8 billion parameters). It's like giving a bicycle the speed of a motorcycle through better aerodynamics.
It learns fast: Even if you only show the AI a few examples (like 32 sentences), ILSE learns the task much faster than other methods.

In a Nutshell
The paper says: "Stop ignoring the middle chapters of the story!" By using a clever mathematical trick to let all parts of the AI talk to each other efficiently, we can make existing AI models much smarter, faster, and more efficient without needing to build bigger, more expensive ones.

1. Problem Statement

Standard practice in Large Language Models (LLMs) relies on the final-layer token representations for downstream tasks (e.g., classification, semantic similarity). However, recent research indicates that:

Intermediate layers often contain substantial, task-relevant information that is lost or diluted in the final layer.
The optimal layer for a specific task is not fixed; it varies depending on the task (e.g., lower layers capture syntax, upper layers capture semantics).
Existing methods for aggregating layer information have limitations:
- ELMo/Weighted Sums: Only capture linear combinations, missing complex non-linear interactions.
- Depth-Wise Attention (DWAtt): Anchors attention to the final layer, failing to treat all layers equally.
- Architecture Modification: Methods that alter the transformer structure require training from scratch, which is computationally infeasible for pre-trained models.

The core challenge is how to effectively combine representations from all internal layers of a frozen LLM into a single, enhanced representation without modifying the base model architecture.

2. Methodology: Inter-Layer Structural Encoders (ILSE)

The authors propose ILSE, a framework that treats the set of layer representations as a structured graph and uses Graph Neural Networks (GNNs) to learn an optimal aggregation.

Core Workflow

Layer Representation: For an LLM with $L$ layers and input sequence $T$ , the hidden states $H_\ell$ are mean-pooled over tokens to create a single vector $z_\ell \in \mathbb{R}^d$ for each layer $\ell$ .
Structural Encoding: These $L$ $L$ vectors are mapped to nodes in a graph. The paper explores three specific graph topologies:
- Set-Encoder: Treats layers as an unordered set (empty graph). Uses DeepSets (permutation-invariant pooling) to aggregate.
- Fully-Connected (FC) Encoder: Connects every layer to every other layer (dense graph). Uses a GNN to allow direct communication between any pair of layers.
- Cayley-Encoder (The Novel Contribution): Maps layers to nodes in a Cayley Graph over the Special Linear Group $SL(2, \mathbb{Z}_n)$ $S L (2, Z_{n})$ .
  - Why Cayley Graphs? They are regular graphs (all nodes have the same degree) and possess logarithmic diameter. This ensures bottleneck-free information flow, allowing efficient global communication between any two layers without the "over-squashing" problem common in standard GNNs.
  - Implementation: If the number of layers $L$ does not match the graph size $|V_n|$ , virtual nodes (zero vectors) are added to preserve the graph's expansion properties. The GNN processes the graph, and only the representations of the non-virtual nodes are used for the final prediction.
Prediction: The final aggregated representation is fed into a task-specific head (linear classifier or similarity head). The base LLM remains frozen; only the encoder and head are trained.

3. Key Contributions

ILSE Framework: A mathematically grounded, efficient method to fuse inter-layer representations using structural encoders (DeepSets and GNNs) without modifying the pre-trained LLM.
Cayley-Encoder: The introduction of Cayley graphs (specifically over $SL(2, \mathbb{Z}_n)$ ) as a structural prior for layer aggregation. This leverages the "bottleneck-free" property of expander graphs to maximize information flow between layers.
Comprehensive Evaluation: Extensive testing across 13 tasks (5 classification, 8 Semantic Text Similarity) and 9 LLMs (ranging from 14M to 8B parameters).
Data Efficiency: Demonstration that ILSE is highly effective in few-shot regimes (as few as 32 samples per label), often outperforming baselines trained on full datasets.
Small Model Empowerment: Showing that ILSE can make small LLMs (e.g., 14M parameters) competitive with significantly larger models (e.g., 2.8B parameters).

4. Experimental Results

The paper evaluates ILSE against baselines: Last-Layer, Best-Single-Layer, Weighted Sum, MLP Last-Layer, and DWAtt.

Classification Tasks:
- ILSE achieved the highest score in all 15 configurations (3 models $\times$ 5 tasks).
- Improvements: Average gains of +30% over Last-Layer and +25% over Best-Layer.
- Emotion Task: ILSE showed massive improvements, up to +44% over Last-Layer.
- Cayley-Encoder was particularly dominant on Gemma2-2B and Llama3-8B, while FC-Encoder excelled on Pythia-410M.
Semantic Text Similarity (STS):
- ILSE outperformed baselines in 17 out of 21 zero-shot transfer combinations.
- Cayley-Encoder achieved the best score in 17 out of 24 STS configurations, suggesting that sparse, regular connectivity is ideal for semantic similarity.
Parameter Efficiency:
- ILSE adds a negligible number of trainable parameters (approx. 0.01% to 0.1% of the base model size).
- It uses 3-5x fewer parameters than DWAtt while achieving superior performance.
Few-Shot Learning:
- With only 32 samples per label, ILSE (specifically Cayley-Encoder and FC-Encoder) surpassed all other baselines, including those trained on full datasets (e.g., ~10,000 samples).
Model Scaling:
- ILSE consistently outperformed baselines across all Pythia model sizes (14M to 2.8B).
- A 14M-parameter model with ILSE reached performance levels comparable to a 2.8B-parameter model without ILSE on specific tasks (e.g., MTOP Domain).

5. Significance and Future Work

Theoretical Insight: The results challenge the assumption that the final layer is always the most informative. They prove that structured aggregation of intermediate layers unlocks complementary signals that are otherwise inaccessible.
Practical Impact: ILSE offers a "plug-and-play" solution to boost frozen LLMs. It is particularly valuable for:
- Low-resource scenarios where labeled data is scarce.
- Edge deployment, where small models need to be made competitive with larger ones without retraining the massive base model.
Future Directions: The authors suggest exploring ILSE in generative contexts (to mitigate hallucinations), integrating these structural encoders directly into the pre-training phase ("geometry-aware" architectures), and applying Cayley aggregation at the token level within layers.

In conclusion, ILSE demonstrates that treating the internal depth of an LLM as a structured graph, specifically using Cayley graphs to ensure efficient information propagation, is a powerful, data-efficient, and mathematically sound approach to improving LLM predictions.