Understanding In-Context Learning Beyond Transformers: An Investigation of State Space and Hybrid Architectures
This paper investigates in-context learning across transformer, state-space, and hybrid architectures, revealing that while these models exhibit similar behavioral performance, their internal mechanisms differ significantly, with function vectors playing a crucial role in parametric knowledge retrieval but not necessarily in contextual understanding.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are teaching a child how to solve a puzzle. You don't give them a textbook; instead, you show them three examples of how to do it, and then you ask them to solve a fourth one. This is In-Context Learning (ICL). It's the magic trick where Large Language Models (LLMs) learn new tasks just by looking at examples in the conversation, without needing to be retrained.
For a long time, scientists thought this magic only worked in Transformers (the current standard AI architecture). They found specific "gears" inside the Transformer's brain, called Function Vectors (FVs), that seemed to be the engine driving this learning.
But recently, new types of AI architectures have arrived: State Space Models (like Mamba) and Hybrid models (a mix of Transformers and Mamba). The big question was: Do these new brains use the same gears to learn, or do they have a secret, different engine?
This paper is like a detective story where the authors take apart these different AI brains to see how they work. Here is the breakdown in simple terms:
1. The Two Types of "Learning"
The researchers realized that not all puzzles are the same. They split the tasks into two categories:
- The "Fact Book" Task (Parametric Knowledge): Imagine asking, "What is the capital of France?" The model just needs to pull a fact it already memorized. It's like looking up a word in a dictionary.
- The "Reading Comprehension" Task (Contextual Knowledge): Imagine reading a short story about a dog and then answering, "Is the dog happy?" The model has to understand the story in front of it, not just pull a fact. It's like understanding a joke.
2. The "Gears" (Function Vectors)
In old-school Transformers, scientists found that specific "gears" (attention heads) act like a Function Vector (FV). Think of an FV as a specialized tool that says, "Hey, look at the examples I just saw, copy that pattern, and apply it to the new question."
3. The Big Discovery: Different Brains, Different Engines
The authors tested Transformers, pure Mamba models, and Hybrid models. Here is what they found:
The "Fact Book" Task:
- Transformers & Hybrids: They use those special FV gears heavily. When you tweak these gears, the model's ability to recall facts changes drastically.
- Mamba (Pure): It also uses FV gears, but they are located in different spots.
- Mamba2 (The New Kid): This is the surprise! Mamba2 barely uses FV gears at all. It seems to have a completely different, secret mechanism for learning facts. It's like finding out that while everyone else uses a key to open a door, Mamba2 just walks through the wall.
The "Reading Comprehension" Task:
- Here, the FV gears are not the main heroes. Whether it's a Transformer or a Hybrid, the "gears" responsible for understanding a story are scattered and messy. The model doesn't rely on one specific tool to understand context; it uses a more general, distributed approach.
4. The Hybrid Models: The "Two-Engine" Plane
Hybrid models (like Hymba and Zamba2) are like airplanes with two engines: one Transformer engine and one Mamba engine.
- The researchers found that for learning new tasks, the Transformer engine is doing almost all the heavy lifting.
- Even though the Mamba engine is there for speed and efficiency, it's the Transformer part that holds the "Function Vector" tools needed to learn from examples.
- It doesn't matter if the engines are stacked one on top of the other or side-by-side; the Transformer part is the one holding the "learning" keys.
5. The "Label Flipping" Test
To prove these models were actually learning and not just memorizing, the researchers played a trick. They told the models: "If the example says 'Hate', the answer is actually 'Love'."
- Result: All the models, including the new Mamba ones, were smart enough to figure out the new rule and follow it. This proved they weren't just reciting memorized facts; they were truly adapting to the context.
The Takeaway
This paper tells us that AI is not a monolith. Just because two models give the same answer doesn't mean they are thinking the same way.
- Transformers are like librarians using a specific index card system (FVs) to find facts.
- Mamba2 is like a wizard using a completely different spell to find those same facts.
- Hybrids are like a team where the librarian does the fact-checking, while the wizard handles the rest.
Why does this matter?
If we want to build better, safer, or more efficient AI, we can't just copy-paste what we learned about Transformers onto Mamba models. We need to understand that different architectures have different "internal mechanics." If we want to fix a bug or improve learning, we need to know which specific "gears" to turn for each specific type of brain.
In short: The paper is a manual for understanding the internal wiring of different AI brains, showing us that while they all can learn from examples, they do it with very different tools.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.