🤖 AI

Multimodal Function Vectors for Visual Relations

This paper demonstrates that specific attention heads in Large Multimodal Models encode visual relational knowledge as manipulatable "function vectors," which can be extracted, fine-tuned, and linearly combined to significantly enhance zero-shot performance, in-context learning, and generalization on relational reasoning tasks.

Original authors: Shuhao Fu, Esther Goldberg, Ying Nian Wu, Hongjing Lu

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shuhao Fu, Esther Goldberg, Ying Nian Wu, Hongjing Lu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but slightly confused, robot assistant. You show it a picture of a kitchen and ask, "What is the boy doing?"

If you just show the picture, the robot might see a disconnected list of items: fridge, boy, cabinet, sink. It struggles to understand the story connecting them. If you give it a few examples first (like, "In this picture, the boy is holding a cup; in that one, the girl is sitting on a chair"), it gets a little better at guessing. This is called "in-context learning," but even with examples, the robot is still guessing in the dark.

This paper is about finding a "magic switch" inside the robot's brain that helps it understand these connections (like "holding," "sitting on," or "next to") without needing to guess.

Here is how the researchers did it, using simple analogies:

1. Finding the "Special Neurons"

The robot's brain is made of billions of tiny processing units called "attention heads." Think of these like thousands of tiny radio stations. Most of them are playing static or music about colors and shapes.

The researchers used a special tool (called Causal Mediation Analysis) to listen in on these stations. They discovered that only a tiny handful of these radio stations (about 10 out of thousands) are actually broadcasting the "relationship" signal. These are the specific channels that know what "above," "below," or "carrying" means.

2. Creating the "Function Vector" (The Magic USB Drive)

Once they found those special radio stations, the researchers recorded their "voice" when the robot was looking at pictures with clear relationships. They averaged these voices together to create a single, compact digital file called a Function Vector.

Think of this vector like a USB drive containing a specific instruction manual.

If you plug in the "Above" USB drive, the robot suddenly knows how to spot things that are on top of other things.
If you plug in the "Carrying" USB drive, it instantly understands who is holding what.

The cool part? They didn't have to re-teach the robot. They just plugged this "USB drive" into the robot's brain while it was looking at a new picture. Suddenly, the robot's performance jumped up, even without any examples to show it first.

3. Tuning the Signal (Fine-Tuning)

The first version of this "USB drive" was good, but the researchers found they could make it even better. They took a small amount of new practice data and tweaked the file slightly (like adjusting the volume or clarity on a radio).

After this quick "tuning," the robot became much smarter at spotting relationships. It didn't need to memorize the whole world; it just needed this specific, optimized instruction file to guide its thinking.

4. Mixing and Matching (The Analogy Trick)

The most magical part of the paper is what happens when the robot faces a relationship it has never seen before, like "above and to the right."

The researchers showed the robot a picture of something "above" and something "to the right." They took the "Above" USB drive and the "Right" USB drive, mixed them together in a specific ratio (like mixing blue and yellow paint to get green), and created a brand new "Above-Right" USB drive.

When they plugged this new, mixed drive into the robot, it could solve a puzzle involving "above-right" relationships it had never been trained on. It was like the robot learned to combine two known concepts to understand a brand new one, just by mixing the digital instructions.

The Big Takeaway

The paper proves that these huge, complex AI models aren't just black boxes. Inside them, there are specific, small, organized parts that handle specific jobs (like understanding relationships).

Before: The robot had to guess based on vague hints.
Now: We can find the specific "relationship switch," pull it out, tweak it, and plug it back in to make the robot understand the world much better.

This helps us understand how these AI models work and gives us a way to control them more precisely, making them better at seeing the connections between things in a picture.

Technical Summary: Multimodal Function Vectors for Visual Relations

Problem Statement

Large Multimodal Models (LMMs) exhibit impressive in-context learning (ICL) capabilities, yet the internal mechanisms enabling them to learn and execute relational reasoning tasks remain opaque. While recent mechanistic interpretability research in Large Language Models (LLMs) has identified "function vectors" (FVs)—compact representations of tasks derived from attention head activations—extending this approach to multimodal settings is challenging. Visual relations (e.g., spatial positioning like "above" or agentic interactions like "holding") are not explicitly signaled by pixels; they must be inferred from the fusion of high-dimensional visual features and natural language. Current LMMs struggle to acquire systematic relational concepts even with demonstrations, often failing to outperform zero-shot baselines significantly. This paper investigates whether the function vector paradigm can be extended to LMMs to extract, manipulate, and generalize visual relational knowledge.

Methodology

The authors propose a framework to identify and extract relation-specific function vectors from the internal representations of LMMs, specifically focusing on OpenFlamingo and Qwen3-VL.

1. Datasets and Task

The study utilizes two datasets:

Synthetic Dataset: 7,000 images generated from object cutouts (Big and Small Objects dataset) to instantiate four spatial relations (above, below, left of, right of).
Real Image Dataset (GQA): Subsets of the GQA dataset containing real-world images annotated with spatial relations (7 types) and agentic relations (5 types: carrying, holding, riding, sitting on, wearing).

The evaluation task is a 4-shot ICL prompt where the model must identify an object that satisfies a specific relation with a query object.

2. Causal Mediation Analysis for Vector Extraction

To identify the internal components responsible for relational reasoning, the authors employ causal mediation analysis:

Activation Averaging: They compute the average activations of attention heads across a set of consistent in-context demonstrations for a specific relation.
Causal Indirect Effect (CIE): They construct perturbed prompts with uninformative contexts (random object pairings) and measure the change in prediction probability when a specific attention head's activation is replaced by the relation-specific mean activation.
Function Vector Definition: Attention heads with the highest Average Indirect Effect (AIE) scores are identified as the causal subnetwork. The function vector ( $v_t$ ) for a relation $t$ is defined as the sum of the mean activations of the top 10 most influential attention heads.

3. Intervention and Fine-Tuning

Zero-Shot Intervention: The extracted function vectors are injected into the hidden states of zero-shot prompts (containing no demonstrations) to test if they can causally elicit correct relational predictions.
Fine-Tuning: The vectors are further optimized using a small set of zero-shot training examples (1,000 synthetic or ~2,100 GQA images) while keeping the LMM parameters frozen. This "Fast Learning" component updates only the function vector to maximize the likelihood of correct answers.

4. Composite Function Vectors for Analogy

Inspired by the linear representation hypothesis, the authors test if untrained relations (e.g., "above-left") can be solved by linearly combining vectors of trained base relations (e.g., "above" + "left"). They compute weights based on a single source analogy example to form a composite vector ( $v_{composite}$ ), which is then used to guide inference on a target analogy.

Key Results

1. Identification of Causal Subnetworks

Causal mediation analysis revealed that only a small, sparse subset of attention heads (concentrated in intermediate layers) exerts a strong causal influence on relational predictions. Injecting the extracted function vectors into these specific layers significantly improved zero-shot accuracy compared to the baseline.

2. Performance Gains via Fine-Tuning

Fine-tuning the function vectors on zero-shot data yielded substantial improvements over standard baselines:

Synthetic Dataset: Fine-tuned vectors (FFVs) significantly outperformed both the zero-shot baseline and the 4-shot ICL baseline for both OpenFlamingo and Qwen3-VL. For Qwen3-VL, FFVs reached 72.2% accuracy compared to 21.3% for the ICL baseline.
GQA Dataset: FFVs more than doubled the accuracy of the zero-shot baseline and outperformed 4-shot ICL across both spatial and agentic relations.
Generalization: The approach generalized to novel objects (unseen during training) and novel relations (e.g., "above-left") when using composite vectors.

3. Analogy and Compositionality

The study demonstrated that relation-specific function vectors can be linearly combined to solve one-shot analogy problems involving untrained spatial relations.

On the OpenFlamingo model, composite function vectors (CFVs) doubled the accuracy of 4-shot ICL (from 8.1% to 16.8%).
On Qwen3-VL, CFVs improved accuracy from 28.7% (4-shot ICL) to 45.1%, outperforming all ICL baselines.

4. Ablation Studies

Injection Layer: Performance peaked when vectors were injected at intermediate layers (e.g., layer 19), suggesting these layers are optimal for triggering relational computations.
Head Set Size: A small subset of heads (6–12) yielded optimal performance; adding more heads introduced noise.
Context Size: Performance was stable across 2-shot and 4-shot contexts, with 8-shot contexts sometimes slightly reducing accuracy, suggesting moderate context is sufficient for extracting robust activations.

Significance and Claims

The paper claims to advance the understanding of model modularity and control in LMMs by demonstrating that:

Localized Encoding: Visual relational knowledge is encoded within localized internal structures (specific attention heads) rather than being distributed diffusely or purely emergent.
Extractability and Manipulability: These representations can be systematically extracted via causal mediation analysis and manipulated (injected or fine-tuned) to alter model behavior without retraining the entire model.
Generalization: The approach supports compositional generalization, allowing models to solve novel relational problems (analogy) by linearly combining learned vectors, suggesting a structured "language of vision" within the model's internal space.
Efficiency: Fine-tuning only the function vectors (a small subset of parameters) is more effective than standard in-context learning, offering a pathway to efficient adaptation of LMMs for relational reasoning.

The authors conclude that this framework provides a proof of concept for interpreting and enhancing relational reasoning in multimodal systems, though they acknowledge limitations regarding the narrow scope of tested relations and the current reliance on contextual demonstrations for inferring vector mixtures.