🤖 AI

FiLoRA: Focus-and-Ignore LoRA for Controllable Feature Reliance

FiLoRA is an instruction-conditioned, parameter-efficient framework that enables controllable modulation of feature reliance in multimodal foundation models by using natural language instructions to selectively amplify or suppress specific internal feature pathways without altering the underlying task semantics.

Original authors: Hyunsuk Chung, Soyeon Caren Han, Seungyeon Ji, Jinwoo Kim, Eun-Jung Holden, Kyungreem Han

Published 2026-08-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hyunsuk Chung, Soyeon Caren Han, Seungyeon Ji, Jinwoo Kim, Eun-Jung Holden, Kyungreem Han

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern artificial intelligence systems that see, hear, and read are becoming incredibly skilled at combining these different senses to understand the world. When you ask such a system to describe a movie poster or identify an emotion in a voice, it draws upon a vast internal network of pathways, some of which process the meaning of words and others that analyze colors, shapes, or sound patterns. For years, researchers have been able to observe which of these internal pathways a model uses to reach a conclusion, often discovering that the system relies on misleading shortcuts—like judging a movie's genre by the color of its poster rather than its plot. However, a significant gap has remained: while scientists could watch the model make these mistakes, they could not easily tell the model to stop using those shortcuts and focus on the right information instead. The ability to observe a behavior does not automatically grant the power to control it.

A new study introduces a method called FiLoRA, which stands for Focus-and-Ignore LoRA, designed to bridge this gap. The researchers developed a way to use simple natural language instructions to directly steer the internal workings of these multimodal models. Instead of just changing the final answer the model gives, this technique allows a user to tell the system exactly which parts of its internal processing to amplify and which to suppress. For instance, a user could instruct the model to "focus on the story" or "ignore the visual appearance," and the system would physically adjust its internal calculations to follow that command. This approach treats the model's reliance on different types of information as something that can be actively managed, rather than a fixed trait that can only be analyzed after the fact.

The researchers tested this idea on several different tasks, ranging from identifying the genre of a movie based on its poster and plot, to recognizing emotions in audio and video clips, and even generating descriptions for images. In a typical scenario, a standard model might look at a movie poster with a dark, moody color scheme and guess that the film is a horror movie, even if the story is actually a romance. This happens because the model has learned a shortcut: dark colors often appear on horror posters. With the new FiLoRA method, researchers could give the instruction to "ignore physical appearance" and "focus on semantic narrative." When they did this, the model successfully shifted its attention away from the misleading colors and instead relied on the text of the plot summary. The result was a correct classification of the movie as a romance, achieved not by changing the task or retraining the entire system from scratch, but by simply gating, or controlling, the flow of information through specific internal pathways.

To make this work, the researchers broke down the model's learning process into distinct groups, each responsible for a different kind of information, such as semantic meaning, visual details, or acoustic features. They then attached a small, adjustable control mechanism to each group. When an instruction is given, the system translates that text into a set of control signals that act like dimmer switches. If the instruction is to focus on the story, the switch for the story-processing pathway is turned up, while the switch for the visual appearance pathway is turned down. This happens in real-time for every single example the model processes. The study showed that this method works consistently across different types of data and different model sizes, proving that the control is not a fluke of a specific setup but a robust feature of the architecture.

The findings suggest that the way these models use information is far more flexible than previously thought. In experiments where the researchers removed the misleading visual cues entirely, models trained with standard methods often failed or became unstable, showing they had become dependent on those shortcuts. In contrast, the models guided by FiLoRA remained stable and accurate, demonstrating that they had learned to rely on the core, meaningful information. The researchers also found that this control works with complex, natural language instructions, not just rigid commands. Whether the user asked the model to "describe the main action without referring to visual style" or "focus on interactions between people while ignoring background cues," the system responded by reshaping its internal focus accordingly.

This work represents a shift from merely analyzing how artificial intelligence makes decisions to actively intervening in the process. By showing that natural language instructions can serve as precise control signals for internal computation, the study offers a new tool for making these systems more reliable and transparent. It suggests that in the future, we might be able to guide complex AI models to avoid biases or errors simply by telling them what to pay attention to, ensuring they use the right kind of evidence to reach their conclusions. The ability to focus on the right reasons for a decision, rather than just getting the right answer by luck or shortcut, marks a significant step toward more trustworthy and controllable artificial intelligence.

Technical Summary: FiLoRA – Focus-and-Ignore LoRA for Controllable Feature Reliance

Problem Statement

Multimodal foundation models integrate heterogeneous signals (text, vision, audio) but often rely on superficial or "shortcut" cues (e.g., poster colors, background patterns, speaker identity) rather than task-relevant semantic information to make predictions. While existing methods can analyze these behaviors post hoc or intervene at the data level, they lack the ability to explicitly control how a model uses internal feature pathways during inference. Current instruction tuning and prompting alter surface-level outputs but do not guarantee corresponding changes in internal computation. Furthermore, methods like FIT or FocalLoRA focus on output alignment or hierarchical adaptation rather than providing mechanisms to selectively activate or suppress specific feature-specific computation paths under a fixed task definition. The central question addressed is: Can natural language instructions be used to explicitly control internal feature reliance in multimodal models without changing the task definition, label space, or base representations?

Methodology: FiLoRA

The authors propose FiLoRA (Focus-and-Ignore LoRA), an instruction-conditioned, parameter-efficient adaptation framework designed to modulate internal feature reliance.

1. Grouped LoRA for Path-Level Control

Standard LoRA applies a single, undifferentiated low-rank update ( $\Delta W = BA^\top$ ) per layer, entangling feature contributions. FiLoRA decomposes these updates into groups ( $\mathcal{G}$ ) aligned with functional computation paths (e.g., semantic reasoning, visual appearance processing, acoustic extraction). The adapted weight matrix becomes:
$W' = W + \sum_{g \in \mathcal{G}} \Delta W_g$
where each $\Delta W_g$ is a low-rank adaptation corresponding to a specific feature pathway. This structural decomposition allows for independent modulation of different feature sources at the parameter level.

2. Instruction-Conditioned Gating

To dynamically control these groups, FiLoRA introduces a gating mechanism. A natural language instruction $I_i$ is encoded into a continuous representation $z_i = h_\phi(I_i)$ and mapped to a gate vector $g(I_i) \in [0, 1]^{|\mathcal{G}|}$ via a sigmoid function. The final adaptation is:
$W' = W + \sum_{g \in \mathcal{G}} g_g(I_i) \Delta W_g$
The gates softly scale the contribution of each group-specific update based on the instruction semantics (e.g., "Focus on Semantic Narrative" vs. "Ignore Physical Appearance"). The base model parameters remain frozen, ensuring that changes in behavior reflect modulation of feature reliance rather than representation drift.

3. Controlled Intervention Training

To associate instructions with specific reliance patterns, the training process utilizes a controlled supervision signal. During training, the supervision target is varied to emphasize different sources of evidence (e.g., semantic vs. appearance-based cues) while remaining within the same label space. This signal is used as an intervention mechanism to bias the model toward specific feature types but is never provided as an input to the model. The model learns to associate instruction semantics with distinct feature usage patterns solely through the interaction between instruction-conditioned gating and the optimization objective. A weak gate regularization term is optionally applied to stabilize gating behavior.

Key Contributions

Formalization of Feature Reliance Control: The paper formalizes instruction following as a problem of controlled intervention on feature reliance, distinguishing it from mere output-level behavior control.
FiLoRA Framework: Introduction of a grouped and gated LoRA framework that enables differentiable, instruction-conditioned modulation of internal computation paths.
Empirical Validation: Demonstration across multimodal benchmarks (classification and generative tasks) that FiLoRA achieves consistent, interpretable, and robust control over feature usage under diverse instructions (natural, paraphrased, and compositional).

Results

The authors evaluated FiLoRA on controlled classification settings (Tiny MM-IMDb, CREMA-D, RAVDESS) and generative tasks (Flickr30K image captioning) using various backbone models (Qwen2.5-VL, Gemma-3n, Qwen2.5-Omni, AV-HuBERT, AudioCLIP).

Controllable Modulation: FiLoRA induces systematic shifts in feature reliance. "Focus-core" instructions consistently amplify gates associated with semantic/narrative pathways while suppressing appearance/demographic pathways. Conversely, "Ignore-core" instructions shift reliance toward shortcut-associated features.
Functional Redistribution: Metrics such as Gate Modulation Range (GMR) and Reliance Sensitivity (RS) confirm that these gate changes produce functional shifts in predictions. Under focus-core instructions, prediction sensitivity concentrates on core pathways; under ignore-core instructions, it shifts to spurious pathways.
Generalization: The method generalizes across instruction formulations (templates, paraphrases, compositional instructions) and model scales/architectures.
Generative Capabilities: In image captioning, FiLoRA successfully alters caption content to emphasize semantic interactions or visual attributes based on instructions, without degrading overall caption quality (CIDEr/CLIPScore).
Robustness: FiLoRA demonstrates higher decision stability and more gradual performance degradation under targeted spurious feature interventions compared to full fine-tuning, standard LoRA, and prompt-only baselines.
Ablation: The effectiveness of FiLoRA depends on semantically meaningful feature grouping; random or coarse grouping significantly reduces controllability.

Significance and Claims

The paper claims that FiLoRA provides a practical mechanism for intervening on internal model behavior, offering a new perspective on controllability that goes beyond output-level prompting or post hoc interpretation. By treating instructions as controlled interventions on feature reliance, the authors argue that multimodal instruction following should involve not just controlling what a model outputs, but regulating how it internally uses information. The work suggests that instruction-conditioned parameter adaptation can serve as a reliable tool for steering internal computation paths, enabling models to perform the same task under different reliance conditions (e.g., "Right for the Right Reasons"). The authors remain modest, noting that their approach operates under fixed task semantics and does not redefine the task itself, but rather provides a structured way to analyze and control the internal reliance on specific modalities or feature groups.