🤖 AI

Different Feedback, Different Updates: Selective Self-Learning from User Interactions for Large Language Models

The paper introduces SLIFT, a selective self-learning framework that decomposes user feedback into task-valid fixes, condition-specific specs, and null components to train complementary Generalist and Specialist LoRA adapters on a frozen backbone, enabling large language models to persistently improve with appropriate generalization scopes.

Original authors: Xuanchen Li, Haitao Li, Yujia Zhou, Qingyi Pan, Heng Wang, Yiqun Liu, Min Zhang, Qingyao Ai

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xuanchen Li, Haitao Li, Yujia Zhou, Qingyi Pan, Heng Wang, Yiqun Liu, Min Zhang, Qingyao Ai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart, but slightly stubborn, robot to be a better conversationalist. You don't just want it to memorize a script; you want it to learn from every chat it has with real people. This is the world of Large Language Models (LLMs), the super-smart AI brains behind chatbots and writing assistants. These models are like giant libraries of human knowledge, but they can make mistakes, miss the point, or give answers that are technically correct but annoyingly boring.

To fix them, we usually rely on user feedback. When a human says, "That's wrong," or "Make it shorter," the model gets a hint. But here's the tricky part: a single comment from a user is often a messy cocktail of different instructions. One sentence might say, "Fix this math error" (a hard rule), while the next says, "I prefer a funny tone" (a nice-to-have preference), and the third says, "By the way, tell me a joke about cats" (which has nothing to do with the original math problem). If the robot tries to learn from all of these at once as if they were the same, it might get confused, start making jokes during math tests, or forget how to do basic arithmetic. The big question researchers are asking is: How do we teach a robot to listen to the right parts of a message and ignore the rest, so it gets smarter over time without getting weird?

This paper introduces a clever new system called SLIFT (which stands for Self-Learning from Interaction Feedback via Task-Relative Specialization) to solve this exact problem. Think of SLIFT as a super-organized team of two robots working together to learn from a messy user comment.

First, the system acts like a detective. When a user sends a feedback message, SLIFT breaks it down into tiny, atomic pieces. It then sorts each piece into one of three buckets:

Fix: These are the "must-haves." If the user said, "The answer is 5, not 3," this is a rule that every future answer to this type of question must follow. It's a permanent change to the robot's default behavior.
Spec: These are the "nice-to-haves" or specific preferences. If the user said, "Use a kitchen analogy," that's great for this specific math problem, but maybe not for a history question. This is a conditional refinement.
Null: These are the "ignore me" parts. If the user suddenly asks for a poem about cats while asking for a math solution, that's unrelated noise. It shouldn't change how the robot learns at all.

Once the feedback is sorted, SLIFT uses two different "learners" (called adapters) to handle the updates, rather than trying to force one brain to do everything.

The first learner is the Generalist. It's like the robot's core personality. It only looks at the Fix items. If the user pointed out a factual error, the Generalist learns to make sure that error never happens again in its default answers. It "bakes" this correction into its permanent memory, so it doesn't have to think about it every time.

The second learner is the Specialist. This one is the "context checker." It doesn't change the Generalist's core personality. Instead, it waits until the Generalist gives an answer, and then it asks: "Hey, did we miss any of those Spec preferences for this specific situation?" If the Generalist gave a correct math answer but forgot to use the kitchen analogy the user wanted, the Specialist steps in. It says, "Keep the math, but add the analogy," and provides a tiny, targeted nudge to fix just that one thing. If the Generalist already did everything right, the Specialist just says, "Keep it as is," and does nothing.

The paper tested this system on two different types of data: simulated chats where a computer played the user, and real-world chats from actual people. The results suggest that SLIFT works really well. On the simulated tests, it improved the model's scores significantly more than other methods. On the real-world tests, it made the model better at following instructions and writing quality responses, without messing up its ability to solve hard reasoning problems.

The authors found that the key to success was not treating all feedback as one big blob. By separating the "must-fix" rules from the "nice-to-have" preferences and the "ignore" noise, the model could learn the right things at the right time. The Generalist became smarter at the basics, while the Specialist learned to be a helpful editor for specific situations. This approach suggests that for AI to truly learn from humans, it needs to be selective, knowing exactly what to remember forever and what to only use for the moment.

Technical Summary: Different Feedback, Different Updates (SLIFT)

Problem Statement

Large Language Models (LLMs) are increasingly deployed in interactive systems where user feedback provides natural supervision for improvement. However, transforming interaction-specific hindsight into persistent, generalizable behavioral changes remains challenging. The core difficulty lies in the nature of user feedback: a single feedback message is often composite, containing corrections, preferences, and new requests simultaneously. Treating such feedback as a single supervision signal obscures what should be learned and how broadly it should generalize.

Specifically, the paper identifies three distinct challenges in learning from user feedback:

Decomposition: Natural feedback rarely takes the form of explicit structured supervision. A single turn may combine task-necessary corrections, compatible but non-necessary refinements, and content that provides no reliable update direction.
Scope Determination: Some feedback identifies properties required for task validity (requiring consolidation into default behavior), while other feedback expresses refinements relevant only to specific contexts (requiring conditional application). Generalizing conditional refinements as default behavior can lead to over-generalization and errors in unsupported contexts.
Signal Construction: Converting raw feedback into a trainable signal is difficult because feedback often lacks a complete target response or explicit scalar reward. Existing methods either compress rich feedback into coarse signals (losing fine-grained guidance) or retain full feedback but provide weak supervision when the model cannot reliably interpret it.

Methodology: SLIFT

The authors introduce SLIFT (Self-Learning from Interaction Feedback via Task-Relative Specialization), a selective self-learning framework. SLIFT operates on a task-relative view of user feedback, decomposing feedback into atomic components and assigning them roles based on their relationship to the original task.

1. Task-Relative Feedback Processing

Using a frozen backbone model ( $M_0$ ), SLIFT processes each user feedback turn ( $u_i$ ) through two steps:

Atomic Extraction: The feedback is decomposed into minimal, self-contained components ( $C_i$ ).
Role Assignment: Each component is classified into one of three roles relative to the original task ( $x_i$ $x_{i}$ ):
- Fix: A requirement that every fully valid response to the task must satisfy. Omitting it renders the response invalid.
- Spec: A compatible refinement that selects a subregion of valid responses. It is task-compatible but not strictly necessary for validity; its relevance depends on the specific task context or the response already produced.
- Null: Content that is ambiguous, irrelevant, conflicting, or raises a separate request, providing no reliable positive update direction.

2. Selective Learning with Dual Pathways

To handle these distinct roles, SLIFT instantiates two complementary LoRA adapters on a shared frozen backbone, inspired by functional specialization in complementary learning systems:

The Generalist (for Fix):
- Objective: Consolidates task-necessary requirements into default behavior.
- Mechanism: It learns from Fix components via feedback-conditioned self-distillation. The frozen backbone generates a "hindsight" distribution conditioned on the task and the Fix components. The Generalist is trained to reproduce this distribution without observing the feedback, effectively internalizing the requirements.
- Loss: A KL-divergence loss between the Generalist's output and the hindsight distribution, regularized by a behavioral anchor (the feedback-free distribution) to prevent over-editing.
The Specialist (for Spec):
- Objective: Supplies residual guidance only for applicable, unmet Spec refinements.
- Mechanism: It observes only the original task ( $x_i$ ) and the Generalist's default response ( $y^G_i$ ). It does not observe the feedback directly during inference.
- Training Target: The frozen backbone constructs a structured target based on the Spec components (used as privileged offline information). The target is either:
  - KEEP: No applicable refinement remains unmet.
  - APPLY: A refinement is needed, accompanied by residual guidance ( $\delta$ ) describing only the missing compatible changes.
- Inference: The Specialist decides whether to KEEP (return $y^G_i$ unchanged) or APPLY (generate $\delta$ ). If APPLY, the frozen Generalist performs a single residual-guided integration pass to implement the changes.
Null Components: These induce no positive update and are discarded.

Key Contributions

Task-Relative Formulation: The paper proposes decomposing composite feedback into atomic Fix, Spec, and Null components, distinguishing between task-necessary requirements, conditional refinements, and noise.
SLIFT Framework: A dual-pathway architecture where a Generalist consolidates Fix requirements into default behavior, and a Specialist provides sparse, conditional guidance for Spec refinements. This separation prevents conditional evidence from becoming default behavior.
Empirical Validation: SLIFT achieves strong performance across different backbones (Qwen3-8B, Ministral3-14B) and datasets (MemoryBench, WildFB), outperforming non-parametric memory systems and parametric baselines (SFT, DPO, SDPO).

Experimental Results

The framework was evaluated on two benchmarks:

MemoryBench (Simulated Feedback): SLIFT achieved the highest mean Norm-Score and Z-score across all input-output length partitions for both backbones. Ablation studies confirmed that atomic extraction, task-relative role assignment, and separate pathways are critical; removing any component significantly degraded performance.
WildFB (Real-World Feedback): SLIFT improved instruction adherence (IFEval) and open-ended response quality (AlpacaEval 2.0) while maintaining performance on knowledge-reasoning benchmarks (MMLU-Pro). It showed modest gains in WildReward win rates, attributed partly to the lower usability of real-world feedback compared to simulated data.

Efficiency: SLIFT incurs lower parameter-update costs than SFT and DPO because it avoids generating full revised targets. At inference, it adds minimal overhead, as the Specialist only triggers an integration pass for a small subset of tasks (low APPLY rates).

Significance and Claims

The paper claims that SLIFT addresses the fundamental challenge of scope generalization in continual learning. By explicitly distinguishing between what must be learned as default behavior (Fix) and what should remain conditional (Spec), the framework avoids the pitfalls of over-generalization common in existing methods.

The authors emphasize that their approach does not require a stronger teacher model or an explicit reward model; supervision is constructed entirely by the frozen backbone using the feedback components. The results suggest that treating composite feedback as a single undifferentiated unit is suboptimal, and that selective self-learning, which respects the appropriate scope of each feedback component, is a viable path for persistent LLM improvement from user interactions.