💬 NLP

Enhancing LLMs through human feedback: a journey towards self-improvement

This paper proposes a novel human-in-the-loop methodology that integrates an auxiliary feedback RAG system to iteratively refine a primary RAG system's performance through continuous user feedback, demonstrating significant improvements in accuracy and relevance across diverse benchmarks via LLM-as-a-Judge evaluation.

Original authors: Tatiana Pelc, Gila Kamhi, Asaf Avrahamy, Adi Fledel-Alon

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tatiana Pelc, Gila Kamhi, Asaf Avrahamy, Adi Fledel-Alon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot librarian named RAG. This librarian is amazing at finding books (documents) and writing answers based on them. But sometimes, the librarian gets things wrong, or the answers feel a bit stiff and robotic. Traditionally, if you didn't like an answer, you'd just give a thumbs down. That's like a binary switch: on or off. It tells the robot "bad," but it doesn't tell the robot why or how to fix it.

This paper introduces a new way to teach this librarian, called FLARE (Feedback-driven LLM Adaptive RAG Enhancement). Instead of just a thumbs down, FLARE lets users write detailed notes, corrections, and even style tips. Think of it as the difference between a student getting a red "X" on a test versus getting a teacher's note that says, "You got the date right, but you forgot to mention the country, and your tone was too grumpy."

The Two-Step Dance: Offline Prep and Online Magic

FLARE works like a two-step dance that happens in the background while the librarian is working.

Step 1: The Offline Prep (The Study Session)
When a user leaves a detailed note, FLARE doesn't just file it away. It puts the note through a "contextual enrichment" process. Imagine a user says, "Wrong, it's Donald Trump." FLARE takes that fragment and, using the chat history, turns it into a clear, complete sentence: "The current president of the US is Donald Trump as of 2025."

Then, a special AI (the "judge") sorts these notes into two buckets:

Fact Checks: "Hey, update the database, the president changed!"
Style Guides: "Hey, next time, be more polite and include more history."

These notes are then turned into digital "search tags" and stored in a special library (a vector database). This is the "offline" part where the system learns from the past.

Step 2: The Online Magic (The Live Show)
Now, a new user asks a question. Before the librarian answers, FLARE does a quick side-check. It asks, "Has anyone asked something similar before? Did they leave a note?" If the answer is yes, FLARE grabs those notes and injects them directly into the librarian's brain while it's writing the answer. It's like the librarian suddenly remembering a sticky note from a previous customer that says, "Don't forget the extra details!"

The Proof: Did It Work?

The authors tested this idea in three different "playgrounds" to see if it actually made the librarian smarter.

The Fictional Playground: They used made-up dialogues about fictional characters. Without FLARE, the librarian was a bit confused, scoring only about 1 out of 5 when context was missing. With FLARE, the librarian got almost perfect scores (5 out of 5) on nearly every question.
The Trivia Playground: This was the tricky part. They asked questions about events in 2024 and 2025—stuff the robot wasn't supposed to know yet because its training data stopped earlier.
- Without FLARE: The librarian guessed wrong or gave vague answers, scoring an average of 3.06 out of 5.
- With FLARE: The librarian used the feedback from previous users to learn the new facts. The average score jumped to 3.91 out of 5.
- Example: When asked about the 2025 Oscars, the basic librarian got the winner wrong. FLARE corrected it to the right person and even added the cool detail that it was a historic win for a specific demographic.
The Tech Support Playground: They tested this on real-world questions about wireless networking (Intel's internal data). Here, they found that FLARE was great at finding the right feedback (it succeeded 95% of the time). However, the system only managed to use that feedback to improve the final answer 61% of the time.

What FLARE Is NOT (And What It Still Needs)

It's important to know what this paper doesn't claim. The authors are careful to say that FLARE isn't a magic wand that solves everything instantly.

It's not a perfect fix: In the tech support tests, even though the system found the right notes 95% of the time, it only used them to fix the answer 61% of the time. The authors suggest this is because the way they "inject" the notes (sticking them into the text) sometimes makes the robot ignore the most important parts.
It's not fully autonomous yet: The system currently treats all feedback as equal. It doesn't know if a note came from a world-expert or a confused user, or if a note is from yesterday or last year. The authors admit they need to build a smarter way to decide which notes matter most.
It's not a "solved" problem: The paper explicitly states that there are "opportunities to further refine" the system. They plan to add features like checking who wrote the feedback and re-reading the notes if the answer looks weak.

The Bottom Line

The main finding is that giving the robot librarian detailed, written feedback—and then having it read those notes while it answers new questions—makes it significantly better at being accurate and helpful. In the simulations and tests they ran, this "human-in-the-loop" approach turned a good system into a great one, especially when dealing with new information the robot didn't know before. But, as the authors point out, the journey to a fully self-improving robot isn't finished; they still need to figure out how to make the robot listen to the best notes, not just any note.

Technical Summary: Enhancing LLMs through Human Feedback via the FLARE Framework

Problem Statement

While Large Language Models (LLMs) have advanced significantly through Reinforcement Learning from Human Feedback (RLHF), traditional AI systems often rely on binary feedback (e.g., like/dislike), which limits the depth of learning. Furthermore, standard Retrieval Augmented Generation (RAG) systems frequently fail to capture evolving user preferences, resolve contextual mismatches, or adapt to dynamic information needs. Existing approaches often suffer from retrieval bias, robustness challenges, and an inability to incorporate rich, textual human corrections into the generative pipeline. There is a need for a mechanism that moves beyond static training to enable continuous, iterative self-improvement in RAG systems through structured, human-in-the-loop feedback.

Methodology: The FLARE Framework

The paper introduces FLARE (Feedback-driven LLM Adaptive RAG Enhancement), an iterative framework designed to refine RAG performance by integrating structured user feedback directly into the inference workflow. The system operates through two parallel phases: Offline Preprocessing and Online Inference.

1. Offline Feedback Management Pipeline

This phase processes raw user feedback to create a structured knowledge base for future retrieval:

Collection & Enrichment: User free-text feedback is stored in a MongoDB database. Raw feedback is enriched with conversation history to convert fragmented inputs into self-contained, interpretable statements.
LLM-Driven Processing: An LLM processes enriched feedback to perform three tasks:
1. Query Synthesis: Refines original user queries to remove extraneous elements while preserving core semantic meaning.
2. Factual Extraction: Generates concise factual statements to correct inaccuracies or update knowledge bases.
3. Guideline Extraction: Identifies stylistic and structural guidelines (e.g., tone, formatting) without altering factual content.
Classification & Aggregation: Feedback is classified into "Factual Data Corrections" and "Stylistic/Structural Guidelines." GPT-4o aggregates feedback to consolidate redundant or conflicting entries, filtering for actionable insights.
Vector Indexing: Synthesized queries are embedded using text-embedding-ada-002 and indexed in an Azure Vector Database to enable fast, similarity-based retrieval.

2. Online Inference Workflow

During live operation, the framework integrates processed feedback in real time:

Dual Retrieval Architecture: Incoming queries are processed by both the primary RAG system and a dedicated Feedback-RAG module.
Adaptive Integration: Azure Hybrid Search identifies previously indexed queries with semantic similarity to the current input.
- Factual Corrections: Injected directly into the primary RAG context.
- Stylistic Guidelines: Integrated as system-level instructions to refine output formatting and tone.
Output Generation: The augmented RAG system generates the final response, which is logged for future offline batch processing, creating a continuous improvement loop.

Experimental Design

The framework was validated using the LLM-as-a-Judge protocol across three diverse datasets:

Fictional Persona Dialogs (FPD): 68 anonymized dialogues from synthetic sources to verify implementation correctness and controlled evaluation.
Trivia Benchmark: 70 training and 30 testing pairs involving time-sensitive events (2024–2025) beyond the LLM's knowledge cutoff, simulating generalization to new information.
Wireless Technical Documentation: 80 training pairs derived from real-world Intel technical support queries, testing domain-specific precision and robustness.

Evaluation metrics included F1-scores for list-based queries and a multi-dimensional qualitative assessment (Factual Accuracy, Contextual Relevance, Completeness) for non-list queries.

Key Results

General Knowledge (FPD): FLARE demonstrated a marked improvement, achieving a maximum score of 5 on nearly all items (Average score: 5.0 vs. 4.0 without FLARE), with a statistically significant p-value of $4 \times 10^{-6}$ .
General Knowledge (Trivia): FLARE achieved an average score of 3.91 compared to 3.06 for the baseline RAG system on a test set of 30 questions. The system successfully adapted to new information beyond its initial training data, whereas the baseline struggled even with access to up-to-date retrieval corpora.
Custom Domain (Wireless):
- Retrieval Success: FLARE successfully retrieved relevant feedback for 95% of test queries (76/80).
- Utilization Gap: While feedback was retrieved, it was incorporated into the final answer only 61% of the time. This highlights a gap between retrieving relevant context and effectively utilizing it during generation.

Significance and Claims

The paper posits that FLARE represents a significant step toward autonomous refinement in information retrieval technologies. Its primary contributions include:

Dynamic Adaptation: Unlike static RLHF pipelines, FLARE classifies, aggregates, and retrieves feedback within the generation loop, enabling continuous system-level adaptation.
Nuanced Learning: By incorporating rich textual corrections and behavioral signals rather than binary feedback, the system achieves more nuanced learning and improved contextual alignment.
Scalable Validation: The use of the LLM-as-a-Judge protocol provides a robust, reproducible framework for assessing feedback-driven improvements, aligning automated scoring with human preferences.

Limitations and Future Directions

The authors modestly acknowledge current limitations:

Retrieval-Utilization Gap: The generation module failed to incorporate retrieved feedback in 39% of cases, suggesting that template-based string concatenation may marginalize important details when model attention is constrained.
Static Weighting: The current system treats all retrieved comments as equal, ignoring cues such as author expertise, recency, or consistency.
Future Work: The authors plan to develop adaptive ranking mechanisms to weight feedback by relevance and credibility, implement iterative decoding loops for sparse contexts, and broaden the feedback schema to include multimodal inputs (screenshots, voice notes). The ultimate goal is a low-latency, fully autonomous, self-improving RAG framework.