💬 NLP

Better with Experience: Self-Evolving LLM Agents for Evidence-Grounded Health Community Notes

The paper introduces EvoNote, a self-evolving LLM agent framework that leverages an experience memory with fine-grained credit assignment to generate evidence-grounded health Community Notes, achieving superior helpfulness and significantly faster production times compared to human-written notes on a new multimodal benchmark.

Original authors: Zihang Fu, Fanxiao Li, Jianyang Gu, Haonan Wang, Preslav Nakov, Bryan Hooi, Min-Yen Kan, Jiaying Wu

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zihang Fu, Fanxiao Li, Jianyang Gu, Haonan Wang, Preslav Nakov, Bryan Hooi, Min-Yen Kan, Jiaying Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Amnesiac" Fact-Checker

Imagine a team of fact-checkers working on social media to stop health misinformation (like false claims about vaccines or miracle cures).

Currently, these AI fact-checkers work like amnesiac interns. Every time they get a new post to fact-check, they start from scratch. They don't remember what they learned yesterday.

Scenario: An intern fact-checks a post about a "miracle cancer cure" and learns that the source is fake.
The Next Day: A different intern (or the same one with a "reset" brain) sees a similar post about a different "miracle cure." They have to re-learn everything from zero, often making the same mistakes or missing the same red flags.

This is slow, inefficient, and leaves people exposed to bad information for longer.

The Solution: EVONOTE (The "Learning Veteran")

The authors created a new system called EVONOTE. Think of this not as an intern, but as a seasoned veteran detective who keeps a detailed, organized case file.

Instead of forgetting everything after a job is done, EVONOTE treats every fact-checking job as a lesson. It builds a "memory bank" of what worked and what didn't, so it gets smarter with every single post it corrects.

How It Works: The Three-Step Loop

EVONOTE operates in a continuous loop, much like a detective solving cases and updating their handbook:

1. The Investigation (The Agent)
When a flagged post arrives, EVONOTE doesn't just guess. It acts like a detective:

Analyzes the Claim: "What exactly is this person saying?"
Gathers Evidence: It searches the web, visits websites, and reads scientific papers to find the truth.
Writes the Note: It drafts a correction for the social media post.

2. The Performance Review (The Judge)
Once the note is written, a "Social Utility Judge" (a smart AI evaluator) reviews the work. It doesn't just ask, "Is this true?" It asks four deeper questions based on health communication:

Understandable: Is it easy for a regular person to read?
Meaningful: Does it explain why this matters for their health?
Usable: Does it tell people what safe steps to take next?
Trustworthy: Does it admit uncertainty if the science isn't 100% clear?

3. The Lesson Learned (The Memory Evolver)
This is the magic part. The Judge gives feedback, and a "Memory Evolver" distills that feedback into a reusable rule.

Example: If the AI missed checking the context of a quote in a previous case, the Memory Evolver creates a rule: "When a post uses a quote, always check the full context before writing."
This rule is stored in the memory bank. The next time a similar post appears, the system automatically pulls up this rule and applies it, preventing the same mistake.

The Results: Faster and Better

The researchers tested EVONOTE on a dataset of 1,200 real health posts (text, images, and videos) that had already been fact-checked by humans.

Beating Humans: In nearly 90% of cases, the notes generated by EVONOTE were rated as better than the original human-written notes.
Speed: While it takes humans an average of 13 hours to get a consensus on a correction, EVONOTE can produce a high-quality draft in under 2 minutes.
Better Evidence: EVONOTE didn't just write faster; it found better sources. It was more likely to cite official health organizations (like the CDC) and less likely to rely on shaky news sites.

The "Caption" Trick

One interesting finding was about images and videos. The system works best when it first converts images/videos into text descriptions (captions) before analyzing them.

Analogy: Imagine trying to solve a puzzle while wearing thick foggy glasses (a direct video AI). It's hard to see the pieces. EVONOTE takes off the foggy glasses, has a human describe the picture clearly, and then solves the puzzle. This made the system much more reliable.

The Bottom Line

The paper argues that fighting health misinformation shouldn't be a game of "reset and retry." Instead, we need systems that learn from their own history.

By turning every fact-checking episode into a reusable lesson, EVONOTE creates a self-improving cycle. It ensures that the next time a similar lie appears, the system is already prepared to debunk it faster and more accurately than before.

What the paper does NOT claim:

It does not claim this system is currently deployed on X (Twitter) for all users.
It does not claim this replaces human doctors or fact-checkers entirely; it suggests it should be a tool to help them.
It does not claim it works for political or financial lies yet; the study was strictly focused on health misinformation.

Technical Summary: Better with Experience: Self-Evolving LLM Agents for Evidence-Grounded Health Community Notes

1. Problem Statement

Health misinformation on social platforms requires timely, evidence-backed, and user-understandable corrections. While X's (formerly Twitter) Community Notes offer a scalable governance paradigm, current Large Language Model (LLM)-augmented approaches suffer from a critical limitation: they operate under a one-pass paradigm. Each post is treated as an isolated instance, resetting the agent's state and discarding useful correction experience from prior cases.

This formulation fails to align with the reality of health misinformation governance, where misleading claims often recur chronologically through similar rhetorical strategies and evidence needs. Consequently, later corrections remain vulnerable to repeated, avoidable errors because the system cannot leverage transferable evidence-seeking and writing strategies derived from earlier, successful human-written notes.

2. Methodology: The EVONOTE Framework

The authors introduce EVONOTE, an agentic framework designed to enable self-evolving health Community Notes generation. Unlike prior methods that rely on passive retrieval of past documents or full trajectories, EVONOTE implements fine-grained credit assignment to distill trajectory-level feedback into actionable, phase-specific memory.

Core Components

Note-Generation Episode Formulation:
Each episode $k$ involves a flagged post $x_k$ (text, image, or video) converted into a unified textual representation $\tilde{x}_k$ . The agent produces a trajectory $\tau_k$ comprising:
- Claim Analysis ( $c_k$ ): Instance-specific identification of check-worthy claims and priority evidence needs.
- Action History ( $H_k$ ): A sequence of search and visit actions.
- Evidence ( $E_k$ ): Collected web evidence.
- Final Note ( $y_k$ ): The generated Community Note.
Social Utility Judge (Trajectory-Level Evaluation):
A specialized agent evaluates the completed trajectory against four health-specific note qualities grounded in prior health communication literature:
- Understandability: Accessibility of language for non-experts.
- Meaningfulness: Clarity on health interpretation and risk perception.
- Usability: Provision of safe, proportionate next-step guidance.
- Trustworthiness: Calibrated use of evidence under uncertainty.
The Judge assigns a utility mode ( $u_k$ ): Unsatisfactory, Satisfactory, or Excellent. This mode acts as a routing signal, determining whether the episode should contribute to failure-avoidance, improvement-oriented, or success-reuse memories.
Memory Evolver (Action-Level Distillation):
The Memory Evolver converts the trajectory and utility mode into a set of reusable memory items $\Delta M_k$ . Each item is structured as a tuple:
- Trigger: A condition identifying when the memory applies (e.g., "Post uses a specific statistic to imply a causal link").
- Action Strategy: A reusable correction strategy (e.g., "Verify the statistic against the primary source").
- Phase: The specific generation stage the strategy guides (Claim Analysis, Evidence Acquisition, or Note Writing).
Memory-Guided Generation:
At inference time, the agent retrieves relevant memories based on the current post and decision state.
- Claim Analyzer: Uses retrieved Claim Analysis memories to generate a verification brief.
- Note Writer: Uses Evidence Acquisition and Note Writing memories to guide search queries, source prioritization, and note framing.
- Budget Constraints: The agent operates under strict search and visit budgets, ensuring efficiency.
Closed-Loop Self-Evolution:
The system forms a closed loop where the trajectory of episode $k+1$ becomes the source of experience for episode $k+2$ . This allows the system to accumulate correction experience without updating model parameters or relying on a fixed bank of human notes.

3. Benchmark: MM-HEALTHCN

To evaluate EVONOTE, the authors curated MM-HEALTHCN, a multimodal benchmark consisting of 1,200 instances:

Composition: 400 Text, 400 Image, and 400 Video posts.
Data: User-flagged health posts paired with human-written Community Notes and crowd-derived helpfulness labels ("Helpful" or "Not Helpful").
Evaluation Protocol: A hierarchical utility judge first filters for length compliance (280 characters), then assesses evidence relevance, evidence correctness, and note-style helpfulness. A pairwise social utility judge compares notes based on the four health-specific qualities (understandability, meaningfulness, usability, trustworthiness).

4. Key Results

Experiments were conducted on MM-HEALTHCN comparing EVONOTE against human-written notes, web-search-enabled LLMs (GPT-4.1, Grok-4.3, Gemini-2.5-Flash), and existing LLM-augmented note generation systems (CrowdNotes+, DeepResearch, ExpRAG, ReMem).

Superiority over Human Notes: Under a human-validated hierarchical utility judge, EVONOTE-generated notes were preferred over corresponding human-written notes in 89.6% of cases.
Performance on Unresolved Cases: On a separate set of "Needs More Ratings" posts (where crowd consensus is not yet reached), EVONOTE produced helpful notes for 82.0% of cases, significantly improving coverage before human consensus is formed.
Efficiency: EVONOTE reduced the median time to produce a candidate correction from over 13 hours (in the human-note pipeline) to under 2 minutes.
Evidence Quality: EVONOTE demonstrated stronger evidence profiles than baselines, achieving higher source quality, domain diversity, and semantic diversity. It utilized more diverse and higher-quality sources without simply increasing the number of citations.
Ablation Studies: Removing the evolving memory (EVONOTE-NM) or the instance-specific claim analyzer (EVONOTE-NM-NA) degraded performance, confirming that both accumulated experience and explicit verification planning are essential.
Multimodal Handling: The paper found that direct multimodal VLM instantiation was unreliable for this task. Using human-validated captions to convert images/videos to text, processed by a medical-capable LLM (MedGemma-27B), yielded significantly more reliable trajectories.

5. Significance and Claims

The paper positions self-evolving note generation as a promising paradigm for health misinformation governance. Its primary contributions and claims are:

Cumulative Correction: The authors demonstrate that misinformation correction can become cumulative. By turning agentic correction trajectories into reusable memory, systems can adapt to emerging claims even when reliable human notes are sparse or delayed.
Phase-Specific Memory: The study argues that generic memory retrieval is insufficient. The key to improvement is distilling feedback into phase-specific strategies (claim analysis, evidence acquisition, writing) that guide specific actions in future episodes.
Actionable Experience: EVONOTE shows that agents can learn actionable strategies (e.g., verifying primary sources for statistics, distinguishing correlation from causation) from their own trajectories, effectively addressing recurring correction failures.
Governance Principle: The work proposes a design principle for misinformation governance: "every flagged post should become experience before the next similar one recurs."

The authors remain modest regarding limitations, noting that the system currently focuses on English health misinformation, learns primarily from its own trajectories (rather than jointly with high-quality human corrections), and operates at the memory level rather than updating the underlying model or workflow. They emphasize that EVONOTE is intended as a human-augmenting system, requiring human oversight for final verification and surfacing of corrections.