💬 NLP

Learning Evidence Highlighting for Frozen LLMs

HiLight is a framework that improves the reasoning of frozen LLMs by training a lightweight "Emphasis Actor" to insert minimal highlight tags around key evidence within long, noisy contexts using reinforcement learning, without requiring evidence labels or modifying the underlying solver.

Original authors: Shaoang Li, Yanhang Shi, Yufei Li, Mingfu Liang, Xiaohan Wei, Yunchen Pu, Fei Tian, Chonglin Sun, Frank Shyu, Luke Simon, Sandeep Pandey, Xi Liu, Jian Li

Published 2026-04-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shaoang Li, Yanhang Shi, Yufei Li, Mingfu Liang, Xiaohan Wei, Yunchen Pu, Fei Tian, Chonglin Sun, Frank Shyu, Luke Simon, Sandeep Pandey, Xi Liu, Jian Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery. You are handed a massive, 500-page stack of messy, disorganized documents: old receipts, random diary entries, newspaper clippings, and grocery lists. Somewhere in that mountain of paper is a single sentence that explains who the thief is.

Even if you are a brilliant detective (a "Large Language Model"), you might get overwhelmed. You might get distracted by a funny recipe on page 42 or a boring weather report on page 200. This is the "Lost in the Middle" problem: when there is too much noise, even the smartest minds miss the signal.

HiLight is like giving that detective a highlighter pen and a smart assistant before they start reading.

The Two Characters in the Story

To understand how HiLight works, think of two different people working together:

The Actor (The Smart Assistant): This person doesn't solve the mystery. Their only job is to skim the 500 pages very quickly and say, "Hey, Boss! These three sentences on page 12, page 88, and page 402 look really important. I’ve put little neon sticky notes on them for you."
The Solver (The Brilliant Detective): This is the heavy hitter. They are incredibly smart at reasoning, but they hate clutter. They take the stack of papers, look only at the parts with the neon sticky notes, and use those clues to solve the case perfectly.

How is this different from other methods?

Before HiLight, people tried two other ways to help the detective, but both had flaws:

The Shredder (Pruning): This method throws away all the "unimportant" pages to save time. The problem? Sometimes, a page that looks boring actually contains a tiny clue that connects two big ideas. If you shred it, the detective can't solve the mystery.
The Summarizer (Compression): This method tries to rewrite the 500 pages into a 5-page summary. The problem? In the process of summarizing, you might accidentally change the meaning or leave out a crucial detail (like changing "The man was not guilty" to "The man was guilty").

HiLight does neither. It leaves the original text exactly as it is. It just adds tiny, invisible "tags" (like digital highlighter marks) so the detective knows exactly where to look.

How does the "Assistant" learn?

The most clever part is how the Assistant (the Actor) learns to highlight. We don't tell the Assistant, "This is the important sentence." We don't have a teacher to grade them.

Instead, we use Trial and Error (Reinforcement Learning).

We let the Assistant highlight some parts, give the papers to the Detective, and see if the Detective solves the case.

If the Detective gets it right, we tell the Assistant: "Great job! Whatever you highlighted, do more of that!"
If the Detective gets it wrong, we tell the Assistant: "That was useless. Try highlighting something else next time."

Over time, the Assistant becomes an expert at spotting the "scent" of evidence, even though it has never actually seen a "correct" answer key.

Why does this matter?

It’s Universal: Because the Assistant is just adding marks to text, you can train it on one type of AI and then use it to help a completely different, much more powerful AI (even ones like GPT-4 that you can't "touch" or modify).
It’s Fast and Cheap: The Assistant is a "lightweight" model. It’s much faster and cheaper to run than the big, heavy Detective.
It’s Transparent: If the Detective makes a mistake, you can look at the highlights and see exactly what they were looking at. It’s not a "black box"; you can see the logic.

In short: HiLight doesn't try to make the detective smarter; it just clears the desk so the detective can focus on the clues that actually matter.

Technical Summary: HiLight: Learning Evidence Highlighting for Frozen LLMs

HiLight is a novel framework designed to improve the reasoning capabilities of Large Language Models (LLMs) in long-context, noisy environments. It addresses the fundamental bottleneck where LLMs struggle to isolate sparse, decisive evidence buried within irrelevant or redundant information—a phenomenon often referred to as the "Lost in the Middle" effect.

1. The Problem: Coupling Selection and Reasoning

The authors identify that long-context failures typically stem from the coupling of two distinct subproblems:

Evidence Selection: Identifying task-relevant spans amidst distractors.
Reasoning: Performing multi-step inference over that evidence.

Current approaches attempt to solve both simultaneously in a single forward pass, or they use "hard selection" (pruning/retrieval) or "soft selection" (summarization/compression). However, hard selection can discard vital connective context, and compression can distort or lose subtle details.

2. Methodology: The HiLight Framework

HiLight proposes decoupling these tasks. It introduces a lightweight Emphasis Actor that performs evidence selection by inserting minimal, non-destructive structural markers (tags) into the original, unaltered text. A frozen Solver LLM then performs reasoning on this "emphasized" context.

Key Technical Components:

Emphasis Actor ( $\pi_\theta$ ): A lightweight, pretrained LM (e.g., Qwen-0.6B to 4B) that acts as an encoder. It maps context tokens to importance probabilities using a learnable weight vector and a temperature-controlled sigmoid function.
Non-Destructive Markup: Instead of deleting text, the Actor selects spans and wraps them in tags (e.g., <start_important> ... <end_important>). This preserves the full context for the Solver, maintaining discourse structure and disambiguation cues.
Budgeted Selection: To ensure the highlights are concise and auditable, a highlight budget ( $\gamma$ ) is enforced, limiting the maximum fraction of emphasized tokens.
Weakly Supervised Reinforcement Learning (RL): Since ground-truth evidence labels are rarely available, the Actor is optimized using Policy Gradient RL. The reward signal is derived solely from the Solver’s downstream task performance (e.g., F1, Accuracy, or NDCG).
- The authors use a Grouped Policy Gradient approach to reduce variance by sampling multiple masks per instance and normalizing rewards within that group.
- The loss function includes an entropy bonus (for exploration) and a target-length regularizer (to stabilize training).

3. Key Contributions

Decoupled Architecture: Separates the "sifting" (Actor) from the "thinking" (Solver), allowing the use of powerful, frozen, or even API-based models (like GPT-5 mini) without needing access to their gradients.
Non-Destructive Intervention: Unlike pruning or summarization, HiLight preserves input fidelity, which is critical for complex reasoning tasks like multi-hop QA.
Zero-Shot Transferability: The learned emphasis policy is not overfitted to a specific backbone; it transfers effectively to different model families and scales (from smaller models to much larger ones).
Interpretability: The explicit tags provide a human-readable "rationale," showing exactly which parts of the context the model used to reach its conclusion.

4. Experimental Results

The framework was evaluated across diverse tasks: Sequential Recommendation (Amazon-Beauty), Multi-hop QA (HotpotQA), Reading Comprehension (SQuAD 2.0), and Biomedical QA (PubMedQA).

Superior Performance: HiLight consistently outperformed strong baselines, including manual instructions and state-of-the-art automated prompt optimization methods (e.g., OPRO, DSPy, APE, PRL).
Sparse-Signal Success: The largest gains were observed in the Amazon-Beauty dataset (up to +27.5% improvement in HR@10), where evidence is extremely sparse and distractors are high.
Robustness to Context Length: In HotpotQA, HiLight maintained its advantage even as context length was artificially extended to 32K tokens.
Efficiency: HiLight requires significantly fewer Solver queries for training compared to search-based prompt optimizers (e.g., 12K vs. 120K for PRL) and adds negligible latency to the inference pipeline.
Alignment with Human Logic: Despite receiving no token-level supervision, the Actor achieved up to 0.78 F1 overlap with human-annotated evidence in HotpotQA.

5. Significance

HiLight represents a shift from "optimizing the prompt" to "optimizing the input structure." By treating evidence selection as a lightweight, learnable markup task, it provides a scalable and compatible way to enhance the utility of long-context LLMs. Its ability to work with frozen, proprietary APIs makes it highly practical for real-world deployments where model weights are inaccessible.