💬 NLP

Less is More: Lightweight Prompt Compression for Question Answering Applications on Edge Devices

The paper introduces CORE, a lightweight, two-stage prompt compression method that eliminates the need for auxiliary small language models to significantly improve accuracy, reduce memory usage, and accelerate inference for retrieval-augmented question answering on resource-constrained edge devices.

Original authors: Zihuai Xu, Ruofei Hou, Yang Xu, Hongli Xu, Yunming Liao, Ying Zhu

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zihuai Xu, Ruofei Hou, Yang Xu, Hongli Xu, Yunming Liao, Ying Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery. You have a massive stack of old newspapers, diaries, and letters (the retrieved context) that might contain the answer to your question. However, 95% of this stack is just weather reports, advertisements, and gossip that has nothing to do with your case.

If you try to read the entire stack to a very smart but busy assistant (the Large Language Model or LLM), two bad things happen:

The Assistant gets overwhelmed: It takes too long to read through all the junk, and it might get confused by the noise, missing the actual clue.
The Assistant runs out of room: Your phone or edge device (like a smart car or robot) doesn't have enough memory to hold the whole stack at once.

The Problem with Current Solutions

Right now, to clean up this stack, people use a "mini-detective" (a Small Language Model or SLM) to read the papers and tell the main assistant what to keep.

The Catch: This mini-detective is heavy. It requires a lot of memory and battery power. Trying to run this mini-detective on a smartphone is like trying to carry a heavy refrigerator in your backpack; it's too slow and drains the battery instantly.

The Solution: CORE (Context Refinement via Entity-aware filtering)

The authors of this paper propose a new method called CORE. Instead of hiring a heavy mini-detective, CORE uses a clever, lightweight two-step process that works like a skilled librarian with a magnifying glass.

Step 1: The "Who, What, Where" Filter (Candidate Filtering)

Instead of reading every word, CORE first looks at the question to guess what kind of answer you need.

The Analogy: If you ask, "Who was the captain?", CORE knows you are looking for a Person. If you ask, "When did it happen?", it knows you are looking for a Date.
The Action: It quickly scans the text to find sentences containing those specific "types" of words (like names or dates). It also finds sentences that sound very similar to your question.
The Result: It creates two short lists:
1. The Answer Set: Sentences that might contain the answer.
2. The Clue Set: Sentences that provide the context or evidence to prove the answer is right.
Why it's cool: This step uses tiny, lightweight tools (like a simple magnifying glass) that fit easily in a pocket, unlike the heavy refrigerator of the old methods.

Step 2: The "Cross-Check" (Evidence Refining)

Now, CORE has two lists, but they might still be too long or contain some fake leads. It needs to clean them up further without using a heavy computer.

Refining the Clues: Imagine the clues are puzzle pieces. CORE arranges them so they don't repeat the same information. If two clues say the same thing, it keeps the best one and discards the duplicate. It does this by checking if a new clue adds something new to the story.
Pruning the Answers: CORE checks if the "Answer" sentences are actually supported by the "Clue" sentences. If an answer sentence is far away from the clues that support it, or if it's just a random mention of a name without context, CORE throws it out. It keeps only the answers that are "standing next to" their proof.

The Results: Fast, Light, and Accurate

The authors tested CORE on real edge devices, like an NVIDIA Jetson (a powerful computer for robots/cars) and a Huawei smartphone.

Speed: CORE is incredibly fast. While other methods took over a minute to clean up a document, CORE did it in seconds.
Memory: It uses a tiny fraction of the memory. If other methods needed a whole hard drive's worth of space, CORE fits in a USB stick.
Battery: On a smartphone, CORE saved 95% of the energy compared to the best existing method. It's like switching from a gas-guzzling truck to an electric scooter.
Accuracy: Despite throwing away so much text, the AI actually got better at answering questions. By removing the noise, the AI focused on the signal. In tests, it improved accuracy by over 30% compared to other compression methods.

Summary

CORE is a smart, lightweight way to shrink massive documents down to their most important parts so they can fit on small devices like phones. It doesn't use a heavy "mini-AI" to do the work; instead, it uses smart rules about entities (names, dates, places) and relationships to filter out the junk. This makes AI assistants on edge devices faster, more accurate, and much less likely to drain your battery.

Technical Summary: CORE – Lightweight Prompt Compression for QA on Edge Devices

Problem Statement
In Retrieval-Augmented Generation (RAG) systems for Question Answering (QA), Large Language Models (LLMs) often suffer from the "long-context problem." Retrieved documents frequently contain substantial redundant information (often exceeding 95% noise), which dilutes key evidence and can degrade response accuracy. Furthermore, the quadratic complexity of self-attention mechanisms means that long prompts significantly increase inference latency and memory usage. This is particularly critical for edge devices (e.g., smartphones, autonomous vehicles), where computational resources and context windows are constrained.

Existing prompt compression methods typically rely on auxiliary Small Language Models (SLMs) to estimate token or sentence importance. However, these approaches introduce prohibitive overhead for edge deployment:

Memory Overhead: SLMs (e.g., 7B parameter models) require 14–24 GB of VRAM, far exceeding the 8–16 GB available on consumer edge GPUs.
Latency: The prefilling of prompts by SLMs introduces delays exceeding one minute on mobile devices, rendering them unsuitable for real-time QA.
Training Requirements: Many methods require extensive fine-tuning on large datasets, which is resource-intensive and impractical for edge scenarios.

Methodology: CORE (Context Refinement via Entity-aware filtering)
The authors propose CORE, a lightweight, two-stage, sentence-level prompt compression framework designed to operate without auxiliary SLMs. CORE utilizes only a lightweight Named Entity Recognition (NER) classifier and an embedding model (totaling ~40 MB in memory).

Stage 1: Candidate Filtering
- Answer Set Construction: CORE analyzes the user query to identify "interested entities" (e.g., dates, locations, persons) by matching the query against a set of canonical template queries in an embedding space. It then filters the retrieved context to construct an Answer Set ( $C_{ans}$ ), consisting of sentences containing at least one of these identified entities.
- Clue Set Construction: Simultaneously, CORE constructs a Clue Set ( $C_{clue}$ ) by filtering context sentences that exhibit high semantic similarity to the user query, serving as bridging evidence.
Stage 2: Evidence Refining
- Clue Refining (Orthogonal Residual Retrieval): To address redundancy within the clue set, CORE employs an iterative orthogonal vector decomposition strategy. It sequentially selects clues that maximize the semantic alignment with the residual (unexplained) part of the query. A dynamic penalty coefficient ( $\mu_t$ ) is calibrated based on how well a selected clue bridges the query to candidate answers, ensuring diverse semantic coverage. The optimal number of clues ( $K$ ) is determined automatically using the "elbow method" on the utility score curve.
- Answer Pruning (Spatial Proximity Metric): To filter the potentially large Answer Set, CORE introduces a metric based on spatial proximity. It calculates the importance of each entity in the answer set by measuring its distance to the high-confidence clues. Entities situated in the immediate vicinity of relevant clues receive higher scores. Answer sentences are then ranked by the sum of their constituent entity scores and pruned to fit the strict token budget.

Key Contributions

SLM-Free Architecture: CORE eliminates the need for heavy auxiliary SLMs, reducing memory overhead to approximately 0.14% of a standard 7B SLM and enabling deployment on resource-constrained edge devices.
Synergistic Refinement: The paper introduces a mechanism that intertwines the refinement of clues and answers. It uses an orthogonal residual strategy to distill clues and a spatial proximity metric to validate answers, ensuring that the compressed context maintains logical coherence and reasoning chains.
Edge-Optimized Implementation: The framework is specifically designed for low latency and energy efficiency, leveraging lightweight components (spaCy NER and MiniLM embeddings) suitable for mobile and edge hardware.

Experimental Results
The authors evaluated CORE on an NVIDIA Jetson AGX Orin (local inference) and a Huawei Nova smartphone (edge-cloud offloading) across six LongBench datasets (e.g., 2WikiMQA, HotpotQA, NarrativeQA) using three 7B LLMs.

Accuracy: Within a 2000-token budget, CORE improved accuracy by at least 30.19% compared to state-of-the-art baselines (LLMLingua, LLMLingua-2, LongLLMLingua). Notably, on the 2WikiMQA dataset with LLaMA2-7B-chat-4k, CORE outperformed the uncompressed baseline.
Efficiency:
- Memory: CORE reduced memory usage by at least 50.47% compared to baselines, with peak usage ranging from 0.7 GB to 1.7 GB.
- Latency: CORE achieved a 48.52% reduction in compression latency compared to LLMLingua-2 and up to a 98.99% speedup compared to methods using 7B SLMs.
- Energy: On the Huawei Nova smartphone, CORE demonstrated a 95.74% reduction in energy consumption compared to LLMLingua-2.
Robustness: CORE maintained superior performance even under severe token budget constraints (down to 500 tokens), whereas baselines suffered significant degradation.

Significance and Claims
The paper claims that CORE addresses the critical bottleneck of deploying RAG-based QA agents on edge devices. By removing the dependency on heavy SLMs, CORE makes prompt compression feasible for real-time, mobile, and industrial edge applications where memory and energy are scarce. The authors emphasize that their method not only reduces computational costs but also enhances answer accuracy by effectively filtering noise and preserving the logical chains necessary for complex reasoning, thereby mitigating the "lost in the middle" phenomenon and hallucinations associated with long contexts. The work positions itself as a practical, generalizable solution for the next generation of edge AI agents.