💻 computer science

Sustainable Local RAG Architectures for Energy Governance: A Green AI Approach via Document Slice Contextualization

This paper proposes a sustainable, local RAG architecture using Document Slice contextualization and a two-tier routing agent to achieve near-baseline retrieval accuracy while drastically reducing energy consumption, emissions, and inference time for energy governance compliance.

Original authors: Martin Tibor Sándor, Róbert Lakatos

Published 2026-10-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Martin Tibor Sándor, Róbert Lakatos

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern world, organizations are drowning in documents. From energy regulations to scientific research, the information needed to make critical decisions is locked inside thousands of pages of text. To make sense of this, many institutions are turning to artificial intelligence, specifically systems that can read a document, find the relevant parts, and answer questions based on them. This process, known as retrieval-augmented generation, works by breaking a long document into smaller pieces, or chunks, and asking a computer model to summarize each piece before searching for answers. However, a significant problem has emerged: the most accurate way to do this requires the computer to read the entire document every single time it summarizes a small piece. This approach is incredibly energy-hungry and demands powerful, expensive computer hardware that most local offices cannot afford. For sectors like energy governance, where laws require that data stays within the country and cannot be sent to a distant cloud server, this creates a bottleneck. The technology exists, but the cost of running it locally is often too high, both in terms of electricity and the physical memory of the computer.

Researchers at the University of Debrecen in Hungary have developed a new method to solve this problem, allowing these powerful AI tools to run efficiently on standard office computers. Their work focuses on a technique called "Document Slice Contextualization." Instead of forcing the computer to read the entire document to understand a single paragraph, the new system intelligently decides how much surrounding text is actually needed. Imagine a librarian who, instead of handing you the entire encyclopedia to find a definition, only opens the specific page and the few pages immediately next to it. The researchers built a smart routing agent that acts like this librarian. Before summarizing a chunk of text, this agent quickly checks if the chunk is self-explanatory, like an abstract or a conclusion, or if it needs help from its neighbors to make sense. If the chunk is self-explanatory, the system skips the summarization step entirely. If it needs context, the agent decides exactly how many neighboring pages to include, rather than blindly including the whole book.

The team tested this approach on a collection of twenty-five scientific papers and four complex policy documents from the European Commission. They compared their new, smart routing method against the traditional approach that reads the whole document every time. The results were striking. The new system achieved nearly the same level of accuracy in finding the right information as the heavy, energy-intensive method, but it did so with a fraction of the effort. Specifically, the new method reduced the energy required to generate answers by 87.6 percent. In practical terms, a task that previously took nearly an hour and consumed over 100 watt-hours of electricity was completed in under nine minutes using only 13.2 watt-hours. This reduction also meant a massive drop in carbon emissions, cutting the output from 21.8 grams of carbon dioxide equivalent down to just 2.8 grams.

A key finding of the study is that the "smart" routing agent did not just save energy; it actually improved the quality of the search in some ways. By skipping the summarization of chunks that were already clear, the system avoided adding unnecessary noise to the search process. This allowed the computer to find the most relevant information faster and more precisely. The researchers measured this using a standard test of 250 difficult questions that required connecting information from multiple parts of a document. Their new system found the correct answers in 95.2 percent of the cases, a figure almost identical to the 95.4 percent achieved by the much slower, energy-heavy method. Furthermore, the system successfully ran on a standard laptop graphics card with only 8 gigabytes of memory, a device that would have crashed under the weight of the traditional method.

The implications of this work extend beyond just saving electricity. For industries like energy governance, where strict laws dictate that data must remain local and under the control of the organization, this technology makes advanced AI feasible. It proves that high-performance artificial intelligence does not require massive, centralized data centers. Instead, it can be performed locally on everyday hardware, making the technology accessible to smaller institutions and ensuring that sensitive data never leaves the premises. The researchers describe their approach as "Green AI," emphasizing that efficiency should be a primary goal alongside intelligence. By making the process precise, explainable, and provable, they have created a pathway for organizations to audit their own compliance and manage their energy systems with tools that are both powerful and sustainable. The study confirms that with the right design, we can have the best of both worlds: the deep understanding of large AI models without the massive environmental and hardware costs that usually come with them.

Technical Summary: Sustainable Local RAG Architectures for Energy Governance

Problem Statement
The paper addresses the tension between the computational demands of Retrieval-Augmented Generation (RAG) and the strict constraints of energy governance and data sovereignty. Current "Red AI" approaches to index-time contextualization—where every document chunk is summarized against the entire document—rely on quadratic self-attention and linearly growing key-value (KV) caches. This pattern pushes memory requirements beyond the 8 GB VRAM of consumer-grade GPUs, forcing organizations to rely on cloud data centers. However, energy regulators and auditors often cannot externalize data processing due to data-sovereignty rules (e.g., GDPR, AI Act) and the need for on-premises, defensible compliance auditing. The challenge is to achieve high-accuracy RAG on commodity hardware without the quadratic memory and energy costs of full-document contextualization.

Methodology
The authors propose a "Green AI" architecture centered on Document Slice Contextualization and a Two-Tier Routing Agent, designed to run entirely on a single workstation with an 8 GB GPU.

Document Slice Contextualization: Instead of feeding the entire document to the summarizer for every chunk, the system defines a bounded "slice" of neighboring chunks around a target chunk $c_i$ . The slice size is determined by a context radius $k$ . To handle document boundaries, the implementation clamps the slice, ensuring the target chunk remains central to the context window. This bounds the prompt length to a fixed size regardless of the total document length, preventing the KV cache from exceeding physical VRAM.
Two-Tier Routing Agent: A routing agent determines the optimal radius $k$ $k$ for each chunk before any summarization occurs, using the same small language model (SLM) that performs the summarization.
- Tier 1 (Deterministic Gate): A regex-based gate identifies "table chunks" (serialized tables or captions) and assigns them a fixed radius ( $k=2$ ) without invoking the model.
- Tier 2 (SLM Router): For non-table chunks, the agent performs a single-token forward pass of a 4-billion-parameter model (Gemma 3 4B-it-qat). The model decides if a chunk is "self-describing" (e.g., abstracts, conclusions, references) and requires no context ( $k=0$ , skipping summarization) or "needs neighbors" ( $k=2$ ).
Pipeline Execution: The pipeline runs sequentially on a single GPU to avoid memory swapping:
- Routing & Summarization: The router assigns radii; the summarizer generates context-enriched summaries for chunks where $k > 0$ .
- Embedding & Indexing: The enriched text is embedded using nomic-embed-text-v1.5 and indexed in LanceDB.
- Retrieval: A hybrid retrieval system (BM25 + dense vector search) re-ranks results using ColBERTv2.

Key Contributions

Formalization of Bounded Slices: The paper provides a formal definition of bounded, boundary-clamped document slices and a complexity model proving that this approach reduces index-time attention work from $O(n^3)$ (full-document) to $O(n)$ (bounded slice), where $n$ is the number of chunks.
Per-Chunk Routing Policy: It introduces a lightweight routing mechanism that dynamically selects context windows, skipping summarization for self-describing chunks entirely. This reduces the number of summarizer calls and total prompt tokens.
Green AI Measurement Protocol: The study implements a rigorous energy accounting protocol using CodeCarbon, tracking GPU, CPU, and memory power to generate per-run energy ledgers and carbon emission estimates, ensuring reproducibility.
Local Feasibility: It demonstrates that a high-fidelity RAG pipeline can operate within the 8 GB memory envelope of a consumer laptop GPU (RTX 4060), a critical requirement for on-premises energy governance.

Results
Evaluated on 25 scientific papers (382 chunks, 250 multi-hop questions) and four European Commission policy documents:

Accuracy: The routed pipeline achieved a Recall@20 of 0.952, statistically indistinguishable from the full-document baseline (0.954) and superior to the static $k=3$ slice (0.945). It also achieved the highest Mean Reciprocal Rank (MRR@20) and Mean Average Rank (MAR@20).
Efficiency:
- Energy: The routed pipeline reduced tracked generation energy by 87.6% (from 106.7 Wh to 13.2 Wh) and CO2 emissions by 87.0% (from 21.8 g to 2.8 g) compared to the full-document baseline.
- Time: Generation time was reduced by 84.1% (from 54:54 to 8:45).
- Memory: The approach kept the KV cache within the 8 GB limit, whereas the full-document baseline required a 30,000-token window that would cause memory paging on the same hardware.
Routing Behavior: The router assigned a mean radius of 1.37, skipping 32% of chunks (mostly abstracts, references, and acknowledgments) and assigning $k=2$ to the rest. This confirmed that not all chunks require full-document context.
Cross-Domain Robustness: On European Commission policy documents, a fixed-radius variant ( $k=3$ ) maintained performance within 0.6 points of the baseline while saving 71.8% of the energy.

Significance and Claims
The paper claims to offer a PEP-AI pathway (Precise, Explainable, Provable) for decentralized compliance auditing. By bounding context and routing decisions, the architecture enables:

Reproducible Metrics: Performance is measured against gold standards with statistical significance testing.
Inspectable Decisions: Every routing decision is logged with its evidence (log-probabilities), allowing for audit trails.
Verifiable Energy: Per-run energy ledgers provide a provable record of the carbon footprint of the AI process.

The authors assert that this approach allows energy governance bodies to run sophisticated, defensible RAG systems on local commodity hardware, adhering to data sovereignty regulations without sacrificing the accuracy of full-document contextualization. The work positions bounded, routed contextualization as a dominant Pareto choice, offering near-optimal accuracy with drastically reduced time and energy costs.

Technical Summary: Sustainable Local RAG Architectures for Energy Governance

More like this