💬 NLP

Contexts are Never Long Enough: Structured Reasoning for Scalable Question Answering over Long Document Sets

SLIDERS is a framework that enables scalable question answering over massive document collections by extracting salient information into a reconciled relational database, allowing for structured reasoning via SQL rather than relying on limited LLM context windows.

Original authors: Harshit Joshi, Priyank Shethia, Jadelynn Dao, Monica S. Lam

Published 2026-04-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Harshit Joshi, Priyank Shethia, Jadelynn Dao, Monica S. Lam

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective tasked with solving a massive mystery. To solve it, you don't just have one single notebook; you have millions of loose scraps of paper scattered across thousands of different rooms in a giant mansion.

The Problem: The "Pile of Scraps" Dilemma

Current AI (like ChatGPT) is like a detective with a very small desk. You can lay out a few scraps of paper, read them, and answer a question. But if you try to dump all 10 million scraps on the desk at once, the desk collapses, the papers fly everywhere, and the detective gets overwhelmed.

Even when AI tries to be clever by reading one scrap at a time and writing a summary, it runs into the "Aggregation Bottleneck." It’s like reading one scrap that says "The suspect is tall" and another that says "The suspect has red hair," but by the time it reads the 1,000th scrap, it has forgotten the first two or gets confused by a scrap that says "The suspect is short" (which was actually talking about a different person).

The Solution: SLIDERS (The Master Librarian)

The researchers at Stanford created SLIDERS. Instead of trying to read everything at once or just making messy summaries, SLIDERS acts like a Master Librarian who turns that mountain of loose scraps into a perfectly organized, searchable Digital Filing Cabinet (a Database).

Here is how SLIDERS works, step-by-step:

1. Contextualized Chunking (The Smart Sorting)
Instead of just ripping papers in half randomly, SLIDERS looks at the "headers" and "titles." It makes sure that if a sentence is part of a table, the whole table stays together. It’s like making sure you don't cut a map in half right where the legend is.

2. Schema Induction (The Blueprint)
Before the Librarian starts filing, they decide on a system. If the question is about "Company Profits," the Librarian creates specific folders labeled: Company Name, Year, Total Revenue, and Currency. This ensures every piece of info goes into a specific, predictable slot.

3. Structured Extraction (The Data Entry Clerk)
The AI goes through the scraps and, instead of writing a paragraph, it fills out a form. It doesn't just say "The company made a lot of money"; it writes: Company: Apple | Year: 2023 | Profit: $97B | Currency: USD. It also keeps a "receipt" (provenance) for every entry, so it can prove exactly which scrap of paper that info came from.

4. Data Reconciliation (The Fact-Checker)
This is the secret sauce. Sometimes, one scrap says "Apple's profit was $97B" and another says "Apple made $97,000,000,000." A normal AI might think these are two different things. SLIDERS has a "Reconciliation Agent" that looks at these, realizes they are the same, and merges them into one clean, perfect record. It resolves conflicts, removes duplicates, and fills in the blanks.

5. SQL Reasoning (The Expert Researcher)
Finally, when you ask a question like "Which company had the lowest debt in 2022?", the AI doesn't "read" the documents anymore. It simply sends a precise command (SQL) to the filing cabinet: "Hey, look at the 'Debt' column, filter for '2022', and tell me the lowest number." The database gives the answer instantly and accurately.

Why is this a big deal?

It’s Scalable: Because it uses a database, it doesn't matter if you have 1,000 documents or 1,000,000. The "desk" never gets too crowded.
It’s Accurate: It doesn't "hallucinate" or get confused by distant information because it’s doing math and logic on structured data, not just guessing based on text.
It’s Auditable: If the AI gives you an answer, you can follow the "receipts" back to the exact page and line of the original document. It’s not just "trust me"; it’s "here is the proof."

In short: SLIDERS turns a chaotic pile of messy text into a high-speed, ultra-accurate, searchable encyclopedia.

Technical Summary: SLIDERS (Scalable Long-document Integration through Decomposed Extraction and Reconciliation System)

1. Problem Statement: The "Aggregation Bottleneck"

Modern Large Language Models (LLMs) face a fundamental limitation when performing question answering (QA) over massive document collections. While context windows are expanding, real-world datasets (e.g., thousands of financial filings or millions of tokens of Wikipedia articles) still exceed these limits.

Current workarounds typically involve chunking documents into smaller fragments and extracting information from each. However, this creates an "Aggregation Bottleneck": as the number of chunks grows, the system must eventually combine and reason over a massive amount of extracted textual evidence. This reintroduces the long-context problem, often leading to:

Inconsistency: Contradictory or duplicated information across chunks.
Incompleteness: Missing evidence due to poor retrieval or fragmented context.
Lack of Auditability: Difficulty in tracing free-form answers back to specific source text.
Computational Expense: High costs associated with processing massive concatenated contexts.

2. Methodology: Structured Reasoning

The authors propose SLIDERS, a framework that shifts the paradigm from reasoning over unstructured text to reasoning over a persistent structured state (a relational database). The process follows five core stages:

Contextualized Chunking: To prevent "orphaned" information (e.g., a table row without its header), documents are augmented with global metadata (titles, descriptions) and local structural signals (section headers) before being partitioned into self-contained chunks.
Schema Induction: Instead of free-form extraction, an LLM derives a task-specific relational schema (tables, fields, data types, units, and normalization rules) based on the user's question and document metadata.
Structured Extraction with Relevance Gating: To minimize hallucinations, a "relevance gate" first determines if a chunk contains useful information. If it does, the model extracts data into the induced schema, explicitly storing provenance (the exact text span) and extraction rationales (the reasoning used to map text to a field).
Automated Data Reconciliation: This is the core innovation. To turn fragmented extractions into a coherent database, SLIDERS uses a two-phase reconciliation process:
- Partitioning: It identifies "primary keys" (e.g., Company Name + Fiscal Year) to group related records.
- Reconciliation Agent: An LLM agent inspects these groups and dynamically generates SQL programs to perform three operations: Deduplication (removing redundant rows), Conflict Resolution (choosing the most authoritative value among contradictions), and Consolidation (merging partial records to fill missing attributes).
Question Answering via SQL: The final QA step uses a coding agent to translate the natural language question into SQL queries. The agent executes these queries against the reconciled database, iteratively refining them until a precise answer is produced.

3. Key Contributions

Structured Reasoning Framework: A novel approach that replaces text-based aggregation with database-based symbolic computation.
Provenance-Driven Reconciliation: The first approach to treat extraction rationales and provenance as first-class signals to resolve data conflicts.
Scalability: By using SQL and key-based partitioning, the system's reasoning complexity is decoupled from the total number of tokens, allowing it to scale to tens of millions of tokens.
New Ultra-Long Benchmarks: The introduction of WikiCeleb100 (3.9M tokens) and FinQ100 (36M tokens) to test the limits of multi-document aggregation.

4. Results

SLIDERS demonstrated superior performance across multiple benchmarks:

Long-Context Benchmarks ( $\le$ 360k tokens): SLIDERS outperformed all baselines (including GPT-4.1 and GraphRAG), exceeding GPT-4.1 by an average of 6.6 points.
Ultra-Long Benchmarks ( $\ge$ 3.9M tokens):
- On WikiCeleb100, it achieved 78.91% accuracy.
- On FinQ100, it achieved 55.22% accuracy, significantly outperforming the next best baseline (which scored only 5% due to context limits).
Efficiency: SLIDERS is significantly more cost-effective than recursive language model (RLM) approaches. For the FinQ100 benchmark, SLIDERS cost approximately $34, whereas a comparable RLM run was estimated to cost $2,000.
Model Agnostic: The framework's gains are attributed to the structured pipeline rather than the power of a specific model; it showed substantial improvements even when using open-source models like Qwen.

5. Significance

SLIDERS represents a significant step toward reliable, auditable, and scalable AI analysts. By transforming the problem of "reading everything" into a problem of "querying a structured database," it provides a blueprint for handling massive, knowledge-intensive corpora in domains like finance, law, and healthcare. Its ability to provide clear provenance makes it suitable for high-stakes environments where humans must verify AI-generated insights.