Beyond Rows to Reasoning: Agentic Retrieval for Multimodal Spreadsheet Understanding and Editing

The paper introduces Beyond Rows to Reasoning (BRTR), a multimodal agentic framework that overcomes the limitations of single-pass retrieval and context compression in enterprise spreadsheet analysis by employing an iterative tool-calling loop, achieving state-of-the-art performance on multiple benchmarks while maintaining full auditability.

The Gist

Imagine you are a detective trying to solve a massive mystery. The clues aren't hidden in a single notebook; they are scattered across thousands of different notebooks, some filled with numbers, some with handwritten notes, and some with blurry photos.

In the past, AI detectives had two main ways to tackle this:

1. The "Photocopy" Method: They tried to photocopy every single page of every notebook and shove it all into their brain at once. But their brains (memory) were too small, so they had to squint, blur the details, or miss entire chapters.
2. The "One-Shot" Method: They took a quick glance at the top of the pile, grabbed the first few pages they saw, and tried to guess the answer immediately. If they missed a crucial clue on page 500, they failed.

"Beyond Rows to Reasoning" (BRTR) is a new, super-smart detective framework that changes the game. Instead of guessing or cramming, it acts like a human expert with a magnifying glass and a notepad.

Here is how it works, broken down into simple concepts:

1. The "Iterative Loop": Don't Guess, Investigate

Imagine you are looking for a specific number in a spreadsheet.

• Old Way: The AI guesses, "It's probably in the 'Sales' tab," looks there, and if it doesn't find it, it gives up or hallucinates a number.
• BRTR Way: The AI says, "Hmm, I don't see it in 'Sales'. Let me check the 'Inventory' tab. Oh, there's a note there pointing to 'Q3 Report'. Let me open that. Ah! There it is."
  It keeps asking questions, checking different tabs, and refining its search until it has enough evidence to be sure. It doesn't stop until the puzzle is solved.

2. The "Smart Librarian" (Indexing)

Before the detective starts working, the system organizes the library. It doesn't just read the text; it understands the structure.

• It knows that a "chart" is different from a "row of numbers."
• It knows that "Cell A1" is connected to "Cell B2."
• The paper tested five different "librarians" (embedding models) to see who was best at finding the right pages. They found that NVIDIA NeMo was the best librarian for this specific job, able to find both text and images perfectly.

3. The "Project Manager" (Planner)

Real-world spreadsheet tasks are messy. You might need to:

1. Find a number in a PDF.
2. Type it into Excel.
3. Create a chart.
4. Check if the math adds up.

A single AI trying to do all this at once gets confused (like a chef trying to chop onions, bake a cake, and wash dishes simultaneously).
BRTR uses a Project Manager. This manager breaks the big job into tiny, manageable steps. It sends the "chopping" task to one worker, the "baking" task to another, and checks their work before moving to the next step. This prevents the AI from getting lost in a long chain of errors.

4. The "Memory Saver" (Context Management)

As the detective gathers clues, the pile of papers on their desk grows. If they keep every photo they ever looked at, the desk gets too cluttered to think.
BRTR has a clever trick: It keeps the notes about the photos (e.g., "This photo showed a red graph") but throws away the actual heavy photo files once it's done looking at them. This keeps the detective's brain fresh and focused on the current clue without running out of space.

Why Does This Matter?

The paper tested this new detective against the old methods on three huge challenges:

• FRTR-Bench: A test of complex, multi-sheet spreadsheets. BRTR scored 99% accuracy, while the old methods scored around 74%.
• SpreadsheetLLM: A test of standard spreadsheet questions. BRTR scored 98%, beating the previous best by a wide margin.
• FINCH: A test of real-world accounting workflows (like balancing a budget across 10 different files). BRTR scored 95%, while the old "naive" methods barely scored 20%.

The Bottom Line

Think of BRTR not as a machine that "reads" a spreadsheet, but as a reasoning partner. It doesn't just look at the data; it thinks about where to look, checks its work, asks for help when it's stuck, and keeps a perfect record of every step it took.

It turns the AI from a "fast guesser" into a "careful investigator," making it reliable enough to handle the messy, complex spreadsheets that real businesses use every day.

Technical Summary

Here is a detailed technical summary of the paper "Beyond Rows to Reasoning: Agentic Retrieval for Multimodal Spreadsheet Understanding and Editing" (BRTR).

1. Problem Statement

Enterprise spreadsheet workbooks often contain millions of cells, complex cross-sheet dependencies, and embedded visual artifacts (charts, images). Current Large Language Model (LLM) approaches struggle to analyze these documents effectively due to three primary limitations:

• Single-Pass Retrieval: Existing methods retrieve context once and generate an answer, lacking the ability to iteratively refine queries or cross-reference data across multiple sheets based on intermediate results.
• Context Loss via Compression: Compression-based methods (e.g., SheetCompressor) reduce token counts but often discard critical cross-sheet relationships and multimodal data, leading to reasoning failures.
• Context Window Exceedance: Naive full-context injection of large workbooks exceeds LLM context windows, while compression loses resolution.
• Lack of Multimodal Guidance: There is no empirical guidance on which multimodal embedding models best capture the semantic structure of mixed tabular and visual spreadsheet data.

2. Methodology: The BRTR Framework

BRTR introduces a multimodal agentic framework that replaces single-pass retrieval with an iterative tool-calling loop. The system is designed to mimic human analysts who incrementally gather evidence, refine their search, and validate findings.

Core Components:

1. Multimodal Indexing and Retrieval:

   • Chunking: Workbooks are decomposed into four chunk types: rows, columns, rectangular windows, and embedded images. Metadata (sheet name, coordinates, headers) is preserved.
   • Embedding: A multimodal embedding model (identified as NVIDIA NeMo Retriever 1B via evaluation) encodes these chunks.
   • Hybrid Search: At query time, the system uses Reciprocal Rank Fusion (RRF) to combine dense cosine-similarity search with BM25 lexical matching (crucial for exact numbers and cell references).

2. Agentic Tool-Calling Loop:

   • Instead of a single generation step, the LLM enters a loop where it:
     1. Analyzes the query to determine relevant content types.
     2. Invokes specific search tools (e.g., search_rows, search_columns, search_images).
     3. Inspects returned chunks and decides if more evidence is needed.
     4. Refines queries or applies coordinate filtering based on intermediate results.
   • Context Management: To prevent token overflow, the system employs an "always-prune" strategy for images, retaining only textual metadata of previous iterations while keeping the most recent batch.

3. Planner-Executor Architecture:

   • Planner: Decomposes complex, multi-step enterprise workflows into a Dependency Graph (DAG) of sub-tasks. Independent branches execute concurrently.
   • Executors: Six specialized executors handle distinct capabilities:
     • Excel: Read/write cells, formulas, formatting, charts.
     • IO: PDFs, CSVs, DOCX.
     • Web: Search and URL fetching.
     • Validation: Accounting invariants (e.g., balance-sheet identity).
     • OCR: Text extraction from scanned documents.
     • Search: The base BRTR retrieval tools.

3. Key Contributions

1. Agentic Iterative Reasoning: BRTR is the first framework to replace single-pass retrieval with a multi-turn tool-calling loop for spreadsheets, enabling end-to-end workflows from analysis to structured editing.
2. Multimodal Embedding Evaluation: The authors provide the first comprehensive empirical comparison of five multimodal embedding models for spreadsheet content, identifying NVIDIA NeMo Retriever 1B as the superior model for mixed tabular/visual data.
3. State-of-the-Art Performance: The framework achieves SOTA results across three major benchmarks, significantly outperforming compression and single-pass retrieval baselines.
4. Full Auditability: The system maintains explicit tool-call traces, allowing for full end-to-end auditability of the reasoning process.

4. Experimental Results

The paper evaluates BRTR across three benchmarks using over 200 hours of expert human evaluation.

A. Multimodal Embedding Evaluation (FRTR-Bench)

• Dataset: 146 queries across 28 files.
• Result: NVIDIA NeMo Retriever 1B achieved the highest Recall@10 (0.60) and nDCG@10 (0.42), outperforming Amazon Titan, Cohere, Amazon Nova, and Jina v4.

B. Framework Evaluation (FRTR-Bench & SpreadsheetLLM)

• FRTR-Bench (Complex Enterprise Workbooks):
  • BRTR with frontier models (GPT-5, Gemini 3 Pro) achieved 99% accuracy.
  • This is a 25 percentage point improvement over the best single-pass FRTR baseline (74%).
  • Compression-based methods (SpreadsheetLLM) failed significantly (6–34% accuracy) due to the loss of cross-sheet context.
• SpreadsheetLLM Benchmark (Single-sheet tasks):
  • BRTR achieved 97–98% accuracy, surpassing the original SpreadsheetLLM compression approach (89–91%) and naive full-context injection (35–68%).
  • Efficiency: GPT-5.2 demonstrated the best efficiency-accuracy trade-off, achieving 98% accuracy with only ~20k tokens per query.

C. Generalization (FINCH Benchmark)

• Dataset: 172 complex, multi-step finance/accounting workflows involving 1,710 spreadsheets and 27 million cells.
• Result: BRTR achieved 95.3% accuracy (Claude Opus 4.6) and 94.8% (GPT-5.2).
• Comparison: This represents a massive improvement over naive single-call agents (e.g., +48.9 points for GPT-5.2) and the previous SOTA of ~42%.
• Ablation: Removing the Planner reduced accuracy by 20 points and increased token usage by 40%, proving that task decomposition is critical for both accuracy and efficiency. Removing the Agent Loop resulted in 0% accuracy, confirming the necessity of iterative reasoning.

5. Significance and Conclusion

• Paradigm Shift: BRTR moves spreadsheet AI from "static compression" to "dynamic reasoning," proving that iterative search and tool use are essential for complex, multi-sheet enterprise tasks.
• Scalability: The framework successfully handles workbooks with millions of cells and cross-file dependencies that exceed standard LLM context windows.
• Practical Impact: By identifying the optimal embedding model and demonstrating high efficiency with GPT-5.2, BRTR provides a viable path for deploying agentic spreadsheet assistants in production environments.
• Future Work: The authors plan to extend the framework to other structured document types and incorporate learned retrieval strategies that adapt to query complexity.

In summary, BRTR demonstrates that for complex enterprise data, iterative agentic reasoning significantly outperforms both compression-based and single-pass retrieval methods, offering a robust, auditable, and highly accurate solution for multimodal spreadsheet understanding.