FACTS: Table Summarization via Offline Template Generation with Agentic Workflows

1. Problem Statement

The paper addresses Query-Focused Table Summarization (QFTS), a task where a system must generate natural language summaries of tabular data conditioned on a specific user query. Unlike generic summarization (which captures all content) or table question answering (which returns short factoid answers), QFTS requires rich, narrative reasoning tailored to user intent.

Key Limitations of Existing Approaches:

• Table-to-Text Models: Require costly fine-tuning and struggle with complex numerical reasoning and logical fidelity.
• Prompt-Based LLMs: Suffer from token limits, efficiency issues, and critical privacy risks because they require sending raw table data to external LLMs.
• Existing Agentic Pipelines: Often rely on decomposition, natural language planning, or manual templates, which lack robustness, scalability, and reusability across different tables with the same schema.

The authors argue that a practical solution must be Fast (reusable), Accurate (grounded in executable operations), and Privacy-Compliant (never exposing raw data to LLMs).

2. Methodology: The FACTS Framework

The proposed solution, FACTS (Fast, Accurate, and Privacy-Compliant Table Summarization), introduces an agentic workflow that generates Offline Templates rather than generating summaries directly from data.

Core Concept: Offline Templates

An offline template is a composite artifact consisting of:

1. Schema-Aware SQL Queries: Executable code to retrieve precise data.
2. Jinja2 Templates: Rendering logic to convert SQL results into natural language.
   Crucially, these templates are bound to the table schema and query semantics, not the raw data values. Once generated, they can be reused across any table sharing the same schema.

The Agentic Workflow

FACTS operates in three iterative stages, supported by an LLM Council (an ensemble of heterogeneous LLMs) for validation:

1. Stage 1: Schema-Guided Specification and Filtering

   • The agent analyzes the user query and table schema (without seeing raw data).
   • It generates guided questions and filtering rules (e.g., "exclude rows where category = 'expense'") to clarify intent.
   • LLM Council: Validates these specifications via majority voting to ensure they align with the query.

2. Stage 2: SQL Queries Generation

   • The agent synthesizes SQL queries based on the validated specifications.
   • Queries are executed locally against the data.
   • LLM Council: Reviews execution results (errors, empty sets) and provides feedback. The agent iteratively refines the SQL until it is robust and executable.

3. Stage 3: Jinja2 Template Generation and Alignment

   • The agent generates a Jinja2 template to render the SQL output into a narrative.
   • LLM Council: Checks for alignment between the SQL output structure and the template variables (e.g., ensuring column names match).
   • Iterative refinement occurs until the pair is consistent.

The LLM Council

To mitigate hallucinations and ensure correctness, FACTS employs an ensemble of LLMs (GPT-4o-mini, Claude-4 Sonnet, DeepSeek v3). Instead of relying on a single model, the Council:

• Independently evaluates intermediate artifacts (specifications, SQL, templates).
• Uses majority voting to accept or reject artifacts.
• Aggregates feedback to guide iterative refinement.

3. Key Contributions

1. Offline Template Generation: A novel paradigm producing reusable, schema-specific artifacts (SQL + Jinja2) that enable scalability to large tables without re-inference.
2. FACTS Framework: An agentic workflow integrating guided specification, SQL synthesis, and template rendering, secured by an iterative LLM Council feedback loop.
3. Privacy-Compliant Design: The system only exposes table schemas and queries to LLMs; raw data values remain local and are never transmitted to external models.
4. Comprehensive Evaluation: Extensive experiments demonstrating superiority over state-of-the-art baselines in accuracy, reusability, and scalability.

4. Experimental Results

The authors evaluated FACTS on three benchmarks: FeTaQA, QTSumm, and QFMTS.

• Performance: FACTS consistently achieved the best or second-best results across BLEU, ROUGE-L, and METEOR scores on all datasets, outperforming both prompt-based methods (e.g., DirectSumm, Reason-then-Summ) and other agentic frameworks (e.g., Binder, SPaGe).
• Human Evaluation:
  • Factuality: 94% SQL execution accuracy and 98% template rendering accuracy.
  • Preference: In side-by-side comparisons with the strongest baseline (SPaGe), FACTS was preferred for completeness (55%), correctness (59%), and hallucination reduction (60%).
• Reusability & Scalability:
  • Reusability: Once an offline template is generated, applying it to new tables with the same schema is significantly faster than re-running full LLM inference.
  • Scalability: As table size increases (100 to 1,000 rows), FACTS runtime remains flat (dependent only on schema), whereas baselines show linearly increasing latency due to token limits.
• Ablation Study: Removing the iterative Council validation (Single-Call variant) resulted in a significant drop in SQL pass rates (83.2% vs. 100% for FACTS) and lower text quality, proving the necessity of the agentic refinement loop.

5. Significance and Impact

FACTS represents a paradigm shift in table summarization by decoupling logic generation from data execution.

• Practicality: It solves the "token limit" and "cost" problems of large tables by using lightweight SQL execution instead of feeding massive datasets to LLMs.
• Trust & Privacy: By adhering to a threat model where raw data never leaves the local environment, it makes LLM-based summarization viable for sensitive domains like finance, healthcare, and law.
• Reliability: The combination of executable SQL grounding and multi-model validation significantly reduces hallucinations, a persistent issue in generative AI.

Limitations: The current approach assumes a static schema; it does not automatically adapt to schema drift (e.g., renamed columns) without regenerating the template. Additionally, while raw data is protected, the schema itself is exposed to the LLM, which may be a concern in zero-trust environments.

In conclusion, FACTS provides a robust, scalable, and privacy-preserving framework for real-world query-focused table summarization, establishing a new standard for agentic workflows in structured data tasks.