An Agentic System for Schema Aware NL2SQL Generation

1. Problem Statement

The Natural Language to SQL (NL2SQL) task is critical for democratizing data access, allowing non-experts to query relational databases. However, current state-of-the-art solutions face three major barriers to real-world deployment:

• High Computational Cost: Reliance on Large Language Models (LLMs) like GPT-4 results in prohibitive operational costs.
• Data Privacy & Compliance: Proprietary LLMs often require sending sensitive data to external clouds, violating on-premise deployment mandates and data regulations.
• Complexity & Accuracy: Existing systems struggle with intricate enterprise schemas, multi-table joins, and domain-specific constraints, often leading to hallucinations or incorrect query generation.

While multi-agent frameworks (e.g., MAC-SQL) have improved accuracy by decomposing tasks, they remain heavily dependent on costly LLMs, limiting scalability.

2. Methodology

The authors propose a hybrid agentic system that strategically prioritizes Small Language Models (SLMs) for primary inference, utilizing LLMs only as an intelligent fallback mechanism for error recovery. The system is built on LangGraph and consists of four specialized agents:

• A. Schema-Aware Extractor Agent:

  • Function: Collects and synthesizes database context to prevent hallucinations.
  • Mechanism: Uses a multi-source retrieval strategy combining:
    1. Database metadata (structure/constraints).
    2. Retrieval-Augmented Generation (RAG) from documentation (table meanings, business rules).
    3. Evidence mappings from historical query-schema alignments.
  • Tech: Uses vector embeddings (all-MiniLM-L6-v2) and Chroma DB for fast retrieval (<1 second), mapping user queries to relevant schema elements.

• B. Decomposer Agent:

  • Function: Breaks complex natural language questions into structured execution plans.
  • Mechanism: Performs hierarchical task planning: entity linking, condition analysis, execution ordering, and output formatting.
  • Tech: Powered by Mistral-7B (SLM). It creates a blueprint for the generator, ensuring complex queries (e.g., nested subqueries) are logically structured before SQL generation.

• C. Generator Agent:

  • Function: Generates the actual SQL query based on the decomposition plan and schema context.
  • Mechanism: A two-stage process:
    1. Primary: Uses Llama-3.1-8B (SLM) to generate the initial SQL.
    2. Fallback: If the query fails validation, GPT-4o (LLM) is invoked to analyze error messages and regenerate the query. This loop allows up to three retries.

• D. Validator and Executor Agent:

  • Function: Ensures syntactic and semantic correctness before final execution.
  • Mechanism: Performs four checks:
    1. Evidence-based Value Validation: Corrects mismatches between natural language terms and exact database values (e.g., "East Bohemia" vs. "east Bohemia").
    2. Syntax Validation: Checks JOIN clauses, column references, and schema compatibility.
    3. Execution Validation: Runs the query in a secure transaction environment to catch runtime errors.
    4. Semantic Validation: Verifies result set structure and aggregation logic.
  • Feedback: If errors occur, diagnostic data is fed back to the Generator to trigger the LLM fallback.

3. Key Contributions

1. Hybrid Agentic Architecture: A novel pipeline where SLMs handle the majority of inference tasks, with LLMs acting strictly as a targeted error-correction fallback.
2. Schema-Aware Prompting: Integration of RAG and evidence mapping to strictly align generated queries with database constraints, reducing hallucinations.
3. Cost-Efficiency Benchmarking: A comprehensive evaluation demonstrating that high-cost LLMs are not always necessary for the majority of queries, achieving significant cost reductions without sacrificing too much accuracy.

4. Experimental Results

The system was evaluated on the BIRD benchmark (1,534 queries across 11 databases), comparing against LLM-centric baselines (MAC-SQL, DAIL-SQL, DIN-SQL).

• Performance Metrics:

  • Execution Accuracy (EX): 47.78% (vs. 59.59% for MAC-SQL).
  • Validation Efficiency Score (VES): 51.05% (vs. 67.68% for MAC-SQL).
  • Note: While accuracy is lower than pure LLM baselines, the system achieves competitive results for a fraction of the cost.

• Cost Efficiency:

  • Query Resolution: Approximately 67% of queries were successfully resolved using only the local SLM (near-zero cost).
  • Average Cost Per Query: $0.0085 for the proposed system vs. $0.094 for LLM-only systems.
  • Savings: Achieved >90% cost reduction compared to state-of-the-art LLM-based approaches.

5. Significance and Limitations

Significance:

• Economic Viability: Makes NL2SQL deployment feasible for organizations with budget constraints by shifting the load to local, open-source models.
• Data Privacy: Enables on-premise deployment, addressing critical data sovereignty and regulatory compliance issues (GDPR, HIPAA) by minimizing data exposure to external APIs.
• Scalability: Reduces the computational barrier for enterprise adoption.

Limitations:

• Accuracy Gap: The system struggles with highly complex reasoning (deep nesting, temporal logic) where SLMs underperform compared to LLMs.
• Latency: While average latency is low, the "tail latency" increases significantly for difficult queries that trigger the LLM fallback loop.
• Error Propagation: Sequential dependencies mean errors in early agents (e.g., Decomposer) can cascade, affecting downstream agents even if they function correctly.
• Schema Dependency: Performance degrades on legacy databases with poor documentation or cryptic column names, as the RAG mechanism relies on semantic similarity.

Conclusion

The paper presents a pragmatic shift in NL2SQL architecture, proving that a hybrid SLM-LLM approach can balance cost, privacy, and performance. By reserving expensive LLM resources for error recovery rather than primary generation, the system offers a secure, scalable, and economically viable solution for real-world enterprise data access.