💬 NLP

SQUiD: Synthesizing Relational Databases from Unstructured Text

The paper introduces SQUiD, a novel neurosymbolic framework that leverages large language models to automatically synthesize relational database schemas and populate tables from unstructured text, demonstrating superior performance across diverse datasets compared to existing baselines.

Original authors: Mushtari Sadia, Zhenning Yang, Yunming Xiao, Ang Chen, Amrita Roy Chowdhury

Published 2026-03-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mushtari Sadia, Zhenning Yang, Yunming Xiao, Ang Chen, Amrita Roy Chowdhury

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, messy pile of handwritten travel journals, medical notes, and business reports. These are unstructured texts—just paragraphs of words with no clear organization. Now, imagine you want to turn this chaotic pile into a perfectly organized Relational Database (like the kind used by banks or airlines), where data is neatly stored in tables, linked together, and ready for instant searching.

Doing this manually is like trying to sort a library by hand while blindfolded. It's slow, error-prone, and boring.

Enter SQUiD (Synthesizing Qualitative Unstructured Data). Think of SQUiD not as a magic wand, but as a super-smart, neuro-symbolic construction crew that builds a database from scratch using a four-step assembly line.

Here is how SQUiD works, explained through a creative analogy:

The Problem: The "Direct Prompt" Disaster

If you just ask a standard AI (a Large Language Model or LLM) to "Make me a database from this text," it's like asking a painter to build a house.

The Result: The AI might hallucinate (make up) fake names, forget to link the "Traveler" table to the "Trip" table, or write SQL code with syntax errors (like a sentence with no verbs). It's messy and often breaks.

The SQUiD Solution: The Four-Stage Assembly Line

SQUiD breaks the impossible task into four manageable jobs, using a mix of AI creativity and rigid, rule-based logic (the "neurosymbolic" part).

Stage 1: The Architect (Schema Generation)

The Job: Before building, you need a blueprint. The AI reads the text and decides: "Okay, we need a table for Travelers, one for Destinations, and one for Trips."
The Trick: It doesn't just guess. It follows strict architectural rules (like a building code). It ensures every table has a unique ID (Primary Key) and that the tables are linked correctly (Foreign Keys).
Analogy: Imagine an architect who refuses to draw a house without making sure the kitchen connects to the dining room and that the roof doesn't float in the air. This stage ensures the database structure is legally and logically sound before any data is added.

Stage 2: The Scavenger Hunt (Value Identification)

The Job: Now, the crew needs to find the actual bricks (data points) hidden in the text.
The Trick: SQUiD uses two methods to find these bricks:
1. The Robot Scanner (Symbolic Tools): A strict, rule-based tool that grabs obvious nouns and numbers (e.g., "Sophia," "34," "Rome"). It's fast but might miss context.
2. The Detective (LLM): A creative AI that reads the sentences to understand context (e.g., realizing "premium package" is a specific type of booking).
The Magic: It combines both lists. If the robot missed "premium," the detective adds it. Then, it cleans up duplicates (so "Rome" isn't listed twice for the same trip).

Stage 3: The Assembly Line (Table Population)

The Job: Now, we have a pile of bricks and a blueprint. We need to put the right bricks in the right places.
The Trick: This is where SQUiD gets clever. Instead of asking the AI to write the whole database at once (which causes confusion), it asks the AI to build one row at a time using a specific tool.
Analogy: Imagine a factory robot arm. Instead of being told "Build a car," it's told "Attach the left front wheel." Then "Attach the right front wheel." By breaking it down, the AI never gets confused about which wheel goes on which car. It also ensures that if "Sophia" is in the Traveler table, her ID in the Trip table matches perfectly. No broken links!

Stage 4: The Mason (Database Materialization)

The Job: Turn the organized bricks into a solid, working building (SQL code).
The Trick: Instead of asking the AI to write the complex SQL code (which is prone to typos), SQUiD takes the clean, organized data from Stage 3 and uses a computer program to automatically write the SQL commands.
Analogy: The AI builds the structure, but a computer script acts as the mason, laying the bricks perfectly according to the blueprint. This guarantees the final database has zero syntax errors and runs immediately.

Why is this a Big Deal?

It's a New Game: Previous AI tools could only answer questions about existing databases (Text-to-SQL) or make simple flat tables. SQUiD is the first to build a complex, multi-table relational database from scratch just by reading text.
It's Reliable: By splitting the work, SQUiD avoids the "hallucinations" (making things up) that plague standard AI.
It Works Everywhere: Whether the text is about tourism, finance, or mental health, SQUiD adapts its construction crew to build the right kind of database.

The Bottom Line

SQUiD is like a translator and architect rolled into one. It takes the messy, unstructured story of your data and turns it into a pristine, queryable, and perfectly linked database, ensuring that every piece of information is in the right place, linked correctly, and ready to be used. It bridges the gap between the "wild west" of text documents and the "ordered city" of relational databases.

1. Problem Definition: Text2R

The paper addresses a critical gap in data management: the inability to directly analyze the vast amount of unstructured text data (e.g., medical records, business reports, academic articles) using standard relational database tools.

The Task (Text2R): The authors define a new task called Text2R (Text to Relational Database). Unlike existing tasks like Text-to-SQL (which assumes a pre-existing schema) or Text-to-Table (which generates flat, non-relational tables), Text2R requires synthesizing a relational database from scratch.
The Goal: Given an unstructured natural language document $D$ $D$ , the system must generate:
1. Schema ( $R$ ): A set of CREATE TABLE statements defining tables, columns, data types, primary keys (PK), and foreign keys (FK) that capture entity-relationship semantics.
2. Data ( $S$ ): A set of INSERT INTO statements populating these tables with extracted tuples, ensuring referential integrity (e.g., consistent entity IDs across tables).
Challenges:
- Semantic Complexity: Identifying interrelated entities and relationships from free text.
- Syntactic Integrity: Ensuring the generated schema adheres to relational database constraints (PK/FK definitions, no reserved keywords).
- Value Consistency: Ensuring the same entity (e.g., "Sophia") is represented consistently across different tables with the same ID.
- Execution: Generating valid, executable SQL without hallucinations or syntax errors.

2. Methodology: The SQUiD Framework

The authors propose SQUiD (SQL on Unstructured Data), a neurosymbolic framework that decomposes the Text2R task into four modular stages. This decomposition allows for specialized techniques at each step, combining the flexibility of Large Language Models (LLMs) with the precision of symbolic tools.

Stage 1: Schema Generation

Goal: Infer a relational schema (tables, columns, PKs, FKs) from the text.
Approach: The system uses LLMs guided by specific prompting strategies (Direct vs. Chain-of-Thought).
- Constraints: The prompts explicitly enforce relational best practices, such as defining primary keys for uniqueness, establishing foreign keys for relationships, and avoiding SQL reserved keywords.
- Decoupling: By separating schema generation from data population, the system can validate the schema's syntactic correctness independently before attempting to populate it.

Stage 2: Value Identification

Goal: Extract granular values from the text and organize them into intermediate representations.
Approach: A hybrid neurosymbolic extraction process using Triplets.
- Symbolic Triplets: Uses deterministic tools (Stanford CoreNLP) to extract (Subject, Relation, Object) triplets. This ensures high recall for specific tokens (e.g., modifiers like "premium") that LLMs might miss.
- Schema-Aligned Triplets: Uses LLMs to extract (Table Column, Value) triplets, ensuring structural alignment with the generated schema.
- Deduplication & Grouping:
  - Deduplication: Uses sentence embeddings (Sentence-T5) and cosine similarity to remove redundant triplets.
  - Grouping: Assigns a unique identifier (Superkey) to each paragraph/entity instance. This ensures that values belonging to "Sophia" are grouped together and distinct from "James," enabling correct cross-table linking later.

Stage 3: Table Population

Goal: Map the identified values to the schema to form tuples.
Approach: The system employs an Ensemble Learning strategy. It runs three independent extraction prompts:
1. Text-only input.
2. Text + Symbolic Triplets.
3. Text + Schema-Aligned Triplets.
- Tool Use: Instead of asking the LLM to generate a massive JSON block (which often leads to formatting errors), the system uses a lightweight tool-use prompting technique. The LLM is instructed to output one structured record at a time, strictly adhering to the schema.
- Integration: The outputs from the three sources are combined to maximize coverage and accuracy.

Stage 4: Database Materialization

Goal: Convert the structured output into executable SQL.
Approach: Symbolic Translation.
- Instead of asking the LLM to generate SQL directly (which is prone to syntax errors and hallucinations), the system programmatically parses the structured tuples generated in Stage 3.
- It automatically constructs valid CREATE TABLE and INSERT INTO statements.
- These statements are executed on a local SQLite instance to instantiate the final database, guaranteeing syntactic correctness.

3. Key Contributions

Definition of Text2R: Formalized the novel task of synthesizing relational databases from unstructured text, distinguishing it from Text-to-SQL and flat Text-to-Table generation.
SQUiD Framework: Introduced a neurosymbolic, four-stage pipeline that effectively handles the complexity of schema inference, value extraction, and referential integrity.
Automated Benchmarking: Created a new dataset generation pipeline using existing relational databases (BIRD, Kaggle) to generate synthetic unstructured text paired with ground-truth databases, as no such benchmark previously existed.
Novel Evaluation Metrics: Proposed a comprehensive suite of metrics covering:
- Schema Quality: Entity Coverage (ECS), Primary Key Coverage (PKC), Foreign Key Coverage (FKC).
- Tuple/Database Quality: Database Construction Success Rate (DBR), Referential Integrity Rate (RRIR), Tuple Coverage (TC), Value Coverage (VC), and Column Consistency (CC).

4. Experimental Results

The authors evaluated SQUiD against direct prompting baselines (zero-shot LLMs) across diverse domains (Tourism, Education, Finance, Mental Health, etc.) using five state-of-the-art models (GPT-4O, DeepSeek-V2.5, Claude 3.7, Llama-3-8B, Qwen-3-8B).

Schema Generation:
- Chain-of-Thought (CoT) prompting generally outperformed direct prompting in Entity Coverage (ECS).
- SQUiD achieved near-perfect Primary Key (PKC) and Foreign Key (FKC) coverage, whereas baselines often failed to define keys or relationships.
Database Construction:
- Success Rate: SQUiD achieved a 100% Database Construction Success Rate (DBR) across almost all models and difficulty levels. In contrast, baseline DBR was as low as 9.7% (GPT-4O) due to syntax errors.
- Referential Integrity: SQUiD significantly improved Referential Integrity Rate (RRIR), with improvements ranging from 1.4x to 46x over baselines.
Tuple Accuracy:
- SQUiD consistently outperformed baselines in Tuple Coverage (TC), Value Coverage (VC), and Column Consistency (CC).
- Notable Finding: Smaller models (8B parameters like Llama-3 and Qwen-3) running under the SQUiD framework outperformed much larger models (like GPT-4O and Claude) running as baselines. This highlights that the framework's design choices (decomposition, tool use, ensemble) are more impactful than raw model size.

5. Significance and Conclusion

Bridging the Gap: SQUiD provides a practical solution to convert the "dark data" of unstructured text into queryable, structured relational databases, unlocking the potential for advanced analytics on text-heavy domains.
Reliability: By decoupling generation from execution and using symbolic translation for SQL, SQUiD eliminates the "hallucination" and syntax error problems common in direct LLM-to-SQL approaches.
Scalability: The framework is model-agnostic and effective even with smaller, open-source models, making it accessible for resource-constrained environments.
Future Work: The authors note limitations in handling highly complex schemas without a central entity for joining and suggest future work on few-shot learning and fine-tuning.

In summary, SQUiD represents a significant advancement in data management by demonstrating that a structured, neurosymbolic approach can reliably automate the creation of relational databases from raw text, outperforming end-to-end LLM approaches in both accuracy and syntactic validity.