💬 NLP

Text-to-SPARQL Generation with Reinforcement Learning: A GRPO-based Approach on DBLP

This paper demonstrates that applying Group-Relative Policy Optimization (GRPO) to a small instruction-tuned language model enables effective zero-shot Text-to-SPARQL generation on the DBLP-QuAD dataset by leveraging execution-based rewards, achieving substantial improvements over zero-shot baselines and competitive generalization despite being outperformed by supervised DoRA fine-tuning.

Original authors: Jann Pfeifer, Debayan Banerjee, Ricardo Usbeck

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jann Pfeifer, Debayan Banerjee, Ricardo Usbeck

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, incredibly organized library of academic papers (the DBLP Knowledge Graph). This library is so complex that asking a librarian a simple question like "Who wrote papers about AI in 2020?" requires you to speak a very difficult, computer-only language called SPARQL. If you get the grammar wrong, the librarian ignores you.

Most computer programs that try to translate your English question into this computer language are either:

Giant Brains: They are huge, expensive, and need to be taught with thousands of perfect examples (Gold Queries) to work well.
Clueless Guessers: Small, cheap models that just guess without any training, getting it wrong most of the time.

This paper asks a simple question: Can we teach a small, cheap "brain" to translate these questions correctly just by letting it try, fail, and learn from the results, without showing it the perfect answers?

Here is how they did it, using a method called GRPO (Group-Relative Policy Optimization).

The Analogy: The "Group Quiz" Approach

Think of the AI model as a student taking a test.

The Old Way (Supervised Learning/DoRA):
The teacher gives the student the question and the perfect answer key. The student memorizes the key. This works great, but you need a teacher with the answer key for every single question.

The New Way (GRPO with Reinforcement Learning):
The teacher doesn't have the answer key. Instead, the teacher asks the student to write four different answers to the same question at the same time.

The teacher takes all four answers and runs them through the library's computer system.
If an answer crashes the system or returns the wrong list of books, it gets a "bad score."
If an answer works and finds the right books, it gets a "good score."
The teacher then tells the student: "Look, Answer #3 was the best of your four tries. Next time, try to write more like Answer #3."

The student learns by comparing their own guesses against each other and the real-world results, rather than by copying a teacher's notes.

The "Hints" (Symbolic Hints)

To make the task fair, the researchers didn't just give the AI the question. They gave it a cheat sheet.

The Question: "Who wrote about AI?"
The Cheat Sheet: "By the way, 'AI' refers to this specific computer code (URI), and 'wrote' refers to this specific relationship (URI)."

This removes the hard part of guessing what the words mean and focuses the AI purely on how to build the sentence structure.

What They Found

They tested this "Group Quiz" method on a small AI model (Qwen3-1.7B) and compared it to two other students:

The Untrained Student: Just guessed randomly. (Terrible results).
The Memorizer (DoRA): Studied the answer keys. (Best results).
The "Group Quiz" Student (GRPO): Learned by trial and error.

The Results:

The Untrained Student failed almost everything.
The Memorizer was the champion, getting about 69% of the answers perfectly correct.
The "Group Quiz" Student was the surprise hero. Without ever seeing the answer keys, it learned to get 47% of the answers correct.

Key Takeaways:

It Works: You can teach a small AI to speak "computer library" just by letting it practice and checking if it gets the right books, even without a teacher holding the answer key.
The "Cheat Sheet" Matters: The biggest boost came from a reward that checked if the AI actually used the specific "cheat sheet" hints (the correct entities and relations) provided in the prompt.
The "Answer Key" Trap: Interestingly, when they did give the "Group Quiz" student a hint about what the perfect answer looked like (Gold Query Shaping), it actually got slightly worse at finding the right books. It seems the AI got too focused on copying the shape of the perfect answer rather than finding the right answer.
Generalization: The "Group Quiz" student was actually better at handling weird, new types of questions it hadn't seen before compared to the "Memorizer," who seemed to have just memorized the specific patterns it studied.

The Bottom Line

This paper proves that for small, affordable AI models, learning from the outcome (did it find the right books?) is a powerful strategy when you don't have perfect answer keys. It's not quite as good as having a teacher with the answer key, but it's a massive improvement over guessing, and it might be the best way to train small models for complex tasks in the future.

Limitations: The study assumed the AI already knew exactly which "book" or "author" the user was talking about (perfect linking). In the real world, figuring out what the user means is often the hardest part, which this study didn't tackle. Also, the "Group Quiz" method was slow because the computer had to run the queries repeatedly to grade them.

Technical Summary: Text-to-SPARQL Generation with Reinforcement Learning

Problem Statement

Knowledge Graph Question Answering (KGQA) aims to translate natural language questions into executable SPARQL queries. While existing approaches often rely on large language models (LLMs) or full supervision via gold query annotations, there is a need to determine if smaller, instruction-tuned models can achieve competitive performance in the scholarly domain using outcome-based reinforcement learning (RL). Specifically, this work investigates whether a small open-weight model can be trained via GRPO (Group-Relative Policy Optimization) to perform zero-shot Text-to-SPARQL generation on the DBLP-QuAD benchmark, and how this compares to supervised fine-tuning when gold queries are unavailable for token-level supervision.

Methodology

Task Formulation and Input

The study treats Text-to-SPARQL as a single-step decision problem on the DBLP-QuAD dataset (10,000 question–query pairs). A central assumption is perfect entity and relation linking: prompts include pre-resolved entity and relation URIs alongside schema-enriched metadata (labels, domains, ranges). This isolates the generation task from the disambiguation and linking challenges typical of end-to-end KGQA.

The input is a two-part conversational prompt:

System Message: Enforces a strict output protocol requiring syntactically correct SPARQL, full URIs, and specific query types (SELECT DISTINCT or ASK). It mandates step-by-step reasoning within <thought> tags followed by the final query.
User Message: Contains the natural language question, linked entity URIs with labels, and linked relation URIs with schema context.

Training Approach: GRPO

The authors apply Group-Relative Policy Optimization (GRPO) to the Qwen3-1.7B model. Unlike PPO, GRPO estimates advantages from groups of sampled trajectories without an explicit value function, reducing memory overhead.

Sampling: For each prompt, the policy samples a group of $G=4$ candidate queries.
Objective: The policy is updated to increase the likelihood of higher-reward queries within the group while staying close to a fixed reference policy via KL regularization.

Reward Design

The reward function is a heuristically weighted linear combination of six components, prioritizing execution feedback:
$R = 3R_{exec} + 2R_{sim} + 1R_{struct} + 0.5R_{format} + 1R_{len} + 1R_{len-ratio}$

Execution Feedback ( $R_{exec}$ ): The primary signal. It executes the generated query against a fixed DBLP endpoint and computes the F1 score against the gold answer set. Execution failures incur a fixed penalty.
Structural Coverage ( $R_{struct}$ ): Rewards the inclusion of all required entity and relation URIs provided in the prompt.
Format ( $R_{format}$ ): Ensures the output adheres to the protocol (e.g., presence of closing tags).
Length ( $R_{len}$ ): Penalizes outputs approaching the maximum generation length to reduce truncation.
Gold-Query Shaping (Optional): Two variants were tested:
- Similarity ( $R_{sim}$ ): Sentence-level BLEU between the generated and gold query.
- Length Ratio ( $R_{len-ratio}$ ): Regularizes query complexity based on token length differences.
- Note: The "without gold-query shaping" variant reflects a weakly supervised scenario where only questions and gold answers are available, not the gold queries themselves.

Baselines

The GRPO variants are compared against:

Zero-shot Base: The unmodified Qwen3-1.7B with and without Chain-of-Thought (CoT) prompting.
Supervised Fine-tuning (DoRA): A parameter-efficient fine-tuned model trained on gold SPARQL targets using DoRA adapters and cross-entropy loss.

Key Results

Overall Performance

Zero-shot Baseline: Performed poorly (EMAcc 0.07, F1 0.14). CoT prompting slightly improved accuracy (0.11) but drastically reduced executability (ExAcc 0.23) due to protocol violations.
Supervised DoRA: Achieved the highest overall performance (EMAcc 0.69, ExAcc 0.91, F1 0.78), confirming the strength of token-level supervision.
GRPO (No Gold Queries): Substantially improved over zero-shot (EMAcc 0.47, ExAcc 0.83, F1 0.48) but remained below the supervised baseline.
GRPO (With Gold Queries): Slightly improved executability (0.85) but reduced answer-level accuracy (0.44) and F1 (0.45) compared to the no-gold variant.

Generalization and Categories

Held-out Templates: GRPO demonstrated superior generalization to unseen query templates (GenAcc 0.53) compared to the DoRA baseline (GenAcc 0.40). The supervised model showed a larger performance drop on held-out data, suggesting a reliance on specific training templates.
Question Categories: Both GRPO and DoRA performed well on local patterns (Single Fact, Disambiguation). However, both struggled with complex compositional patterns, particularly Union (Disjunction), Superlative/Comparative, and Multi-intent questions.
Temporal Questions: GRPO improved significantly over zero-shot (TempAcc 0.52 vs 0.09) but did not close the gap to DoRA (0.69).

Reward Ablations

Execution Feedback: Accounted for the majority of performance gains.
Structural Coverage: Provided the most significant additional improvement in answer correctness and subset performance.
Gold-Query Shaping: Adding similarity and length-ratio rewards increased executability slightly but degraded answer-level accuracy and generalization, suggesting a conflict between reproducing surface forms and achieving semantic correctness.

Significance and Claims

The paper claims that outcome-based reinforcement learning is a viable training strategy for small models when gold queries are unavailable for token-level supervision.

Viability of Small Models: The study demonstrates that a 1.7B parameter model, when paired with GRPO and execution-based rewards, can close a considerable portion of the gap between zero-shot prompting and supervised fine-tuning.
Generalization vs. Supervision: While supervised fine-tuning (DoRA) yields higher aggregate accuracy, GRPO exhibits better generalization to held-out templates. This suggests that outcome-based optimization rewards semantic correctness independently of the exact surface form of the query, whereas token-level supervision may overfit to specific template structures.
Limitations of Shaping: The inclusion of gold-query-based shaping terms did not improve answer-level performance and may introduce competing objectives. The authors conclude that execution feedback and structural constraints are the dominant drivers of success in this setting.
Error Analysis: Failures in GRPO tend to be structural (syntax, schema violations, UNION handling), whereas DoRA failures are often subtle semantic mismatches within well-formed queries.

The authors maintain a modest stance, noting that the study assumes perfect linking (removing a major source of error in real-world KGQA) and is limited to a single benchmark and model family. They conclude that while GRPO is a strong alternative for weakly supervised scenarios, it does not yet substitute for supervised fine-tuning when gold targets are available.