💬 NLP

QFS-Composer: Query-focused summarization pipeline for less resourced languages

This paper introduces QFS-Composer, a novel framework that integrates query decomposition, question generation, and question answering to enhance the factual alignment and relevance of query-focused summaries in less-resourced languages, demonstrated through improved performance on Slovenian text.

Original authors: Vuk {\DJ}uranović, Marko Robnik Šikonja

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vuk {\DJ}uranović, Marko Robnik Šikonja

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of books (the internet), but most of them are written in languages that big, famous AI robots don't speak very well. These "less-resourced" languages, like Slovenian, often get ignored because the AI hasn't read enough books in them to learn the nuances.

This paper introduces a new tool called QFS-Composer to help these AI robots understand and summarize text in these smaller languages, specifically when a human asks a specific question about the text.

Here is the breakdown of how it works, using some everyday analogies:

The Problem: The "Overwhelmed Librarian"

Imagine you ask a librarian (the AI) to summarize a 500-page book about a specific topic, like "How did the weather affect the local fishing industry?"

The Old Way: You just ask the librarian, "Summarize the fishing part." The librarian might skim the book, get confused, make things up (hallucinate), or miss the most important details because they are trying to guess what you want without a clear map.
The Language Barrier: If the book is in a language the librarian doesn't know well, they might just guess or give a very generic answer.

The Solution: QFS-Composer (The "Detective Team")

Instead of asking the librarian to do the whole job alone, the authors created a team of specialists who work together to get the perfect answer. Think of it as a detective squad solving a case.

Here are the four steps of their process:

1. The Interrogator (Query Decomposition)

The user's question is often vague, like "Tell me about the fishing."

What happens: The system breaks this big, messy question into smaller, specific clues.
Analogy: Instead of asking "Who stole the cookie?", the Interrogator breaks it down: "Was the cookie eaten by the cat?" "Was the cat in the kitchen?" "Did the cat have crumbs?"
How they did it: Since they didn't have a pre-trained "Interrogator" for Slovenian, they used two tricks:
- The "Smart Guess" (LLM): Asking a powerful AI to break the question down.
- The "Name Finder" (NER): Looking for specific names (like "Fisherman John" or "Lake Bled") in the question and turning those names into questions.

2. The Evidence Hunter (Question Answering)

Now that they have specific questions, they need to find the answers in the big book.

What happens: The system scans the text to find the exact sentences that answer those small questions.
The Challenge: The book is too long for the AI to read all at once (it's like trying to swallow a whole elephant).
The Fix: They chop the book into small bite-sized chunks. They use a "sniff test" (called BERTScore) to see which chunks smell like the answer. They only keep the top chunks that smell right and ignore the rest. This prevents the AI from getting lost in the middle of a long story.

3. The Storyteller (Abstractive Summarization)

Now the team has the original question, the list of small questions, and the specific answers found in the text.

What happens: They feed all this "evidence" to the main AI (the Storyteller).
The Result: The Storyteller doesn't just guess anymore. It has a cheat sheet. It writes a summary that is factually accurate because it is literally reading the answers it just found. It's like a student taking a test with an open book and a highlighted study guide.

4. The Judge (Evaluation)

How do they know if the summary is good? Usually, you need a human to read it and compare it to a "perfect" summary. But writing perfect summaries is expensive and slow.

The Innovation: They built a "Robot Judge" that doesn't need a human.
How it works: The Robot Judge asks the AI, "If I ask you this question based only on your summary, do you get the same answer as if I asked you based on the whole book?"
The Metaphor: It's like a teacher checking if a student's study notes are accurate. If the student's notes lead to the right answer, the notes are good. If the notes lead to a wrong answer, the notes are bad. They adapted these judging tools to work for the Slovenian language.

The Results: Why It Matters

When they tested this on Slovenian news articles:

Shorter but Smarter: The summaries created by this "Detective Team" were actually shorter than the ones made by the AI working alone.
Less Hallucination: The AI made up fewer fake facts.
Better Relevance: The summaries stuck closer to what the user actually asked.

The Bottom Line

This paper is a blueprint for helping AI speak "smaller" languages better. Instead of hoping the AI magically knows everything, they gave it a step-by-step workflow: Break the question down, find the specific facts, and then write the story.

It's the difference between asking a friend to "tell me about the party" (who might forget the details) and asking them to "check the guest list, find out who brought the cake, and see who left early, then tell me the story" (who will give you a perfect, factual report).

By making these tools open-source and adaptable, the authors hope that speakers of less-resourced languages won't be left behind in the AI revolution.

1. Problem Statement

Large Language Models (LLMs) have demonstrated strong capabilities in text summarization, particularly for high-resource languages like English. However, their performance degrades significantly in less-resourced languages (e.g., Slovenian) due to a lack of:

Labeled training datasets.
Robust evaluation tools (most metrics are English-centric).
Query-Focused Summarization (QFS) specific resources, where the goal is to summarize a text based on a specific user query (e.g., "How does this article portray Company X?").

Current evaluation methods often rely on reference summaries created by humans, which is expensive and time-consuming. Furthermore, standard automatic metrics (like ROUGE) often fail to capture factual consistency or relevance to the user's specific intent in non-English contexts.

2. Methodology: QFS-Composer

The authors propose QFS-Composer, a novel pipeline designed to improve the factual alignment and relevance of summaries for less-resourced languages. The system integrates four key stages: Query Decomposition, Question Generation (QG), Question Answering (QA), and Abstractive Summarization.

A. System Architecture

The pipeline operates as follows:

Query Decomposition: The user's initial query is often broad. The system splits this into smaller, manageable sub-questions. Two approaches were tested:
- LLM-based Decomposition (decomp aug): Using a few-shot prompt with an LLM to generate sub-questions.
- NER-based Decomposition (ner aug): Using a Named Entity Recognition (NER) model to extract entities from the query, which are then fed to a QG model to generate specific questions.
Question Answering (QA): The system uses a fine-tuned Slovenian QA model (based on the GaMS-9B-Instruct LLM) to answer the generated sub-questions using the source document as context.
- Technical Constraint: Due to memory limits, the QA model cannot process long documents at once. The authors implemented a chunking strategy: the document is split into chunks, and chunks are ranked by BERTScore against the question. Only the top $n$ chunks (with a score > 0.85) are processed to find the answer.
Augmented Prompting: The original user query is combined with the extracted Question-Answer (QA) pairs to form an "augmented prompt."
Final Summarization: An LLM generates the final summary based on the source text and the augmented prompt (query + QA pairs). This forces the model to focus on facts relevant to the user's intent.

B. Adaptation for Less-Resourced Languages

To enable training and evaluation in Slovenian, the authors developed several assets:

QA & QG Models: Fine-tuned the GaMS-9B-Instruct model (based on Gemma) on the Slovenian translation of the SQuAD v2 dataset.
Dataset: Translated the MOCHA dataset (a QFS dataset) into Slovenian.
Evaluation Metrics: Adapted reference-free evaluation frameworks QAGS and QuestEval for the Slovenian language. These metrics evaluate summary quality by generating questions from the summary and checking if the answers match those derived from the source text.

3. Key Contributions

QFS-Composer Framework: A modular, QA-guided pipeline that enhances factual consistency in summarization for low-resource languages.
Slovenian NLP Assets:
- Fine-tuned Slovenian QA and QG models.
- Slovenian translation of the MOCHA dataset.
- Adapted implementations of QuestEval, QAGS, and RQUGE for Slovenian.
Novel Evaluation Strategy: Demonstrated the effectiveness of using QA-based metrics (reference-free) to evaluate QFS in languages lacking human-annotated reference summaries.
Empirical Validation: Provided a comprehensive comparison of different decomposition strategies (ner aug vs. decomp aug) across multiple LLMs.

4. Experimental Results

The system was evaluated on 21 Slovenian news articles using four LLMs: GPT-4.1-mini, Gemma 2 9B, Llama 3.1 8B, and GaMS 9B.

Quantitative Findings

Performance Gain: The ner aug (NER-based decomposition) setting consistently outperformed the baseline (no aug) and the LLM-based decomposition (decomp aug) in most cases.
- Best Model: Llama-3.1-8B-Instruct with ner aug achieved the highest scores in QAGS (F1, EM, BERTScore).
- Best QuestEval Score: GaMS-9B-Instruct with ner aug achieved the highest QuestEval score (0.427).
Metric Analysis:
- QAGS: Showed that augmented prompts led to summaries with higher factual consistency (answers derived from the summary matched answers derived from the source text more closely).
- QuestEval: Showed improvements in information coverage, though scores were generally lower due to the metric's reliance on exact matches and the limitations of the QA model in long contexts.
Summary Length: Contrary to expectations, summaries generated with augmented prompts were shorter but more information-dense than those generated from plain queries. This suggests the QA-guided approach filters out irrelevant information effectively.

Qualitative Findings

Conciseness: Augmented prompts produced plain text summaries rather than unordered lists, which were more coherent.
Hallucinations: While improved, hallucinations still occurred (e.g., misinterpreting a fisherman's stance on law changes).
Limitations: The QA model sometimes failed to find answers in long documents because the chunking/ranking mechanism (based on BERTScore) might prioritize a chunk that doesn't actually contain the answer, leading to "unanswerable" tokens or incorrect answers.

5. Significance and Future Work

Inclusivity: This work bridges the gap between resource-rich and resource-poor languages, proving that high-quality QFS is achievable without massive labeled datasets by leveraging modular QA/QG pipelines.
Reproducibility: All code, models, and datasets are open-sourced, encouraging further research in Slovenian and other low-resource NLP.
Future Directions:
- Improving the precision of the Query Decomposer and QA models to prevent error propagation.
- Overcoming hardware constraints to process longer contexts without aggressive chunking.
- Conducting human evaluation studies to better quantify user satisfaction and alignment.

In conclusion, QFS-Composer establishes a robust, extensible methodology for advancing query-focused summarization in less-resourced languages, demonstrating that integrating question-answering mechanisms significantly improves the factual reliability and relevance of LLM-generated summaries.