💻 computer science

STELLAR-E: a Synthetic, Tailored, End-to-end LLM Application Rigorous Evaluator

STELLAR-E is a fully automated, two-stage framework that generates high-quality, customizable synthetic datasets for rigorous LLM application evaluation, offering a scalable and efficient alternative to manual data collection while achieving comparable assessment quality to existing benchmarks.

Original authors: Alessio Sordo, Lingxiao Du, Meeka-Hanna Lenisa, Evgeny Bogdanov, Maxim Romanovsky

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alessio Sordo, Lingxiao Du, Meeka-Hanna Lenisa, Evgeny Bogdanov, Maxim Romanovsky

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a head chef trying to train a new, very smart kitchen robot (a Large Language Model, or LLM) to cook specific dishes. To teach it, you need a recipe book (a dataset) with thousands of questions and answers.

The Problem:
Usually, getting these recipes is a nightmare. You have to hire human experts to write them, which is slow, expensive, and often impossible if the recipes involve secret family ingredients (private data) or strict health regulations. Also, most recipe books are only written in English, leaving out chefs who speak Italian, Swahili, or other languages.

The Solution: STELLAR-E
The authors of this paper built a machine called STELLAR-E. Think of it as an automated "Recipe Factory" that can instantly print out high-quality, custom recipe books in any language, without needing any human writers or existing secret recipes.

Here is how the factory works, broken down into simple steps:

1. The Blueprint (Topic Generation)

Instead of guessing what to ask, the factory first asks a smart AI to brainstorm a list of "topics" (like "Italian pasta" or "German banking rules"). It then acts like a strict editor, throwing away any topics that are too vague or unrelated.

2. Writing the Questions (Instruction Generation)

Once it has good topics, the factory generates the actual questions. But it doesn't just write them once and hope for the best. It uses a "Feedback Loop":

The AI writes a question.
A "Judge AI" grades it.
If the grade is too low, the AI has to rewrite it.
This happens over and over until the question is perfect.
Analogy: It's like a student rewriting an essay until the teacher gives it an A+, rather than just submitting the first draft.

3. Making it Harder (Difficulty Enhancement)

Sometimes, AI-generated questions are too easy, like asking "What color is the sky?" The factory has a special module that paraphrases the questions to make them trickier, ensuring the robot chef actually has to think hard to answer them.

4. Checking for Variety (Diversity Enhancement)

If the factory accidentally prints 100 questions that all mean the same thing, it's a waste of time. The system uses a "semantic scanner" (a tool that understands the meaning of words) to check the distance between questions. If two questions are too similar, it deletes one. This ensures the final book has a wide variety of unique challenges.

5. The Final Exam (Evaluation)

The factory doesn't just stop at making the book; it also tests the robot chef. They used this system to create test books in English and Italian and compared them to real, human-written test books.

What Did They Find?

The "Good Enough" Standard: The synthetic (AI-made) test books were very close in quality to the real human-written ones. In fact, the AI-made books were only about 5.7% "easier" than the real ones.
The Trap for Small Robots: The study found that while big, powerful AI models handled the synthetic tests well, smaller, weaker AI models found the real human tests much harder than the AI-made ones. It seems the AI-made tests accidentally contained "cheat codes" or patterns that the smaller robots could exploit to get high scores, whereas the real human tests were more genuinely difficult.
Language Matters: The system worked well for both English and Italian, proving you don't need to translate English tests to get good results in other languages; you can generate them natively.

The Bottom Line
STELLAR-E is a tool that lets companies create their own custom, private, and multilingual test suites instantly. It solves the problem of "we don't have enough data to test our AI." While the tests aren't perfectly identical to human-made ones (especially for smaller AI models), they are good enough to be a fast, cheap, and reliable alternative to hiring armies of human editors.

1. Problem Statement

The rapid adoption of Large Language Models (LLMs) across regulated sectors (e.g., finance, healthcare) has created a critical bottleneck in evaluation and benchmarking. Current challenges include:

Data Scarcity & Privacy: Collecting high-quality, domain-specific, and language-specific datasets is hindered by privacy regulations, data sensitivity, and the high cost/time of manual annotation.
Limitations of Existing Automation: Existing automated benchmarks often rely on augmenting or anonymizing existing data (unsuitable for high-security domains), focus on narrow domains, or lack multilingual support.
Multilingual Gaps: Non-English benchmarks are scarce. Relying on machine translation introduces "translationese," cultural artifacts, and loss of nuance, widening the disparity between high-resource and low-resource languages.
Scalability: Current methods struggle to generate large-scale, diverse datasets without human intervention, making continuous monitoring in LLMOps (CI/CD pipelines) difficult.

2. Methodology: The STELLAR-E Pipeline

STELLAR-E is a fully automated, end-to-end system designed to generate high-quality, customizable synthetic instruction-answer (I&A) datasets and evaluate LLMs against them. The system is built on a two-stage architecture:

Stage 1: Synthetic Data Generation Engine

The system modifies the TGRT Self-Instruct framework to create a controllable generation pipeline. It operates through the following modular stages:

Question Type (QT) Definition: Researchers define specific domains or adversarial question types (e.g., reasoning, safety) to guide the generation.
Topic Generation & Filtering: An LLM generates candidate topics (noun phrases) for each QT. These are filtered using a custom G-Eval framework to ensure semantic correctness and relevance.
Instruction Generation & Quality Improvement:
- Feedback Loop: Instructions are generated iteratively. Low-quality instances are not discarded immediately but are fed back to an LLM with specific feedback to regenerate them.
- Quality Criteria: Instructions are scored on Diversity, Relevance, Conciseness, and Correctness.
Difficulty Enhancement (DFE): To prevent the generation of trivial tasks, a module paraphrases instructions to increase complexity, ensuring they challenge the target LLM (inspired by adversarial generation).
Answer Generation & Quality Improvement: Answers are generated separately from instructions (sample-wise decomposition) to reduce hallucinations. They are evaluated on Correctness, Relevance, Conciseness, Completeness, and Safety.
Diversity Enhancement (DVE): A multilingual embedding model (bge-m3) computes embeddings for all I&A pairs. A pairwise distance algorithm filters out redundant pairs, ensuring the final dataset is semantically and linguistically diverse.

Stage 2: Evaluation Pipeline

The system includes a built-in evaluation component to validate the synthetic data's fidelity against human-curated benchmarks.

Metric: It uses a custom G-Eval framework (LLM-as-a-judge) with Chain-of-Thought reasoning.
Models: It employs Gemini 2.0 Flash for generation/filtering and Gemini 2.5 Pro for evaluation to ensure robustness and minimize bias.
Thresholds: A score threshold ( $\tau = 8$ on a 1–10 scale) is used to accept high-quality data.

3. Key Contributions

Fully Automated, Customizable Pipeline: A system that generates synthetic datasets of custom size, language, and domain without relying on pre-existing real-world data.
Toggleable Quality Optimizations:
- Diversity Enhancement (DVE): Uses embedding distances to eliminate redundancy.
- Difficulty Enhancement (DFE): Increases prompt complexity to better test model capabilities.
Robust Supervision Mechanisms: Implementation of a feedback loop for re-generation and a multi-stage filtering process (Topic $\to$ Instruction $\to$ Answer) to ensure high fidelity.
Multilingual Support: The system natively supports generating and evaluating data in multiple languages (demonstrated in English and Italian), avoiding the pitfalls of simple translation.

4. Experimental Results

The authors evaluated STELLAR-E using the Mintaka dataset (a QA benchmark) in English and Italian, comparing synthetic data against real human-curated data and machine-translated data. They tested two models: a Strong Model (Gemini 2.5 Flash) and a Weak Model (Llama 2 13B).

Key Findings:

Fidelity to Real Data: The synthetic datasets with both DVE and DFE enabled achieved the closest performance match to real-world data.
- Strong Model: The gap in G-Eval scores between synthetic and real data was reduced to +5.7%.
- Weak Model: The gap was +10.9% in English and +0.7% in Italian, indicating the synthetic data is highly effective even for smaller models.
Quality Metrics:
- Synthetic data generally showed higher Accuracy and Completeness scores than real data, suggesting the generated instructions might still be slightly easier or more structured than human queries.
- Relevance scores dropped significantly when DVE/DFE were applied, bringing them closer to real-world baselines, proving these modules effectively test a model's ability to stay on topic.
- ROUGE-L scores were lower for synthetic data, indicating that models generate semantically similar but syntactically different answers compared to "golden" human answers.
Translation vs. Synthesis: In Italian, machine-translated data performed slightly better than real data (suggesting translation removed ambiguity), but the synthetic data remained a robust alternative that avoids cultural artifacts.

5. Significance and Conclusion

STELLAR-E addresses the critical need for scalable, privacy-compliant, and domain-adaptable evaluation for LLMs.

Efficiency: It offers a faster alternative to manual dataset creation, enabling rapid quality assurance cycles in LLMOps.
Fairness: By generating custom datasets, it allows for the fair evaluation of LLM applications in specific domains and languages without relying on sensitive real data.
Future Impact: The system is modular, allowing components to be reused for training, fine-tuning (via RL), or RAG evaluation. The authors note that while synthetic data is a strong substitute, future work should include human evaluation to detect subtle systemic biases introduced by the LLMs themselves.

In summary, STELLAR-E demonstrates that high-quality, rigorous benchmarking for LLMs can be achieved through fully automated, synthetic data generation, bridging the gap between the need for massive evaluation datasets and the constraints of privacy and manual effort.