💻 computer science

ReqGenX: An Empirical Study of Atomic Decomposition, Artifact Regeneration, and Reconstruction for Legacy SRS Documents

This paper introduces ReqGenX, an empirical study demonstrating that decomposing legacy Software Requirements Specification (SRS) documents into traceable, atomic statements and regenerating them as synthetic pre-SRS artifacts enables fine-grained evaluation of LLM-based SRS generation while revealing critical trade-offs between faithfulness, information retention, and artifact completeness.

Original authors: Ragib Shahariar Ayon, Rayed Fahmi, Sumon Biswas, Shibbir Ahmed

Published 2026-07-21

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Ragib Shahariar Ayon, Rayed Fahmi, Sumon Biswas, Shibbir Ahmed

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant, hyper-fast robot how to write a rulebook for a new video game. You hand the robot a messy, handwritten notebook from the 1990s filled with vague ideas, scribbled notes, and half-finished thoughts. Your goal is for the robot to turn that messy notebook into a perfect, professional rulebook. But here's the catch: the robot is so eager to be helpful that it might accidentally invent new rules, forget old ones, or mix up two different ideas into one confusing sentence. This is the world of Software Requirements Specification (SRS) generation. In the tech world, an SRS is the "master blueprint" that tells engineers exactly what a piece of software should do. While Large Language Models (LLMs)—the same kind of AI that writes poems and answers questions—are getting better at writing these blueprints, it's incredibly hard to check if they are actually telling the truth or just making things up. We need a way to trace every single sentence the robot writes back to the original, messy notebook to see if it's faithful or if it's hallucinating.

This is exactly what the researchers behind ReqGenX set out to solve. They didn't just ask the AI to write a new rulebook; they built a clever, step-by-step "factory" to test how well AI can break down old, messy documents into tiny, undeniable facts, turn those facts into specific types of notes (like "User Stories" or "Quality Checks"), and then see if those notes can be used to rebuild the original idea without losing anything. Think of it like taking a complex Lego castle, carefully snapping every single brick apart to label them, sorting them into specific boxes, and then trying to rebuild the castle using only those sorted boxes. The researchers wanted to know: Can the AI do this without losing a single brick or inventing a new color that didn't exist?

The Factory: ReqGenX

The team created a pipeline called ReqGenX to act as a controlled laboratory for this experiment. Instead of letting the AI just "write" a new document, they forced it to go through three strict stages, acting like a quality-control inspector at every step.

Stage 1: The Atomic Breakdown
First, the AI takes a chunk of text from an old document and tries to break it down into "atoms." Imagine a sentence like, "The product uses the Qt library and must run on all supported operating systems." To a human, that's one sentence. To the AI, this is actually two different ideas mashed together. The goal is to split them into "atomic statements"—tiny, self-contained truths that can't be broken down further. One atom becomes "The product uses the Qt library," and the other becomes "The product must run on all supported operating systems." The researchers found that the AI is actually quite good at this. When they checked if these tiny atoms were faithful to the original text, the scores were incredibly high, usually between 0.96 and 0.99 on a scale where 1.0 is perfect. However, they also noticed that sometimes the AI gets a little too eager and drops a piece of information, or conversely, it might keep a piece that is too vague to be useful.

Stage 2: The Sorting Hat
Once the text is broken into atoms, the AI has to decide what kind of note each atom belongs to. This is like a sorting hat in a wizard school. Does this fact belong in a "User Story" (a note about what a person wants to do)? Does it belong in a "Quality Note" (a rule about how fast or safe the system must be)? Or is it just a "System Context" note (background info)? The researchers used a team of different AI models to vote on these categories. They found that while the AI is great at sorting general ideas, it sometimes struggles with very specific technical labels. For example, when trying to decide if a sentence was a "functional requirement" or a "quality requirement," the AI models disagreed with each other quite a bit, with agreement scores dropping as low as 0.31 in some cases. This suggests that while the AI can see the forest, it sometimes gets confused about the specific types of trees.

Stage 3: The Reconstruction
Finally, the AI takes these sorted, labeled atoms and tries to write them into a new, professional-looking document. This is the moment of truth: can the AI turn these tiny, sorted facts back into a coherent story? The researchers tested this by taking the AI-generated notes and asking a different AI to rebuild the original document from them. They found that the AI could indeed rebuild the document, but it wasn't a perfect copy. The "semantic similarity" (how much the meaning matched) was decent, ranging from 0.69 to 0.75, but the "factual support" (did the new text actually prove the old facts?) was a bit more variable.

The Verdict: Faithful but Flawed

So, what did the researchers actually discover?

First, they confirmed that AI can be a very good librarian. It can take a messy, old document and break it down into tiny, truthful facts with a very high success rate. Most of the time, the atoms it creates are 96% to 99% faithful to the source. This means the AI isn't just making things up out of thin air; it's usually sticking to the script.

However, they also found a few important "buts."

The "Missing Brick" Problem: Even though the AI is faithful, it isn't perfect at keeping everything. Depending on the document and the AI model used, the amount of original information successfully preserved (coverage) varied widely, ranging from 35.6% to 93.5%. This doesn't mean the AI lied; it means that for some documents, the AI decided a large portion of the original text was too vague, redundant, or structural to be kept as a standalone "atom," while for others, it preserved almost everything.
The "Perfect Template" Trap: When the AI tried to turn these atoms into specific document types (like a "Quality Note"), it sometimes struggled to fill in all the blanks without adding fake details. While the AI passed a basic "judge" test nearly 100% of the time, a stricter, more critical test (called Prometheus) showed that only 54.8% to 97.1% of the artifacts were truly complete and high-quality. This suggests that the AI can make things look right, but sometimes they are missing the deep details needed to be truly useful.
The Reconstruction Gap: When they tried to rebuild the original document from the AI's notes, the new version was a "traceable draft," not a perfect replica. It captured the main ideas, but some of the nuance was lost in the translation.

What This Means for the Future

The researchers aren't saying that AI has solved the problem of writing software rulebooks. Instead, they are saying that AI is a powerful tool for checking and organizing information, but it needs a human guide.

They found that the biggest bottleneck isn't the AI's ability to write; it's the granularity of the evidence. If the original document is vague or mixes too many ideas together, the AI struggles to turn it into a clean, usable format. The AI can't magically invent the missing context; it can only work with what it's given.

In the end, this study suggests that we can use AI to create "traceable synthetic artifacts"—digital notes that link back to the original source so we can see exactly where every idea came from. This is a huge step forward for checking if AI is telling the truth. But it also serves as a reminder: just because the AI produces a clean, professional-looking document doesn't mean it has captured every single detail of the original. The AI is a fantastic assistant, but it still needs a human to hold the map and make sure no bricks are left behind.

Technical Summary: ReqGenX – An Empirical Study of Atomic Decomposition, Artifact Regeneration, and Reconstruction for Legacy SRS Documents

1. Problem Statement

Evaluating Large Language Model (LLM)-based Software Requirements Specification (SRS) generation is challenging due to a lack of datasets providing fine-grained traceability between source requirements, intermediate elicitation artifacts, and generated specifications. Existing resources often fail to jointly link heterogeneous SRS evidence, intermediate requirements engineering (RE) artifacts, and traceability relations. Consequently, evaluating generated SRSs often relies on document-level fluency, which may mask the omission of source information, the merging of distinct concerns, or the introduction of unsupported requirements.

The paper addresses the need for a mechanism to transform legacy SRS documents into traceable synthetic pre-SRS artifacts. The goal is to create a substrate that supports fine-grained evaluation of LLM-based SRS compilation, specifically examining whether legacy content can be decomposed into atomic, source-grounded statements and routed to standard artifact types without losing critical information or introducing hallucinations.

2. Methodology: The ReqGenX Pipeline

The authors introduce ReqGenX, a controlled, reproducible pipeline designed to operationalize the transformation of legacy SRSs into synthetic artifacts. The pipeline operates in three primary stages:

Stage 1: Decomposition of Requirements

Normalization: Legacy SRS documents (typically PDFs) are converted to text (Markdown), with non-requirements material (cover pages, headers, footers) removed and structural errors repaired.
Chunking: Documents are segmented into section-level chunks based on restored hierarchy.
Atomization: An LLM-based "atomizer" decomposes chunks into atomic statements. An atom is defined as a minimal, source-supported proposition that is faithful to the source, atomic (expressing one coherent idea), and useful for downstream artifact generation.
Judging: A separate judge LLM evaluates atoms against a rubric for faithfulness, atomicity, and utility, proposing actions (keep, rewrite, split, merge, remove, add) to refine the set.

Stage 2: Semantic Routing

Tagging: Atoms are assigned semantic tags based on an IEEE 29148-inspired schema (e.g., Purpose, Scope, Functional, Quality Attributes).
Multi-LLM Voting: To ensure reliability, three independent LLM agents propose tags for each atom. The final tag is selected via plurality voting. Unresolved cases undergo Human-in-the-Loop (HITL) review.
Granular Classification: Atoms tagged as "Requirements" undergo a secondary classification (Functional vs. Quality) and, if Quality, a sub-type classification (e.g., Portability, Security) to guide artifact generation.

Stage 3: Artifact Generation and Review

Generation: Validated atoms are converted into typed pre-SRS artifacts (e.g., User Research Notes, System Context Notes, Quality Attribute Constraints) using specific templates. The atom text serves as the primary source of truth; no unsupported enrichment is allowed.
Iterative Refinement: A judge LLM evaluates generated artifacts on five dimensions: faithfulness, support, contextuality, structural alignment, and sparsity. Artifacts failing the threshold are revised iteratively.
Traceability: The pipeline maintains provenance metadata, linking every generated artifact back to its source atom, chunk, and original SRS section.

3. Experimental Design

Dataset: Seven SRS documents from the PURE dataset (Get Real, Library, Puget Sound, QHeadache, VUB, Libra, Home), totaling 31,454 words and 1,408 sentences.
Models: Experiments utilized a mix of closed-source (Claude Sonnet 4.6, GPT-4o) and open-source models (GPT-OSS, Gemma 3, Phi-4, Mistral, Llama 3, Qwen3).
Evaluation Metrics:
- AlignScore: Measures semantic support between source and generated content.
- Judge/Prometheus Scores: Rubric-based quality assessment (1–5 scale) for faithfulness, atomicity, and structural alignment.
- Embedding Similarity (SBERT): Used to measure information retention and downstream recoverability.
- Coverage Analysis: Measured the percentage of source information units retained through the pipeline.

4. Key Results

RQ1: Reliability of LLM-Based RE Decisions

Semantic Routing: LLMs can support semantic routing, but reliability varies. Claude Sonnet 4.6 achieved the highest agreement with human labels ( $\kappa = 0.83$ for tags, $\kappa = 0.80$ for requirements). Fine-grained requirement classification showed lower agreement between models (e.g., GPT-4o vs. GPT-OSS $\kappa = 0.31$ ).
Quality Judgment: Gemma 3 was identified as the best judge model, offering a strong tradeoff between F1 score and low false-positive rates for both atoms and artifacts.

RQ2: Quality of Generated Atoms

Faithfulness: Generated atoms are generally faithful to source chunks, with median AlignScore values between 0.96 and 0.99. Unsupported rates were typically below 3%.
Coverage vs. Faithfulness: While atoms are faithful, source coverage varies significantly (35.6% to 93.5%). Some information is lost not due to hallucination but because the decomposition process discards implicit, redundant, or weakly actionable content to ensure atomicity.
Quality: Post-revision atoms achieved high pass rates (76%–100% for atomicity; 70%–97% for usability) with average Prometheus scores ranging from 4.34 to 4.85.

RQ3: Quality of Generated Artifacts

Grounding: Generated artifacts remain strongly grounded in source atoms, with mean AlignScore values typically between 0.80 and 0.94.
Refinement: Judge pass rates approached 100% after iterative refinement. However, stricter Prometheus evaluation revealed wider variation (54.8% to 97.1% pass rates), indicating that grounding does not guarantee structural completeness or sparsity.

RQ4: Downstream SRS Reconstruction

Recoverability: In a case study reconstructing SRSs from generated artifacts, source atoms were semantically recoverable (SBERT means: 0.69–0.75).
Factual Support: AlignScore medians for reconstruction ranged from 0.76 to 0.84. However, lower means and wider interquartile ranges suggest that factual support is more variable than semantic similarity, often due to abstraction or rephrasing during compilation.

RQ5: Information Loss and Failure Patterns

Granularity Issues: Information loss is primarily driven by evidence granularity rather than generation failures.
- SRS $\to$ Atoms: Loss occurs when statements are implicit, bundled, or context-dependent.
- Atoms $\to$ Artifacts: Loss occurs when valid atoms are too narrow to fill constrained artifact templates without adding unsupported content.
Revision Patterns: The most frequent revision actions were "Rewrite" and "Remove" (addressing bundling or weak wording), while "Split" was common for dense sections. "Merge" and "Add" were rare.

5. Key Contributions

Empirical Characterization: The study characterizes LLM-based atomization of legacy SRSs, demonstrating that high atom-level faithfulness can coexist with significant variation in source coverage.
Stage-wise Analysis: It provides a granular analysis of information loss across the SRS-to-Atoms-to-Artifacts transformation, distinguishing between evidence-granularity issues and hallucination.
Routing Reliability: It shows that LLM semantic routing (specifically for high-level tags) is more reliable than fine-grained requirement classification, with Claude achieving high human agreement.
Judge Calibration: It presents a human-calibrated procedure for selecting LLM judges for RE artifact evaluation, based on 291 annotated triples.
Replication Package: The authors release a traceable replication package containing processed SRSs, generated atoms/artifacts, prompts, codebooks, and traceability metadata.

6. Significance and Claims

The paper claims that traceable synthetic pre-SRS artifacts can support more fine-grained evaluation of LLM-based SRS generation. By decomposing legacy documents into atomic, provenance-linked units, the study exposes critical tradeoffs among:

Faithfulness: How well the output matches the source.
Information Retention: How much source content is preserved.
Artifact Completeness: Whether the output satisfies structural requirements without hallucination.

The authors conclude that while LLMs can produce faithful and usable atoms and grounded artifacts, these transformations are not lossless. The study highlights that the primary bottleneck is evidence granularity: legacy SRSs often contain implicit or bundled information that resists clean decomposition into standard artifact templates. The authors modestly position their work not as a production-ready SRS generation tool, but as an experimental instrument to observe and evaluate how information changes as it flows through decomposition, routing, and reconstruction stages. They suggest that future systems must account for these tradeoffs, potentially by allowing partially specified artifacts or grouping related atoms rather than forcing every source statement into a rigid template.