ReqElicitGym: An Evaluation Environment for Interview Competence in Conversational Requirements Elicitation

1. Problem Statement

With the rapid advancement of Large Language Models (LLMs) in code generation, the primary bottleneck in automated software development has shifted from writing code to eliciting accurate and complete user requirements. While LLMs excel at generating code from clear specifications, they struggle with conversational requirements elicitation—the multi-turn process of interviewing stakeholders to uncover implicit, missing, or ambiguous requirements.

Current evaluation methods for this capability suffer from three critical limitations:

1. Lack of Scenarios: Existing benchmarks cover too few scenarios to support systematic comparison.
2. Reliance on Real Users: Evaluations often depend on human role-playing or student participants, which introduces high variability, high costs, and lack of reproducibility.
3. Subjective Metrics: Assessments often rely on post-hoc human scoring rather than objective, process-aware metrics that track what information is elicited at each turn.

2. Methodology: ReqElicitGym

The authors propose ReqElicitGym, an interactive, automatic, and reproducible evaluation environment designed to assess the "interview competence" of LLMs. The system operates as a pipeline consisting of three core modules:

A. Evaluation Dataset

• Scope: Contains 101 website requirements elicitation scenarios spanning 10 application types (e.g., e-commerce, dashboards, learning platforms).
• Structure: Each scenario is defined by three components:
  1. Initial Req: An underspecified user description (e.g., "I want a stock search website").
  2. Implicit Req (Ground Truth): A set of critical missing details (categorized into Interaction, Content, and Style) that must be uncovered.
  3. Final Req: The complete, structured specification derived after all implicit requirements are elicited.
• Construction: Built from the WebGen-Bench dataset, refined by PhD-level annotators through a rigorous multi-step process involving annotation, gap analysis, and iterative quality control.

B. Oracle User (Simulated Stakeholder)

• Role: Simulates a realistic human stakeholder to interact with the LLM interviewer.
• Mechanism: Powered by an LLM (GPT-5.1) with controlled prompts.
• Design Principles:
  • Groundedness: Only reveals information explicitly annotated as "Implicit Req" in the dataset.
  • Passivity: Does not volunteer extra information unless explicitly asked (preventing the LLM from getting "free" answers).
  • Context Awareness: Maintains dialogue consistency.
• Validation: Validated against 33 real-world interview sessions (555 turns). Achieved a Cohen's $\kappa$ of 0.73 with real users regarding disclosure behavior, confirming high fidelity.

C. Task Evaluator (Automated Judge)

• Role: Analyzes the dialogue turn-by-turn to evaluate the interviewer's performance.
• Functionality:
  • Classifies the interviewer's action into three strategies: Clarify (resolving ambiguity), Probe (uncovering new info), or Finish.
  • Determines if the action successfully elicited a remaining implicit requirement.
• Validation: Validated against human expert annotations with a Cohen's $\kappa$ of 0.72 for relevance judgment.

D. Evaluation Metrics

The environment computes quantitative metrics for each interaction:

1. Implicit Requirement Elicitation Rate (IRE): The proportion of ground-truth implicit requirements successfully uncovered ($|\hat{R}| / |R|$).
2. Turn-discounted Key Question Rate (TKQR): Adapted from nDCG, this measures the efficiency of asking key questions early in the dialogue.
3. Effective Strategy Ratio (ESR): The proportion of "Clarify" or "Probe" turns that successfully elicited at least one requirement.

3. Key Contributions

1. ReqElicitGym Environment: The first automated, reproducible benchmark specifically for evaluating interview competence in conversational requirements elicitation.
2. High-Quality Dataset: A dataset of 101 scenarios explicitly modeling the gap between initial, implicit, and final requirements across diverse web domains.
3. Automated Simulation & Evaluation: Novel designs for an "Oracle User" and "Task Evaluator" that eliminate the need for real human participants while maintaining high agreement with human behavior and expert judgment.
4. Systematic Empirical Study: A comprehensive evaluation of 7 representative LLMs (including GPT-5.2, Claude Opus, DeepSeek, Qwen, etc.) under both Non-CoT and Chain-of-Thought (CoT) settings.

4. Experimental Results & Findings

The study evaluated 7 LLMs across the 101 scenarios. Key findings include:

• Limited Overall Competence: Current LLMs perform poorly in uncovering implicit requirements. Even the best-performing model (DeepSeek V3.2) achieved an IRE of only 0.32, meaning they missed nearly 70% of implicit requirements.
• CoT Impact: Chain-of-Thought (CoT) prompting consistently improved questioning efficiency (TKQR) and reduced the number of dialogue turns, but did not reliably increase coverage (IRE). In some cases, CoT led to lower IRE.
• Strategy Imbalance: LLMs overwhelmingly favor Probing over Clarification.
  • Probing turns vastly outnumber clarification turns.
  • Clarification questions are consistently less effective (low ESR) than probing questions.
  • CoT improves the effectiveness of both strategies but does not change the preference for probing.
• Requirement Type Disparity:
  • Interaction & Content: LLMs show moderate ability to elicit functional and content-related requirements.
  • Style: LLMs consistently fail to elicit Style-related requirements (visual preferences, layout), with IRE for style remaining near 0.01 across almost all models.
• Domain Sensitivity: Performance varies significantly by application type. Structured domains like Dashboards (IRE ~0.27) are easier for LLMs to handle than unstructured domains like E-commerce or Learning Platforms (IRE ~0.05–0.06).

5. Significance and Future Directions

• Significance: The paper highlights a critical gap between LLMs' conversational fluency and their ability to perform systematic, structured reasoning required for requirements engineering. It provides the necessary infrastructure (dataset + environment) to drive progress in this specific sub-field of AI-assisted software development.
• Future Directions:
  1. Ontology-Guided Interviewing: Integrating domain-specific requirement ontologies to guide LLMs in systematically traversing requirement dimensions rather than relying on pattern completion.
  2. Reinforcement Learning (RL): Formulating requirements elicitation as a sequential decision-making problem to train agents on optimal strategy selection (when to probe, clarify, or finish) based on coverage and efficiency rewards.

In conclusion, ReqElicitGym establishes a new standard for evaluating LLMs in requirements engineering, revealing that while current models are capable of generating code, they are not yet competent enough to autonomously define the problems that code should solve.