💻 computer science

From Chat to Interview: Agentic Requirements Elicitation with an Experience Ontology

This paper introduces OntoAgent, an interview agent that leverages an automatically constructed experience ontology to guide large language models in conducting structured, systematic, and efficient requirements elicitation interviews, significantly outperforming existing baselines in both effectiveness and questioning efficiency.

Original authors: Dongming Jin, Zhi Jin, Yaotian Yang, Linyu Li, Zheng Fang, Yuanpeng He, Wenchun Jing, Xiaohong Chen

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dongming Jin, Zhi Jin, Yaotian Yang, Linyu Li, Zheng Fang, Yuanpeng He, Wenchun Jing, Xiaohong Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Chatty" Interviewer vs. The "Experienced" Detective

Imagine you are trying to build a custom house. You hire a contractor to ask you questions about what you need.

The Old Way (Free-Form Chat): You hire a contractor who just chats with you. They might ask, "Do you like blue?" then "Do you like red?" then "Do you like blue?" They forget to ask about the plumbing, the roof, or the garage because they are just "talking." They miss the hidden details you didn't think to mention (like, "I need a soundproof room for my band"). This is like using a standard AI chatbot to gather requirements—it's friendly but often misses the big picture and asks the same questions twice.
The Human Expert Way: A veteran contractor has a mental checklist. They know that before talking about paint colors, they must check the foundation. They know to ask about the kitchen layout before the bathroom. They follow a hidden map of "what needs to be asked" based on years of experience.

The Paper's Goal: The authors wanted to teach an AI to act like that veteran contractor, not just a chatty friend. They built a system called OntoAgent.

The Solution: The "Experience Map" (Ontology)

The core idea of this paper is that you shouldn't just let an AI "wing it." Instead, you give it a structured map based on real-world experience.

The authors call this map an "Experience Ontology." Think of it like a tree or a family tree for requirements:

The Trunk (Aspects): The big categories (e.g., "How the user interacts," "What content is shown," "How it looks").
The Branches (Dimensions): Specific areas under those categories (e.g., under "Interaction," you have "Search" and "Login").
The Leaves (Slots): The tiny, specific details that need to be clarified (e.g., under "Search," you need to know: "Do they want to filter by date?").

How OntoAgent Builds the Map:
Before interviewing anyone, the AI reads thousands of past project descriptions (like reading old blueprints). It uses a Large Language Model (LLM) to automatically organize these past experiences into that "Tree" structure. This becomes its "Experience Map."

How the Interview Works: The 4-Step Dance

When OntoAgent interviews a user, it doesn't just guess the next question. It follows a strict 4-step routine for every turn of the conversation:

Listen (ParseUser): It listens to what the user just said. Did they confirm they need a login? Did they say they don't need a search bar?
Score the Map (ScoreOnto): It looks at its "Experience Map." Based on what the user said, it gives points to different branches. If the user mentioned "stocks," the "Search" branch gets a high score. If they didn't mention "style," the "Style" branch gets a low score.
Re-Rank (ReRankOnto): As the conversation goes on, the map updates. Maybe the user said, "I don't care about the color," so the AI lowers the priority of "Style" questions and raises the priority of "Function" questions. It constantly reshuffles the list of what to ask next.
Prune the Branches (GatePrune): This is the efficiency booster. If the AI asks three questions about "Login" and the user says "No, we don't need login," the AI immediately cuts off that whole branch of the tree. It stops asking about login passwords, email formats, etc., because that path is closed. This saves time.

Finally, it takes the best question from that list and asks it in natural language.

The Results: Why It's Better

The authors tested this system against other AI interviewers (the "chatty" ones) using 101 different scenarios (like building a website for a salon or a ski resort).

Finding the Hidden Stuff (Effectiveness): The "chatty" AIs missed a lot of hidden requirements (like specific rules for how data is sorted). OntoAgent found 33% more of these hidden details. It was like the detective who remembered to check the basement, while the others only looked in the living room.
Asking the Right Questions Faster (Efficiency): The "chatty" AIs wasted time asking obvious questions or asking the same thing twice. OntoAgent asked the most important questions 21% faster. It didn't get stuck in loops.
Working with Different Brains: The system worked well even when they swapped the "brain" (the underlying AI model) for different types of AI. It proved that the "Experience Map" is the secret sauce, not just the specific AI model used.
Human Feedback: When real people (students and experts) acted as the "clients" and talked to the AI, they rated OntoAgent higher. They felt it was more effective, efficient, and adaptable than the other methods.

The Bottom Line

The paper claims that by giving an AI a structured "Experience Map" (an ontology) derived from past work, we can turn a chaotic, free-flowing chat into a systematic, expert-level interview.

It's the difference between asking a friend, "What do you want for dinner?" (who might just say "Pizza" and miss the fact that you're vegetarian) versus asking a professional chef who has a checklist of dietary restrictions, flavor profiles, and cooking methods to ensure nothing is missed.

Key Takeaway: To get the best results from AI in complex tasks, don't just let it chat freely; give it a structured map of human experience to guide its questions.

Technical Summary: OntoAgent for Requirements Elicitation

Problem Statement

Requirements elicitation interviews are a cornerstone of requirements engineering but remain heavily dependent on the experience of human analysts. This reliance incurs significant time and training costs and introduces risks of human bias and communication misunderstandings. While Large Language Models (LLMs) offer a pathway to automate this process, existing "free-form" approaches—where an LLM acts as an interviewer without structural guidance—suffer from critical limitations. Empirical studies indicate that these models struggle to uncover implicit requirements, often generate generic or redundant questions in early dialogue stages, and lack interpretability and controllability. Consequently, they fail to systematically explore the requirement space, leading to incomplete requirement capture.

Methodology: OntoAgent

To address these limitations, the authors propose OntoAgent, an interview agent that integrates LLMs with a structured Experience Ontology. The core design principle is to explicitly model the structured cognitive framework that experienced analysts use, decoupling what to ask (guided by the ontology) from how to ask (generated by the LLM).

The framework operates in two interrelated stages:

1. Experience Ontology Induction

OntoAgent automatically constructs a hierarchical, three-level tree structure from domain-specific requirement texts. This ontology serves as a structured questioning space:

Aspects (Level 1): High-level abstractions of system requirements (e.g., Interaction, Content, Style). These are provided by domain experts to ensure stability.
Dimensions (Level 2): Core functional points within an aspect (e.g., Search, Display, Login). These are induced automatically using an LLM with a conservative expansion strategy that prioritizes semantic merging over creating new dimensions.
Slots (Level 3): Fine-grained, clarifiable requirement details (e.g., filtering options, sorting rules). These are extracted as tuples of normalized items and candidate clarification questions.

2. Ontology-Guided Interviewing

During the multi-turn interview, OntoAgent executes a decision loop involving four key operations to dynamically select the most relevant yet insufficiently explored requirement slot:

ParseUser: Interprets stakeholder responses to support pruning decisions at both the aspect and dimension levels.
ScoreOnto: Assigns initial priority scores to ontology nodes based on the initial requirement description to ensure the interview begins with high-relevance branches.
ReRankOnto: Dynamically re-evaluates and reorders candidate slots based on the updated dialogue history to adapt to new context.
GatePrune: Implements two pruning mechanisms to reduce redundancy:
- Aspect-level: Triggers a macro-confirmation question after $N$ inquiries yield no new requirements; if negative, the branch is pruned.
- Dimension-level: Immediately prunes slots if a dimension is explicitly rejected by the stakeholder.
QuestionGen: Combines the selected slot with the current dialogue context to generate a natural language, context-aware elicitation question.

Key Contributions

OntoAgent Framework: The proposal of an interview agent that explicitly models interview experience as an ontology and integrates it with LLMs to facilitate structured question generation.
Experience Ontology: The development of a three-level hierarchical tree (Aspects, Dimensions, Slots) that serves as an interpretable and structured questioning space.
Ontology-Guided Mechanism: The design of a decision-making mechanism comprising four operations (ParseUser, ScoreOnto, ReRankOnto, GatePrune) to dynamically select requirement slots and generate context-aware questions.
Comprehensive Evaluation: Extensive quantitative and qualitative evaluations demonstrating the framework's effectiveness and practical applicability.

Experimental Results

The authors evaluated OntoAgent on 101 website application scenarios using the ReqElicitGym environment, comparing it against five baselines (including Non-CoT, CoT, LLMREI-short/long, and Mistake-guided Prompting).

Effectiveness (IRE): OntoAgent achieved an Implicit Requirements Elicitation Ratio (IRE) of 0.69, representing a 33% relative improvement over the best baseline (Mistake-guided Prompting at 0.52). This demonstrates a superior ability to uncover implicit requirements.
Efficiency (TKQR): OntoAgent achieved a Turn-discounted Key Question Rate (TKQR) of 0.59, a 21% relative improvement over the best baseline. This indicates higher questioning efficiency and the ability to identify key requirements earlier in the dialogue.
Ablation Studies: Incremental addition of components confirmed that the Experience Ontology provides the largest performance gain, while ScoreOnto, ReRankOnto, and GatePrune further refine alignment, efficiency, and redundancy reduction.
Robustness: The framework demonstrated strong generalization across six different base LLMs (including Claude, Gemini, DeepSeek, Kimi, GLM, and Qwen), maintaining competitive performance regardless of the underlying model.
Type-wise Analysis: OntoAgent provided balanced coverage across interaction, content, and style requirements. Notably, it significantly outperformed baselines in eliciting style requirements (IRE 0.55 vs. <0.01 for some baselines), a category often neglected by free-form approaches.
Human Evaluation: In a study with 6 participants acting as stakeholders, OntoAgent received significantly higher Likert scores (p < 0.05) in elicitation effectiveness, questioning efficiency, and adaptability compared to baselines.

Significance and Claims

The paper claims that OntoAgent successfully bridges the gap between the generative capabilities of LLMs and the structured cognitive processes of experienced analysts. By transforming free-form LLM chats into systematic and interpretable inquiry processes, OntoAgent addresses the critical issues of implicit requirement omission and redundant questioning.

The authors assert that their work demonstrates the importance of ontology-guided LLM agents for advancing automated requirements development. They conclude that explicitly modeling interview experience is essential for achieving high elicitation coverage and efficiency, and that their approach offers a practical, extensible solution for real-world requirements elicitation tasks beyond the tested web application domain.