💻 computer science

A Comparative Study of Competency Question Elicitation Methods from Ontology Requirements

This paper presents an empirical comparative evaluation of manual, pattern-based, and LLM-driven Competency Question (CQ) elicitation methods, revealing that while LLMs can effectively generate initial CQs, the resulting outputs vary significantly in quality and typically require human refinement before being used for ontology modeling.

Original authors: Reham Alharbi, Valentina Tamma, Terry R. Payne, Jacopo de Berardinis

Published 2026-04-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Reham Alharbi, Valentina Tamma, Terry R. Payne, Jacopo de Berardinis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a massive, digital library for a museum's collection of music memorabilia. Before you can organize the books, records, and photos, you need a blueprint. In the world of computer science, this blueprint is called an Ontology. It's a map that tells the computer how everything is connected (e.g., "This song was written by that artist").

But how do you know if your blueprint is good? You need a checklist of questions that the computer must be able to answer. These are called Competency Questions (CQs).

Example: "Who wrote this song?" or "Which albums were released in 1990?"

If the questions are vague, the blueprint is useless. If they are too complicated, the computer gets confused.

The Big Question: Who is the best "Question Writer"?

The authors of this paper wanted to find out: Who is better at writing these checklist questions?
They compared three different "writers":

The Human Expert: A seasoned computer scientist who knows the rules of the game.
The Template Filler: A method that uses pre-made sentence structures (like a "Mad Libs" game) where you just plug in the words.
The AI Robot: A super-smart Large Language Model (like GPT-4 or Gemini) that writes questions automatically.

They gave all three writers the exact same story about the museum's music collection and asked them to write their lists of questions. Then, they graded the results.

The Results: The Race for the Best Questions

Here is what they found, using some fun analogies:

1. The Human Expert (The Master Chef) 🧑‍🍳

Performance: The humans wrote the best questions.
Why? Their questions were clear, easy to read, and hit the nail on the head. They didn't just ask what was written in the story; they asked what was implied.
Analogy: Think of the Human Expert as a Master Chef. They know exactly what ingredients are needed to make a perfect dish. They don't just follow a recipe; they know why you need salt and how to balance the flavors. Their questions were short, punchy, and exactly what the computer needed to understand the museum.

2. The Template Filler (The Assembly Line Worker) 🏭

Performance: This method was okay, but a bit stiff and repetitive.
Why? It used pre-made patterns (like "Which [Item] has [Feature]?"). It worked, but it felt robotic and sometimes missed the nuance of the story.
Analogy: This is like an Assembly Line Worker stamping out parts. They are fast and consistent, but they can't improvise. If the story needed a creative twist, the template just didn't fit, leaving some gaps in the blueprint.

3. The AI Robot (The Over-Enthusiastic Intern) 🤖

Performance: The AI wrote a lot of questions, but they were often too long, too complicated, and sometimes missed the point.
Why? The AI tried too hard. It used big words, complex sentence structures, and asked questions that were technically correct but hard for a human to understand. It also missed the "hidden" requirements that the human experts caught.
Analogy: The AI is like an Over-Enthusiastic Intern who wants to impress the boss. They write a 10-page essay when a one-sentence answer would do. They use fancy vocabulary to sound smart, but they sometimes miss the simple, practical needs of the job. They also tend to go off on tangents, asking questions that aren't really in the story.

The "Secret Sauce" Discovery

The study found something really interesting about Relevance:

Humans asked questions about things that weren't explicitly written in the story but were obviously necessary (e.g., "What is the file format?" even if the story didn't say "file format" explicitly). They used their brain to fill in the blanks.
AI stuck strictly to what was written. If the story didn't say it, the AI didn't ask about it. It lacked the "common sense" or "domain knowledge" that humans have.

The Verdict

Can we just let the AI do all the work?
Not yet.

The study concludes that while AI is a great tool for brainstorming (getting the ball rolling), it cannot replace the human expert.

AI is like a rough draft. It gives you a pile of ideas, but they need editing.
Humans are the editors. They need to take the AI's messy, complex draft, simplify it, fix the grammar, and add the "common sense" questions that the AI missed.

The Takeaway

If you are building a digital map of knowledge, don't just ask the robot to do it alone. Use the robot to get ideas, but let the human expert refine them. The best results come from a hybrid team: the speed of the AI combined with the wisdom of the human.

In short: The AI is a fast typist with a big vocabulary, but the Human is the one who actually knows what the story is really about.

1. Problem Statement

Competency Questions (CQs) are fundamental to ontology engineering, serving to scope knowledge models, guide design, and validate ontologies. Despite their importance, CQs are often underutilized because they are time-consuming to formulate and lack standardized tool support. While recent literature proposes various generation methods—ranging from manual formulation and pattern-based instantiation to the use of Large Language Models (LLMs)—there is a significant gap in systematic, empirical comparisons of these approaches. Existing studies often lack:

Controlled comparisons using identical source requirements.
Multi-dimensional assessments of CQ quality (e.g., ambiguity, readability, complexity, and semantic overlap).
A comprehensive dataset generated by different methods from the same source material.

2. Methodology

The authors conducted a controlled comparative study using a single user story from the cultural heritage domain (involving a music archivist and collection curator) as the source of requirements. They generated CQs using three distinct approaches:

Manual Formulation: Two experienced ontology engineers (HA-1, HA-2) independently formulated CQs without constraints.
Pattern-Based: An engineer instantiated predefined CQ templates (archetypes) based on the entities and relations in the user story.
LLM-Based: Two state-of-the-art models (GPT-4.1 and Gemini 2.5 Pro) generated CQs from the user story using a neutral prompt (no specific constraints on number or style to avoid bias).

Dataset Creation (AskCQ):
The study resulted in AskCQ, a novel multi-annotator dataset containing 204 CQs across five sets (HA-1: 44, HA-2: 54, Pattern: 38, GPT: 26, Gemini: 42). All CQs were anonymized to prevent evaluator bias.

Evaluation Framework:
The study addressed three Research Questions (RQs) using a mixed-methods approach:

RQ1 (Suitability): Expert evaluation by three ontology engineers to rate CQ suitability (accept/reject) and identify ambiguity.
RQ2 (Characteristics): Quantitative analysis of:
- Ambiguity: Based on expert comments and consensus.
- Relevance: Rated by an LLM (Gemini 2.5 Pro) on a 4-point Likert scale (explicit vs. inferential requirements).
- Readability: Measured via Flesch-Kincaid Grade Level (FKGL) and Dale-Chall Readability (DCR).
- Complexity: Assessed across four dimensions: Character length ( $c_0$ ), Requirement complexity ( $c_1$ , ontological primitives), Linguistic complexity ( $c_2$ , NLP features), and Syntactic complexity ( $c_3$ , dependency parsing).
RQ3 (Semantic Overlap): Analysis using Sentence-BERT embeddings to calculate centroid similarity, directional coverage, and bidirectional coverage between CQ sets.

3. Key Contributions

AskCQ Dataset: The first publicly available multi-annotator dataset of CQs generated from the same source material using manual, pattern-based, and LLM-based methods.
Systematic Comparative Framework: A multi-faceted evaluation framework assessing CQs across suitability, ambiguity, relevance, readability, complexity, and semantic diversity.
Empirical Evidence on LLMs: A rigorous analysis demonstrating that while LLMs can generate CQs, they differ significantly in characteristics compared to human experts, often requiring refinement.

4. Key Results

Suitability and Expert Agreement (RQ1)

Human-Generated CQs: Achieved the highest suitability scores (HA-2: 98% acceptance; HA-1: 91% acceptance) and the lowest proportion of negative comments.
LLM-Generated CQs: Showed moderate suitability (GPT: 85%; Gemini: 67%) but received significantly more critical comments.
Pattern-Based: Had the lowest suitability (50% acceptance) and the highest variability, suggesting templates alone are insufficient without deep contextualization.

Readability and Complexity (RQ2)

Readability: Human CQs were the most readable (lowest FKGL/DCR scores, equivalent to early secondary school level). LLM CQs were significantly less readable, requiring late secondary or university-level comprehension.
Complexity: LLMs produced markedly more complex CQs:
- Length: LLM CQs were roughly double the character length of human CQs (e.g., GPT avg. 111 chars vs. HA-1 avg. 42 chars).
- Requirement/Linguistic/Syntactic: LLMs scored highest in all complexity dimensions, indicating they generate questions with more ontological primitives, elaborate phrasing, and complex grammatical structures.
Relevance: Human experts were better at identifying inferential requirements (Score 3 on the relevance scale)—questions not explicitly stated but functionally necessary. LLMs tended to stick strictly to explicit text (Score 4) or introduce irrelevant requirements.

Semantic Overlap (RQ3)

Thematic Alignment: All sets showed high centroid similarity (0.61–0.85), indicating they addressed the same core themes.
Granular Overlap: Despite thematic alignment, specific CQ overlap was very low.
- Human sets (HA-1 vs. HA-2) showed the highest bidirectional coverage (15.3%).
- LLM sets showed negligible overlap with human sets (often 0% coverage) and low overlap with each other (e.g., GPT vs. Gemini: 2.9%).
- Gemini produced the most distinct set of CQs, with almost no overlap with human or pattern-based sets.

5. Significance and Conclusion

Human Expertise is Irreplaceable (for now): The study confirms that ontology engineers produce CQs that are more readable, less complex, and better at capturing implicit (inferential) requirements.
LLMs as Assistants, Not Replacements: LLMs are useful for initial brainstorming but generate outputs that are often too complex, less readable, and semantically divergent from human expectations. They generally require a refinement step by humans.
Variability of LLMs: Different LLMs (or even the same model with different prompts) produce vastly different CQ sets for the same input, posing a risk of inconsistency if used as the sole basis for ontology design.
Future Directions: The authors suggest a hybrid approach where LLMs assist in generation, but human experts refine the output. Future work should focus on prompt engineering to guide LLMs toward the "desiderata" of human experts (e.g., lower complexity, higher inferential relevance) and expanding the study to other domains.

Limitations: The study is limited to a single user story in the cultural heritage domain, and the complexity metrics are proxies rather than formal definitions. However, the controlled nature of the study isolates the generation method as the primary variable, providing robust comparative insights.