💻 computer science

The Role of Cognitive Abilities in Requirements Inspection: Comparing UML and Textual Representations

This study reveals that the effectiveness of UML diagrams in requirements inspection is not uniform but significantly interacts with individual cognitive abilities, where high working memory and mental rotation skills lead to reduced violation detection but improved justification accuracy when using visual aids.

Original authors: Giovanna Broccia, Sira Vegas, Alessio Ferrari

Published 2026-01-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Giovanna Broccia, Sira Vegas, Alessio Ferrari

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a quality control inspector for a video game. Your job is to read the rulebook (the requirements) and find mistakes, like a rule that says "jump high" without saying how high, or a rule that contradicts another one.

This paper is like a science experiment to figure out: Does giving inspectors a picture (a UML diagram) alongside the text help them find mistakes better? And does the inspector's brain power change the answer?

Here is the breakdown of what the researchers found, using simple analogies.

The Setup: Text vs. Pictures

The researchers gave 38 students two different "rulebooks" to inspect:

The Text-Only Book: Just words.
The Picture Book: The same words, but with a flowchart (a UML sequence diagram) showing how the game moves.

They wanted to see if the pictures helped people find errors (detection) and explain why something was wrong (justification).

The "Brain Power" Variables

The researchers also measured two specific types of brain power in the students:

Working Memory: Think of this as the size of your "mental sticky note." It's how many pieces of information you can hold in your head at once while doing a task.
Mental Rotation: Think of this as your ability to spin a 3D object in your mind. It's how good you are at understanding shapes and spatial relationships.

The Surprising Results

The researchers expected that having a picture would always help, or at least that people with "super brains" (high working memory and high mental rotation) would do the best job. They were wrong.

Here is what actually happened, broken down by the two tasks:

1. Finding the Mistakes (Detection)

The Analogy: Imagine trying to find a specific needle in a haystack.

The Result: For people with very high brain power (high working memory AND high mental rotation), the picture actually hurt their performance.
Why? These smart students tried to do too much at once. They looked at the text and the picture, trying to hold all the details in their heads simultaneously. It was like trying to juggle five balls while riding a unicycle. Their brains got "overloaded," and they missed mistakes they would have caught if they just looked at the text alone.
The Takeaway: For the hardest part of the job (spotting errors), having a picture didn't help the smartest students; it actually made them slower and less accurate because they tried to process too much information.

2. Explaining the Mistakes (Justification)

The Analogy: Imagine you found the needle and now you have to write a report explaining exactly why it was the wrong needle.

The Result: For the same group of very high brain-power students, the picture helped them write better explanations.
Why? Once they found the error, their strong brains could use the picture to organize their thoughts. The visual map helped them connect the dots and write a clearer, more logical reason for why the rule was broken.
The Takeaway: The picture acted like a "thinking aid" for the smart students, helping them articulate their ideas better, even if it made the initial search harder.

The "Goldilocks" Zone

The study suggests there isn't a "one-size-fits-all" tool.

For some people: The picture was confusing and made them miss errors.
For others (the high-capacity group): The picture was a double-edged sword. It made finding errors harder (due to overload) but made explaining errors easier (due to better reasoning).

The Main Conclusion

You cannot just say "Pictures are good" or "Pictures are bad." It depends on the person's brain.

If you give a complex visual tool (like a UML diagram) to someone with a very powerful brain, they might get too excited and try to process everything at once, causing them to stumble during the search.
However, that same powerful brain can use the visual tool to explain things very well once they have found the problem.

In short: The paper warns us that adding more information (like diagrams) doesn't always make a task easier. Sometimes, for very capable people, it adds so much "noise" that they get overwhelmed, even though their brains are strong enough to handle it in other ways. The best tool depends on the specific task and the specific person using it.

Technical Summary: The Role of Cognitive Abilities in Requirements Inspection: Comparing UML and Textual Representations

Problem Statement
Software inspection is a critical process for identifying defects in artifacts, yet its effectiveness is debated regarding the representation of requirements. While visual representations like UML diagrams are widely used alongside text, their impact on inspection accuracy remains inconsistent in empirical literature. Furthermore, existing research often treats inspection factors in isolation, failing to account for the interplay between the representation format and the inspector's cognitive profile. Specifically, the role of cognitive abilities—such as Working Memory (WM) capacity and mental rotation skills—in modulating the effectiveness of UML support during requirements inspection is under-explored. Prior studies suggest that graphics can either compensate for cognitive limitations or impose additional load, but the specific interaction between these abilities and visual notations in inspection tasks has not been systematically modeled.

Methodology
The authors conducted a controlled crossover experiment involving 38 master's students in Software Engineering. The study employed a within-subjects design where participants performed requirements inspections under two treatments:

Textual Requirements: Requirements presented exclusively in text.
UML-Supported Requirements: The same requirements accompanied by UML sequence diagrams.

The experiment utilized Checklist-Based Reading (CBR) as the fixed reading technique to ensure uniformity. Participants were divided into two groups with balanced cognitive profiles to counterbalance the order of treatments (Text $\to$ UML vs. UML $\to$ Text), with a one-week washout period between sessions to mitigate carryover effects.

Cognitive Assessment:
Prior to the inspection tasks, participants underwent two standardized cognitive assessments:

Working Memory (WM) Capacity: Measured via an Operation Span task (PCU scoring method).
Mental Rotation Ability: Measured via a 3D Mental Rotation task.

Performance Metrics:
Inspection performance was evaluated using two dependent variables:

Issue Detection Accuracy: Measured by the F1-score (harmonic mean of precision and recall) for identifying requirements violating a quality checklist.
Issue Justification Accuracy: Measured by "Accuracy Why," a custom metric assessing the quality of the rationale provided for identified violations on a 0–4 Likert scale.

Statistical Analysis:
The authors employed Linear Mixed-Effects Models (LMM) for F1-score and Generalized Linear Models (GZLM) for Accuracy Why. These models analyzed the main effects of treatment and cognitive abilities, as well as their two-way and three-way interactions. A hierarchical approach to multiplicity adjustment (Benjamini–Hochberg) was used to control Type I errors.

Key Results
The study revealed that the impact of UML diagrams is not uniform but is significantly moderated by the interaction of cognitive abilities:

Three-Way Interaction (Detection): A significant three-way interaction was found between treatment, WM capacity, and mental rotation ability ( $p = 0.0774$ ) for issue detection. Contrary to the hypothesis that high cognitive abilities would universally improve performance, participants with high scores in both WM and mental rotation experienced a reduction in detection accuracy (F1-score) when using UML support compared to text alone. The data suggests an "optimal-capacity window" where performance peaks at moderate-to-high WM levels but declines when capacity is extremely high under UML conditions, potentially due to cognitive overload.
Three-Way Interaction (Justification): The same three-way interaction showed a positive effect on justification accuracy ( $p = 0.0013$ ). Participants with high WM and mental rotation abilities provided significantly better justifications for violations when supported by UML diagrams. This indicates that while high cognitive load may hinder the initial detection of issues in multi-modal environments, these same abilities support deeper reasoning and articulation once an issue is identified.
Isolated Effects: When analyzed in isolation (two-way interactions), the effects were either non-significant or moved in the opposite direction, highlighting that analyzing cognitive factors separately can lead to misleading conclusions.
Carryover Effects: No significant carryover effects were detected, validating the crossover design.

Contributions and Significance
The paper makes three primary contributions:

Empirical Investigation of Interaction: It provides novel evidence that cognitive abilities and representation formats do not operate independently. The study demonstrates that the effectiveness of UML diagrams is contingent upon the specific combination of an inspector's WM capacity and spatial ability.
Differential Impact on Performance Measures: The work reveals that cognitive profiles moderate inspection performance differently depending on the outcome measure. High cognitive abilities can be detrimental to detection accuracy under UML support (likely due to overload) while simultaneously enhancing justification accuracy (likely due to improved reasoning capabilities).
Practical Implications for Task Assignment: The findings suggest that "one-size-fits-all" approaches to requirements inspection are suboptimal. The authors argue that inspection tasks and tool support should be tailored to individual cognitive profiles. Specifically, in educational or training settings, the introduction of visual notations like UML may require scaffolding strategies to prevent cognitive overload, even for students with high cognitive abilities.

Conclusion
The authors conclude that while UML diagrams do not universally improve requirements inspection accuracy, they can benefit specific cognitive profiles, particularly for the reasoning and justification phases of inspection. The study cautions against the assumption that higher cognitive abilities always yield better performance in complex, multi-modal tasks. Instead, it highlights the need to consider individual cognitive profiles when designing inspection processes, noting that multiple means of representation are not always beneficial and can sometimes hinder performance if they exceed the user's optimal cognitive load.