Understanding on the Edge: LLM-generated Boundary Test Explanations

Technical Summary: Understanding on the Edge: LLM-generated Boundary Test Explanations

Problem Statement
Boundary Value Analysis and Testing (BVT) is a fundamental technique in software quality assurance, operating on the premise that faults cluster at the extremes of input domains. While BVT is widely taught and used, it lacks a precise formal definition, forcing practitioners to rely on intuition or informal rules to identify meaningful boundaries. Recent advancements have introduced automated tools capable of discovering boundary candidates without human guidance; however, identification alone is insufficient. Testers require explanations to understand why a specific input-output pair represents a behavioral boundary. Without transparent justification, automated suggestions are often discarded by skeptical developers, similar to auto-generated test cases that lack human readability. While Large Language Models (LLMs) have transformed tasks like test generation and code review, their potential to generate natural-language rationales for boundary testing has not been empirically assessed.

Methodology
The authors conducted an exploratory, sequential mixed-methods study involving 27 software professionals (13 researchers and 14 practitioners) to evaluate LLM-generated boundary explanations.

• Context: The study utilized four Functions Under Test (FUTs): Email validation, Bytecount conversion, Date creation, and BMI calculation. These were treated as black boxes.
• Data Generation: Twenty boundary pairs were selected (five per FUT) from manually selected or automatically discovered candidates. Explanations for these pairs were generated in May 2025 using OpenAI's GPT-4.1 model with a temperature of 0.2. Prompts framed the model as an expert tester, limiting explanations to 150 characters to manage cognitive load.
• Evaluation:
  • Survey: Participants rated the 20 explanations on a five-point Likert scale across four dimensions: Clarity (ease of reading), Correctness (factual accuracy), Completeness (coverage of key aspects), and Perceived Usefulness (added value). The LLM origin of the explanations was concealed during the survey to avoid bias.
  • Interviews: Six participants (three researchers, three practitioners) engaged in semi-structured interviews to elaborate on their ratings, discuss trust, and identify use cases.
• Analysis: Quantitative data were analyzed using descriptive statistics. Qualitative data from survey comments and interview transcripts were analyzed using reflexive thematic analysis.

Key Contributions

1. First Empirical Study: This is the first systematic investigation into whether LLMs can provide meaningful explanations for software boundary cases, establishing a baseline for explainability in BVT.
2. Evaluation Framework: The study validates four criteria (clarity, correctness, completeness, usefulness) as effective dimensions for assessing boundary explanation quality.
3. Requirement Checklist: The authors distilled a seven-item requirement list to guide the design of future LLM-based boundary explanation tools.

Results

• Quantitative Findings: Overall, 63.5% of ratings were positive (4 or 5 on the Likert scale), while 17% were negative (1 or 2). Mean scores across all categories ranged from 3.5 to 4.0.
  • Clarity, Completeness, Usefulness: Averaged 3.7.
  • Correctness: Averaged 3.8.
  • Variability: High standard deviations (1.1–1.3) indicated significant variance in participant perceptions.
  • Specific Issues: The "Date" FUT received notably lower scores for correctness and usefulness due to a specific hallucination where the LLM incorrectly claimed a year field changed from 3 to 4 digits (both were actually 4 digits).
• Qualitative Findings:
  • Design: Participants favored structured explanations that defined technical terms, cited authoritative sources (e.g., RFCs), and adapted depth to user expertise.
  • Usefulness: Explanations were seen as valuable for debugging, documentation, and onboarding junior engineers. However, usefulness was compromised by hallucinations, unrealistic examples (e.g., negative years, 0kg weight), or lack of context.
  • Trust: Trust was fragile; even a single factual error or hallucination significantly undermined confidence. Participants emphasized that explanations should support, not replace, human reasoning.
• Requirements for Future Tools: Based on the findings, the authors propose seven design criteria:
  1. Adaptive Detail Levels: Adjust verbosity based on user expertise.
  2. Authoritative References: Link to standards and documentation to verify logic.
  3. Explicit Structure: Follow a predictable sequence (valid condition, invalid condition, I/O pairs, justification).
  4. Multiple Examples: Include additional boundary pairs on either side of the primary boundary to illustrate behavior changes.
  5. Contextual Justification: Explicitly state underlying assumptions to prevent hallucinations.
  6. Interactive Conversations: Allow users to query or challenge specific elements.
  7. Tool Integration: Surface explanations within IDEs or static analysis dashboards.

Significance and Claims
The paper claims that LLM-generated boundary explanations show promise for supporting automated testing workflows, with the majority of professionals finding them generally clear, correct, and useful. The authors position this work as a "low-stakes yet rigorous entry point" toward using LLMs for boundary identification and assessing their conceptual understanding.

The study modestly asserts that while the results are encouraging, they are indicative rather than definitive. The authors note that the study used a single model (GPT-4.1) with minimal prompt engineering and a limited set of functions. They emphasize that for LLM-based tools to be adopted, they must be refined to ensure accuracy, provide actionable examples, and integrate seamlessly into developer workflows. The ultimate goal is to enhance the explainability and trustworthiness of automated boundary testing through human-AI collaboration, rather than replacing human judgment with black-box automation.