Comparing LLM-Based Conversational and Graphical Interfaces for Industrial Decision Tasks: An Exploratory Mixed-Methods Study

Technical Summary: Comparing LLM-Based Conversational and Graphical Interfaces for Industrial Decision Tasks

Problem Statement
In industrial decision-making contexts, particularly within Smart Manufacturing and Industry 4.0, decision support systems (DSS) rely heavily on graphical dashboards. While dashboards consolidate operational data, they often impose high cognitive loads on users due to the need for visual searching, manual integration of information across multiple views, and the management of working memory during complex comparisons. This can lead to information overload, especially when data is dense or poorly integrated.

Conversational User Interfaces (CUIs) powered by Large Language Models (LLMs) have emerged as a potential alternative, promising direct access to data through natural language and the automation of reasoning tasks. However, it remains unclear whether LLM-based CUIs can effectively mitigate information overload or improve decision performance compared to traditional dashboards in high-stakes industrial settings. Existing empirical comparisons are scarce, particularly regarding how these interfaces perform across tasks with varying cognitive demands (e.g., simple retrieval vs. complex problem solving).

Methodology
The authors conducted an exploratory mixed-methods study involving 20 participants (primarily Computer Science students) to compare an LLM-based Conversational Agent (CA) against a state-of-the-art graphical dashboard.

• Experimental Design: A 2×2 within-subjects factorial design was employed. The independent variables were Interface Type (Dashboard vs. CA) and Task Complexity (Information Retrieval [IR] vs. Problem Solving [PS]).
• Tasks: Participants completed four simulated industrial decision tasks within a synthetic factory scenario.
  • IR Tasks: Required locating specific facts or data points.
  • PS Tasks: Required interpreting data, comparing multiple factors, and making optimal decisions (e.g., selecting machine configurations).
  • Complexity Metric: Task complexity was quantified using Wood's (1986) model, excluding dynamic complexity, resulting in a complexity score of 3 for IR tasks and 9 for PS tasks.
• Interfaces:
  • Dashboard: The Zerynth platform, providing standard visual analytics (charts, tables).
  • CA: A custom chatbot based on GPT-4o, delivered via Telegram, equipped with tools to autonomously retrieve, filter, compare, and aggregate data from the same underlying dataset.
• Measures:
  • Objective: Task completion time, decision accuracy (binary for IR; Percent of Maximum Possible scoring for PS), and interaction logs.
  • Subjective: NASA-TLX for mental workload, post-study questionnaires for perceived usefulness and confidence, and semi-structured interviews.
• Analysis: Quantitative data were analyzed using non-parametric and parametric tests (Mann-Whitney U, Wilcoxon signed-rank, t-tests) appropriate for the data distributions. Qualitative data underwent a codebook-based thematic analysis, assisted by multiple LLMs (ChatGPT 5.2, Gemini 3 Pro, Claude 4.5) to ensure coding consistency, achieving a Krippendorff's $\alpha$ of 0.738.

Key Contributions

1. Empirical Comparison: The study provides direct empirical evidence comparing LLM-based CUIs and dashboards in industrial decision tasks, addressing a gap in the literature regarding their relative merits in terms of workload and performance.
2. Task-Contingent Findings: It demonstrates that the advantages of conversational interfaces are not uniform but depend heavily on task demands and the specific nature of the interaction (retrieval vs. synthesis).
3. Divergence of Usefulness and Confidence: The research highlights a critical distinction where users perceive CUIs as more useful and efficient but do not necessarily trust them more than dashboards, particularly regarding the auditability of outputs.
4. Hybrid Workflow Insight: The study identifies that users often envision a complementary role for both interfaces rather than a replacement, utilizing dashboards for overview and verification and CUIs for direct synthesis.

Results

• Mental Workload: Participants reported significantly lower mental workload with the CA in specific comparisons, particularly when tasks involved direct retrieval. Qualitative themes indicated that the CA reduced "mental bookkeeping" by compressing multi-step navigation into single queries. However, cross-interface differences in workload were not statistically significant across all tasks.
• Performance (Time & Accuracy):
  • Time: The CA was significantly faster than the dashboard in two of the four tasks (one IR and one PS). In other tasks, differences were not statistically significant.
  • Accuracy: The CA showed significantly higher decision accuracy only in Task 3 (a PS task involving energy cost minimization). No significant differences were found in other tasks.
• User Preference and Confidence:
  • Usefulness: The CA was rated significantly higher in perceived usefulness.
  • Confidence: There was no significant difference in confidence levels; however, when "confidence" was explicitly the criterion for future use, participants overwhelmingly preferred the dashboard.
  • Task Preference: For Problem Solving tasks, participants significantly preferred the CA. For Information Retrieval, preferences were mixed.
• Qualitative Themes:
  • Compression: The CA reduced interaction effort by synthesizing data.
  • Auditability: Users trusted the dashboard more because it allowed direct inspection of raw data, whereas the CA's "black box" nature required verification efforts (e.g., re-asking questions) that could offset speed gains.
  • Skill Gatekeeping: Successful use of the CA depended on prompt formulation skills, while dashboard use depended on visual literacy and interface familiarity.
  • Breakdowns: Navigation dead-ends in the dashboard and opaque reasoning in the CA both increased cognitive load.

Significance and Claims
The paper modestly claims that its findings suggest LLM-based CUIs are not a universal replacement for traditional dashboards but rather a complementary tool. The significance lies in identifying the conditions under which each interface excels:

• CUIs are advantageous for reducing interactional effort in tasks requiring direct retrieval, filtering, or synthesis, provided users can formulate clear prompts.
• Dashboards remain superior for tasks requiring persistent overview, explicit comparison, and verification of decision bases, offering higher auditability.

The authors conclude that the most plausible design direction is not the substitution of one interface by the other, but the development of hybrid systems. In such systems, conversational agents could handle avoidable interaction costs (search and synthesis), while graphical interfaces preserve overview, transparency, and user control. The study emphasizes that the relative value of these interfaces depends on task demands and the user's ability to interpret and rely on the resulting information, noting that further large-scale studies with experienced industrial decision-makers are needed to validate these exploratory findings.