Exploring Creativity in Human-Human-LLM Collaborative Software Design

1. Problem Statement

While Large Language Models (LLMs) are extensively studied for their ability to accelerate software development tasks (e.g., code generation, debugging), there is a significant gap in understanding how LLMs influence collaborative creativity within software engineering. Software design is inherently a creative, problem-solving activity requiring the generation of novel, surprising, and valuable artifacts. The authors investigate whether LLMs act as catalysts for creativity or if they merely automate routine tasks, potentially hindering the creative process by anchoring designers to conventional solutions or causing unproductive digressions.

Research Question: How, and where, does creativity appear naturally when designing with an LLM?

2. Methodology

The authors conducted a controlled laboratory study simulating a remote workplace environment.

• Participants: 18 pairs of software professionals ($N=36$) with design experience (ranging from 5 to 20+ years). Pairs were recruited to mimic real-world team dynamics.
• Task: Pairs were given 90 minutes to produce a software design document for a "Bicycle Parking Application" based on a Product Requirements Document (PRD). The PRD included specific features (e.g., finding parking, reviews, directions) but left room for architectural and feature expansion.
• Conditions:
  • LLM Access: All pairs had access to a custom LLM wrapper (OpenAI's ChatGPT 3.5 Turbo) but were not required to use it.
  • Priming: Participants were not instructed to be creative; they were told the design would be judged on completeness. This ensured creativity emerged naturally rather than being forced.
• Data Collection:
  • Screen recordings of design sessions and exit interviews.
  • Final design documents.
  • Logs of all LLM prompts and responses.
• Analysis Framework:
  • Process Analysis: Used Dorst and Cross's concept of "Creative Events" (moments where problem and solution spaces co-evolve via a "leap" or insight).
  • Product Analysis: Used Boden's definition of creativity: artifacts that are New, Surprising, and Valuable.
  • Coding: Two researchers qualitatively coded transcripts and design artifacts to identify sources of creativity (Human-initiated vs. LLM-assisted).

3. Key Contributions

The paper makes three primary contributions to the field of Human-Computer Interaction (HCI) and Software Engineering:

1. Empirical Evidence of Natural Creativity: It demonstrates that creativity emerges naturally in collaborative software design even without explicit priming, occurring in both the design process and the final product.
2. Source Attribution: It provides a granular taxonomy of creativity sources, distinguishing between human-driven insights (experience, empathy, analogies) and LLM-driven contributions (sparking, elaborating, providing).
3. The "Double-Edged Sword" of LLMs: It reveals that while LLMs can automate routine work to free up cognitive space for deep thinking, they can also hinder creativity by introducing complexity, anchoring designers to suboptimal solutions, or causing unproductive digressions.

4. Key Results

A. Prevalence of Creativity

• Process: Creativity (in the form of creative events) was present in all 18 pairs.
• Product: 13 out of 18 pairs produced design documents containing at least one creative element (New, Surprising, and Valuable).
• LLM Usage: Two pairs did not use the LLM. Three pairs used it to generate the entire design. The rest used it for specific tasks (information seeking, partial generation).

B. Sources of Creativity (The "Who" and "How")

The study identified 8 sources of creative elements (Table 1 in the paper):

• Human-Initiated (Dominant):
  • Prior Knowledge/Experience: The most frequent source (e.g., applying industry standards for security or scalability).
  • Empathy: Designers imagining themselves as users (e.g., adding a "home screen pin" for quick access).
  • Analogies: Comparing the problem to existing apps (e.g., Google Maps, Yelp) to solve design challenges.
  • Ambiguity & Underlying Goals: Interpreting vague requirements or focusing on the root problem rather than just the stated requirements.
• LLM-Assisted (Minority):
  • LLM Sparks: An LLM response triggered a new human idea (e.g., suggesting a CDN led to a performance insight).
  • LLM Elaborates: The LLM expanded on a human-initiated idea (e.g., fleshing out cost metrics).
  • LLM Provides: The LLM suggested a novel idea directly (e.g., content moderation).

Statistical Finding: Of the 29 creative elements identified in the final designs, 15 were human-initiated, 5 were joint human-LLM, and 9 were directly provided by the LLM. Human traits (experience, empathy) were the primary drivers.

C. Impact of LLM on the Design Process

• Positive: LLMs often automated "boilerplate" work (e.g., API definitions, data models), freeing up cognitive space for designers to engage in deeper problem-solving and strategic thinking.
• Negative:
  • Anchoring: Designers sometimes accepted the first LLM solution without exploring alternatives.
  • Digressions: LLM responses occasionally triggered unproductive discussions (e.g., re-debating user accounts already settled).
  • Over-complication: LLMs sometimes suggested overly complex architectures (e.g., mixing multiple cloud providers) that designers had to simplify.

D. Product Creativity Findings

• Novelty vs. Surprise: LLM-generated ideas were often "new" relative to the PRD but rarely "surprising" or groundbreaking. They tended to be commodity features (e.g., standard content moderation).
• The "Fixed Requirements" Trap: Pairs that pasted the entire PRD into the LLM prompts tended to produce constrained, non-creative designs. Conversely, pairs that used open-ended prompts or relied on their own judgment produced more creative outcomes.
• Non-LLM Success: Interestingly, the two pairs that did not use the LLM still produced creative designs, proving that LLM availability does not guarantee creativity.

5. Significance and Implications

• Human-Centric Creativity: The study reinforces that human insight remains central to software design. LLMs act as amplifiers rather than replacements for human creativity.
• Prompt Engineering Strategy: To foster creativity, practitioners should use open-ended, expansive prompts with minimal context rather than rigidly feeding detailed requirements to the LLM. This prevents the LLM from constraining the solution space.
• Cognitive Offloading: Using LLMs to handle routine tasks (boilerplate code, documentation) is beneficial as it creates the "space" necessary for deep, creative thought.
• Critical Engagement: Designers must remain critical of LLM outputs to avoid anchoring bias and unproductive digressions.
• Future Tooling: The authors suggest the need for "Creativity Support Tools" built on LLMs that are specifically designed to encourage exploration of the problem space, mitigate design fixation, and maintain human agency.

Conclusion:
The paper concludes that while LLMs can support collaborative software design by generating novel ideas and elaborating on human concepts, they do not inherently drive high-level creativity. The most significant creative leaps came from human designers leveraging experience, empathy, and analogies. Effective integration of LLMs requires intentional engagement strategies that prioritize human agency and critical thinking over passive acceptance of AI suggestions.