"Should I Give Up Now?" Investigating LLM Pitfalls in Software Engineering

This study investigates the pitfalls of integrating large language models into software engineering workflows, revealing that persistent inaccuracies and cognitive overload often lead developers to abandon AI tools, though strategic prompt refinement can significantly mitigate the risk of such abandonment.

The Gist

Imagine you are trying to build a complex Lego castle, but instead of doing it alone, you have a very smart, very chatty robot assistant named "ChatGPT" helping you. You tell the robot what you want, and it tries to build the pieces for you.

This research paper is essentially a report card on how that partnership went. The researchers asked: "When does this robot helper become more trouble than it's worth, causing people to throw up their hands and say, 'Should I give up now?'"

Here is the breakdown of their findings, translated into everyday language:

1. The Setup: A Real-World Test

The researchers didn't just ask people what they thought about AI. They put 26 people (a mix of students and professional software engineers) in a room (virtually) and gave them a tough job: build a website from scratch using a new tool they weren't experts with. They had to use ChatGPT to help them.

The goal wasn't to see if the AI was perfect; it was to see how real humans handle the AI when things go wrong.

2. The Nine Ways the Robot Fails

The researchers found that the robot doesn't just make one mistake; it has nine specific "bad habits" that get annoying. They grouped them into three main categories:

• The "Half-Baked" Cookie (Incomplete or Incorrect Responses):

  • Missing Pieces: You ask for a whole cake, and the robot gives you the frosting but forgets the cake.
  • The "Forgot the Oven" Mistake: The robot tells you how to bake the cake but forgets to tell you to preheat the oven first (missing setup steps).
  • The "Wrong Recipe": The robot gives you a recipe that looks good but tastes terrible (buggy code).

• The "Information Overload" (Cognitive Overload):

  • The Wall of Text: You ask a simple question, and the robot writes a novel. You have to read 50 pages to find the one sentence you needed.
  • The Over-Engineer: You ask how to hang a picture, and the robot suggests building a crane. It makes simple things complicated.
  • The Hallucination: The robot draws a picture of a cat when you asked for a dog, and it insists it's a dog.

• The "Short Memory" (Context Loss):

  • The Amnesiac: You tell the robot, "Make the button blue." Then, in the next sentence, you say, "Make it red." The robot forgets you just asked for blue and gets confused, or it forgets the whole conversation and starts over.
  • The Stubborn Mule: You tell the robot, "No, that's wrong, try again," and it gives you the exact same wrong answer.

3. How People Tried to Fix It (The Mitigation Strategies)

When the robot messed up, people didn't just quit immediately. They tried to fix it, like a mechanic trying to tune a car engine:

• Rephrasing the Request: "Maybe I didn't ask nicely enough."
• Breaking it Down: "Let's build one wall at a time instead of the whole house."
• Manual Debugging: "Okay, I'll take your code, but I have to fix the mistakes myself."
• The "Google" Switch: "You know what? I'm going to look this up on Google instead."

4. The Breaking Point: When Do People Quit?

This is the most important part of the study. 17 out of 26 people eventually gave up on the robot.

The researchers found two main reasons why people quit:

1. The "Unhelpful" Factor: If the robot gave a bad answer, the person was 11 times more likely to quit. It's like if your GPS sends you into a lake three times in a row; you stop trusting it.
2. The "Prompting" Factor: Surprisingly, the more people kept trying (asking more questions), the less likely they were to quit. It's like if you keep arguing with a stubborn friend, you might eventually get the answer. But if the friend keeps giving you nonsense, you walk away.

The Experience Gap:

• Students tended to keep trying longer, even when the robot was failing. They were willing to "fight" the robot to get it to work.
• Professional Engineers were quicker to quit. They knew their time was valuable. If the robot wasn't helping, they just did the work themselves or looked it up elsewhere. They didn't have time to waste on a broken tool.

5. The "Newer Model" Surprise

The researchers tested this again with a newer, "smarter" version of the robot (GPT-5.1).

• The Good News: The new robot was better at the first try.
• The Bad News: Once the conversation got long and complex, the new robot made the exact same mistakes as the old one. It still forgot context, still gave long confusing answers, and still made people quit.

The Big Takeaway

Using AI for coding isn't like using a calculator (where you press a button and get the right answer). It's more like co-piloting a plane with a pilot who sometimes forgets how to fly.

• AI is great for simple, one-off tasks.
• AI struggles with long, complex projects where you need to remember what you did five minutes ago.
• The biggest problem isn't that the AI is "dumb"; it's that the partnership requires so much human effort to fix the AI's mistakes that, eventually, it's faster to just do it yourself.

In short: If you are building something simple, AI is a great helper. If you are building something complex and the AI keeps messing up, don't be afraid to say, "Should I give up now?" and take the wheel yourself.

Technical Summary

Here is a detailed technical summary of the paper "Should I Give Up Now?" Investigating LLM Pitfalls in Software Engineering.

1. Problem Statement

While Large Language Models (LLMs) like ChatGPT are increasingly integrated into software engineering (SE) workflows to boost productivity, their reliability in complex, multi-step tasks remains inconsistent. Existing research often focuses on:

• Short, isolated tasks (e.g., single-function generation) rather than complex, multi-file development.
• Expert-optimized prompting, which does not reflect the "low-scaffolding" reality where developers use LLMs without formal prompt engineering training.
• Static tool assumptions, failing to account for the stochastic, multi-turn, and co-adaptive nature of LLM interactions.

The core problem is a lack of understanding regarding how and why developers abandon LLM assistance during complex tasks. When failures accumulate (e.g., hallucinations, context loss, incorrect code), developers often disengage, rendering the tool counterproductive. This paper investigates the specific failure modes, user mitigation strategies, and the statistical drivers behind this abandonment behavior.

2. Methodology

The study employed a mixed-methods approach combining observational studies, qualitative thematic analysis, and quantitative logistic regression.

• Participants: 26 participants total (21 students with limited web dev experience and 5 professional Software Developers (SDEs)). A replication study was conducted with 6 additional SDEs using GPT-5.1 to ensure findings were robust across model versions.
• Task Design: Participants were tasked with building a complex web application (involving HTML, CSS, JavaScript, and deployment) based on a tutorial. The task required multi-file edits and visual integration, simulating real-world "unfamiliar" work scenarios.
• Protocol:
  • Low-Scaffolding: No formal prompt engineering training was provided to simulate natural usage.
  • Think-Aloud: Participants verbalized their reasoning while working.
  • Data Collection: Screen recordings, interaction logs (prompts/responses), and post-task semi-structured interviews.
• Analysis:
  • Qualitative: Inductive thematic analysis of interaction logs and transcripts to categorize failures and mitigation strategies.
  • Quantitative: Logistic regression modeling to test hypotheses regarding factors influencing the likelihood of abandoning the tool (dependent variable: Abandon_ChatGPT).

3. Key Contributions

1. Taxonomy of LLM Failures: Identification of 9 distinct failure types grouped into three categories:
   • Incomplete or Incorrect Responses: Missing details, missing preconditions, and erroneous code.
   • Cognitive Overload & Information Mismanagement: Overwhelmingly long answers, overcomplicated solutions, and incorrect image generation.
   • Context Loss & Redundant Effort: Loss of conversation context, irrelevant answers, and unresponsiveness to prompt changes.
2. Root Cause Analysis: Differentiation between User-Contributed (UC) factors (e.g., vague prompts, skimming responses) and ChatGPT-Contributed (CC) factors (e.g., ignoring user expertise, poor file interaction, lack of code localization).
3. Mitigation Strategies: Documentation of 7 user strategies (e.g., task scaffolding, prompt clarification, manual debugging, consulting external sources) and their varying effectiveness.
4. Statistical Evidence on Abandonment: A quantitative model revealing the specific odds ratios associated with abandoning LLM assistance.
5. Model Robustness: Demonstration that these failure patterns persist even in newer models (GPT-5.1), indicating that improvements in one-shot generation do not necessarily resolve multi-turn interactional breakdowns.

4. Key Results

A. Failure Patterns (RQ1 & RQ2)

• Prevalence: 58% of interactions resulted in failure cases (92 total cases).
• Top Failures:
  • Incomplete Answers (40%): Omitting crucial details when processing large prompts.
  • Overwhelming Answers (24%): Providing excessive, unstructured text that forces users to sift through irrelevant code.
  • Lack of Context (17%): Failing to remember previous turns, requiring users to re-state details.
• Causes: Failures were rarely due to a single factor. They often stemmed from an interplay where a user's vague prompt (UC) triggered a model's inability to infer context (CC), leading to a cascading breakdown.
• GPT-5.1 Findings: While one-shot performance improved, multi-turn failures (context loss, hallucination during edits) persisted. The model still struggled with "edit locality" (making small changes without rewriting the whole file).

B. Mitigation Strategies (RQ3)

Users attempted to recover from failures using:

1. Further Scaffolding: Breaking tasks into smaller subtasks.
2. Prompt Clarification: Rephrasing questions.
3. Manual Debugging: Testing and fixing the AI's code directly.
4. Consulting External Sources: Switching to Google, StackOverflow, or tutorials.

• Outcome: Despite these efforts, 17 out of 26 participants (65%) ultimately abandoned ChatGPT during the task. Abandonment often occurred after a "tipping point" of repeated unhelpful responses.

C. User Perceptions & Abandonment Drivers (RQ4 & RQ5)

• Perception: Users rated ChatGPT as generally helpful (3-5/5) but expressed frustration with code readability and the effort required to prompt.
• Statistical Findings (Logistic Regression):
  • Unhelpful Responses: Increased the odds of abandonment by a factor of 11 (Odds Ratio = 11.0).
  • Prompt Frequency: Each additional prompt reduced the probability of abandonment by 17% (suggesting persistence correlates with lower abandonment).
  • Coding Experience: Experienced developers were less likely to abandon the tool than novices, though both groups faced similar failure types.
  • Task Completeness: Surprisingly, the objective score of the final task did not significantly predict abandonment; the perception of unhelpfulness was the stronger driver.

5. Significance and Implications

• Theoretical Shift: The paper challenges the view of LLMs as static tools. It posits that LLM failures are interactional trajectories that accumulate over turns, leading to disengagement. This requires new HCI frameworks that account for stochastic, co-adaptive systems.
• Design Recommendations:
  • Context Retention: Models must better maintain state across multi-turn conversations.
  • Edit Locality: Models should be able to make targeted code changes without regenerating entire files.
  • Adaptive Expertise: Responses should adapt to the user's skill level (avoiding over-complication for novices or under-detail for experts).
  • Rabbit-Hole Detection: Systems should detect when a user is stuck in an unproductive prompting loop and suggest alternative strategies (e.g., "Try consulting a tutorial").
• Educational Impact: While LLMs aid code generation, they may hinder deep learning if users rely on them without understanding the underlying structure. Tools need to support "scaffolded learning" rather than just "code completion."

Conclusion: The study concludes that while LLMs are powerful, their current limitations in multi-step, complex reasoning lead to significant cognitive overhead. Without addressing the root causes of interactional breakdowns (context loss, hallucination, and poor edit localization), developers will continue to face a high risk of abandoning these tools, negating their potential productivity gains.