Understanding the Human-LLM Dynamic: A Literature Survey of LLM Use in Programming Tasks

1. Problem Statement

Large Language Models (LLMs) have rapidly transformed programming practices, offering capabilities in code generation, analysis, and repair. However, while technical benchmarks exist, there is a lack of comprehensive understanding regarding how humans actually interact with LLMs in real-world programming scenarios and the net impact of these interactions on human capability and task performance.

Existing literature often focuses on model performance against static datasets or isolated usability studies. There is a gap in synthesizing user studies to understand:

• The specific interaction patterns and prompting strategies employed by programmers.
• Whether LLMs genuinely enhance human productivity and learning or introduce new cognitive burdens.
• How specific interaction behaviors correlate with task success or failure.
• The variability in results across different user expertise levels (novices vs. experts) and task types.

2. Methodology

The authors conducted a systematic literature survey focusing on user studies and experience reports involving LLMs in programming tasks.

• Search Strategy:
  • Databases: ACM, IEEE, Scopus, and arXiv.
  • Keywords: Combinations of LLM identifiers (e.g., "ChatGPT," "Copilot," "LLM") and programming terms (e.g., "programmers," "code generation," "program synthesis").
  • Scope: Papers published post-2017 (coinciding with the Transformer architecture).
  • Inclusion Criteria: Papers must involve LLMs based on the Transformer architecture, pertain to programming, and include a user study or experience report with observed human-LLM interactions.
  • Exclusion Criteria: Studies relying solely on dataset evaluations without human interaction, or studies not focused on the programming process.
• Data Collection:
  • Initial search yielded a broad set, which was refined through title/abstract screening and full-text review.
  • Snowballing: Backward and forward snowballing (citations) were used to augment the dataset.
  • Final Corpus: 88 papers were selected for analysis.
• Analysis Framework:
  • The papers were categorized by study goals (e.g., Human Enhancement, Task Performance, Interaction Analysis).
  • Participants were classified as Academia vs. Industry and Novices vs. Experts (using a modified Dreyfus Model).
  • Four Research Questions (RQs) guided the qualitative and quantitative synthesis.

3. Key Contributions

The paper makes four primary contributions to the field of Human-Computer Interaction (HCI) and Software Engineering:

1. Taxonomy of Interaction Patterns: A detailed classification of how programmers interact with LLMs, including request types, prompting strategies, and review behaviors.
2. Synthesis of Performance Metrics: A consolidated view of metrics used to evaluate human enhancement (productivity, learning) and task performance (correctness, security, readability).
3. Identification of Contextual Factors: Evidence showing that LLM effectiveness is highly dependent on user expertise, task complexity, and the structure of the LLM implementation (guided vs. unstructured).
4. Practical Guidelines & Standardization Proposal: Recommendations for researchers to standardize evaluation metrics and for practitioners to adopt effective interaction techniques.

4. Key Results

RQ1: Interaction Observations

• Request Types: Users make four types of requests:
  1. Learning/Exploration: Asking for explanations or implementation ideas (common in novices).
  2. Solution-Oriented: Requesting specific code generation or test cases.
  3. Error-Correcting: Debugging code or refining incorrect LLM outputs.
  4. Unrelated: Social prompts or off-topic queries.
• Prompting Strategies:
  • Single-Prompt (Zero-shot): Asking for a solution in one go.
  • Multi-Prompt: Breaking tasks into sub-tasks.
  • Re-prompting Techniques: When initial attempts fail, users employ strategies like re-asking, verification (asking the LLM to check itself), task transformation (changing the problem entirely), word restructuring, and elaboration (adding context). Elaboration was the most common and effective strategy.
• Interaction Modes:
  • Only LLM: Users rely entirely on the model (indicates dependence).
  • No LLM: Users revert to manual coding (rare, often due to cognitive load).
  • Hybrid: The most common mode; users combine LLM generation with manual coding and review to manage cognitive load.
• Time Allocation: Users spend the majority of time (>50%) on understanding LLM responses and prompt crafting, rather than writing code. This signifies a shift from "code writing" to "code reviewing."

RQ2: Human Enhancement Evaluation

• Time Productivity: Results are mixed. LLMs significantly improve productivity for simple tasks and syntax-heavy coding. However, for complex tasks, productivity often decreases due to the time required to understand, debug, and verify LLM output.
• Learning:
  • Structured/Guided: When LLMs are used as tutoring systems with clear goals, learning gains are significant.
  • Unstructured: When students have free access to general LLMs, learning outcomes are often neutral or negative. Some studies showed students using LLMs performed worse on manual coding post-tests, suggesting a potential "syntax atrophy" or over-reliance.

RQ3: Task Performance Evaluation

• Collaborative Correctness: LLMs show high accuracy on basic programming and algorithmic tasks but exhibit significant variance on complex or specific domain tasks.
• Code Security & Reliability: Findings are contradictory. Some studies show LLMs generate more secure code; others indicate they introduce more vulnerabilities (bugs, insecure libraries). Explicit evaluation of security remains a research gap.
• Readability: LLM-generated code is generally readable but may use concepts unknown to the user, requiring further prompting to understand.

RQ4: Interaction Effects on Performance

• Behavioral Impact: Specific interaction patterns correlate with outcomes.
  • Learning/Exploration modes often reduce time productivity but increase understanding.
  • Implementation modes improve time productivity.
  • Elaboration (providing detailed context) and precise error reporting are the strongest predictors of successful task completion.
• Expertise Gap: Experts tend to spend less time reviewing code and more time on manual integration, while novices spend more time trying to understand the generated code, sometimes leading to "copy-paste" behaviors without verification.

5. Significance and Implications

• For Researchers:
  • Standardization: The paper proposes a standardized set of metrics (Table 8) for evaluating human-LLM interactions (e.g., Success Rate, Completion Time, Pre/Post-test scores) to allow for better cross-study comparison.
  • Future Directions: Highlights the need to study interaction patterns across different LLM models (beyond just ChatGPT/Copilot) and to investigate how specific prompting strategies quantitatively affect performance.
• For Practitioners & Educators:
  • Prompting Skills: Effective LLM usage requires "prompt engineering" skills (elaboration, context provision) rather than just syntax knowledge.
  • Learning vs. Understanding: Educators must distinguish between using LLMs to understand code (beneficial) versus using them to learn to program (potentially detrimental if unstructured).
  • Hybrid Workflows: The most effective workflow is "Hybrid," where the human acts as the architect and reviewer, and the LLM acts as a component generator.
• System Design: LLM interfaces should be designed to reduce cognitive load, perhaps by segmenting responses or providing context-aware suggestions to aid the "reviewing" phase, which is currently the most time-consuming part of the interaction.

Conclusion

The paper concludes that the Human-LLM dynamic is non-deterministic and highly context-dependent. While LLMs offer significant potential for enhancing programming productivity and learning, these benefits are not automatic. They depend heavily on the user's expertise, the complexity of the task, and the user's ability to engage in effective interaction strategies (specifically elaboration and verification). The authors call for a shift in research focus from purely model-centric benchmarks to user-centric interaction studies to fully realize the potential of LLMs in software engineering.