🤖 AI

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions

This tertiary review synthesizes 30 secondary studies on large language model-based code generation to reveal a rapidly growing but unevenly evaluated field where strong benchmark performance contrasts with significant challenges in real-world generalization, robustness, efficiency, and socio-technical integration.

Original authors: Muslim Chochlov, Michael English, Jim Buckley

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Muslim Chochlov, Michael English, Jim Buckley

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of software development as a massive, bustling construction site. For years, workers (programmers) have built buildings (software) by hand, brick by brick. Recently, a new kind of robot has arrived: the Large Language Model (LLM). These robots can read a blueprint (a description) and instantly lay down bricks, write code, or even fix broken walls.

This paper is not a report on a single robot's performance. Instead, it is a "Tertiary Review." Think of it as a meta-report. The authors didn't go out and test the robots themselves. Instead, they gathered and analyzed 30 existing reports (secondary studies) that had tested these robots. They wanted to see the big picture: How fast is this field growing? What do the reports say works? Where are the robots failing? And what do experts think we need to do next?

Here is the breakdown of their findings, translated into everyday language:

1. The Landscape: A Field Exploding with Activity

The authors looked at the "construction site" of research.

The Boom: In 2022, there was only one report about these robots. By 2024, that number jumped to 17. It's like a sudden gold rush where everyone is writing about the new robots.
Growing Up: At first, the reports were just "surveys" (looking around and saying, "Wow, there are lots of robots!"). Now, the reports are becoming "systematic reviews" (deep, rigorous investigations). This suggests the field is maturing from a fad into a serious science.
The Redundancy Trap: Even though the number of reports tripled, the number of actual robot tests they covered didn't grow as much. It's like having 100 people write reviews of the same 10 movies. The authors warn that researchers might be repeating the same work rather than discovering new things.

2. The Performance: The "Heim" Scorecard

The authors used a special scorecard called HELM (Holistic Evaluation of Language Models) to grade the robots. Imagine grading a student not just on their test scores, but on their safety, speed, and fairness.

Accuracy (The Test Score): The robots are getting A's on practice exams (benchmarks). They can solve specific coding puzzles very well. However, the high-quality reports warn that these scores might be inflated. It's like a student who memorized the answer key but might fail if the questions change slightly. In the real world, they often make mistakes.
Robustness (The "Stress Test"): This is where the robots are fragile. If you change the wording of a request slightly, or ask them to work in a different programming language, they often break. They are like a high-performance sports car that drives great on a track but stalls on a muddy road.
Efficiency (The Gas Bill): The robots are expensive. They require massive amounts of computer power, electricity, and money to run. Using them in a small office or on a mobile phone is currently like trying to power a toaster with a nuclear reactor.
Toxicity & Bias (The Safety Hazard): The robots sometimes generate code that is insecure, leaks private passwords, or copies copyrighted material. They also tend to favor certain styles of coding over others, just because that's what they saw most often in their training data.

3. The Scenarios: Where Are They Working?

The reports mostly focus on the robots doing basic construction tasks:

Code Generation: Writing new code from scratch.
Repair: Fixing bugs or security holes.
Completion: Finishing a sentence of code the human started.

Interestingly, the reports rarely mention where this code is being used (e.g., is it for a hospital system? A video game? A car?). They also rarely mention which programming languages are being used, other than the big ones like Python and Java. It's like knowing the robots can lay bricks, but not knowing if they are building a house, a bridge, or a castle.

4. The Challenges: Why We Can't Just Flip the Switch

The authors identified three main groups of problems stopping these robots from taking over the construction site:

The Economics (The Price Tag): It costs too much to train and run these models. Small companies or individual developers can't afford the "gas bill."
The Evaluation (The Fake Grades): The tests we use to grade these robots don't match real life. They are too simple and don't account for the messy, complex reality of building software.
The Integration (The Friction): The robots don't fit well into the tools developers already use. They are slow, hard to control, and sometimes give answers that are confusing or untrustworthy. Developers don't fully trust them yet, and new developers might rely on them too much without understanding the code.

5. The Future: What's Next?

The reports suggest a clear path forward, which the authors summarize as:

Specialize the Robots: Instead of one giant robot that knows a little about everything, we need robots trained specifically on our company's code and rules.
Better Tests: We need to stop using simple practice exams and start testing the robots in realistic, messy scenarios.
Teamwork: The future isn't just the robot; it's the robot working with traditional tools and human experts (a "hybrid" approach).
Safety First: We need to fix the security and privacy leaks before we let these robots loose in the real world.

The Bottom Line

This paper concludes that LLM-based code generation is a fast-maturing technology that is currently unevenly evaluated.

It's like a new, incredibly powerful engine that has been installed in a car. The engine runs fast on the test track (benchmarks), but we aren't sure if it will handle a rainy day (real-world robustness), if it will cost a fortune in fuel (efficiency), or if it might accidentally drive off a cliff (security). The field is moving fast, but we need to slow down, build better safety standards, and figure out how to make this technology actually useful for everyday builders, not just for passing tests.

Technical Summary: A Tertiary Review of Large Language Model-Based Code Generating Tasks

Problem Statement

Large Language Models (LLMs) are increasingly applied to Code Generating Tasks (CGTs) in software engineering (SE), showing promising results in benchmarks and specific trials (e.g., GitHub Copilot, AlphaCode). However, the broader effects of these applications, their integration into real-world development workflows, and the validity of reported results remain insufficiently understood. Existing secondary studies and tertiary reviews are often too broad (covering general ML in SE), too narrow (focusing only on specific tasks like testing or prompt engineering), or lack a unified definition of CGTs. Consequently, there is a lack of consolidated evidence regarding the landscape, effects, challenges, and future directions specifically for LLM-based CGTs. This paper addresses this gap by conducting a tertiary systematic literature review (SLR) to synthesize existing secondary evidence.

Methodology

The authors followed Kitchenham's systematic review guidelines and the SEGRESS reporting standards for structured, transparent reporting. The methodology involved:

Search Strategy: A comprehensive search was conducted across five digital libraries (IEEE Xplore, ACM DL, Scopus, DBLP, Web of Science) and Google Scholar. The search utilized a PICOC-based query construction process, validated against a test set, and combined database querying with backward-and-forward snowballing.
Screening and Selection: The process included title/abstract screening and full-text review. To manage the volume of records (particularly from snowballing), the authors employed a novel semi-automated screening approach using GPT-4o as a first-pass screener, validated against human judgment.
Inclusion/Exclusion Criteria: The review focused on secondary studies (SLRs, systematic mapping studies, surveys, meta-analyses) published between 2017 and 2025 (up to June 2025) that explicitly covered LLMs applied to CGTs. Primary studies, non-LLM AI, and non-code-generation tasks (e.g., code summarization) were excluded.
Quality Assessment: Selected studies were evaluated using the DARE criteria (Database of Abstracts of Reviews of Effects), resulting in a quality distribution of 43.3% High, 26.7% Medium, and 30.0% Low quality.
Data Extraction and Synthesis: Data was extracted using a structured form covering publication metadata, AI/SE scopes, and specific evidence related to the HELM (Holistic Evaluation of Language Models) framework. The authors adapted the HELM framework (originally for natural language) to the SE context by redefining "scenarios" to include task, programming language, application type, and application domain, while mapping "measures" to SE-specific interpretations (e.g., Accuracy as functional correctness, Toxicity as security vulnerabilities).
Reliability: Inter-rater agreement was assessed using Fleiss' $\kappa$ and Cohen's $\kappa$ , yielding almost perfect agreement (0.86 for screening, 0.83 for AI screening validation, and high reliability for data extraction).

Key Contributions

Tertiary Synthesis of LLM-based CGTs: The paper identifies and synthesizes 30 secondary studies (2017–2025), quantifying the publication landscape, mapping AI and SE scopes, and analyzing trends. It provides the first consolidated view of evidence specifically for LLM-based code generation.
Novel Application of LLMs in Literature Review: The study demonstrates the reliable use of GPT-4o to improve the efficiency of the snowballing process (screening references), achieving high precision and recall compared to human screening, potentially saving significant human effort.
Adaptation of the HELM Framework: The authors adapt the HELM framework for software engineering contexts. They retain the multi-measure design but redefine scenarios to reflect SE-specific dimensions (task, language, application type/domain) consistent with SWEBOK v4 and ISO/IEC 25010, providing a structured way to evaluate LLM performance in CGTs.

Results

RQ1: Landscape of Secondary Studies

Growth and Maturity: The number of secondary studies has grown rapidly, tripling from 2023 (6 studies) to 2024 (17 studies). There is a notable shift from exploratory surveys/reviews toward more rigorous systematic literature reviews (SLRs) and systematic mapping studies (SMS), with 83% of 2025 studies being SLRs/SMSs.
Redundancy: While the number of secondary studies increased significantly, the total number of unique primary studies reviewed grew less dramatically, suggesting emerging redundancy and a maturing field where future contributions may need to focus on scope refinement rather than volume.
Scopes: The field is dominated by LLM-focused studies (21/30). In terms of SE scope, "Software Construction" is the primary focus (25 studies), followed by "Software Maintenance" (14) and "Software Security" (11). Most studies are narrow in scope (covering 1–2 SWEBOK knowledge areas).
Venues: High-quality journals (e.g., TOSEM, TSE) host the majority of peer-reviewed studies, though a significant portion (9/30) appears on arXiv, reflecting the fast-paced nature of the field.

RQ2: HELM Measures Reported

Accuracy: The most reported measure (18 studies). While many studies report high accuracy on benchmarks (e.g., HumanEval, MBPP), higher-quality studies express caution, noting that benchmark scores may be inflated by data leakage and do not always reflect real-world generalization or semantic correctness.
Robustness: Reported in 16 studies. Evidence suggests robustness is fragile; performance varies significantly across tasks, languages, and configurations. Models often fail to generalize beyond specific datasets or exhibit sensitivity to prompt perturbations.
Efficiency: Discussed in 14 studies. There is a consistent recognition of high computational, economic, and energy costs associated with training, inference, and evaluation, particularly for large models and resource-constrained environments.
Toxicity and Bias: Toxicity (8 studies) and Bias (2 studies) are under-reported but consistently identified as risks, primarily linked to training data (e.g., privacy leakage, insecure code patterns, and dataset imbalances).
Calibration and Fairness: These measures were not explicitly addressed in the reviewed secondary studies.

RQ3: HELM Scenarios Reported

Tasks: The most frequently reported tasks are Patch Generation/Repair (22 studies), Code Generation (20), Code Translation (9), and Code Completion (8).
Languages: A wide range of languages is covered, with Java, Python, C, C++, and JavaScript being the most common.
Domains: Application domains are rarely reported, and application types are mentioned in only two studies, limiting the external validity of many findings.

RQ4: Challenges

The synthesis identifies three dominant challenge clusters:

Economic Feasibility: High costs related to training, inference, and energy consumption, alongside hardware constraints.
Evaluation Validity: Current benchmarks are often unrepresentative of real-world development (narrow scope, single-language, data leakage), leading to poor generalizability.
Socio-Technical Integration: Challenges include insufficient integration with existing tools (IDEs), lack of operational controls, responsiveness issues, and the need for human oversight and safety training for developers.

Other challenges include Data & Context limitations (token limits, domain shifts), Security & Safety (vulnerabilities in generated code), and Legal/Privacy risks.

RQ5: Future Directions

The most frequently recommended future directions are:

Model Improvement & Training: Specifically, domain-aware adaptation (specialized datasets, fine-tuning) and improving efficiency and interpretability.
Benchmark & Evaluation Improvement: Calls for comprehensive, multi-metric, and cross-language evaluation frameworks that go beyond functional correctness.
Ensembling & Hybridization: Combining LLMs with other models or traditional SE methods (e.g., static analysis) to improve performance and reliability.
Integration into Practice: Moving from standalone tools to workflow-integrated solutions (e.g., RAG, IDE plugins).
Security, Privacy, and Trustworthiness: Addressing vulnerabilities and data leakage.

Significance and Claims

The paper claims to provide the first systematic tertiary synthesis of secondary evidence on LLM-based CGTs. It highlights that the field is fast-maturing yet unevenly evaluated. While LLMs show promise, the authors emphasize that:

Reported accuracy gains are often benchmark-specific and may not translate to real-world scenarios.
Robustness is fragile, and efficiency constraints are pervasive.
There is a critical need for domain-aware model improvements and holistic, standardized evaluation frameworks that address efficiency, security, and socio-technical integration.
The field is transitioning from exploratory mapping to rigorous synthesis, but risks of redundancy and limited coverage of underrepresented SE areas (like testing and management) remain.

The authors conclude that for LLM-based CGTs to be successfully integrated into real-world software engineering, future research must address economic feasibility, develop more representative benchmarks, and focus on the socio-technical aspects of human-in-the-loop integration.