💻 computer science

Teaching Prompt-Based Programming with LLMs: A 45-Minute Lesson with Guided Practice for End-User Programmers

This paper presents a randomized controlled study demonstrating that a brief, 45-minute guided lesson on prompt-based programming significantly improves engineering students' self-efficacy and shows a modest, though not statistically significant, improvement in their ability to specify computational goals to large language models compared to traditional code tracing activities.

Original authors: Keith Tran, Samiha Marwan, Thomas Price

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Keith Tran, Samiha Marwan, Thomas Price

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very talented, but slightly literal-minded robot assistant. You want it to bake a cake, but instead of giving it a recipe, you have to describe exactly what you want in plain English. If you say, "Make a cake," the robot might give you a brick of flour. If you say, "Mix two cups of flour, three eggs, and bake at 350 degrees for 30 minutes," you get a cake.

This is the core idea of Prompt-Based Programming. Instead of writing complex computer code (the "recipe"), you write natural language instructions (the "prompt") for an AI to write the code for you.

The paper by Keith Tran, Samiha Marwan, and Thomas Price asks a simple question: Can we teach people how to talk to this robot assistant effectively in just one class period (45 minutes)?

Here is a breakdown of their study using everyday analogies:

The Problem: The "Vague Order" Issue

Many people, especially those who aren't computer experts, try to use AI to write code. But they often struggle to be specific. It's like going to a restaurant and saying, "I want something good," and getting confused when the waiter brings you a salad instead of a burger. The AI isn't broken; the instructions were just too vague.

The Experiment: A Race Between Two Lessons

The researchers gathered 55 engineering students (who had very little coding experience) and split them into two groups for a 45-minute session.

Group A (The Experimental Group): Learned how to talk to the AI.
- The Lesson: They were taught a specific "template" for giving orders. Instead of just saying "do this," they learned to say: "Here is the input (the ingredients), here is the output (the finished dish), and here is exactly how to change the ingredients into the dish." They practiced this with three different tasks.
- The Analogy: They learned the art of writing a perfect recipe card for the robot chef.
Group B (The Control Group): Learned how to read code.
- The Lesson: They spent the same 45 minutes looking at existing code and tracing through it step-by-step to see what it does. This is what a normal computer class usually does.
- The Analogy: They learned how to read a recipe card that someone else already wrote, but they didn't learn how to write their own instructions for the robot.

The Test: The "Promptly" Challenge

Before and after the lesson, both groups took a test. They were given a puzzle (like "filter out all the numbers bigger than 10") and had to write a prompt for an AI to solve it. They couldn't just copy the puzzle description; they had to translate it into a clear instruction for the AI.

The Results: Confidence vs. Skill

Here is what happened:

Confidence (Self-Efficacy): Group A (the prompt learners) felt much more confident in their ability to talk to the AI after the lesson. Their confidence score jumped significantly higher than Group B's. It's like Group A felt, "I know how to order at this restaurant now!"
Actual Skill (Task Performance): Group A did get better at solving the puzzles, but the improvement was small and not statistically "proven" to be a huge win. They solved about 1 more problem out of 8 on average compared to Group B.
- The Takeaway: The lesson made them feel like they could do it, and they did slightly better, but they didn't become masters in just 45 minutes. It's like learning the basics of driving in one day: you feel ready to start the car, but you aren't ready for a cross-country road trip yet.

Why This Matters

The researchers found that a short, focused lesson can help "end-user programmers" (people who need code to solve a specific problem in their job, but don't want to be professional coders) communicate better with AI.

The Good News: You don't need a whole semester to learn the basics of talking to AI. A single class period can boost confidence and give a slight edge in performance.
The Catch: 45 minutes isn't enough to make someone an expert. Just like you can't learn to be a master chef in one hour, you can't master prompt engineering in one class. The students needed more practice to truly get fluent.

The Bottom Line

The paper suggests that teaching people how to ask the AI for help is a valuable skill that can be squeezed into a busy schedule. It acts as a bridge, helping non-coders get started with AI tools. However, to really get good at it, you need more than just a quick lesson; you need time to practice and make mistakes.

In short: Teaching people how to write clear instructions for AI works well enough to make them feel capable and slightly more effective, but true mastery takes more time than a single class period.

Technical Summary: Teaching Prompt-Based Programming with LLMs

Problem Statement
The emergence of Large Language Models (LLMs) has introduced "prompt-based programming," a modality where users express computational goals in natural language rather than writing traditional code. While this lowers barriers to entry for end-user programmers (non-CS majors or those using code as a means to an end), it does not eliminate the need for foundational computational thinking skills. Research indicates that learners, particularly novices, struggle to communicate intent clearly to LLMs, often producing vague or underspecified prompts, lacking debugging strategies, and conflating technical vocabulary with colloquial terms. A critical gap exists in understanding how to integrate explicit instruction on prompting into existing, time-constrained curricula. Specifically, it is unclear how much instructional time is required to produce meaningful gains in prompting ability and self-efficacy.

Methodology
The authors conducted a randomized controlled study (RCT) with 55 first-year engineering students at a university. The study compared a 45-minute experimental intervention against a "business-as-usual" control condition of equal length.

Experimental Condition: A 45-minute lesson on prompt-based programming paired with guided practice. The lesson utilized a "backward design" approach, focusing on five learning objectives derived from the "Gulf of Envisioning" framework: specifying function structure, describing data transformation (mapping, filtering, reducing), expressing conditional logic, handling edge cases, and debugging failed prompts. Students were taught a reusable prompt template and practiced on three Promptly problems (tasks requiring learners to infer function behavior from input/output pairs and prompt an LLM to solve them).
Control Condition: A 45-minute lesson on code tracing (a standard introductory CS lab activity). This group received instruction on Python syntax, control flow, and code tracing using Python Tutor, practicing on three equivalent problems focused on reading and understanding code rather than writing prompts.
Design: Both groups completed pre- and post-tests consisting of 8 Promptly problems and surveys measuring prior programming experience and prompting self-efficacy. The study used linear mixed-effects models to analyze data, accounting for repeated measures and session-level clustering.

Key Contributions

A Lightweight Intervention: The authors present a publicly available, 45-minute lesson plan with guided practice designed to teach CS-specific prompting strategies (e.g., structured templates, data transformation patterns) without requiring prior CS background.
Empirical Evidence on Short Interventions: The study provides controlled experimental data on the efficacy of a brief intervention. It demonstrates that a single 45-minute session can significantly increase learners' self-efficacy in prompting and produce meaningful, though statistically modest, improvements in task performance compared to standard code-tracing activities.

Results

Task Performance (RQ1): The experimental group showed greater average gains in solving Promptly problems from pre- to post-test compared to the control group (+10.8 percentage points vs. +1.1 percentage points). However, this difference was not statistically significant at the $\alpha=0.05$ level ( $p=0.056$ ). The effect size was comparable to the impact of one full standard deviation of prior programming experience.
Prompting Self-Efficacy (RQ2): The experimental group demonstrated a statistically significant greater increase in prompting self-efficacy compared to the control group (+35.4 percentage points vs. +21.9 percentage points; $p=0.029$ ).
Predictors: Prior programming experience was a significant predictor of task performance in both conditions.

Significance and Claims
The paper claims that a brief, targeted intervention is likely sufficient to improve learners' ability to specify computational goals to LLMs and significantly boosts their confidence in doing so. The authors position prompt-based programming not as a replacement for traditional coding instruction, but as a complementary pathway that may offer a more accessible entry point for end-user programmers intimidated by syntax.

However, the authors maintain a modest tone regarding the magnitude of these gains. They acknowledge that while the intervention produced meaningful improvements, the effect on task performance was small (less than one additional problem solved on average), suggesting that 45 minutes is likely insufficient for learners to achieve fluency or mastery. The results indicate that prompting skills require more time and practice to develop fully. The study concludes that while short interventions can be integrated into existing courses without substantial disruption, future research is needed to determine how to extend these gains to complex, open-ended, multi-step programming tasks.