Structured AI Demonstrations and Student LLM Use in Engineering Mechanics: Study Design and Preliminary Results
This paper presents a study design and preliminary findings from a Spring 2026 engineering mechanics course that introduces a structured framework of nine instructor-led AI demonstrations and a reproducible survey instrument to empirically investigate student LLM usage patterns, attitudes, and their impact on academic performance.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine the classroom as a giant, bustling workshop where students are learning to build complex machines. For decades, the tools in this workshop were textbooks, calculators, and a teacher who walked around offering hints. But recently, a new kind of "super-assistant" has arrived: Large Language Models (LLMs). Think of these as incredibly fast, chatty robots that can read almost any book and write almost any sentence. They are like a genie that can answer your homework questions instantly, but with a catch: sometimes the genie gets the math wrong, or it makes up facts that sound real but aren't.
The big question for teachers right now is: Do we ban this genie, or do we teach students how to work with it without letting it do the thinking for them? It's a bit like teaching someone to drive. You don't want them to just sit in the passenger seat while the car drives itself, but you also don't want them to crash because they don't know how the brakes work. This paper is a report from a group of engineers who decided to stop guessing and start watching. They set up a special experiment in a college engineering class to see exactly how students are using these AI tools, whether they are checking the work, and if showing them how to use the tools changes the way they learn.
The Experiment: A Semester of AI Watch
In the spring of 2026, researchers at Boston University decided to peek behind the curtain of a tough engineering class called "Engineering Mechanics." This is the class where students learn how to keep bridges from falling down and buildings from tipping over. It's heavy on math, diagrams, and logic. The researchers wanted to know three things: How are students actually using AI? Does using it help or hurt their grades? And, if teachers give a series of short, structured lessons on how to use AI wisely, does that change the students' behavior?
To find out, they split the class into two groups. One group (the "Intervention" group) got nine special, 15-minute mini-lessons throughout the semester. These weren't just lectures; they were live demonstrations where the teacher showed the students how to use AI as a "Socratic tutor" (a robot that asks questions instead of giving answers), how to use it to build computer programs, and how to test if the AI was lying. The other group just went about their normal business.
The researchers then asked the students to fill out surveys at the beginning and end of the semester, asking them to be honest about how often they used AI, what they thought of it, and how they checked if the AI was right. They also looked at the students' grades on midterms, which were taken in class without phones or internet, to see if the AI habits matched up with how well they understood the material.
What They Found: The Shift from "Unverified Use" to "Checking"
The story the data tells is a bit like watching a group of kids figure out a new video game. At the start of the semester, about one-third of the students said they had never used AI for this specific class. By the end, that number dropped to almost zero. Everyone was using it.
But the way they used it changed.
- The "Self-Attempt" Shift: At the beginning, most students said they would try to solve a problem themselves first, and then use AI to check their answer or get unstuck. By the end of the semester, this was still the most popular method, but a new trend appeared: more students started using AI to guide them step-by-step while they were working, rather than just at the end.
- The "Trust" Issue: Even though students were using AI more, they didn't blindly trust it. In fact, the data suggests they became more skeptical. More students reported that they were actively checking the AI's work against textbooks or re-solving the problem themselves to make sure the robot wasn't hallucinating.
- The "Grades" Mystery: Here is where it gets tricky. The researchers found a pattern: students who were already struggling (the bottom 25% of the class) were the ones using AI the most for this specific class right from the start. The top students used it the least. However, the paper is very careful to say this doesn't prove that AI caused the low grades. It's just as likely that the struggling students were turning to AI because they needed help, or that relying on AI too much prevented them from practicing enough to get better. The study can't tell which way the cause-and-effect arrow points.
The Lessons: Did the Teacher's Demos Help?
The nine mini-lessons were designed to show students how to be "smart users" of AI. The results were a mixed bag, like a concert where half the audience loved the show and the other half thought it was a waste of time.
- The Good: Many students said the lessons helped them understand how to use AI for their specific projects, like building a truss bridge. They liked seeing how to make the AI act like a tutor that asks questions instead of just giving answers.
- The Bad: A significant number of students felt the lessons were "redundant" (they already knew the stuff) or "lacked utility" (they didn't see how it helped with their homework). Some felt the timing was off; for example, a lesson on building computer interfaces was given before the students fully understood the math behind the project, so it felt confusing.
- The Verdict: The lessons didn't magically make everyone use AI perfectly. Instead, they highlighted that students have very different starting points. Some were ready to use AI as a powerful tool, while others were still figuring out the basics. The researchers noted that for these lessons to work better in the future, they need to be tied much more closely to the actual homework and exams students are doing that week.
The Big Picture: A Work in Progress
The most important takeaway from this paper isn't a final answer, but a new way of looking at the problem. The researchers found that AI is no longer a "maybe" in engineering class; it's a "definitely." Students are using it, and they are using it in complex ways—sometimes to learn, sometimes to bypass independent work, and often somewhere in between.
The study suggests that simply banning AI is impossible because it's too embedded in how students work. But simply letting them run wild is risky because they might not learn the core skills. The "middle path" seems to be teaching students how to verify the AI's work and how to use it as a partner rather than a replacement.
However, the authors are very humble about their findings. They admit that their study was small (only about 100 students) and that they couldn't prove that the lessons caused any specific change. They describe their work as a "preliminary snapshot"—a first look at a rapidly moving target. As the AI tools get smarter and the students get more used to them, the rules of the game will keep changing. The paper's main goal is to provide a toolkit (the surveys and the lesson plans) so that other teachers can run their own experiments and help figure out the best way to teach engineering in the age of the robot genie.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.