CodeStylist: Supporting Early Undergraduate Programmers with Course-Aware Code Style Feedback

Imagine you are learning to write a story. You know the plot needs to make sense, the characters need to act right, and the ending needs to be satisfying. But there's another layer: your teacher also wants you to use specific fonts, indent your paragraphs just so, and avoid starting sentences with "And." If you get the plot right but the formatting wrong, you might lose points. In the world of computer programming, this "formatting" is called code style. It's not about whether the computer can run the program (that's the plot), but whether the code is neat, organized, and easy for other humans to read.

For a long time, teachers have struggled with this. They want students to learn these neatness rules, but giving feedback on them takes forever. If a teacher waits until the end of the semester to say, "Hey, you used the wrong font for your variable names," the student has already forgotten the lesson. Recently, a new type of "smart assistant" called a Large Language Model (LLM) has appeared. Think of these as super-smart robots that can read your code and chat with you about it. But here's the catch: these robots are great at general advice, but they often don't know the specific rules of your particular class. They might tell you to write like a professional in a different country, when your teacher just wants you to follow the local school rules.

This is the problem a team of researchers at Purdue University tried to solve with a new tool called CodeStylist. They wanted to see if they could build a robot that knows exactly what your specific teacher wants, so it can give you helpful tips while you are still writing your code, not after you've already turned it in. They didn't just build it; they asked a group of teachers and teaching assistants to try it out and tell them if it was actually useful or if it was just a fancy toy that would confuse students.

The Experiment: A Robot That Knows the Syllabus

The researchers built CodeStylist, a web app that acts like a personal editor for coding students. Instead of giving generic advice like "make your code cleaner," CodeStylist is configured with the specific "style guide" of a particular course. It's like having a robot that has been given the exact text of your teacher's syllabus and rubric to reference.

Here is how it works in the real world: A student uploads their code files (which might be a whole project with many different files, not just one). They tell the tool, "This is for my C programming class," and the tool sends the code to a Large Language Model (an AI). The AI reads the code and generates a report pointing to specific lines. It says things like, "Hey, on line 42, you named this variable 'x', but our class rule says variables should be descriptive, like 'totalScore'. Also, you forgot a comment here explaining what this part does."

Crucially, in the version tested in this study, the tool relied entirely on the AI to make these judgments. It did not have a separate, strict computer program (a "deterministic" checker) to double-check the facts. It asked the AI to find the rules and the mistakes all at once.

The goal wasn't to fix the code for the student or to grade them. The goal was to act as a "revision buddy"—a tool to help students catch mistakes before they hit the submit button. The researchers wanted to know: Would teachers trust this robot? Would students actually use it? And most importantly, would the robot give advice that matches what the teachers actually want?

What the Teachers Said: A Mixed Bag

To find out, the researchers gathered 18 instructional staff members (teaching assistants and one instructor) from a computer science course at Purdue. These were the people who actually read the students' code every week. They were given the tool and asked to test it with real code examples they had seen before. They then filled out a survey and shared their honest thoughts.

The results were a bit like a school play where the actors are trying their best, but the script needs some editing.

The Good News:
The teachers agreed that the idea was great. They saw that CodeStylist could make invisible rules visible. One teacher noted that it was like having a "second pair of eyes" that could spot style mistakes instantly. They believed that students would actually use the tool, with most predicting that students would check their code at least once a week. The teachers felt it was a helpful way to catch style errors before the final grade, saving everyone time in the long run.

The Bad News (and the Big Worry):
The main problem was trust. The teachers didn't fully trust the robot's answers.

• Accuracy: When asked how often the robot got things right, the teachers guessed an average of 60.7%. That means nearly 4 out of 10 times, the robot might be wrong, confusing, or giving advice that didn't match the class rules. Because the tool relied solely on the AI without a strict "spell-checker" backup, it sometimes missed things or made things up.
• Helpfulness: On a scale of 1 to 5, the teachers rated how helpful the answers were at 3.50, and how useful they were at 3.33. These are "okay" scores, but not "amazing" scores.
• Learning: The biggest concern was whether the tool would actually help students learn. The average rating for "anticipated student learning" was 2.61 out of 5. This is below the middle point. The teachers worried that if the robot gave wrong advice, students might just blindly follow it without thinking, or get confused when the robot's advice clashed with the teacher's real rules.

One teaching assistant was so skeptical they gave the tool a 4% accuracy rating and said students would "Never" use it. Another worried that if the robot gave bad advice, it would create more work for the teachers, who would then have to fix the mess the robot made.

The Verdict: A Promising Draft, Not a Finished Product

The researchers concluded that CodeStylist is a promising start, but it's not ready to be the final authority in the classroom. The teachers saw it as a useful tool for checking style and finding potential issues, but they did not see it as a tool that could reliably teach students the deep principles of good coding yet.

The paper suggests that for this tool to truly work, it needs a "hybrid" approach. Instead of letting the AI guess everything, it should use strict, unchangeable computer rules (like a spell-checker) to find the obvious errors, and then use the AI only to explain why those errors matter. The teachers also asked for better visual aids, like highlighting the exact line of code and showing the specific rule the student broke, so students could verify the advice themselves.

In short, the study suggests that while course-aware feedback is a great idea, the current version of CodeStylist is a bit too unreliable to be a teacher's best friend because it lacks that strict computer backup. It's a helpful draft that needs more work to ensure it doesn't accidentally teach students the wrong way to write code. The researchers believe that if they can fix the accuracy issues and make the feedback easier to verify, this kind of tool could become a game-changer for helping students write cleaner, more professional code.