RECAP: An End-to-End Platform for Capturing, Replaying, and Analyzing AI-Assisted Programming Interactions
The paper introduces RECAP, an open-source platform that passively captures and unifies AI chat sessions with fine-grained code edits to enable interactive replay and advanced analysis of developer-AI interaction patterns, as validated by a deployment involving 41 students.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to understand how a student learns to cook by watching them use a smart kitchen assistant. If you only look at the final dish (the code they submitted) or just read the text messages they sent to the assistant (the chat logs), you miss the whole story. You don't see the ingredients they tried and threw away, the moments they ignored the assistant's advice, or how their cooking strategy changed over time.
RECAP is a new tool designed to fix this blind spot. It acts like a "black box" recorder for programmers using AI coding assistants (like GitHub Copilot) inside their code editor (VS Code).
Here is how it works, broken down into simple parts:
1. The "Shadow" Recorder (The Camera)
Think of the programmer's computer as a busy kitchen. Usually, a programmer only saves their work when they hit "commit" (like taking a photo of the dish every hour). This misses all the chopping, tasting, and burning that happened in between.
RECAP installs a tiny, invisible extension called the Copilot Interaction Archiver.
- It watches the chat: It records every question the student asks the AI and every answer the AI gives.
- It watches the code: It creates a "shadow" version of the code that saves a snapshot every time the student hits save, or even when they are just typing and haven't saved yet.
- The Magic Link: It connects the two. It knows that at 2:00 PM, the student asked the AI for a specific fix, and at 2:02 PM, that exact fix appeared in the code. It links the "prompt" to the "result."
2. The Time-Travel Viewer (The Replay)
Once the data is collected, researchers can use the Session Replay Viewer. This is like a video game replay feature, but for coding.
- The Timeline: Instead of just seeing a list of files, you see a single, colorful timeline.
- Blue dots are questions asked to the AI.
- Yellow dots are code the AI suggested.
- Green dots are code the human typed themselves.
- The Walkthrough: A researcher can click play and watch the student's entire session unfold. They can see: "Ah, here the student asked the AI to fix a bug, the AI suggested a solution, the student rejected it, tried it themselves, failed, and then went back to the AI."
3. What Did They Find? (The Detective Work)
The researchers tested this tool in a university class with 41 students working on a two-week project. They didn't just count how many times students used AI; they used the replay to spot interesting patterns that would have been invisible otherwise:
- The "Stuck" Loop: One student spent 11 minutes stuck in a loop. They pasted an error message, the AI fixed it, a new error appeared, they pasted that, and the cycle repeated. The replay showed this circular struggle clearly, whereas a final code file would just look like a finished project.
- The "Frankenstein" Approach: One student used one AI (ChatGPT) to come up with a big plan, then pasted that plan into another AI (Copilot) to write the code. The replay linked these two different tools, showing how students are mixing and matching AI assistants.
- Learning to Drive: As the project went on, students relied less on the AI to write the actual code. They used the AI more for explaining concepts and planning, suggesting they were learning to drive the car themselves rather than letting the AI steer.
Why This Matters
Before RECAP, researchers were like detectives trying to solve a crime by only looking at the crime scene (the final code) or the suspect's diary (the chat logs), but never seeing the actual event.
RECAP provides the full movie. It allows educators and researchers to see exactly how humans and AI work together over long periods, revealing the messy, trial-and-error process that leads to the final result.
Important Note: The paper emphasizes that this tool is currently designed for VS Code and GitHub Copilot. It is an open-source tool for researchers and teachers to study how people learn and work, not a tool to grade students or a medical device. The data collected is strictly anonymized and protected to ensure student privacy.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.