💬 NLP

Moving Beyond Review: Applying Language Models to Planning and Translation in Reflection

This paper introduces Pensée, a novel AI tool that applies the Cognitive Process Theory of writing to support learners during the planning and translation stages of reflective writing, demonstrating through a controlled experiment that such theory-driven scaffolding significantly enhances reflection depth and structural quality compared to traditional feedback-focused approaches.

Original authors: Seyed Parsa Neshaei, Richard Lee Davis, Tanja Käser

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Seyed Parsa Neshaei, Richard Lee Davis, Tanja Käser

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to write a story about a difficult day at work. You want to do more than just list what happened; you want to figure out why it happened, what you learned, and how you'll do better next time. This is called reflective writing. It's a superpower for learning, but most people struggle to do it well. They get stuck in "just the facts" mode and miss the deep thinking part.

For a long time, researchers have tried to fix this with AI. Usually, the AI acts like a strict editor who waits until you finish your essay, then points out your mistakes. But this paper asks: What if we used AI not just to grade the final product, but to help you build the house from the foundation up?

Here is the story of their experiment, explained simply.

The Problem: The "Blank Page" Panic

When students try to reflect, they often feel overwhelmed. They have a messy pile of thoughts in their heads but don't know how to organize them into a clear story.

The Old Way: You sit down, stare at a blank page, and try to force your thoughts out.
The New Idea: What if an AI could help you plan (organize your thoughts) and translate (turn those thoughts into words) before you even start writing the final draft?

The Solution: Meet "Pensée"

The researchers built a tool called Pensée (French for "thought"). They based it on a theory called Cognitive Process Theory, which says writing happens in three steps:

Planning: Figuring out what to say.
Translation: Turning those ideas into sentences.
Reviewing: Checking your work.

Pensée uses a chatbot (an AI conversational agent) to guide students through these steps:

The Planner (Planning): Instead of a blank page, the student talks to the AI. The AI asks smart questions like, "Why did that situation go wrong?" or "What skills did you use?" As the student answers, the AI automatically pulls out the "Key Concepts" (the main ideas) and puts them in a sidebar, like sticky notes on a wall.
The Translator (Translation): When the student starts writing, those "sticky notes" are still there. They act as a safety net, reminding the student of their best ideas so they don't forget them while typing.
The Reviewer (Reviewing): Once the essay is done, the AI highlights parts of the text to show, "Great job, you included an analysis here!" or "Oops, you missed a conclusion."

The Experiment: The "Robot" vs. The "Worksheet"

To see if the fancy AI chatbot was actually better than just having a list of questions, they tested it on 93 vocational students (future medical assistants).

Group A (The AI Team): Used Pensée with the chatbot that talked to them and extracted key ideas.
Group B (The Control Team): Used a version of Pensée that looked the same but had no AI. Instead of a chatbot, they just saw static text boxes with the same questions. Instead of automatic "sticky notes," they just saw their own typed answers.

The Big Question: Did the talking, thinking AI help them write better reflections than the simple, non-talking worksheet?

The Results: A Surprise Twist

Here is what happened:

Both Groups Got Much Better: When using the tool, both groups wrote significantly deeper and better-structured reflections than they did before. The act of having a structured process (planning first, then writing, then checking) worked wonders for everyone.
The AI Didn't Win the Race: Surprisingly, the group with the fancy talking AI did not perform better than the group with the simple worksheet. Their scores were almost identical.
The "Short-Term" Effect: When the students were tested a week later without any tool, the improvements faded. They didn't seem to have permanently "learned" how to reflect on their own yet.

Why Did the AI Fail to Shine? (The "Robot Butler" Analogy)

The researchers found a funny reason why the AI didn't beat the worksheet. They looked at how students actually used the chatbot.

Imagine you hire a Robot Butler to help you cook dinner. You expect it to be interactive, maybe suggesting recipes or chatting about ingredients. But in this study, the students treated the Robot Butler exactly like a static recipe card.

They answered the questions, but they didn't ask follow-up questions.
They didn't have deep conversations.
They just treated the chatbot like a digital form to fill out.

Because the students didn't "talk" to the AI in a deep way, the AI couldn't do its magic. It was like having a Ferrari but driving it in a 20 mph school zone; the engine was powerful, but the driver wasn't using it.

The Takeaway

This paper teaches us three important lessons:

Structure is King: The biggest help wasn't the AI itself, but the structure it forced students to follow. Breaking writing down into "Plan, Write, Review" helps everyone, whether they use a robot or a pencil.
AI Needs a Different Job: If we want AI to be a true learning partner, we can't just make it ask questions like a worksheet. We need to design it to encourage real conversation, like a coach who challenges your thinking, not just a form-filler.
Habits Take Time: One session with a tool makes you perform better while you have the tool. But to actually learn the skill so you can do it alone later, you need more practice and time.

In short: The researchers built a cool new AI tool that helps students think better. It worked great at the moment, but the "magic" of the AI wasn't the secret sauce—the secret sauce was simply giving students a clear map to follow. To make AI truly special, we need to teach students how to have real conversations with it, not just fill out digital forms.

1. Problem Statement

Reflective writing is a critical pedagogical tool for developing metacognitive skills, yet learners often struggle to move beyond simple descriptive recounting to deep, analytical reflection. While Large Language Models (LLMs) have shown promise in providing feedback on finished texts, their application has been limited in two key areas:

Scope: Most existing AI tools focus on the "reviewing" phase (feedback on the final draft) rather than the earlier cognitive stages of writing.
Theory: Current implementations often lack grounding in established writing theories, treating reflection as a single activity rather than a multi-stage cognitive process.

The authors argue that to improve reflective writing quality, AI support must explicitly target the planning and translation stages of the writing process, not just the review stage.

2. Methodology

Theoretical Framework

The study is grounded in the Cognitive Process Theory (CPT) of writing, which conceptualizes writing as an interplay of three stages:

Planning: Generating ideas, setting goals, and organizing thoughts.
Translation: Converting those ideas into written text.
Reviewing: Evaluating and revising the text.

System Design: Pensée

The authors developed Pensée, an interactive tool designed to scaffold these three CPT stages using LLMs:

Planning Phase (Conversational Agent): An LLM-based chatbot (built on GPT-4o) guides learners through a structured dialogue based on a four-state cycle (Metacognition, Connection, Organization, Metacognition). It asks probing questions to help learners externalize and organize their thoughts before writing.
Translation Phase (Key Concept Extraction): As the conversation progresses, an LLM agent automatically extracts "key concepts" and relevant quotes from the dialogue. These are displayed in a persistent sidebar, acting as a dynamic planning board to help learners translate their planned ideas into coherent text without relying solely on memory.
Reviewing Phase (Structural Feedback): Upon drafting, an LLM agent classifies text excerpts against the six components of the Gibbs Reflective Cycle (Description, Feelings, Evaluation, Analysis, Conclusion, Action Plan). It provides visual feedback (highlighting and checkmarks) indicating which components are present or missing.

Experimental Design

Study Type: Controlled between-subjects experiment with $N=93$ vocational students (medical assistance training, ages 15–19).
Conditions:
- Treatment Group (TG): Used Pensée with full AI support (Conversational Agent for planning + Auto-extraction for translation + AI feedback for review).
- Control Group (CG): Used a version of Pensée where the AI conversational agent was replaced with static text boxes containing the same questions, and the key concept extraction was replaced with a static display of the user's own Q&A. Both groups received the same AI feedback in the reviewing phase to isolate the effects of the planning/translation support.
Procedure: Three sessions over several weeks:
1. Pre-intervention: Baseline reflection task (no tool support).
2. Learning Intervention: Participants used their assigned tool version to reflect on a workplace situation.
3. Delayed Post-test: Participants wrote a reflection on a new topic without any tool support (to measure transfer of learning).
Metrics:
- Depth: Scored based on Metacognition, Connection, and Organization (based on a rubric adapted from Glogger et al.).
- Structure: Scored based on the presence of the 6 Gibbs Reflective Cycle components (0–6 points).
- Behavior: Word counts, interaction patterns, and follow-up question usage.
- Perception: Self-reported metrics (usefulness, ease of use, excitement, etc.) via Likert scales.

3. Key Contributions

First CPT-Oriented LLM Application: This is the first study to explicitly apply LLMs to the planning and translation stages of reflective writing, moving beyond the dominant "feedback-on-draft" paradigm.
Pensée Tool: A novel architecture that integrates a state-machine-driven conversational agent with automatic key-concept extraction to bridge the gap between ideation and writing.
Empirical Evidence on AI vs. Static Scaffolding: The study provides rigorous data comparing AI-driven adaptive support against static, non-AI scaffolding within the same theoretical framework.

4. Results

RQ1: Effects on Reflection Depth and Structure

Immediate Impact: Both the Treatment (AI) and Control (Static) groups showed significant improvements in reflection depth and structural quality during the tool usage session compared to their pre-intervention baseline.
- Depth: Scores increased significantly for Metacognition and Connection.
- Structure: Scores improved significantly (from ~2.1 to ~4.5 out of 6).
AI vs. Static: There was no statistically significant difference between the AI-supported group and the static scaffolding group. The explicit CPT support (the structure of the questions and the organization of the interface) drove the improvement, not the AI adaptivity itself.
Delayed Transfer: In the delayed post-test (without tool support), the gains diminished. While the Control group showed a slight trend toward improvement, the Treatment group's scores reverted closer to baseline. This suggests the scaffolding boosted performance during use but did not result in robust, long-term skill transfer in this short-term intervention.

RQ2: Interaction Behavior

Passive Usage: Contrary to the hypothesis that AI would encourage deeper dialogue, students in the Treatment group largely treated the Conversational Agent like a static worksheet.
Response Length: There was no significant difference in word count between groups. Both groups showed a sharp decline in response length over the course of the conversation.
Follow-up Questions: Only 7% of users in the AI group asked follow-up questions. The CA's design (ending every turn with a question) inadvertently mimicked the static questionnaire format, limiting genuine conversational engagement.

RQ3: User Perception

Positive Trends: The AI group consistently rated the system higher on ease of use, excitement, usefulness, and willingness to reuse compared to the static group.
Statistical Significance: These differences were not statistically significant after correction for multiple comparisons, likely due to the sample size ( $N=93$ ). However, the consistent trend suggests a potential advantage in user experience.

5. Significance and Implications

Theory-Driven Design is Crucial: The study demonstrates that grounding AI tools in cognitive theories (like CPT) is highly effective for improving writing quality. The structure of the support matters more than the "AI-ness" of the support in the short term.
Limitations of Current LLM CAs: The findings highlight a gap between the potential of conversational agents and their actual usage. Users tend to treat them as static forms unless the interface explicitly encourages richer interaction (e.g., via follow-up buttons or open-ended prompts).
Evaluation Methodology: The lack of transfer to the delayed post-test underscores a common issue in writing interventions: short-term scaffolding improves performance but may not build lasting skills. The authors advocate for delayed post-tests as a standard evaluation metric to distinguish between performance boosts and genuine learning.
Future Directions: To unlock the full potential of LLMs in education, future designs must move beyond "question-answer" loops to foster genuine conversational engagement and focus on longitudinal studies to measure sustained learning gains.

In conclusion, while Pensée successfully demonstrated that explicit process-oriented scaffolding improves reflective writing, the study serves as a reality check: LLMs are powerful tools, but they are not magic. Their educational value depends on careful, theory-driven design and rigorous evaluation of long-term impact.