💬 NLP

Memory-Based vs. Context-Only Conditioning Produces Distinct Behavioral Patterns in Stateful Personalization

This paper demonstrates that in teacher-facing educational recommender systems, memory-based conditioning produces distinct, history-dependent personalization behaviors that differ significantly from the question-level responsiveness of context-only conditioning, highlighting the need for behavior-level diagnostics to evaluate such effects.

Original authors: Junsoo Park, Youssef Medhat, Htet Phyo Wai, Ploy Thajchayapong, Ashok K. Goel

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Junsoo Park, Youssef Medhat, Htet Phyo Wai, Ploy Thajchayapong, Ashok K. Goel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to help a student who just asked a tricky question. You have two different ways to decide how to answer them. This paper compares those two ways to see how they change the advice a computer gives to the teacher.

Here is the breakdown of the study using simple analogies:

The Two Approaches: "The Stranger" vs. "The Old Friend"

The researchers tested two different "modes" for an AI system that helps teachers:

Context-Only Conditioning (The Stranger):
Imagine a helpful stranger walking into the room. They only know what you are saying right now. If you ask, "What is 2+2?", they tell you "4." If you ask the same question five minutes later, they still just say "4." They don't know who you are, what you struggled with yesterday, or what you are good at. They react purely to the immediate question.
- In the paper: This is the "Contextual" system. It looks only at the current student question.
Memory-Based Conditioning (The Old Friend):
Now imagine a teacher who has known the student for a whole semester. They know the student is great at math but struggles with reading instructions. If the student asks "What is 2+2?", this teacher might say, "Remember how you solved that last week? Let's try it again, but this time, read the instructions out loud first." They are using the student's history to change the answer, even though the question is the same.
- In the paper: This is the "Memory-Based" system. It looks at the current question plus a "memory file" of the student's past struggles and needs.

What Did They Find?

The researchers asked: Does giving the AI a "memory" change how it acts?

1. The "Stranger" is faster to react to the specific question.
When the AI only looks at the current question, its answers are very tightly linked to that specific question. If the question changes slightly, the answer changes immediately. The researchers measured this with a "responsiveness score," and the "Stranger" mode scored higher. It was very good at reacting to the now.

2. The "Old Friend" creates unique answers for the same question.
This is the big discovery. When the AI had access to the student's history, it gave different advice for the exact same question depending on who the student was.

Example from the paper: Two students asked the exact same True/False question about "cognitive tasks."
- Student A (who needs help staying interested) got advice to focus on the goal and why the task matters.
- Student B (who needs help with accuracy) got advice to focus on fixing mistakes and checking facts.
- The "Stranger" AI would have given both students the exact same generic advice. The "Old Friend" AI tailored the advice to the person, not just the question.

The "Measuring Tape" Problem

The researchers also found a problem with how we usually test these AI systems.

The Old Way: We often use a "similarity ruler" (embedding metrics) to see if the AI's answer matches the question. If the answer is very similar to the question, we think it's good.
The Problem: This ruler is great for the "Stranger" mode because it measures how well the AI reacts to the current question. But it fails to measure the "Old Friend" mode. The "Old Friend" might give an answer that looks less like the question on the surface (because it's adding history), but it's actually more personalized.
The Takeaway: You can't use a simple "similarity ruler" to judge personalization. You have to look at the behavior to see if the AI is treating different students differently.

Did the Teachers Like It?

The researchers showed these AI-generated suggestions to real teaching assistants (human teachers).

The teachers found the suggestions useful and easy to understand.
They felt the "Old Friend" (Memory-Based) suggestions gave them better clues on how to help specific students, making the AI feel more like a helpful partner rather than just a search engine.

The Bottom Line

This paper proves that if you want an AI to act like a personalized tutor who knows a student's history, you have to give it that history.

Without memory, the AI is just a reactive machine that answers the question in front of it.
With memory, the AI becomes a differentiating machine that changes its strategy based on who is asking.

The authors conclude that we need new ways to test these systems. We can't just ask, "Does the answer match the question?" We have to ask, "Does the AI treat Student A differently than Student B when they ask the same thing?" That is the true sign of personalization.

Technical Summary: Memory-Based vs. Context-Only Conditioning in Stateful Personalization

Problem Statement
Educational AI systems aim to support personalized learning, yet teacher-facing systems require interpretable and actionable insights to augment instructional decision-making. While Large Language Models (LLMs) have advanced through Retrieval-Augmented Generation (RAG), these approaches remain largely episodic, relying on the current query or retrieved information without persistent memory. There is a gap in understanding how shifting from purely contextual conditioning (based on the immediate student question) to memory-based conditioning (utilizing persistent learner information derived from interaction history) alters the behavioral patterns of educational recommenders. Specifically, it is unclear whether embedding-based similarity metrics, which capture responsiveness to the current input, can adequately characterize personalization grounded in accumulated learner states.

Methodology
The authors conducted a controlled comparative study using a teacher-facing recommender system deployed in graduate-level computer science courses at Georgia Institute of Technology. The study utilized anonymized interaction logs containing 8,838 student questions from approximately 200 students.

Experimental Design: Two system configurations were compared using the same underlying language model and generation procedure. The only variable was the conditioning input:
1. Contextual (Question-Conditioned): The system received only the current student question.
2. Memory-Based (Learner-Conditioned): The system received the current question plus a persistent learner state (derived from prior interactions) and a selected pedagogical tactic.
Learner State Construction: The persistent state included a "need vector" (performance, engagement, skill progression), a dominant instructional need, and a persona label. These were mapped to one of three pedagogical tactics from Hattie and Timperley's framework: Feed Up, Feed Back, or Feed Forward.
Data Sampling: For the controlled comparison, 50 student questions were randomly sampled. Both system configurations generated recommendations for these identical inputs, differing only in the conditioning context.
Metrics and Analysis:
- Deviation Correlation: Used to measure question-level responsiveness. This metric calculates the correlation between the deviations of question embeddings and recommendation embeddings from the classroom mean. Higher values indicate stronger alignment with the specific current question.
- Statistical Testing: Paired t-tests and Wilcoxon signed-rank tests were used to compare the two conditions.
- Teacher Evaluation: Five teaching fellows rated the recommendations on usefulness and usability (5-point Likert scale) and participated in semi-structured interviews to assess interpretability and actionability.

Key Results

Divergence in Responsiveness: Contextual recommendations exhibited significantly higher deviation correlation ( $\rho = 0.689$ ) compared to memory-based recommendations ( $\rho = 0.578$ ). Statistical tests confirmed this reduction was significant ( $p \ll 0.01$ ) with large effect sizes. This indicates that memory-based conditioning systematically reduces strict alignment with the current question in favor of history-dependent behavior.
Learner-Specific Differentiation: Under memory-based conditioning, the system generated distinct recommendations for different learners receiving the exact same question. For example, a learner with a dominant need for "Engagement" received a Feed Up tactic focusing on goal clarity, while a learner with a "Performance" need received a Feed Back tactic focusing on accuracy and misconception identification. In contrast, the contextual system produced identical outputs for the same input regardless of learner history.
Teacher Feedback: Teacher evaluations indicated high perceived usefulness and usability (mean ratings > 4.0). Instructors found the recommendations interpretable and actionable for identifying intervention opportunities, validating the practical relevance of the memory-based approach.

Key Contributions

Behavioral Diagnostics: The paper demonstrates that standard embedding-based similarity metrics (like deviation correlation) are insufficient for evaluating personalization grounded in learner history. These metrics favor contextual responsiveness but fail to capture stateful differentiation.
Stateful vs. Episodic Distinction: The study provides empirical evidence that conditioning on persistent learner states induces qualitatively different system behaviors compared to episodic conditioning. It establishes that memory-based personalization enables differentiation under identical inputs, a capability absent in context-only systems.
Design Variable Framework: The authors frame "conditioning context" as a primary design variable. They show that adding learner history at generation time allows for stateful personalization without modifying the underlying language model or training procedure.
Non-Identity-Based Approach: The work distinguishes memory-based personalization (derived from interaction history and behavioral signals) from identity-based personalization, offering a practical, pseudo-anonymized approach that avoids the risks of misrepresentation associated with identity attributes.

Significance and Claims
The paper claims that conditioning on persistent learner state produces qualitatively different personalization behaviors compared to contextual conditioning alone, supporting the hypothesis that memory-based systems enable learner-specific differentiation under identical inputs.

The authors explicitly state that they do not claim these behavioral shifts result in improved learning outcomes. Instead, the significance lies in the shift of system behavior: moving from "episodic responses driven by surface-level similarity metrics" to "structured behavior informed by prior interactions." This transition is analogous to the difference between shallow and deep similarity in knowledge-based AI.

The study concludes that evaluating personalization requires analyzing behavioral patterns rather than relying on a single similarity metric. It proposes behavior-level diagnostics as a necessary tool for studying how conditioning context shapes system outputs, complementing future work on representation distinctiveness. The findings motivate a broader shift in educational AI design from purely contextual responsiveness toward memory-based personalization.