Technical Summary: Cognitive Twins and the Personalized Thought Model (PTM)

1. Problem Statement

Effective personalized education requires more than just tracking performance metrics or activity logs; it demands an understanding of how learners interpret tasks, regulate effort, and apply knowledge. Existing approaches to learner modeling often rely on "black-box" predictions or superficial metrics incapable of distinguishing between knowledge deficits, misconceptions, and metacognitive difficulties. Furthermore, current knowledge-tracking models typically estimate concept mastery without explicitly representing how learners organize relationships between concepts or how they regulate their own learning processes. There is a critical need for interpretable learner models capable of representing reasoning, metacognitive regulation, and self-system motivation to provide tailored support and function as continuous learning companions.

2. Methodology

2.1 Theoretical Framework

The Personalized Thought Model (PTM) is based on Marzano's New Taxonomy of Educational Objectives, which describes three interacting systems: the Cognitive System (task execution), the Metacognitive System (goal setting and regulation), and the Self-System (motivational priorities and beliefs). This framework guides the model's hierarchical structure, progressing from concrete behavioral instances to abstract core values.

2.2 Model Architecture

The PTM organizes evidence from learner journals into a five-layer hierarchical knowledge graph:

• L0 (Behavioral Instance): Atomic 5W1H (What, When, Where, Who, Why, How) event instances directly extracted from raw text.
• L1 (Behavioral Patterns): Recurring patterns synthesized from L0 instances via semantic clustering, assigned to Marzano's Cognitive System for Understanding.
• L2 (Cognitive Utilization): Strategic routines, habit formation, and trigger responses assigned to the Cognitive System for Knowledge Utilization.
• L3 (Metacognitive):** Goal prioritization, planning logic, and decision-making reasons assigned to the Metacognitive System.
• L4 (Core Value): Fundamental beliefs, deep motivations, and problem-solving philosophies assigned to the Self-System.

2.3 Construction Pipeline

The model is constructed using a three-phase pipeline involving inference via Large Language Models (LLMs) (Gemini 2.5 Pro), sentence embedding, dimensionality reduction, and clustering:

1. Phase 1 (Data Collection & L1 Creation): Raw data from journal entries are analyzed into atomic L0 instances. These are converted into dense vectors using Sentence BERT (all-MiniLM-L6-v2), reduced to 25 dimensions via UMAP, and clustered with HDBSCAN. A weighted consensus clustering algorithm with a temporal penalty is applied to identify stable inter-day patterns, which are then synthesized into L1 nodes.
2. Phase 2 (Higher-Layer Synthesis): Analytical dimensions are generated by sampling 50 L1 nodes. These dimensions guide the grouping of L1 nodes into L2, L3, and L4 clusters via prompt-based clustering. A graph-based synthesis prompt then generates abstract interpretations for these clusters and creates the higher-layer nodes.
3. Phase 3 (Human-in-the-Loop Refinement): A HITL phase allows users to verify generated nodes through fact-checking questions. User feedback is utilized to refine node content and ensure the model remains accurate and understandable.

2.4 Evaluation Strategy

A seven-week field study was conducted with 40 Information Technology bachelor students in Indonesia. The fidelity of the PTM was evaluated through three methods:

• Automatic Evaluation: Atomic information point matching against ground-truth journal sentences to calculate precision, recall, and F1 scores.
• User Evaluation: A Likert-scale survey (1–5) assessing the perceived accuracy of synthesized statements before and after HITL refinement.
• Semantic Alignment Verification: Analysis of internal semantics (topic coherence, similarity, silhouette coefficient, topic count) and external anchoring (Jaccard similarity between model vocabulary and source journals) to verify the abstraction hierarchy.

3. Key Results

3.1 Model Fidelity and Accuracy

• Automatic Evaluation: The PTM achieved an overall F1 score of 74.57% before HITL refinement and 75.48% after refinement. Both values exceed the 0.70 benchmark for acceptable accuracy in complex information extraction. The improvement was primarily driven by a reduction in false positives (hallucinations).
• Layer Performance: All layers maintained F1 scores above 70%. Lower layers (L1, L2) showed modest improvements after refinement, while higher layers (L3, L4) exhibited minimal changes, likely due to the abstract nature of the constructs, which makes precise verification difficult for users.

3.2 User Perception

• Acceptance: Users rated the model highly, with average scores of 4.26 (pre-HITL) and 4.30 (post-HITL) on a 5-point scale.
• Divergence: User ratings were consistently approximately 7% higher than the automatic F1 scores. This suggests that users prioritize semantic "core messages" and thematic alignment over literal matching, whereas automatic metrics penalize lexical variations more strictly.
• Qualitative Feedback: Participants noted that while higher-level terms were sometimes more academic than their own language, the core message felt accurate and offered "cleaner explanations" of their habits.

3.3 Semantic Hierarchy and Abstraction

• Internal Semantics: Topic coherence increased from 0.436 (L1) to 0.626 (L4), indicating that higher layers form more coherent and interpretable thematic units.
• External Anchoring: A clear inverse relationship was observed between semantic coherence and vocabulary overlap. Jaccard similarity decreased from 0.114 (L1) to 0.007 (L4). This confirms that the model successfully performs semantic abstraction rather than simple extractive summarization, transitioning from direct event extraction to conceptual representation.
• Structure: Node distribution followed a hierarchical pattern (approximately a 4:1 ratio between L1 and L4), and the density of connections between layers showed that higher-layer nodes were supported by multiple lower-level observations (average in-degree ~3.06).

4. Key Contributions

• Interpretable Cognitive Twin: The article presents the PTM as a "cognitive twin" that models a learner's internal thinking, planning, and motivation, not just physical or performance data. It serves as a verifiable, hierarchical representation of learning processes.
• Hierarchical Architecture: It operationalizes Marzano's Taxonomy in a computational pipeline that successfully transitions from atomic behavioral data to abstract self-system values.
• Validation of Abstraction: The study provides empirical evidence that AI-generated learner models can achieve a valid semantic hierarchy, where higher layers become more coherent and less reliant on source vocabulary without losing perceived relevance by users.
• HITL Dynamics: It demonstrates that while HITL refinement improves model accuracy (particularly by reducing hallucinations), its effectiveness depends on the depth and complexity of user feedback.

5. Implications and Claims

The article claims that the PTM successfully addresses the "black-box" problem in educational AI by providing a verifiable, interpretable representation of learner thinking. The results suggest that:

1. Fidelity is achievable: AI can construct personalized thought models with acceptable factual fidelity (F1 > 75%) from unstructured journal data.
2. Abstraction is valid: The model successfully abstracts user data into higher-order cognitive and self-system concepts, evidenced by increasing coherence and decreasing lexical overlap.
3. User trust is established: Learners perceive these synthesized profiles as accurate reflections of their thinking, even when the language is more formal than their own.
4. Foundation for lifelong learning: By functioning as a persistent digital counterpart that continuously adapts, the PTM lays the groundwork for "lifelong learning"—a systematic approach to capturing and perpetuating human wisdom through digital repositories, transcending the natural limits of human knowledge exchange.

The authors maintain a modest stance and acknowledge limitations, such as the homogeneous demographics of the study (Indonesian IT bachelor students) and the short duration of the study, which may not capture long-term shifts in self-system beliefs. They conclude that while the PTM is a robust foundation for personalized support, future work must address interface design to better support user refinement of abstract concepts and validate the model across diverse populations.