Transition Matrix Analysis Analyzing Students Use of Cognitive Resources in Physics

Technical Summary: Transition Matrix Analysis of Students' Cognitive Resources in Physics

Problem Statement
Traditional analysis of conceptual surveys in physics education, such as the Force Concept Inventory (FCI) or the Determining and Interpreting Resistive Electric Circuit Concepts Test (DIRECT), typically focuses on pre- and post-test performance gains to evaluate pedagogical interventions. This approach aggregates data into correct/incorrect metrics, thereby obscuring the specific incorrect options selected by students. Consequently, it fails to reveal the stability, coherency, and context-dependency of students' underlying knowledge structures. While the "misconception" framework views student errors as stable alternative theories, the "resources framework" posits that student knowledge is fragmented, consisting of small-grained cognitive resources activated inconsistently depending on the context. The authors argue that focusing solely on correct answers ignores the incorrect options which could inform the stability of these cognitive resources.

Methodology
The authors propose a Transition Matrix Analysis (TMA) guided by the resources framework to analyze data from conceptual surveys. The methodology involves the following steps:

1. Identification of Resources-Equivalent Questions: A subset of questions is selected from a validated survey. These questions must share the same underlying physics principle but differ in context. Crucially, the set of answer choices across these questions must correspond to the same set of cognitive resources.
2. Mapping Choices to Resources: Each multiple-choice option is mapped to a specific cognitive resource. In this study, the authors mapped choices from three DIRECT questions (Q3, Q7, Q12) regarding DC circuit power to four resources:
   • $S$: Scientifically productive resource.
   • $R_1$: Constant current source resource (battery supplies constant current regardless of circuit arrangement).
   • $R_2$: Superposition resource (more batteries always equal more brightness/power).
   • $Null$: Resources used by very few students (to maintain vector completeness).
3. State Vector Construction: For a student $k$ answering $m$ questions, a probability distribution vector $\vec{Q}_k$ is defined, where elements $\rho_i$ represent the probability of activating resource $i$ ($\rho_i = n_i/m$).
4. Transition Matrix Calculation: The change in resource activation from a pre-test (initial state $\vec{Q}_i$) to a post-test (final state $\vec{Q}_f$) is modeled by a transition matrix $T$ such that $\vec{Q}_f = T\vec{Q}_i$. The matrix element $T_{ij}$ represents the conditional probability that a student who used resource $j$ on the pre-test switches to resource $i$ on the post-test.
5. Aggregation: Individual student matrices are averaged to generate a class-level transition matrix, allowing researchers to track average shifts in resource activation across the cohort.

Experimental Application
The method was applied to data from $N=135$ elementary education majors enrolled in a conceptual physics course. The study utilized the DIRECT survey administered as both an individual and a group pre-test and post-test, separated by a 7-week instructional unit. The analysis focused specifically on the transition between the individual session and the group session for both pre- and post-tests.

Key Results
Analysis of the transition matrices revealed several critical insights that were not apparent from overall performance scores:

• Overall Performance vs. Resource Stability: While a paired samples t-test showed a significant improvement in overall class scores from pre-test to post-test, the TMA revealed that students did not necessarily solidify their understanding of the specific concept (DC circuit power).
• Group Work Effects: In both pre- and post-tests, students frequently shifted from the scientifically productive resource ($S$) to unproductive resources ($R_1$ and $R_2$) during group sessions.
• Persistence of Unproductive Resources: The diagonal elements of the transition matrix ($T_{R1R1}$ and $T_{R2R2}$) indicated that students retained their unproductive resources at high rates.
• Worsening Trends: Comparing pre-test and post-test transition matrices via t-tests showed that in the post-test group sessions, students retained unproductive resources ($R_1$ and $R_2$) significantly more often than in the pre-test, and retained the productive resource ($S$) significantly less often. Specifically, the transition $T_{SS}$ decreased significantly, while $T_{R1R1}$ and $T_{R2R2}$ increased significantly.
• Context Dependence: The data supports the view that resource activation is context-dependent; students did not consistently apply the correct reasoning across the resource-equivalent questions, and group interaction sometimes reinforced incorrect reasoning.

Significance and Claims
The paper claims that Transition Matrix Analysis offers a more nuanced perspective than traditional gain analysis by exposing the dynamics of cognitive resource activation.

• Diagnostic Value: TMA can detect when instructional interventions or group work fail to stabilize productive resources, even when overall test scores improve. In this study, it highlighted that group work may inadvertently reinforce unproductive resources ($R_1$ and $R_2$) regarding circuit power.
• Proof of Concept: The study serves as a proof of concept for applying mathematical modeling (specifically transition matrices) to physics education research to describe learning processes.
• Future Directions: The authors modestly suggest that if this approach proves fruitful, it could motivate experts to revisit existing conceptual assessments to identify underlying cognitive resources for each option and encourage the intentional design of "resource-equivalent" questions in future surveys.
• Limitations: The authors acknowledge that mapping choices to resources is an assumption, as the DIRECT survey was originally developed under the misconception framework. They note that the assumption that different students selecting the same option are activating similar resources requires further validation through methods like interviews.