A Deep Investigation Using Factorial Analysis and Rasch Model: An Optimized Short-Version of the Computational Thinking Scale (COTS)

Technical Summary: An Optimized Short-Version of the Computational Thinking Scale (COTS)

Problem Statement
Computational thinking (CT) is recognized as a foundational 21st-century competency essential for systematic problem-solving across diverse disciplines. Despite its growing importance in education, the field lacks psychometrically robust instruments specifically designed to assess CT skills, particularly in non-ICT contexts such as social sciences. Existing tools often rely on closed-ended questionnaires that may not adequately account for cultural differences or possess sufficient validity and reliability. This gap limits the ability of researchers and educators to accurately diagnose student strengths and weaknesses, thereby hindering the design of effective educational interventions. The study addresses the need for a high-quality, validated measurement tool grounded in rigorous psychometric analysis.

Methodology
The study employed a two-stage sequential validation process to develop and refine a Computational Thinking Skills instrument (COTS), originally adapted from a 22-item scale.

• Participants and Data Collection: Data were collected anonymously from undergraduate students in Makassar, Indonesia. The process involved two distinct samples: 364 participants for the initial Confirmatory Factor Analysis (CFA) and 434 participants for the subsequent Rasch model analysis.
• Stage 1: Confirmatory Factor Analysis (CFA): Using JASP software, the researchers evaluated the factorial validity of the 22-item instrument. The analysis focused on factor loadings (requiring >0.50), model fit indices (CFI, GFI, TLI, NNFI, IFI, RNI >0.90; RMSEA and SRMR <0.08), and internal consistency reliability (McDonald's ω and Cronbach's α >0.80 for the total scale). Items failing to meet these criteria were removed.
• Stage 2: Rasch Rating Scale Model (RSM): The refined instrument was analyzed using J.Metrik software. This stage assessed fundamental measurement assumptions, including:
  • Unidimensionality: Verified via Principal Component Analysis (PCA) of residuals (eigenvalue of the first contrast <3.0).
  • Local Independence: Evaluated through residual correlations between item pairs (threshold <0.40).
  • Rating Scale Functioning: Examined for monotonic increasing patterns in average measures and ordered threshold parameters.
  • Model Fit: Assessed using Infit and Outfit Mean Square statistics (acceptable range 0.5–1.5).
  • Reliability and Separation: Calculated for both items and persons to determine measurement precision and the ability to distinguish between different ability levels.
  • Item Difficulty: Analyzed to determine the distribution of items across the latent trait continuum.

Key Results

• Instrument Refinement: The CFA stage resulted in the removal of two items from the "abstraction" dimension due to factor loadings below 0.50. The final instrument consists of 20 items across four dimensions: Abstraction (3 items), Decomposition (6 items), Algorithmic Thinking (6 items), and Generalization/Pattern Recognition (5 items).
• CFA Outcomes: The final model demonstrated excellent fit. All fit indices exceeded 0.90 (e.g., CFI = 0.922, TLI = 0.908), and error indices were below 0.06 (RMSEA = 0.057, SRMR = 0.046). Factor loadings ranged from 0.52 to 0.75. The overall reliability was 0.89, with subscale reliabilities ranging from 0.62 to 0.85.
• Rasch Outcomes:
  • Assumptions: The instrument satisfied unidimensionality (first contrast eigenvalue = 2.54) and local independence (maximum residual correlation = 0.40).
  • Model Fit: All 20 items fell within the acceptable Infit/Outfit range (0.5–1.5), and the rating scale functioned optimally with monotonically increasing thresholds.
  • Reliability: Item reliability was excellent (0.97–0.98), and person reliability was good (0.87–0.88). Separation indices indicated the instrument could distinguish approximately 9.89 strata of item difficulty and 3.80 strata of person ability.
  • Item Difficulty: The difficulty range spanned from −0.79 to 0.82 logits. The distribution indicates the instrument is most effective for assessing individuals with moderate levels of CT skills.

Significance and Claims
The paper claims to provide a psychometrically sound, short-version instrument (COTS) for assessing computational thinking among university students. By integrating CFA and Rasch modeling, the study offers a comprehensive validation that addresses both structural validity and item-level measurement properties.

The authors assert that the COTS is a valid and reliable tool for educational contexts, capable of identifying student strengths and weaknesses to inform targeted interventions and curriculum design. However, the study modestly acknowledges a limitation: the instrument is primarily targeted at individuals with moderate CT abilities. Due to the clustering of item difficulty in the low-to-moderate range, the tool has limited precision for differentiating individuals with advanced or high-level CT skills. Consequently, the paper concludes that while the current instrument is robust for general assessment, future development should focus on creating more challenging items to improve coverage at the upper end of the ability spectrum.