Assessing Problem-Solving in HR Contexts: A Comparison Between Game-Based and Self-Report Measures

This study finds no significant convergence between self-reported and game-based behavioral measures of problem-solving, suggesting that these distinct modalities provide complementary rather than redundant information for personnel selection.

The Gist

Imagine you are hiring a new employee for a job that requires solving tricky puzzles under pressure. You have two ways to find out if they are good at this:

1. The "Resume Interview" (Self-Report): You ask the candidate, "On a scale of 1 to 10, how good are you at solving problems?" They think about their life, maybe feel a bit nervous, and say, "I'm a 10! I'm a genius at fixing things!"
2. The "Obstacle Course" (Game-Based Assessment): You put them in a room with a digital puzzle game. You watch them play for five minutes. You don't ask them how they feel; you just watch how they actually move the pieces, how fast they think, and whether they get stuck.

This paper is basically a report card on what happens when you compare these two methods side-by-side.

The Big Surprise: They Don't Match!

The researchers took 72 people (mostly university students) and gave them both tests. They expected that the people who said, "I'm a problem-solving wizard!" would be the same people who actually crushed the video game puzzle.

They were wrong.

The results showed zero connection between the two.

• Some people who claimed to be "Very High" problem-solvers on the questionnaire actually struggled with the game.
• Some people who said, "I'm just average," actually played the game like a pro.

It's like asking a chef, "How good are you at cooking?" and they say, "I'm the best!" but then watching them burn toast. Or, someone saying, "I'm a terrible cook," but then watching them whip up a gourmet meal in seconds.

Why Did This Happen? (The Analogy)

The authors explain that these two tests are measuring two completely different things, like comparing a map to the actual terrain.

• The Questionnaire (The Map): This measures your confidence and your belief in your skills. It's like looking at a map and thinking, "I know this route perfectly." It's based on your memory and how you think you behave in real life. It can be influenced by how you want to look (social desirability) or how nervous you are.
• The Game (The Terrain): This measures your actual performance in the moment. It's like driving the route. Even if you know the map perfectly, you might hit a traffic jam, get distracted, or make a wrong turn. The game captures how you handle pressure, time limits, and unexpected twists right now.

The paper suggests that knowing you are good (confidence) is not the same thing as being good (performance).

What Does This Mean for Hiring?

If you are a boss or an HR manager, this study gives you a very important warning: Don't rely on just one method.

• If you only use questionnaires: You might hire the loudest, most confident people who are actually bad at the job. You might miss out on quiet, skilled people who are too humble to brag.
• If you only use games: You might miss out on people who have great strategic thinking but just aren't good at video games or get nervous in front of a screen.

The Best Solution: Use both!
Think of it like a medical checkup. You wouldn't just ask a patient, "Do you feel healthy?" (Self-report). You also wouldn't just run an X-ray (Game/Behavior) without asking about their symptoms. You need both to get the full picture.

The Takeaway

This study proves that Game-Based Assessments (like the puzzle game used here) are not just "fun distractions." They are powerful tools that see things traditional tests miss. However, they aren't a magic replacement for asking people about themselves.

To find the best employees, companies should stop trying to choose between "talking" and "doing." Instead, they should combine them. Let the candidate tell you what they think they can do, and then let them prove it by actually doing it. That way, you get the full story, not just the highlight reel.

Technical Summary

1. Problem Statement

The study addresses a critical gap in Human Resources (HR) and organizational psychology: the lack of empirical evidence comparing Game-Based Assessments (GBAs) with traditional self-report measures for evaluating transversal soft skills, specifically problem-solving.

• Context: Organizations increasingly rely on GBAs to capture dynamic, behavioral data, while traditional recruitment still heavily utilizes self-report psychometric tools (e.g., the Problem Solving Inventory).
• The Conflict: It is unclear whether these two modalities measure the same underlying construct or distinct facets of competence. Traditional self-reports are susceptible to social desirability bias, response distortion, and test anxiety, and they measure perceived competence rather than enacted behavior. Conversely, GBAs measure real-time behavior but their convergence with established psychometric scales remains unverified.
• Research Question: Do self-perceived problem-solving abilities (measured via PSI-B) converge with behaviorally enacted problem-solving performance (measured via a GBA)?

2. Methodology

The study employed a cross-sectional method-comparison design involving 72 participants.

• Participants:
  • $N = 72$ (final sample after excluding incomplete data).
  • Demographics: Undergraduate and graduate Computer Science students from the University of Camerino (Italy), aged 18+.
  • Recruitment: Voluntary participation during class sessions; no incentives offered.
• Instruments:
  1. Self-Report Measure: The Problem Solving Inventory (PSI-B), a revised version of the classic PSI.
     • Format: 6-point Likert scale (1 = Strongly Agree to 6 = Strongly Disagree).
     • Focus: Measures beliefs, confidence, and perceived control regarding problem-solving.
     • Scoring: Raw scores were categorized into four ordinal levels (Base, Medium, High, Very High) based on quartiles to align with the GBA output.
  2. Game-Based Assessment (GBA): Behaveme-PS, a 5-minute interactive task developed by Emaze Gaming.
     • Mechanism: Based on principles from Raven's Progressive Matrices and the Tower of Hanoi. Candidates must reorganize elements under time pressure.
     • Dimensions Measured: Strategic planning, sequential thinking, and adaptability.
     • Data Capture: System recorded valid combinations, total moves, and average time per combination.
     • Output: Performance classified into four behavioral proficiency levels (Base, Medium, Advanced, Very Advanced).
• Procedure:
  • Participants completed the PSI-B first (supervised by a licensed psychologist) via a Microsoft Form.
  • Participants then completed the 5-minute Behaveme-PS session.
  • Participants manually entered their GBA result level into the form.
  • Data analysis was conducted independently by the academic team; the game developer had no role in data processing.
• Statistical Analysis:
  • Software: Jamovi (v2.6.26.0).
  • Primary Test: Spearman's rank-order correlation (chosen due to the ordinal nature of the categorized variables).
  • Secondary Analysis: Contingency tables and box plots to examine distribution and within-group variability.

3. Key Results

The study found no significant convergence between the two measurement modalities.

• Correlation Analysis:
  • Spearman's $\rho = -0.09$ ($p = 0.45$).
  • This indicates no significant monotonic relationship between self-reported problem-solving competence and behavioral performance in the game.
• Distribution Analysis (Contingency Table):
  • High Heterogeneity: Participants classified as "Very High" in the PSI-B were distributed across all four GBA performance levels (Base to Very Advanced).
  • Inverse Patterns: Many participants with "Medium" or "High" self-reported scores demonstrated "Advanced" or "Very Advanced" behavioral performance.
  • Low Self-Reporters: Participants in the "Base" PSI-B quartile were not predominantly low performers in the game; most were distributed across Medium to Very Advanced levels.
• Visual Evidence: Box plots confirmed overlapping distributions with no clear trend, reinforcing that high self-perception does not guarantee high behavioral execution, and vice versa.

4. Key Contributions

• Empirical Evidence of Divergence: The study provides robust data supporting the hypothesis that self-report and behavioral measures capture distinct, non-interchangeable facets of problem-solving.
• Theoretical Clarification: It distinguishes between Perceived Competence (what individuals believe they can do, captured by PSI) and Enacted Performance (how individuals actually behave under constraints, captured by GBA).
• Methodological Validation: It validates the use of GBAs not as replacements for psychometric tests, but as complementary tools that mitigate biases (e.g., social desirability, faking) inherent in self-reports.
• HR Practice Implications: The findings argue against relying on a single assessment modality. Exclusive reliance on self-reports risks false positives (overestimating ability) or false negatives (underestimating ability).

5. Significance and Limitations

Significance:

• Recruitment Strategy: The study underscores the necessity of multi-method assessment frameworks in personnel selection. Integrating GBAs with traditional tools provides a more holistic view of candidate capability.
• Bias Reduction: GBAs offer a mechanism to observe "maximal performance" in a low-anxiety, engaging environment, potentially revealing competencies that self-reports miss due to introspection limits or bias.
• Construct Validity: It challenges the assumption that different tools measuring "problem-solving" are tapping into the same construct, suggesting a need for more nuanced definitions of the skill in organizational psychology.

Limitations:

• Sample Homogeneity: The sample consisted entirely of young university students (CS majors), which may limit generalizability to diverse professional populations.
• Context: The study was conducted in an academic setting rather than a high-stakes real-world recruitment environment.
• Missing Variables: The study did not control for individual differences such as prior gaming experience, self-efficacy, personality traits, or motivation, which could influence the discrepancy between self-report and behavior.

Conclusion:
The paper concludes that self-report and game-based assessments are complementary, not redundant. To accurately assess complex, context-dependent soft skills like problem-solving, organizations should integrate behavioral observations (GBAs) with self-perception data (PSI) to mitigate the risks of single-modality assessment.