The challenging task of investigating student thinking: an example from quantum computing

This paper chronicles the extensive development and revision of a single multiple-choice item on the Quantum Computing Conceptual Survey to illustrate the complexities of probing student reasoning in quantum mechanics, specifically regarding phase kickback, while offering a cautionary tale for researchers and instructors using such assessments.

The Gist

Imagine you are a detective trying to figure out what a student is actually thinking when they solve a physics problem. You can't just read their minds; you have to look at the clues they leave behind.

This paper is the story of one specific clue—a single multiple-choice question on a test about Quantum Computing—that caused the researchers so much trouble, debate, and revision that it took up more energy than the other 19 questions on the test combined.

Here is the story of "Item 15," told in simple terms.

The Big Picture: The "Mind-Reading" Problem

Physics Education Researchers (PER) want to know how students think about physics. But unlike a chemist who can mix two chemicals and see a reaction, researchers can't easily see inside a student's brain.

• The Problem: Students might get an answer right for the wrong reason (like guessing), or get it wrong because they misunderstood a tiny word in the question.
• The Goal: The team wanted to build a "Quantum Computing Conceptual Survey" (QCCS) to test if students really understood the weird, counter-intuitive rules of quantum computers.

The Villain: "Phase Kickback"

The specific question (Item 15) was designed to test a concept called Phase Kickback.

• The Analogy: Imagine you have two magic coins. One is the "Control" coin, and the other is the "Target" coin.
• In a normal world, if you flip the Target coin, the Control coin doesn't care.
• In the Quantum World, if the Target coin is in a special "superposition" (spinning both heads and tails at once), flipping it actually sends a "kick" of information backwards to the Control coin, changing its state.
• The Trap: This is so weird that it has no real-life equivalent. It's like if you sneezed, and it changed the color of your neighbor's shirt. The researchers wanted to see if students could spot this invisible "kick."

The Saga of Item 15: A Tale of Four Versions

The team tried to write this question four times. Each time, they thought they had fixed it, but the students' answers told a different story.

Version 1.0: The "Too Vague" Attempt

• The Setup: They drew a circuit diagram with a generic "input state" and asked what happened.
• The Glitch: Students were confused by the wording. Some thought "measurement" meant something specific; others thought it meant nothing. The word "effect" was too fuzzy.
• The Lesson: You can't ask a physics question if the students don't agree on what the words mean. It's like asking, "Did the car move?" without defining if you mean the wheels turned or the car changed location.

Version 2.0: The "Guessing Game"

• The Fix: They made the question more specific. They gave a concrete input state and clarified the diagram.
• The Glitch: The statistics showed a weird pattern. The "smart" students got it right, but the "struggling" students also got it right at the same rate!
• The Realization: The struggling students weren't understanding the physics; they were just guessing. They saw the answer choices and picked the one that sounded "scientific" without doing the math. The question wasn't measuring knowledge; it was measuring luck.

Version 2.1: The "Impossible" Trap

• The Fix: They split the question into two parts to stop the guessing. They added an option that said, "None of the above."
• The Glitch: Only 3% of students got it right. The question was now too hard.
• The Realization: Students were terrified of the "None of the above" option. Even if they knew the other answers were wrong, they were too scared to pick "None of the above" because it felt like a trick. They would rather guess a wrong answer than admit the question might be broken. The question was measuring test-taking anxiety, not quantum physics.

Version 2.2: The Final Solution

• The Fix: They changed the "None of the above" option to something very specific: "The state cannot be written as a single-qubit ket." (In plain English: "The two coins are so tangled together you can't describe one without the other.")
• The Result: Finally, the data made sense!
  • The smartest students got it right.
  • The struggling students got it wrong.
  • The "wrong" answers revealed specific misconceptions (e.g., thinking the Control coin never changes, or thinking you can always describe one coin separately).
• The Victory: The question finally stopped being a riddle and started being a true test of understanding.

The Moral of the Story

This paper teaches us three big lessons for anyone trying to teach or test complex ideas:

1. Students are tricky: They will use "test-taking strategies" (like avoiding "None of the above") that have nothing to do with the subject matter.
2. Small words matter huge: Changing a word from "can" to "must," or clarifying a diagram, can completely change how a student thinks.
3. Iterate, Iterate, Iterate: You cannot just write a test and hand it out. You have to try it, watch students struggle, listen to their confusion, and rewrite it. The "messy" process of fixing a bad question often teaches you more about your students than the final, perfect question ever could.

In short: The researchers spent years trying to build a single question that could see inside a student's mind. They failed three times before they finally succeeded, proving that understanding how people learn is just as complex as the physics they are trying to learn.

Technical Summary

1. Problem Statement

Physics Education Research (PER) aims to understand student reasoning, but probing the nuances of this thinking is notoriously difficult. While qualitative methods (interviews, observations) provide depth, they are hard to scale. Quantitative tools (conceptual surveys) are necessary for broad assessment but often fail to capture complex reasoning or are susceptible to "test-taking strategies" rather than genuine conceptual understanding.

The specific problem addressed in this paper is the development of a valid, reliable multiple-choice item (Item 15) for the Quantum Computing Conceptual Survey (QCCS). The item targets phase kickback, a counter-intuitive phenomenon in quantum algorithms where the state of a control qubit changes due to the measurement of a target qubit in an entangled system. The authors found that despite rigorous design, the item initially failed to discriminate between high and low-performing students, often measuring guessing strategies or confusion over notation rather than the intended physics concepts.

2. Methodology

The authors employed an iterative design and validation process over several academic years (Fall 2023 – Fall 2025) to refine Item 15. The methodology combined quantitative statistical analysis with qualitative diagnostic data.

• Instrument: The QCCS, a closed-form, multiple-choice assessment designed to measure student understanding of quantum computing concepts.
• Data Collection:
  • Large-scale administration: Data collected from $N \approx 2,000$ students across 46 US institutions over four pilot rounds (Versions 1.0, 2.0, 2.1, 2.2).
  • Qualitative Diagnostics: Think-aloud interviews with students ($N=36$ total across versions) and interviews with faculty ($N=6$) to identify structural ambiguities and reasoning errors.
  • Statistical Metrics:
    • Item Difficulty ($p_i$): Fraction of students answering correctly (target range 0.3–0.9).
    • Item Discrimination ($\rho^*$): Point-biserial correlation between the item and the total test score (target $\ge 0.2$, ideally $\ge 0.3$).
• Iterative Refinement: The team analyzed response patterns, interview transcripts, and statistical plots to identify why the item failed, then modified the question stem, circuit diagram, or answer choices for the next version.

3. Key Contributions & Development Timeline

The paper provides a "behind-the-scenes" case study of how a single test item evolved through four major versions to overcome specific cognitive and methodological hurdles:

• Version 1.0 (Fall 2023):

  • Issue: Ambiguity in measurement notation (Z-basis vs. unspecified) and the word "effect." Students were confused about whether the top qubit's state was "ill-defined" or simply unchanged in probability.
  • Fix: Added specific input states and clarified measurement notation.

• Version 2.0 (Spring 2024):

  • Issue: Low discrimination ($\rho^* = 0.14$). Low-performing students guessed the correct answer ("b"), while middle-performing students chose a conceptual distractor ("a"). The item was non-monotonic.
  • Diagnosis: Interviews revealed students could guess correctly without understanding the intermediate state of the top qubit.
  • Fix: Split the item into two sub-parts to force a step-by-step reasoning process.

• Version 2.1 (Fall 2024):

  • Issue: Extreme difficulty ($p = 0.03$). Only 3% answered correctly.
  • Diagnosis: The correct answer for part (i) was "e" (None of the above). Students, even high achievers, were reluctant to select "None of the above" due to test-taking anxiety, preferring to guess among options a-d.
  • Fix: Removed the "None of the above" option and replaced it with a specific, conceptually accurate description of the state.

• Version 2.2 (Spring/Fall 2025):

  • Result: The item finally functioned as intended.
  • Statistics: Difficulty $p = 0.23$ (challenging but above the guess floor); Discrimination $\rho^* \gtrsim 0.3$ (strong).
  • Mechanism: The final item successfully forced students to integrate two concepts: (1) recognizing that the CNOT gate creates an entangled state that cannot be written as a single-qubit ket, and (2) understanding the effect of partial measurement on that entangled state.

4. Results

• Student Reasoning Patterns:
  • Naive Conceptions: Many students incorrectly believe the control qubit of a CNOT gate never changes (a classical XOR analogy error).
  • State vs. Probability: Students often conflate the quantum state vector with measurement probabilities, failing to recognize that a state can change (phase kickback) without altering Z-basis measurement probabilities.
  • Test-Taking Strategies: The evolution of the item highlighted how students use meta-cognitive strategies (e.g., avoiding "None of the above," guessing based on distractor patterns) that can mask true conceptual understanding.
• Statistical Validation: The final version (2.2) showed a monotonically increasing probability of a correct answer as overall test scores increased, confirming the item measures the intended construct rather than guessing.

5. Significance

• For PER Practitioners: The paper serves as a cautionary tale and a methodological guide. It demonstrates that even expert-designed items can fail due to subtle factors like notation conventions, "None of the above" biases, or the inability of multiple-choice formats to capture intermediate reasoning steps.
• On Student Thinking: It reveals that "phase kickback" is a significant conceptual hurdle. Students struggle to move beyond classical analogies (XOR gates) and to visualize entangled states where individual qubits do not have well-defined pure states (kets).
• Curriculum Implications: The study highlights the importance of notational consistency across courses. The authors found that some courses teach measurement as basis-agnostic (meaningless without specification), while others implicitly assume the Z-basis. This divergence confuses students when they encounter standardized assessments.
• Validation Process: It underscores that iterative validation with actual students is irreplaceable. Expert intuition and whiteboard discussions are insufficient to predict how a question will perform in a real classroom setting.

In conclusion, the paper argues that the "saga" of Item 15 is not just about a specific quantum concept, but a microcosm of the broader challenges in PER: measuring complex, non-linear student reasoning through the rigid constraints of multiple-choice testing requires rigorous, iterative, and often surprising refinement.