📊 statistics

The Behavioural Reflection Test: A time-efficient measure of reflective reasoning in morally and epistemically charged decisions

The paper introduces the Behavioural Reflection Test (BRT) and a bespoke Cognitive Reflection Test (bCRT) as time-efficient, low-exposure measures that successfully predict evidence-sensitive, ethically driven decision-making and specific linguistic patterns in online adults, outperforming traditional familiarity-adjusted CRT metrics.

Original authors: Sion Weatherhead, Flora Salim, Aaron Belbasis, Ben R. Newell

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sion Weatherhead, Flora Salim, Aaron Belbasis, Ben R. Newell

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your brain has two modes. One is the "Fast Lane": a speedy, automatic autopilot that grabs the first answer that pops into your head. The other is the "Slow Lane": a reflective mode where you hit the brakes, check the map, and think things through. For years, scientists have used a famous puzzle game called the Cognitive Reflection Test (CRT) to see who uses the Slow Lane. But here's the catch: the old puzzles have become so famous that everyone knows the "trick" answers. It's like playing a video game where everyone has already memorized the cheat codes; scoring high just means you've seen the level before, not that you're actually thinking deeply right now.

This paper introduces a fresh set of tools to fix that problem: the Behavioural Reflection Test (BRT) and a new, shorter puzzle called the bespoke CRT (bCRT).

The New Puzzle: Fresh Meat

First, the authors built a new four-question puzzle (the bCRT) that most people haven't seen yet. They tested it on 473 adults online. The results showed that this new puzzle is a better "cheat-code detector." While the old puzzles were familiar to 40–74% of people, the new ones were only familiar to 5–32% of them.

When they looked at who scored high on this new puzzle, they found something interesting: these people weren't just smart; they were reflective. They correlated with traits like "Need for Cognition" (loving to think) and "Conscientiousness" (being organized and careful). The paper suggests that this new puzzle measures a genuine tendency to override your first instinct, rather than just testing how well you remember old riddles.

The Real-World Test: The Restaurant and The Doctor

To see if this "reflective" brain actually changes how people act in real life, the researchers created the Behavioural Reflection Test (BRT). Instead of solving math riddles, participants had to make decisions in two messy, real-life scenarios:

The Restaurant Dilemma: You have a non-refundable birthday dinner booked. A friend sends you scary, low-quality reviews from social media saying the food is terrible, contradicting professional critics. Do you cancel?
The Medicine Dilemma: Your doctor prescribes a standard medicine. A popular social media influencer warns it's dangerous based on a few scary stories. Do you take it?

The paper found that people who scored higher on the new bCRT puzzle made better, more evidence-based choices.

In the restaurant: They were more likely to keep the booking, trusting the professional critics over the noisy social media rumors. They also used language that showed they cared about fairness and loyalty, and they were more willing to call out their unhelpful friend for being selfish.
In the medicine scenario: They were more likely to stick with the doctor's advice or, if they hesitated, they planned to look for high-quality evidence (like peer-reviewed studies) rather than just scrolling through more influencer posts.

The Language Clues: How They Spoke

The researchers also analyzed the words these people used while explaining their choices. They found that more reflective people didn't just think differently; they spoke differently.

They used more words related to risk (like "danger" or "hazard"), showing they were paying attention to the potential downsides.
They used more negative emotion words (like "frustrated" or "annoyed"). The authors suggest this isn't because they were angry, but because they were emotionally engaged with the unfairness of the situation.
They used fewer "filler" words (auxiliary verbs like "is," "have," or "will"). This suggests their thinking was more direct and economical, cutting out the fluff.

What the Paper Rules Out

It's important to note what this study says is NOT the answer.

It's not just about being smart: The paper argues that the old puzzles often just measured math skills or memory of the trick, not true reflection.
It's not about using "moral" words: The authors explicitly found that while reflective people made fairer decisions, they didn't necessarily use more words from a "moral" dictionary (like "right" or "wrong"). Their morality showed up in their actions and reasoning structure, not just in fancy vocabulary.
It's not a magic bullet for everything: The paper notes that for the medicine scenario, the link between the puzzle score and the final decision was a bit fuzzy. This suggests that in some complex situations, being reflective doesn't always lead to one single "correct" choice, because both accepting the doctor and delaying to check facts can be reasonable.

How Sure Are They?

The authors are measured and statistically confident in their main findings. They ran the numbers on 473 people and found that the new puzzle (bCRT) consistently predicted these better decisions and language patterns. They even checked to make sure the results weren't just a fluke by using a strict statistical correction (Benjamini–Hochberg), and the results held up for four out of five key outcomes.

However, they are cautious about some details. For instance, they found that reflective people used more "negative emotion" words, but they admit they can't be 100% sure if this means the people were actually feeling bad, or just using those words to describe the situation sharply. They suggest this is a clue worth investigating further, not a final proof.

The Bottom Line

Think of the old CRT puzzles as a worn-out map that everyone has memorized. The BRT and bCRT are like a new, uncharted territory. The study suggests that when you give people fresh, tricky situations, the ones who truly "reflect" are the ones who ignore the noisy crowd, trust the experts, check the risks, and speak with a clear, direct voice. They don't just solve puzzles; they navigate the messy, noisy world of real-life decisions with a bit more care.

Problem Statement
The paper addresses two interrelated limitations in the measurement of reflective reasoning. First, the Cognitive Reflection Test (CRT), a prominent measure of the tendency to override intuitive responses, suffers from widespread item familiarity. As classic items become known, scores increasingly reflect prior exposure rather than the immediate cognitive tendency to inhibit intuitive answers, weakening psychometric interpretation and predictive validity. Second, existing measures often fail to capture how reflective tendencies manifest in "messier," real-world decision-making and linguistic expression outside of decontextualized puzzles. While prior research links CRT scores to specific linguistic patterns and informational hygiene on social media, these findings have not been leveraged to create an efficient, rich behavioral measure of reflection that operates beyond "trick questions."

Methodology
The study utilized two independent online samples: a pilot sample of ~240 undergraduate students (UNSW SONA) for item selection and a validation sample of 473 adults (Prolific, UK/USA) for primary analysis.

Bespoke CRT (bCRT): The authors developed a four-item CRT designed to minimize prior exposure while maintaining the classic "intuitive lure vs. correct answer" structure. Items were selected from a larger pool using classical item statistics and two-parameter logistic item response models (2PL IRT) to ensure acceptable difficulty and discrimination.
Behavioural Reflection Test (BRT): This is a brief, open-ended behavioral measure consisting of two scenarios:
- Restaurant Scenario: Participants must decide whether to keep a non-refundable booking despite negative crowd reviews contradicting professional critics, involving a friend's unhelpful intervention.
- Medication Scenario: Participants must decide on a doctor-prescribed medication challenged by a health influencer's anecdotal warnings.
  Participants provided open-ended justifications for their decisions.
Measures and Coding:
- Linguistic Analysis: Responses were analyzed using LIWC-22 to quantify specific categories (e.g., negative emotion, risk, insight, moral, auxiliary verbs).
- Decisional Analysis: Responses were coded using a large language model (GPT-5-nano) under a pre-specified rubric to extract binary outcomes regarding evidence weighting, fairness/loyalty reasoning, norm enforcement, and epistemic search quality.
- Covariates: The study included the standard CRT2, Need for Cognition (NFC), and Conscientiousness (BFI-10) to establish convergent validity.

Key Contributions

The Behavioural Reflection Test (BRT): A time-efficient (median 11.8 minutes), open-ended measure that captures reflective reasoning in morally and epistemically charged contexts. It assesses how individuals integrate evidence, navigate social norms, and justify decisions without relying on fixed "correct" answers.
The Bespoke CRT (bCRT): A low-exposure, four-item instrument that serves as a fresher anchor for the construct of cognitive reflection. It demonstrates that a brief measure can retain predictive power when legacy items are saturated.
Integration of Language and Behavior: The study bridges the gap between abstract cognitive traits and their enactment in decision style and linguistic expression, moving beyond laboratory puzzles to ecologically plausible scenarios.

Results

Psychometrics of bCRT: The bCRT showed modest internal consistency ( $\alpha = .539$ ) and a unidimensional structure. It correlated positively with the CRT2 ( $r = .259$ ), NFC, and Conscientiousness, supporting convergent validity while indicating it is not redundant with the CRT2. Item Response Theory (IRT) analysis revealed that bCRT items provided complementary information to the CRT2, particularly at and above mean ability levels.
Prior Exposure: bCRT items had significantly lower prior-exposure rates (5–32%) compared to CRT2 items (40–74%), validating the low-exposure rationale.
Decisional Outcomes: Higher bCRT scores significantly predicted four of five primary decisional outcomes (after Benjamini–Hochberg correction):
- Fair, evidence-based retention of the restaurant booking.
- Spontaneous appeals to fairness and loyalty.
- Hostile rebuke of the unhelpful friend (norm enforcement).
- High-quality epistemic search plans in the medication scenario.
- (Note: Evidence-based acceptance of the medication was directionally consistent but did not survive correction, likely due to the scenario's ambiguity).
Linguistic Markers: Higher bCRT scores predicted:
- Greater use of negative emotion and risk words.
- Reduced use of auxiliary verbs, discrepancy (e.g., "should," "could"), and differentiation markers.
- Greater present-focus.
- Crucially, the familiarity-adjusted CRT2 failed to recover these linguistic or decisional patterns, suggesting saturation has eroded the CRT2's ability to predict these specific behavioral signatures.

Significance and Claims
The paper claims that the BRT and bCRT together offer a pragmatic framework for studying reflection as both a latent trait and an enacted pattern of reasoning. The significance lies in demonstrating that:

Reflection is Expressive: More reflective individuals do not just solve puzzles better; they exhibit distinct behavioral and linguistic signatures, including a willingness to sanction norm violations, reliance on high-quality institutional evidence, and a more direct, economical, yet emotionally engaged writing style.
Legacy Measures are Saturated: The failure of the familiarity-adjusted CRT2 to predict the same linguistic and behavioral outcomes as the bCRT suggests that classic CRT items no longer capture the full spectrum of reflective tendencies in contemporary samples.
Efficiency and Validity: The BRT remains a time-efficient behavioral assay that recovers theoretically relevant signatures of reflection (informational hygiene, risk attention, norm sensitivity) without the time cost of longer cognitive batteries or the saturation issues of legacy trick questions.

The authors conclude that while the BRT scenarios are stylized, they successfully reproduce key aspects of the language-behavior signature previously observed in large-scale social media data, providing a foundation for future work on interventions, individual differences, and computational modeling of reflective judgment.