Intuition First or Reflection Before Judgment? The Impact of Evaluation Sequence on Consumer Ratings

This research demonstrates that the sequence of rating and reviewing significantly polarizes consumer evaluations—amplifying high ratings for good services and low ratings for poor ones—by triggering affective heuristics and varying cognitive effort, a finding validated through experiments and real-world data from platforms like Yelp and Letterboxd.

The Gist

Imagine you just finished a meal at a restaurant. You loved it (or hated it), and now you're on your phone to leave a review. But before you type a single word, the app asks you to tap a star rating. Or maybe, the app forces you to write a paragraph about your meal before you can tap those stars.

Does it matter which order you do these things? According to this research paper, yes, it matters a huge amount.

Here is the story of the research, explained simply with some everyday analogies.

The Big Idea: The "Order of Operations" for Reviews

The researchers (He Wang and his team) wanted to know if the sequence of filling out a review changes the final score. They compared two types of apps:

1. The "Star-First" App: You tap your stars (1 to 5) before you write anything.
2. The "Story-First" App: You write your review before you tap your stars.

They found that the order acts like a traffic light for your brain, changing how you process your feelings.

The Two Brain Modes: The "Gut" vs. The "Brain"

To understand why the order matters, imagine your brain has two modes:

• System 1 (The Gut): Fast, emotional, and intuitive. It's like a reflex. You feel happy, you give 5 stars. You feel angry, you give 1 star.
• System 2 (The Brain): Slow, analytical, and effortful. It's like a calculator. It weighs the pros and cons.

The "Star-First" sequence triggers System 1. You haven't thought deeply yet; you just react to your immediate feeling.
The "Story-First" sequence forces you to use System 2. To write a review, you have to recall details, organize your thoughts, and explain why you felt a certain way. This "cognitive effort" cools down your immediate emotional reaction.

The Results: The "Polarization" Effect

The study found a fascinating "polarization" effect. The "Star-First" approach makes ratings more extreme (very high or very low), while "Story-First" makes them more moderate and balanced.

Here is how it played out in two different scenarios:

Scenario A: The Amazing Meal (High Quality)

• Star-First: You just had a great time. Your gut says "Yay!" You tap 5 stars immediately. Because you haven't stopped to think about the slightly cold fries, you stick with that 5. Result: A perfect score.
• Story-First: You have to write, "The food was great, but the fries were cold." As you write, your brain realizes, "Well, it wasn't perfect." You might drop it to a 4. Result: A slightly lower, but more realistic score.

Scenario B: The Terrible Meal (Low Quality)

• Star-First: You are furious. Your gut screams "Angry!" You tap 1 star immediately. You don't stop to think, "Well, the waiter was nice," so you stick with the 1. Result: A rock-bottom score.
• Story-First: You have to write, "The food was terrible, but the waiter tried his best." As you force yourself to write the review, your brain calms down a bit. You realize it wasn't all bad. You might bump it up to a 2. Result: A slightly higher, more nuanced score.

The Analogy:
Think of the "Star-First" method like taking a sneeze. It's a sudden, explosive reaction. It's pure emotion.
Think of the "Story-First" method like writing a diary entry. You have to sit down, think, and organize your thoughts. This process naturally smooths out the jagged edges of your emotions.

The "Flavor" of the Product Matters

The researchers also found that this effect depends on what you are reviewing.

• Hedonic Products (Fun stuff): Like movies, fancy dinners, or vacations. These are about feelings. The "Star-First" method makes the ratings even more extreme because you are riding an emotional wave.
• Utilitarian Products (Useful stuff): Like a hammer, a battery, or a basic lunch. These are about function. Even if you tap stars first, your brain is already thinking about "did it work?" so the emotional swing isn't as wild.

Real-World Proof: Yelp vs. Letterboxd

The researchers didn't just run experiments; they looked at real data from millions of users.

• Yelp (Star-First): Their rating graph looks like a U-shape. Lots of 5s and lots of 1s, with very few middling scores. It's very polarized.
• Letterboxd (Story-First for movies): Their rating graph looks like a hill. Most people give 3s, 4s, and 5s, but fewer extreme 1s. The ratings are more concentrated and balanced.

Why Should You Care?

This isn't just about math; it's about honesty.

• If you are a business owner selling a fun, emotional product (like a theme park), you might want a "Star-First" interface to capture those happy, immediate feelings.
• If you are a business owner in a service industry where things can go wrong (like a hotel), you might want a "Story-First" interface. It forces customers to think before they judge, leading to fairer, more constructive feedback rather than just angry outbursts.

In a nutshell: The order in which we are asked to judge something changes how we judge it. Asking for a score first captures our heart; asking for a story first engages our head. And depending on what you're selling, you might want one or the other.

Technical Summary

Here is a detailed technical summary of the paper "Intuition First or Reflection Before Judgment? The Impact of Evaluation Sequence on Consumer Ratings."

1. Problem Statement

The research addresses a critical gap in online review literature: while extensive studies exist on how review content influences consumer decisions, little is known about how the interface design of the evaluation process itself (specifically the sequence of rating vs. writing a review) shapes the authenticity and extremity of the ratings generated.

• The Gap: Existing literature treats ratings as exogenous inputs or focuses on user self-selection bias (e.g., why only extreme users post). It overlooks the psychological processing mechanisms triggered by the order in which users are asked to provide a numerical score versus a textual review.
• The Core Question: Does requiring a "Rating-First" (numerical score before text) versus "Review-First" (text before score) interface alter consumer ratings through affective (emotional) and cognitive (analytical) processing paths, and how do service quality and product attributes moderate this effect?

2. Methodology

The authors employed a multi-method approach combining three controlled behavioral experiments and a large-scale secondary data analysis to ensure both internal and external validity.

A. Experimental Design (Experiments 1–3)

• Participants: Recruited via the Credamo platform (Total N ≈ 714 across three studies).
• Design:
  • Experiment 1: 2 (Sequence: Rating-First vs. Review-First) × 2 (Service Quality: High vs. Low) between-subjects design. Focused on the main effect of sequence on rating extremity.
  • Experiment 2: Replicated Experiment 1 but added mediators: Affective Heuristics (reliance on feelings) and Cognitive Effort (mental energy invested). Used a serial mediation model.
  • Experiment 3: 2 × 2 × 2 mixed design (Sequence × Service Quality × Product Attributes: Hedonic vs. Utilitarian). Investigated product attributes as a moderator.
• Procedure: Participants recalled a specific dining experience (simulated via textual stimuli) and were forced into either the "Rating-First" or "Review-First" condition. They then rated the service and wrote a review.
• Measures:
  • Dependent Variable: Overall rating (5-point Likert scale).
  • Mediators: Scales for Affective Heuristics (e.g., "I relied on feelings") and Cognitive Effort (single-item scale).
  • Moderator: Product attributes (Hedonic vs. Utilitarian scales).

B. Secondary Data Analysis

• Data Sources: ~7.7 million consumer review records from two distinct platforms:
  • Yelp: "Rating-First" interface (users rate before writing).
  • Letterboxd: "Review-First" interface (users write reviews before rating).
• Analysis: Compared rating distributions (bimodal vs. concentrated) using Kolmogorov-Smirnov, Chi-square, and Anderson-Darling tests to validate experimental findings in a real-world setting.

3. Key Contributions

1. Theoretical Extension of Dual-Process Theory: The study applies Kahneman's System 1 (fast, intuitive) and System 2 (slow, analytical) frameworks to the generation of reviews. It posits that "Rating-First" activates System 1 (affective heuristics), while "Review-First" forces System 2 engagement (cognitive effort).
2. Identification of a Serial Mediation Mechanism: The paper establishes a specific psychological pathway: Sequence $\rightarrow$ Affective Heuristics $\rightarrow$ Cognitive Effort $\rightarrow$ Final Rating.
3. Contextual Boundary Conditions: It demonstrates that the impact of sequence is not uniform; it is contingent on Service Quality (High vs. Low) and Product Attributes (Hedonic vs. Utilitarian).
4. Empirical Validation via Big Data: By correlating experimental results with real-world data from Yelp and Letterboxd, the study bridges the gap between lab simulations and actual platform behavior, confirming that interface design drives distributional polarization.

4. Key Results

A. Main Effect: The Polarization Phenomenon

• High-Quality Contexts: "Rating-First" leads to significantly higher ratings than "Review-First."
• Low-Quality Contexts: "Rating-First" leads to significantly lower ratings than "Review-First."
• Conclusion: The "Rating-First" sequence amplifies rating extremity (polarization), whereas "Review-First" leads to more moderate, concentrated ratings.

B. Mediation Mechanism (Serial Path)

• Path: Rating-First $\rightarrow$ Increased Affective Heuristics $\rightarrow$ Decreased Cognitive Effort $\rightarrow$ Extreme Ratings.
• High-Quality: High affect + Low effort $\rightarrow$ Ratings are "rescued" or boosted by positive emotion.
• Low-Quality: High affect + Low effort $\rightarrow$ Ratings are driven down by negative emotion without the "buffer" of rational analysis.
• Review-First: Forces cognitive effort, which acts as a regulatory mechanism, tempering extreme emotional reactions and leading to more balanced scores.

C. Moderation by Product Attributes

• Hedonic Products (e.g., entertainment, luxury dining): The polarization effect of "Rating-First" is stronger. Emotional engagement is high, making users more susceptible to intuitive, extreme judgments.
• Utilitarian Products (e.g., basic meals, functional services): The effect is weaker. Users rely more on attribute-based trade-offs, making the sequence less impactful on extremity.

D. Secondary Data Findings

• Yelp (Rating-First): Exhibits a U-shaped bimodal distribution (high concentration of 1-star and 5-star reviews), confirming the polarization hypothesis.
• Letterboxd (Review-First): Exhibits a right-skewed inverted U-shape (more concentrated around the mean), confirming that requiring text first dampens extremity.

5. Significance and Implications

Theoretical Significance

• Shifts the Focus: Moves the study of online reviews from "content analysis" to "process design." It proves that the act of evaluation is malleable and subject to interface-induced biases.
• Explains Rating Skew: Offers a new explanation for the ubiquitous "J-shaped" or "U-shaped" rating distributions on platforms, suggesting they are not just due to self-selection bias but are structurally induced by the rating interface.
• Integrates Emotion and Cognition: Provides a nuanced model showing how emotional heuristics and cognitive effort interact dynamically during judgment formation.

Managerial/Practical Implications

• Platform Design:
  • For High-Quality/Hedonic Services: Use "Rating-First" to capture peak positive emotions and amplify positive word-of-mouth.
  • For Low-Quality/Utilitarian Services: Use "Review-First" to encourage rational reflection, potentially reducing the severity of negative ratings and providing more constructive feedback.
• Algorithmic Optimization: Platforms can dynamically adjust the evaluation sequence based on the product category or historical user sentiment to generate more authentic and less polarized data.
• Credibility Management: Understanding that "Rating-First" may artificially inflate or deflate scores based on immediate affect helps stakeholders interpret rating distributions more accurately.

Conclusion

The paper concludes that the evaluation sequence is a powerful regulatory force in consumer behavior. By altering the order of tasks, platforms can fundamentally shift the balance between intuitive emotion and rational analysis, thereby shaping the authenticity and distribution of consumer ratings. This offers a strategic lever for optimizing interface design to align with specific business goals (e.g., maximizing positive sentiment vs. ensuring balanced feedback).