💬 NLP

Using Text-Based Causal Inference to Disentangle Factors Influencing Online Review Ratings

This paper proposes an enhanced CausalBERT framework with temperature scaling, hyperparameter optimization, and interpretability methods to disentangle the causal effects of specific aspects on online review ratings, demonstrating through a study of over 600,000 U.S. K-12 school reviews that school administration and benchmark performance are significant drivers of overall ratings.

Original authors: Linsen Li, Aron Culotta, Nicholas Mattei

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Linsen Li, Aron Culotta, Nicholas Mattei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out why a specific restaurant got a 5-star rating. You read the reviews and see people raving about the "steak" and the "service." But here's the problem: the restaurant also happens to be in a fancy neighborhood, and the reviews mention the "view" and the "valet parking" constantly.

If you just look at the words, you might think the steak alone caused the 5 stars. But maybe the steak is just good, and the real reason for the high rating is that rich people in that neighborhood love to write glowing reviews. In data science, we call this "confounding." It's like trying to taste the salt in a soup while someone is constantly adding pepper; you can't tell which spice is doing what.

This paper is about a new tool called CausalBERT that helps us separate the "salt" (the specific factor we care about) from the "pepper" (all the other confusing factors) in online reviews. The authors tested this on over 600,000 reviews of U.S. schools to see what really drives a school's overall rating.

Here is how they did it, broken down into simple steps:

1. The Problem: The "Noise" in the Reviews

Usually, computers analyze reviews by just counting how many positive words are used. But that's flawed.

The Analogy: Imagine a school is famous for having a great football team. Parents might write, "The football team is amazing, and the teachers are great!"
The Trap: A simple computer might think the football team is the only reason the school is rated highly. But maybe the football team just brings in a crowd of happy, wealthy parents who also happen to love the teachers. The computer needs to know: Did the teachers get the rating, or did the football team just bring the crowd?

2. The Solution: A "Time-Traveling" Computer

The authors used a method called Causal Inference. Think of this as a time-travel simulation.

They ask the computer: "If this school had exactly the same reviews, but we magically erased the mention of 'football,' what would the rating be?"
Then they ask: "What if we kept the 'football' mention but erased the 'teachers' mention?"
By comparing these two imaginary worlds, they can isolate the true effect of just one thing (like "administration" or "bullying") on the final rating.

3. The Upgrade: Making the Computer Smarter

The base tool they used (CausalBERT) was already good, but the authors gave it three specific upgrades to make it more reliable:

Temperature Scaling (The "Confidence Thermostat"):
Sometimes, the computer gets too confident. It might say, "I am 99.9% sure this review is about bullying," when it's actually only 60% sure. This over-confidence messes up the math.
- The Fix: They added a "temperature" knob. Turning it up "softens" the computer's confidence, making it admit, "I'm pretty sure, but not 100%." This prevents the math from crashing when the computer is wrong.
Hyperparameter Tuning (The "Volume Knob"):
The computer has to listen to two things: the specific topic (treatment) and the rest of the text (confounders). If it listens too hard to the "rest of the text," it might accidentally cancel out the signal it's trying to measure.
- The Fix: They found a way to automatically turn the "volume" up or down on the background noise depending on how tricky the data is, so the main signal stays clear.
Interpretability (The "X-Ray Vision"):
Deep learning models are usually "black boxes"—you put data in, and a number comes out, but you don't know why.
- The Fix: They added tools to highlight exactly which words the computer was looking at to make its decision. This lets humans verify, "Yes, the computer is actually looking at 'administration' and not just random words like 'the' or 'and'."

4. The Experiment: Testing with Fake and Real Data

Before trusting the tool with real schools, they tested it on semi-synthetic data.

The Setup: They took real school reviews and secretly injected fake sentences about "academic challenges" into some of them. They knew exactly how much these fake sentences should change the rating.
The Result: The upgraded CausalBERT was much better at guessing the true effect than the old methods. It successfully ignored the "noise" and found the "signal."

5. The Real-World Findings: What Actually Matters for Schools?

After testing, they applied the tool to 600,000 real reviews of U.S. K-12 schools. They looked at topics like bullying, curriculum, extracurriculars, and administration.

The Big Reveal:
When they removed the "confounding noise" (like the fact that schools with good facilities often have good teachers), they found:

Administration: How people feel about the school's leadership and staff is a massive driver of the overall rating.
Academic Performance: How well the school does on tests and benchmarks is the other huge driver.

The "Bullying" Surprise:
They found that mentions of "bullying" did lower ratings, but the effect wasn't as huge as a simple count of negative words would suggest. Why? Because schools with bullying often also had complaints about poor administration. The simple math thought bullying was the only problem, but the causal math showed that while bullying hurts, the administration was often the bigger underlying issue dragging the rating down.

Summary

This paper didn't just count words; it built a smarter way to understand cause and effect in text. By using a "thermostat" for confidence, a "volume knob" for noise, and "X-ray vision" to see what the computer is thinking, they proved that how a school is run (administration) and how well students perform (academics) are the true engines behind a school's reputation, not just the presence of other factors like sports or facilities.

Technical Summary: Using Text-Based Causal Inference to Disentangle Factors Influencing Online Review Ratings

Problem Statement

Online reviews offer rich insights into the perceived quality of products and services, yet traditional aspect-based sentiment analysis often fails to quantify the causal impact of specific aspects on overall ratings. A primary challenge is the presence of confounding variables within the text. For instance, a school praised for academic performance might also be praised for extracurriculars or parental involvement; a standard correlation analysis might incorrectly attribute the high overall rating solely to academics, ignoring the socioeconomic advantages or other factors driving the sentiment.

The goal of this work is to estimate the Average Treatment Effect (ATE) of specific textual aspects (e.g., mentions of "bullying" or "administration") on overall entity ratings while rigorously controlling for confounding factors present in the remaining text. The authors treat textual mentions as proxies for real-world attributes, aiming to isolate the causal influence of a specific factor from the noise of correlated factors.

Methodology

Framework and Assumptions

The authors adopt the potential outcomes framework (Neyman, 1923). For each entity (school) $i$ , they observe a tuple $(X_i, Y_i, T_i)$ :

$T_i \in \{0, 1\}$ : Treatment assignment (presence of a specific topic/keyword in reviews).
$Y_i$ : Continuous outcome (average review rating).
$X_i$ : Text covariates (the remaining text of the reviews, excluding the treatment-inducing keywords).

The objective is to estimate $\tau = E[Y_i(1) - Y_i(0)]$ . To address the violation of ignorability (where treatment assignment correlates with confounds), the authors assume Conditional Ignorability: treatment assignment is independent of potential outcomes given the covariates $X_i$ .

The CausalBERT Architecture

The core of the methodology is an enhanced CausalBERT (Veitch et al., 2020), which extends BERT (specifically DistilBERT) to learn text representations that predict both the propensity score and conditional expected outcomes.

Input Processing: Reviews are split into treatment-determining text (keywords) and covariate text (all other reviews concatenated).
Multi-Objective Learning: The model learns a dense representation $h_i$ $h_{i}$ from the covariate text to simultaneously optimize:
- Propensity Score Prediction ( $g$ ): Predicting the probability of treatment $T_i$ .
- Outcome Prediction ( $Q$ ): Predicting potential outcomes $Y_i(1)$ and $Y_i(0)$ under treatment and control scenarios.
Loss Function: A weighted sum of Mean Squared Error (MSE) for outcome prediction and Cross-Entropy (CE) for treatment assignment, balanced by a hyperparameter $\alpha$ .

Key Enhancements

The paper introduces three specific improvements to the standard CausalBERT framework:

Temperature Scaling: To address the instability of Inverse Probability Weighting (IPW) caused by extreme propensity scores (values near 0 or 1), the authors apply temperature scaling to the propensity score logits. A temperature parameter $M > 1$ "softens" the probability distribution, aligning predicted probabilities with their actual confidence levels and mitigating the risk of extreme weights in IPW and Augmented IPW (AIPW) estimators.
Hyperparameter Optimization for Overadjustment: The parameter $\alpha$ controls the trade-off between treatment prediction and outcome prediction. The authors propose an empirical strategy to select $\alpha$ based on the treatment prediction accuracy. They posit that lower treatment accuracy indicates weaker confounding; in such cases, a higher $\alpha$ (focusing more on treatment classification) yields better ATE estimates by preventing overadjustment for non-existent confounds.
Interpretability Methods: To validate the "black box" nature of the embeddings, the authors employ:
- CLS Comparative Analysis: Comparing attention weights on the CLS token between the fine-tuned CausalBERT and a baseline DistilBERT to identify tokens that become significant only after controlling for confounds.
- Integrated Gradients (IG): Attributing prediction changes to specific input tokens to characterize which words drive propensity scores and outcome predictions.

Estimators

The study evaluates the ATE using several estimators derived from the CausalBERT outputs:

Naive Estimator ( $\hat{\tau}_{unadjust}$ ): Simple difference in means.
Q-only ( $\hat{\tau}_Q$ ): Outcome regression adjustment.
IPW ( $\hat{\tau}_{IPW}$ ): Inverse Probability Weighting.
AIPW ( $\hat{\tau}_{AIPW}$ ): Augmented IPW, combining propensity scores and outcome models for double robustness.

Experimental Setup

Dataset: 677,210 reviews from GreatSchools.org covering 83,795 U.S. K-12 schools (2002–2019).
Topics: "Bullying," "Administration," "Academic Performance," "Extracurricular Activities," and "Curriculum."
Treatment Definition:
- Neutral/Positive topics (e.g., Administration): Treatment is defined by sentiment (all positive vs. all negative).
- Negative topics (e.g., Bullying): Treatment is defined by the presence of the topic.
Semi-Synthetic Validation: To evaluate performance under controlled conditions, the authors injected "academic challenge" text (confounders) into a subset of reviews. They varied the confounder strength ( $p$ ) from 0.5 (random assignment) to 0.9 (strong correlation) and the true ATE ( $u$ ) to test estimator robustness.

Key Results

Semi-Synthetic Data Findings

Confounder Strength: As confounder strength increases, the performance of naive estimators degrades significantly, while CausalBERT-based estimators (IPW, AIPW) maintain robustness.
Temperature Scaling: Scaling significantly reduces the error ratio of IPW estimators, particularly at high confounder strengths ( $p=0.9$ ), by preventing extreme propensity scores. The benefit is less pronounced for AIPW due to its inherent double robustness.
Hyperparameter $\alpha$ : A linear relationship exists between treatment prediction accuracy and confounder strength.
- Low Accuracy (Weak Confounding): Higher $\alpha$ (e.g., 0.8) yields better ATE estimates.
- High Accuracy (Strong Confounding): Moderate $\alpha$ (e.g., 0.5) performs best.
- This allows for an empirical selection of $\alpha$ based on initial model performance.
Interpretability: CLS analysis confirmed that CausalBERT successfully shifts attention to injected confounder tokens when confounding is strong, whereas it focuses on treatment-specific tokens when confounding is weak.

Real-World Data Findings

Applying the optimized framework (using $\alpha=0.8$ and temperature scaling) to the school reviews:

Confounder Adjustment: Adjusted estimates consistently showed smaller effect magnitudes than unadjusted estimates. For example, the negative impact of "bullying" was estimated at -0.16 (adjusted) versus -0.26 (unadjusted), suggesting that poor administration often co-occurs with bullying, inflating the unadjusted correlation.
Significant Drivers: After controlling for confounds, Administration and Academic Performance were identified as the most significant positive drivers of overall school ratings.
Bullying: Remained a significant negative driver, but its estimated impact was moderated by the presence of other factors.

Significance and Claims

The paper claims to advance the field of text-based causal inference by:

Validating CausalBERT: Demonstrating its efficacy in disentangling causal effects from correlated textual features in a real-world, high-stakes domain (education).
Methodological Refinement: Providing concrete, empirically validated strategies (Temperature Scaling and $\alpha$ selection) to improve the stability and accuracy of causal estimates derived from deep learning models.
Interpretability: Offering qualitative methods to verify that the model is indeed learning to control for confounds rather than simply memorizing correlations.
Domain Insight: Revealing that while aspects like bullying are negatively perceived, the perception of school quality is most significantly driven by administration and academic performance, and that failing to control for confounding leads to overestimation of the impact of negative factors.

The authors maintain a modest tone, acknowledging that true causal effects in observational data are unknown and that the reliance on keyword-based treatment identification introduces potential noise. They emphasize that their results should be interpreted as estimates of perceived influence rather than definitive causal truths, and caution against over-reliance on these metrics for administrative decision-making without considering unique school contexts.