From Means to Slopes: Rethinking International Comparisons of Questionnaire Indicators in PISA
This study argues that for non-cognitive PISA questionnaire indicators, international comparisons should shift from relying on potentially misleading national mean rankings to analyzing within-school slopes, which offer a more robust and policy-relevant understanding of how educational systems shape student experiences and achievement.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a detective trying to solve a mystery about how different schools around the world are doing. For years, the main tool detectives used was a giant scoreboard, or a "league table," that simply listed countries from first to last place based on their average scores. It's like ranking basketball teams just by looking at the total points they scored in a season, without caring about who played, how they played, or if the teams were even playing the same game. This works okay for things like math or reading, where the answers are usually right or wrong. But when it comes to feelings, motivation, or how happy students are, it gets tricky. These aren't things you can measure with a ruler; they are like flavors or colors. If you ask people in different cultures how "spicy" their food is, a "very spicy" rating in one country might mean something totally different than in another. This paper is about a group of researchers who decided to stop just looking at the scoreboard and start looking at the relationships between the players and the game. Instead of asking, "Which country has the highest average happiness?", they asked, "How does a student's background or their test scores change their feelings about school?" They wanted to see if the rules of the game change when you zoom out from a single student to a whole country.
The researchers, led by Kit-Tai Hau and his team, took a massive snapshot of data from 612,004 students across 80 countries, collected during the 2018 PISA test. They looked at 31 different questionnaires about things like motivation, well-being, and how students feel about their classrooms. Their first big discovery was that the scoreboard was mostly useless for these feelings. When they crunched the numbers, they found that the differences between countries were tiny—only about 6% of the variation in these feelings happened because of which country a student was in. The other 94% happened within the countries, mostly between different schools. It's like trying to compare the average height of people in France versus Italy and finding that the difference is so small it doesn't matter, while the difference between a basketball team and a gymnastics team in the same city is huge.
But the real magic happened when they stopped looking at averages and started looking at "slopes." Imagine a graph where the line goes up or down. A "slope" tells you how one thing changes as another thing changes. The researchers found that if you just look at the country's average, the story often flips upside down. For example, at the individual student level, doing well in reading usually makes a student feel good about themselves. But when you look at the country level, countries with higher average reading scores sometimes had students who reported lower self-confidence. It's a bit like a "Big Fish, Little Pond" effect: if you are a smart fish in a pond full of other smart fish, you might feel less special than if you were the smartest fish in a small, quiet pond. The country's average hides this because it mixes up all the ponds together.
The paper argues that instead of ranking countries by their average "mood," we should look at these slopes to understand how education systems work. They found that the relationship between a student's background (like their family's wealth) and their school experience is a much better way to judge a system. They discovered that these relationships (the slopes) were mostly consistent in direction—meaning the theory held up—but the strength of these relationships varied from country to country. Interestingly, most of the differences in these slopes weren't between countries, but between schools within the same country. This suggests that if we want to fix problems or improve things, we shouldn't just look at national policies; we need to look at what's happening inside specific schools.
The authors also checked to make sure their findings weren't just a fluke caused by how they counted the data. They tried different ways of weighting the numbers, and the results stayed the same. They also found that some structural features of a country, like how segregated schools are (how much students from different backgrounds are separated), could explain why the slopes were different in some places. For instance, in countries where schools were more mixed, the link between doing well and feeling good was often stronger.
In the end, the paper suggests a major shift in how we think about international education reports. We shouldn't throw away the scoreboard entirely, but we shouldn't rely on it to tell us how happy or motivated students are. The scoreboard is a weak tool for feelings because it ignores the context. The "slope" approach is like switching from a blurry group photo to a high-definition video that shows how students interact with their world. It tells us that the most important differences aren't between nations, but between the schools inside them. By focusing on these relationships rather than just the averages, we can get a clearer, more honest picture of how education systems shape the lives of young people, moving us from simply ranking who is "best" to understanding how to make things better for everyone.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.