Who Gets Heeded? An Obligation-Level Audit of Responsiveness in EPA Rulemaking

Imagine the U.S. government as a giant, bustling kitchen where the chefs (the agencies) are constantly cooking up new recipes called "rules." These rules tell everyone from big factories to backyard gardeners exactly what they can and cannot do. To make sure these recipes are fair, the government has a special tradition called "notice-and-comment." It's like putting a draft recipe on the table and saying, "Hey everyone, tell us what you think!" In theory, this is a perfect democracy: a single person with a handwritten note has the exact same right to speak as a massive corporation with a team of lawyers.

But here's the tricky part: just because everyone can speak doesn't mean everyone is heard. For decades, scientists have been trying to figure out if these public comments actually change the final recipe or if the government just nods politely and keeps cooking the same way. The big question is: Does the government listen to the loudest voices, the most organized groups, or the people with the best arguments? And more importantly, does the system treat a regular person the same as a giant corporation? This paper dives deep into that mystery, not by looking at the whole recipe, but by zooming in on the tiny, specific ingredients—the "obligations"—that make up the rules.

---

The Great Recipe Audit: Who Actually Gets Heard?

Imagine you are trying to fix a massive, 200-page instruction manual for a video game. Instead of reading the whole thing, you decide to look at every single specific rule inside it: "You must jump over this pit," "You cannot use this weapon," "You must collect 50 coins." These specific rules are what the authors call regulatory obligations.

The researchers, Jianing Fan and Yue Yao, decided to audit the U.S. Environmental Protection Agency (EPA) to see if the public comments actually changed these specific rules. They didn't just look at the final manual; they looked at the draft version, the final version, and the thousands of notes people wrote in between. They built a high-tech, AI-assisted system to act like a super-fast librarian. This system read 786,197 comments (that's nearly 800,000 notes!) and matched them to the specific rules people were talking about. Then, it checked: Did the rule change? Did it stay the same? Did it get deleted?

Here is what they found, broken down into three big discoveries:

1. Speaking Up Helps, But Not by Much
The first finding is a bit of a "yes, but..." The researchers found that when a specific rule got a lot of attention (five or more people commenting on it), it was more likely to get tweaked. Specifically, rules with high engagement were revised 19.5 percentage points more often than rules nobody talked about.

• The Analogy: Imagine a teacher grading a test. If five students raise their hands and say, "Hey, question 4 is confusing," the teacher is almost twice as likely to change the answer key for that specific question compared to a question nobody mentioned.
• The Catch: Even with all that noise, most rules that people complained about didn't change, and most rules that did change were ones nobody even commented on. The comments act more like a "Hey, look at this!" signal than a magic wand that forces a change.

2. Being Angry or Happy Doesn't Matter
You might think that if everyone is screaming "NO!" at a rule, the government will change it, but if everyone is cheering "YES!", the rule will stay. The researchers tested this, and the answer was a flat no.

• The Finding: Whether people were supporting a rule or opposing it, the rate of change was basically the same. About 62% of rules with opponents were revised, and about 68% of rules with supporters were revised. The difference wasn't statistically significant.
• The Takeaway: The government isn't just tallying votes like a popularity contest. They seem to be looking at other things, like technical details or internal reviews, rather than just counting how many people are mad or happy.

3. The "Big Players" Play a Different Game
This is the most surprising and important finding. The researchers looked at who was commenting. They tried to figure out if the comment came from a regular person or a big organization (like a company or a non-profit).

• The Discovery: Big organizations and regular people don't just disagree on different rules; they seem to be playing on completely different fields. The data shows that organizations tend to focus on rules that end up getting editorial tweaks (small wording changes, like fixing a typo or clarifying a sentence). However, the study could not confirm if regular people were more likely to be associated with substantive changes (big, meaningful shifts) within the same rulemaking. Instead, the pattern suggests that organizations and regular people simply choose to engage with different types of rules in the first place. The "inequity" isn't that the government treats a specific rule differently depending on who comments on it; it's that regular people often lack the resources to even find and fight the specific, tiny rules where the real changes happen, while organizations dominate the rules that only get minor edits.
• The Analogy: Imagine a game of chess. The big organizations are the grandmasters who are very good at spotting tiny, technical mistakes in the rules and getting them fixed. Regular people are the ones trying to change the game entirely, but they often end up playing on different boards where the rules are already set. The study suggests that the "inequity" isn't that the government ignores regular people within a specific rule; it's that regular people often lack the resources to even find and fight the specific, tiny rules where the real changes happen.

The "Magic" Tool and Its Glitch
To do all this, the researchers used a powerful AI tool (a Large Language Model) to read the rules and match them to comments. They were very careful to check if the AI was doing a good job. They hired human experts to double-check the AI's work.

• The Good News: The AI was amazing at finding the rules and matching them to comments. It agreed with human experts almost perfectly.
• The Bad News: The AI was terrible at telling the difference between a "small wording change" and a "big meaningful change." It thought a rule was just "edited" when humans could see it was actually "changed."
• The Fix: When the researchers corrected the AI's mistakes with human judgment, the main finding about organizations vs. regular people actually got stronger (the odds ratio jumped from 3.07 to 4.90), proving that the pattern was real and not just a computer glitch.

The Bottom Line
The paper concludes that the "fairness" problem in rulemaking isn't just about who gets to speak. It's about who has the tools to understand the tiny, complex legal ingredients in the recipe. Big organizations have the resources to find and fix the small, technical details, while regular people often struggle to even find the right place to shout. The government listens to the volume of the crowd, but the type of change that happens depends on who has the resources to navigate the complex legal maze in the first place.

So, while everyone has the same right to speak, the paper suggests that the playing field is still tilted. The real inequality isn't that the government ignores you; it's that you might not even know which specific rule you need to fight to make a difference.