Is the ACL Responsible NLP Checklist a Box-Ticking Exercise? A Large-Scale Analysis of EMNLP 2025

Imagine the world of Artificial Intelligence (AI) as a massive, bustling construction site. Every day, teams of engineers are building incredible new machines—some that write poetry, others that diagnose diseases, and some that translate languages instantly. But just like any construction project, there are rules. You wouldn't want a builder to ignore safety codes, use unstable materials, or build a bridge that collapses on the first car. In the world of AI research, these "safety codes" are called Responsible NLP Practices. They are a set of guidelines asking researchers to be transparent about how they built their models, to think about who might get hurt by their work, and to make sure their data was gathered ethically.

To make sure everyone follows these rules, the big AI conferences (like the one where this paper was presented) introduced a Checklist. Think of this checklist like a final inspection form a builder must fill out before a building is approved. It asks questions like, "Did you check for safety risks?" "Did you credit the people who made your tools?" and "Did you get permission from the people you studied?" The goal was to turn these big, scary ethical ideas into simple "Yes" or "No" boxes that researchers could tick off, ensuring their work was safe and honest for everyone.

But here is the big question: Are these researchers actually thinking deeply about the answers, or are they just rushing to tick the boxes so they can get their papers published? That is exactly what this paper investigates. The authors, a team of researchers from the University of Aberdeen, decided to play detective. They gathered the checklists from over 3,000 papers submitted to the EMNLP 2025 conference and analyzed them like a forensic team examining a crime scene. They didn't just look at the "Yes" or "No" marks; they looked at the tiny notes (justifications) authors wrote to explain their choices. They wanted to see if the checklist was actually making researchers more responsible, or if it had just become a boring, mindless game of "check-the-box."

The Great "Check-the-Box" Heist

The authors of this study treated the EMNLP 2025 checklists like a giant puzzle. They built a special computer pipeline (a set of automated tools) to read thousands of PDF papers and their corresponding checklists, linking the answers back to the specific sections of the papers where the authors claimed to have done the work. They analyzed a massive 73,922 individual answers and justifications. What they found suggests that for many researchers, the checklist has become less of a serious safety inspection and more of a "box-ticking exercise"—a task done just to get it over with.

The "Afterthought" Problem
One of the most striking findings is that researchers often treat ethics as an afterthought, like remembering to lock the front door only after you've already left the house. The study found that for the "Main Track" of the conference, authors tended to isolate the ethics questions from the rest of their paper. Instead of weaving safety and ethics into the story of their research, they tacked it on at the end. It's as if a chef wrote a recipe for a delicious cake but only mentioned the safety of the oven in a tiny footnote at the very bottom, rather than explaining how they checked the temperature while baking.

The "No" Justification Disaster
When researchers answered "No" to a question (meaning, "No, I didn't do this specific ethical thing"), they were supposed to explain why. This is the most critical part of the checklist, as it forces you to admit what you didn't do. However, the authors found that 44.9% of these "No" justifications were either empty (blank) or incredibly brief (just one or two words like "Not applicable").

Imagine a driver getting pulled over for a broken headlight. When the police officer asks, "Why didn't you fix it?", the driver just shrugs and says, "Whatever," or says nothing at all. That is what nearly half of the researchers did. They didn't explain why they skipped the safety step; they just checked the box and moved on. This is a major problem because without a good explanation, no one knows if the risk was truly low or if the researcher just didn't care.

The "Risks" Blind Spot
Perhaps the most worrying finding concerns the question about potential risks. This is the part where researchers are asked to think: "Could my AI be used to hurt people? Could it spread bias?" The study found that 53% of authors dismissed these risks entirely, claiming their work had "no risks" or "no social impact."

The authors argue this is like a car manufacturer saying, "Our new car has no risk of crashing," without ever testing the brakes. The paper suggests that by simply ticking "No risks" without a deep, honest reflection, researchers are ignoring the real-world dangers of their creations. The checklist design didn't force them to think hard enough; it let them off the hook too easily.

The Logic Loophole
The study also found that the checklists were full of logical contradictions. The checklist is structured like a tree: if you answer "No" to a big question (the parent), all the smaller questions underneath it (the children) should also be "No" or "Not Applicable." But the authors found that 6% of all checklists had logical errors.

For example, a researcher might answer "No" to the question "Did you use any data?" but then answer "Yes" to the child question "Did you describe the data you used?" It's like saying, "I didn't bake a cake," and then immediately saying, "Yes, I used chocolate chips." These contradictions suggest that many authors were filling out the form carelessly, not realizing that their answers didn't make sense. This confusion makes it hard for reviewers to trust the checklist.

Main Track vs. Findings Track
The researchers compared the "Main Track" (the most prestigious papers) with the "Findings Track" (papers with solid but narrower contributions). They expected the Main Track to be more careful. While the Main Track did have slightly better compliance, the bad habits were present in both. Even in the top-tier papers, researchers were skipping ethical reflections and writing empty justifications. This suggests the problem isn't just about the quality of the papers, but about how the checklist itself is being used (or misused) by everyone.

The Verdict: A Broken Inspection Form?

The paper concludes that the current checklist design is failing to do its job. It is not successfully forcing researchers to think about the ethics and risks of their work. Instead, it has become a bureaucratic hurdle that people rush through. The authors point out that the checklist encourages "surface compliance"—where researchers look like they are following the rules without actually doing the hard work of reflection.

The study suggests that the checklist needs a serious makeover. They recommend:

1. Enforcing a minimum word count: If you say "No," you must write a real explanation, not just a shrug.
2. Better design: The form should prevent logical contradictions (like the "No cake, but yes chocolate" problem) by automatically hiding questions that don't apply.
3. More scrutiny: Reviewers should look much closer at the "No risks" answers, because claiming a new technology has zero risk is often a red flag.

In short, the paper argues that we cannot rely on a simple checklist to make AI safe. If the people building the future of technology are just ticking boxes without thinking, the "safety inspection" is a fake. The authors hope that by exposing these flaws, the community can build a better system—one that actually makes researchers pause and ask, "Is this safe?" before they hit publish.