Beyond Explanation: Evidentiary Rights for Algorithmic Accountability
This paper argues that algorithmic accountability requires shifting from a focus on explanation to the establishment of evidentiary and counterfactual interrogation rights, as empirical analysis of 168 legal cases reveals that meaningful contestation is nearly impossible without access to evidence, regardless of transparency.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The Big Problem: The "Black Box" Dinner
Imagine you go to a fancy restaurant and order a steak. The waiter brings it out, but it's burnt. You ask, "Why is this burnt?"
The chef (the algorithm) sends a note: "The steak was cooked for 15 minutes at 400 degrees."
You read the note. You understand the explanation. But you still can't fix it. You don't know:
- Was the thermometer broken?
- Did the chef use the wrong cut of meat?
- If you asked for 10 minutes instead of 15, would it have been perfect?
Because you can't see the kitchen, you can't prove the chef made a mistake. You just have to eat the burnt steak or leave without a refund.
This is the current state of AI. When an algorithm denies you a loan, a job, or a house, companies give you a "reason" (e.g., "Your debt-to-income ratio is too high"). You understand the reason, but you can't contest it. You can't check the math, you can't test if a small change would fix it, and you can't prove the system is biased.
The Paper's Solution: The "Test Kitchen"
The author, Matthew Stewart, argues that we need to stop focusing just on explaining the decision and start giving people the right to test the decision.
He proposes a new right called Counterfactual Interrogation.
The Analogy:
Instead of just reading the chef's note, imagine the law says: "If you are denied a steak, you get 30 minutes in the kitchen. You can't see the secret recipe (the code), but you can try cooking the steak with different settings to see what happens."
- You try: "What if I cook it for 10 minutes?" -> Result: Perfect.
- You try: "What if I use a different cut?" -> Result: Perfect.
- You try: "What if I change my name on the order form?" -> Result: Still burnt.
If the outcome changes just by changing a tiny, irrelevant detail (like the name or the cooking time), you have evidence that the system is broken or biased. You don't need to know how the chef thinks; you just need to see that the result is unfair.
The Evidence: What the Data Says
The author looked at 168 real-life court cases where people fought against algorithmic decisions (like being fired, denied a loan, or arrested).
He found a massive pattern, which he calls the "Two-Gate" System:
Gate 1: The Evidence Gate. Can the person get into the kitchen to test the system?
- Without access: 91% of people lost. They couldn't prove anything, so the judge threw out the case.
- With access: 93% of people won. Once they could test the system and show the results, the companies had to admit they were wrong.
Gate 2: The Law Gate. Even if you prove the system is broken, does the law actually care?
- The author found one exception: Social Media Platforms. Even when people proved Facebook or Google's algorithms were harmful, they lost because of a specific law (Section 230) that protects platforms from being sued for what users post. This shows that having evidence is necessary, but sometimes the law needs to change too.
Why "Explanation" Isn't Enough
The paper makes a clear distinction between two things:
- Explanation: "Here is why we said no." (Like the waiter's note).
- Contestation: "Here is proof that your 'no' was wrong." (Like the test kitchen results).
Currently, laws like GDPR (in Europe) and FCRA (in the US) focus on Explanation. They say, "You must tell the person why."
The author says: That's not enough. You can give a perfect explanation for a wrong decision. You need Evidentiary Rights—the right to verify, compare, and test.
How It Would Work in Real Life
Imagine you are a 58-year-old woman applying for a job. The AI rejects you with a score of 0.42 (you needed 0.60).
Under the new system:
- You get a link to a "Test Portal."
- You submit a fake application where you change your graduation year from 1991 to 2011 (making you look younger).
- The Result: Your score jumps to 0.71, and you get the job!
- The Proof: You now have a document showing the AI is judging you based on your age, not your skills.
- The Outcome: You file an appeal with this proof. The company has to explain why age matters, or they have to fix the system.
The Bottom Line
We are trying to fix a broken system by just asking for better instructions. This paper says we need to give people tools to break the system and prove it's broken.
- Current State: "Trust us, the computer said no."
- Proposed State: "Show us the test results. If the computer fails the test, we fix it."
It's about moving from blind trust to verifiable fairness. You don't need to know how the engine works to know if the car is safe; you just need to be able to take it for a test drive.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.