Opening the Black Box or Responding to Demands? Rethinking and Reconstructing the European Standard of Algorithmic Explanation
This paper critiques the limitations of the EU's current algorithmic explanation standard and existing mitigation or substitution approaches, proposing instead a demand-oriented framework that reconstructs the standard by categorizing explanation needs based on actor control points and realizing them through feasible substitute schemes.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern world, computers make countless decisions that shape our daily lives, from determining whether we get a loan to deciding which news stories we see. For decades, the prevailing idea was that if a computer made a choice, we should be able to ask it to show its work, much like a student showing their math homework to prove the answer is correct. This concept is known as algorithmic explanation, and it is a cornerstone of privacy laws in the European Union and other regions. The law assumes that if a person is affected by an automated decision, they have a right to understand the logic behind it. However, a significant problem has emerged: the computers making these decisions have changed. The old systems followed clear, step-by-step rules that could be easily traced. Today's most powerful systems, often called neural networks, learn from vast amounts of data in ways that are so complex and interconnected that even their creators cannot fully map out every internal step. This creates a "black box" situation where the input goes in and a decision comes out, but the path in between remains hidden. When the law demands to see the inside of a box that technology cannot fully open, the system breaks down, leading to confusion, legal evasion, and a loss of trust.
A researcher named Feng Zhi, working at the School of Artificial Intelligence and Law in Chongqing, China, has proposed a new way to solve this deadlock. Instead of trying to force open a black box that cannot be opened, the study suggests we should stop asking to see the internal mechanics and start asking questions that match what the people in charge can actually answer. The core finding of this work is that we need to split the demand for an explanation into two different moments in time, based on who is in control of the computer system. The first moment is when the system is being built and set up; the second moment is when the system is actually making a decision for a specific person. By separating these two stages, the author argues we can replace the impossible demand to see the code with practical demands that are just as effective at protecting people's rights.
The paper argues that the current approach, which tries to provide different levels of detail depending on who is asking, fails because it still requires the same impossible technical feat: peering inside the black box. Whether a user asks a simple question or a complex one, the computer still has to reveal its internal logic to answer, which it often cannot do without losing accuracy or revealing trade secrets. Another common idea is to simply trust the system if it seems safe or if the people who built it are qualified. The author rejects this as well, pointing out that a system can be built by good people and still make bad decisions later, or that trusting the builder does not explain why a specific person was denied a loan. The study suggests that simply checking if the final result can be challenged after the fact is also insufficient, because by then, the harm might already be done and impossible to undo.
To fix this, the paper introduces a framework based on "temporal nodes," which are simply the specific points in time when different people hold the reins of control. The first stage is the "behavior stage." This happens before the system is ever used on the public. During this time, the developers and the companies deploying the system have full control. They decide what data to use, how to train the model, and what safety measures to put in place. Because they have control, the demand for an explanation at this stage should not be about the final decision, but about risk. The study proposes that instead of showing the code, these actors should be required to file detailed statements about how they assessed the risks and what steps they took to prevent harm. They must also keep a record of the system's activity logs. This shifts the focus from "how does the math work?" to "did you do your job to make sure this is safe?" If a problem occurs later, these records allow regulators to trace back and see if the creators followed the rules, effectively allocating responsibility without needing to understand the unexplainable math.
The second stage is the "outcome stage." This is when the system is running and making decisions that affect real people. At this point, the developers have lost control; the system is reacting to new data in real-time. Here, the demand for explanation changes. The goal is no longer to check the risk controls, but to build trust in the result. The paper suggests that people do not need to understand the complex internal logic to trust a decision; they need to know if the decision makes sense based on common human experience. For example, if a credit scoring system denies a loan, the user does not need to see the neural network's weights; they need to know if the decision aligns with what society generally considers fair, such as income levels or debt ratios. The study proposes that companies should publish these general social rules so people can compare the decision against them. If a decision deviates from what common sense would suggest, the system must trigger a human review. A real person must then look at the case and provide a clear, reasoned explanation for why the unusual decision was made.
This approach does not ask for less protection; it asks for a different kind of protection that is actually possible to deliver. By replacing the demand to see the internal code with a demand for risk records and a demand for common-sense consistency, the system can function without breaking. The author notes that this is not a new invention but a reorganization of existing legal tools, such as the requirement for companies to keep records of their processing activities. The study emphasizes that this method respects the reality of modern technology while still fulfilling the legal promise of accountability. It moves the conversation away from the impossible task of opening the black box and toward the practical task of ensuring that those who build and use these systems are answerable for their actions and that the results they produce make sense to the people living with them. The research concludes that this shift in perspective is necessary to restore the practical value of algorithmic explanation, ensuring that the law remains relevant in an age of intelligent machines.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.