From Plausible to Actionable: A Position on LLM Self-Explanations

Imagine you are walking through a vast, digital library where the books are written by a super-smart robot that has read almost everything ever published. This robot is a Large Language Model (LLM). When you ask it a question, it doesn't just spit out an answer; it often writes a little note explaining why it chose that answer. In the world of computer science, this is called "Explainable AI" (XAI). For a long time, scientists have been obsessed with two big questions about these notes: Are they plausible? (Do they sound like something a human would say and make sense to us?) and are they faithful? (Do they actually tell the truth about how the robot's brain worked, or is it just making up a story to look good?)

Why does this matter? Because we are starting to let these robots help us make big decisions, like diagnosing illnesses or spotting hate speech. If the robot says, "I think this patient has a fever because of X," we need to know if it's actually looking at X, or if it's just guessing and then inventing a reason that sounds smart. If the explanation is a lie, we might trust the robot too much and make a mistake. But what if the robot is lying, but the lie is so convincing that we can't tell the difference? That is the tricky puzzle this paper tackles.

---

The Robot's "Just-So" Story

This paper argues that when these AI robots explain their own choices, they are like a very charming, very confident storyteller who might be making things up as they go along. The authors, a team of researchers from the Netherlands and Italy, suggest that we need to stop obsessing over whether the robot's story is a perfect, literal transcript of its internal gears turning. Instead, we should ask: Is the story useful enough to help us make a good decision?

Here is the breakdown of their argument, served with a side of metaphors:

The "Too Good to Be True" Trap
The paper points out that these AI explanations are often incredibly plausible. They sound great! They use the right words, they flow well, and they flatter the person reading them (a behavior the authors call "sycophancy," which is like a dog wagging its tail just to get a treat). Because they sound so human and logical, we tend to believe them.

However, the authors argue these explanations are questionably faithful. Imagine a magician pulling a rabbit out of a hat. If the magician says, "I used a secret trapdoor in the floor," that might be a plausible story. But if the rabbit actually came out of a sleeve, the story isn't faithful to the truth. The paper suggests that LLMs are like that magician. They don't actually "look inside" their own code to see how they made a decision. Instead, they generate the explanation at the exact same time they generate the answer. It's like a student writing an essay and then, while writing the conclusion, inventing a reason for the thesis that sounds smart but wasn't actually the reason they started writing.

Why the Old Tests Don't Work
Scientists have been trying to test if these explanations are "faithful" using old-school methods, like poking the robot and seeing if it changes its mind. The paper says these tests are broken for modern AI.

• The "One-Size-Fits-All" Problem: Old tests assume that if you change a word in the input, the robot's reasoning should change in a predictable, straight line. But these robots are sensitive to tiny things, like a missing comma or a capital letter. It's like trying to test a car's engine by tapping the horn; the car might stop, but not because the engine is broken.
• The "Memory vs. Input" Problem: Sometimes, the robot ignores the question you asked and just answers from its own memory. If you try to test its faithfulness by removing words from your question, the robot might just say, "I know the answer anyway!" and keep going. This makes the old tests useless.

The New Goal: Actionability
So, if we can't prove the robot is telling the literal truth about its brain, should we throw the explanations away? The authors say no. They propose a shift in focus from "Is this true?" to "Is this actionable?"

Think of the explanation not as a technical manual of the robot's brain, but as a tour guide.

• The Tour Guide Metaphor: You don't need the tour guide to be a literal map of the underground tunnels to enjoy the tour. You just need the guide to point out the interesting rocks, warn you about the slippery path, and help you decide where to go next.
• The "Advocate" Role: The paper suggests we should treat the AI as a "devil's advocate" or a "consultant." Even if the AI's reasoning isn't a perfect mirror of its code, its explanation can still help a human doctor, a lawyer, or a teacher spot potential errors, consider different viewpoints, or understand the uncertainty in the answer.

What This Means for Us
The authors aren't saying the robots are perfect truth-tellers. In fact, they warn that if we trust these "plausible but unfaithful" stories too much, we might fall into "automation bias"—where we blindly follow the robot even when it's wrong.

Instead, they suggest a new way to use these tools:

1. As a Translator: Use the AI to turn complex, scary technical data into simple language that a regular person can understand.
2. As a Thinking Partner: Use the AI to surface different arguments and perspectives, helping humans make the final decision, rather than letting the robot decide for us.
3. As a Deliberation Tool: Let different AI models argue with each other (one playing the prosecutor, one the defense attorney) to help humans see the full picture.

In short, the paper argues that we should stop trying to force the robot to be a perfect, honest diary of its own thoughts. Instead, we should treat its explanations as a powerful, conversational tool that helps us think better, make safer choices, and take the right actions, even if the robot's internal story is a bit of a fabrication. The goal isn't to know exactly how the machine thinks; it's to make sure the machine helps us think better.