💬 NLP

Robust Explanations for User Trust in Enterprise NLP Systems

This paper proposes a unified black-box robustness evaluation framework for token-level explanations in enterprise NLP, demonstrating through a large-scale cross-architecture study that decoder-based LLMs offer significantly more stable explanations than encoder models, with stability improving alongside model scale, thereby enabling practical cost-robustness tradeoff decisions for compliance-sensitive deployments.

Original authors: Guilin Zhang, Kai Zhao, Jeffrey Friedman, Xu Chu, Amine Anoun, Jerry Ting

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Guilin Zhang, Kai Zhao, Jeffrey Friedman, Xu Chu, Amine Anoun, Jerry Ting

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a busy office. You have two types of assistants helping you make important decisions:

The "Old School" Assistant (Encoder Models): They are fast, cheap, and great at reading short notes. But, if you slightly change a word in a sentence they just read, they might suddenly point to a completely different word as the reason for their decision.
The "New Genie" Assistant (Decoder LLMs): These are the newer, more powerful AI models (like the ones powering advanced chatbots). They are a bit more expensive to run, but they seem to have a much steadier hand. Even if you shuffle the words around or make a typo, they usually stick to the same reason for their decision.

This paper is a report from Workday AI (a company that manages HR and finance software) testing exactly this scenario. They wanted to know: When we move from the old assistants to the new Genies, do the reasons they give us for their decisions become more reliable?

Here is the breakdown of their findings using simple analogies:

1. The Problem: The "Whimsical" Assistant

In the corporate world (like banks or HR departments), you can't just ask an AI, "Why did you reject this loan?" and hope for a consistent answer. You need a written record that makes sense.

The researchers found that the "Old School" assistants were whimsical.

The Analogy: Imagine you ask the assistant, "Why is this employee a good fit?" They point to the word "creative." But if you delete the word "very" from the sentence or swap "creative" with "innovative," the assistant suddenly points to "punctual" instead.
The Result: For the old models, nearly 50% of the time, a tiny change in the input caused the "main reason" to flip to something else. This is bad for trust. If an auditor asks, "Why did you change your mind?" and the AI says, "Because the text changed slightly," that's a compliance nightmare.

2. The Solution: The "Steady" Genie

The researchers tested the new "Genie" models (like Llama and Qwen).

The Analogy: These assistants are like a seasoned judge. If you rephrase a question or make a small typo, the judge still looks at the same core evidence to make a ruling.
The Result: The new models were 73% more stable. If you changed the input, they kept pointing to the same "main reason" most of the time.
The "Size" Effect: They also found that bigger Genies are steadier. A "70 Billion parameter" model (a very large brain) was even more consistent than a "7 Billion parameter" one. It's like having a senior partner review the decision; they are less likely to be swayed by a typo.

3. The Test: How They Did It

Since these companies often can't see the "inside" of the AI (it's a "Black Box" accessed only via an API), they couldn't look at the code to see why the AI decided what it did.

So, they used a "Leave-One-Out" game:

They showed the AI a sentence.
They asked, "What is the most important word here?"
Then, they secretly deleted that word (or swapped it, or shuffled the sentence) and asked again.
The Metric: If the AI pointed to a different word as the most important one, that was a "Flip." They counted how many flips happened.

They ran this test 64,800 times across different types of text (short tweets, news articles, long reviews) to get a massive amount of data.

4. The Trade-Off: Cost vs. Trust

The paper acknowledges that the "Steady Genies" cost more money and take longer to run than the "Old School" assistants.

They propose a Three-Tier Strategy for companies to choose their AI:

Tier 1: Speed First (The Old School): Use the cheap, fast models if you just need a quick guess and don't need to explain the "why" to a regulator. (e.g., "Is this email spam?")
Tier 2: Balanced (The Mid-Size Genie): Use a medium-sized new model if you need to talk to customers and need a decent explanation. (e.g., "Why was this claim denied?")
Tier 3: Regulatory Gold (The Big Genie): Use the massive, expensive models if you are dealing with high-stakes laws, audits, or legal risks. Here, stability is worth the extra cost because you need to prove your decision-making process is consistent.

The Bottom Line

If you are building an AI system for a bank, hospital, or HR department where trust and audits matter, don't just look at accuracy. Look at stability.

The paper concludes that switching to the newer, larger "Decoder" models is like upgrading from a fickle weather vane to a solid compass. It might cost a bit more to buy the compass, but when the wind (or user input) changes, you know exactly which way is North.

1. Problem Statement

Enterprise NLP systems (particularly in finance and HR) increasingly rely on AI for high-stakes decisions (e.g., risk screening, policy compliance). As organizations migrate from encoder-based models (e.g., BERT, RoBERTa) to decoder-based Large Language Models (LLMs) (e.g., Llama, Qwen) for improved accuracy, a critical gap remains: explanation robustness under black-box constraints.

The Constraint: In production, models are often accessed via APIs where internal gradients or hidden states are unavailable. This precludes gradient-based explainers (like Integrated Gradients) and forces reliance on query-based, model-agnostic methods.
The Risk: Existing studies do not sufficiently characterize whether explanations remain stable when inputs undergo realistic user-generated noise (typos, rephrasing, reordering). Unstable explanations undermine auditability, user trust, and regulatory compliance (e.g., EU AI Act requirements for consistent rationales).
The Question: Do decoder LLMs provide more stable explanations than encoder baselines under black-box access, and what are the associated cost-latency tradeoffs?

2. Methodology

The authors propose a unified black-box robustness evaluation framework designed to simulate real-world enterprise deployment constraints.

A. Evaluation Setting

Black-Box Assumption: Models are treated as opaque functions $M: X \to \Delta(Y)$ , accessible only via input-output queries. No internal parameters, gradients, or attention weights are used.
Determinism: Inference is run deterministically (e.g., greedy decoding, temperature 0) to ensure explanation changes are due to input perturbations, not sampling noise.

B. Perturbation Framework

The study introduces realistic, user-like perturbations across three linguistic levels at three severity levels ( $s \in \{0.05, 0.10, 0.20\}$ ):

Character-level: Adjacent character swaps and deletions (simulating typos).
Word-level: Synonym replacement (via WordNet), random word deletion, and local word shuffling.
Sentence-level: Back-translation (e.g., English $\to$ German $\to$ English) to simulate paraphrasing while preserving semantics.

C. Explanation Mechanism

To ensure cross-architecture comparability, the authors use a Unified Leave-One-Out (LOO) Occlusion explainer:

Process: For an input $x$ with $n$ words, the model predicts the class. Then, each word $w_i$ is removed one by one ( $x_{-i}$ ), and the model is queried again.
Metric: The importance score of $w_i$ is the drop in confidence for the original predicted class: $Imp(w_i) = P(\hat{y}|x) - P(\hat{y}|x_{-i})$ .
Output: A ranked list of tokens by importance.

D. Robustness Metric

The primary metric is the Flip Rate (FR):

Defined as the proportion of cases where the top-1 most important token changes after perturbation.
Lower FR indicates higher stability.
Statistical significance is ensured via paired bootstrap resampling (10,000 iterations) with 95% confidence intervals.

E. Experimental Setup

Models: 6 models total.
- Encoders: BERT-base, RoBERTa-base.
- Decoders: Qwen-2.5 (7B, 14B), Llama-3.1 (8B, 70B).
Datasets: SST-2 (short), AG News (medium), IMDB (long).
Scale: 64,800 paired evaluations across 18 model-dataset configurations.

3. Key Results

A. Decoder vs. Encoder Stability

Significant Improvement: Decoder LLMs produce substantially more stable explanations than encoder baselines.
- Average Flip Rate: Encoders average 0.470, while Decoders average 0.125.
- Improvement: A 73.4% relative reduction in flip rates for decoders.
Dataset Impact: The gap is most pronounced on longer texts (IMDB), where encoders degrade significantly (FR ~0.60) due to bidirectional attention distributing importance across many tokens, whereas decoders remain stable.

B. Scaling Effects

Model Size Correlation: Explanation stability improves with model scale within the decoder family.
- 7B/8B models: Average FR $\approx$ 0.149.
- 14B models: Average FR $\approx$ 0.119.
- 70B models: Average FR $\approx$ 0.083.
Gain: A 44% improvement in stability is observed when scaling from 7B to 70B parameters.

C. Perturbation Sensitivity

Destructive Perturbations: Decoders show massive robustness gains against deletion and shuffling (e.g., 76% improvement on word deletion).
Semantic Preservation: Both architectures perform well on synonym replacement and back-translation (low flip rates for both), though decoders still maintain a slight edge.

4. Contributions

Unified Black-Box Protocol: A standardized framework for evaluating token-level explanation robustness without access to model internals, applicable to both encoder and decoder families.
Systematic Cross-Architecture Comparison: The first large-scale study (64,800 cases) comparing encoder classifiers and decoder LLMs specifically on explanation stability under realistic noise.
Empirical Evidence of Stability: Demonstrating that decoder LLMs are not only more accurate but also significantly more robust in their reasoning traces, challenging the assumption that larger models are merely "black boxes" with unpredictable behavior.
Cost-Robustness Tradeoff Curve: Mapping stability gains to inference costs to provide actionable deployment guidelines.

5. Significance and Deployment Framework

The paper proposes a Three-Tier Decision Framework for enterprise NLP deployment based on the tradeoff between Flip Rate (Stability) and Cost:

Tier	Target Use Case	Recommended Model	Avg Flip Rate	Relative Cost
Regulatory	High-stakes audit, compliance, legal	70B Decoder	0.083	636x
Balanced	Customer-facing, general enterprise	7-8B Decoder	0.149	64x
Speed-first	Real-time, informational only	Encoders (BERT)	0.470	1x

Implications:

Compliance: For industries requiring auditable rationales (Finance, Healthcare), encoder models may fail consistency requirements (flipping in ~50% of cases). Decoder models, particularly larger ones, offer the consistency needed for regulatory adherence.
Trust: Stable explanations prevent "reason drift" where minor user edits change the AI's stated rationale, thereby preserving user trust.
Caveat: The authors note that robustness $\neq$ faithfulness. A model can be consistently wrong. However, robustness is a necessary first-order criterion for trustworthy systems.

Conclusion

The study concludes that the migration to decoder-based LLMs in enterprise settings offers a dual benefit: improved predictive accuracy and, crucially, significantly more robust explanations under black-box constraints. This stability scales with model size, providing a quantitative basis for selecting models that balance regulatory compliance, user trust, and computational cost.