Robust Explanations for User Trust in Enterprise NLP Systems

[Submitted on 13 Apr 2026 (v1), last revised 16 Jul 2026 (this version, v4)]

View PDF HTML (experimental)

Abstract:Robust explanations are increasingly required for user trust in enterprise NLP, yet pre-deployment validation is difficult in the common case of black-box deployment (API-only access) where representation-based explainers are infeasible and existing studies provide limited guidance on whether explanations remain stable under real user noise, especially when organizations migrate from encoder classifiers to decoder LLMs. To close this gap, we propose a unified black-box robustness evaluation framework for token-level explanations based on leave-one-out occlusion, and operationalize explanation robustness with top-token flip rate under realistic perturbations (swap, deletion, shuffling, and back-translation) at multiple severity levels. Using this protocol, we conduct a systematic cross-architecture comparison across three benchmark datasets and six models spanning encoder and decoder families (BERT, RoBERTa, Qwen 7B/14B, Llama 8B/70B; 64,800 cases). We find that decoder LLMs produce substantially more stable explanations than encoder baselines (73% lower flip rates on average), and that stability improves with model scale (44% gain from 7B to 70B). Finally, we relate robustness improvements to inference cost, yielding a practical cost-robustness tradeoff curve that supports model and explanation selection prior to deployment in compliance-sensitive applications.

Submission history

From: Guilin Zhang [view email]
[v1] Mon, 13 Apr 2026 21:19:59 UTC (358 KB)
[v2] Sun, 26 Apr 2026 01:13:29 UTC (361 KB)
[v3] Tue, 14 Jul 2026 19:25:49 UTC (360 KB)
[v4] Thu, 16 Jul 2026 18:45:20 UTC (360 KB)