Evaluation Cards for XAI Metrics

Technical Summary: Evaluation Cards for XAI Metrics

Problem Statement
The field of Explainable AI (XAI) evaluation is currently fragmented and lacks standardization. Despite a growing body of literature attempting to systematize XAI evaluation metrics, three primary issues persist:

1. Terminological Inconsistency: Identical metric names are used for different operational definitions, while the same underlying properties are labeled differently across research groups.
2. Contextual Omission: Metrics are frequently introduced without declaring the specific properties they target, the grounding level (functionally-, human-, or application-grounded), or the evaluation context (model architecture, data modality). This leads to results reported without the necessary information for reproducibility or meaningful comparison.
3. Validation and Implementation Gaps: Many proposed metrics remain theoretical definitions without accompanying code. Furthermore, validation evidence (such as sensitivity or stability analysis) is rarely provided, and risks of "gaming" the metric (achieving high scores without improving the target property) are seldom acknowledged.

These issues result in a field where evaluation practices are skewed toward functionally-grounded proxy tasks, while human-grounded and application-grounded methods remain underrepresented. The lack of structured reporting prevents meta-analysis and obscures the validity of claims regarding explanation quality.

Methodology
This paper employs a meta-review approach, analyzing eleven surveys on XAI evaluation metrics published between 2021 and 2025. The authors synthesize recurring gaps identified across these surveys to characterize the current state of the field. Rather than proposing a new evaluation metric or a new taxonomy, the authors propose a documentation framework.

The core methodology is the design of the XAI Evaluation Card, a structured documentation template analogous to Model Cards and Datasheets for Datasets. The card is designed to accompany any study that introduces a new XAI evaluation metric, forcing authors to explicitly declare critical information that is currently often omitted.

Key Contributions
The primary contribution is the XAI Evaluation Card, a four-section template designed to standardize the reporting of XAI evaluation metrics:

1. Identity: Requires the metric name, a list of all explainability properties it operationalizes (with references to definitions), and a declaration of its grounding level (functionally-grounded, human-grounded, or application-grounded).
2. Scope and Context: Mandates a description of the evaluation context (model architecture, data modality, local vs. global scope) and an explicit enumeration of all metric assumptions (e.g., feature independence, linearity, availability of baselines).
3. Implementation and Validation: Requires authors to state implementation availability (with links), summarize validation evidence (sensitivity, stability, correlation), describe gaming risks, and document known failure cases.
4. Relationships and Limitations: Asks authors to list relationships to other metrics targeting the same property, note agreements or disagreements, prioritize properties for specific deployment scenarios in case of conflict, and explicitly state limitations and contexts where the metric should not be used.

The paper also outlines a strategy for adoption, suggesting integration with existing reproducibility checklists at major venues, the use of lightweight templates (Markdown, LaTeX, JSON), and LLM-assisted drafting to reduce authoring costs.

Results
As a proposal paper, this work does not present experimental results or empirical validation of a new metric. Instead, the "results" are the identification of specific, recurring failure modes in the literature that the Evaluation Card is designed to address:

• The conflation of properties and metrics.
• The reporting of metric scores in isolation without context.
• The lack of sensitivity and stability analysis in original proposals.
• The inconsistency in reporting implementation availability.
• The failure to acknowledge metric disagreements or gaming risks.

The paper argues that the template directly operationalizes solutions to these five identified problems.

Significance and Claims
The paper claims that the XAI Evaluation Card addresses a central, under-addressed problem: the lack of transparency in evaluation reporting. By adopting this template as a community norm, the authors argue that the field can achieve:

• Reduced Fragmentation: Standardizing documentation to make metrics comparable.
• Improved Accountability: Making explicit the assumptions, limitations, and validity conditions of evaluation claims.
• Enhanced Meta-Analysis: Creating structured, machine-readable records that allow for the aggregation and comparison of evaluation results across studies.
• Contextual Validity: Ensuring that metric values are never reported in isolation, but always accompanied by the context required for interpretation.

The authors emphasize that the card is non-prescriptive; it does not mandate specific metrics or grounding levels but rather requires the explicit declaration of whatever choices the authors have made. The ultimate goal is to shift evaluation norms toward greater rigor and reproducibility without imposing a single evaluation paradigm.