Related Work

In this section, we first provide a brief introduction to the primary approaches for saliency methods. Next, we discuss the evaluation metrics commonly used for assessing the performance of saliency methods. Finally, we review existing datasets and benchmarks for XAI and saliency and highlight the gaps that our proposed benchmark aims to address.

Saliency Methods for Visual Explanation

Saliency methods are essential tools for explaining model decisions in image classification tasks by identifying the regions of an image that most influence a model’s prediction. These methods can be broadly classified into two categories: post-hoc explanations and intrinsic explanations.

Post-hoc saliency methods are applied after the model has been trained, providing explanations that highlight important regions in the image based on the model’s output. Gradient-based methods, such as GradCAM [48], Integrated Gradients [53], and DeconvNet [66], calculate the gradient of the output with respect to the input image, emphasizing regions with the highest gradients. These methods are efficient but can be sensitive to noise in the model and the intricacies of training [52].

Another class of post-hoc methods includes perturbation-based approaches like RISE [44], LIME [39], and MAPLE [37], which perturb parts of the image and measure how the model’s output changes as a result.

Intrinsic saliency methods, on the other hand, are integrated directly into the model’s architecture, providing explanations from within the model itself. Attention-based methods, such as those used in Vision Transformers [15] and Swin Transformers [36], leverage attention mechanisms to reveal which regions of the image the model focuses on during the decision-making process. These methods can be highly informative as they provide direct insight into the model’s internal workings [18, 57].

Evaluation Metrics for Saliency Methods

Various metrics have been proposed to evaluate the alignment and faithfulness of saliency methods, each providing distinct insights into model explanations. Alignment metrics gauge how well saliency maps correspond to human understanding of the model’s decision-making process. Common alignment metrics include mIoU, Pointing Game [63], Shared Interest [7], MAE Error [40], and Human Judgment [14, 20, 44, 48]. Faithfulness metrics assess how accurately saliency maps reflect the model’s true reasoning. Examples of faithfulness metrics include the Insertion Curve (iAUC) [44], Deletion [44], AOPC, and Faithfulness F [56]. In addition to alignment and faithfulness, other evaluation methods, such as False-positives [59], Sensitivity [22], and Stability [60], have been proposed but are less commonly adopted. Additionally, toolkits such as Xplique [17], Captum [32], and Quantus [24] offer automated implementations and evaluations of XAI methods.

Despite the availability of these metrics, there is still no standardized benchmark, making it difficult to assess and compare saliency methods consistently, especially when ground-truth annotations are lacking.

Datasets for XAI and Saliency Benchmarking

Several datasets have been developed to evaluate saliency methods. CLEVR-XAI [5] and VQA-HAT [11] focus on evaluating visual explanations for visual question answering (VQA) tasks, using questions and ground-truth masks. Previous studies have utilized the PASCAL VOC [10] and ImageNet [12] datasets, incorporating multilayer human attention masks aggregated from multiple annotators to evaluate saliency methods [40]. Additionally, some works have employed bounding boxes [7] as a proxy for ground-truth annotations in assessing model explanations. In the medical domain, chest X-ray segmentation datasets, such as CheXpert [27], provide radiologist-annotated segmentations for evaluating saliency methods on medical imaging tasks [47]. FunnyBirds [25] introduces a synthetic vision dataset designed for the automatic and quantitative analysis of XAI methods through image interventions.

Efforts in other domains have focused on datasets for benchmarking saliency methods in NLP, tabular, and graph data. ERASER [13] provides human-annotated rationales for NLP, with additional textual rationales from forums [8, 31, 61]. XAI-Bench [35] offers synthetic datasets for feature attribution in tabular data, while OpenXAI [2] provides real-world datasets and metrics. For graph data, SHAPEGGEN [3] and G-XAI Bench [3] offer synthetic and real-world datasets for evaluating GNN explainers, and Hruska et al. [26] introduced a dataset for chemical property prediction on molecular graphs.

Despite existing efforts, unified, large-scale, diverse, and consistently annotated datasets for evaluating saliency methods, particularly in image classification, remain scarce. Our work fills this gap by providing eight diverse image datasets with human-annotated explanations and a standardized evaluation framework.