A Comprehensive Benchmark for Evaluating Visual Explanations

In this section, we introduce Saliency-Bench, our proposed benchmark for evaluating visual explanations in XAI. We begin with an overview of Saliency-Bench in Section 4.1, followed by a detailed description of the dataset collection in Section 4.2, which includes both an overview of the datasets and the annotation process. Finally, we discuss the proposed standardized evaluation pipeline to assess visual explanation in Section 4.3.

Overview of Saliency-Bench

To advance research in XAI and saliency methods, we introduce Saliency-Bench, a comprehensive benchmark and dataset collection for evaluating visual explanations, as shown in Figure 2. This framework provides a standardized evaluation paradigm that assesses both the faithfulness and alignment of generated saliency maps. Saliency-Bench comprises a diverse set of datasets across various domains, with sizes ranging from small to large. To ensure consistency and facilitate widespread use, we have unified the data format and developed a modular evaluation pipeline. This standardized pipeline unifies both the implementation of evaluation methods and the format of the visual explanations generated by different saliency methods, ensuring seamlessly reproducible experiments, efficient comparisons, and streamlined iterations on existing approaches. In addition to the benchmarking experiments, we provide an in-depth analysis of the performance of multiple saliency methods, addressing both alignment and faithfulness. Overall, Saliency-Bench offers a scalable, modular, and comprehensive framework for evaluating and improving saliency methods in XAI.

Dataset Collection

In constructing our dataset collection, we curated images from eight datasets across diverse domains, including gender classification, object classification, scene recognition, nodule classification, tumor detection, and action classification, each varying in task complexity and annotation methods to enable a comprehensive evaluation of saliency methods. The annotation approaches were tailored to the nature of each task. For some datasets, ground-truth explanations were provided through human annotation, while others utilized foreground extraction techniques to highlight the most relevant regions in the images. We developed specialized user interfaces (UIs) to facilitate the human annotation process, ensuring consistency and high-quality explanations (see the Appendix for a detailed description of the dataset construction and annotation process). Figure 1 presents example images from each dataset. Table 1 offers a summary of the key characteristics of the datasets, including the class types, sizes, and annotation methods used. The following sections provide more specific details on each dataset.

Gender Classification Dataset (Gender-XAI)

The gender classification dataset is derived from the Microsoft COCO dataset [34]. To construct the dataset, we extracted images from the COCO dataset that contained captions with the terms man or woman. Further filtering was performed to remove images that mentioned both genders in the caption, depicted multiple individuals, or featured unrecognizable human figures. Furthermore, a subset of the images was manually annotated by human annotators using human annotation UIs. The dataset comprises a total of 5,000 images with class labels and 3,454 human explanation annotations, evenly distributed between females and males.

Environment Classification Dataset (Environment-XAI)

The environment classification dataset used in our study is derived from the Places365 dataset [67] and further annotated manually with human annotation UIs. The task of this dataset involves binary classification for scene recognition, specifically distinguishing between natural and urban scenes. To create the dataset, we selectively sampled images from specific categories. Specifically, the categories used to sample the data are: Nature: mountain, pond, waterfall, field wild, forest broadleaf, rainforest; and Urban: house, bridge, campus, tower, street, and driveway. In total, the dataset comprises 5,000 images with class labels and 3,052 human annotations.

Disease Risk Estimation Dataset (Disease-XAI)

We constructed the disease risk estimation dataset from the LIDC-IDRI [4], which comprises thoracic computed tomography (CT) scans from lung cancer screenings annotated with lesion markers. We converted the 3D nodule images into 2D by selecting the central slice along the z-axis and resizing it to 224×224 pixels. Up to four experienced thoracic radiologists provided annotations in XML format for each scan. The ground truth explanation was established by computing a consensus volume from these annotations, with a nodule considered positive if agreed upon by at least 50% of the radiologists. Conversely, negative samples were derived by slicing surrounding areas without nodules. Post-preprocessing [64], the dataset includes 2,625 positive nodule images with human explanation annotations and 2625 negative non-nodule images. The primary objective of utilizing this dataset is to determine the presence or absence of nodules.

Cancer Diagnosis Dataset (Cancer-XAI)

We sourced normal pancreas images from the Cancer Imaging Archive [46]. Abnormal scans, featuring pancreatic tumors, were derived from the Medical Segmentation Decathlon dataset, where initial ground-truth annotations by a medical student were rigorously reviewed and refined by a skilled radiologist. The final cancer diagnosis dataset includes 281 positive scans, identified by the presence of tumors, and 80 negative scans without tumor indications. In a preprocessing approach akin to that used for the LIDC-IDRI dataset, we converted the 3D scans into 2D slices by randomly selecting along the z-axis, thus setting the stage for a binary classification task to discern between positive (tumorous) and negative (normal) pancreatic samples.

Security Check Classification Dataset (Security-XAI)

The security check classification task in our study is constructed using the Sixray dataset [38]. The Sixray dataset, partitioned based on the recognition of prohibited items, comprises an extensive suite of 1,059,231 X-ray images. Each image is annotated at the image level by experienced security inspectors, whose expert annotations we repurpose as human explanation annotations. This approach capitalizes on professional insight, ensuring that our dataset’s annotations reflect real-world classification scenarios and provide a reliable basis for the binary classification of prohibited items. Consequently, the dataset has 17,654 images evenly distributed in positive and negative classes, and each positive image has a corresponding human explanation annotation.

Pet Type Classification Dataset (Pet-XAI)

The pet type classification dataset used in our study is constructed from The Oxford-IIIT Pet Dataset [43]. This dataset, tailored for pet image analysis, contains over 7,000 images across 37 unique categories, each corresponding to different breeds of dogs or cats. For the purposes of our research, we treat pixel-level foreground extractions, which isolate the pet from the background, as proxies for human explanation annotations. These extractions effectively highlight the subject of interest in alignment with the class label, mirroring the focus areas a human annotator might identify when asked to explain the basis for classifying an image as either a dog or a cat. The resulting dataset contains 2,400 images for cat and 4,990 images for dog, with a total of 7,349 human explanation annotations.

Action Classification Dataset (Action-XAI)

The Action Classification Dataset, Action-XAI, is derived from the VQA-based visual and textual explanations dataset, Activity Explanation (ACT-X) [42, 65]. We extracted samples from the ACT-X dataset that could be converted from a VQA task into a classification task. To determine whether a question-answer pair could be restructured into a classification problem, we employed LLMs to verify the suitability of each pair for conversion. The resulting Action-XAI dataset is designed for object classification, where each image sample includes visual explanations that justify the assigned class label. The dataset consists of 127 distinct action class labels, providing a diverse set of actions for classification tasks. In total, the dataset contains 11,511 image samples.

Object Classification Dataset (Object-XAI)

The Object Classification Dataset utilized in our study is constructed from the PASCAL VOC 2012 Dataset [16]. The dataset comprises roughly 11,540 images and covers 20 diverse object categories. These include aeroplanes, bicycles, birds, boats, bottles, buses, cars, cats, chairs, cows, dining tables, dogs, horses, motorbikes, people, potted plants, sheep, sofas, trains, and TV monitors. Each image in the VOC 2012 Dataset is manually annotated with pixel-level region masks and corresponding class labels for identified objects, providing a robust resource for object classification research. In our study, explanation annotations were generated by extracting the pixel-level foreground corresponding to the image label class, leveraging these precise regions as effective proxies for human explanations by directly highlighting the areas most relevant to the object’s classification. The finalized dataset contains 4,318 images, each with a class label and corresponding human explanation annotations.

Standardized Evaluation Pipeline

Saliency-Bench adopts a standardized evaluation pipeline that incorporates a broad range of quantitative metrics for assessing explanation quality. It integrates both alignment-based and faithfulness-based metrics. Our pipeline standardizes the evaluation process by unifying both the implementation of these evaluation methods and the format of the visual explanations generated by different saliency methods. This ensures seamlessly reproducible experiments, efficient comparisons, and streamlined iterations on existing approaches. Below, we provide a detailed overview of these metric categories and their role within our comprehensive evaluation framework.

Alignment-based metrics.

Alignment-based metrics evaluate how well the generated visual explanation aligns with ground-truth explanation annotations. Common alignment metrics include mean Intersection over Union (mIoU) and the Pointing Game (PG).

mIoU

To calculate mIoU, the generated saliency map E is first converted into a binary map B∈{0,1}m×n for each sample, based on a threshold θ. Each pixel in B is assigned a value of 1 if the corresponding value in E exceeds θ, 0 otherwise. The mIoU is then calculated by comparing this binary map with the binary human explanation annotations A, with the formula: where N is the number of samples and Bi and Ai are the binary maps for the i-th sample.

mIoU(E,A)=1N∑i=1NBi∩AiBi∪Ai,

PG

The Pointing Game [63] evaluates whether the peak of the saliency map E for each sample falls within the human-annotated explanation region A. It is defined as: where N is the number of samples, and MaxLoc Ei represents the location of the highest activation in the saliency map E for the i-th sample.

PointingGame=∑i=1N1MaxLocEi∈AiN,

An example of mIoU and PG evaluations is shown in Figure 3.

Faithfulness-based metrics.

Faithfulness-based metrics evaluate the causal influence of highlighted features of the visual explanation on the model’s output. These metrics test whether manipulating the input regions identified as important by the explanation leads to predictable changes in the model’s predictions. The objective is to determine whether the highlighted features are causally linked to the model’s output. In this paper, we implement the iAUC metric to assess faithfulness.

Insertion (iAUC):

The Insertion metric [44] evaluates how much the model’s prediction confidence increases as the most important features, as identified by the explanation, are progressively inserted into a blank image. Starting with an image that contains no features, the most important parts of the image are gradually added. If the explanation is accurate, inserting the highlighted features should significantly boost the model’s prediction confidence. Formally, the iAUC is calculated as: where Py∣Iinsert(r) represents the probability of predicting class y given the input image I, with a fraction r of the most important pixels inserted.

iAUC=∫01Py∣Iinsert(r)dr,