Introduction

Deep Neural Networks (DNNs) have achieved significant success in automated decision-making tasks, particularly in image classification tasks. However, their “black box” nature presents challenges in ensuring transparency and interpretability [1, 55]. To address these challenges, explainable AI (XAI) techniques have emerged, providing insight into the rationale behind the model prediction process [9, 45]. Among these techniques, saliency methods have gained considerable attention for their ability to generate visual explanations, enhancing user understanding and trust. By highlighting the regions most influential in model predictions, saliency maps provide valuable information about the model’s focus and rationale [28, 58, 68]. For example, when diagnosing an image as “cancerous,” the model’s reasoning should focus on the lesion areas, rather than on artifacts, ensuring that its prediction is based on correct reasoning.

Given the importance of saliency methods in improving model transparency, it is crucial to assess the quality of the explanations they generate. Evaluation of model explanations involves two key aspects: 1) whether the explanation reflects the true underlying reasoning of the model, and 2) how closely the explanation aligns with the ground truth. To assess these aspects, we evaluate both faithfulness and alignment [14, 29, 33, 41] of visual explanations. Faithfulness measures how accurately the explanation represents the model’s true reasoning process, while alignment gauges how well the explanation corresponds to human understanding or ground truth. Together, these evaluations ensure that the model’s predictions are not only accurate but also interpretable and aligned with human understanding and the true decision-making process.

However, it is challenging to obtain explanation annotations because of prohibitive human effort and additional challenges: (1) Lack of a standardized evaluation framework: Saliency maps are inherently continuous representations, while human annotations are usually discrete or categorical in nature [50, 54]. Such discrepancies, however, hinder meaningful comparisons between AI-generated explanations and human annotations and reduce the reproducibility of results, complicating the integration of saliency methods into real-world applications [6, 30]. (2) Absence of comprehensive and diverse datasets: There are limited benchmark datasets focusing on limited domains, basically medical imaging and object classification [40, 47], which are too narrow in scope to holistically evaluate XAI. Furthermore, those relying on user interfaces [20, 48] or questionnaires [14, 44, 48], are expensive and not scalable. This gap in high-quality, annotated datasets significantly hampers both the development and reliable evaluation of saliency methods. (3) Lack of comprehensive benchmarks and analysis: First, there are very few existing works, each of which, however, provides only one or a couple of datasets for evaluation. For example, [20, 21] introduces two datasets under medical imaging types for evaluating faithfulness but not alignment, while [48] provides a low-resolution image dataset for evaluating object detection but not classification. However, the formats and evaluation metrics of these datasets are arbitrarily different from each other, making it difficult to test saliency methods across different domains and tasks.

In this work, we establish Saliency-Bench, a comprehensive benchmark for visual explanation of image classification tasks. Saliency-Bench is a collection of eight datasets with annotated ground-truth explanations, covering a wide range of topics including gender classification, environment classification, action classification, object classification, cancer diagnosis, disease estimation, pet type classification, and security check classification. These datasets are processed into a unified format, enabling consistent evaluation across different tasks. We conducted extensive benchmarking experiments to evaluate several saliency methods—GradCAM [48], GradCAM++[10], Integrated Gradients [53], InputXGradient [49], Occlusion [62], and RISE [44]—across different backbone classifier architectures, including ResNet-18 and VGG-19. We also benchmark the ViT-B/16’s attention mechanism [15] as a saliency method. For alignment evaluation, we use two key metrics: mean Intersection over Union and Pointing Game [63]. Additionally, we assess the faithfulness of these methods using the insertion Area Under the Curve (iAUC) [44] and conduct an inter-method reliability analysis. Through these experiments, we aim to address the challenges outlined earlier by providing a standardized evaluation framework, a diverse collection of annotated datasets, and comprehensive benchmarks for saliency methods. Our contributions are summarized as follows:

• Comprehensive Dataset Collection: We curated, constructed, and annotated a dataset collection designed to benchmark saliency methods for image classification tasks. The collection includes eight datasets spanning topics such as action classification, tumor classification, and object classification. Each dataset comes with class labels and ground-truth explanation annotations, ranging from small to large scales, and covering both binary and multiclass classification problems.
• Standardized Evaluation Pipeline: We developed a standardized pipeline for holistically evaluating the quality of the explanations. This pipeline unifies both the implementation of evaluation methods and the format of visual explanations generated by different saliency methods, ensuring seamlessly reproducible experiments, efficient comparisons, and streamlined iteration on existing approaches.
• Extensive Benchmarking and Analysis: We conducted extensive benchmarking across our dataset collection to assess the quality of the explanations generated by various saliency methods. Our analysis focuses on key evaluation criteria, including alignment and faithfulness, providing insights into the effectiveness of these visual explanations.
• User-Friendly Evaluation Toolkit: We offer an easy-to-use toolkit with an API for querying and accessing our proposed datasets, standardized dataset loading, and performance evaluation, simplifying the evaluation process for researchers. A detailed tutorial for implementation is also provided in the Appendix A.