Computer Vision XAI libraries: the complete guide
In this post, we review the most widely used libraries built for computer vision and point out their strengths and weaknesses.
Introduction
Before a machine learning model can be put to use, it is more and more often required to verify that it is valid by means of explainable AI techniques. Writing the code that covers this part of a project is complex and takes a lot of time. Luckily for machine learning engineers, there are a number of open-source libraries built primarily to explain trained models with a wide range of algorithms.
In this post, we review the most widely used libraries built for computer vision and point out their strengths and weaknesses.
Methodology
I picked the libraries based on 4 criteria. To begin with, a library had to be written for Python – the most widely used language in machine learning, for which the neural network training libraries relied on by a big community of engineers and researchers have been built. Next, it had to be related to computer vision tasks. Third, I considered the libraries that are most popular as measured by their number of GitHub stars. The final criterion was that the repository is actively maintained: at the time of writing, its most recent commit could be no older than 1 year. The one exception is the M3d-CAM library, where the latest commit is nearly 2 years old.
The repositories covered, ordered from the highest to the lowest number of GitHub ★:
- pytoch-grad-cam
- captum
- pytorch-cnn-visualizations
- torch-cam
- innvestigate
- tf-explain
- OmniXAI
- xplique
- M3d-Cam
Grab a coffee, because this blog post is on the longer side. Happy reading!
If you just want a quick comparison of the libraries, take a look at the tables summarizing their features in the following section.
Comparison
Repository features
Algorithms
Metrics
pytorch-grad-cam 6.7k
Source: the project’s GitHub repository
This repository concentrates primarily on gradient class activations maps (CAM) algorithms for PyTorch computer vision models. It provides 11 different algorithms and has been tested on various architectures: CNNs and Vision Transformers. The authors state that it is suitable for many computer vision tasks, such as classification, object detection, semantic segmentation, and plenty more. A nice feature is attribute heatmap smoothing, which produces a smoother image for the user. One more plus is that the repository includes metrics of trust in the model.Algorithms
The library offers attribute methods for the whole model and for an individual layer, as well as the deep feature factorization method. The repository also includes metrics for XAI methods.
Visualization
For visualization, the developers concentrated on placing the heatmap over the original image. You will also find examples of visualizing attribute heatmaps for semantic segmentation and object detection tasks.
Documentation
The library comes with fairly thorough documentation, full of examples and tutorials. It shows sample outputs of the algorithms, giving you a general idea of what results to expect. Examples are available in the documentation as well. What is missing, though, is a standard API description in which the user could look up a method’s parameters.
Captum 3.7k★
source.
The creators of the library describe their project like this: “Captum is a PyTorch library for interpreting and understanding models.” That one sentence captures what the library is meant for. It implements a number of algorithms in a task-agnostic way, meaning they can be applied to NLP, computer vision, and other tasks alike.
Algorithms
The library offers attribute methods for the whole model, for an individual layer, and for an individual neuron. It also provides perturbation and noise tunnel methods for computing the model’s attributes.
Visualization
The authors of the library describe a widget called “Captum insights” for interactively visualizing model explanations. Sadly, I could not get their example to work after installing the package, either via pypi or with the setup.py script from the cloned repository.
For visualization, the developers chose to show attributes alongside the original image rather than placing the heatmap over it.
Documentation
The authors made sure to deliver clear documentation along with usage examples. The first tab describes each algorithm in detail and includes a summary comparing them. The tutorials tab offers a fairly extensive set of examples showing the library in various domains (CV, NLP) and tasks (image classification and semantic segmentation). The examples cover attribute-based explanation methods as well as techniques like adversarial attacks, influential examples, and concept-based explanations. The library additionally provides several metrics and methods for checking model robustness. An API reference is available too.
pytorch-cnn-visualizations 3.7k★
Source: the project’s GitHub repository
This repository is a collection of examples demonstrating various explainability algorithms for CNN networks. Note: it is not a library. To use it in your own project, you would need to write suitable functions and classes yourself, based on the code provided, that carry out similar operations. Another issue is the rather outdated dependencies: torch in version 0.4.1, etc.
Interface
There is no API – only a set of example scripts.
Algorithms
The library offers attribute methods for the whole model and for an individual layer. On top of that, the repository includes feature visualization algorithms.
Visualization
The example scripts save their artifacts as attribute images or as heat maps placed over the original image.
Documentation
There is no documentation.
torch-cam 1.2k★
Source: the project’s GitHub repository
This is yet another library centered on CAM methods for computer vision. It is simple to use and implements 9 different algorithms.
Algorithms
The library offers attribute methods for the whole model and for an individual layer.
Visualization
A single method is available for visualizing the model’s attributes – a function that shows an attribute heatmap placed over the original image.
Documentation
The repository’s documentation is published via Github Pages. It contains installation instructions, code samples for running the explanation algorithms, tutorials, and an API description.
innvestigate 1.1k★
Source: the project’s GitHub repository
This library aims to make analyzing machine learning models for Keras/TensorFlow2 easier.
Algorithms
The library offers attribute methods for the whole model, usable for image classification tasks only. Besides the algorithms found in nearly every library described here, it provides several variants of the LRP algorithm.
Visualization
A few methods are available for visualizing the model’s attributes, a heatmap among them.
Documentation
Only the API reference has been published for this repository, but the examples directory holds a number of tutorials.
tf-explain 964★
source
This library implements interpretability methods for TensorFlow 2 in the form of a trainer callback, with 7 explainability algorithms available. Its documentation offers handy examples of using the explainers, together with the result shown as an image of the computed attributes. What distinguishes it from the others is its integration with the training process and the option to store artifacts of the explanation algorithms in a training monitoring tool – here, TensorBoard. This can be very helpful when diagnosing issues while a network is being trained.
Interface
Here, too, the library provides a simple, unified, and parameterizable interface for all explanation algorithms and metrics.
There is also a second interface for integrating with the training process. Every class that implements an explanation algorithm serves as a callback (it inherits from tensorflow.keras.callbacks.Callback) for the trainer.
Algorithms
The library offers attribute methods for the whole model, usable for image classification tasks only.
Visualization
The model’s attributes can be visualized in two ways: as an attribute image or as an attributes heatmap placed over the original image.
Documentation
Its documentation is published on ReadtheDocs and includes basic usage examples and a description of the core API, but sadly it contains bugs and the API reference section is empty.
OmniXAI 517★
source
This library offers XAI algorithms for explaining machine learning models built with PyTorch and TensorFlow. It supports tabular data, text, images, and a time series in classification and regression tasks.
Interface
The API here is more complex. The library offers a number of wrapper classes for different data types, such as tabular data, text, images, etc. Each explainer class for a given data type takes many parameters: a list of explainer names, the task type, a preprocessing function, and further parameters.
Algorithms
The repository includes explanation algorithms for numerous data types, but this blog post covers only the computer vision ones. In this area, there are 8 different algorithms.
Visualization
The repository provides a Dash app for visualizing explanations interactively. The authors built the visualizations on the plotly package. Attributes can be shown either as a separate image or as a layer placed over the original image.
Documentation
The repository’s documentation is published via ReadtheDocs. It walks you through the library’s design, architecture, and installation process, and includes examples and an API reference.
Xplique 348★
source
This one is a choice for projects built on the TensorFlow framework. The library’s authors put their goal well: “Xplique (…) is a Python toolkit devoted to explainability, at present built on Tensorflow. The aim of this library is to gather the state of the art in Explainable AI so you can better understand your complex neural network models.” Besides feature attributes and concept-based explanations, the library can also visualize the features a model has learned.
Algorithms
The library offers attribute methods for the whole model and for an individual layer. It also provides metrics indicating how far we can trust the methods that explain the model’s decisions.
Visualization
For visualization, the developers concentrated on placing the heatmap over the original image.
Documentation
The authors delivered clear documentation and many examples of using the library. The developers kept to a more detailed API description, which explains how the individual algorithms and metrics work.
A fairly extensive set of interactive examples in Google Colaboratory is worth praising. Well done! 👏 They show how to use the library to visualize attributes, features, and concepts, and to compute metrics. On top of that, every metric and method has its own example, where the creators explain the algorithm, the API and available parameters, answer FAQs, and share how each parameter influences the algorithm’s behavior. It is a concentrated body of knowledge on how to apply the algorithm correctly.
M3d-Cam 216★
source
This PyTorch library makes it possible to explain a model’s decisions on 3D/ 2D medical images, for classification as well as segmentation tasks.
Algorithms
The library provides 4 algorithms: guided backpropagation, GradCAM, Guided GradCAM, and GradCAM++.
Visualization
The model’s attributes can be visualized in two ways: as an attribute image or as an attributes heatmap placed over the original image.
Documentation
Documentation for the repository has been published, but it consists of the API reference alone. The README links to several tutorials on Google Colab.
Summary
The libraries covered here differ considerably. Some are general-purpose, while others focus on a single kind of task, algorithm, or particular data type. At present, no universal library offers all the algorithms and metrics at once. When picking a library, you have to determine which features your current project actually needs.This shows there is room for new libraries that would address this gap. Clearly, new libraries still have some space to fill.
Only a few libraries offer metrics that indicate how much we can trust explanation algorithms. Integration with experiment trackers, which would let users monitor how a network is being trained, is missing. On top of that, nearly all libraries are restricted to the basic case of image classification. Support for segmentation is absent, and support for object detection is very limited.