tf-keras-vis

Web documents

https://keisen.github.io/tf-keras-vis-docs/

Overview

tf-keras-vis is a visualization toolkit for debugging tf.keras.Model in Tensorflow2.0+. Currently supported methods for visualization include:

• Feature Visualization
  • ActivationMaximization (web, github)
• Class Activation Maps
  • GradCAM (paper)
  • GradCAM++ (paper)
  • ScoreCAM (paper, github)
  • Faster-ScoreCAM (github)
  • LayerCAM (paper, github) 🆕⚡
• Saliency Maps
  • Vanilla Saliency (paper)
  • SmoothGrad (paper)

tf-keras-vis is designed to be light-weight, flexible and ease of use. All visualizations have the features as follows:

• Support N-dim image inputs, that's, not only support pictures but also such as 3D images.
• Support batch wise processing, so, be able to efficiently process multiple input images.
• Support the model that have either multiple inputs or multiple outputs, or both.
• Support the mixed-precision model.

And in ActivationMaximization,

• Support Optimizers that are built to tf.keras.

Visualizations

Dense Unit

Convolutional Filter

Class Activation Map

The images above are generated by GradCAM++.

Saliency Map

The images above are generated by SmoothGrad.

Usage

ActivationMaximization (Visualizing Convolutional Filter)

import tensorflow as tf
from tensorflow.keras.applications import VGG16
from matplotlib import pyplot as plt
from tf_keras_vis.activation_maximization import ActivationMaximization
from tf_keras_vis.activation_maximization.callbacks import Progress
from tf_keras_vis.activation_maximization.input_modifiers import Jitter, Rotate2D
from tf_keras_vis.activation_maximization.regularizers import TotalVariation2D, Norm
from tf_keras_vis.utils.model_modifiers import ExtractIntermediateLayer, ReplaceToLinear
from tf_keras_vis.utils.scores import CategoricalScore

# Create the visualization instance.
# All visualization classes accept a model and model-modifier, which, for example,
#     replaces the activation of last layer to linear function so on, in constructor.
activation_maximization = \
   ActivationMaximization(VGG16(),
                          model_modifier=[ExtractIntermediateLayer('block5_conv3'),
                                          ReplaceToLinear()],
                          clone=False)

# You can use Score class to specify visualizing target you want.
# And add regularizers or input-modifiers as needed.
activations = \
   activation_maximization(CategoricalScore(FILTER_INDEX),
                           steps=200,
                           input_modifiers=[Jitter(jitter=16), Rotate2D(degree=1)],
                           regularizers=[TotalVariation2D(weight=1.0),
                                         Norm(weight=0.3, p=1)],
                           optimizer=tf.keras.optimizers.RMSprop(1.0, 0.999),
                           callbacks=[Progress()])

## Since v0.6.0, calling `astype()` is NOT necessary.
# activations = activations[0].astype(np.uint8)

# Render
plt.imshow(activations[0])

Gradcam++

import numpy as np
from matplotlib import pyplot as plt
from matplotlib import cm
from tf_keras_vis.gradcam_plus_plus import GradcamPlusPlus
from tf_keras_vis.utils.model_modifiers import ReplaceToLinear
from tf_keras_vis.utils.scores import CategoricalScore

# Create GradCAM++ object
gradcam = GradcamPlusPlus(YOUR_MODEL_INSTANCE,
                          model_modifier=ReplaceToLinear(),
                          clone=True)

# Generate cam with GradCAM++
cam = gradcam(CategoricalScore(CATEGORICAL_INDEX),
              SEED_INPUT)

## Since v0.6.0, calling `normalize()` is NOT necessary.
# cam = normalize(cam)

plt.imshow(SEED_INPUT_IMAGE)
heatmap = np.uint8(cm.jet(cam[0])[..., :3] * 255)
plt.imshow(heatmap, cmap='jet', alpha=0.5) # overlay

Please see the guides below for more details:

Getting Started Guides

• Saliency and CAMs
• Visualize Dense Layer
• Visualize Convolutional Filer

[NOTES] If you have ever used keras-vis, you may feel that tf-keras-vis is similar with keras-vis. Actually tf-keras-vis derived from keras-vis, and both provided visualization methods are almost the same. But please notice that tf-keras-vis APIs does NOT have compatibility with keras-vis.

Requirements

• Python 3.7+
• Tensorflow 2.0+

Installation

• PyPI

$ pip install tf-keras-vis tensorflow

• Source (for development)

$ git clone https://github.com/keisen/tf-keras-vis.git
$ cd tf-keras-vis
$ pip install -e .[develop] tensorflow

Use Cases

• chitra
  • A Deep Learning Computer Vision library for easy data loading, model building and model interpretation with GradCAM/GradCAM++.

Known Issues

• With InceptionV3, ActivationMaximization doesn't work well, that's, it might generate meaninglessly blur image.
• With cascading model, Gradcam and Gradcam++ don't work well, that's, it might occur some error. So we recommend to use FasterScoreCAM in this case.
• channels-first models and data is unsupported.

ToDo

• Guides
  • Visualizing multiple attention or activation images at once utilizing batch-system of model
  • Define various score functions
  • Visualizing attentions with multiple inputs models
  • Visualizing attentions with multiple outputs models
  • Advanced score functions
  • Tuning Activation Maximization
  • Visualizing attentions for N-dim image inputs
• We're going to add some methods such as below
  • Deep Dream
  • Style transfer