Feature/dick/anomaly score normalization #35

djdameln · 2021-12-17T09:41:02Z

Description

This PR adds anomaly score and anomaly normalization. A substantial refactor was needed to make this functionality compatible with the TorchMetrics evaluation design.

Changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Adds NormalizationCallback, which normalizes the predicted scores and error maps based on the distribution of normal data in the training set.
A test case was added for the normalization callback, which asserts that the (image-level) performance with and without normalization is the same.
Normalization is repeated with numpy operations in the post-process method of the openvino inferencer.
Adaptive threshold and training set statistics are now implemented as TorchMetrics classes, which de-clutters the base anomaly module and allows automatic saving and loading of the values in the model's state dict.
F1 score is now computed by using TorchMetrics F1 class configured with the computed threshold, instead of OptimalF1.
Our custom FeatureExtractor was replaced by torchvision's create_feature_extractor. For this I had to bump up the torch and torchvision versions.

Checklist:

My code follows the pre-commit style and check guidelines of this project.
I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing tests pass locally with my changes

samet-akcay

Thanks a lot @djdameln! Looks really good, only minor stuff.

samet-akcay · 2021-12-17T13:10:10Z

anomalib/core/callbacks/__init__.py

@@ -9,6 +9,7 @@

 from .compress import CompressModelCallback
 from .model_loader import LoadModelCallback
+from .normalization import NormalizationCallback


I think NormalizationCallback is not sufficient to understand what the callback does. I think it should be something like ScoreNormalizationCallback or something similar.

Yeah, I tried AnomalyScoreNormalizationCallback but found it too verbose. ScoreNormalizationCallback would be better, though maybe a bit vague. How about OutputNormalizationCallback, because it normalizes the outputs of the model?

I changed it to OutputNormalizationCallback. Let me know what you think

anomalib/core/metrics/training_stats.py

anomalib/data/tiler.py

requirements/requirements.txt

tests/core/callbacks/normalization_callback/test_normalization_callback.py

anomalib/models/padim/model.py

djdameln · 2021-12-17T16:03:20Z

Thanks for the review @samet-akcay. I addressed your comments and updated the PR

anomalib/core/model/anomaly_module.py

samet-akcay · 2021-12-17T16:11:26Z

requirements/requirements.txt

@@ -1,8 +1,9 @@
 pillow==8.3.2
 nncf>=2.0.0
 pytorch-lightning==1.3.6
-torch==1.8.1
-torchvision==0.9.1
+-f https://download.pytorch.org/whl/torch_stable.html


I think we need to remove this line too.

samet-akcay · 2021-12-17T16:14:23Z

anomalib/core/callbacks/normalization.py

+from torch.distributions import LogNormal, Normal
+
+
+class OutputNormalizationCallback(Callback):


How about the name of your PR? :) Like AnomalyScoreNormalizationCallback or ScoreNormalizationCallback. It doesn't matter if it's verbose. What's more important is its readability

samet-akcay

Done with the second round. This will be super cool once merged.

This reverts commit 5d46732.

samet-akcay

Thanks a lot for the effort @djdameln! This is a crucial addition!

samet-akcay · 2021-12-20T14:13:33Z

Maybe we could use version 0.2.3 for this one?

…o feature/dick/anomaly-score-normalization

ashwinvaidya17 · 2021-12-21T10:14:54Z

tests/core/callbacks/normalization_callback/test_normalization_callback.py

+
+
+def test_normalizer():
+    config = get_configurable_parameters(model_config_path="anomalib/models/padim/config.yaml")


Can you add get_dataset_path() to config.dataset.path? Without this, the CI is downloading the entire dataset again in the current directory. Due to the permissions of the original files, the CI cannot remove the dataset once the test runs.

…:openvinotoolkit/anomalib into feature/dick/anomaly-score-normalization

djdameln added 26 commits December 14, 2021 20:55

implement anomaly score and anomaly map normalization

1e2ac6e

switch back to callback design

95c15c1

fix docstrings and typing

cd1612b

always cast thresholds to float

149e026

improve logic of training stats metric

0cdb25c

switch to torchvision feature extractor

122bbb6

update configs

5d82316

merge development

b4246ab

switch back to saving training stats in model

891c709

Tensor -> tensor

95de8fc

loading checkpoint no longer needed

d84ea11

add normalization to inferencer

1cb7a15

subtract image mean from anomaly maps

21dfefb

small refactor

ae70160

switch to torchmetrics design for threshold computation

8416336

import training stats from init

a41d7c1

small fix

1494e5a

switch to torchmetrics for persisting training stats

9a196b2

add test case for normalization callback

ac4573e

fix visualizer

d05acb9

fix mypy issues

54cd83e

fix compression tests

4c1a756

remove print statement

cecb23e

revert checkpoint loading

721ea0b

revert changing weight path

4a3b2b1

Merge branch 'development' into feature/dick/anomaly-score-normalization

416a4c8

djdameln requested review from ashwinvaidya17 and samet-akcay December 17, 2021 09:41

samet-akcay reviewed Dec 17, 2021

View reviewed changes

ashwinvaidya17 approved these changes Dec 17, 2021

View reviewed changes

djdameln added 5 commits December 17, 2021 16:46

rename normalization callback

95067fe

rename anomaly score dsitrbution class

8453260

change function ordering

e30d870

remove cuda version from torch and torchvision

895bf1e

add deprecation warning to feature extractor.

5d46732

samet-akcay reviewed Dec 17, 2021

View reviewed changes

anomalib/core/model/anomaly_module.py Outdated Show resolved Hide resolved

samet-akcay reviewed Dec 17, 2021

View reviewed changes

djdameln added 7 commits December 17, 2021 17:35

training_stats -> training_distribution

918f422

update requirements

a55e6d5

revert to anomalib feature extractor

7a07fa4

workaround for torch 1.8.1 compatibility

9853828

rename normalization callback

aee4802

merge development

497d3bc

Revert "add deprecation warning to feature extractor."

c002f2a

This reverts commit 5d46732.

samet-akcay approved these changes Dec 20, 2021

View reviewed changes

samet-akcay added 3 commits December 21, 2021 00:32

Add batch size support to patchcore

fc01635

Score Normalization doesnt work for Patchcore

fc8b3d9

Merge branch 'development' of github.com:openvinotoolkit/anomalib int…

00dc01f

…o feature/dick/anomaly-score-normalization

ashwinvaidya17 reviewed Dec 21, 2021

View reviewed changes

samet-akcay mentioned this pull request Dec 21, 2021

[OTE_SDK] Update visualizing probability for labels openvinotoolkit/training_extensions#802

Closed

djdameln added 3 commits December 21, 2021 13:44

check to prevent using both normalization and nncf

a56ba72

use get_dataset_path

e23f82d

Merge branch 'feature/dick/anomaly-score-normalization' of github.com…

1ffb0fb

…:openvinotoolkit/anomalib into feature/dick/anomaly-score-normalization

samet-akcay merged commit f14d195 into development Dec 21, 2021

samet-akcay deleted the feature/dick/anomaly-score-normalization branch December 21, 2021 13:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/dick/anomaly score normalization #35

Feature/dick/anomaly score normalization #35

djdameln commented Dec 17, 2021 •

edited

Loading

samet-akcay left a comment

samet-akcay Dec 17, 2021

djdameln Dec 17, 2021

djdameln Dec 17, 2021

djdameln commented Dec 17, 2021

samet-akcay Dec 17, 2021

samet-akcay Dec 17, 2021

samet-akcay left a comment

samet-akcay left a comment

samet-akcay commented Dec 20, 2021

ashwinvaidya17 Dec 21, 2021

		from torch.distributions import LogNormal, Normal


		class OutputNormalizationCallback(Callback):



		def test_normalizer():
		config = get_configurable_parameters(model_config_path="anomalib/models/padim/config.yaml")

Feature/dick/anomaly score normalization #35

Feature/dick/anomaly score normalization #35

Conversation

djdameln commented Dec 17, 2021 • edited Loading

Description

Changes

Checklist:

samet-akcay left a comment

Choose a reason for hiding this comment

samet-akcay Dec 17, 2021

Choose a reason for hiding this comment

djdameln Dec 17, 2021

Choose a reason for hiding this comment

djdameln Dec 17, 2021

Choose a reason for hiding this comment

djdameln commented Dec 17, 2021

samet-akcay Dec 17, 2021

Choose a reason for hiding this comment

samet-akcay Dec 17, 2021

Choose a reason for hiding this comment

samet-akcay left a comment

Choose a reason for hiding this comment

samet-akcay left a comment

Choose a reason for hiding this comment

samet-akcay commented Dec 20, 2021

ashwinvaidya17 Dec 21, 2021

Choose a reason for hiding this comment

djdameln commented Dec 17, 2021 •

edited

Loading