Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Re introduce adaptive scheduling for training #2541

Merged
merged 2 commits into from
Oct 11, 2023
Merged

Re introduce adaptive scheduling for training #2541

merged 2 commits into from
Oct 11, 2023

Conversation

jaegukhyun
Copy link
Contributor

Summary

CVS-122171

How to test

Unit, Intg

Checklist

  • I have added unit tests to cover my changes.​
  • I have added integration tests to cover my changes.​
  • I have added e2e tests for validation.
  • I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
  • I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
  • I have linked related issues.

License

  • I submit my code changes under the same Apache License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below).
# Copyright (C) 2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

@github-actions github-actions bot added ALGO Any changes in OTX Algo Tasks implementation TEST Any changes in tests labels Oct 11, 2023
@goodsong81 goodsong81 added this to the 1.4.3 milestone Oct 11, 2023
@goodsong81 goodsong81 added FIX defect fix RELEASE labels Oct 11, 2023
@jaegukhyun jaegukhyun marked this pull request as ready for review October 11, 2023 03:20
@jaegukhyun jaegukhyun requested a review from a team as a code owner October 11, 2023 03:20
goodsong81
goodsong81 approved these changes Oct 11, 2023
View reviewed changes
Copy link
Contributor

@goodsong81 goodsong81 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the quick fix :)

@jaegukhyun jaegukhyun merged commit b152d9e into openvinotoolkit:releases/1.4.0 Oct 11, 2023
13 checks passed
@jaegukhyun jaegukhyun mentioned this pull request Oct 11, 2023
Merged
8 tasks
sovrasov added a commit that referenced this pull request Nov 27, 2023
@sovrasov @yunchu
@sungchul2 @vinnamkim @negvet @goodsong81 @eunwoosh @jaegukhyun @sungmanc @eugene123tw @wonjuleee @djdameln @GalyaZalesskaya @harimkang @ashwinvaidya17
Update Intel devices branch to the latest develop state (#2666)
2239ac1 
* Update base.txt

updated dependency version of datumaro

* Update __init__.py

update version string

* Update requirements.txt

* Temporarily skip visual prompting openvino integration test (#2323)

* Fix import dm.DatasetSubset (#2324)

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Fix semantic segmentation soft prediction dtype (#2322)

* Fix semantic segmentation soft prediction dtype

* relax ref sal vals check

---------

Co-authored-by: Songki Choi <songki.choi@intel.com>

* Contrain yapf verison lesser than 0.40.0 (#2328)

contrain_yapf_version

* Fix detection e2e tests (#2327)

Fix for detection

* Mergeback: Label addtion/deletion 1.2.4 --> 1.4.0 (#2326)

* Make black happy

* Fix conflicts

* Merge-back: add test datasets and edit the test code

* Make black happy

* Fix mis-merge

* Make balck happy

* Fix typo

* Fix typoi

---------

Co-authored-by: Songki Choi <songki.choi@intel.com>

* Bump datumaro up to 1.4.0rc2 (#2332)

bump datumaro up to 1.4.0rc2

* Tiling Doc for releases 1.4.0 (#2333)

* Add tiling documentation

* Bump otx version to 1.4.0rc2 (#2341)

* OTX deploy for visual prompting task  (#2311)

* Enable `otx deploy`

* (WIP) integration test

* Docstring

* Update args for create_model

* Manually set image embedding layout

* Enable to use model api for preprocessing
- `fit_to_window` doesn't work expectedly, so newly implemented `VisualPromptingOpenvinoAdapter` to use new resize function

* Remove skipped test

* Updated

* Update unit tests on model wrappers

* Update

* Update configuration

* Fix not to patch pretrained path

* pylint & update model api version in docstring

---------

Co-authored-by: Wonju Lee <wonju.lee@intel.com>

* Bump albumentations version in anomaly requirements (#2350)

increment albumentations version

* Update action detection (#2346)

* Remove skip mark for PTQ test of action detection

* Update action detection documentation

* Fix e2e (#2348)

* Change classification dataset from dummy to toy

* Revert test changes

* Change label name for multilabel dataset

* Revert e2e test changes

* Change ov test cases' threshold

* Add parent's label

* Update ModelAPI in 1.4 release (#2347)

* Upgrade model API

* Update otx in exportable code

* Fix unit tests

* Fix black

* Fix detection inference

* Fix det tiling

* Fix mypy

* Fix demo

* Fix visualizer in demo

* Fix black

* Add OTX optimize for visual prompting task (#2318)

* Initial commit

* Update block

* (WIP) otx optimize

* Fix

* WIP

* Update configs & exported outputs

* Remove unused modules for torch

* Add unit tests

* pre-commit

* Update CHANGELOG

* Update detection docs (#2335)

* Update detection docs

* Revert template id changes

* Fix wrong template id

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

---------

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Add visual prompting documentation (#2354)

* (WIP) write docs

* Add visual prompting documentation

* Update CHANGELOG

---------

Co-authored-by: sungchul.kim <sungchul@ikvensx010>

* Remove custom modelapi patch in visual prompting (#2359)

* Remove custom modelapi patch

* Update test

* Fix graph metric order and label issues (#2356)

* Fix graph metric going backward issue
* Add license notice
* Fix pre-commit issue
* Add rename items & logic for metric
---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update multi-label document and conversion script (#2358)

Update docs, label convert script

* Update third party programs (#2365)

* Make anomaly task compatible with older albumentations versions (#2363)

* fix transforms export in metadata

* wrap transform dict

* add todo for updating to_dict call

* Fixing detection saliency map for one class case (#2368)

* fix softmax

* fix validity tests

* Add e2e test for visual prompting (#2360)

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* Delete unused configuration.yaml

* Edit test_name

* Add to limit activation range

* Update from `vp` to `visprompt`

* Fix about no returning the first label

* pre-commit

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* pre-commit

* Add actions

* Update tests/e2e/cli/visual_prompting/test_visual_prompting.py

Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>

* Skip PTQ e2e test

* Change task name

* Remove skipped tc

---------

Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>

* Fix e2e (#2366)

* Change e2e reference name

* Update openvino eval threshold for multiclass classification

* Change comment message

* Fix tiling e2e tests

---------

Co-authored-by: GalyaZalesskaya <galina.zalesskaya@intel.com>

* Add Dino head unit tests (#2344)

Recover DINO head unit tests

* Update for release 1.4.0rc2 (#2370)

* update for release 1.4.0rc2

* Add skip mark for unstable unit tests

---------

Co-authored-by: jaegukhyun <jaeguk.hyun@intel.com>

* Fix NNCF training on CPU (#2373)

* Align label order between Geti and OTX (#2369)

* align label order

* align with pre-commit

* update CHANGELOG.md

* deal with edge case

* update type hint

* Remove CenterCrop from Classification test pipeline and editing missing docs link (#2375)

* Fix missing link for docs and removing centercrop for classification data pipeline

* Revert the test threshold

* Fix H-label classification (#2377)

* Fix h-labelissue

* Update unit tests

* Make black happy

* Fix unittests

* Make black happy

* Fix update heades information func

* Update the logic: consider the loss per batch

* Update for release 1.4 (#2380)

* updated for 1.4.0rc3

* update changelog & release note

* bump datumaro version up

---------

Co-authored-by: Songki Choi <songki.choi@intel.com>

* Switch to PTQ for sseg (#2374)

* Switch to PTQ for sseg

* Update log messages

* Fix invalid import structures in otx.api (#2383)

Update tiler.py

* Update for 1.4.0rc4 (#2385)

update for release 1.4.0rc4

* [release 1.4.0] XAI: Return saliency maps for Mask RCNN IR async infer (#2395)

* Return saliency maps for openvino async infer

* add workaround to fix yapf importing error

---------

Co-authored-by: eunwoosh <eunwoo.shin@intel.com>

* Update for release 1.4.0 (#2399)

update version string

Co-authored-by: Sungman Cho <sungman.cho@intel.com>

* Fix broken links in documentation (#2405)

* fix docs links to datumaro's docs
* fix docs links to otx's docs
* bump version to 1.4.1

* Update exportable code README (#2411)

* Updated for release 1.4.1 (#2412)

updated for release 1.4.1

* Add workaround for the incorrect meta info M-RCNN (used for XAI) (#2437)

Add workaround for the incorrect mata info

* Add model category attributes to model template (#2439)

Add model category attributes to model template

* Add model category & status fields in model template

* Add is_default_for_task attr to model template

* Update model templates with category attrs

* Add integration tests for model templates consistency

* Fix license & doc string

* Fix typo

* Refactor test cases

* Refactor common tests by generator

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update for 1.4.2rc1 (#2441)

update for release 1.4.2rc1

* Fix label list order for h-label classification (#2440)

* Fix label list for h-label cls
* Fix unit tests

* Modified fq numbers for lite HRNET (#2445)

modified fq numbers for lite HRNET

* Update PTQ ignored scope for hrnet 18  mod2 (#2449)

Update ptq ignored scope for hrnet 18  mod2

* Fix OpenVINO inference for legacy models (#2450)

* bug fix for legacy openvino models

* Add tests

* Specific exceptions

---------

* Update for 1.4.2rc2 (#2455)

update for release 1.4.2rc2

* Prevent zero-sized saliency map in tiling if tile size is too big (#2452)

* Prevent zero-sized saliency map in tiling if tile size is too big

* Prevent zero-sized saliency in tiling (PyTorch)

* Add unit tests for Tiler merge features methods

---------

Co-authored-by: Galina <galina.zalesskaya@intel.com>

* Update pot fq reference number (#2456)

update pot fq reference number to 15

* Bump datumaro version to 1.5.0rc0 (#2470)

bump datumaro version to 1.5.0rc0

* Set tox version constraint (#2472)

set tox version constraint - tox-dev/tox#3110

* Bug fix for albumentations (#2467)

* bug fix for legacy openvino models

* Address albumentation issue

---------

Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>

* update for release 1.4.2rc3

* Add a dummy hierarchical config required by MAPI (#2483)

* bump version to 1.4.2rc4

* Bump datumaro version (#2502)

* bump datumaro version

* remove deprecated/reomved attribute usage of the datumaro

* Upgrade nncf version for 1.4 release (#2459)

* Upgrade nncf version

* Fix nncf interface warning

* Set the exact nncf version

* Update FQ refs after NNCF upgrade

* Use NNCF from pypi

* Update version for release 1.4.2rc5 (#2507)

update version for release 1.4.2rc5

* Update for 1.4.2 (#2514)

update for release 1.4.2

* create branch release/1.5.0

* Delete mem cache handler after training is done (#2535)

release mem cache handler after training is done

* Fix bug that auto batch size doesn't consider distributed training (#2533)

* consider distributed training while searching batch size

* update unit test

* reveret gpu memory upper bound

* fix typo

* change allocated to reserved

* add unit test for distributed training

* align with pre-commit

* Apply fix progress hook to release 1.5.0 (#2539)

* Fix hook's ordering issue. AdaptiveRepeatHook changes the runner.max_iters before the ProgressHook

* Change the expression

* Fix typo

* Fix multi-label, h-label issue

* Fix auto_bs issue

* Apply suggestions from code review

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Reflecting reviews

* Refactor the name of get_data_cfg

* Revert adaptive hook sampler init

* Refactor the function name: get_data_cfg -> get_subset_data_cfg

* Fix unit test errors

* Remove adding AdaptiveRepeatDataHook for autobs

* Remove unused import

* Fix detection and segmentation case in Geti scenario

---------

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Re introduce adaptive scheduling for training (#2541)

* Re-introduce adaptive patience for training

* Revert unit tests

* Update for release 1.4.3rc1 (#2542)

* Mirror Anomaly ModelAPI changes (#2531)

* Migrate anomaly exportable code to modelAPI (#2432)

* Fix license in PR template

* Migrate to modelAPI

* Remove color conversion in streamer

* Remove reverse_input_channels

* Add float

* Remove test as metadata is no longer used

* Remove metadata from load method

* remove anomalib openvino inferencer

* fix signature

* Support logacy OpenVINO model

* Transform image

* add configs

* Re-introduce adaptive training (#2543)

* Re-introduce adaptive patience for training

* Revert unit tests

* Fix auto input size mismatch in eval & export (#2530)

* Fix auto input size mismatch in eval & export

* Re-enable E2E tests for Issue#2518

* Add input size check in export testing

* Format float numbers in log

* Fix NNCF export shape mismatch

* Fix saliency map issue

* Disable auto input size if tiling enabled

---------

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update ref. fq number for anomaly e2e2 (#2547)

* Skip e2e det tests by issue2548 (#2550)

* Add skip to chained TC for issue #2548 (#2552)

* Update for release 1.4.3 (#2551)

* Update MAPI for 1.5 release (#2555)

Upgrade MAPI to v 0.1.6 (#2529)

* Upgrade MAPI

* Update exp code demo commit

* Fix MAPI imports

* Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Disable QAT for SegNexts (#2565)

* Disable NNCF QAT for SegNext

* Del obsolete pot configs

* Move NNCF skip marks to test commands to avoid duplication

* Add Anomaly modelAPI changes to releases/1.4.0 (#2563)

* bug fix for legacy openvino models

* Apply otx anomaly 1.5 changes

* Fix tests

* Fix compression config

* fix modelAPI imports

* update integration tests

* Edit config types

* Update keys in deployed model

---------

Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>
Co-authored-by: Kim, Sungchul <sungchul.kim@intel.com>

* Fix the CustomNonLinearClsHead when the batch_size is set to 1 (#2571)

Fix bn1d issue

Co-authored-by: sungmanc <sungmanc@intel.com>

* Update ModelAPI configuration (#2564 from 1.4) (#2568)

Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Update for 1.4.4rc1 (#2572)

* Hotfix DatasetEntity.get_combined_subset function loop (#2577)

Fix get_combined_subset function

* Revert default input size to `Default` due to YOLOX perf regression (#2580)

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix for the degradation issue of the classification task (#2585)

* Revert to sync with 1.4.0

* Remove repeat data

* Convert to the RGB value

* Fix color conversion logic

* Fix precommit

* Bump datumaro version to 1.5.1rc3 (#2587)

* Add label ids to anomaly OpenVINO model xml (#2590)

* Add label ids to model xml

---------

* Fix DeiT-Tiny model regression during class incremental training (#2594)

* enable IBloss for DeiT-Tiny

* update changelog

* add docstring

* Add label ids to model xml in release 1.5 (#2591)

Add label ids to model xml

* Fix DeiT-Tiny regression test for release/1.4.0 (#2595)

* Fix DeiT regression test

* update changelog

* temp

* Fix mmcls bug not wrapping model in DataParallel on CPUs (#2601)

Wrap multi-label and h-label classification models by MMDataParallel in case of CPU training.
---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix h-label loss normalization issue w/ exclusive label group of singe label (#2604)

* Fix h-label loss normalization issue w/ exclusive label group with signle label

* Fix non-linear version

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Boost up Image numpy accessing speed through PIL (#2586)

* boost up numpy accessing speed through PIL

* update CHANGELOG

* resolve precommit error

* resolve precommit error

* add fallback logic with PIL open

* use convert instead of draft

* Add missing import pathlib for cls e2e testing (#2610)

* Fix division by zero in class incremental learning for classification (#2606)

* Add empty label to reproduce zero-division error

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix minor typo

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix empty label 4 -> 3

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Prevent division by zero

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update license

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update CHANGELOG.md

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix inefficient sampling

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Revert indexing

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix minor typo

Signed-off-by: Songki Choi <songki.choi@intel.com>

---------

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Unify logger usage (#2612)

* unify logger

* align with pre-commit

* unify anomaly logger to otx

* change logger file path

* align with pre-commit

* change logger file path in missing file

* configure logger after ConfigManager is initialized

* configure logger when ConfigManager instance is initialized

* update unit test code

* move config_logger to each cli file

* align with pre-commit

* change part still using mmcv logger

* Fix XAI algorithm for Detection (#2609)

* Impove saliency maps algorithm for Detection

* Remove extra changes

* Update unit tests

* Changes for 1 class

* Fix pre-commit

* Update CHANGELOG

* Tighten dependency constraint only adapting latest patches (#2607)

* tighten dependency constratint only adapting latest patches

* adjust scikit-image version w.r.t python version

* adjust tensorboard version w.r.t python version

* remove version specifier for scikit-image

* Add metadata to optimized model (#2618)

* bug fix for legacy openvino models

* Add metadata to optimized model

* Revert formatting changes

---------

Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>

* modify omegaconf version constraint

* [release 1.5.0] Fix XAI algorithm for Detection (#2617)

Update detection XAI algorithm

* Update dependency constraint (#2622)

* Update tpp (#2621)

* Fix h-label bug of missing parent labels in output (#2626)

* Fix h-label bug of missing parent labels in output

* Fix h-label test data label schema

* Update CHANGELOG.md

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update publish workflow (#2625)

update publish workflow to push whl to internal pypi

* bump datumaro version to ~=1.5.0

* fixed mistake while mergeing back 1.4.4

* modifiy readme

* remove openvino model wrapper class

* remove openvino model wrapper tests

* [release 1.5.0] DeiT: enable tests + add ViTFeatureVectorHook (#2630)

Add ViT feature vector hook

* Fix docs broken link to datatumaro_h-label

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix wrong label settings for non-anomaly task ModelAPIs

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

Fix e2e XAI ref value

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* update release note and readme

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

* fix datumaro version to 1.6.0rc0

* Mergeback 1.5.0 to develop (#2642)

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

---------

Co-authored-by: Galina Zalesskaya <galina.zalesskaya@intel.com>
Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>

* Revert "Mergeback 1.5.0 to develop" (#2645)

Revert "Mergeback 1.5.0 to develop (#2642)"

This reverts commit 2f67686.

* Add a tool to help conduct experiments (#2651)

* implement run and experiment

* implement experiment result aggregator

* refactor experiment.py

* refactor run.py

* get export model speed

* add var collumn

* refactor experiment.py

* refine a way to update argument in cmd

* refine resource tracker

* support anomaly on research framework

* refine code aggregating exp result

* bugfix

* make other task available

* eval task save avg_time_per_images as result

* Add new argument to track CPU&GPU utilization and memory usage (#2500)

* add argument to track resource usage

* fix bug

* fix a bug in a multi gpu case

* use total cpu usage

* add unit test

* add mark to unit test

* cover edge case

* add pynvml in requirement

* align with pre-commit

* add license comment

* update changelog

* refine argument help

* align with pre-commit

* add version to requirement and raise an error if not supported values are given

* apply new resource tracker format

* refactor run.py

* support optimize in research framework

* cover edge case

* Handle a case where fail cases exist

* make argparse raise error rather than exit if problem exist

* revert tensorboard aggregator

* bugfix

* save failed cases as yaml file

* deal with integer in variables

* add epoch to metric

* use latest log.json file

* align with otx logging method

* move experiment.py from cli to tools

* refactor experiment.py

* merge otx run feature into experiment.py

* move set_arguments_to_cmd definition into experiment.py

* refactor experiment.py

* bugfix

* minor bugfix

* use otx.cli instead of each otx entry

* add feature to parse single workspace

* add comments

* fix bugs

* align with pre-commit

* revert parser argument

* align with pre-commit

* Revert inference batch size to 1 for instance segmentation (#2648)

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Remove unnecessary log while building a model (#2658)

* revert logger in otx/algorithms/detection/adapters/mmdet/utils/builder.py

* revert logger in otx/algorithms/classification/adapters/mmcls/utils/builder.py

* make change more readable

* Fix a minor bug of experiment.py (#2662)

fix bug

* Not check avg_time_per_image during test (#2665)

* ignore avg_time_per_image during test

* do not call stdev when length of array is less than 2

* ignore avg_time_per_image during regerssion test

* Update device selection logic in classificaiton

---------

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Songki Choi <songki.choi@intel.com>
Co-authored-by: Yunchu Lee <yunchu.lee@intel.com>
Co-authored-by: Kim, Sungchul <sungchul.kim@intel.com>
Co-authored-by: Vinnam Kim <vinnam.kim@intel.com>
Co-authored-by: Evgeny Tsykunov <evgeny.tsykunov@intel.com>
Co-authored-by: Songki Choi <songki.choi@intel.com>
Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>
Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>
Co-authored-by: Sungman Cho <sungman.cho@intel.com>
Co-authored-by: Eugene Liu <eugene.liu@intel.com>
Co-authored-by: Wonju Lee <wonju.lee@intel.com>
Co-authored-by: Dick Ameln <dick.ameln@intel.com>
Co-authored-by: sungchul.kim <sungchul@ikvensx010>
Co-authored-by: GalyaZalesskaya <galina.zalesskaya@intel.com>
Co-authored-by: Harim Kang <harim.kang@intel.com>
Co-authored-by: Ashwin Vaidya <ashwin.vaidya@intel.com>
Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>
Co-authored-by: sungmanc <sungmanc@intel.com>
sovrasov added a commit that referenced this pull request Nov 30, 2023
@sovrasov @yunchu
@sungchul2 @vinnamkim @negvet @goodsong81 @eunwoosh @jaegukhyun @sungmanc @eugene123tw @wonjuleee @djdameln @GalyaZalesskaya @harimkang @ashwinvaidya17
Add XPU support to anomaly task (#2677)
c1beb05 
* Update base.txt

updated dependency version of datumaro

* Update __init__.py

update version string

* Update requirements.txt

* Temporarily skip visual prompting openvino integration test (#2323)

* Fix import dm.DatasetSubset (#2324)

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Fix semantic segmentation soft prediction dtype (#2322)

* Fix semantic segmentation soft prediction dtype

* relax ref sal vals check

---------

Co-authored-by: Songki Choi <songki.choi@intel.com>

* Contrain yapf verison lesser than 0.40.0 (#2328)

contrain_yapf_version

* Fix detection e2e tests (#2327)

Fix for detection

* Mergeback: Label addtion/deletion 1.2.4 --> 1.4.0 (#2326)

* Make black happy

* Fix conflicts

* Merge-back: add test datasets and edit the test code

* Make black happy

* Fix mis-merge

* Make balck happy

* Fix typo

* Fix typoi

---------

Co-authored-by: Songki Choi <songki.choi@intel.com>

* Bump datumaro up to 1.4.0rc2 (#2332)

bump datumaro up to 1.4.0rc2

* Tiling Doc for releases 1.4.0 (#2333)

* Add tiling documentation

* Bump otx version to 1.4.0rc2 (#2341)

* OTX deploy for visual prompting task  (#2311)

* Enable `otx deploy`

* (WIP) integration test

* Docstring

* Update args for create_model

* Manually set image embedding layout

* Enable to use model api for preprocessing
- `fit_to_window` doesn't work expectedly, so newly implemented `VisualPromptingOpenvinoAdapter` to use new resize function

* Remove skipped test

* Updated

* Update unit tests on model wrappers

* Update

* Update configuration

* Fix not to patch pretrained path

* pylint & update model api version in docstring

---------

Co-authored-by: Wonju Lee <wonju.lee@intel.com>

* Bump albumentations version in anomaly requirements (#2350)

increment albumentations version

* Update action detection (#2346)

* Remove skip mark for PTQ test of action detection

* Update action detection documentation

* Fix e2e (#2348)

* Change classification dataset from dummy to toy

* Revert test changes

* Change label name for multilabel dataset

* Revert e2e test changes

* Change ov test cases' threshold

* Add parent's label

* Update ModelAPI in 1.4 release (#2347)

* Upgrade model API

* Update otx in exportable code

* Fix unit tests

* Fix black

* Fix detection inference

* Fix det tiling

* Fix mypy

* Fix demo

* Fix visualizer in demo

* Fix black

* Add OTX optimize for visual prompting task (#2318)

* Initial commit

* Update block

* (WIP) otx optimize

* Fix

* WIP

* Update configs & exported outputs

* Remove unused modules for torch

* Add unit tests

* pre-commit

* Update CHANGELOG

* Update detection docs (#2335)

* Update detection docs

* Revert template id changes

* Fix wrong template id

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

---------

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Add visual prompting documentation (#2354)

* (WIP) write docs

* Add visual prompting documentation

* Update CHANGELOG

---------

Co-authored-by: sungchul.kim <sungchul@ikvensx010>

* Remove custom modelapi patch in visual prompting (#2359)

* Remove custom modelapi patch

* Update test

* Fix graph metric order and label issues (#2356)

* Fix graph metric going backward issue
* Add license notice
* Fix pre-commit issue
* Add rename items & logic for metric
---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update multi-label document and conversion script (#2358)

Update docs, label convert script

* Update third party programs (#2365)

* Make anomaly task compatible with older albumentations versions (#2363)

* fix transforms export in metadata

* wrap transform dict

* add todo for updating to_dict call

* Fixing detection saliency map for one class case (#2368)

* fix softmax

* fix validity tests

* Add e2e test for visual prompting (#2360)

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* Delete unused configuration.yaml

* Edit test_name

* Add to limit activation range

* Update from `vp` to `visprompt`

* Fix about no returning the first label

* pre-commit

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* pre-commit

* Add actions

* Update tests/e2e/cli/visual_prompting/test_visual_prompting.py

Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>

* Skip PTQ e2e test

* Change task name

* Remove skipped tc

---------

Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>

* Fix e2e (#2366)

* Change e2e reference name

* Update openvino eval threshold for multiclass classification

* Change comment message

* Fix tiling e2e tests

---------

Co-authored-by: GalyaZalesskaya <galina.zalesskaya@intel.com>

* Add Dino head unit tests (#2344)

Recover DINO head unit tests

* Update for release 1.4.0rc2 (#2370)

* update for release 1.4.0rc2

* Add skip mark for unstable unit tests

---------

Co-authored-by: jaegukhyun <jaeguk.hyun@intel.com>

* Fix NNCF training on CPU (#2373)

* Align label order between Geti and OTX (#2369)

* align label order

* align with pre-commit

* update CHANGELOG.md

* deal with edge case

* update type hint

* Remove CenterCrop from Classification test pipeline and editing missing docs link (#2375)

* Fix missing link for docs and removing centercrop for classification data pipeline

* Revert the test threshold

* Fix H-label classification (#2377)

* Fix h-labelissue

* Update unit tests

* Make black happy

* Fix unittests

* Make black happy

* Fix update heades information func

* Update the logic: consider the loss per batch

* Update for release 1.4 (#2380)

* updated for 1.4.0rc3

* update changelog & release note

* bump datumaro version up

---------

Co-authored-by: Songki Choi <songki.choi@intel.com>

* Switch to PTQ for sseg (#2374)

* Switch to PTQ for sseg

* Update log messages

* Fix invalid import structures in otx.api (#2383)

Update tiler.py

* Update for 1.4.0rc4 (#2385)

update for release 1.4.0rc4

* [release 1.4.0] XAI: Return saliency maps for Mask RCNN IR async infer (#2395)

* Return saliency maps for openvino async infer

* add workaround to fix yapf importing error

---------

Co-authored-by: eunwoosh <eunwoo.shin@intel.com>

* Update for release 1.4.0 (#2399)

update version string

Co-authored-by: Sungman Cho <sungman.cho@intel.com>

* Fix broken links in documentation (#2405)

* fix docs links to datumaro's docs
* fix docs links to otx's docs
* bump version to 1.4.1

* Update exportable code README (#2411)

* Updated for release 1.4.1 (#2412)

updated for release 1.4.1

* Add workaround for the incorrect meta info M-RCNN (used for XAI) (#2437)

Add workaround for the incorrect mata info

* Add model category attributes to model template (#2439)

Add model category attributes to model template

* Add model category & status fields in model template

* Add is_default_for_task attr to model template

* Update model templates with category attrs

* Add integration tests for model templates consistency

* Fix license & doc string

* Fix typo

* Refactor test cases

* Refactor common tests by generator

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update for 1.4.2rc1 (#2441)

update for release 1.4.2rc1

* Fix label list order for h-label classification (#2440)

* Fix label list for h-label cls
* Fix unit tests

* Modified fq numbers for lite HRNET (#2445)

modified fq numbers for lite HRNET

* Update PTQ ignored scope for hrnet 18  mod2 (#2449)

Update ptq ignored scope for hrnet 18  mod2

* Fix OpenVINO inference for legacy models (#2450)

* bug fix for legacy openvino models

* Add tests

* Specific exceptions

---------

* Update for 1.4.2rc2 (#2455)

update for release 1.4.2rc2

* Prevent zero-sized saliency map in tiling if tile size is too big (#2452)

* Prevent zero-sized saliency map in tiling if tile size is too big

* Prevent zero-sized saliency in tiling (PyTorch)

* Add unit tests for Tiler merge features methods

---------

Co-authored-by: Galina <galina.zalesskaya@intel.com>

* Update pot fq reference number (#2456)

update pot fq reference number to 15

* Bump datumaro version to 1.5.0rc0 (#2470)

bump datumaro version to 1.5.0rc0

* Set tox version constraint (#2472)

set tox version constraint - tox-dev/tox#3110

* Bug fix for albumentations (#2467)

* bug fix for legacy openvino models

* Address albumentation issue

---------

Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>

* update for release 1.4.2rc3

* Add a dummy hierarchical config required by MAPI (#2483)

* bump version to 1.4.2rc4

* Bump datumaro version (#2502)

* bump datumaro version

* remove deprecated/reomved attribute usage of the datumaro

* Upgrade nncf version for 1.4 release (#2459)

* Upgrade nncf version

* Fix nncf interface warning

* Set the exact nncf version

* Update FQ refs after NNCF upgrade

* Use NNCF from pypi

* Update version for release 1.4.2rc5 (#2507)

update version for release 1.4.2rc5

* Update for 1.4.2 (#2514)

update for release 1.4.2

* create branch release/1.5.0

* Delete mem cache handler after training is done (#2535)

release mem cache handler after training is done

* Fix bug that auto batch size doesn't consider distributed training (#2533)

* consider distributed training while searching batch size

* update unit test

* reveret gpu memory upper bound

* fix typo

* change allocated to reserved

* add unit test for distributed training

* align with pre-commit

* Apply fix progress hook to release 1.5.0 (#2539)

* Fix hook's ordering issue. AdaptiveRepeatHook changes the runner.max_iters before the ProgressHook

* Change the expression

* Fix typo

* Fix multi-label, h-label issue

* Fix auto_bs issue

* Apply suggestions from code review

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Reflecting reviews

* Refactor the name of get_data_cfg

* Revert adaptive hook sampler init

* Refactor the function name: get_data_cfg -> get_subset_data_cfg

* Fix unit test errors

* Remove adding AdaptiveRepeatDataHook for autobs

* Remove unused import

* Fix detection and segmentation case in Geti scenario

---------

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Re introduce adaptive scheduling for training (#2541)

* Re-introduce adaptive patience for training

* Revert unit tests

* Update for release 1.4.3rc1 (#2542)

* Mirror Anomaly ModelAPI changes (#2531)

* Migrate anomaly exportable code to modelAPI (#2432)

* Fix license in PR template

* Migrate to modelAPI

* Remove color conversion in streamer

* Remove reverse_input_channels

* Add float

* Remove test as metadata is no longer used

* Remove metadata from load method

* remove anomalib openvino inferencer

* fix signature

* Support logacy OpenVINO model

* Transform image

* add configs

* Re-introduce adaptive training (#2543)

* Re-introduce adaptive patience for training

* Revert unit tests

* Fix auto input size mismatch in eval & export (#2530)

* Fix auto input size mismatch in eval & export

* Re-enable E2E tests for Issue#2518

* Add input size check in export testing

* Format float numbers in log

* Fix NNCF export shape mismatch

* Fix saliency map issue

* Disable auto input size if tiling enabled

---------

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update ref. fq number for anomaly e2e2 (#2547)

* Skip e2e det tests by issue2548 (#2550)

* Add skip to chained TC for issue #2548 (#2552)

* Update for release 1.4.3 (#2551)

* Update MAPI for 1.5 release (#2555)

Upgrade MAPI to v 0.1.6 (#2529)

* Upgrade MAPI

* Update exp code demo commit

* Fix MAPI imports

* Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Disable QAT for SegNexts (#2565)

* Disable NNCF QAT for SegNext

* Del obsolete pot configs

* Move NNCF skip marks to test commands to avoid duplication

* Add Anomaly modelAPI changes to releases/1.4.0 (#2563)

* bug fix for legacy openvino models

* Apply otx anomaly 1.5 changes

* Fix tests

* Fix compression config

* fix modelAPI imports

* update integration tests

* Edit config types

* Update keys in deployed model

---------

Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>
Co-authored-by: Kim, Sungchul <sungchul.kim@intel.com>

* Fix the CustomNonLinearClsHead when the batch_size is set to 1 (#2571)

Fix bn1d issue

Co-authored-by: sungmanc <sungmanc@intel.com>

* Update ModelAPI configuration (#2564 from 1.4) (#2568)

Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Update for 1.4.4rc1 (#2572)

* Hotfix DatasetEntity.get_combined_subset function loop (#2577)

Fix get_combined_subset function

* Revert default input size to `Default` due to YOLOX perf regression (#2580)

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix for the degradation issue of the classification task (#2585)

* Revert to sync with 1.4.0

* Remove repeat data

* Convert to the RGB value

* Fix color conversion logic

* Fix precommit

* Bump datumaro version to 1.5.1rc3 (#2587)

* Add label ids to anomaly OpenVINO model xml (#2590)

* Add label ids to model xml

---------

* Fix DeiT-Tiny model regression during class incremental training (#2594)

* enable IBloss for DeiT-Tiny

* update changelog

* add docstring

* Add label ids to model xml in release 1.5 (#2591)

Add label ids to model xml

* Fix DeiT-Tiny regression test for release/1.4.0 (#2595)

* Fix DeiT regression test

* update changelog

* temp

* Fix mmcls bug not wrapping model in DataParallel on CPUs (#2601)

Wrap multi-label and h-label classification models by MMDataParallel in case of CPU training.
---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix h-label loss normalization issue w/ exclusive label group of singe label (#2604)

* Fix h-label loss normalization issue w/ exclusive label group with signle label

* Fix non-linear version

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Boost up Image numpy accessing speed through PIL (#2586)

* boost up numpy accessing speed through PIL

* update CHANGELOG

* resolve precommit error

* resolve precommit error

* add fallback logic with PIL open

* use convert instead of draft

* Add missing import pathlib for cls e2e testing (#2610)

* Fix division by zero in class incremental learning for classification (#2606)

* Add empty label to reproduce zero-division error

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix minor typo

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix empty label 4 -> 3

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Prevent division by zero

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update license

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update CHANGELOG.md

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix inefficient sampling

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Revert indexing

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix minor typo

Signed-off-by: Songki Choi <songki.choi@intel.com>

---------

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Unify logger usage (#2612)

* unify logger

* align with pre-commit

* unify anomaly logger to otx

* change logger file path

* align with pre-commit

* change logger file path in missing file

* configure logger after ConfigManager is initialized

* configure logger when ConfigManager instance is initialized

* update unit test code

* move config_logger to each cli file

* align with pre-commit

* change part still using mmcv logger

* Fix XAI algorithm for Detection (#2609)

* Impove saliency maps algorithm for Detection

* Remove extra changes

* Update unit tests

* Changes for 1 class

* Fix pre-commit

* Update CHANGELOG

* Tighten dependency constraint only adapting latest patches (#2607)

* tighten dependency constratint only adapting latest patches

* adjust scikit-image version w.r.t python version

* adjust tensorboard version w.r.t python version

* remove version specifier for scikit-image

* Add metadata to optimized model (#2618)

* bug fix for legacy openvino models

* Add metadata to optimized model

* Revert formatting changes

---------

Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>

* modify omegaconf version constraint

* [release 1.5.0] Fix XAI algorithm for Detection (#2617)

Update detection XAI algorithm

* Update dependency constraint (#2622)

* Update tpp (#2621)

* Fix h-label bug of missing parent labels in output (#2626)

* Fix h-label bug of missing parent labels in output

* Fix h-label test data label schema

* Update CHANGELOG.md

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update publish workflow (#2625)

update publish workflow to push whl to internal pypi

* bump datumaro version to ~=1.5.0

* fixed mistake while mergeing back 1.4.4

* modifiy readme

* remove openvino model wrapper class

* remove openvino model wrapper tests

* [release 1.5.0] DeiT: enable tests + add ViTFeatureVectorHook (#2630)

Add ViT feature vector hook

* Fix docs broken link to datatumaro_h-label

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix wrong label settings for non-anomaly task ModelAPIs

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

Fix e2e XAI ref value

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* update release note and readme

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

* fix datumaro version to 1.6.0rc0

* Mergeback 1.5.0 to develop (#2642)

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

---------

Co-authored-by: Galina Zalesskaya <galina.zalesskaya@intel.com>
Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>

* Revert "Mergeback 1.5.0 to develop" (#2645)

Revert "Mergeback 1.5.0 to develop (#2642)"

This reverts commit 2f67686.

* Add a tool to help conduct experiments (#2651)

* implement run and experiment

* implement experiment result aggregator

* refactor experiment.py

* refactor run.py

* get export model speed

* add var collumn

* refactor experiment.py

* refine a way to update argument in cmd

* refine resource tracker

* support anomaly on research framework

* refine code aggregating exp result

* bugfix

* make other task available

* eval task save avg_time_per_images as result

* Add new argument to track CPU&GPU utilization and memory usage (#2500)

* add argument to track resource usage

* fix bug

* fix a bug in a multi gpu case

* use total cpu usage

* add unit test

* add mark to unit test

* cover edge case

* add pynvml in requirement

* align with pre-commit

* add license comment

* update changelog

* refine argument help

* align with pre-commit

* add version to requirement and raise an error if not supported values are given

* apply new resource tracker format

* refactor run.py

* support optimize in research framework

* cover edge case

* Handle a case where fail cases exist

* make argparse raise error rather than exit if problem exist

* revert tensorboard aggregator

* bugfix

* save failed cases as yaml file

* deal with integer in variables

* add epoch to metric

* use latest log.json file

* align with otx logging method

* move experiment.py from cli to tools

* refactor experiment.py

* merge otx run feature into experiment.py

* move set_arguments_to_cmd definition into experiment.py

* refactor experiment.py

* bugfix

* minor bugfix

* use otx.cli instead of each otx entry

* add feature to parse single workspace

* add comments

* fix bugs

* align with pre-commit

* revert parser argument

* align with pre-commit

* Revert inference batch size to 1 for instance segmentation (#2648)

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Remove unnecessary log while building a model (#2658)

* revert logger in otx/algorithms/detection/adapters/mmdet/utils/builder.py

* revert logger in otx/algorithms/classification/adapters/mmcls/utils/builder.py

* make change more readable

* Fix a minor bug of experiment.py (#2662)

fix bug

* Not check avg_time_per_image during test (#2665)

* ignore avg_time_per_image during test

* do not call stdev when length of array is less than 2

* ignore avg_time_per_image during regerssion test

* Update device selection logic in classificaiton

* Add xpu accelerator

* Tmp patch for anomaly trainer

* Use XPU callback for anomaly training

* Update xpu accelerator

* Fix for anomaly xpu callback

* Fix validation batch logic

* Cleanup, add docstrings

* Refine xpu callback

---------

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Songki Choi <songki.choi@intel.com>
Co-authored-by: Yunchu Lee <yunchu.lee@intel.com>
Co-authored-by: Kim, Sungchul <sungchul.kim@intel.com>
Co-authored-by: Vinnam Kim <vinnam.kim@intel.com>
Co-authored-by: Evgeny Tsykunov <evgeny.tsykunov@intel.com>
Co-authored-by: Songki Choi <songki.choi@intel.com>
Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>
Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>
Co-authored-by: Sungman Cho <sungman.cho@intel.com>
Co-authored-by: Eugene Liu <eugene.liu@intel.com>
Co-authored-by: Wonju Lee <wonju.lee@intel.com>
Co-authored-by: Dick Ameln <dick.ameln@intel.com>
Co-authored-by: sungchul.kim <sungchul@ikvensx010>
Co-authored-by: GalyaZalesskaya <galina.zalesskaya@intel.com>
Co-authored-by: Harim Kang <harim.kang@intel.com>
Co-authored-by: Ashwin Vaidya <ashwin.vaidya@intel.com>
Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>
Co-authored-by: sungmanc <sungmanc@intel.com>
sovrasov added a commit that referenced this pull request Dec 14, 2023
@eunwoosh @yunchu
@sungchul2 @vinnamkim @negvet @goodsong81 @jaegukhyun @sungmanc @eugene123tw @wonjuleee @djdameln @sovrasov @GalyaZalesskaya @harimkang @ashwinvaidya17
Merge develop to develop-idev (#2727)
927fab6 
* Update base.txt

updated dependency version of datumaro

* Update __init__.py

update version string

* Update requirements.txt

* Temporarily skip visual prompting openvino integration test (#2323)

* Fix import dm.DatasetSubset (#2324)

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Fix semantic segmentation soft prediction dtype (#2322)

* Fix semantic segmentation soft prediction dtype

* relax ref sal vals check

---------

Co-authored-by: Songki Choi <songki.choi@intel.com>

* Contrain yapf verison lesser than 0.40.0 (#2328)

contrain_yapf_version

* Fix detection e2e tests (#2327)

Fix for detection

* Mergeback: Label addtion/deletion 1.2.4 --> 1.4.0 (#2326)

* Make black happy

* Fix conflicts

* Merge-back: add test datasets and edit the test code

* Make black happy

* Fix mis-merge

* Make balck happy

* Fix typo

* Fix typoi

---------

Co-authored-by: Songki Choi <songki.choi@intel.com>

* Bump datumaro up to 1.4.0rc2 (#2332)

bump datumaro up to 1.4.0rc2

* Tiling Doc for releases 1.4.0 (#2333)

* Add tiling documentation

* Bump otx version to 1.4.0rc2 (#2341)

* OTX deploy for visual prompting task  (#2311)

* Enable `otx deploy`

* (WIP) integration test

* Docstring

* Update args for create_model

* Manually set image embedding layout

* Enable to use model api for preprocessing
- `fit_to_window` doesn't work expectedly, so newly implemented `VisualPromptingOpenvinoAdapter` to use new resize function

* Remove skipped test

* Updated

* Update unit tests on model wrappers

* Update

* Update configuration

* Fix not to patch pretrained path

* pylint & update model api version in docstring

---------

Co-authored-by: Wonju Lee <wonju.lee@intel.com>

* Bump albumentations version in anomaly requirements (#2350)

increment albumentations version

* Update action detection (#2346)

* Remove skip mark for PTQ test of action detection

* Update action detection documentation

* Fix e2e (#2348)

* Change classification dataset from dummy to toy

* Revert test changes

* Change label name for multilabel dataset

* Revert e2e test changes

* Change ov test cases' threshold

* Add parent's label

* Update ModelAPI in 1.4 release (#2347)

* Upgrade model API

* Update otx in exportable code

* Fix unit tests

* Fix black

* Fix detection inference

* Fix det tiling

* Fix mypy

* Fix demo

* Fix visualizer in demo

* Fix black

* Add OTX optimize for visual prompting task (#2318)

* Initial commit

* Update block

* (WIP) otx optimize

* Fix

* WIP

* Update configs & exported outputs

* Remove unused modules for torch

* Add unit tests

* pre-commit

* Update CHANGELOG

* Update detection docs (#2335)

* Update detection docs

* Revert template id changes

* Fix wrong template id

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

---------

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Add visual prompting documentation (#2354)

* (WIP) write docs

* Add visual prompting documentation

* Update CHANGELOG

---------

Co-authored-by: sungchul.kim <sungchul@ikvensx010>

* Remove custom modelapi patch in visual prompting (#2359)

* Remove custom modelapi patch

* Update test

* Fix graph metric order and label issues (#2356)

* Fix graph metric going backward issue
* Add license notice
* Fix pre-commit issue
* Add rename items & logic for metric
---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update multi-label document and conversion script (#2358)

Update docs, label convert script

* Update third party programs (#2365)

* Make anomaly task compatible with older albumentations versions (#2363)

* fix transforms export in metadata

* wrap transform dict

* add todo for updating to_dict call

* Fixing detection saliency map for one class case (#2368)

* fix softmax

* fix validity tests

* Add e2e test for visual prompting (#2360)

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* Delete unused configuration.yaml

* Edit test_name

* Add to limit activation range

* Update from `vp` to `visprompt`

* Fix about no returning the first label

* pre-commit

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* pre-commit

* Add actions

* Update tests/e2e/cli/visual_prompting/test_visual_prompting.py

Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>

* Skip PTQ e2e test

* Change task name

* Remove skipped tc

---------

Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>

* Fix e2e (#2366)

* Change e2e reference name

* Update openvino eval threshold for multiclass classification

* Change comment message

* Fix tiling e2e tests

---------

Co-authored-by: GalyaZalesskaya <galina.zalesskaya@intel.com>

* Add Dino head unit tests (#2344)

Recover DINO head unit tests

* Update for release 1.4.0rc2 (#2370)

* update for release 1.4.0rc2

* Add skip mark for unstable unit tests

---------

Co-authored-by: jaegukhyun <jaeguk.hyun@intel.com>

* Fix NNCF training on CPU (#2373)

* Align label order between Geti and OTX (#2369)

* align label order

* align with pre-commit

* update CHANGELOG.md

* deal with edge case

* update type hint

* Remove CenterCrop from Classification test pipeline and editing missing docs link (#2375)

* Fix missing link for docs and removing centercrop for classification data pipeline

* Revert the test threshold

* Fix H-label classification (#2377)

* Fix h-labelissue

* Update unit tests

* Make black happy

* Fix unittests

* Make black happy

* Fix update heades information func

* Update the logic: consider the loss per batch

* Update for release 1.4 (#2380)

* updated for 1.4.0rc3

* update changelog & release note

* bump datumaro version up

---------

Co-authored-by: Songki Choi <songki.choi@intel.com>

* Switch to PTQ for sseg (#2374)

* Switch to PTQ for sseg

* Update log messages

* Fix invalid import structures in otx.api (#2383)

Update tiler.py

* Update for 1.4.0rc4 (#2385)

update for release 1.4.0rc4

* [release 1.4.0] XAI: Return saliency maps for Mask RCNN IR async infer (#2395)

* Return saliency maps for openvino async infer

* add workaround to fix yapf importing error

---------

Co-authored-by: eunwoosh <eunwoo.shin@intel.com>

* Update for release 1.4.0 (#2399)

update version string

Co-authored-by: Sungman Cho <sungman.cho@intel.com>

* Fix broken links in documentation (#2405)

* fix docs links to datumaro's docs
* fix docs links to otx's docs
* bump version to 1.4.1

* Update exportable code README (#2411)

* Updated for release 1.4.1 (#2412)

updated for release 1.4.1

* Add workaround for the incorrect meta info M-RCNN (used for XAI) (#2437)

Add workaround for the incorrect mata info

* Add model category attributes to model template (#2439)

Add model category attributes to model template

* Add model category & status fields in model template

* Add is_default_for_task attr to model template

* Update model templates with category attrs

* Add integration tests for model templates consistency

* Fix license & doc string

* Fix typo

* Refactor test cases

* Refactor common tests by generator

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update for 1.4.2rc1 (#2441)

update for release 1.4.2rc1

* Fix label list order for h-label classification (#2440)

* Fix label list for h-label cls
* Fix unit tests

* Modified fq numbers for lite HRNET (#2445)

modified fq numbers for lite HRNET

* Update PTQ ignored scope for hrnet 18  mod2 (#2449)

Update ptq ignored scope for hrnet 18  mod2

* Fix OpenVINO inference for legacy models (#2450)

* bug fix for legacy openvino models

* Add tests

* Specific exceptions

---------

* Update for 1.4.2rc2 (#2455)

update for release 1.4.2rc2

* Prevent zero-sized saliency map in tiling if tile size is too big (#2452)

* Prevent zero-sized saliency map in tiling if tile size is too big

* Prevent zero-sized saliency in tiling (PyTorch)

* Add unit tests for Tiler merge features methods

---------

Co-authored-by: Galina <galina.zalesskaya@intel.com>

* Update pot fq reference number (#2456)

update pot fq reference number to 15

* Bump datumaro version to 1.5.0rc0 (#2470)

bump datumaro version to 1.5.0rc0

* Set tox version constraint (#2472)

set tox version constraint - tox-dev/tox#3110

* Bug fix for albumentations (#2467)

* bug fix for legacy openvino models

* Address albumentation issue

---------

Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>

* update for release 1.4.2rc3

* Add a dummy hierarchical config required by MAPI (#2483)

* bump version to 1.4.2rc4

* Bump datumaro version (#2502)

* bump datumaro version

* remove deprecated/reomved attribute usage of the datumaro

* Upgrade nncf version for 1.4 release (#2459)

* Upgrade nncf version

* Fix nncf interface warning

* Set the exact nncf version

* Update FQ refs after NNCF upgrade

* Use NNCF from pypi

* Update version for release 1.4.2rc5 (#2507)

update version for release 1.4.2rc5

* Update for 1.4.2 (#2514)

update for release 1.4.2

* create branch release/1.5.0

* Delete mem cache handler after training is done (#2535)

release mem cache handler after training is done

* Fix bug that auto batch size doesn't consider distributed training (#2533)

* consider distributed training while searching batch size

* update unit test

* reveret gpu memory upper bound

* fix typo

* change allocated to reserved

* add unit test for distributed training

* align with pre-commit

* Apply fix progress hook to release 1.5.0 (#2539)

* Fix hook's ordering issue. AdaptiveRepeatHook changes the runner.max_iters before the ProgressHook

* Change the expression

* Fix typo

* Fix multi-label, h-label issue

* Fix auto_bs issue

* Apply suggestions from code review

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Reflecting reviews

* Refactor the name of get_data_cfg

* Revert adaptive hook sampler init

* Refactor the function name: get_data_cfg -> get_subset_data_cfg

* Fix unit test errors

* Remove adding AdaptiveRepeatDataHook for autobs

* Remove unused import

* Fix detection and segmentation case in Geti scenario

---------

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Re introduce adaptive scheduling for training (#2541)

* Re-introduce adaptive patience for training

* Revert unit tests

* Update for release 1.4.3rc1 (#2542)

* Mirror Anomaly ModelAPI changes (#2531)

* Migrate anomaly exportable code to modelAPI (#2432)

* Fix license in PR template

* Migrate to modelAPI

* Remove color conversion in streamer

* Remove reverse_input_channels

* Add float

* Remove test as metadata is no longer used

* Remove metadata from load method

* remove anomalib openvino inferencer

* fix signature

* Support logacy OpenVINO model

* Transform image

* add configs

* Re-introduce adaptive training (#2543)

* Re-introduce adaptive patience for training

* Revert unit tests

* Fix auto input size mismatch in eval & export (#2530)

* Fix auto input size mismatch in eval & export

* Re-enable E2E tests for Issue#2518

* Add input size check in export testing

* Format float numbers in log

* Fix NNCF export shape mismatch

* Fix saliency map issue

* Disable auto input size if tiling enabled

---------

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update ref. fq number for anomaly e2e2 (#2547)

* Skip e2e det tests by issue2548 (#2550)

* Add skip to chained TC for issue #2548 (#2552)

* Update for release 1.4.3 (#2551)

* Update MAPI for 1.5 release (#2555)

Upgrade MAPI to v 0.1.6 (#2529)

* Upgrade MAPI

* Update exp code demo commit

* Fix MAPI imports

* Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Disable QAT for SegNexts (#2565)

* Disable NNCF QAT for SegNext

* Del obsolete pot configs

* Move NNCF skip marks to test commands to avoid duplication

* Add Anomaly modelAPI changes to releases/1.4.0 (#2563)

* bug fix for legacy openvino models

* Apply otx anomaly 1.5 changes

* Fix tests

* Fix compression config

* fix modelAPI imports

* update integration tests

* Edit config types

* Update keys in deployed model

---------

Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>
Co-authored-by: Kim, Sungchul <sungchul.kim@intel.com>

* Fix the CustomNonLinearClsHead when the batch_size is set to 1 (#2571)

Fix bn1d issue

Co-authored-by: sungmanc <sungmanc@intel.com>

* Update ModelAPI configuration (#2564 from 1.4) (#2568)

Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Update for 1.4.4rc1 (#2572)

* Hotfix DatasetEntity.get_combined_subset function loop (#2577)

Fix get_combined_subset function

* Revert default input size to `Default` due to YOLOX perf regression (#2580)

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix for the degradation issue of the classification task (#2585)

* Revert to sync with 1.4.0

* Remove repeat data

* Convert to the RGB value

* Fix color conversion logic

* Fix precommit

* Bump datumaro version to 1.5.1rc3 (#2587)

* Add label ids to anomaly OpenVINO model xml (#2590)

* Add label ids to model xml

---------

* Fix DeiT-Tiny model regression during class incremental training (#2594)

* enable IBloss for DeiT-Tiny

* update changelog

* add docstring

* Add label ids to model xml in release 1.5 (#2591)

Add label ids to model xml

* Fix DeiT-Tiny regression test for release/1.4.0 (#2595)

* Fix DeiT regression test

* update changelog

* temp

* Fix mmcls bug not wrapping model in DataParallel on CPUs (#2601)

Wrap multi-label and h-label classification models by MMDataParallel in case of CPU training.
---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix h-label loss normalization issue w/ exclusive label group of singe label (#2604)

* Fix h-label loss normalization issue w/ exclusive label group with signle label

* Fix non-linear version

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Boost up Image numpy accessing speed through PIL (#2586)

* boost up numpy accessing speed through PIL

* update CHANGELOG

* resolve precommit error

* resolve precommit error

* add fallback logic with PIL open

* use convert instead of draft

* Add missing import pathlib for cls e2e testing (#2610)

* Fix division by zero in class incremental learning for classification (#2606)

* Add empty label to reproduce zero-division error

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix minor typo

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix empty label 4 -> 3

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Prevent division by zero

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update license

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update CHANGELOG.md

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix inefficient sampling

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Revert indexing

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix minor typo

Signed-off-by: Songki Choi <songki.choi@intel.com>

---------

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Unify logger usage (#2612)

* unify logger

* align with pre-commit

* unify anomaly logger to otx

* change logger file path

* align with pre-commit

* change logger file path in missing file

* configure logger after ConfigManager is initialized

* configure logger when ConfigManager instance is initialized

* update unit test code

* move config_logger to each cli file

* align with pre-commit

* change part still using mmcv logger

* Fix XAI algorithm for Detection (#2609)

* Impove saliency maps algorithm for Detection

* Remove extra changes

* Update unit tests

* Changes for 1 class

* Fix pre-commit

* Update CHANGELOG

* Tighten dependency constraint only adapting latest patches (#2607)

* tighten dependency constratint only adapting latest patches

* adjust scikit-image version w.r.t python version

* adjust tensorboard version w.r.t python version

* remove version specifier for scikit-image

* Add metadata to optimized model (#2618)

* bug fix for legacy openvino models

* Add metadata to optimized model

* Revert formatting changes

---------

Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>

* modify omegaconf version constraint

* [release 1.5.0] Fix XAI algorithm for Detection (#2617)

Update detection XAI algorithm

* Update dependency constraint (#2622)

* Update tpp (#2621)

* Fix h-label bug of missing parent labels in output (#2626)

* Fix h-label bug of missing parent labels in output

* Fix h-label test data label schema

* Update CHANGELOG.md

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update publish workflow (#2625)

update publish workflow to push whl to internal pypi

* bump datumaro version to ~=1.5.0

* fixed mistake while mergeing back 1.4.4

* modifiy readme

* remove openvino model wrapper class

* remove openvino model wrapper tests

* [release 1.5.0] DeiT: enable tests + add ViTFeatureVectorHook (#2630)

Add ViT feature vector hook

* Fix docs broken link to datatumaro_h-label

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix wrong label settings for non-anomaly task ModelAPIs

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

Fix e2e XAI ref value

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* update release note and readme

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

* fix datumaro version to 1.6.0rc0

* Mergeback 1.5.0 to develop (#2642)

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

---------

Co-authored-by: Galina Zalesskaya <galina.zalesskaya@intel.com>
Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>

* Revert "Mergeback 1.5.0 to develop" (#2645)

Revert "Mergeback 1.5.0 to develop (#2642)"

This reverts commit 2f67686.

* Add a tool to help conduct experiments (#2651)

* implement run and experiment

* implement experiment result aggregator

* refactor experiment.py

* refactor run.py

* get export model speed

* add var collumn

* refactor experiment.py

* refine a way to update argument in cmd

* refine resource tracker

* support anomaly on research framework

* refine code aggregating exp result

* bugfix

* make other task available

* eval task save avg_time_per_images as result

* Add new argument to track CPU&GPU utilization and memory usage (#2500)

* add argument to track resource usage

* fix bug

* fix a bug in a multi gpu case

* use total cpu usage

* add unit test

* add mark to unit test

* cover edge case

* add pynvml in requirement

* align with pre-commit

* add license comment

* update changelog

* refine argument help

* align with pre-commit

* add version to requirement and raise an error if not supported values are given

* apply new resource tracker format

* refactor run.py

* support optimize in research framework

* cover edge case

* Handle a case where fail cases exist

* make argparse raise error rather than exit if problem exist

* revert tensorboard aggregator

* bugfix

* save failed cases as yaml file

* deal with integer in variables

* add epoch to metric

* use latest log.json file

* align with otx logging method

* move experiment.py from cli to tools

* refactor experiment.py

* merge otx run feature into experiment.py

* move set_arguments_to_cmd definition into experiment.py

* refactor experiment.py

* bugfix

* minor bugfix

* use otx.cli instead of each otx entry

* add feature to parse single workspace

* add comments

* fix bugs

* align with pre-commit

* revert parser argument

* align with pre-commit

* Make `max_num_detections` configurable (#2647)

* Make max_num_detections configurable

* Fix RCNN case with integration test

* Apply max_num_detections to train_cfg, too

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Revert inference batch size to 1 for instance segmentation (#2648)

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix CPU training issue on non-CUDA system (#2655)

Fix bug that auto adaptive batch size raises an error if CUDA isn't available (#2410)

---------
Co-authored-by: Sungman Cho <sungman.cho@intel.com>
Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Remove unnecessary log while building a model (#2658)

* revert logger in otx/algorithms/detection/adapters/mmdet/utils/builder.py

* revert logger in otx/algorithms/classification/adapters/mmcls/utils/builder.py

* make change more readable

* Fix a minor bug of experiment.py (#2662)

fix bug

* Not check avg_time_per_image during test (#2665)

* ignore avg_time_per_image during test

* do not call stdev when length of array is less than 2

* ignore avg_time_per_image during regerssion test

* Update docs for enabling sphinx.ext.autosummary (#2654)

*  fix some errors/warnings on docs source

* enable sphinx-autosummary for API reference documentation

* Update Makefile

* update sphinx configuration

* Update PTQ docs (#2672)

* Replace POT -> PTQ

* Fixes from comments

* Update regression tests for develop (#2652)

* Update regression tests (#2556)

* update reg tests

* update test suit

* update regression criteria

---------

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Exclude py37 target config for cibuildwheel (#2673)

* Add `--dryrun` option to tools/experiment.py (#2674)

* Fix variable override bug

* Add --dryrun option to see experiment list

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update OTX explain CLI arguments (#2671)

* Change int8 to uint8 to XAI tests

* Add probabilities for CLI demo

* Rename arguments for explain

* Fix pre-commit

* Remove extra changes

* Fix integration tests

* Fix integration "explain_all_classes" test for OV

* Fix e2e tests for explain (#2681)

* Add README.md for experiment.py (#2688)

* write draft readme

* refine readme

* align with pre-commit

* Fix typo in reg test cmd (#2691)

* Select more proper model weight file according to commands run just before (#2696)

* consider more complex case when prepare eval and optimize

* update readme

* align with pre-commit

* add comment

* Add visual prompting zero-shot learning (`learn` & `infer`) (#2616)

* Add algobackend & temp configs

* Update config

* WIP

* Fix to enable `algo_backend`

* (WIP) Update dataset

* (WIP) Update configs

* (WIP) Update tasks

* (WIP) Update models

* Enable `learn` task through otx.train

* (WIP) enable `infer` (TODO : normalize points)

* Fix when `state_dict` is None

* Enable `ZeroShotInferenceCallback`

* Enable otx infer

* Enable to independently use processor

* Revert max_steps

* Change `postprocess_masks` to `staticmethod`

* Add `PromptGetter` & Enable `learn` and `infer`

* precommit

* Fix args

* Fix typo

* Change `id` to `id_`

* Fix import

* Fix args

* precommit

* (WIP) Add unit tests

* Fix

* Add unit tests

* Fix

* Add integration tests

* precommit

* Update CHANGELOG.md

* Update docstring and type annotations

* Fix

* precommit

* Fix unused args

* precommit

* Fix

* Fix unsupported dtype in ov graph constant converter (#2676)

* Fix unsupported dtype in ov graph constant converter

* Fix more ov-graph related unit tests

* Skip failure TC with adding issue number ref. (#2717)

* Fix visual prompting e2e test (#2719)

Skip zero-shot e2e

* Remove duplicated variable combination in experiment.py (#2713)

* Enhance detection & instance segmentation experiment (#2710)

* Compute precision and recall along with f-measure

* Log performance

* Accept ellipse annotation from datumaro format

* Fix dataset adapter condition for det/iset

* Insert garbage collection btw experiments

---------

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Songki Choi <songki.choi@intel.com>
Co-authored-by: Yunchu Lee <yunchu.lee@intel.com>
Co-authored-by: Kim, Sungchul <sungchul.kim@intel.com>
Co-authored-by: Vinnam Kim <vinnam.kim@intel.com>
Co-authored-by: Evgeny Tsykunov <evgeny.tsykunov@intel.com>
Co-authored-by: Songki Choi <songki.choi@intel.com>
Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>
Co-authored-by: Sungman Cho <sungman.cho@intel.com>
Co-authored-by: Eugene Liu <eugene.liu@intel.com>
Co-authored-by: Wonju Lee <wonju.lee@intel.com>
Co-authored-by: Dick Ameln <dick.ameln@intel.com>
Co-authored-by: Vladislav Sovrasov <sovrasov.vlad@gmail.com>
Co-authored-by: sungchul.kim <sungchul@ikvensx010>
Co-authored-by: GalyaZalesskaya <galina.zalesskaya@intel.com>
Co-authored-by: Harim Kang <harim.kang@intel.com>
Co-authored-by: Ashwin Vaidya <ashwin.vaidya@intel.com>
Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>
Co-authored-by: sungmanc <sungmanc@intel.com>
sovrasov added a commit that referenced this pull request Feb 20, 2024
@kprokofi @yunchu
@sungchul2 @vinnamkim @negvet @goodsong81 @eunwoosh @jaegukhyun @sungmanc @eugene123tw @wonjuleee @djdameln @sovrasov @GalyaZalesskaya @harimkang @ashwinvaidya17
Merge develop 2 (#2936)
6914551 
* Update base.txt

updated dependency version of datumaro

* Update __init__.py

update version string

* Update requirements.txt

* Temporarily skip visual prompting openvino integration test (#2323)

* Fix import dm.DatasetSubset (#2324)

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Fix semantic segmentation soft prediction dtype (#2322)

* Fix semantic segmentation soft prediction dtype

* relax ref sal vals check

---------

Co-authored-by: Songki Choi <songki.choi@intel.com>

* Contrain yapf verison lesser than 0.40.0 (#2328)

contrain_yapf_version

* Fix detection e2e tests (#2327)

Fix for detection

* Mergeback: Label addtion/deletion 1.2.4 --> 1.4.0 (#2326)

* Make black happy

* Fix conflicts

* Merge-back: add test datasets and edit the test code

* Make black happy

* Fix mis-merge

* Make balck happy

* Fix typo

* Fix typoi

---------

Co-authored-by: Songki Choi <songki.choi@intel.com>

* Bump datumaro up to 1.4.0rc2 (#2332)

bump datumaro up to 1.4.0rc2

* Tiling Doc for releases 1.4.0 (#2333)

* Add tiling documentation

* Bump otx version to 1.4.0rc2 (#2341)

* OTX deploy for visual prompting task  (#2311)

* Enable `otx deploy`

* (WIP) integration test

* Docstring

* Update args for create_model

* Manually set image embedding layout

* Enable to use model api for preprocessing
- `fit_to_window` doesn't work expectedly, so newly implemented `VisualPromptingOpenvinoAdapter` to use new resize function

* Remove skipped test

* Updated

* Update unit tests on model wrappers

* Update

* Update configuration

* Fix not to patch pretrained path

* pylint & update model api version in docstring

---------

Co-authored-by: Wonju Lee <wonju.lee@intel.com>

* Bump albumentations version in anomaly requirements (#2350)

increment albumentations version

* Update action detection (#2346)

* Remove skip mark for PTQ test of action detection

* Update action detection documentation

* Fix e2e (#2348)

* Change classification dataset from dummy to toy

* Revert test changes

* Change label name for multilabel dataset

* Revert e2e test changes

* Change ov test cases' threshold

* Add parent's label

* Update ModelAPI in 1.4 release (#2347)

* Upgrade model API

* Update otx in exportable code

* Fix unit tests

* Fix black

* Fix detection inference

* Fix det tiling

* Fix mypy

* Fix demo

* Fix visualizer in demo

* Fix black

* Add OTX optimize for visual prompting task (#2318)

* Initial commit

* Update block

* (WIP) otx optimize

* Fix

* WIP

* Update configs & exported outputs

* Remove unused modules for torch

* Add unit tests

* pre-commit

* Update CHANGELOG

* Update detection docs (#2335)

* Update detection docs

* Revert template id changes

* Fix wrong template id

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

---------

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Add visual prompting documentation (#2354)

* (WIP) write docs

* Add visual prompting documentation

* Update CHANGELOG

---------

Co-authored-by: sungchul.kim <sungchul@ikvensx010>

* Remove custom modelapi patch in visual prompting (#2359)

* Remove custom modelapi patch

* Update test

* Fix graph metric order and label issues (#2356)

* Fix graph metric going backward issue
* Add license notice
* Fix pre-commit issue
* Add rename items & logic for metric
---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update multi-label document and conversion script (#2358)

Update docs, label convert script

* Update third party programs (#2365)

* Make anomaly task compatible with older albumentations versions (#2363)

* fix transforms export in metadata

* wrap transform dict

* add todo for updating to_dict call

* Fixing detection saliency map for one class case (#2368)

* fix softmax

* fix validity tests

* Add e2e test for visual prompting (#2360)

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* Delete unused configuration.yaml

* Edit test_name

* Add to limit activation range

* Update from `vp` to `visprompt`

* Fix about no returning the first label

* pre-commit

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* pre-commit

* Add actions

* Update tests/e2e/cli/visual_prompting/test_visual_prompting.py

Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>

* Skip PTQ e2e test

* Change task name

* Remove skipped tc

---------

Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>

* Fix e2e (#2366)

* Change e2e reference name

* Update openvino eval threshold for multiclass classification

* Change comment message

* Fix tiling e2e tests

---------

Co-authored-by: GalyaZalesskaya <galina.zalesskaya@intel.com>

* Add Dino head unit tests (#2344)

Recover DINO head unit tests

* Update for release 1.4.0rc2 (#2370)

* update for release 1.4.0rc2

* Add skip mark for unstable unit tests

---------

Co-authored-by: jaegukhyun <jaeguk.hyun@intel.com>

* Fix NNCF training on CPU (#2373)

* Align label order between Geti and OTX (#2369)

* align label order

* align with pre-commit

* update CHANGELOG.md

* deal with edge case

* update type hint

* Remove CenterCrop from Classification test pipeline and editing missing docs link (#2375)

* Fix missing link for docs and removing centercrop for classification data pipeline

* Revert the test threshold

* Fix H-label classification (#2377)

* Fix h-labelissue

* Update unit tests

* Make black happy

* Fix unittests

* Make black happy

* Fix update heades information func

* Update the logic: consider the loss per batch

* Update for release 1.4 (#2380)

* updated for 1.4.0rc3

* update changelog & release note

* bump datumaro version up

---------

Co-authored-by: Songki Choi <songki.choi@intel.com>

* Switch to PTQ for sseg (#2374)

* Switch to PTQ for sseg

* Update log messages

* Fix invalid import structures in otx.api (#2383)

Update tiler.py

* Update for 1.4.0rc4 (#2385)

update for release 1.4.0rc4

* [release 1.4.0] XAI: Return saliency maps for Mask RCNN IR async infer (#2395)

* Return saliency maps for openvino async infer

* add workaround to fix yapf importing error

---------

Co-authored-by: eunwoosh <eunwoo.shin@intel.com>

* Update for release 1.4.0 (#2399)

update version string

Co-authored-by: Sungman Cho <sungman.cho@intel.com>

* Fix broken links in documentation (#2405)

* fix docs links to datumaro's docs
* fix docs links to otx's docs
* bump version to 1.4.1

* Update exportable code README (#2411)

* Updated for release 1.4.1 (#2412)

updated for release 1.4.1

* Add workaround for the incorrect meta info M-RCNN (used for XAI) (#2437)

Add workaround for the incorrect mata info

* Add model category attributes to model template (#2439)

Add model category attributes to model template

* Add model category & status fields in model template

* Add is_default_for_task attr to model template

* Update model templates with category attrs

* Add integration tests for model templates consistency

* Fix license & doc string

* Fix typo

* Refactor test cases

* Refactor common tests by generator

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update for 1.4.2rc1 (#2441)

update for release 1.4.2rc1

* Fix label list order for h-label classification (#2440)

* Fix label list for h-label cls
* Fix unit tests

* Modified fq numbers for lite HRNET (#2445)

modified fq numbers for lite HRNET

* Update PTQ ignored scope for hrnet 18  mod2 (#2449)

Update ptq ignored scope for hrnet 18  mod2

* Fix OpenVINO inference for legacy models (#2450)

* bug fix for legacy openvino models

* Add tests

* Specific exceptions

---------

* Update for 1.4.2rc2 (#2455)

update for release 1.4.2rc2

* Prevent zero-sized saliency map in tiling if tile size is too big (#2452)

* Prevent zero-sized saliency map in tiling if tile size is too big

* Prevent zero-sized saliency in tiling (PyTorch)

* Add unit tests for Tiler merge features methods

---------

Co-authored-by: Galina <galina.zalesskaya@intel.com>

* Update pot fq reference number (#2456)

update pot fq reference number to 15

* Bump datumaro version to 1.5.0rc0 (#2470)

bump datumaro version to 1.5.0rc0

* Set tox version constraint (#2472)

set tox version constraint - tox-dev/tox#3110

* Bug fix for albumentations (#2467)

* bug fix for legacy openvino models

* Address albumentation issue

---------

Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>

* update for release 1.4.2rc3

* Add a dummy hierarchical config required by MAPI (#2483)

* bump version to 1.4.2rc4

* Bump datumaro version (#2502)

* bump datumaro version

* remove deprecated/reomved attribute usage of the datumaro

* Upgrade nncf version for 1.4 release (#2459)

* Upgrade nncf version

* Fix nncf interface warning

* Set the exact nncf version

* Update FQ refs after NNCF upgrade

* Use NNCF from pypi

* Update version for release 1.4.2rc5 (#2507)

update version for release 1.4.2rc5

* Update for 1.4.2 (#2514)

update for release 1.4.2

* create branch release/1.5.0

* Delete mem cache handler after training is done (#2535)

release mem cache handler after training is done

* Fix bug that auto batch size doesn't consider distributed training (#2533)

* consider distributed training while searching batch size

* update unit test

* reveret gpu memory upper bound

* fix typo

* change allocated to reserved

* add unit test for distributed training

* align with pre-commit

* Apply fix progress hook to release 1.5.0 (#2539)

* Fix hook's ordering issue. AdaptiveRepeatHook changes the runner.max_iters before the ProgressHook

* Change the expression

* Fix typo

* Fix multi-label, h-label issue

* Fix auto_bs issue

* Apply suggestions from code review

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Reflecting reviews

* Refactor the name of get_data_cfg

* Revert adaptive hook sampler init

* Refactor the function name: get_data_cfg -> get_subset_data_cfg

* Fix unit test errors

* Remove adding AdaptiveRepeatDataHook for autobs

* Remove unused import

* Fix detection and segmentation case in Geti scenario

---------

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Re introduce adaptive scheduling for training (#2541)

* Re-introduce adaptive patience for training

* Revert unit tests

* Update for release 1.4.3rc1 (#2542)

* Mirror Anomaly ModelAPI changes (#2531)

* Migrate anomaly exportable code to modelAPI (#2432)

* Fix license in PR template

* Migrate to modelAPI

* Remove color conversion in streamer

* Remove reverse_input_channels

* Add float

* Remove test as metadata is no longer used

* Remove metadata from load method

* remove anomalib openvino inferencer

* fix signature

* Support logacy OpenVINO model

* Transform image

* add configs

* Re-introduce adaptive training (#2543)

* Re-introduce adaptive patience for training

* Revert unit tests

* Fix auto input size mismatch in eval & export (#2530)

* Fix auto input size mismatch in eval & export

* Re-enable E2E tests for Issue#2518

* Add input size check in export testing

* Format float numbers in log

* Fix NNCF export shape mismatch

* Fix saliency map issue

* Disable auto input size if tiling enabled

---------

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update ref. fq number for anomaly e2e2 (#2547)

* Skip e2e det tests by issue2548 (#2550)

* Add skip to chained TC for issue #2548 (#2552)

* Update for release 1.4.3 (#2551)

* Update MAPI for 1.5 release (#2555)

Upgrade MAPI to v 0.1.6 (#2529)

* Upgrade MAPI

* Update exp code demo commit

* Fix MAPI imports

* Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Disable QAT for SegNexts (#2565)

* Disable NNCF QAT for SegNext

* Del obsolete pot configs

* Move NNCF skip marks to test commands to avoid duplication

* Add Anomaly modelAPI changes to releases/1.4.0 (#2563)

* bug fix for legacy openvino models

* Apply otx anomaly 1.5 changes

* Fix tests

* Fix compression config

* fix modelAPI imports

* update integration tests

* Edit config types

* Update keys in deployed model

---------

Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>
Co-authored-by: Kim, Sungchul <sungchul.kim@intel.com>

* Fix the CustomNonLinearClsHead when the batch_size is set to 1 (#2571)

Fix bn1d issue

Co-authored-by: sungmanc <sungmanc@intel.com>

* Update ModelAPI configuration (#2564 from 1.4) (#2568)

Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Update for 1.4.4rc1 (#2572)

* Hotfix DatasetEntity.get_combined_subset function loop (#2577)

Fix get_combined_subset function

* Revert default input size to `Default` due to YOLOX perf regression (#2580)

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix for the degradation issue of the classification task (#2585)

* Revert to sync with 1.4.0

* Remove repeat data

* Convert to the RGB value

* Fix color conversion logic

* Fix precommit

* Bump datumaro version to 1.5.1rc3 (#2587)

* Add label ids to anomaly OpenVINO model xml (#2590)

* Add label ids to model xml

---------

* Fix DeiT-Tiny model regression during class incremental training (#2594)

* enable IBloss for DeiT-Tiny

* update changelog

* add docstring

* Add label ids to model xml in release 1.5 (#2591)

Add label ids to model xml

* Fix DeiT-Tiny regression test for release/1.4.0 (#2595)

* Fix DeiT regression test

* update changelog

* temp

* Fix mmcls bug not wrapping model in DataParallel on CPUs (#2601)

Wrap multi-label and h-label classification models by MMDataParallel in case of CPU training.
---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix h-label loss normalization issue w/ exclusive label group of singe label (#2604)

* Fix h-label loss normalization issue w/ exclusive label group with signle label

* Fix non-linear version

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Boost up Image numpy accessing speed through PIL (#2586)

* boost up numpy accessing speed through PIL

* update CHANGELOG

* resolve precommit error

* resolve precommit error

* add fallback logic with PIL open

* use convert instead of draft

* Add missing import pathlib for cls e2e testing (#2610)

* Fix division by zero in class incremental learning for classification (#2606)

* Add empty label to reproduce zero-division error

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix minor typo

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix empty label 4 -> 3

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Prevent division by zero

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update license

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update CHANGELOG.md

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix inefficient sampling

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Revert indexing

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix minor typo

Signed-off-by: Songki Choi <songki.choi@intel.com>

---------

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Unify logger usage (#2612)

* unify logger

* align with pre-commit

* unify anomaly logger to otx

* change logger file path

* align with pre-commit

* change logger file path in missing file

* configure logger after ConfigManager is initialized

* configure logger when ConfigManager instance is initialized

* update unit test code

* move config_logger to each cli file

* align with pre-commit

* change part still using mmcv logger

* Fix XAI algorithm for Detection (#2609)

* Impove saliency maps algorithm for Detection

* Remove extra changes

* Update unit tests

* Changes for 1 class

* Fix pre-commit

* Update CHANGELOG

* Tighten dependency constraint only adapting latest patches (#2607)

* tighten dependency constratint only adapting latest patches

* adjust scikit-image version w.r.t python version

* adjust tensorboard version w.r.t python version

* remove version specifier for scikit-image

* Add metadata to optimized model (#2618)

* bug fix for legacy openvino models

* Add metadata to optimized model

* Revert formatting changes

---------

Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>

* modify omegaconf version constraint

* [release 1.5.0] Fix XAI algorithm for Detection (#2617)

Update detection XAI algorithm

* Update dependency constraint (#2622)

* Update tpp (#2621)

* Fix h-label bug of missing parent labels in output (#2626)

* Fix h-label bug of missing parent labels in output

* Fix h-label test data label schema

* Update CHANGELOG.md

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update publish workflow (#2625)

update publish workflow to push whl to internal pypi

* bump datumaro version to ~=1.5.0

* fixed mistake while mergeing back 1.4.4

* modifiy readme

* remove openvino model wrapper class

* remove openvino model wrapper tests

* [release 1.5.0] DeiT: enable tests + add ViTFeatureVectorHook (#2630)

Add ViT feature vector hook

* Fix docs broken link to datatumaro_h-label

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix wrong label settings for non-anomaly task ModelAPIs

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

Fix e2e XAI ref value

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* update release note and readme

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

* fix datumaro version to 1.6.0rc0

* Mergeback 1.5.0 to develop (#2642)

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

---------

Co-authored-by: Galina Zalesskaya <galina.zalesskaya@intel.com>
Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>

* Revert "Mergeback 1.5.0 to develop" (#2645)

Revert "Mergeback 1.5.0 to develop (#2642)"

This reverts commit 2f67686.

* Add a tool to help conduct experiments (#2651)

* implement run and experiment

* implement experiment result aggregator

* refactor experiment.py

* refactor run.py

* get export model speed

* add var collumn

* refactor experiment.py

* refine a way to update argument in cmd

* refine resource tracker

* support anomaly on research framework

* refine code aggregating exp result

* bugfix

* make other task available

* eval task save avg_time_per_images as result

* Add new argument to track CPU&GPU utilization and memory usage (#2500)

* add argument to track resource usage

* fix bug

* fix a bug in a multi gpu case

* use total cpu usage

* add unit test

* add mark to unit test

* cover edge case

* add pynvml in requirement

* align with pre-commit

* add license comment

* update changelog

* refine argument help

* align with pre-commit

* add version to requirement and raise an error if not supported values are given

* apply new resource tracker format

* refactor run.py

* support optimize in research framework

* cover edge case

* Handle a case where fail cases exist

* make argparse raise error rather than exit if problem exist

* revert tensorboard aggregator

* bugfix

* save failed cases as yaml file

* deal with integer in variables

* add epoch to metric

* use latest log.json file

* align with otx logging method

* move experiment.py from cli to tools

* refactor experiment.py

* merge otx run feature into experiment.py

* move set_arguments_to_cmd definition into experiment.py

* refactor experiment.py

* bugfix

* minor bugfix

* use otx.cli instead of each otx entry

* add feature to parse single workspace

* add comments

* fix bugs

* align with pre-commit

* revert parser argument

* align with pre-commit

* Make `max_num_detections` configurable (#2647)

* Make max_num_detections configurable

* Fix RCNN case with integration test

* Apply max_num_detections to train_cfg, too

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Revert inference batch size to 1 for instance segmentation (#2648)

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix CPU training issue on non-CUDA system (#2655)

Fix bug that auto adaptive batch size raises an error if CUDA isn't available (#2410)

---------
Co-authored-by: Sungman Cho <sungman.cho@intel.com>
Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Remove unnecessary log while building a model (#2658)

* revert logger in otx/algorithms/detection/adapters/mmdet/utils/builder.py

* revert logger in otx/algorithms/classification/adapters/mmcls/utils/builder.py

* make change more readable

* Fix a minor bug of experiment.py (#2662)

fix bug

* Not check avg_time_per_image during test (#2665)

* ignore avg_time_per_image during test

* do not call stdev when length of array is less than 2

* ignore avg_time_per_image during regerssion test

* Update docs for enabling sphinx.ext.autosummary (#2654)

*  fix some errors/warnings on docs source

* enable sphinx-autosummary for API reference documentation

* Update Makefile

* update sphinx configuration

* Update PTQ docs (#2672)

* Replace POT -> PTQ

* Fixes from comments

* Update regression tests for develop (#2652)

* Update regression tests (#2556)

* update reg tests

* update test suit

* update regression criteria

---------

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Exclude py37 target config for cibuildwheel (#2673)

* Add `--dryrun` option to tools/experiment.py (#2674)

* Fix variable override bug

* Add --dryrun option to see experiment list

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update OTX explain CLI arguments (#2671)

* Change int8 to uint8 to XAI tests

* Add probabilities for CLI demo

* Rename arguments for explain

* Fix pre-commit

* Remove extra changes

* Fix integration tests

* Fix integration "explain_all_classes" test for OV

* Fix e2e tests for explain (#2681)

* Add README.md for experiment.py (#2688)

* write draft readme

* refine readme

* align with pre-commit

* Fix typo in reg test cmd (#2691)

* Select more proper model weight file according to commands run just before (#2696)

* consider more complex case when prepare eval and optimize

* update readme

* align with pre-commit

* add comment

* Add visual prompting zero-shot learning (`learn` & `infer`) (#2616)

* Add algobackend & temp configs

* Update config

* WIP

* Fix to enable `algo_backend`

* (WIP) Update dataset

* (WIP) Update configs

* (WIP) Update tasks

* (WIP) Update models

* Enable `learn` task through otx.train

* (WIP) enable `infer` (TODO : normalize points)

* Fix when `state_dict` is None

* Enable `ZeroShotInferenceCallback`

* Enable otx infer

* Enable to independently use processor

* Revert max_steps

* Change `postprocess_masks` to `staticmethod`

* Add `PromptGetter` & Enable `learn` and `infer`

* precommit

* Fix args

* Fix typo

* Change `id` to `id_`

* Fix import

* Fix args

* precommit

* (WIP) Add unit tests

* Fix

* Add unit tests

* Fix

* Add integration tests

* precommit

* Update CHANGELOG.md

* Update docstring and type annotations

* Fix

* precommit

* Fix unused args

* precommit

* Fix

* Fix unsupported dtype in ov graph constant converter (#2676)

* Fix unsupported dtype in ov graph constant converter

* Fix more ov-graph related unit tests

* Skip failure TC with adding issue number ref. (#2717)

* Fix visual prompting e2e test (#2719)

Skip zero-shot e2e

* Remove duplicated variable combination in experiment.py (#2713)

* Enhance detection & instance segmentation experiment (#2710)

* Compute precision and recall along with f-measure

* Log performance

* Accept ellipse annotation from datumaro format

* Fix dataset adapter condition for det/iset

* Insert garbage collection btw experiments

* Upgrade NNCF & OpenVINO (#2656)

* Upgrade OV MAPI and NNCF version

* Update demo requirements

* Update changelog

* Update datumaro

* Add rust installation

* Update NNCF configs for IS models

* Update more fqs

* Exclude nncf from upgrade

* Revert "Update NNCF configs for IS models"

This reverts commit 7c8db8c.

* Revert "Update more fqs"

This reverts commit 5b91c32.

* Revert "Exclude nncf from upgrade"

This reverts commit 8926c51.

* Update FQs

* Revert "Revert "Update NNCF configs for IS models""

This reverts commit f904c0c.

* Disable explain for NNCF detection task

* Update FQs for anomaly

* Update cls FQs

* Update datumaro

* Update exportable code requirements

* Add unit test to cover the changes

* Fix multilabel classification class index (#2736)

Fix multilabel cls

* Refine parsing final score of training in experiment.py (#2738)

refine val parser

* Make mean teacher algorithm consider distributed training (#2729)

* make mean_teacher consider distributed training

* align with pre-commit

* re-enable test case

* move tensor not to cuda but current device

* apply comment

* Add visual prompting zero-shot learning (`export`, IR inference) (#2706)

* Add algobackend & temp configs

* Update config

* WIP

* Fix to enable `algo_backend`

* (WIP) Update dataset

* (WIP) Update configs

* (WIP) Update tasks

* (WIP) Update models

* Enable `learn` task through otx.train

* (WIP) enable `infer` (TODO : normalize points)

* Fix when `state_dict` is None

* Enable `ZeroShotInferenceCallback`

* Enable otx infer

* Enable to independently use processor

* Revert max_steps

* Change `postprocess_masks` to `staticmethod`

* Add `PromptGetter` & Enable `learn` and `infer`

* precommit

* Fix args

* Fix typo

* Change `id` to `id_`

* Fix import

* Fix args

* precommit

* (WIP) Add unit tests

* Fix

* Add unit tests

* Fix

* Add integration tests

* precommit

* Update CHANGELOG.md

* Update docstring and type annotations

* Fix

* precommit

* Reuse SAM modules for `export` & Add dataset

* Fix

* Enable `export`

* Convert fp32

* Update logic & tests

* Fix & Add prompt getter in `model_adapter_keys`

* Initial `Inferencer`, `Task`, and `Model`

* Fix to use original mask decoder during inference

* Remove internal loop in `PromptGetter`

* Update IO

* (WIP) Add unit tests for export

* Update `PromptGetter` to use only tensor ops

* Fix issue about `original_size` disappear in onnx graph

* (WIP) Add export unit test

* Update

* Fix typo

* Update

* Fix unexpected IF & Update inputs to avoid issues which OV on CPU doesn't support dynamic operations

* Enable `PromptGetter` to handle #labels itself

* Add ov inferencer

* Fix overflow during casting dtype & duplicated cast

* Fix

* Add unit&integration tests

* pre-commit

* Fix original vpms

* Fix intg & e2e tests

* Change mo CLI to API

* precommit

* Remove blocks

* Update CHANGELOG.md

* Avoid repeatedly assigning constant tensors/arrays

* Fix typo

* Automate performance benchmark (#2742)

* Add parameterized perf test template

* Split acccuracy / perf tests

* Automate speed test setting

* Add benchmark summary fixture

* Add multi/h-label tests

* Add detection tests

* Add instance segmentationt tests

* Add tiling tests

* Add semantic segmenation tests

* Add anomaly test

* Update tools/expreiment.py (#2751)

* have constant exp directory name

* support to parse dynamic eval output

* align with pre-commit

* fix minor unit test bug

* Add performance benchmark github action workflow (#2762)

* Split accuracy & speed benchmark github workflows (#2763)

* Fix a bug that error is raised when train set size is greater than minimumof batch size in HPO by exactly 1 (#2760)

deal with HPO edge case

* Fix a bug that a process tracking resource usage doesn't exit when main process raises an error (#2765)

* termiate a process tracking resource usage if main process raises an error

* call stop() only if ExitStack is None

* Skip large datasets for iSeg perf benchmark (#2766)

* Support multiple experiments in single recipe for tools/experiment.py (#2757)

* implement draft version

* update logging failed cases

* align with pre-commit

* add doc string

* Update README file

* fix bugs: single command, failed case output

* exclude first epoch from calculating iter time

* fix weird name used when there is no variables

* align with pre-commit

* initialize iter_time and data_time at first

* Enable perf benchmark result logging to mlflow server (#2768)

* Bump datumaro version to 1.6.0rc1 (#2784)

* bump datumaro version to 1.6.0rc1

* remove rust toolchain installation step from workflows

* Update perf logging (#2785)

* Update perf logging workflow to get branch+sha from gh context (#2791)

* update perf logging workflow to get branch+sha from gh context
* skip logging when tracking server uri is not configured

* Add visual prompting zero-shot learning (optimize, documentation, bug fixes) (#2753)

* Fix to resize bbox

* (WIP) Add post-checking between masks with different labels

* Fix to use the first mask in the first loop

* Add post-checking between masks with different labels

* pre-commit

* Add optimize task

* pre-commit

* Add e2e

* Update documentation

* Update CHANGELOG

* Check performance benchmark result with reference (#2821)

* Average 'Small' (/1 /2 /3) dataset benchmark results

* Load perf result with indexing

* Add speed ref check for all tasks

* Add accuracy ref check for all tasks

* Mergeback releases/1.5.0 to develop (#2830)

* Update MAPI version (#2730)

* Update dependency for exportable code (#2732)

* Filter invalid polygon shapes (#2795)

---------

Co-authored-by: Vladislav Sovrasov <sovrasov.vlad@gmail.com>
Co-authored-by: Eugene Liu <eugene.liu@intel.com>

* Create OSSF scorecard workflow (#2831)

* Fix ossf/scorecard-action version (#2832)

* Update scorecard.yml

* Update perf benchmark reference (#2843)

* Set default wf permission to read-all (#2882)

* Remedy token permission issue (#2888)

* remedy token-permission issues - part2

* removed dispatch event from scorecard wf

* Add progress callback interface to HPO (#2889)

* add progress callback as HPO argument

* deal with edge case

* Restrict configurable parameters to avoid unreasonable cost for SaaS trial (#2891)

* Reduce max value of POT samples to 1k

* Reduce max value of num_iters to 1k

* Fix pre-commit

* Fix more token-permission issues - part3 (#2893)

* Resolve pinned-dependency issues on publish_internal workflow (#2907)

* Forward unittest workloads to AWS (#2887)

* Resolve pinned dependency issues on workflows (#2909)

* Fix pinned-dependency issues - part2 (#2911)

* Add pinning dependencies (#2916)

* Update pip install cmd to use hashes (#2919)

* Fix HPO progress callback bug (#2908)

fix minor bug

* Fix pinned-dependencies issues (#2929)

* Remove unused test files (#2930)

* Update weekly workflow to run perf tests (#2920)

* update weekly workflow to run perf tests

* Fix missing fixture in perf test

* update input to perf tests for weekly

---------

Co-authored-by: Songki Choi <songki.choi@intel.com>

* Adjust permission of documentation workflows from pages to contents for writing (#2933)

* remove unused import

---------

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Songki Choi <songki.choi@intel.com>
Co-authored-by: Yunchu Lee <yunchu.lee@intel.com>
Co-authored-by: Kim, Sungchul <sungchul.kim@intel.com>
Co-authored-by: Vinnam Kim <vinnam.kim@intel.com>
Co-authored-by: Evgeny Tsykunov <evgeny.tsykunov@intel.com>
Co-authored-by: Songki Choi <songki.choi@intel.com>
Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>
Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>
Co-authored-by: Sungman Cho <sungman.cho@intel.com>
Co-authored-by: Eugene Liu <eugene.liu@intel.com>
Co-authored-by: Wonju Lee <wonju.lee@intel.com>
Co-authored-by: Dick Ameln <dick.ameln@intel.com>
Co-authored-by: Vladislav Sovrasov <sovrasov.vlad@gmail.com>
Co-authored-by: sungchul.kim <sungchul@ikvensx010>
Co-authored-by: GalyaZalesskaya <galina.zalesskaya@intel.com>
Co-authored-by: Harim Kang <harim.kang@intel.com>
Co-authored-by: Ashwin Vaidya <ashwin.vaidya@intel.com>
Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>
Co-authored-by: sungmanc <sungmanc@intel.com>
eunwoosh added a commit that referenced this pull request Mar 11, 2024
@kprokofi @sovrasov
@goodsong81 @sungchul2 @eunwoosh @yunchu @vinnamkim @negvet @jaegukhyun @sungmanc @eugene123tw @wonjuleee @djdameln @GalyaZalesskaya @harimkang @ashwinvaidya17 @chuneuny-emily
Add support of the Intel dGPUs devices (#3058)
ccbfac1 
* POC: xpu support (#2553)

* Upgrade torch==2.0.1 torchvision==0.15.2 mmcv-full=1.7.1

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Rollback to mmcv-full==1.7.0 due to mmaction conflict

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix unit test

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update cls to run on xpu

* Enable XPU for detection

* Disable FP16 for detection models

* Add xpu data parallel to core utils

* Del copys of XPUDP

* Update sseg XPU support

* Update XPUDP

* added fix for OD and IS. Not debugged

* Upgrade torch==2.0.1 torchvision==0.15.2 mmcv-full=1.7.1

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Rollback to mmcv-full==1.7.0 due to mmaction conflict

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix unit test

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update cls to run on xpu

* Enable XPU for detection

* Disable FP16 for detection models

* Add xpu data parallel to core utils

* Del copys of XPUDP

* Update sseg XPU support

* Update XPUDP

* Fix import of assigner

* Add mmdet ops patching to inference

* Add xpu seed

* Fix linters

* Cleanup in cls train runner

* Cleanup in segm train function

* Fix linters

* Fix linters

* Cleanup

* Update NMS patch

* Use GPU id to move model on target XPU

* Disable fp16 for swinT

* Add XPU device info to OTX env report

* Fix linters

* Update new inits

* Update autocast

* Update fallback to CPU

* Update target device in XPU data parallel

* Revert "Update autocast"

This reverts commit 0ff53ebccd2007fc38e387919e306d476a695635.

* Revert "Patch to enable FP16 on XPU"

This reverts commit fd91f6744f238546c92ae46a7d193369aebc17b4.

* FP16 support for XPU

* Enable FP 16

* Patch to enable FP16 on XPU

* Fix linters

---------

Signed-off-by: Songki Choi <songki.choi@intel.com>
Co-authored-by: Songki Choi <songki.choi@intel.com>
Co-authored-by: kprokofi <kirill.prokofiev@intel.com>

* Fix tv nms check (#2582)

* POC: HPU support (#2574)

* Add hpu.is_available

* Add hpu in `configure_device`

* Add hpu optimizers

* Update `train_model`

* Add `HPUDataParallel`

* Fix

* Enable HPUOptimizerHooks

* Enable autocast

* Update

* (WIP) Enable dataloader

* (tmp) Disenable habana loader

* Fix to convert bfloat16 tensor to float32 for numpy

* Update `htcore.mark_step` location

* Move setting optimizer hooks into configurer

* Fix to enable bf16

* (WIP) Enable gpu migration

* enable ocr_lite_hrnet_18_mod2 training

* register hpu optimizer once

* detch seg logit for calculating acc only when hpu is used

* change hpu_optimizer path

* cache whether hpu_available

* align with other args

* move hpu_migration into otx.algo.common.util init file

* Refactoring

* move the code changing adam to adamw into mmseg.apis.train.py

* move the code model.to in mmseg.api.train.py

* use use_autocast properly

* remove HPUDistOptimizerHook

* precommit

* Fix unit test

* Use `self.src_device_obj`

* Add function for repeated pattern

* precommit

---------

Co-authored-by: Shin, Eunwoo <eunwoo.shin@intel.com>

* Merge develop & fix YOLOX inference in BFP16 on XPU (#2602)

Fix yolox with bfp16 dtype on xpu

* Object Detection with Gaudi2 (#2608)

* added support for OD on habana

* added hpu_opt

* added OD support.

* optimize a bit YOLOX. Now, inference is fast. Training still freezes

* SSD, ATSS e2e training

* stabilize mask rcnn a bit

* don't put gt on hpu for OD

* minor fix

* Enable e2e training for Instance Segmentation.

* clean the code stage 1

* clean code 2

* fix pre-commit

* minor

* change cast of bf16

* reply comments

* align with pre-commit

---------

Co-authored-by: eunwoosh <eunwoo.shin@intel.com>

* Update XPU detection (#2623)

* Add `ModuleCacher` for HPU graphs (#2624)

* Add `ModuleCacher` on classification and segmentation

* Fix

* Update Intel devices branch to the latest develop state (#2666)

* Update base.txt

updated dependency version of datumaro

* Update __init__.py

update version string

* Update requirements.txt

* Temporarily skip visual prompting openvino integration test (#2323)

* Fix import dm.DatasetSubset (#2324)

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Fix semantic segmentation soft prediction dtype (#2322)

* Fix semantic segmentation soft prediction dtype

* relax ref sal vals check

---------

Co-authored-by: Songki Choi <songki.choi@intel.com>

* Contrain yapf verison lesser than 0.40.0 (#2328)

contrain_yapf_version

* Fix detection e2e tests (#2327)

Fix for detection

* Mergeback: Label addtion/deletion 1.2.4 --> 1.4.0 (#2326)

* Make black happy

* Fix conflicts

* Merge-back: add test datasets and edit the test code

* Make black happy

* Fix mis-merge

* Make balck happy

* Fix typo

* Fix typoi

---------

Co-authored-by: Songki Choi <songki.choi@intel.com>

* Bump datumaro up to 1.4.0rc2 (#2332)

bump datumaro up to 1.4.0rc2

* Tiling Doc for releases 1.4.0 (#2333)

* Add tiling documentation

* Bump otx version to 1.4.0rc2 (#2341)

* OTX deploy for visual prompting task  (#2311)

* Enable `otx deploy`

* (WIP) integration test

* Docstring

* Update args for create_model

* Manually set image embedding layout

* Enable to use model api for preprocessing
- `fit_to_window` doesn't work expectedly, so newly implemented `VisualPromptingOpenvinoAdapter` to use new resize function

* Remove skipped test

* Updated

* Update unit tests on model wrappers

* Update

* Update configuration

* Fix not to patch pretrained path

* pylint & update model api version in docstring

---------

Co-authored-by: Wonju Lee <wonju.lee@intel.com>

* Bump albumentations version in anomaly requirements (#2350)

increment albumentations version

* Update action detection (#2346)

* Remove skip mark for PTQ test of action detection

* Update action detection documentation

* Fix e2e (#2348)

* Change classification dataset from dummy to toy

* Revert test changes

* Change label name for multilabel dataset

* Revert e2e test changes

* Change ov test cases' threshold

* Add parent's label

* Update ModelAPI in 1.4 release (#2347)

* Upgrade model API

* Update otx in exportable code

* Fix unit tests

* Fix black

* Fix detection inference

* Fix det tiling

* Fix mypy

* Fix demo

* Fix visualizer in demo

* Fix black

* Add OTX optimize for visual prompting task (#2318)

* Initial commit

* Update block

* (WIP) otx optimize

* Fix

* WIP

* Update configs & exported outputs

* Remove unused modules for torch

* Add unit tests

* pre-commit

* Update CHANGELOG

* Update detection docs (#2335)

* Update detection docs

* Revert template id changes

* Fix wrong template id

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

---------

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Add visual prompting documentation (#2354)

* (WIP) write docs

* Add visual prompting documentation

* Update CHANGELOG

---------

Co-authored-by: sungchul.kim <sungchul@ikvensx010>

* Remove custom modelapi patch in visual prompting (#2359)

* Remove custom modelapi patch

* Update test

* Fix graph metric order and label issues (#2356)

* Fix graph metric going backward issue
* Add license notice
* Fix pre-commit issue
* Add rename items & logic for metric
---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update multi-label document and conversion script (#2358)

Update docs, label convert script

* Update third party programs (#2365)

* Make anomaly task compatible with older albumentations versions (#2363)

* fix transforms export in metadata

* wrap transform dict

* add todo for updating to_dict call

* Fixing detection saliency map for one class case (#2368)

* fix softmax

* fix validity tests

* Add e2e test for visual prompting (#2360)

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* Delete unused configuration.yaml

* Edit test_name

* Add to limit activation range

* Update from `vp` to `visprompt`

* Fix about no returning the first label

* pre-commit

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* pre-commit

* Add actions

* Update tests/e2e/cli/visual_prompting/test_visual_prompting.py

Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>

* Skip PTQ e2e test

* Change task name

* Remove skipped tc

---------

Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>

* Fix e2e (#2366)

* Change e2e reference name

* Update openvino eval threshold for multiclass classification

* Change comment message

* Fix tiling e2e tests

---------

Co-authored-by: GalyaZalesskaya <galina.zalesskaya@intel.com>

* Add Dino head unit tests (#2344)

Recover DINO head unit tests

* Update for release 1.4.0rc2 (#2370)

* update for release 1.4.0rc2

* Add skip mark for unstable unit tests

---------

Co-authored-by: jaegukhyun <jaeguk.hyun@intel.com>

* Fix NNCF training on CPU (#2373)

* Align label order between Geti and OTX (#2369)

* align label order

* align with pre-commit

* update CHANGELOG.md

* deal with edge case

* update type hint

* Remove CenterCrop from Classification test pipeline and editing missing docs link (#2375)

* Fix missing link for docs and removing centercrop for classification data pipeline

* Revert the test threshold

* Fix H-label classification (#2377)

* Fix h-labelissue

* Update unit tests

* Make black happy

* Fix unittests

* Make black happy

* Fix update heades information func

* Update the logic: consider the loss per batch

* Update for release 1.4 (#2380)

* updated for 1.4.0rc3

* update changelog & release note

* bump datumaro version up

---------

Co-authored-by: Songki Choi <songki.choi@intel.com>

* Switch to PTQ for sseg (#2374)

* Switch to PTQ for sseg

* Update log messages

* Fix invalid import structures in otx.api (#2383)

Update tiler.py

* Update for 1.4.0rc4 (#2385)

update for release 1.4.0rc4

* [release 1.4.0] XAI: Return saliency maps for Mask RCNN IR async infer (#2395)

* Return saliency maps for openvino async infer

* add workaround to fix yapf importing error

---------

Co-authored-by: eunwoosh <eunwoo.shin@intel.com>

* Update for release 1.4.0 (#2399)

update version string

Co-authored-by: Sungman Cho <sungman.cho@intel.com>

* Fix broken links in documentation (#2405)

* fix docs links to datumaro's docs
* fix docs links to otx's docs
* bump version to 1.4.1

* Update exportable code README (#2411)

* Updated for release 1.4.1 (#2412)

updated for release 1.4.1

* Add workaround for the incorrect meta info M-RCNN (used for XAI) (#2437)

Add workaround for the incorrect mata info

* Add model category attributes to model template (#2439)

Add model category attributes to model template

* Add model category & status fields in model template

* Add is_default_for_task attr to model template

* Update model templates with category attrs

* Add integration tests for model templates consistency

* Fix license & doc string

* Fix typo

* Refactor test cases

* Refactor common tests by generator

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update for 1.4.2rc1 (#2441)

update for release 1.4.2rc1

* Fix label list order for h-label classification (#2440)

* Fix label list for h-label cls
* Fix unit tests

* Modified fq numbers for lite HRNET (#2445)

modified fq numbers for lite HRNET

* Update PTQ ignored scope for hrnet 18  mod2 (#2449)

Update ptq ignored scope for hrnet 18  mod2

* Fix OpenVINO inference for legacy models (#2450)

* bug fix for legacy openvino models

* Add tests

* Specific exceptions

---------

* Update for 1.4.2rc2 (#2455)

update for release 1.4.2rc2

* Prevent zero-sized saliency map in tiling if tile size is too big (#2452)

* Prevent zero-sized saliency map in tiling if tile size is too big

* Prevent zero-sized saliency in tiling (PyTorch)

* Add unit tests for Tiler merge features methods

---------

Co-authored-by: Galina <galina.zalesskaya@intel.com>

* Update pot fq reference number (#2456)

update pot fq reference number to 15

* Bump datumaro version to 1.5.0rc0 (#2470)

bump datumaro version to 1.5.0rc0

* Set tox version constraint (#2472)

set tox version constraint - https://github.com/tox-dev/tox/issues/3110

* Bug fix for albumentations (#2467)

* bug fix for legacy openvino models

* Address albumentation issue

---------

Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>

* update for release 1.4.2rc3

* Add a dummy hierarchical config required by MAPI (#2483)

* bump version to 1.4.2rc4

* Bump datumaro version (#2502)

* bump datumaro version

* remove deprecated/reomved attribute usage of the datumaro

* Upgrade nncf version for 1.4 release (#2459)

* Upgrade nncf version

* Fix nncf interface warning

* Set the exact nncf version

* Update FQ refs after NNCF upgrade

* Use NNCF from pypi

* Update version for release 1.4.2rc5 (#2507)

update version for release 1.4.2rc5

* Update for 1.4.2 (#2514)

update for release 1.4.2

* create branch release/1.5.0

* Delete mem cache handler after training is done (#2535)

release mem cache handler after training is done

* Fix bug that auto batch size doesn't consider distributed training (#2533)

* consider distributed training while searching batch size

* update unit test

* reveret gpu memory upper bound

* fix typo

* change allocated to reserved

* add unit test for distributed training

* align with pre-commit

* Apply fix progress hook to release 1.5.0 (#2539)

* Fix hook's ordering issue. AdaptiveRepeatHook changes the runner.max_iters before the ProgressHook

* Change the expression

* Fix typo

* Fix multi-label, h-label issue

* Fix auto_bs issue

* Apply suggestions from code review

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Reflecting reviews

* Refactor the name of get_data_cfg

* Revert adaptive hook sampler init

* Refactor the function name: get_data_cfg -> get_subset_data_cfg

* Fix unit test errors

* Remove adding AdaptiveRepeatDataHook for autobs

* Remove unused import

* Fix detection and segmentation case in Geti scenario

---------

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Re introduce adaptive scheduling for training (#2541)

* Re-introduce adaptive patience for training

* Revert unit tests

* Update for release 1.4.3rc1 (#2542)

* Mirror Anomaly ModelAPI changes (#2531)

* Migrate anomaly exportable code to modelAPI (#2432)

* Fix license in PR template

* Migrate to modelAPI

* Remove color conversion in streamer

* Remove reverse_input_channels

* Add float

* Remove test as metadata is no longer used

* Remove metadata from load method

* remove anomalib openvino inferencer

* fix signature

* Support logacy OpenVINO model

* Transform image

* add configs

* Re-introduce adaptive training (#2543)

* Re-introduce adaptive patience for training

* Revert unit tests

* Fix auto input size mismatch in eval & export (#2530)

* Fix auto input size mismatch in eval & export

* Re-enable E2E tests for Issue#2518

* Add input size check in export testing

* Format float numbers in log

* Fix NNCF export shape mismatch

* Fix saliency map issue

* Disable auto input size if tiling enabled

---------

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update ref. fq number for anomaly e2e2 (#2547)

* Skip e2e det tests by issue2548 (#2550)

* Add skip to chained TC for issue #2548 (#2552)

* Update for release 1.4.3 (#2551)

* Update MAPI for 1.5 release (#2555)

Upgrade MAPI to v 0.1.6 (#2529)

* Upgrade MAPI

* Update exp code demo commit

* Fix MAPI imports

* Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Disable QAT for SegNexts (#2565)

* Disable NNCF QAT for SegNext

* Del obsolete pot configs

* Move NNCF skip marks to test commands to avoid duplication

* Add Anomaly modelAPI changes to releases/1.4.0 (#2563)

* bug fix for legacy openvino models

* Apply otx anomaly 1.5 changes

* Fix tests

* Fix compression config

* fix modelAPI imports

* update integration tests

* Edit config types

* Update keys in deployed model

---------

Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>
Co-authored-by: Kim, Sungchul <sungchul.kim@intel.com>

* Fix the CustomNonLinearClsHead when the batch_size is set to 1 (#2571)

Fix bn1d issue

Co-authored-by: sungmanc <sungmanc@intel.com>

* Update ModelAPI configuration (#2564 from 1.4) (#2568)

Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Update for 1.4.4rc1 (#2572)

* Hotfix DatasetEntity.get_combined_subset function loop (#2577)

Fix get_combined_subset function

* Revert default input size to `Default` due to YOLOX perf regression (#2580)

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix for the degradation issue of the classification task (#2585)

* Revert to sync with 1.4.0

* Remove repeat data

* Convert to the RGB value

* Fix color conversion logic

* Fix precommit

* Bump datumaro version to 1.5.1rc3 (#2587)

* Add label ids to anomaly OpenVINO model xml (#2590)

* Add label ids to model xml

---------

* Fix DeiT-Tiny model regression during class incremental training (#2594)

* enable IBloss for DeiT-Tiny

* update changelog

* add docstring

* Add label ids to model xml in release 1.5 (#2591)

Add label ids to model xml

* Fix DeiT-Tiny regression test for release/1.4.0 (#2595)

* Fix DeiT regression test

* update changelog

* temp

* Fix mmcls bug not wrapping model in DataParallel on CPUs (#2601)

Wrap multi-label and h-label classification models by MMDataParallel in case of CPU training.
---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix h-label loss normalization issue w/ exclusive label group of singe label (#2604)

* Fix h-label loss normalization issue w/ exclusive label group with signle label

* Fix non-linear version

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Boost up Image numpy accessing speed through PIL (#2586)

* boost up numpy accessing speed through PIL

* update CHANGELOG

* resolve precommit error

* resolve precommit error

* add fallback logic with PIL open

* use convert instead of draft

* Add missing import pathlib for cls e2e testing (#2610)

* Fix division by zero in class incremental learning for classification (#2606)

* Add empty label to reproduce zero-division error

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix minor typo

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix empty label 4 -> 3

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Prevent division by zero

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update license

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update CHANGELOG.md

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix inefficient sampling

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Revert indexing

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix minor typo

Signed-off-by: Songki Choi <songki.choi@intel.com>

---------

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Unify logger usage (#2612)

* unify logger

* align with pre-commit

* unify anomaly logger to otx

* change logger file path

* align with pre-commit

* change logger file path in missing file

* configure logger after ConfigManager is initialized

* configure logger when ConfigManager instance is initialized

* update unit test code

* move config_logger to each cli file

* align with pre-commit

* change part still using mmcv logger

* Fix XAI algorithm for Detection (#2609)

* Impove saliency maps algorithm for Detection

* Remove extra changes

* Update unit tests

* Changes for 1 class

* Fix pre-commit

* Update CHANGELOG

* Tighten dependency constraint only adapting latest patches (#2607)

* tighten dependency constratint only adapting latest patches

* adjust scikit-image version w.r.t python version

* adjust tensorboard version w.r.t python version

* remove version specifier for scikit-image

* Add metadata to optimized model (#2618)

* bug fix for legacy openvino models

* Add metadata to optimized model

* Revert formatting changes

---------

Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>

* modify omegaconf version constraint

* [release 1.5.0] Fix XAI algorithm for Detection (#2617)

Update detection XAI algorithm

* Update dependency constraint (#2622)

* Update tpp (#2621)

* Fix h-label bug of missing parent labels in output (#2626)

* Fix h-label bug of missing parent labels in output

* Fix h-label test data label schema

* Update CHANGELOG.md

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update publish workflow (#2625)

update publish workflow to push whl to internal pypi

* bump datumaro version to ~=1.5.0

* fixed mistake while mergeing back 1.4.4

* modifiy readme

* remove openvino model wrapper class

* remove openvino model wrapper tests

* [release 1.5.0] DeiT: enable tests + add ViTFeatureVectorHook (#2630)

Add ViT feature vector hook

* Fix docs broken link to datatumaro_h-label

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix wrong label settings for non-anomaly task ModelAPIs

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

Fix e2e XAI ref value

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* update release note and readme

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

* fix datumaro version to 1.6.0rc0

* Mergeback 1.5.0 to develop (#2642)

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

---------

Co-authored-by: Galina Zalesskaya <galina.zalesskaya@intel.com>
Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>

* Revert "Mergeback 1.5.0 to develop" (#2645)

Revert "Mergeback 1.5.0 to develop (#2642)"

This reverts commit 2f67686103df873d020681f6d504f9595ce4a963.

* Add a tool to help conduct experiments (#2651)

* implement run and experiment

* implement experiment result aggregator

* refactor experiment.py

* refactor run.py

* get export model speed

* add var collumn

* refactor experiment.py

* refine a way to update argument in cmd

* refine resource tracker

* support anomaly on research framework

* refine code aggregating exp result

* bugfix

* make other task available

* eval task save avg_time_per_images as result

* Add new argument to track CPU&GPU utilization and memory usage (#2500)

* add argument to track resource usage

* fix bug

* fix a bug in a multi gpu case

* use total cpu usage

* add unit test

* add mark to unit test

* cover edge case

* add pynvml in requirement

* align with pre-commit

* add license comment

* update changelog

* refine argument help

* align with pre-commit

* add version to requirement and raise an error if not supported values are given

* apply new resource tracker format

* refactor run.py

* support optimize in research framework

* cover edge case

* Handle a case where fail cases exist

* make argparse raise error rather than exit if problem exist

* revert tensorboard aggregator

* bugfix

* save failed cases as yaml file

* deal with integer in variables

* add epoch to metric

* use latest log.json file

* align with otx logging method

* move experiment.py from cli to tools

* refactor experiment.py

* merge otx run feature into experiment.py

* move set_arguments_to_cmd definition into experiment.py

* refactor experiment.py

* bugfix

* minor bugfix

* use otx.cli instead of each otx entry

* add feature to parse single workspace

* add comments

* fix bugs

* align with pre-commit

* revert parser argument

* align with pre-commit

* Revert inference batch size to 1 for instance segmentation (#2648)

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Remove unnecessary log while building a model (#2658)

* revert logger in otx/algorithms/detection/adapters/mmdet/utils/builder.py

* revert logger in otx/algorithms/classification/adapters/mmcls/utils/builder.py

* make change more readable

* Fix a minor bug of experiment.py (#2662)

fix bug

* Not check avg_time_per_image during test (#2665)

* ignore avg_time_per_image during test

* do not call stdev when length of array is less than 2

* ignore avg_time_per_image during regerssion test

* Update device selection logic in classificaiton

---------

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Songki Choi <songki.choi@intel.com>
Co-authored-by: Yunchu Lee <yunchu.lee@intel.com>
Co-authored-by: Kim, Sungchul <sungchul.kim@intel.com>
Co-authored-by: Vinnam Kim <vinnam.kim@intel.com>
Co-authored-by: Evgeny Tsykunov <evgeny.tsykunov@intel.com>
Co-authored-by: Songki Choi <songki.choi@intel.com>
Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>
Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>
Co-authored-by: Sungman Cho <sungman.cho@intel.com>
Co-authored-by: Eugene Liu <eugene.liu@intel.com>
Co-authored-by: Wonju Lee <wonju.lee@intel.com>
Co-authored-by: Dick Ameln <dick.ameln@intel.com>
Co-authored-by: sungchul.kim <sungchul@ikvensx010>
Co-authored-by: GalyaZalesskaya <galina.zalesskaya@intel.com>
Co-authored-by: Harim Kang <harim.kang@intel.com>
Co-authored-by: Ashwin Vaidya <ashwin.vaidya@intel.com>
Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>
Co-authored-by: sungmanc <sungmanc@intel.com>

* Fix mixed & lower precision training (#2668)

* remove dtype argument in torch.xpu.optimize

* Add `custom_auto_fp16` to use xpu autocast

* Update `forward`s to use `custom_auto_fp16`

* precommit

* Disable FP16 training

* Add `custom_force_fp32`

* Removed what force casting tensors to bf16

* Add `XPUOptimizerHook` and `XPUGradScaler`

* precommit

* Enable lower precision training

* Remove dtype check for lower precision

* Add `bf16_training` in recipe

* fix

* Remove unused module

* Change `XPUOptimizerHook` to `BFp16XPUOptimizerHook`

* Fix for common devices which don't use bf16

* precommit

* Enable to use `auto_fp16` as it is

* Add try-except avoiding mmcv import error

* Fix error type

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

---------

Co-authored-by: Shin, Eunwoo <eunwoo.shin@intel.com>

* Add XPU support to anomaly task (#2677)

* Update base.txt

updated dependency version of datumaro

* Update __init__.py

update version string

* Update requirements.txt

* Temporarily skip visual prompting openvino integration test (#2323)

* Fix import dm.DatasetSubset (#2324)

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Fix semantic segmentation soft prediction dtype (#2322)

* Fix semantic segmentation soft prediction dtype

* relax ref sal vals check

---------

Co-authored-by: Songki Choi <songki.choi@intel.com>

* Contrain yapf verison lesser than 0.40.0 (#2328)

contrain_yapf_version

* Fix detection e2e tests (#2327)

Fix for detection

* Mergeback: Label addtion/deletion 1.2.4 --> 1.4.0 (#2326)

* Make black happy

* Fix conflicts

* Merge-back: add test datasets and edit the test code

* Make black happy

* Fix mis-merge

* Make balck happy

* Fix typo

* Fix typoi

---------

Co-authored-by: Songki Choi <songki.choi@intel.com>

* Bump datumaro up to 1.4.0rc2 (#2332)

bump datumaro up to 1.4.0rc2

* Tiling Doc for releases 1.4.0 (#2333)

* Add tiling documentation

* Bump otx version to 1.4.0rc2 (#2341)

* OTX deploy for visual prompting task  (#2311)

* Enable `otx deploy`

* (WIP) integration test

* Docstring

* Update args for create_model

* Manually set image embedding layout

* Enable to use model api for preprocessing
- `fit_to_window` doesn't work expectedly, so newly implemented `VisualPromptingOpenvinoAdapter` to use new resize function

* Remove skipped test

* Updated

* Update unit tests on model wrappers

* Update

* Update configuration

* Fix not to patch pretrained path

* pylint & update model api version in docstring

---------

Co-authored-by: Wonju Lee <wonju.lee@intel.com>

* Bump albumentations version in anomaly requirements (#2350)

increment albumentations version

* Update action detection (#2346)

* Remove skip mark for PTQ test of action detection

* Update action detection documentation

* Fix e2e (#2348)

* Change classification dataset from dummy to toy

* Revert test changes

* Change label name for multilabel dataset

* Revert e2e test changes

* Change ov test cases' threshold

* Add parent's label

* Update ModelAPI in 1.4 release (#2347)

* Upgrade model API

* Update otx in exportable code

* Fix unit tests

* Fix black

* Fix detection inference

* Fix det tiling

* Fix mypy

* Fix demo

* Fix visualizer in demo

* Fix black

* Add OTX optimize for visual prompting task (#2318)

* Initial commit

* Update block

* (WIP) otx optimize

* Fix

* WIP

* Update configs & exported outputs

* Remove unused modules for torch

* Add unit tests

* pre-commit

* Update CHANGELOG

* Update detection docs (#2335)

* Update detection docs

* Revert template id changes

* Fix wrong template id

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

---------

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Add visual prompting documentation (#2354)

* (WIP) write docs

* Add visual prompting documentation

* Update CHANGELOG

---------

Co-authored-by: sungchul.kim <sungchul@ikvensx010>

* Remove custom modelapi patch in visual prompting (#2359)

* Remove custom modelapi patch

* Update test

* Fix graph metric order and label issues (#2356)

* Fix graph metric going backward issue
* Add license notice
* Fix pre-commit issue
* Add rename items & logic for metric
---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update multi-label document and conversion script (#2358)

Update docs, label convert script

* Update third party programs (#2365)

* Make anomaly task compatible with older albumentations versions (#2363)

* fix transforms export in metadata

* wrap transform dict

* add todo for updating to_dict call

* Fixing detection saliency map for one class case (#2368)

* fix softmax

* fix validity tests

* Add e2e test for visual prompting (#2360)

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* Delete unused configuration.yaml

* Edit test_name

* Add to limit activation range

* Update from `vp` to `visprompt`

* Fix about no returning the first label

* pre-commit

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* pre-commit

* Add actions

* Update tests/e2e/cli/visual_prompting/test_visual_prompting.py

Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>

* Skip PTQ e2e test

* Change task name

* Remove skipped tc

---------

Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>

* Fix e2e (#2366)

* Change e2e reference name

* Update openvino eval threshold for multiclass classification

* Change comment message

* Fix tiling e2e tests

---------

Co-authored-by: GalyaZalesskaya <galina.zalesskaya@intel.com>

* Add Dino head unit tests (#2344)

Recover DINO head unit tests

* Update for release 1.4.0rc2 (#2370)

* update for release 1.4.0rc2

* Add skip mark for unstable unit tests

---------

Co-authored-by: jaegukhyun <jaeguk.hyun@intel.com>

* Fix NNCF training on CPU (#2373)

* Align label order between Geti and OTX (#2369)

* align label order

* align with pre-commit

* update CHANGELOG.md

* deal with edge case

* update type hint

* Remove CenterCrop from Classification test pipeline and editing missing docs link (#2375)

* Fix missing link for docs and removing centercrop for classification data pipeline

* Revert the test threshold

* Fix H-label classification (#2377)

* Fix h-labelissue

* Update unit tests

* Make black happy

* Fix unittests

* Make black happy

* Fix update heades information func

* Update the logic: consider the loss per batch

* Update for release 1.4 (#2380)

* updated for 1.4.0rc3

* update changelog & release note

* bump datumaro version up

---------

Co-authored-by: Songki Choi <songki.choi@intel.com>

* Switch to PTQ for sseg (#2374)

* Switch to PTQ for sseg

* Update log messages

* Fix invalid import structures in otx.api (#2383)

Update tiler.py

* Update for 1.4.0rc4 (#2385)

update for release 1.4.0rc4

* [release 1.4.0] XAI: Return saliency maps for Mask RCNN IR async infer (#2395)

* Return saliency maps for openvino async infer

* add workaround to fix yapf importing error

---------

Co-authored-by: eunwoosh <eunwoo.shin@intel.com>

* Update for release 1.4.0 (#2399)

update version string

Co-authored-by: Sungman Cho <sungman.cho@intel.com>

* Fix broken links in documentation (#2405)

* fix docs links to datumaro's docs
* fix docs links to otx's docs
* bump version to 1.4.1

* Update exportable code README (#2411)

* Updated for release 1.4.1 (#2412)

updated for release 1.4.1

* Add workaround for the incorrect meta info M-RCNN (used for XAI) (#2437)

Add workaround for the incorrect mata info

* Add model category attributes to model template (#2439)

Add model category attributes to model template

* Add model category & status fields in model template

* Add is_default_for_task attr to model template

* Update model templates with category attrs

* Add integration tests for model templates consistency

* Fix license & doc string

* Fix typo

* Refactor test cases

* Refactor common tests by generator

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update for 1.4.2rc1 (#2441)

update for release 1.4.2rc1

* Fix label list order for h-label classification (#2440)

* Fix label list for h-label cls
* Fix unit tests

* Modified fq numbers for lite HRNET (#2445)

modified fq numbers for lite HRNET

* Update PTQ ignored scope for hrnet 18  mod2 (#2449)

Update ptq ignored scope for hrnet 18  mod2

* Fix OpenVINO inference for legacy models (#2450)

* bug fix for legacy openvino models

* Add tests

* Specific exceptions

---------

* Update for 1.4.2rc2 (#2455)

update for release 1.4.2rc2

* Prevent zero-sized saliency map in tiling if tile size is too big (#2452)

* Prevent zero-sized saliency map in tiling if tile size is too big

* Prevent zero-sized saliency in tiling (PyTorch)

* Add unit tests for Tiler merge features methods

---------

Co-authored-by: Galina <galina.zalesskaya@intel.com>

* Update pot fq reference number (#2456)

update pot fq reference number to 15

* Bump datumaro version to 1.5.0rc0 (#2470)

bump datumaro version to 1.5.0rc0

* Set tox version constraint (#2472)

set tox version constraint - https://github.com/tox-dev/tox/issues/3110

* Bug fix for albumentations (#2467)

* bug fix for legacy openvino models

* Address albumentation issue

---------

Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>

* update for release 1.4.2rc3

* Add a dummy hierarchical config required by MAPI (#2483)

* bump version to 1.4.2rc4

* Bump datumaro version (#2502)

* bump datumaro version

* remove deprecated/reomved attribute usage of the datumaro

* Upgrade nncf version for 1.4 release (#2459)

* Upgrade nncf version

* Fix nncf interface warning

* Set the exact nncf version

* Update FQ refs after NNCF upgrade

* Use NNCF from pypi

* Update version for release 1.4.2rc5 (#2507)

update version for release 1.4.2rc5

* Update for 1.4.2 (#2514)

update for release 1.4.2

* create branch release/1.5.0

* Delete mem cache handler after training is done (#2535)

release mem cache handler after training is done

* Fix bug that auto batch size doesn't consider distributed training (#2533)

* consider distributed training while searching batch size

* update unit test

* reveret gpu memory upper bound

* fix typo

* change allocated to reserved

* add unit test for distributed training

* align with pre-commit

* Apply fix progress hook to release 1.5.0 (#2539)

* Fix hook's ordering issue. AdaptiveRepeatHook changes the runner.max_iters before the ProgressHook

* Change the expression

* Fix typo

* Fix multi-label, h-label issue

* Fix auto_bs issue

* Apply suggestions from code review

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Reflecting reviews

* Refactor the name of get_data_cfg

* Revert adaptive hook sampler init

* Refactor the function name: get_data_cfg -> get_subset_data_cfg

* Fix unit test errors

* Remove adding AdaptiveRepeatDataHook for autobs

* Remove unused import

* Fix detection and segmentation case in Geti scenario

---------

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Re introduce adaptive scheduling for training (#2541)

* Re-introduce adaptive patience for training

* Revert unit tests

* Update for release 1.4.3rc1 (#2542)

* Mirror Anomaly ModelAPI changes (#2531)

* Migrate anomaly exportable code to modelAPI (#2432)

* Fix license in PR template

* Migrate to modelAPI

* Remove color conversion in streamer

* Remove reverse_input_channels

* Add float

* Remove test as metadata is no longer used

* Remove metadata from load method

* remove anomalib openvino inferencer

* fix signature

* Support logacy OpenVINO model

* Transform image

* add configs

* Re-introduce adaptive training (#2543)

* Re-introduce adaptive patience for training

* Revert unit tests

* Fix auto input size mismatch in eval & export (#2530)

* Fix auto input size mismatch in eval & export

* Re-enable E2E tests for Issue#2518

* Add input size check in export testing

* Format float numbers in log

* Fix NNCF export shape mismatch

* Fix saliency map issue

* Disable auto input size if tiling enabled

---------

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update ref. fq number for anomaly e2e2 (#2547)

* Skip e2e det tests by issue2548 (#2550)

* Add skip to chained TC for issue #2548 (#2552)

* Update for release 1.4.3 (#2551)

* Update MAPI for 1.5 release (#2555)

Upgrade MAPI to v 0.1.6 (#2529)

* Upgrade MAPI

* Update exp code demo commit

* Fix MAPI imports

* Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Disable QAT for SegNexts (#2565)

* Disable NNCF QAT for SegNext

* Del obsolete pot configs

* Move NNCF skip marks to test commands to avoid duplication

* Add Anomaly modelAPI changes to releases/1.4.0 (#2563)

* bug fix for legacy openvino models

* Apply otx anomaly 1.5 changes

* Fix tests

* Fix compression config

* fix modelAPI imports

* update integration tests

* Edit config types

* Update keys in deployed model

---------

Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>
Co-authored-by: Kim, Sungchul <sungchul.kim@intel.com>

* Fix the CustomNonLinearClsHead when the batch_size is set to 1 (#2571)

Fix bn1d issue

Co-authored-by: sungmanc <sungmanc@intel.com>

* Update ModelAPI configuration (#2564 from 1.4) (#2568)

Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Update for 1.4.4rc1 (#2572)

* Hotfix DatasetEntity.get_combined_subset function loop (#2577)

Fix get_combined_subset function

* Revert default input size to `Default` due to YOLOX perf regression (#2580)

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix for the degradation issue of the classification task (#2585)

* Revert to sync with 1.4.0

* Remove repeat data

* Convert to the RGB value

* Fix color conversion logic

* Fix precommit

* Bump datumaro version to 1.5.1rc3 (#2587)

* Add label ids to anomaly OpenVINO model xml (#2590)

* Add label ids to model xml

---------

* Fix DeiT-Tiny model regression during class incremental training (#2594)

* enable IBloss for DeiT-Tiny

* update changelog

* add docstring

* Add label ids to model xml in release 1.5 (#2591)

Add label ids to model xml

* Fix DeiT-Tiny regression test for release/1.4.0 (#2595)

* Fix DeiT regression test

* update changelog

* temp

* Fix mmcls bug not wrapping model in DataParallel on CPUs (#2601)

Wrap multi-label and h-label classification models by MMDataParallel in case of CPU training.
---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix h-label loss normalization issue w/ exclusive label group of singe label (#2604)

* Fix h-label loss normalization issue w/ exclusive label group with signle label

* Fix non-linear version

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Boost up Image numpy accessing speed through PIL (#2586)

* boost up numpy accessing speed through PIL

* update CHANGELOG

* resolve precommit error

* resolve precommit error

* add fallback logic with PIL open

* use convert instead of draft

* Add missing import pathlib for cls e2e testing (#2610)

* Fix division by zero in class incremental learning for classification (#2606)

* Add empty label to reproduce zero-division error

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix minor typo

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix empty label 4 -> 3

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Prevent division by zero

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update license

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update CHANGELOG.md

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix inefficient sampling

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Revert indexing

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix minor typo

Signed-off-by: Songki Choi <songki.choi@intel.com>

---------

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Unify logger usage (#2612)

* unify logger

* align with pre-commit

* unify anomaly logger to otx

* change logger file path

* align with pre-commit

* change logger file path in missing file

* configure logger after ConfigManager is initialized

* configure logger when ConfigManager instance is initialized

* update unit test code

* move config_logger to each cli file

* align with pre-commit

* change part still using mmcv logger

* Fix XAI algorithm for Detection (#2609)

* Impove saliency maps algorithm for Detection

* Remove extra changes

* Update unit tests

* Changes for 1 class

* Fix pre-commit

* Update CHANGELOG

* Tighten dependency constraint only adapting latest patches (#2607)

* tighten dependency constratint only adapting latest patches

* adjust scikit-image version w.r.t python version

* adjust tensorboard version w.r.t python version

* remove version specifier for scikit-image

* Add metadata to optimized model (#2618)

* bug fix for legacy openvino models

* Add metadata to optimized model

* Revert formatting changes

---------

Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>

* modify omegaconf version constraint

* [release 1.5.0] Fix XAI algorithm for Detection (#2617)

Update detection XAI algorithm

* Update dependency constraint (#2622)

* Update tpp (#2621)

* Fix h-label bug of missing parent labels in output (#2626)

* Fix h-label bug of missing parent labels in output

* Fix h-label test data label schema

* Update CHANGELOG.md

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update publish workflow (#2625)

update publish workflow to push whl to internal pypi

* bump datumaro version to ~=1.5.0

* fixed mistake while mergeing back 1.4.4

* modifiy readme

* remove openvino model wrapper class

* remove openvino model wrapper tests

* [release 1.5.0] DeiT: enable tests + add ViTFeatureVectorHook (#2630)

Add ViT feature vector hook

* Fix docs broken link to datatumaro_h-label

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Fix wrong label settings for non-anomaly task ModelAPIs

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

Fix e2e XAI ref value

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* update release note and readme

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

* fix datumaro version to 1.6.0rc0

* Mergeback 1.5.0 to develop (#2642)

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

---------

Co-authored-by: Galina Zalesskaya <galina.zalesskaya@intel.com>
Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>

* Revert "Mergeback 1.5.0 to develop" (#2645)

Revert "Mergeback 1.5.0 to develop (#2642)"

This reverts commit 2f67686103df873d020681f6d504f9595ce4a963.

* Add a tool to help conduct experiments (#2651)

* implement run and experiment

* implement experiment result aggregator

* refactor experiment.py

* refactor run.py

* get export model speed

* add var collumn

* refactor experiment.py

* refine a way to update argument in cmd

* refine resource tracker

* support anomaly on research framework

* refine code aggregating exp result

* bugfix

* make other task available

* eval task save avg_time_per_images as result

* Add new argument to track CPU&GPU utilization and memory usage (#2500)

* add argument to track resource usage

* fix bug

* fix a bug in a multi gpu case

* use total cpu usage

* add unit test

* add mark to unit test

* cover edge case

* add pynvml in requirement

* align with pre-commit

* add license comment

* update changelog

* refine argument help

* align with pre-commit

* add version to requirement and raise an error if not supported values are given

* apply new resource tracker format

* refactor run.py

* support optimize in research framework

* cover edge case

* Handle a case where fail cases exist

* make argparse raise error rather than exit if problem exist

* revert tensorboard aggregator

* bugfix

* save failed cases as yaml file

* deal with integer in variables

* add epoch to metric

* use latest log.json file

* align with otx logging method

* move experiment.py from cli to tools

* refactor experiment.py

* merge otx run feature into experiment.py

* move set_arguments_to_cmd definition into experiment.py

* refactor experiment.py

* bugfix

* minor bugfix

* use otx.cli instead of each otx entry

* add feature to parse single workspace

* add comments

* fix bugs

* align with pre-commit

* revert parser argument

* align with pre-commit

* Revert inference batch size to 1 for instance segmentation (#2648)

Signed-off-by: Songki Choi <songki.choi@intel.com>

* Remove unnecessary log while building a model (#2658)

* revert logger in otx/algorithms/detection/adapters/mmdet/utils/builder.py

* revert logger in otx/algorithms/classification/adapters/mmcls/utils/builder.py

* make change more readable

* Fix a minor bug of experiment.py (#2662)

fix bug

* Not check avg_time_per_image during test (#2665)

* ignore avg_time_per_image during test

* do not call stdev when length of array is less than 2

* ignore avg_time_per_image during regerssion test

* Update device selection logic in classificaiton

* Add xpu accelerator

* Tmp patch for anomaly trainer

* Use XPU callback for anomaly training

* Update xpu accelerator

* Fix for anomaly xpu callback

* Fix validation batch logic

* Cleanup, add docstrings

* Refine xpu callback

---------

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Songki Choi <songki.choi@intel.com>
Co-authored-by: Yunchu Lee <yunchu.lee@intel.com>
Co-authored-by: Kim, Sungchul <sungchul.kim@intel.com>
Co-authored-by: Vinnam Kim <vinnam.kim@intel.com>
Co-authored-by: Evgeny Tsykunov <evgeny.tsykunov@intel.com>
Co-authored-by: Songki Choi <songki.choi@intel.com>
Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>
Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>
Co-authored-by: Sungman Cho <sungman.cho@intel.com>
Co-authored-by: Eugene Liu <eugene.liu@intel.com>
Co-authored-by: Wonju Lee <wonju.lee@intel.com>
Co-authored-by: Dick Ameln <dick.ameln@intel.com>
Co-authored-by: sungchul.kim <sungchul@ikvensx010>
Co-authored-by: GalyaZalesskaya <galina.zalesskaya@intel.com>
Co-authored-by: Harim Kang <harim.kang@intel.com>
Co-authored-by: Ashwin Vaidya <ashwin.vaidya@intel.com>
Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>
Co-authored-by: sungmanc <sungmanc@intel.com>

* Disable mixed precision training on XPU (#2683)

* disable mixed precision training on XPU

* fix optimize error

* refactor XPUDataParallel

* align with pre-commit

* align with pre-commit

* fix bug

* Update anomaly XPU integration (#2697)

* Update anomaly XPU integration

* Update strategy and accelerator

* Cleanup in strategy

* Fix mypy

* remove XPU callback

* Add XPU mixed precision plugin for lightning (#2714)

* Update anomaly XPU integration

* Update strategy and accelerator

* Cleanup in strategy

* Fix mypy

* remove XPU callback

* Add XPU mixed precision lightning training

* Fix linters

* Handle default plugins value

* Update code to support other features than 'train' on XPU (#2704)

* move where patching code for xpu in detection

* implement xpu resource manager in HPO

* consider xpu during adaptive_num_workers

* consider xpu while deciding asynchronous_sha

* align with pre-commit

* change variable name to proper one

* change patching code into configure_device

* update unit test code

* align with pre-commit

* Merge develop to develop-idev (#2727)

* Update base.txt

updated dependency version of datumaro

* Update __init__.py

update version string

* Update requirements.txt

* Temporarily skip visual prompting openvino integration test (#2323)

* Fix import dm.DatasetSubset (#2324)

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Fix semantic segmentation soft prediction dtype (#2322)

* Fix semantic segmentation soft prediction dtype

* relax ref sal vals check

---------

Co-authored-by: Songki Choi <songki.choi@intel.com>

* Contrain yapf verison lesser than 0.40.0 (#2328)

contrain_yapf_version

* Fix detection e2e tests (#2327)

Fix for detection

* Mergeback: Label addtion/deletion 1.2.4 --> 1.4.0 (#2326)

* Make black happy

* Fix conflicts

* Merge-back: add test datasets and edit the test code

* Make black happy

* Fix mis-merge

* Make balck happy

* Fix typo

* Fix typoi

---------

Co-authored-by: Songki Choi <songki.choi@intel.com>

* Bump datumaro up to 1.4.0rc2 (#2332)

bump datumaro up to 1.4.0rc2

* Tiling Doc for releases 1.4.0 (#2333)

* Add tiling documentation

* Bump otx version to 1.4.0rc2 (#2341)

* OTX deploy for visual prompting task  (#2311)

* Enable `otx deploy`

* (WIP) integration test

* Docstring

* Update args for create_model

* Manually set image embedding layout

* Enable to use model api for preprocessing
- `fit_to_window` doesn't work expectedly, so newly implemented `VisualPromptingOpenvinoAdapter` to use new resize function

* Remove skipped test

* Updated

* Update unit tests on model wrappers

* Update

* Update configuration

* Fix not to patch pretrained path

* pylint & update model api version in docstring

---------

Co-authored-by: Wonju Lee <wonju.lee@intel.com>

* Bump albumentations version in anomaly requirements (#2350)

increment albumentations version

* Update action detection (#2346)

* Remove skip mark for PTQ test of action detection

* Update action detection documentation

* Fix e2e (#2348)

* Change classification dataset from dummy to toy

* Revert test changes

* Change label name for multilabel dataset

* Revert e2e test changes

* Change ov test cases' threshold

* Add parent's label

* Update ModelAPI in 1.4 release (#2347)

* Upgrade model API

* Update otx in exportable code

* Fix unit tests

* Fix black

* Fix detection inference

* Fix det tiling

* Fix mypy

* Fix demo

* Fix visualizer in demo

* Fix black

* Add OTX optimize for visual prompting task (#2318)

* Initial commit

* Update block

* (WIP) otx optimize

* Fix

* WIP

* Update configs & exported outputs

* Remove unused modules for torch

* Add unit tests

* pre-commit

* Update CHANGELOG

* Update detection docs (#2335)

* Update detection docs

* Revert template id changes

* Fix wrong template id

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

---------

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Add visual prompting documentation (#2354)

* (WIP) write docs

* Add visual prompting documentation

* Update CHANGELOG

---------

Co-authored-by: sungchul.kim <sungchul@ikvensx010>

* Remove custom modelapi patch in visual prompting (#2359)

* Remove custom modelapi patch

* Update test

* Fix graph metric order and label issues (#2356)

* Fix graph metric going backward issue
* Add license notice
* Fix pre-commit issue
* Add rename items & logic for metric
---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update multi-label document and conversion script (#2358)

Update docs, label convert script

* Update third party programs (#2365)

* Make anomaly task compatible with older albumentations versions (#2363)

* fix transforms export in metadata

* wrap transform dict

* add todo for updating to_dict call

* Fixing detection saliency map for one class case (#2368)

* fix softmax

* fix validity tests

* Add e2e test for visual prompting (#2360)

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* Delete unused configuration.yaml

* Edit test_name

* Add to limit activation range

* Update from `vp` to `visprompt`

* Fix about no returning the first label

* pre-commit

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* pre-commit

* Add actions

* Update tests/e2e/cli/visual_prompting/test_visual_prompting.py

Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>

* Skip PTQ e2e test

* Change task name

* Remove skipped tc

---------

Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com>

* Fix e2e (#2366)

* Change e2e reference name

* Update openvino eval threshold for multiclass classification

* Change comment message

* Fix tiling e2e tests

---------

Co-authored-by: GalyaZalesskaya <galina.zalesskaya@intel.com>

* Add Dino head unit tests (#2344)

Recover DINO head unit tests

* Update for release 1.4.0rc2 (#2370)

* update for release 1.4.0rc2

* Add skip mark for unstable unit tests

---------

Co-authored-by: jaegukhyun <jaeguk.hyun@intel.com>

* Fix NNCF training on CPU (#2373)

* Align label order between Geti and OTX (#2369)

* align label order

* align with pre-commit

* update CHANGELOG.md

* deal with edge case

* update type hint

* Remove CenterCrop from Classification test pipeline and editing missing docs link (#2375)

* Fix missing link for docs and removing centercrop for classification data pipeline

* Revert the test threshold

* Fix H-label classification (#2377)

* Fix h-labelissue

* Update unit tests

* Make black happy

* Fix unittests

* Make black happy

* Fix update heades information func

* Update the logic: consider the loss per batch

* Update for release 1.4 (#2380)

* updated for 1.4.0rc3

* update changelog & release note

* bump datumaro version up

---------

Co-authored-by: Songki Choi <songki.choi@intel.com>

* Switch to PTQ for sseg (#2374)

* Switch to PTQ for sseg

* Update log messages

* Fix invalid import structures in otx.api (#2383)

Update tiler.py

* Update for 1.4.0rc4 (#2385)

update for release 1.4.0rc4

* [release 1.4.0] XAI: Return saliency maps for Mask RCNN IR async infer (#2395)

* Return saliency maps for openvino async infer

* add workaround to fix yapf importing error

---------

Co-authored-by: eunwoosh <eunwoo.shin@intel.com>

* Update for release 1.4.0 (#2399)

update version string

Co-authored-by: Sungman Cho <sungman.cho@intel.com>

* Fix broken links in documentation (#2405)

* fix docs links to datumaro's docs
* fix docs links to otx's docs
* bump version to 1.4.1

* Update exportable code README (#2411)

* Updated for release 1.4.1 (#2412)

updated for release 1.4.1

* Add workaround for the incorrect meta info M-RCNN (used for XAI) (#2437)

Add workaround for the incorrect mata info

* Add model category attributes to model template (#2439)

Add model category attributes to model template

* Add model category & status fields in model template

* Add is_default_for_task attr to model template

* Update model templates with category attrs

* Add integration tests for model templates consistency

* Fix license & doc string

* Fix typo

* Refactor test cases

* Refactor common tests by generator

---------
Signed-off-by: Songki Choi <songki.choi@intel.com>

* Update for 1.4.2rc1 (#2441)

update for release 1.4.2rc1

* Fix label list order for h-label classification (#2440)

* Fix label list for h-label cls
* Fix unit tests

* Modified fq numbers for lite HRNET (#2445)

modified fq numbers for lite HRNET

* Update PTQ ignored scope for hrnet 18  mod2 (#2449)

Update ptq ignored scope for hrnet 18  mod2

* Fix OpenVINO inference for legacy models (#2450)

* bug fix for legacy openvino models

* Add tests

* Specific exceptions

---------

* Update for 1.4.2rc2 (#2455)

update for release 1.4.2rc2

* Prevent zero-sized saliency map in tiling if tile size is too big (#2452)

* Prevent zero-sized saliency map in tiling if tile size is too big

* Prevent zero-sized saliency in tiling (PyTorch)

* Add unit tests for Tiler merge features methods

---------

Co-authored-by: Galina <galina.zalesskaya@intel.com>

* Update pot fq reference number (#2456)

update pot fq reference number to 15

* Bump datumaro version to 1.5.0rc0 (#2470)

bump datumaro version to 1.5.0rc0

* Set tox version constraint (#2472)

set tox version constraint - https://github.com/tox-dev/tox/issues/3110

* Bug fix for albumentations (#2467)

* bug fix for legacy openvino models

* Address albumentation issue

---------

Co-authored-by: Ashwin Vaidya <ashwinitinvaidya@gmail.com>

* update for release 1.4.2rc3

* Add a dummy hierarchical config required by MAPI (#2483)

* bump version to 1.4.2rc4

* Bump datumaro version (#2502)

* bump datumaro version

* remove deprecated/reomved attribute usage of the datumaro

* Upgrade nncf version for 1.4 release (#2459)

* Upgrade nncf version

* Fix nncf interface warning

* Set the exact nncf version

* Update FQ refs after NNCF upgrade

* Use NNCF from pypi

* Update version for release 1.4.2rc5 (#2507)

update version for release 1.4.2rc5

* Update for 1.4.2 (#2514)

update for release 1.4.2

* create branch release/1.5.0

* Delete mem cache handler after training is done (#2535)

release mem cache handler after training is done

* Fix bug that auto batch size doesn't consider distributed training (#2533)

* consider distributed training while searching batch size

* update unit test

* reveret gpu memory upper bound

* fix typo

* change allocated to reserved

* add unit test for distributed training

* align with pre-commit

* Apply fix progress hook to release 1.5.0 (#2539)

* Fix hook's ordering issue. AdaptiveRepeatHook changes the runner.max_iters before the ProgressHook

* Change the expression

* Fix typo

* Fix multi-label, h-label issue

* Fix auto_bs issue

* Apply suggestions from code review

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Reflecting reviews

* Refactor the name of get_data_cfg

* Revert adaptive hook sampler init

* Refactor the function name: get_data_cfg -> get_subset_data_cfg

* Fix unit test errors

* Remove adding AdaptiveRepeatDataHook for autobs

* Remove unused import

* Fix detection and segmentation case in Geti scenario

---------

Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>

* Re introduce adaptive scheduling for training (#2541)

* Re-introduce adaptive patience for training

* Revert unit tests

* Update for release 1.4.3rc1 (#2542)

* Mirror Anomaly ModelAPI changes (#2531)

* Migrate anomaly exportable code to modelAPI (#2432)

* Fix license in PR template

* Migrate to modelAPI

* Remove color conversion in streamer

* Remove reverse_input_channels

* Add float

* Remove test as metadata is no longer used

* Remove metadata from load method

* remove anomalib openvino inferencer

* fix signature

* Support logacy OpenVINO model

* Transform image

* add configs

* Re-introduce adaptive training (#2543)

* Re-introduce adaptive patience for training

* Revert unit t…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sungmanc sungmanc sungmanc approved these changes

@goodsong81 goodsong81 goodsong81 approved these changes

Assignees
No one assigned
Labels
ALGO Any changes in OTX Algo Tasks implementation FIX defect fix RELEASE TEST Any changes in tests
Projects
None yet
Milestone
1.4.3
Development

Successfully merging this pull request may close these issues.
3 participants
@jaegukhyun @goodsong81 @sungmanc