diff --git a/CHANGELOG.md b/CHANGELOG.md index c7f815292d9..f471a0f4039 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,6 +22,20 @@ All notable changes to this project will be documented in this file. - OpenVINO(==2022.3) IR inference is not working well on 2-stage models (e.g. Mask-RCNN) exported from torch==1.13.1 (working well up to torch==1.12.1) () +## \[v1.2.1\] + +### Enhancements + +- Upgrade mmdeploy==0.14.0 from official PyPI () +- Integrate new ignored loss in semantic segmentation () +- Optimize YOLOX data pipeline () +- Tiling Spatial Concatenation for OpenVINO IR () + +### Bug fixes + +- Bug fix: value of validation variable is changed after auto decrease batch size () +- Fix tiling 0 stride issue in parameter adapter () + ## \[v1.2.0\] ### New features diff --git a/docs/source/guide/explanation/additional_features/fast_data_loading.rst b/docs/source/guide/explanation/additional_features/fast_data_loading.rst new file mode 100644 index 00000000000..46767e2c8bd --- /dev/null +++ b/docs/source/guide/explanation/additional_features/fast_data_loading.rst @@ -0,0 +1,73 @@ +Fast Data Loading +================= + +OpenVINO™ Training Extensions provides several ways to boost model training speed, +one of which is fast data loading. + + +=================== +Faster Augmentation +=================== + + +****** +AugMix +****** +AugMix [1]_ is a simple yet powerful augmentation technique +to improve robustness and uncertainty estimates of image classification task. +OpenVINO™ Training Extensions implemented it in `Cython `_ for faster augmentation. +Users do not need to configure anything as cythonized AugMix is used by default. + + + +======= +Caching +======= + + +***************** +In-Memory Caching +***************** +OpenVINO™ Training Extensions provides in-memory caching for decoded images in main memory. +If the batch size is large, such as for classification tasks, or if dataset contains +high-resolution images, image decoding can account for a non-negligible overhead +in data pre-processing. +One can enable in-memory caching for maximizing GPU utilization and reducing model +training time in those cases. + + +.. code-block:: + + $ otx train --mem-cache-size=8GB .. + + + +*************** +Storage Caching +*************** + +OpenVINO™ Training Extensions uses `Datumaro `_ +under the hood for dataset managements. +Since Datumaro `supports `_ +`Apache Arrow `_, OpenVINO™ Training Extensions +can exploit fast data loading using memory-mapped arrow file at the expanse of storage consumtion. + + +.. code-block:: + + $ otx train .. params --algo_backend.storage_cache_scheme JPEG/75 + + +The cache would be saved in ``$HOME/.cache/otx`` by default. +One could change it by modifying ``OTX_CACHE`` environment variable. + + +.. code-block:: + + $ OTX_CACHE=/path/to/cache otx train .. params --algo_backend.storage_cache_scheme JPEG/75 + + +Please refere `Datumaro document `_ +for available schemes to choose but we recommend ``JPEG/75`` for fast data loaidng. + +.. [1] Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. "AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty" International Conference on Learning Representations. 2020. diff --git a/docs/source/guide/explanation/additional_features/index.rst b/docs/source/guide/explanation/additional_features/index.rst index 57add22bcb1..b9b24ddc43e 100644 --- a/docs/source/guide/explanation/additional_features/index.rst +++ b/docs/source/guide/explanation/additional_features/index.rst @@ -11,3 +11,4 @@ Additional Features auto_configuration xai noisy_label_detection + fast_data_loading diff --git a/docs/source/guide/explanation/additional_features/noisy_label_detection.rst b/docs/source/guide/explanation/additional_features/noisy_label_detection.rst index 410e1cab1d4..d55271c86ef 100644 --- a/docs/source/guide/explanation/additional_features/noisy_label_detection.rst +++ b/docs/source/guide/explanation/additional_features/noisy_label_detection.rst @@ -1,4 +1,4 @@ -Noisy label detection +Noisy Label Detection ===================== OpenVINO™ Training Extensions provide a feature for detecting noisy labels during model training. diff --git a/docs/source/guide/get_started/quick_start_guide/cli_commands.rst b/docs/source/guide/get_started/quick_start_guide/cli_commands.rst index be079b8b0cc..5a74f1655e9 100644 --- a/docs/source/guide/get_started/quick_start_guide/cli_commands.rst +++ b/docs/source/guide/get_started/quick_start_guide/cli_commands.rst @@ -273,6 +273,18 @@ For example, that is how you can change the learning rate and the batch size for --learning_parameters.batch_size 16 \ --learning_parameters.learning_rate 0.001 +You could also enable storage caching to boost data loading at the expanse of storage: + +.. code-block:: + + (otx) ...$ otx train SSD --train-data-roots \ + --val-data-roots \ + params \ + --algo_backend.storage_cache_scheme JPEG/75 + +.. note:: + Not all templates support stroage cache. We are working on extending supported templates. + As can be seen from the parameters list, the model can be trained using multiple GPUs. To do so, you simply need to specify a comma-separated list of GPU indices after the ``--gpus`` argument. It will start the distributed data-parallel training with the GPUs you have specified. diff --git a/docs/source/guide/tutorials/base/how_to_train/instance_segmentation.rst b/docs/source/guide/tutorials/base/how_to_train/instance_segmentation.rst index 84012c69999..9df3cf5f1a6 100644 --- a/docs/source/guide/tutorials/base/how_to_train/instance_segmentation.rst +++ b/docs/source/guide/tutorials/base/how_to_train/instance_segmentation.rst @@ -9,7 +9,7 @@ To learn more about Instance Segmentation task, refer to :doc:`../../../explanat .. note:: - To learn deeper how to manage training process of the model including additional parameters and its modification, refer to :doc:`./detection`. + To learn deeper how to manage training process of the model including additional parameters and its modification. To learn how to deploy the trained model, refer to: :doc:`../deploy`. @@ -20,7 +20,7 @@ The process has been tested on the following configuration. - Ubuntu 20.04 - NVIDIA GeForce RTX 3090 - Intel(R) Core(TM) i9-11900 -- CUDA Toolkit 11.4 +- CUDA Toolkit 11.7 ************************* Setup virtual environment @@ -43,40 +43,76 @@ environment: Dataset preparation *************************** -1. Let's use the simple toy dataset `Car, Tree, Bug dataset `_ -provided by OpenVINO™ Training Extensions. +.. note:: -This dataset contains images of simple car, tree, bug with the annotation for instance segmentation. + Currently, we support the following instance segmentation dataset formats: -- ``car`` - Car Shape Illustration -- ``tree`` - Tree Shape Illustration -- ``bug`` - Bug Shape Illustration + - `COCO `_ -This allows us to look at the structure of the dataset used in instance segmentation, and can be a good starting point for how to start an instance segmentation task with OpenVINO™ Training Extensions. +1. Clone a repository with +`WGISD dataset `_. -.. image:: ../../../../../utils/images/car_tree_bug_gt_sample.png - :width: 400 +.. code-block:: + + mkdir data ; cd data + git clone https://github.com/thsant/wgisd.git + cd wgisd + git checkout 6910edc5ae3aae8c20062941b1641821f0c30127 + + +This dataset contains images of grapevines with the annotation for different varieties of grapes. + +- ``CDY`` - Chardonnay +- ``CFR`` - Cabernet Franc +- ``CSV`` - Cabernet Sauvignon +- ``SVB`` - Sauvignon Blanc +- ``SYH`` - Syrah + +| + +.. image:: ../../../../../utils/images/wgisd_dataset_sample.jpg + :width: 600 + :alt: this image uploaded from this `source `_ +| 2. Check the file structure of downloaded dataset, we will need the following file structure: .. code-block:: - car_tree_bug + wgisd ├── annotations/ - ├── instances_train.json - └── instances_val.json + ├── instances_train.json + ├── instances_val.json + (Optional) + └── instances_test.json ├──images/ - └── - ... + (Optional) + ├── train + ├── val + └── test + (There may be more extra unrelated folders) -.. warning:: - There may be features that don't work properly with the current toy dataset. We recommend that you proceed with a proper training and validation dataset, - the tutorial and dataset here are for reference only. +We can do that by running these commands: + +.. code-block:: + + # format images folder + mv data images + + # format annotations folder + mv coco_annotations annotations + + # rename annotations to meet *_train.json pattern + mv annotations/train_bbox_instances.json annotations/instances_train.json + mv annotations/test_bbox_instances.json annotations/instances_val.json - We will update this tutorial with larger public datasets soon. + cd ../.. + +.. note:: + We can use this dataset in the detection tutorial. refer to :doc:`./detection`. ********* Training @@ -109,7 +145,7 @@ Let's prepare an OpenVINO™ Training Extensions instance segmentation workspace .. code-block:: - (otx) ...$ otx build --task instance_segmentation --model MaskRCNN-ResNet50 + (otx) ...$ otx build --task instance_segmentation [*] Workspace Path: otx-workspace-INSTANCE_SEGMENTATION [*] Load Model Template ID: Custom_Counting_Instance_Segmentation_MaskRCNN_ResNet50 @@ -124,6 +160,14 @@ Let's prepare an OpenVINO™ Training Extensions instance segmentation workspace (otx) ...$ cd ./otx-workspace-INSTANCE_SEGMENTATION +.. note:: + The default model for instance segmentation is MaskRCNN-ResNet50. + If you want to use a different model, use the commands below. + + .. code-block:: + + (otx) ...$ otx build --task instance_segmentation --model + It will create **otx-workspace-INSTANCE_SEGMENTATION** with all necessary configs for MaskRCNN-ResNet50, prepared ``data.yaml`` to simplify CLI commands launch and splitted dataset. .. note:: @@ -134,14 +178,12 @@ It will create **otx-workspace-INSTANCE_SEGMENTATION** with all necessary config .. code-block:: (otx) ...$ otx train Custom_Counting_Instance_Segmentation_MaskRCNN_ResNet50 \ - --train-data-roots data/car_tree_bug \ - --val-data-roots data/car_tree_bug \ + --train-data-roots /wgisd \ + --val-data-roots /wgisd \ params --learning_parameters.num_iters 8 The command above also creates an ``otx-workspace-INSTANCE_SEGMENTATION``, just like running build. This also updates ``data.yaml`` with data-specific commands. - For more information, see :doc:`quick start guide <../../../get_started/quick_start_guide/cli_commands>` or :ref:`detection example `. - .. warning:: Note, that we can't run CLI commands for instance segmentation via model name, since the same models are utilized for different algorithm and the behavior can be unpredictable. Please, use the template path or template ID instead. @@ -149,58 +191,86 @@ It will create **otx-workspace-INSTANCE_SEGMENTATION** with all necessary config To simplify the command line functions calling, we may create a ``data.yaml`` file with annotations info and pass it as a ``--data`` parameter. The content of the ``otx-workspace-INSTANCE_SEGMENTATION/data.yaml`` for dataset should have absolute paths and will be similar to that: -.. note:: - - When a workspace is created, ``data.yaml`` is always generated. - - You can modify the required arguments in ``data.yaml`` or use the command to provide the required arguments. +Check ``otx-workspace-INSTANCE_SEGMENTATION/data.yaml`` to ensure, which data subsets will be used for training and validation, and update it if necessary. .. code-block:: - {'data': - { - 'train': - {'data-roots': 'otx-workspace-INSTANCE_SEGMENTATION/splitted_dataset/car_tree_bug'}, - 'val': - {'data-roots': 'otx-workspace-INSTANCE_SEGMENTATION/splitted_dataset/car_tree_bug'}, - 'test': - {'data-roots': 'otx-workspace-INSTANCE_SEGMENTATION/splitted_dataset/car_tree_bug'} - } - } - -4. To start training we need to call ``otx train`` + data: + train: + ann-files: null + data-roots: /wgisd + val: + ann-files: null + data-roots: /wgisd + test: + ann-files: null + data-roots: null + unlabeled: + file-list: null + data-roots: null + +3. To start training we need to call ``otx train`` command in our workspace: .. code-block:: - (otx) .../otx-workspace-INSTANCE_SEGMENTATION$ otx train \ - params --learning_parameters.num_iters 10 + (otx) .../otx-workspace-INSTANCE_SEGMENTATION$ otx train -.. warning:: - Since this is a very small dataset, we adjusted ``num_iters`` to avoid overfitting in this tutorial. + ... + 2023-04-26 10:55:29,312 | INFO : Update LrUpdaterHook patience: 3 -> 3 + 2023-04-26 10:55:29,312 | INFO : Update CheckpointHook interval: 1 -> 2 + 2023-04-26 10:55:29,312 | INFO : Update EvalHook interval: 1 -> 2 + 2023-04-26 10:55:29,312 | INFO : Update EarlyStoppingHook patience: 10 -> 5 + 2023-04-26 10:55:46,681 | INFO : Epoch [1][28/28] lr: 5.133e-04, eta: 2:54:03, time: 1.055, data_time: 0.658, memory: 7521, current_iters: 27, loss_rpn_cls: 0.2227, loss_rpn_bbox: 0.1252, loss_cls: 1.0220, acc: 77.4606, loss_bbox: 0.7682, loss_mask: 1.1534, loss: 3.2915, grad_norm: 14.0078 - In other general datasets, OpenVINO™ Training Extensions ends training at the right time without adjusting ``num_iters``. + ... + 2023-04-26 11:32:36,162 | INFO : called evaluate() + 2023-04-26 11:32:36,511 | INFO : F-measure after evaluation: 0.5576271186440678 + 2023-04-26 11:32:36,511 | INFO : Evaluation completed + Performance(score: 0.5576271186440678, dashboard: (1 metric groups)) + otx train time elapsed: 0:20:23.541362 +The training time highly relies on the hardware characteristics, for example on 1 NVIDIA GeForce RTX 3090 the training took about 20 minutes with full dataset. -The training results are ``weights.pth`` and ``label_schema.json`` files that located in ``otx-workspace-INSTANCE_SEGMENTATION/models`` folder, while training logs and tf_logs for `Tensorboard` visualization can be found in the ``otx-workspace-INSTANCE_SEGMENTATION`` dir. +4. ``(Optional)`` Additionally, we can tune training parameters such as batch size, learning rate, patience epochs or warm-up iterations. +Learn more about template-specific parameters using ``otx train params --help``. -``weights.pth`` and ``label_schema.json``, which are needed as input for the further commands: ``export``, ``eval``, ``optimize``, etc. +It can be done by manually updating parameters in the ``template.yaml`` file in your workplace or via the command line. + +For example, to decrease the batch size to 4, fix the number of epochs to 100 and disable early stopping, extend the command line above with the following line. .. code-block:: + + otx train params --learning_parameters.batch_size 4 \ + --learning_parameters.num_iters 100 \ + --learning_parameters.enable_early_stopping false - ... - 2023-02-21 22:34:53,474 | INFO : Update LrUpdaterHook patience: 5 -> 2 - 2023-02-21 22:34:53,474 | INFO : Update CheckpointHook interval: 1 -> 5 - 2023-02-21 22:34:53,474 | INFO : Update EvalHook interval: 1 -> 5 - 2023-02-21 22:34:53,474 | INFO : Update EarlyStoppingHook patience: 10 -> 3 - 2023-02-21 22:34:54,320 | INFO : Epoch [1][2/2] lr: 3.400e-04, eta: 3:14:44, time: 1.180, data_time: 0.784, memory: 7322, current_iters: 1, loss_rpn_cls: 0.0720, loss_rpn_bbox: 0.0250, loss_cls: 2.6643, acc: 89.3066, loss_bbox: 0.3984, loss_mask: 3.5540, loss: 6.7136, grad_norm: 66.2921 +5. The training results are ``weights.pth`` and ``label_schema.json`` files located in ``outputs/**_train/models`` folder, +while training logs can be found in the ``outputs/**_train/logs`` dir. + +- ``weights.pth`` - a model snapshot +- ``label_schema.json`` - a label schema used in training, created from a dataset + +These are needed as inputs for the further commands: ``export``, ``eval``, ``optimize``, ``deploy`` and ``demo``. +.. note:: + We also can visualize the training using ``Tensorboard`` as these logs are located in ``outputs/**/logs/**/tf_logs``. + +.. code-block:: + + otx-workspace-INSTANCE_SEGMENTATION + ├── outputs/ + ├── 20230403_134256_train/ + ├── logs/ + ├── models/ + ├── weights.pth + └── label_schema.json + └── cli_report.log + ├── latest_trained_model + ├── logs/ + ├── models/ + └── cli_report.log ... - 2023-02-21 22:35:07,908 | INFO : Inference completed - 2023-02-21 22:35:07,908 | INFO : called evaluate() - 2023-02-21 22:35:07,909 | INFO : F-measure after evaluation: 0.33333333333333326 - 2023-02-21 22:35:07,909 | INFO : Evaluation completed - Performance(score: 0.33333333333333326, dashboard: (1 metric groups)) After that, we have the PyTorch instance segmentation model trained with OpenVINO™ Training Extensions, which we can use for evaluation, export, optimization and deployment. @@ -217,13 +287,11 @@ Please note, ``label_schema.json`` file contains meta information about the data ``otx eval`` will output a F-measure for instance segmentation. 2. The command below will run validation on our dataset -and save performance results in ``outputs/performance.json`` file: +and save performance results in ``outputs/**_eval/performance.json`` file: .. code-block:: - (otx) ...$ otx eval --test-data-roots otx-workspace-INSTANCE_SEGMENTATION/splitted_dataset/car_tree_bug \ - --load-weights models/weights.pth \ - --outputs outputs + (otx) ...$ otx eval --test-data-roots /wgisd We will get a similar to this validation output: @@ -231,26 +299,25 @@ We will get a similar to this validation output: ... - 2023-02-21 22:37:10,263 | INFO : Inference completed - 2023-02-21 22:37:10,263 | INFO : called evaluate() - 2023-02-21 22:37:10,265 | INFO : F-measure after evaluation: 0.33333333333333326 - 2023-02-21 22:37:10,265 | INFO : Evaluation completed - Performance(score: 0.33333333333333326, dashboard: (1 metric groups)) + 2023-04-26 12:46:27,856 | INFO : Inference completed + 2023-04-26 12:46:27,856 | INFO : called evaluate() + 2023-04-26 12:46:28,453 | INFO : F-measure after evaluation: 0.5576271186440678 + 2023-04-26 12:46:28,453 | INFO : Evaluation completed + Performance(score: 0.5576271186440678, dashboard: (1 metric groups)) .. note:: You can omit ``--test-data-roots`` if you are currently inside a workspace and have test-data stuff written in ``data.yaml``. - Also, if you're inside a workspace and ``weights.pth`` exists in ``models`` dir, you can omit ``--load-weights`` as well, assuming those weights are the default as ``models/weights.pth``. - - If you omit ``--output``, it will create a ``performance.json`` in the folder for those weights. + Also, if you're inside a workspace and ``weights.pth`` exists in ``outputs/latest_train_model/models`` dir, + you can omit ``--load-weights`` as well, assuming those weights are the default as ``latest_train_model/models/weights.pth``. -The output of ``./outputs/performance.json`` consists of a dict with target metric name and its value. +The output of ``./outputs/**_eval/performance.json`` consists of a dict with target metric name and its value. .. code-block:: - {"f-measure": 0.33333333333333326} + {"f-measure": 0.5576271186440678} ********* Export @@ -262,42 +329,24 @@ OpenVINO™ Intermediate Representation (IR) format. It allows running the model on the Intel hardware much more efficient, especially on the CPU. Also, the resulting IR model is required to run POT optimization. IR model consists of 2 files: ``openvino.xml`` for weights and ``openvino.bin`` for architecture. 2. We can run the below command line to export the trained model -and save the exported model to the ``openvino_model`` folder. - -.. code-block:: - - (otx) ...$ otx export --load-weights models/weights.pth \ - --output openvino_model - - ... - [ SUCCESS ] Generated IR version 11 model. - [ SUCCESS ] XML file: /tmp/OTX-task-51omlxb0/stage00_DetectionExporter-train/model.xml - [ SUCCESS ] BIN file: /tmp/OTX-task-51omlxb0/stage00_DetectionExporter-train/model.bin +and save the exported model to the ``outputs/**_export/openvino`` folder. - 2023-02-21 22:38:21,893 - mmdeploy - INFO - Successfully exported OpenVINO model: /tmp/OTX-task-51omlxb0/stage00_DetectionExporter-train/model_ready.xml - 2023-02-21 22:38:21,894 | INFO : run task done. - 2023-02-21 22:38:21,940 | INFO : Exporting completed - -3. We can check the accuracy of the IR model and the consistency between -the exported model and the PyTorch model. +.. note:: -You can use ``otx train`` directly without ``otx build``. It will be required to add ``--train-data-roots`` and ``--val-data-roots`` in the command line: + if you're inside a workspace and ``weights.pth`` exists in ``outputs/latest_train_model/models`` dir, + you can omit ``--load-weights`` as well, assuming those weights are the default as ``latest_train_model/models/weights.pth``. .. code-block:: - (otx) ...$ otx eval --test-data-roots otx-workspace-INSTANCE_SEGMENTATION/splitted_dataset/car_tree_bug \ - --load-weights openvino_model/openvino.xml \ - --output openvino_model + (otx) ...$ otx export ... + [ SUCCESS ] Generated IR version 11 model. + [ SUCCESS ] XML file: otx-workspace-INSTANCE_SEGMENTATION/outputs/20230426_124738_export/logs/model.xml + [ SUCCESS ] BIN file: otx-workspace-INSTANCE_SEGMENTATION/outputs/20230426_124738_export/logs/model.bin - 2023-02-21 22:39:13,423 | INFO : Loading OpenVINO OTXDetectionTask - 2023-02-21 22:39:17,014 | INFO : OpenVINO task initialization completed - 2023-02-21 22:39:17,015 | INFO : Start OpenVINO inference - 2023-02-21 22:39:18,309 | INFO : OpenVINO inference completed - 2023-02-21 22:39:18,309 | INFO : Start OpenVINO metric evaluation - 2023-02-21 22:39:18,310 | INFO : OpenVINO metric evaluation completed - Performance(score: 0.33333333333333326, dashboard: (1 metric groups)) + 2023-04-26 12:47:48,293 - mmdeploy - INFO - Successfully exported OpenVINO model: outputs/20230426_124738_export/logs/model_ready.xml + 2023-04-26 12:47:48,670 | INFO : Exporting completed ************* Optimization @@ -309,38 +358,27 @@ It uses NNCF or POT depending on the model format. Please, refer to :doc:`optimization explanation <../../../explanation/additional_features/models_optimization>` section to get the intuition of what we use under the hood for optimization purposes. 2. Command example for optimizing -a PyTorch model (`.pth`) with OpenVINO™ NNCF. - -.. code-block:: +a PyTorch model (`.pth`) with OpenVINO™ `NNCF `_. - (otx) ...$ otx optimize --load-weights models/weights.pth --output nncf_model +.. note:: - ... + if you're inside a workspace and ``weights.pth`` exists in ``outputs/latest_train_model/models`` dir, + you can omit ``--load-weights`` as well (nncf only), assuming those weights are the default as ``latest_train_model/models/weights.pth``. - 2023-02-21 22:45:35,996 | INFO : run task done. - 2023-02-21 22:45:36,012 | INFO : Inference completed - 2023-02-21 22:45:36,013 | INFO : called evaluate() - 2023-02-21 22:45:36,014 | INFO : F-measure after evaluation: 0.33333333333333326 - 2023-02-21 22:45:36,014 | INFO : Evaluation completed - Performance(score: 0.33333333333333326, dashboard: (1 metric groups)) +.. code-block:: -The optimization time relies on the hardware characteristics, for example on 1 GeForce 3090 and Intel(R) Core(TM) i9-11900 it took about 1 minutes. + (otx) ...$ otx optimize 3. Command example for optimizing OpenVINO™ model (.xml) with OpenVINO™ POT. .. code-block:: - (otx) ...$ otx optimize --load-weights openvino_model/openvino.xml \ - --output pot_model - - ... - - Performance(score: 0.33333333333333326, dashboard: (3 metric groups)) + (otx) ...$ otx optimize --load-weights openvino_model/openvino.xml Please note, that POT will take some time (generally less than NNCF optimization) without logging to optimize the model. 4. Now we have fully trained, optimized and exported an efficient model representation ready-to-use instance segmentation model. -The following tutorials provide further steps on how to :doc:`deploy <../deploy>` and use your model in the :doc:`demonstration mode <../demo>` and visualize results. \ No newline at end of file +The following tutorials provide further steps on how to :doc:`deploy <../deploy>` and use your model in the :doc:`demonstration mode <../demo>` and visualize results. diff --git a/otx/algorithms/common/adapters/mmcv/utils/automatic_bs.py b/otx/algorithms/common/adapters/mmcv/utils/automatic_bs.py index c45b490ff30..a4578d35094 100644 --- a/otx/algorithms/common/adapters/mmcv/utils/automatic_bs.py +++ b/otx/algorithms/common/adapters/mmcv/utils/automatic_bs.py @@ -64,10 +64,15 @@ def train_func_single_iter(batch_size): else: copied_cfg.runner["max_epochs"] = 1 - if not validate: # disable validation - for hook in copied_cfg.custom_hooks: - if hook["type"] == "AdaptiveTrainSchedulingHook": - hook["enable_eval_before_run"] = False + otx_prog_hook_idx = None + for i, hook in enumerate(copied_cfg.custom_hooks): + if not validate and hook["type"] == "AdaptiveTrainSchedulingHook": + hook["enable_eval_before_run"] = False + elif hook["type"] == "OTXProgressHook": + otx_prog_hook_idx = i + + if otx_prog_hook_idx is not None: + del copied_cfg.custom_hooks[otx_prog_hook_idx] new_datasets = [SubDataset(datasets[0], batch_size)] diff --git a/otx/algorithms/common/configs/training_base.py b/otx/algorithms/common/configs/training_base.py index 1b7032bb9b9..cfaed062776 100644 --- a/otx/algorithms/common/configs/training_base.py +++ b/otx/algorithms/common/configs/training_base.py @@ -359,7 +359,7 @@ class BaseTilingParameters(ParameterGroup): description="Overlap between each two neighboring tiles.", default_value=0.2, min_value=0.0, - max_value=1.0, + max_value=0.9, affects_outcome_of=ModelLifecycle.NONE, ) @@ -372,4 +372,20 @@ class BaseTilingParameters(ParameterGroup): affects_outcome_of=ModelLifecycle.NONE, ) + tile_ir_scale_factor = configurable_float( + header="OpenVINO IR Scale Factor", + description="The purpose of the scale parameter is to optimize the performance and " + "efficiency of tiling in OpenVINO IR during inference. By controlling the increase in tile size and " + "input size, the scale parameter allows for more efficient parallelization of the workload and " + "improve the overall performance and efficiency of the inference process on OpenVINO.", + warning="Setting the scale factor value too high may cause the application " + "to crash or result in out-of-memory errors. It is recommended to " + "adjust the scale factor value carefully based on the available " + "hardware resources and the needs of the application.", + default_value=2.0, + min_value=1.0, + max_value=4.0, + affects_outcome_of=ModelLifecycle.NONE, + ) + tiling_parameters = add_parameter_group(BaseTilingParameters) diff --git a/otx/algorithms/detection/adapters/mmdet/datasets/pipelines/load_pipelines.py b/otx/algorithms/detection/adapters/mmdet/datasets/pipelines/load_pipelines.py index c61ad0e0be3..c92a252fe4f 100644 --- a/otx/algorithms/detection/adapters/mmdet/datasets/pipelines/load_pipelines.py +++ b/otx/algorithms/detection/adapters/mmdet/datasets/pipelines/load_pipelines.py @@ -78,8 +78,8 @@ def _load_masks(results, ann_info): def __call__(self, results: Dict[str, Any]): """Callback function of LoadAnnotationFromOTXDataset.""" - dataset_item = results["dataset_item"] - label_list = results["ann_info"]["label_list"] + dataset_item = results.pop("dataset_item") + label_list = results.pop("ann_info")["label_list"] ann_info = get_annotation_mmdet_format(dataset_item, label_list, self.domain, self.min_size) if self.with_bbox: results = self._load_bboxes(results, ann_info) diff --git a/otx/algorithms/detection/adapters/mmdet/datasets/tiling.py b/otx/algorithms/detection/adapters/mmdet/datasets/tiling.py index 21925c506da..06b27fce2a3 100644 --- a/otx/algorithms/detection/adapters/mmdet/datasets/tiling.py +++ b/otx/algorithms/detection/adapters/mmdet/datasets/tiling.py @@ -356,7 +356,7 @@ def __getitem__(self, idx): result = copy.deepcopy(self.tiles[idx]) dataset_idx = result["dataset_idx"] x_1, y_1, x_2, y_2 = result["tile_box"] - ori_img = self.cached_results[dataset_idx]["dataset_item"].media.numpy + ori_img = self.cached_results[dataset_idx]["img"] cropped_tile = ori_img[y_1:y_2, x_1:x_2, :] if self.img2fp32: cropped_tile = cropped_tile.astype(np.float32) diff --git a/otx/algorithms/detection/adapters/mmdet/task.py b/otx/algorithms/detection/adapters/mmdet/task.py index cd23a7d43bc..289880694fe 100644 --- a/otx/algorithms/detection/adapters/mmdet/task.py +++ b/otx/algorithms/detection/adapters/mmdet/task.py @@ -41,7 +41,6 @@ from otx.algorithms.common.adapters.mmcv.utils import ( adapt_batch_size, build_data_parallel, - get_configs_by_pairs, patch_data_pipeline, patch_from_hyperparams, ) @@ -62,7 +61,12 @@ from otx.algorithms.detection.adapters.mmdet.hooks.det_class_probability_map_hook import ( DetClassProbabilityMapHook, ) -from otx.algorithms.detection.adapters.mmdet.utils import patch_tiling +from otx.algorithms.detection.adapters.mmdet.utils import ( + patch_input_preprocessing, + patch_input_shape, + patch_ir_scale_factor, + patch_tiling, +) from otx.algorithms.detection.adapters.mmdet.utils.builder import build_detector from otx.algorithms.detection.adapters.mmdet.utils.config_utils import ( should_cluster_anchors, @@ -620,67 +624,10 @@ def _init_deploy_cfg(self, cfg) -> Union[Config, None]: if os.path.exists(deploy_cfg_path): deploy_cfg = MPAConfig.fromfile(deploy_cfg_path) - def patch_input_preprocessing(deploy_cfg): - normalize_cfg = get_configs_by_pairs( - cfg.data.test.pipeline, - dict(type="Normalize"), - ) - assert len(normalize_cfg) == 1 - normalize_cfg = normalize_cfg[0] - - options = dict(flags=[], args={}) - # NOTE: OTX loads image in RGB format - # so that `to_rgb=True` means a format change to BGR instead. - # Conventionally, OpenVINO IR expects a image in BGR format - # but OpenVINO IR under OTX assumes a image in RGB format. - # - # `to_rgb=True` -> a model was trained with images in BGR format - # and a OpenVINO IR needs to reverse input format from RGB to BGR - # `to_rgb=False` -> a model was trained with images in RGB format - # and a OpenVINO IR does not need to do a reverse - if normalize_cfg.get("to_rgb", False): - options["flags"] += ["--reverse_input_channels"] - # value must be a list not a tuple - if normalize_cfg.get("mean", None) is not None: - options["args"]["--mean_values"] = list(normalize_cfg.get("mean")) - if normalize_cfg.get("std", None) is not None: - options["args"]["--scale_values"] = list(normalize_cfg.get("std")) - - # fill default - backend_config = deploy_cfg.backend_config - if backend_config.get("mo_options") is None: - backend_config.mo_options = ConfigDict() - mo_options = backend_config.mo_options - if mo_options.get("args") is None: - mo_options.args = ConfigDict() - if mo_options.get("flags") is None: - mo_options.flags = [] - - # already defiend options have higher priority - options["args"].update(mo_options.args) - mo_options.args = ConfigDict(options["args"]) - # make sure no duplicates - mo_options.flags.extend(options["flags"]) - mo_options.flags = list(set(mo_options.flags)) - - def patch_input_shape(deploy_cfg): - resize_cfg = get_configs_by_pairs( - cfg.data.test.pipeline, - dict(type="Resize"), - ) - assert len(resize_cfg) == 1 - resize_cfg = resize_cfg[0] - size = resize_cfg.size - if isinstance(size, int): - size = (size, size) - assert all(isinstance(i, int) and i > 0 for i in size) - # default is static shape to prevent an unexpected error - # when converting to OpenVINO IR - deploy_cfg.backend_config.model_inputs = [ConfigDict(opt_shapes=ConfigDict(input=[1, 3, *size]))] - - patch_input_preprocessing(deploy_cfg) + patch_input_preprocessing(cfg, deploy_cfg) if not deploy_cfg.backend_config.get("model_inputs", []): - patch_input_shape(deploy_cfg) + patch_input_shape(cfg, deploy_cfg) + patch_ir_scale_factor(deploy_cfg, self._hyperparams) return deploy_cfg diff --git a/otx/algorithms/detection/adapters/mmdet/utils/__init__.py b/otx/algorithms/detection/adapters/mmdet/utils/__init__.py index 18238cfbb1b..77b125b0ca5 100644 --- a/otx/algorithms/detection/adapters/mmdet/utils/__init__.py +++ b/otx/algorithms/detection/adapters/mmdet/utils/__init__.py @@ -9,6 +9,9 @@ patch_config, patch_datasets, patch_evaluation, + patch_input_preprocessing, + patch_input_shape, + patch_ir_scale_factor, patch_tiling, prepare_for_training, set_hyperparams, @@ -23,4 +26,7 @@ "set_hyperparams", "build_detector", "patch_tiling", + "patch_input_preprocessing", + "patch_input_shape", + "patch_ir_scale_factor", ] diff --git a/otx/algorithms/detection/adapters/mmdet/utils/config_utils.py b/otx/algorithms/detection/adapters/mmdet/utils/config_utils.py index 16e8ff8f518..fe3e77c3592 100644 --- a/otx/algorithms/detection/adapters/mmdet/utils/config_utils.py +++ b/otx/algorithms/detection/adapters/mmdet/utils/config_utils.py @@ -373,3 +373,98 @@ def patch_tiling(config, hparams, dataset=None): config.update(dict(evaluation=dict(iou_thr=[0.5]))) return config + + +def patch_input_preprocessing(cfg: ConfigDict, deploy_cfg: ConfigDict): + """Update backend configuration with input preprocessing options. + + - If `"to_rgb"` in Normalize config is truthy, it adds `"--reverse_input_channels"` as a flag. + + The function then sets default values for the backend configuration in `deploy_cfg`. + + Args: + cfg (mmcv.ConfigDict): Config object containing test pipeline and other configurations. + deploy_cfg (mmcv.ConfigDict): DeployConfig object containing backend configuration. + + Returns: + None: This function updates the input `deploy_cfg` object directly. + """ + normalize_cfgs = get_configs_by_pairs(cfg.data.test.pipeline, dict(type="Normalize")) + assert len(normalize_cfgs) == 1 + normalize_cfg: dict = normalize_cfgs[0] + + # Set options based on Normalize config + options = { + "flags": ["--reverse_input_channels"] if normalize_cfg.get("to_rgb", False) else [], + "args": { + "--mean_values": list(normalize_cfg.get("mean", [])), + "--scale_values": list(normalize_cfg.get("std", [])), + }, + } + + # Set default backend configuration + mo_options = deploy_cfg.backend_config.get("mo_options", ConfigDict()) + mo_options = ConfigDict() if mo_options is None else mo_options + mo_options.args = mo_options.get("args", ConfigDict()) + mo_options.flags = mo_options.get("flags", []) + + # Override backend configuration with options from Normalize config + mo_options.args.update(options["args"]) + mo_options.flags = list(set(mo_options.flags + options["flags"])) + + deploy_cfg.backend_config.mo_options = mo_options + + +def patch_input_shape(cfg: ConfigDict, deploy_cfg: ConfigDict): + """Update backend configuration with input shape information. + + This function retrieves the Resize config from `cfg.data.test.pipeline`, checks + that only one Resize then sets the input shape for the backend model in `deploy_cfg` + + ``` + { + "opt_shapes": { + "input": [1, 3, *size] + } + } + ``` + + Args: + cfg (Config): Config object containing test pipeline and other configurations. + deploy_cfg (DeployConfig): DeployConfig object containing backend configuration. + + Returns: + None: This function updates the input `deploy_cfg` object directly. + """ + resize_cfgs = get_configs_by_pairs( + cfg.data.test.pipeline, + dict(type="Resize"), + ) + assert len(resize_cfgs) == 1 + resize_cfg: ConfigDict = resize_cfgs[0] + size = resize_cfg.size + if isinstance(size, int): + size = (size, size) + assert all(isinstance(i, int) and i > 0 for i in size) + # default is static shape to prevent an unexpected error + # when converting to OpenVINO IR + deploy_cfg.backend_config.model_inputs = [ConfigDict(opt_shapes=ConfigDict(input=[1, 3, *size]))] + + +def patch_ir_scale_factor(deploy_cfg: ConfigDict, hyper_parameters: DetectionConfig): + """Patch IR scale factor inplace from hyper parameters to deploy config. + + Args: + deploy_cfg (ConfigDict): mmcv deploy config + hyper_parameters (DetectionConfig): OTX detection hyper parameters + """ + + if hyper_parameters.tiling_parameters.enable_tiling: + scale_ir_input = deploy_cfg.get("scale_ir_input", False) + if scale_ir_input: + tile_ir_scale_factor = hyper_parameters.tiling_parameters.tile_ir_scale_factor + logger.info(f"Apply OpenVINO IR scale factor: {tile_ir_scale_factor}") + ir_input_shape = deploy_cfg.backend_config.model_inputs[0].opt_shapes.input + ir_input_shape[2] = int(ir_input_shape[2] * tile_ir_scale_factor) # height + ir_input_shape[3] = int(ir_input_shape[3] * tile_ir_scale_factor) # width + deploy_cfg.ir_config.input_shape = (ir_input_shape[3], ir_input_shape[2]) # width, height diff --git a/otx/algorithms/detection/adapters/openvino/task.py b/otx/algorithms/detection/adapters/openvino/task.py index 5ca6ef8e7cc..659112dd1d2 100644 --- a/otx/algorithms/detection/adapters/openvino/task.py +++ b/otx/algorithms/detection/adapters/openvino/task.py @@ -261,6 +261,7 @@ class OpenVINOTileClassifierWrapper(BaseInferencerWithConverter): tile_size (int): tile size overlap (float): overlap ratio between tiles max_number (int): maximum number of objects per image + tile_ir_scale_factor (float, optional): scale factor for tile size tile_classifier_model_file (Union[str, bytes, None], optional): tile classifier xml. Defaults to None. tile_classifier_weight_file (Union[str, bytes, None], optional): til classifier weight bin. Defaults to None. device (str, optional): device to run inference on, such as CPU, GPU or MYRIAD. Defaults to "CPU". @@ -274,6 +275,7 @@ def __init__( tile_size: int = 400, overlap: float = 0.5, max_number: int = 100, + tile_ir_scale_factor: float = 1.0, tile_classifier_model_file: Union[str, bytes, None] = None, tile_classifier_weight_file: Union[str, bytes, None] = None, device: str = "CPU", @@ -293,7 +295,7 @@ def __init__( classifier = Model(model_adapter=adapter, preload=True) self.tiler = Tiler( - tile_size=tile_size, + tile_size=int(tile_size * tile_ir_scale_factor), overlap=overlap, max_number=max_number, detector=inferencer.model, @@ -372,6 +374,10 @@ def load_config(self) -> ADDict: if self.model is not None and self.model.get_data("config.json"): json_dict = json.loads(self.model.get_data("config.json")) flatten_config_values(json_dict) + # NOTE: for backward compatibility + json_dict["tiling_parameters"]["tile_ir_scale_factor"] = json_dict["tiling_parameters"].get( + "tile_ir_scale_factor", 1.0 + ) config = merge_a_into_b(json_dict, config) except Exception as e: # pylint: disable=broad-except logger.warning(f"Failed to load config.json: {e}") @@ -418,6 +424,7 @@ def load_inferencer( self.config.tiling_parameters.tile_size, self.config.tiling_parameters.tile_overlap, self.config.tiling_parameters.tile_max_number, + self.config.tiling_parameters.tile_ir_scale_factor, tile_classifier_model_file, tile_classifier_weight_file, ) diff --git a/otx/algorithms/detection/configs/detection/configuration.yaml b/otx/algorithms/detection/configs/detection/configuration.yaml index 619c7e59f22..85b4cb3b826 100644 --- a/otx/algorithms/detection/configs/detection/configuration.yaml +++ b/otx/algorithms/detection/configs/detection/configuration.yaml @@ -507,7 +507,7 @@ tiling_parameters: affects_outcome_of: TRAINING default_value: 0.2 min_value: 0.0 - max_value: 1.0 + max_value: 0.9 type: FLOAT editable: true ui_rules: diff --git a/otx/algorithms/detection/configs/detection/cspdarknet_yolox/data_pipeline.py b/otx/algorithms/detection/configs/detection/cspdarknet_yolox/data_pipeline.py index 811445ef098..83358159eb7 100644 --- a/otx/algorithms/detection/configs/detection/cspdarknet_yolox/data_pipeline.py +++ b/otx/algorithms/detection/configs/detection/cspdarknet_yolox/data_pipeline.py @@ -74,7 +74,7 @@ ann_file=__data_root + "annotations/instances_train2017.json", img_prefix=__data_root + "train2017/", pipeline=[ - dict(type="LoadImageFromFile", to_float32=True), + dict(type="LoadImageFromFile", to_float32=False), dict(type="LoadAnnotations", with_bbox=True), ], ), diff --git a/otx/algorithms/detection/configs/instance_segmentation/configuration.yaml b/otx/algorithms/detection/configs/instance_segmentation/configuration.yaml index 16eb6cb0571..028df9ce94b 100644 --- a/otx/algorithms/detection/configs/instance_segmentation/configuration.yaml +++ b/otx/algorithms/detection/configs/instance_segmentation/configuration.yaml @@ -523,7 +523,7 @@ tiling_parameters: affects_outcome_of: TRAINING default_value: 0.2 min_value: 0.0 - max_value: 1.0 + max_value: 0.9 type: FLOAT editable: true ui_rules: @@ -553,5 +553,23 @@ tiling_parameters: visible_in_ui: true warning: null + tile_ir_scale_factor: + header: OpenVINO IR Scale Factor + description: The purpose of the scale parameter is to optimize the performance and efficiency of tiling in OpenVINO IR during inference. By controlling the increase in tile size and input size, the scale parameter allows for more efficient parallelization of the workload and improve the overall performance and efficiency of the inference process on OpenVINO. + affects_outcome_of: TRAINING + default_value: 2.0 + min_value: 1.0 + max_value: 4.0 + type: FLOAT + editable: true + ui_rules: + action: DISABLE_EDITING + operator: AND + rules: [] + type: UI_RULES + value: 2.0 + visible_in_ui: true + warning: null + type: PARAMETER_GROUP visible_in_ui: true diff --git a/otx/algorithms/detection/configs/instance_segmentation/efficientnetb2b_maskrcnn/deployment.py b/otx/algorithms/detection/configs/instance_segmentation/efficientnetb2b_maskrcnn/deployment.py index fa981b687fc..f9701f38ac0 100644 --- a/otx/algorithms/detection/configs/instance_segmentation/efficientnetb2b_maskrcnn/deployment.py +++ b/otx/algorithms/detection/configs/instance_segmentation/efficientnetb2b_maskrcnn/deployment.py @@ -2,8 +2,11 @@ _base_ = ["../../base/deployments/base_instance_segmentation_dynamic.py"] +scale_ir_input = True + ir_config = dict( output_names=["boxes", "labels", "masks"], + input_shape=(1024, 1024), ) backend_config = dict( diff --git a/otx/algorithms/detection/configs/instance_segmentation/resnet50_maskrcnn/deployment.py b/otx/algorithms/detection/configs/instance_segmentation/resnet50_maskrcnn/deployment.py index 715e77ef995..8ef82f1ca34 100644 --- a/otx/algorithms/detection/configs/instance_segmentation/resnet50_maskrcnn/deployment.py +++ b/otx/algorithms/detection/configs/instance_segmentation/resnet50_maskrcnn/deployment.py @@ -2,8 +2,11 @@ _base_ = ["../../base/deployments/base_instance_segmentation_dynamic.py"] +scale_ir_input = True + ir_config = dict( output_names=["boxes", "labels", "masks"], + input_shape=(1344, 800), ) backend_config = dict( diff --git a/otx/algorithms/detection/configs/rotated_detection/configuration.yaml b/otx/algorithms/detection/configs/rotated_detection/configuration.yaml index 8a20f17e9d1..14213c8fc26 100644 --- a/otx/algorithms/detection/configs/rotated_detection/configuration.yaml +++ b/otx/algorithms/detection/configs/rotated_detection/configuration.yaml @@ -469,7 +469,7 @@ tiling_parameters: affects_outcome_of: TRAINING default_value: 0.2 min_value: 0.0 - max_value: 1.0 + max_value: 0.9 type: FLOAT editable: true ui_rules: diff --git a/otx/algorithms/detection/utils/data.py b/otx/algorithms/detection/utils/data.py index 2eccb16d706..82b49992f9e 100644 --- a/otx/algorithms/detection/utils/data.py +++ b/otx/algorithms/detection/utils/data.py @@ -481,6 +481,10 @@ def adaptive_tile_params( tile_size = int(math.sqrt(object_area / object_tile_ratio)) tile_overlap = max_area / (tile_size**2) + if tile_overlap >= tiling_parameters.get_metadata("tile_overlap")["max_value"]: + # Use the average object area if the tile overlap is too large to prevent 0 stride. + tile_overlap = object_area / (tile_size**2) + # validate parameters are in range tile_size = max( tiling_parameters.get_metadata("tile_size")["min_value"], diff --git a/otx/api/usecases/exportable_code/demo/demo_package/model_container.py b/otx/api/usecases/exportable_code/demo/demo_package/model_container.py index 75c485657e9..6caeb1b4909 100644 --- a/otx/api/usecases/exportable_code/demo/demo_package/model_container.py +++ b/otx/api/usecases/exportable_code/demo/demo_package/model_container.py @@ -70,14 +70,18 @@ def setup_tiler(self, model_dir, device) -> Optional[Tiler]: if not self.parameters.get("tiling_parameters") or not self.parameters["tiling_parameters"]["enable_tiling"]: return None + tile_size = self.parameters["tiling_parameters"]["tile_size"] + tile_overlap = self.parameters["tiling_parameters"]["tile_overlap"] + max_number = self.parameters["tiling_parameters"]["tile_max_number"] + classifier = {} if self.parameters["tiling_parameters"].get("enable_tile_classifier", False): adapter = OpenvinoAdapter(create_core(), get_model_path(model_dir / "tile_classifier.xml"), device=device) classifier = Model(model_adapter=adapter, preload=True) - tile_size = self.parameters["tiling_parameters"]["tile_size"] - tile_overlap = self.parameters["tiling_parameters"]["tile_overlap"] - max_number = self.parameters["tiling_parameters"]["tile_max_number"] + if self.parameters["tiling_parameters"].get("tile_ir_scale_factor", False): + tile_size = int(tile_size * self.parameters["tiling_parameters"]["tile_ir_scale_factor"]) + tiler = Tiler(tile_size, tile_overlap, max_number, self.core_model, classifier, self.segm) return tiler diff --git a/otx/api/usecases/exportable_code/demo/requirements.txt b/otx/api/usecases/exportable_code/demo/requirements.txt index 7c8ea680276..fbd58e11114 100644 --- a/otx/api/usecases/exportable_code/demo/requirements.txt +++ b/otx/api/usecases/exportable_code/demo/requirements.txt @@ -1,4 +1,4 @@ openvino==2022.3.0 openmodelzoo-modelapi==2022.3.0 -otx==1.2.0 +otx @ git+https://github.com/openvinotoolkit/training_extensions/@554d0939e677528c41b83d79f61fd0c514746c6c#egg=otx numpy>=1.21.0,<=1.23.5 # np.bool was removed in 1.24.0 which was used in openvino runtime diff --git a/otx/api/utils/tiler.py b/otx/api/utils/tiler.py index cbbabae68ae..8f5bba63507 100644 --- a/otx/api/utils/tiler.py +++ b/otx/api/utils/tiler.py @@ -70,7 +70,7 @@ def tile(self, image: np.ndarray) -> List[List[int]]: return coords def filter_tiles_by_objectness( - self, image: np.ndarray, tile_coords: List[List[int]], confidence_threshold: float = 0.45 + self, image: np.ndarray, tile_coords: List[List[int]], confidence_threshold: float = 0.35 ): """Filter tiles by objectness score by running tile classifier. diff --git a/otx/core/data/adapter/base_dataset_adapter.py b/otx/core/data/adapter/base_dataset_adapter.py index 4250ab8714c..9190fcbbcb4 100644 --- a/otx/core/data/adapter/base_dataset_adapter.py +++ b/otx/core/data/adapter/base_dataset_adapter.py @@ -214,14 +214,14 @@ def _get_subset_data(self, subset: str, dataset: DatumDataset) -> DatumDatasetSu for s in [subset, "default"]: if subset == "val" and s != "default": s = "valid" - exact_subset = get_close_matches(s, subsets) + exact_subset = get_close_matches(s, subsets, cutoff=0.5) if exact_subset: return dataset.subsets()[exact_subset[0]].as_dataset() elif subset == "test": # If there is not test dataset in data.yml, then validation set will be test dataset s = "valid" - exact_subset = get_close_matches(s, subsets) + exact_subset = get_close_matches(s, subsets, cutoff=0.5) if exact_subset: return dataset.subsets()[exact_subset[0]].as_dataset() diff --git a/tests/unit/algorithms/detection/tiling/test_tiling_detection.py b/tests/unit/algorithms/detection/tiling/test_tiling_detection.py index 70f7a8db998..c1462e8e05b 100644 --- a/tests/unit/algorithms/detection/tiling/test_tiling_detection.py +++ b/tests/unit/algorithms/detection/tiling/test_tiling_detection.py @@ -8,10 +8,14 @@ import numpy as np import pytest import torch -from mmcv import ConfigDict +from mmcv import Config, ConfigDict from mmdet.datasets import build_dataloader, build_dataset +from mmdet.models import DETECTORS +from openvino.model_zoo.model_api.adapters import OpenvinoAdapter, create_core +from torch import nn from otx.algorithms.common.adapters.mmcv.utils.config_utils import MPAConfig +from otx.algorithms.common.adapters.mmdeploy.apis import MMdeployExporter from otx.algorithms.detection.adapters.mmdet.task import MMDetectionTask from otx.algorithms.detection.adapters.mmdet.utils import build_detector, patch_tiling from otx.api.configuration.helper import create @@ -32,8 +36,26 @@ ) +@DETECTORS.register_module(force=True) +class MockDetModel(nn.Module): + def __init__(self, backbone, train_cfg=None, test_cfg=None, init_cfg=None): + super().__init__() + self.conv = torch.nn.Conv2d(3, 3, 3) + self.box_dummy = torch.nn.AdaptiveAvgPool2d((1, 5)) + self.label_dummy = torch.nn.AdaptiveAvgPool2d((1)) + self.mask_dummy = torch.nn.AdaptiveAvgPool2d((28, 28)) + + def forward(self, *args, **kwargs): + img = args[0] + x = self.conv(img) + boxes = self.box_dummy(x).mean(1) + labels = self.label_dummy(x).mean(1) + masks = self.mask_dummy(x).mean(1) + return boxes, labels, masks + + def create_otx_dataset(height: int, width: int, labels: List[str]): - """Create a random OTX dataset + """Create a random OTX dataset. Args: height (int): The height of the image @@ -54,11 +76,11 @@ def create_otx_dataset(height: int, width: int, labels: List[str]): class TestTilingDetection: - """Test the tiling functionality""" + """Test the tiling detection algorithm.""" @pytest.fixture(autouse=True) def setUp(self) -> None: - """Setup the test case""" + """Setup the test case.""" self.height = 1024 self.width = 1024 self.label_names = ["rectangle", "ellipse", "triangle"] @@ -132,7 +154,7 @@ def setUp(self) -> None: @e2e_pytest_unit def test_tiling_train_dataloader(self): - """Test that the training dataloader is built correctly for tiling""" + """Test that the training dataloader is built correctly for tiling.""" dataset = build_dataset(self.train_data_cfg) train_dataloader = build_dataloader(dataset, **self.dataloader_cfg) @@ -143,7 +165,7 @@ def test_tiling_train_dataloader(self): @e2e_pytest_unit def test_tiling_test_dataloader(self): - """Test that the testing dataloader is built correctly for tiling""" + """Test that the testing dataloader is built correctly for tiling.""" dataset = build_dataset(self.test_data_cfg) stride = int((1 - self.tile_cfg["overlap_ratio"]) * self.tile_cfg["tile_size"]) @@ -160,7 +182,7 @@ def test_tiling_test_dataloader(self): @e2e_pytest_unit def test_inference_merge(self): - """Test that the inference merge works correctly""" + """Test that the inference merge works correctly.""" dataset = build_dataset(self.test_data_cfg) # create simulated inference results @@ -222,7 +244,7 @@ def test_load_tiling_parameters(self, tmp_dir_path): @e2e_pytest_unit def test_patch_tiling_func(self): - """Test that patch_tiling function works correctly""" + """Test that patch_tiling function works correctly.""" cfg = MPAConfig.fromfile(os.path.join(DEFAULT_ISEG_TEMPLATE_DIR, "model.py")) model_template = parse_model_template(os.path.join(DEFAULT_ISEG_TEMPLATE_DIR, "template.yaml")) hyper_parameters = create(model_template.hyper_parameters.data) @@ -235,6 +257,60 @@ def test_patch_tiling_func(self): patch_tiling(cfg, hyper_parameters, self.otx_dataset) @e2e_pytest_unit - def test_openvino(self): - # TODO[EUGENE]: implement unittest for tiling prediction with openvino - pass + @pytest.mark.parametrize("scale_factor", [1, 1.5, 2, 3, 4]) + def test_tile_ir_scale_deploy(self, tmp_dir_path, scale_factor): + """Test that the IR scale factor is correctly applied during inference.""" + model_template = parse_model_template(os.path.join(DEFAULT_ISEG_TEMPLATE_DIR, "template.yaml")) + hyper_parameters = create(model_template.hyper_parameters.data) + hyper_parameters.tiling_parameters.enable_tiling = True + hyper_parameters.tiling_parameters.tile_ir_scale_factor = scale_factor + task_env = init_environment(hyper_parameters, model_template) + img_norm_cfg = dict(mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True) + task = MMDetectionTask(task_env) + pipeline = [ + dict(type="LoadImageFromFile"), + dict( + type="MultiScaleFlipAug", + img_scale=(800, 800), + flip=False, + transforms=[ + dict(type="Resize", keep_ratio=False), + dict(type="RandomFlip"), + dict(type="Normalize", **img_norm_cfg), + dict(type="Pad", size_divisor=32), + dict(type="DefaultFormatBundle"), + dict(type="Collect", keys=["img"]), + ], + ), + ] + config = Config( + dict(model=dict(type="MockDetModel", backbone=dict(init_cfg=None)), data=dict(test=dict(pipeline=pipeline))) + ) + + deploy_cfg = task._init_deploy_cfg(config) + onnx_path = MMdeployExporter.torch2onnx( + tmp_dir_path, + np.zeros((50, 50, 3), dtype=np.float32), + config, + deploy_cfg, + ) + assert isinstance(onnx_path, str) + assert os.path.exists(onnx_path) + + openvino_paths = MMdeployExporter.onnx2openvino( + tmp_dir_path, + onnx_path, + deploy_cfg, + ) + for openvino_path in openvino_paths: + assert os.path.exists(openvino_path) + + task._init_task() + original_width, original_height = task._recipe_cfg.data.test.pipeline[0].img_scale # w, h + + model_adapter = OpenvinoAdapter(create_core(), openvino_paths[0], openvino_paths[1]) + + ir_input_shape = model_adapter.get_input_layers()["image"].shape + _, _, ir_height, ir_width = ir_input_shape + assert ir_height == original_height * scale_factor + assert ir_width == original_width * scale_factor diff --git a/tests/unit/core/data/adapter/test_detection_adapter.py b/tests/unit/core/data/adapter/test_detection_adapter.py index 51856cd05a4..7222d3c7c82 100644 --- a/tests/unit/core/data/adapter/test_detection_adapter.py +++ b/tests/unit/core/data/adapter/test_detection_adapter.py @@ -63,6 +63,73 @@ def test_detection(self): assert isinstance(det_test_dataset_adapter.get_otx_dataset(), DatasetEntity) assert isinstance(det_test_dataset_adapter.get_label_schema(), LabelSchemaEntity) + @e2e_pytest_unit + def test_get_subset_data(self): + class MockDatumDataset: + def __init__(self, subsets): + self._subsets = subsets + self.eager = None + + def subsets(self): + return self._subsets + + class MockDataset: + def __init__(self, string): + self.string = string + + def as_dataset(self): + return self.string + + task = "detection" + + task_type: TaskType = TASK_NAME_TO_TASK_TYPE[task] + data_root_dict: dict = TASK_NAME_TO_DATA_ROOT[task] + + train_data_roots: str = os.path.join(self.root_path, data_root_dict["train"]) + val_data_roots: str = os.path.join(self.root_path, data_root_dict["val"]) + + det_train_dataset_adapter = DetectionDatasetAdapter( + task_type=task_type, + train_data_roots=train_data_roots, + val_data_roots=val_data_roots, + ) + + dataset = MockDatumDataset( + { + "train": MockDataset("train"), + "val": MockDataset("val"), + "test": MockDataset("test"), + } + ) + assert det_train_dataset_adapter._get_subset_data("val", dataset) == "val" + + dataset = MockDatumDataset( + { + "train": MockDataset("train"), + "validation": MockDataset("validation"), + "test": MockDataset("test"), + } + ) + assert det_train_dataset_adapter._get_subset_data("val", dataset) == "validation" + + dataset = MockDatumDataset( + { + "train": MockDataset("train"), + "trainval": MockDataset("trainval"), + "val": MockDataset("val"), + } + ) + assert det_train_dataset_adapter._get_subset_data("val", dataset) == "val" + + dataset = MockDatumDataset( + { + "train2017": MockDataset("train2017"), + "val2017": MockDataset("val2017"), + "test2017": MockDataset("test2017"), + } + ) + assert det_train_dataset_adapter._get_subset_data("val", dataset) == "val2017" + @e2e_pytest_unit def test_instance_segmentation(self): task = "instance_segmentation"