Skip to content

Commit

Permalink
Update Quickstart guide to reference Object Detection 2D example (#298)
Browse files Browse the repository at this point in the history
* Update quickstart to reference Object Detection 2d example

* update example on extending annotations

* improve wording of annoation expansion section

* revert update to expanding annotation section

* update based on PR comments

* rename mask_cnn to mask_rcnn

* fix whitespace

* fix spacing
  • Loading branch information
sandalns authored Nov 6, 2023
1 parent c85ef45 commit af89ff4
Show file tree
Hide file tree
Showing 5 changed files with 20 additions and 22 deletions.
2 changes: 1 addition & 1 deletion docs/core-concepts/workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ can be tested can be modeled as a workflow in Kolena.
Examples of workflows include:

<div class="grid cards" markdown>
- [:kolena-keypoint-detection-20: Keypoint Detection](https://github.com/kolenaIO/kolena/tree/trunk/examples/keypoint_detection) using images
- [:kolena-widget-20: Object Detection (2D)](https://github.com/kolenaIO/kolena/tree/trunk/examples/object_detection_2d) using images
- [:kolena-text-summarization-20: Text Summarization](https://github.com/kolenaIO/kolena/tree/trunk/examples/text_summarization) using articles/documents
- [:kolena-age-estimation-20: Age Estimation](https://github.com/kolenaIO/kolena/tree/trunk/examples/age_estimation) (regression) using images
- [:kolena-video-20: Video Retrieval](https://paperswithcode.com/task/video-retrieval) using text queries on a corpus of videos
Expand Down
32 changes: 15 additions & 17 deletions docs/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ icon: kolena/flame-16
Install Kolena to set up rigorous and repeatable model testing in minutes.

In this quickstart guide, we'll use the
[`age_estimation`](https://github.com/kolenaIO/kolena/tree/trunk/examples/age_estimation) example integration to
[`object_detection_2d`](https://github.com/kolenaIO/kolena/tree/trunk/examples/object_detection_2d) example integration to
demonstrate the how to curate test data and test models in Kolena.

## Install `kolena`
Expand Down Expand Up @@ -135,10 +135,10 @@ git clone https://github.com/kolenaIO/kolena.git
```

With the repository cloned, let's set up the
[`age_estimation`](https://github.com/kolenaIO/kolena/tree/trunk/examples/age_estimation) example:
[`object_detection_2d`](https://github.com/kolenaIO/kolena/tree/trunk/examples/object_detection_2d) example:

```shell
cd kolena/examples/age_estimation
cd kolena/examples/object_detection_2d
poetry update && poetry install
```

Expand All @@ -152,7 +152,7 @@ Each of the example integrations comes with scripts for two flows:
1. `seed_test_suite.py`: Create test cases and test suite(s) from a source dataset
2. `seed_test_run.py`: Test model(s) on the created test suites

Before running [`seed_test_suite.py`](https://github.com/kolenaIO/kolena/blob/trunk/examples/age_estimation/age_estimation/seed_test_suite.py),
Before running [`seed_test_suite.py`](https://github.com/kolenaIO/kolena/blob/trunk/examples/object_detection_2d/object_detection_2d/seed_test_suite.py),
let's first configure our environment by populating the `KOLENA_TOKEN`
environment variable. Visit the [:kolena-developer-16: Developer](https://app.kolena.io/redirect/developer) page to
generate an API token and copy and paste the code snippet into your environment:
Expand All @@ -164,40 +164,38 @@ export KOLENA_TOKEN="********"
We can now create test suites using the provided seeding script:

```shell
poetry run python3 age_estimation/seed_test_suite.py
poetry run python3 object_detection_2d/seed_test_suite.py
```

After this script has completed, we can visit the [:kolena-test-suite-16: Test Suites](https://app.kolena.io/redirect/testing)
page to view our newly created test suites.

In this `age_estimation` example, we've created test suites stratifying the LFW dataset (which is stored as a CSV in
S3) into test cases by age, estimated race, and estimated gender.
In this `object_detection_2d` example, we've created test suites stratifying the [COCO](https://cocodataset.org/#overview) 2014 validation set (which is stored as a CSV in
S3) into test cases by brightness and bounding box size. In this example will be looking at the following labels:

`["bicycle", "car", "motorcycle", "bus", "train", "truck", "traffic light", "fire hydrant", "stop sign"]`

## Test a Model

After we've created test suites, the final step is to test models on these test suites. The `age_estimation` example
provides the `ssrnet` model for this step:
After we've created test suites, the final step is to test models on these test suites. The `object_detection_2d` example
provides the following models to choose from `{yolo_r, yolo_x, mask_rcnn, faster_rcnn, yolo_v4s, yolo_v3}` for this step:

```shell
poetry run python3 age_estimation/seed_test_run.py \
"ssrnet" \
"age :: labeled-faces-in-the-wild [age estimation]" \
"race :: labeled-faces-in-the-wild [age estimation]" \
"gender :: labeled-faces-in-the-wild [age estimation]"
poetry run python3 object_detection_2d/seed_test_run.py "yolo_v4s"
```

!!! note "Note: Testing additional models"
In this example, model results have already been extracted and are stored in CSV files in S3. To run a new model,
plug it into the `infer` method in [`seed_test_run.py`](https://github.com/kolenaIO/kolena/blob/trunk/examples/age_estimation/age_estimation/seed_test_run.py).
plug it into the `infer` method in [`seed_test_run.py`](https://github.com/kolenaIO/kolena/blob/trunk/examples/object_detection_2d/object_detection_2d/seed_test_run.py).

Once this script has completed, click the results link in your console or visit
[:kolena-results-16: Results](https://app.kolena.io/redirect/results) to view the test results for this newly tested model.

## Conclusion

In this quickstart, we used an example integration from [kolenaIO/kolena](https://github.com/kolenaIO/kolena) to create
test suites from the [Labeled Faces in the Wild (LFW)](http://vis-www.cs.umass.edu/lfw/) dataset and test the
open-source `ssrnet` model on these test suites.
test suites from the [COCO](https://cocodataset.org/#overview) dataset and test the
open-source `yolo_v4s` model on these test suites.

This example shows us how to define an ML problem as a workflow for testing in Kolena, and can be arbitrarily extended
with additional metrics, plots, visualizations, and data.
4 changes: 2 additions & 2 deletions examples/object_detection_2d/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,15 +30,15 @@ This project defines two scripts that perform the following operations:
`small`, `medium` and `large`.

2. [`seed_test_run.py`](object_detection_2d/seed_test_run.py) tests the following models on the above test suites:
`yolo_r`, `yolo_x`, `mask_cnn`, `faster_rcnn`, `yolo_v4s`, and `yolo_v3`. Information about these models can be
`yolo_r`, `yolo_x`, `mask_rcnn`, `faster_rcnn`, `yolo_v4s`, and `yolo_v3`. Information about these models can be
found in [`constants.py`](object_detection_2d/constants.py).

Command line arguments are defined within each script to specify what model to use and what test suite to
seed/evaluate. Run a script using the `--help` flag for more information:

```shell
$ poetry run python3 object_detection_2d/seed_test_run.py --help
usage: seed_test_run.py [-h] [--test_suite TEST_SUITE] {yolo_r,yolo_x,mask_cnn,faster_rcnn,yolo_v4s,yolo_v3}
usage: seed_test_run.py [-h] [--test_suite TEST_SUITE] {yolo_r,yolo_x,mask_rcnn,faster_rcnn,yolo_v4s,yolo_v3}

positional arguments:
{yolo_r,yolo_x,mask_cnn,faster_rcnn,yolo_v4s,yolo_v3}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@
MODEL_METADATA: Dict[str, Dict[str, str]] = {
"yolo_r": YOLO_R,
"yolo_x": YOLO_X,
"mask_cnn": MASK_RCNN,
"mask_rcnn": MASK_RCNN,
"faster_rcnn": FAST_RCNN,
"yolo_v4s": YOLO_V4,
"yolo_v3": YOLO_V3,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@
MODEL_LIST: Dict[str, str] = {
"yolo_r": f"YOLOR-D6 (modified CSP, {WORKFLOW})",
"yolo_x": f"YOLOX (modified CSP-v5, {WORKFLOW})",
"mask_cnn": f"Mask R-CNN (Inception-ResNet-v2, {WORKFLOW})",
"mask_rcnn": f"Mask R-CNN (Inception-ResNet-v2, {WORKFLOW})",
"faster_rcnn": f"Faster R-CNN (Inception-ResNet-v2, {WORKFLOW})",
"yolo_v4s": f"Scaled YOLOv4 (CSP-DarkNet-53, {WORKFLOW})",
"yolo_v3": f"YOLOv3 (DarkNet-53, {WORKFLOW})",
Expand Down

0 comments on commit af89ff4

Please sign in to comment.