From 8583a9545e3f30542ca053a5c68a4abfcd88c2f6 Mon Sep 17 00:00:00 2001 From: Jonas Schult Date: Tue, 8 Aug 2023 16:09:17 -0700 Subject: [PATCH 1/3] Update index.html --- index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.html b/index.html index 6471316..92e1cd8 100644 --- a/index.html +++ b/index.html @@ -106,7 +106,7 @@

Important Dates 🗓️

  • Challenge Track:
    • From e7a068c9490b7a9eb6640ef475904d733d80f3e4 Mon Sep 17 00:00:00 2001 From: Ayca Takmaz Date: Fri, 11 Aug 2023 09:53:58 +0200 Subject: [PATCH 2/3] Update index.html --- index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.html b/index.html index 92e1cd8..a2140ae 100644 --- a/index.html +++ b/index.html @@ -115,7 +115,7 @@

    Important Dates 🗓️

  • Data Instructions & Helper Scripts: GitHub
  • Dev Phase Start: July 13, 2023
  • Submission Portal Start: July 17, 2023
    • -
  • Test Phase Start: August 13, 2023
    • +
  • Test Phase Start: August 16, 2023
  • Test Phase End: September 30, 2023 (Winners are decided on this date)
    • From f9c5a41ce89521edc5e4d6af6e63f03812b1ff76 Mon Sep 17 00:00:00 2001 From: Ayca Takmaz Date: Wed, 16 Aug 2023 07:48:08 +0200 Subject: [PATCH 3/3] add scenes for the test phase --- challenge/README.md | 11 ++++---- .../challenge_test_scenes.csv | 27 ++++++++++++++++++- .../queries_test_scenes.csv | 26 ++++++++++++++++++ 3 files changed, 57 insertions(+), 7 deletions(-) diff --git a/challenge/README.md b/challenge/README.md index 19ee2ad..fce1cc7 100644 --- a/challenge/README.md +++ b/challenge/README.md @@ -32,7 +32,7 @@ Our challenge consists of two phases: *Development Phase*, and *Test Phase*. - In the first phase, the *Development Phase*, the challenge participants can download and use the whole *Training* split of the ARKitScenes dataset for their experiments. From these *Training* scenes, we annotate a few example scenes for development purposes. More specifically, for each example scene, we first specify an open-vocabulary query, and then manually annotate objects corresponding to the given query, by segmenting them in the point cloud of the given scene. We refer to this subset as the *Challenge Development* set. The participants can upload their predictions on scenes from the *Challenge Development* set to the [Phase 1 - Development Benchmark](https://eval.ai/web/challenges/challenge-page/2102/submission) in the challenge website. -- In the second phase, the *Test Phase*, we provide a subset of scenes from the *Validation* split of the ARKitScenes dataset, we refer to this subset as the *Challenge Test* set. For each of these scenes, we provide an input text-query. The participants are expected to upload their predictions for each scene from the *Challenge Test* to the [Phase 2 - Test Benchmark](https://eval.ai/web/challenges/challenge-page/2102/submission) in the challenge website. Data for this phase will be made available for downloading in August 2023, this will be announced on our website. +- In the second phase, the *Test Phase*, we provide a subset of scenes from the *Validation* split of the ARKitScenes dataset, we refer to this subset as the *Challenge Test* set. For each of these scenes, we provide an input text-query. The participants are expected to upload their predictions for each scene from the *Challenge Test* to the [Phase 2 - Test Benchmark](https://eval.ai/web/challenges/challenge-page/2102/submission) in the challenge website. ### **Download Instructions** @@ -47,16 +47,15 @@ Download the data using ``` python3 challenge/download_data_opensun3d.py --data_type=challenge_development_set --download_dir PATH/TO/ARKITSCENES/DOWNLOAD/DIR ``` -Queries for each scene are available in [`queries_development_scenes.csv`](benchmark_file_lists/queries_test_scenes.csv). +Queries for each scene are available in [`queries_development_scenes.csv`](benchmark_file_lists/queries_development_scenes.csv). Furthermore, we provide ground truth instance masks for the development scenes [here](benchmark_data/gt_development_scenes), whose data format is explained [here](benchmark_data/gt_data_format.md). Please note that submission of the predicted masks require a different file format, explained in more detail [here](#submission-instructions). #### **Phase 2 - Download Challenge Test Set (~30GB)** ->NOTE: Data for this phase will be made available for downloading in August 2023, this will be announced on our website. ``` python3 challenge/download_data_opensun3d.py --data_type=challenge_test_set --download_dir PATH/TO/ARKITSCENES/DOWNLOAD/DIR ``` -Queries for each scene will be made available in [`queries_test_scenes.csv`](benchmark_file_lists/queries_test_scenes.csv). +Queries for each scene are available in [`queries_test_scenes.csv`](benchmark_file_lists/queries_test_scenes.csv). #### *(Optional, needed only if you want to train a model) Download Full Training Set (Several hundreds of GBs)* ``` @@ -69,7 +68,7 @@ Some of these scenes do not have `wide` assets (see below). If you want to downl python3 challenge/download_data_opensun3d.py --data_type=full_training_set_w_wide_assets --download_dir PATH/TO/ARKITSCENES/DOWNLOAD/DIR ``` -NOTE: If you need to download other assets from the ARKitScenes, please see the [data instructions](https://github.com/apple/ARKitScenes/blob/main/DATA.md) in the [original ARKitScenes repository](https://github.com/apple/ARKitScenes) for further details. For this challenge, in order to develop or train your model, you can only use the scenes provided in the ARKitScenes Training set or the Development set we provide. +NOTE: If you need to download other assets from the ARKitScenes, please see the [data instructions](https://github.com/apple/ARKitScenes/blob/main/DATA.md) in the [original ARKitScenes repository](https://github.com/apple/ARKitScenes) for further details. --- ## **Data Organization and Format of Input Data** @@ -129,7 +128,7 @@ You can explore the [`demo_dataloader_lowres.py`](demo_dataloader_lowres.py) fil scene_id = "42445173" # an example scene ID ``` -Queries for each scene are available in [`queries_development_scenes.csv`](obenchmark_file_lists/queries_test_scenes.csv). First column is `video_id`, second column is `visit_id` and the last column is the open vocabulary query. What we refer to as `{SCENE_ID}` in this document is the `video_id`. +Queries for each development scene are available in [`queries_development_scenes.csv`](benchmark_file_lists/queries_development_scenes.csv), and queries for each test scene are available in [`queries_test_scenes.csv`](benchmark_file_lists/queries_test_scenes.csv). First column is `video_id`, second column is `visit_id` and the last column is the open vocabulary query. What we refer to as `{SCENE_ID}` in this document is the `video_id`. --- ## **Submission Instructions** diff --git a/challenge/benchmark_file_lists/challenge_test_scenes.csv b/challenge/benchmark_file_lists/challenge_test_scenes.csv index 8ae21b3..2dd8fb8 100644 --- a/challenge/benchmark_file_lists/challenge_test_scenes.csv +++ b/challenge/benchmark_file_lists/challenge_test_scenes.csv @@ -1 +1,26 @@ -video_id,visit_id,fold +video_id,visit_id,fold +42446165,422014,Validation +42446151,422018,Validation +42445991,422024,Validation +42446100,422034,Validation +42446540,422381,Validation +42446529,422386,Validation +42897564,422785,Validation +42897545,422813,Validation +42897688,423441,Validation +42898818,434641,Validation +42898817,434650,Validation +42899459,435385,Validation +45261150,466801,Validation +45261133,466803,Validation +47333934,467301,Validation +47331262,468267,Validation +45662987,468711,Validation +47332916,469249,Validation +47430036,470512,Validation +47115474,470652,Validation +47115526,470659,Validation +47115452,470661,Validation +47204573,471940,Validation +47895732,482328,Validation +48018367,483313,Validation \ No newline at end of file diff --git a/challenge/benchmark_file_lists/queries_test_scenes.csv b/challenge/benchmark_file_lists/queries_test_scenes.csv index e69de29..680a181 100644 --- a/challenge/benchmark_file_lists/queries_test_scenes.csv +++ b/challenge/benchmark_file_lists/queries_test_scenes.csv @@ -0,0 +1,26 @@ +video_id,visit_id,query +42446165,422014,watercolor painting on the table +42446151,422018,hoverboard +42445991,422024,picture of a horse +42446100,422034,post-it +42446540,422381,red switch +42446529,422386,package of paper towel rolls +42897564,422785,victorian upholstered chair +42897545,422813,watering can +42897688,423441,Dyson air purifier +42898818,434641,seashell +42898817,434650,Persil +42899459,435385,large letter M +45261150,466801,Star Wars sheet +45261133,466803,portrait of Queen Elizabeth +47333934,467301,furniture than can seat three people +47331262,468267,musical instrument +45662987,468711,wall art hanging across the fireplace +47332916,469249,heart shaped pillow +47430036,470512,Winnie-the-Pooh +47115474,470652,folded towel stack +47115526,470659,hippopotamus +47115452,470661,giant decorative fork +47204573,471940,lego head +47895732,482328,gamepad +48018367,483313,oven mitts \ No newline at end of file