Releases screenshots for SIMMC-Furniture and API calls with better heuristics (SIMMC v1.2)

.gitattributes

@@ -30,3 +30,7 @@ data/simmc_fashion/fashion_teststd_dials_retrieval_candidates_public.json filter
 data/simmc_furniture/furniture_teststd_dials_api_calls.json filter=lfs diff=lfs merge=lfs -text
 data/simmc_furniture/furniture_teststd_dials_public.json filter=lfs diff=lfs merge=lfs -text
 data/simmc_furniture/furniture_teststd_dials_retrieval_candidates_public.json filter=lfs diff=lfs merge=lfs -text
+data/simmc_furniture/furniture_screenshots_part_0.zip filter=lfs diff=lfs merge=lfs -text
+data/simmc_furniture/furniture_screenshots_part_1.zip filter=lfs diff=lfs merge=lfs -text
+data/simmc_furniture/furniture_screenshots_part_2.zip filter=lfs diff=lfs merge=lfs -text
+data/simmc_furniture/furniture_screenshot_map.json filter=lfs diff=lfs merge=lfs -text

.lfsconfig

@@ -0,0 +1,2 @@
+[lfs]
+fetchexclude = data/simmc_furniture/*.zip

README.md

@@ -17,12 +17,17 @@ Our challenge focuses on our SIMMC datasets, both of which are shopping domains:
 </figure>
 
 
+
 ### Latest News
 
+* **[Sept 28, 2020]** Released screenshots for SIMMC-Furniture [here][screenshot_link]. Also released improved API calls with newer heuristics as SIMMC v1.2 ([PR][screenshot_pr]).
+* **[Dec 29, 2020]** Fixed the errors in text spans for both SIMMC-Furniture and SIMMC-Fashion, released new JSON files as SIMMC v1.1 ([PR][span_fix_pr]).
 * **[Sept 28, 2020]** Test-Std data released, End of Challenge Phase 1.
-* **[June 22, 2020]** Challenge announcement. Training / development datasets are released.
-* **[July 8, 2020]** Evaluation scripts and code to train baselines for 
-Sub-Task #1, Sub-Task #2 released.
+* **[July 8, 2020]** Evaluation scripts and code to train baselines for Sub-Task #1, Sub-Task #2 released.
+* **[June 22, 2020]** Challenge announcement. Training / development datasets (SIMMC v1.0) are released.
+
+**Note:** DSTC9 SIMMC Challenge was conducted on SIMMC v1.0. Thus all the results and baseline performances are on SIMMC v1.0.
+
 
 ## Important Links
 
@@ -149,3 +154,6 @@ SIMMC is released under [CC-BY-NC-SA-4.0](https://creativecommons.org/licenses/b
 
 [dstc9]:https://sites.google.com/dstc.community/dstc9/home
 [simmc_arxiv]:https://arxiv.org/abs/2006.01460
+[screenshot_link]:./data/simmc_furniture/furniture_screenshots.zip
+[span_fix_pr]:https://github.com/facebookresearch/simmc/pull/54
+[screenshot_pr]:https://github.com/facebookresearch/simmc/pull/60

mm_action_prediction/README.md

@@ -16,6 +16,7 @@ We also measure the correctness of the predicted action (API) arguments using at
 Specifically, the following API classes and attributes are evaluated.
 
 **SIMMC-Furniture**
+
 |  API   |  API Attributes |
 |:------:| :--------: |
 | `SearchFurniture`  | `furnitureType`, `color` |
@@ -30,6 +31,7 @@ Each of the above attributes is a categorical variable, modeled as multiclass cl
 **Note:** `minPrice` and `maxPrice` attributes corresponding to the `SpecifyInfo` action for Furniture are excluded in the current evaluation.
 
 **SIMMC-Fashion**
+
 | API  |   API Attributes |
 |:--------: | :------: |
 | `SearchDatabase` | `attributes` |
@@ -205,6 +207,8 @@ The baselines trained through the code obtain the following results for Sub-Task
 | MN     | 81.6 | 1.74 | 61.6 |
 | T-HAE  | 81.4 | 1.78 | 62.1 |
 
+**Note:** DSTC9 SIMMC Challenge was conducted on SIMMC v1.0. Thus all the results and baseline performances are on SIMMC v1.0.
+
 
 ## Rules for Sub-task #1 Submissions
 * Disallowed Input: `belief_state`, `system_transcript`, `system_transcript_annotated`, `state_graph_1`, `state_graph_2`, and anything from future turns.

mm_action_prediction/tools/extract_actions.py

@@ -723,6 +723,13 @@ def extract_actions(input_json_file, save_root, furniture_db, subtask):
             insert_item["current_search_results"] = copy.deepcopy(search_results)
             chat_utterances.append(insert_item)
 
+        # if dialog_id == 18702 or dialog_id == "18702":
+        #     print("Here:")
+        #     for turn_datum in chat_utterances:
+        #         print([ii["api"] for ii in turn_datum["raw_action_with_args"]])
+        #     import pdb; pdb.set_trace()
+        #     pass
+
         roundwise_actions = get_roundwise_dialog_actions(
             subtask,
             chat_utterances
@@ -1073,7 +1080,6 @@ def get_roundwise_dialog_actions(subtask, dialog_actions):
                 # for GET_INFO_ACTION and ADD_TO_CART_ACTION subtask create a
                 # single reference relative to the carousel
                 furniture_ids = cur_supervision[ARGS][FURNITURE_ID]
-                # {'focus': '', 'carousel': ['901712', '1215555']}
                 if turn_carousel_state["focus"]:
                     if turn_carousel_state["focus"] in furniture_ids:
                         cur_supervision[ARGS][FURNITURE_ID] = "focus"
@@ -1108,15 +1114,20 @@ def get_roundwise_dialog_actions(subtask, dialog_actions):
         action_datum["action"] = cur_action
         action_datum["action_supervision"] = cur_supervision
         action_datum["action_output_state"] = action_output_state
-        roundwise_actions.append(action_datum)
 
-        # Go through all the raw_actions to get the next turn_state.
+        # Go through all the raw_actions to get the next turn_state 
+        # (only if Share is the last action)
         raw_actions = turn_datum["raw_action_with_args"]
-        if len(raw_actions):
-            turn_carousel_state = {
-                "focus": raw_actions[-1][NEXT_STATE][SHARED_FOCUS],
-                "carousel": raw_actions[-1][NEXT_STATE][SHARED_CAROUSEL],
-            }
+        for raw_act in raw_actions:
+            if raw_act[API] == SHARE:
+                turn_carousel_state = {
+                    "focus": raw_act[NEXT_STATE][SHARED_FOCUS],
+                    "carousel": raw_act[NEXT_STATE][SHARED_CAROUSEL],
+                }
+
+        # Also record the final carousel_state.
+        action_datum["final_carousel_state"] = copy.deepcopy(turn_carousel_state)
+        roundwise_actions.append(action_datum)
     return roundwise_actions
 
 
@@ -1131,12 +1142,11 @@ def update_carousel_state(filtered_action, current_state):
         insert_carousal_state: Carousel state to insert for the action
         new_current_state: State of carousel after executing the action
     """
-    if filtered_action[API] != GET_INFO_ACTION:
-        current_state = get_carousel_state(
-            filtered_action["previousState"], filtered_action
-        )
+    # NOTE: For GetInfo, AddToCart, None actions, do not update the states.
+    ignore_actions = [GET_INFO_ACTION, ADD_TO_CART_ACTION, NONE_ACTION]
+    # action shouldn't effect the view of the previous state
     insert_carousel_state = copy.deepcopy(current_state)
-    if filtered_action[API] != GET_INFO_ACTION:
+    if filtered_action[API] not in ignore_actions:
         current_state = get_carousel_state(
             filtered_action["nextState"], filtered_action
         )
@@ -1174,20 +1184,27 @@ def get_carousel_state(state=None, action_args=None):
         return new_state
 
     # Get items in the carousel.
+    search_order_carousel = ["prefabsInCarousel", "sharedPrefabsInCarousel"]
+    search_order_focus = ["prefabInFocus", "sharedPrefabInFocus"]
     action = action_args["api"]
-    if action == "SearchFurniture":
-        new_state["carousel"] = state["sharedPrefabsInCarousel"]
-    elif action in ["Rotate", "FocusOnFurniture"]:
+    if action in [
+        SEARCH_FURNITURE_ACTION,
+        NAVIGATE_CAROUSEL_ACTION
+    ]:
+        for order in search_order_carousel:
+            if state[order]:
+                new_state["carousel"] = state[order]
+                # break loop and return - don't return focus item
+                # SearchFurniture and NavigateCarousel actions
+                return new_state
+    elif action in [ROTATE_ACTION, FOCUS_ON_FURNITURE_ACTION]:
         focus_id = action_args["args"]["furniture_id"]
-        search_order = ["prefabsInCarousel", "sharedPrefabsInCarousel"]
-        for order in search_order:
+        for order in search_order_carousel:
             if focus_id in state[order]:
                 new_state["carousel"] = state[order]
-    elif action in ["Previous", "Next"]:
-        new_state["carousel"] = state["prefabsInCarousel"]
+
     # Focus object.
-    search_order = ["prefabInFocus", "sharedPrefabInFocus"]
-    for key in search_order:
+    for key in search_order_focus:
         if state[key] != "":
             new_state["focus"] = state[key]
             break

mm_dst/README.md

@@ -186,6 +186,8 @@ Below is the summary of the [published models](https://github.com/facebookresear
 | GPT2 - Fashion (text-only) | 61.2 | 52.1 |
 | GPT2 - Fashion (multimodal) | 61.1 | 60.6 |
 
+**Note:** DSTC9 SIMMC Challenge was conducted on SIMMC v1.0. Thus all the results and baseline performances are on SIMMC v1.0.
+
 ## Rules for Sub-task #3 Submissions
 * Disallowed input per each turn: `belief_state`, `system_transcript`, `system_transcript_annotated`, `state_graph_1`, `state_graph_2`, and anything from future turns.
 * If you would like to use any other external resources, please consult with the track organizers (simmc@fb.com). Generally, we allow the use of publicly available pre-trained language models, such as BERT, GPT-2, etc.