Merge pull request #1257 from facebookresearch/remote-procedure-examp…

…le-image-prompts

Added example task 'Interactive Image Generation'

docs/web/docs/explanations/architecture_overview.md

@@ -33,7 +33,7 @@ This is a quick guide over file directories in Mephisto project. Note that some
         - `webapp` - task's React app containing FC
             - `src` - React code to display FC component (example of using FC plugin in a project)
             - `webpack.config.js` - webpack config for React app containing FC
-            - `webpack.config.presigned_urls.js` - webpack config for React app containing FC with presigned S3 URLs feature
+            - `webpack.config.remote_procedure.js` - webpack config for React app containing FC with remote procedures (e.g. presigned S3 URLs feature)
             - `webpack.config.review.js` - webpack config for React app displayin read-only task page inside iframe (for TaskReview app)
       - `static_react_task` - simple React application (with Onboarding and Screening)
       - `static_react_task_with_worker_opinion` - simple React application with Worker Opinion widget

docs/web/docs/guides/how_to_contribute/documentation.md

@@ -22,9 +22,15 @@ docker-compose -f docker/docker-compose.dev.yml up
 docker exec -it mephisto_dc bash
 mephisto scripts local_db auto_generate_all_docs_reference_md
 cd /mephisto/docs/web/
+
 yarn
 yarn install
 yarn start-dev:docker
+
+# or
+
+npm install
+npm run start-dev:docker
 ```
 
 Then run `mephisto scripts local_db auto_generate_all_docs_reference_md` to auto-generate `.md` files in `docs/web/docs/reference` directory. (They will describe parameters of some Python classes.) Noe that on production this command is called by Github Actions.
@@ -42,8 +48,15 @@ Sometimes development server do not raise exceptions which production one can.
 docker-compose -f docker/docker-compose.dev.yml up
 docker exec -it mephisto_dc bash
 cd /mephisto/docs/web/
+
 yarn
 yarn install
 yarn build
 yarn serve-prod:docker
+
+# or
+
+npm install
+npm run build
+npm run serve-prod:docker
 ```

docs/web/docs/guides/how_to_contribute/known_issues.md

@@ -12,3 +12,6 @@ sidebar_position: 6
 While we strive to make Mephisto work its best, there are a few "perennial" issues to be aware of:
 
 1. Sometimes Tasks consisting of only 1 Unit don't shut themselves down upon that Unit completion.
+2. After restarting a Task numerous times within the same Docker container, 
+remote procedures will eventually fail to respond and websockets will be automatically closed. 
+(Reloading Task page will re-establish a socket and you will get the old responses, but the new page code won't have the context to process them)

docs/web/docs/guides/how_to_use/form_composer/configuration/config_files.md

@@ -165,6 +165,8 @@ TBD: Other classes and styles insertions
 - `show_instructions_as_modal` - Enables showing `instruction` content as a modal (opened by clicking a sticky button in top-right corner); this make lengthy task instructions available from any place of a lengthy form without scrolling the page (Boolean, Optional, Default: false)
 - `title` - HTML header of the form (String)
 - `submit_button` - Button to submit the whole form and thus finish a task (Object)
+    - `classes` - Custom classes that you can use to restyle element or refer to it from custom handlers code (String, Optional)
+    - `classes_button_element` - Custom classes that you can use to restyle button element or refer to it from custom handlers code (String, Optional)
     - `id` - Unique HTML id of the button, in case we need to refer to it from custom handlers code (String, Optional)
     - `instruction` - Text shown above the "Submit" button (String, Optional)
     - `text` - Label shown on the button (String)
@@ -206,6 +208,7 @@ Each item of `fieldsets` list is an object with the following attributes:
 
 Each item of `rows` list is an object with the following attributes:
 
+- `help` - HTML explanation of the row - displayed in small font below the field (String, Optional)
 - `id` - Unique HTML id of the row, in case we need to refer to it from custom handlers code (String, Optional)
 - `classes` = Custom classes that you can use to restyle element or refer to it from custom handlers code (String, Optional)
 - `fields` - **List of fields** that will be lined up into one horizontal line
@@ -214,7 +217,7 @@ Each item of `rows` list is an object with the following attributes:
 
 #### Config level: field
 
-Each item of `fields` list is an object that corresponds to the actual form field displayed in the resulting Task UI page.
+Each item of `fields` list is an object corresponding to the actual form field that's displayed in the resulting Task UI page.
 
 Here's example of a single field config:
 
@@ -257,12 +260,12 @@ The `value` attribute specifies initial value of a field, and has the following
 
 The most important attributes are: `label`, `name`, `type`, `validators`
 
-- `help` - HTML explanation of the field/fieldset displayed in small font below the field (String, Optional)
+- `help` - HTML explanation of the field/fieldset - displayed in small font below the field (String, Optional)
 - `id` - Unique HTML id of the field, in case we need to refer to it from custom handlers code (String, Optional)
 - `classes` = Custom classes that you can use to restyle element or refer to it from custom handlers code (String, Optional)
-- `label` - Field name displayed above the field (String)
+- `label` - Field name - displayed above the field (String)
 - `name` - Unique name under which this field's data will be sent to the server (String)
-- `placeholder` - Text faintly displayed in the field before user provides a value (String, Optional)
+- `placeholder` - Text faintly - displayed in the field before user provides a value (String, Optional)
 - `tooltip` - Text shown in browser tooltip on mouseover (String, Optional)
 - `type` - Type of the field (`input`, `email`, `select`, `textarea`, `checkbox`, `radio`, `file`, `hidden`) (String)
 - `validators` - Validators preventing incorrect data from being submitted (Object[<String\>: String|Boolean|Number], Optional). Supported key-value pairs for the `validators` object:

docs/web/docs/guides/how_to_use/form_composer/configuration/insertions.md

@@ -176,14 +176,24 @@ The `myTriggerEventType` parameter can take one of the following values, dependi
 Example in `custom_triggers.js`...
 
 ```js
-export function onClickSectionHeader(
+export function onClickSectionHeader({
   formData, // React state for the entire form
   updateFormData, // callback to set the React state
   element, // "field", "section", or "submit button" element that invoked this trigger
   fieldValue, // (optional) current field value, if the `element` is a form field
-  formFields, // (optional) Object containing all form fields as defined in 'unit_config.json'
-  sectionName // Argument for this trigger (taken from form config)
-) {
+  formFields, // Object containing all form fields as defined in 'unit_config.json'
+  remoteProcedureCollection, // (optional) Collection or remote procedures
+  setDynamicFormElementsConfig, // (optional) Setter for React state with dynamic elements
+  setInvalidFormFields, // (optional) Setter for React state with form fields errors
+  setSubmitErrors, // (optional) Setter for React state with form submit errors
+  setSubmitSuccessText, // (optional) Setter for React state with success text next to submit button
+  submitForm, // (optional) Function that submits form
+  triggerArgs, // Trigger-specific arguments (taken from form config)
+}) {
+  const [
+    sectionName, // Argument for this trigger (taken from form config)
+  ] = triggerArgs;
+
   alert(`${sectionName} section was clicked!`);
 }
 ```

docs/web/docs/guides/how_to_use/task_creation/task_run.md

@@ -433,3 +433,69 @@ We also specify two `units_per_assignment`, meaning that Mephisto creates one `U
 meaning in this case that different workers can complete the same job, usually to get inter-annotator agreement.
 (In some cases Mephisto can use an `Assignment` to connect multiple workers each with one `Unit` on a collaborative live task).
 As we had one assignment, it makes sense that each worker `x` and your second worker could only complete one task each.
+
+
+### 3.5 Server Callbacks
+
+To make your Tasks more dynamic and flexible, we have server callbacks or remote procedures how we call them.
+
+In two words, remote procedures are server Pythong functions that can be called from the UI via JS in the middle of the Task progress.
+
+To start working with them, you need to follow several rules:
+1. Use blueprint `remote_procedure`
+2. Use agent state `SharedRemoteProcedureTaskState`
+3. Write your own remote procedure(-s) and pass them in the Task intialization into agent state
+    ```python
+    from mephisto.abstractions.blueprints.remote_procedure.remote_procedure_agent_state import (
+        RemoteProcedureAgentState,
+    )
+    from mephisto.abstractions.blueprints.remote_procedure.remote_procedure_blueprint import (
+        SharedRemoteProcedureTaskState,
+    )
+    
+    ...
+    
+    def my_remote_procedure(
+        request_id: str,
+        args: dict,
+        agent_state: RemoteProcedureAgentState,
+    ):
+        # Your logic here
+        pass
+        
+    ...    
+    
+    task_data = ...
+    shared_state = SharedRemoteProcedureTaskState(
+        static_task_data=task_data,
+        function_registry={
+            "nameProcedureNameForJs": my_remote_procedure,
+        },
+    )
+    ```
+4. Use `useMephistoRemoteProcedureTask` in yout React-app
+   ```js
+   const {
+     isLoading,
+     initialTaskData,
+     remoteProcedure,
+     handleSubmit,
+     handleFatalError,
+   } = useMephistoRemoteProcedureTask();
+    
+   const nameProcedureNameForJsFn = remoteProcedure("nameProcedureNameForJs");
+    
+   const requestArgs = {
+     // your args that will be passed into `my_remote_procedure` under `args` argument
+   };
+   nameProcedureNameForJsFn(requestArgs)
+     .then((response) => {
+       // Your logic here
+     })
+     .catch((error) => {
+       // JS errors
+     });
+   ```
+
+We recommend to pay attention to our comprehensive [example](/docs/guides/tutorials/model_in_the_loop/), 
+where we touched all related aspects of remote procedures feature.

docs/web/docs/guides/tutorials/form_composer.md

@@ -24,7 +24,7 @@ To launch a form-based Task from within a Docker container follow these steps:
 1. Prepare and validate your config files as described in [FormComposer config files](/docs/guides/how_to_use/form_composer/configuration/setup/) and
 [form_composer config command](/docs/guides/how_to_use/form_composer/configuration/form_composer_config_command/) sections
 
-2. Launch your form-based Task (see details in [Running FormComposer task](/docs/guides/how_to_use/form_composer/running/)). For a lcal testing scenario, we will run:
+2. Launch your form-based Task (see details in [Running FormComposer task](/docs/guides/how_to_use/form_composer/running/)). For a local testing scenario, we will run:
 
 ```shell
 docker-compose -f docker/docker-compose.dev.yml run \

docs/web/docs/guides/tutorials/model_in_the_loop.md

@@ -0,0 +1,85 @@
+---
+# Copyright (c) Meta Platforms and its affiliates.
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+sidebar_position: 7
+---
+
+# Run model-in-the-loop tasks
+
+Mephisto allows to create interactive Tasks where worker can interact with the server during completion of their Task.
+
+
+-----
+
+
+## Interactive Image Generation Task
+
+A good example Task demonstrating this is "Interactive Generation of a Look-Alike Image"
+
+
+#### Task Flow
+
+At a high level, here's what this Task entails:
+
+- Worker's goal is to compose a Generative AI text prompt to produce an image that closely resembles initially displayed "target" image
+- Worker is given up to `max_answer_loops` attempts to achieve the goal
+    - Each attempt consists of judging the generated image (with the resemblance `score`), and updating the text prompt
+        - Worker cannot reuse exactly the same text prompt from any of the previous attempts
+    - Goal is reached if worker indicates `score=10` in their response (perfect resemblance is achieved)
+- After receiving each attempt, based on the updated text prompt the server produces a new image and extra instruction
+    - Here we're picking a random locally stored image, but this could also be a request to a 3rd party API
+    - Extra instruction shown to worker will vary, depending on provided score
+
+
+#### Technical Notes
+
+Some technical aspects are worth pointing out, to understand this example better:
+
+- This Task is based on our [FormComposer feature](/docs/guides/tutorials/form_composer/) since worker input is form-based
+- We make use of [custom triggers](/docs/guides/how_to_use/form_composer/configuration/insertions/#js-trigger-insertion)
+to send server requests invoking server-side (called "remote") procedures
+
+We recommend examining [source code](https://github.com/facebookresearch/Mephisto/blob/main/examples/remote_procedure/interactive_image_generation) for a deeper look, in particular:
+
+- [run_task__local__inhouse.py](https://github.com/facebookresearch/Mephisto/blob/main/examples/remote_procedure/interactive_image_generation/run_task__local__inhouse.py) -
+see how task is being prepared and the logic of remote procedure (that simulates the image generation part)
+- [data](https://github.com/facebookresearch/Mephisto/blob/main/examples/remote_procedure/interactive_image_generation/data/) -
+see FormComposer config used for this Task
+- [custom_triggers.js](https://github.com/facebookresearch/Mephisto/blob/main/examples/remote_procedure/interactive_image_generation/data/insertions/custom_triggers.js) -
+see the UI logic behind clicking "Start" and "Update" buttons in the form:
+    - preparing request data
+    - sending server request
+    - processing server response
+    - updating form's React state
+
+
+#### Run with Docker
+
+To launch this example from within a Docker container follow these steps:
+
+1. (Optional) You can modify, then prepare and validate your config files as described in [FormComposer config files](/docs/guides/how_to_use/form_composer/configuration/setup/) and
+[form_composer config command](/docs/guides/how_to_use/form_composer/configuration/form_composer_config_command/) sections
+
+2. Launch the example for a local testing scenario:
+
+```shell
+docker-compose -f docker/docker-compose.dev.yml run \
+    --build \
+    --publish 8081:8000 \
+    --publish 3001:3000 \
+    --rm mephisto_dc \
+    python /mephisto/examples/remote_procedure/interactive_image_generation/run_task__local__inhouse.py
+```
+
+3. Once your Task launches, your console will display you URLs like this: `http://localhost:3001?worker_id=WORKER_USERNAME&id=1`
+
+
+#### Further Details
+
+You can learn more in these sections:
+- [Run form-based tasks](/docs/guides/tutorials/form_composer/)
+- [FormComposer overview](/docs/guides/how_to_use/form_composer/overview/)
+- [Run FormComposer tasks](/docs/guides/how_to_use/form_composer/running/)
+- [JS trigger insertion](/docs/guides/how_to_use/form_composer/configuration/insertions/#js-trigger-insertion)

examples/form_composer_demo/README.md

@@ -6,12 +6,18 @@ These form-based questionnaires are example of FormComposer task generator.
 
 1. In repo root, launch containers: `docker-compose -f docker/docker-compose.dev.yml up`
 2. SSH into running container to run server: `docker exec -it mephisto_dc bash`
-3. Inside the container, run the project with either of these commands:
-    - Simple form: `cd /mephisto/examples/form_composer_demo && python ./run_task.py`
-    - Dynamic form: `cd /mephisto/examples/form_composer_demo && python ./run_task_dynamic.py`
-    - Dynamic form with Prolific on EC2: `cd /mephisto/examples/form_composer_demo && python ./run_task_dynamic_ec2_prolific.py`
-    - Dynamic form with Mturk on EC2: `cd /mephisto/examples/form_composer_demo && python ./run_task_dynamic_ec2_mturk_sandbox.py`
-    - Simple form with Gold Units: `cd /mephisto/examples/form_composer_demo && python ./run_task_with_gold_unit.py`
+3. Inside the container, go to FormComposer examples directory: `cd /mephisto/examples/form_composer_demo`
+4. Inside the examples directory, run a desired example with one of these commands:
+   - Simple form with Mock provider: `python ./run_task.py`
+   - Simple form with Inhouse provider: `python ./run_task__local__inhouse.py`
+   - Dynamic form with Mock provider: `python ./run_task_dynamic.py`
+   - Dynamic form with Mturk on EC2: `python ./run_task_dynamic_ec2_mturk_sandbox.py`
+   - Dynamic form with Prolific on EC2: `python ./run_task_dynamic_ec2_prolific.py`
+   - Dynamic form with Presigned URLs: `python ./run_task_dynamic_presigned_urls_ec2_prolific.py`
+   - Simple form with Gold Units: `python ./run_task_with_gold_unit.py`
+   - Simple form with Onboarding: `python ./run_task_with_onboarding.py`
+   - Simple form with Screening: `python ./run_task_with_screening.py`
+   - Simple form with Worker Opinion: `python ./run_task_with_worker_opinion.py`
 
 ---
 
@@ -21,7 +27,7 @@ These form-based questionnaires are example of FormComposer task generator.
 2. For dynamic form configs you need two JSON files in `examples/form_composer_demo/data/dynamic` directory:
    - Unit configuration `unit_config.json`
    - Token sets values `token_sets_values_config.json`
-   - To generate extrapolated `task_data.json` config, run this command: `mephisto form_composer config --extrapolate-token-sets True`
+   - To generate extrapolated `task_data.json` config, run this command: `mephisto form_composer config --extrapolate-token-sets`
        - Note that `task_data.json` file will be overwritten with the resulting config
 3. To generate `token_sets_values_config.json` file from token values permutations in `separate_token_values_config.json`, run this command: `mephisto form_composer config --permutate-separate-tokens`
     - Note that `token_sets_values_config.json` file will be overwriten with new sets of tokens values
@@ -31,3 +37,26 @@ These form-based questionnaires are example of FormComposer task generator.
 #### Unit config
 
 For details on how form config is composed, and how its data fields are validated, please see the main FormComposer's [README.md](/mephisto/generators/form_composer/README.md).
+
+---
+
+## End-to-end example
+
+Now let's see how the whole end-to-end list of commands looks for the example `Sample Questionnaire` with presigned URLs:
+
+```shell
+# 1. In your console
+
+docker-compose -f docker/docker-compose.dev.yml up
+docker exec -it mephisto_dc bash
+
+# 2.Inside Docker container
+
+# 2a. Optionally, prepare config (re-generate `task_data.json`)
+mephisto form_composer config --directory /mephisto/examples/form_composer_demo/data/dynamic_presigned_urls/ --permutate-separate-tokens
+mephisto form_composer config --directory /mephisto/examples/form_composer_demo/data/dynamic_presigned_urls/ --extrapolate-token-sets
+
+# 2b. Run the Task
+cd /mephisto/examples/form_composer_demo 
+python ./run_task_dynamic_presigned_urls_ec2_prolific.py
+```