Better contribution guide (#368)

* update docs

* update test cases

* update dependencies

* update PR template

.github/pull_request_template.md

@@ -11,22 +11,23 @@
 ## Checklist:
 <!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
 <!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
-- [ ] I've read the [CONTRIBUTION](https://github.com/vwxyzjn/cleanrl/blob/master/CONTRIBUTING.md) guide (**required**).
+- [ ] I've read the [CONTRIBUTION](https://docs.cleanrl.dev/contribution/) guide (**required**).
 - [ ] I have ensured `pre-commit run --all-files` passes (**required**).
-- [ ] I have updated the documentation and previewed the changes via `mkdocs serve`.
 - [ ] I have updated the tests accordingly (if applicable).
-
-If you are adding new algorithm variants or your change could result in performance difference, you may need to (re-)run tracked experiments. See https://github.com/vwxyzjn/cleanrl/pull/137 as an example PR. 
-- [ ] I have contacted [vwxyzjn](https://github.com/vwxyzjn) to obtain access to the [openrlbenchmark W&B team](https://wandb.ai/openrlbenchmark) (**required**).
-- [ ] I have tracked applicable experiments in [openrlbenchmark/cleanrl](https://wandb.ai/openrlbenchmark/cleanrl) with `--capture-video` flag toggled on (**required**).
-- [ ] I have added additional documentation and previewed the changes via `mkdocs serve`.
+- [ ] I have updated the documentation and previewed the changes via `mkdocs serve`.
     - [ ] I have explained note-worthy implementation details.
     - [ ] I have explained the logged metrics.
-    - [ ] I have added links to the original paper and related papers (if applicable).
-    - [ ] I have added links to the PR related to the algorithm variant.
-    - [ ] I have created a table comparing my results against those from reputable sources (i.e., the original paper or other reference implementation).
-    - [ ] I have added the learning curves (in PNG format).
-    - [ ] I have added links to the tracked experiments.
-    - [ ] I have updated the overview sections at the [docs](https://docs.cleanrl.dev/rl-algorithms/overview/) and the [repo](https://github.com/vwxyzjn/cleanrl#overview)
-- [ ] I have updated the tests accordingly (if applicable).
+    - [ ] I have added links to the original paper and related papers.
+
+If you need to run benchmark experiments for a performance-impacting changes:
+
+- [ ] I have contacted @vwxyzjn to obtain access to the [openrlbenchmark W&B team](https://wandb.ai/openrlbenchmark).
+- [ ] I have used the [benchmark utility](/get-started/benchmark-utility/) to submit the tracked experiments to the [openrlbenchmark/cleanrl](https://wandb.ai/openrlbenchmark/cleanrl) W&B project, optionally with `--capture-video`.
+- [ ] I have performed RLops with `python -m openrlbenchmark.rlops`.
+    - For new feature or bug fix:
+        - [ ] I have used the RLops utility to understand the performance impact of the changes and confirmed there is no regression.
+    - For new algorithm:
+        - [ ] I have created a table comparing my results against those from reputable sources (i.e., the original paper or other reference implementation).
+    - [ ] I have added the learning curves generated by the `python -m openrlbenchmark.rlops` utility to the documentation.
+    - [ ] I have added links to the tracked experiments in W&B, generated by `python -m openrlbenchmark.rlops ....your_args... --report`,  to the documentation.
 

.github/workflows/tests.yaml

@@ -11,7 +11,7 @@ jobs:
       fail-fast: false
       matrix:
         python-version: [3.8]
-        poetry-version: [1.2]
+        poetry-version: [1.3]
         os: [ubuntu-22.04, macos-latest, windows-latest]
     runs-on: ${{ matrix.os }}
     steps:
@@ -50,7 +50,7 @@ jobs:
       fail-fast: false
       matrix:
         python-version: [3.8]
-        poetry-version: [1.2]
+        poetry-version: [1.3]
         os: [ubuntu-22.04, macos-latest, windows-latest]
     runs-on: ${{ matrix.os }}
     steps:
@@ -82,7 +82,7 @@ jobs:
       fail-fast: false
       matrix:
         python-version: [3.8]
-        poetry-version: [1.2]
+        poetry-version: [1.3]
         os: [ubuntu-22.04, macos-latest, windows-latest]
     runs-on: ${{ matrix.os }}
     steps:
@@ -110,7 +110,7 @@ jobs:
       fail-fast: false
       matrix:
         python-version: [3.8]
-        poetry-version: [1.2]
+        poetry-version: [1.3]
         os: [ubuntu-22.04, macos-latest, windows-latest]
     runs-on: ${{ matrix.os }}
     steps:
@@ -136,7 +136,7 @@ jobs:
       fail-fast: false
       matrix:
         python-version: [3.8]
-        poetry-version: [1.2]
+        poetry-version: [1.3]
         os: [ubuntu-22.04]
     runs-on: ${{ matrix.os }}
     steps:
@@ -166,7 +166,7 @@ jobs:
       fail-fast: false
       matrix:
         python-version: [3.8]
-        poetry-version: [1.2]
+        poetry-version: [1.3]
         os: [macos-latest, windows-latest]
     runs-on: ${{ matrix.os }}
     steps:
@@ -193,7 +193,7 @@ jobs:
       fail-fast: false
       matrix:
         python-version: [3.8]
-        poetry-version: [1.2]
+        poetry-version: [1.3]
         os: [ubuntu-22.04]
     runs-on: ${{ matrix.os }}
     steps:
@@ -226,7 +226,7 @@ jobs:
       fail-fast: false
       matrix:
         python-version: [3.8]
-        poetry-version: [1.2]
+        poetry-version: [1.3]
         os: [ubuntu-22.04]
     runs-on: ${{ matrix.os }}
     steps:
@@ -252,7 +252,7 @@ jobs:
       fail-fast: false
       matrix:
         python-version: [3.8]
-        poetry-version: [1.2]
+        poetry-version: [1.3]
         os: [ubuntu-22.04]
     runs-on: ${{ matrix.os }}
     steps:
@@ -278,7 +278,7 @@ jobs:
       fail-fast: false
       matrix:
         python-version: [3.8]
-        poetry-version: [1.2]
+        poetry-version: [1.3]
         os: [ubuntu-22.04, macos-latest]
     runs-on: ${{ matrix.os }}
     steps:

.github/workflows/utils_test.yaml

@@ -11,7 +11,7 @@ jobs:
       fail-fast: false
       matrix:
         python-version: [3.8]
-        poetry-version: [1.2]
+        poetry-version: [1.3]
         os: [ubuntu-22.04]
     runs-on: ${{ matrix.os }}
     steps:

.pre-commit-config.yaml

@@ -38,7 +38,7 @@ repos:
           - --ignore-words-list=nd,reacher,thist,ths,magent,ba
           - --skip=docs/css/termynal.css,docs/js/termynal.js,docs/get-started/CleanRL_Huggingface_Integration_Demo.ipynb
   - repo: https://github.com/python-poetry/poetry
-    rev: 1.2.1
+    rev: 1.3.2
     hooks:
       - id: poetry-export
         name: poetry-export requirements.txt

CONTRIBUTING.md

@@ -1,73 +1 @@
-## Contributing to CleanRL
-
-👍🎉 Thank you for taking the time to contribute! 🎉👍
-
-Feel free to open an issue or a Pull Request if you have any questions or suggestions. You can also [join our Discord](https://discord.gg/D6RCjA6sVT) and ask questions there. If you plan to work on an issue, let us know in the issue thread to avoid duplicate work.
-
-Good luck and have fun!
-
-## Dev Setup
-
-```bash
-poetry install
-poetry install --with atari
-poetry install --with pybullet
-```
-
-Then you can run the scripts under the poetry environment in two ways: `poetry run` or `poetry shell`. 
-
-* `poetry run`:
-    By prefixing `poetry run`, your command will run in poetry's virtual environment. For example, try running
-    ```bash
-    poetry run python ppo.py
-    ```
-* `poetry shell`:
-    First, activate the poetry's virtual environment by executing `poetry shell`. Then, the name of the poetry's
-    virtual environment (e.g. `(cleanrl-ghSZGHE3-py3.9)`) should appear in the left side of your shell.
-    Afterwards, you can directly run
-    ```bash
-    (cleanrl-ghSZGHE3-py3.9) python ppo.py
-    ```
-
-
-## Pre-commit utilities
-
-We use [pre-commit](https://pre-commit.com/) to helps us automate a sequence of short tasks (called pre-commit "hooks") such as code formatting. In particular, we always use the following hooks when submitting code to the main repository.
-
-* [**pyupgrade**](https://github.com/asottile/pyupgrade): pyupgrade upgrades syntax for newer versions of the language. 
-* [**isort**](https://github.com/PyCQA/isort): isort sorts imported dependencies according to their type (e.g, standard library vs third-party library) and name.
-* [**black**](https://black.readthedocs.io/en/stable/): black enforces an uniform code style across the codebase.
-* [**autoflake**](https://github.com/PyCQA/autoflake): autoflake helps remove unused imports and variables.
-* [**codespell**](https://github.com/codespell-project/codespell): codespell helps avoid common incorrect spelling.
-
-You can run the following command to run the following hooks:
-
-```bash
-poetry run pre-commit run --all-files
-```
-
-which in most cases should automatically fix things as shown below: 
-
-![](docs/static/pre-commit.png)
-
-## Contributing new algorithms
-
-We welcome the contributions of new algorithms.
-
-**Before opening a pull request**, please open an issue first to discuss with us since this is likely a sizable effort. Once we agree on the plan, feel free to make a PR to include the new algorithm.
-
-To help ease the review process, here is a checklist template when contributing a new algorithm. See https://github.com/vwxyzjn/cleanrl/pull/137 as an example.
-
-- [ ] I've read the [CONTRIBUTION](https://github.com/vwxyzjn/cleanrl/blob/master/CONTRIBUTING.md) guide (**required**).
-- [ ] I have ensured `pre-commit run --all-files` passes (**required**).
-- [ ] I have contacted @vwxyzjn to obtain access to the [openrlbenchmark W&B team](https://wandb.ai/openrlbenchmark) (**required**).
-- [ ] I have tracked applicable experiments in [openrlbenchmark/cleanrl](https://wandb.ai/openrlbenchmark/cleanrl) with `--capture-video` flag toggled on (**required**).
-- [ ] I have updated the documentation and previewed the changes via `mkdocs serve`.
-    - [ ] I have explained note-worthy implementation details.
-    - [ ] I have explained the logged metrics.
-    - [ ] I have added links to the original paper and related papers (if applicable).
-    - [ ] I have added links to the PR related to the algorithm.
-    - [ ] I have created a table comparing my results against those from reputable sources (i.e., the original paper or other reference implementation).
-    - [ ] I have added the learning curves (in PNG format with `width=500` and `height=300`).
-    - [ ] I have added links to the tracked experiments.
-- [ ] I have updated the tests accordingly (if applicable).
+## Please check out https://docs.cleanrl.dev/contribution/ for more detail.