Add Gymnasium support

[araffin]

Description

Like #780 but with gymnasium as default and compat wrapper on (would close #871 too).

Installation:

pip install git+https://github.com/DLR-RM/stable-baselines3@feat/gymnasium-support
pip install git+https://github.com/Stable-Baselines-Team/stable-baselines3-contrib@feat/gymnasium-support

Documentation is available here: https://stable-baselines3.readthedocs.io/en/feat-gymnasium-support/

There is an alpha version on pypi:

# sb3 contrib installs sb3 too
pip install "sb3_contrib>=2.0.0a1" --upgrade
pip install "stable_baselines3>=2.0.0a1" --upgrade

Some new issues:

• typing annotations for wrappers is broken (since Update the type hinting for core.py Farama-Foundation/Gymnasium#39), not clear what should be the new annotations as gymnasium doesn't even do it for its own wrappers, should be fixed by Fix the wrapper type hints Farama-Foundation/Gymnasium#337
• warnings when importing gymnasium, should be fixed by shimmy 0.2.1
• slower import (see [Bug Report] Jax, pytorch and dependency imports when importing gymnasium Farama-Foundation/Gymnasium#314), should be fixed in next Gymnasium release
• shimmy doesn't allow gym 0.21 but should (Add check when registering gym compatibility envs to avoid gymnasium warning Farama-Foundation/Shimmy#29), should be fixed in shimmy 0.2.1
• test fails when checking for discrete env because of Change discrete dtype to np.int64 Farama-Foundation/Gymnasium#141
• spec behavior is broken (and would break RL Zoo too), see Add Gymnasium support #1327 (comment)

Missing:

• update changelog
• update notebooks (rename imports as they should already work with this version)
• update tutorials
• Cleanup GymStep26Return

Related but due to pygame: pygame/pygame#3572

Motivation and Context

• I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (update in the documentation)

closes #840 #871
closes #271
closes #1156
closes #1324
closes #1406
closes #1315

Checklist

• I've read the CONTRIBUTION guide (required)
• I have updated the changelog accordingly (required).
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.
• I have opened an associated PR on the SB3-Contrib repository (if necessary)
• I have opened an associated PR on the RL-Zoo3 repository (if necessary)
• I have reformatted the code using make format (required)
• I have checked the codestyle using make check-codestyle and make lint (required)
• I have ensured make pytest and make type both pass. (required)
• I have checked that the documentation builds using make doc (required)

Note: You can run most of the checks using make commit-checks.

Note: we are using a maximum length of 127 characters per line

[jkterry1]

Just so I know- what's the plan for making a release that includes this PR?

[araffin]

Just so I know- what's the plan for making a release that includes this PR?

Alpha release is already on pypi since 15 days, stable release will be in some weeks if everything is fine.