Changes to support Gym 0.26.0

[Markus28]

Generally, I am trying to keep compatibility with Gym < 0.26.0 and envpool.

• Vector environments will accept environments complying either with the old or the new step API and behave analogously.
• Replay buffers now contain two additional fields: terminated and truncated. The done field does no longer have to be specified when data is added to the buffer.
• Collectors will work with environments complying either with the old or the new step API. If the environment uses the old API, the collector will infer truncated from info and set terminated = done and not truncated before adding the transition to the replay buffer.
• The preprocess_fn accepts either done or terminated and truncated.
• Everything should work exactly as before with old-style environments, due to the way we infer terminated

This should not break any user code, unless the user is directly accessing the replay buffer. In particular, I didn't need to modify any of the training tests.

[Markus28]

• I believe the PEP8 check is failing due to an update of flake8 (add exception or fix the problem?)
• Tests are failing due to changes in PettingZoo (additional PR for this?)
• I disabled wandb for now because they aren't supporting Gym 0.26.0 yet. This should be probably be turned on once wandb fixes it
• It may be prudent to add a CI run with Gym==0.23.0.

[codecov-commenter]

Codecov Report

Merging #748 (04fcf7b) into master (278c91a) will decrease coverage by 2.75%.
The diff coverage is 87.80%.

@@            Coverage Diff             @@
##           master     #748      +/-   ##
==========================================
- Coverage   93.42%   90.66%   -2.76%     
==========================================
  Files          72       70       -2     
  Lines        4895     4917      +22     
==========================================
- Hits         4573     4458     -115     
- Misses        322      459     +137     

Flag	Coverage Δ
unittests	90.66% <87.80%> (-2.76%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
tianshou/data/collector.py	90.63% <61.53%> (-3.61%)	⬇️
tianshou/data/buffer/base.py	98.93% <100.00%> (+0.02%)	⬆️
tianshou/data/buffer/cached.py	100.00% <100.00%> (ø)
tianshou/data/buffer/manager.py	84.05% <100.00%> (+0.35%)	⬆️
tianshou/env/utils.py	83.33% <100.00%> (+5.55%)	⬆️
tianshou/env/venv_wrappers.py	83.60% <100.00%> (+0.27%)	⬆️
tianshou/env/venvs.py	94.15% <100.00%> (+0.07%)	⬆️
tianshou/env/worker/base.py	65.45% <100.00%> (+0.63%)	⬆️
tianshou/env/worker/dummy.py	85.29% <100.00%> (+8.82%)	⬆️
tianshou/env/worker/ray.py	83.33% <100.00%> (+6.73%)	⬆️
... and 21 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[Trinkle23897]

Hey @Markus28 , I adjust GPU test to gym==0.25.2, but it seems some of the tests are failed to pass. Could you please fix those?

[Markus28]

Thanks! I think the issue was caused by _SetAttrWrapper. In Gym 0.25, wrappers automatically convert environments to the old step API. Thus, RayVectorEnv will always follow the old step API in Gym 0.25, even if the underlying environment uses the new API. I changed _SetAttrWrapper to pass the right settings to the super constructor, depending on the API of the underlying environment. It's somewhat inconvenient, but I don't see a better way without getting rid of _SetAttrWrapper entirely. The test in question is now passing locally for me