Changes in DEP-RL library after myosuite Update

[YusseffRuiz]

After the last myosuite Update, the Tutorials for training a new policy for MyoSuite locomotion in MyoLeg, and i'm thinking in all other environments, there are a few changes needed to be performed so it can work fine:

Reproduce the Issue:
Install Myosuite.
Prepare Tutorial based on the following:
https://myosuite.readthedocs.io/en/latest/tutorials.html
"DEP-RL baselines"
And Training file with the following:
https://deprl.readthedocs.io/en/latest/loading_checkpoints.html#continuing-training-with-deprl

A new environment is required to be registered in init file in site-packages/deprl/

Afterwards, two error pops out:

Expected float and received Double.

Which can be solved by modifying every torch and arrays of the type float32, and changing them to "double" in the following files inside:
deprl/custom_distributed

Then, an issue with compatibility with the new gym update appears, since "step" and "reset" returns 5 values, instead of 4.

We need to change the following in the file:
deprl/env_wrappers/wrappers

From:

     def step(self, action):
        try:
            observation, reward, done, info= self._inner_step(action)
...

    def reset(self, **kwargs):
        observation = super().reset(**kwargs)

To

    def step(self, action):
        try:
            observation, reward, done, info, extras = self._inner_step(action)

       def reset(self, **kwargs):
        observation, *_ = super().reset(**kwargs)

After these changes, the repository works correctly with different environments.

[P-Schumacher]

Thanks for the helpful suggestions :)
I have implemented some updates in the deprl repo and the current state of the issue is being tracked here:
martius-lab/depRL#6