How to disable preprocessing for a policy?

[houcharlie]

What is your question?

I have a few learned policies as well as one hard-coded policy. I instantiated the hard-coded policy using the Policy interface. I'd like to know how to disable the observation preprocessing for this policy only; couldn't find anything for this in the docs.

For instance, in the rock-paper-scissors example, the observations are automatically converted to one-hot-encodings. https://github.com/ray-project/ray/blob/master/rllib/examples/policy/rock_paper_scissors_dummies.py

But I'd like to be able to operate directly on the states 0,1,2 for rock, paper scissors.

[houcharlie]

Answer: you can make the state space for the hard-coded policy be a gym.spaces.Box regardless of whether the state space is actually continuous or not; rllib does not perform any preprocessing for Box spaces. This is a pretty hacky solution though, would be nice to see a cleaner one.

[sven1977]

Got you. Yes, setting preprocessing to off shouldn't even do one-hot'ing anymore. I'll take a look.

[andras-kth]

Has this been addressed in a non-hacky way?
Why is automatic preprocessing turned on by default, anyway?

I see a preprocessor_pref in TrainerConfDict, but based on the rather sparse documentation,
only 2 values are (or, may be?) supported: "deepmind" or "rllib". Would setting this to None
turn off preprocessing? I guess I better test...
Setting this to None doesn't appear to accomplish anything...

Maybe "NoPreprocessor" might help...

ray/rllib/agents/trainer.py

Line 1003 in 7916500

if type(pp).__name__ != "NoPreprocessor":

NOPE...

Also tried custom_preprocessor, but NoPreprocessor doesn't appear to be registered;
plus, there's a deprecation warning:

2021-09-10 09:25:46,996	WARNING catalog.py:695 -- DeprecationWarning: Custom preprocessors are deprecated, since they sometimes conflict with the built-in preprocessors for handling complex observation spaces. Please use wrapper classes around your environment instead of preprocessors.

Hmm... it looks like this is being worked on, but not quite ready for prime time:

• [RLlib] POC: Run RLlib w/o Preprocessors setup. #17656
• [RLlib] No Preprocessors; preparatory PR #1 #18367
• [RLlib] No Preprocessors (part 2). #18468

[andras-kth]

In the end, the custom_preprocessor approach worked. Eventually, deprecation may force the use of a wrapper.

Just in case anyone finds it useful, here's a snippet that transforms boolean vector observations from
the original gym.spaces.Tuple((gym.spaces.Discrete(2),)*dimension) to gym.spaces.Box(0, 1, (dimension,), dtype=int):

import gym
import numpy as np
from ray import rllib

class BooleanVectorPreprocessor(rllib.models.preprocessors.Preprocessor):
    def _init_shape(self, observation_space, options=None):
        return (len(observation_space.spaces),)

    def transform(self, observation):
        return np.array(observation)
    
    @property
    def observation_space(self):
        space = gym.spaces.Box(0, 1, self.shape, dtype='int8')
        space.original_space = self._obs_space
        return space

rllib.models.ModelCatalog.register_custom_preprocessor('boolean_vector', BooleanVectorPreprocessor)

[ArturNiederfahrenhorst]

Thanks for adding to this @andras-kth !