This project will focus primarily on the implementation and benchmark of different approaches to domain and task transfer learning in reinforcement learning. The focus lies on a diverse set of simplified domestic robot tasks using ai2thor, a realistic household 3D environment. To provide an example, an agent could learn to pick up a cup under particular conditions and then zero/few shot transfer to pick up many different cups in many different situations.
We included our own wrapper for the environment as well to support the modification of the tasks within an openAI gym interface, so that new and more complex tasks can be developed efficiently to train and test the agent.
We have begun a long-running blog series which will go into more detail about this repo and how to use all of the features of our wrapper.
- From Cups to Consciousness (Part 1): How are cups related to intelligence?
- From Cups to Consciousness (Part 2): From simulation to the real world
Currently using ai2thor version 0.0.44 and up. More detailed information on ai2thor environment can be found on their website.
A3C agent learning during training on NaturalLanguagePickUpMultipleObjectTask in one of our customized scenes and tasks with the target object being CUPS!
This project will include implementations and adaptations of the following papers as a benchmark of the current state of the art approaches to the problem:
- Ikostrikov's A3C
- Gated-Attention Architectures for Task-Oriented Language Grounding -- Original code available on DeepRL-Grounding based on Ikostrikov's A3C
- Rainbow: Combining Improvements in Deep Reinforcement Learning -- Original code available on Rainbow from Kaixhin
Implementations of these can be found in the algorithms folder and can be run on AI2ThorEnv with:
python algorithms/a3c/main.py
python algorithms/rainbow/main.py
Check the argparse help for more details and variations of running the algorithm with different hyperparams and on the atari environment as well.
Clone cups-rl repository:
# CUPS_RL=/path/to/clone/cups-rl
git clone https://github.com/TheMTank/cups-rl.git $CUPS_RL
Install Python dependencies (Currently only supporting python 3.5+):
pip install -r $CUPS_RL/requirements.txt
Finally, add CUPS_RL to your PYTHONPATH environment variable and you are done.
The wrapper is based on OpenAI gym interfaces as described in gym documentation. Here is a simple example with the default configuration, that will place the agent in a "Kitchen" for the task of picking up and putting down mugs.
from gym_ai2thor.envs.ai2thor_env import AI2ThorEnv
N_EPISODES = 20
env = AI2ThorEnv()
max_episode_length = env.task.max_episode_length
for episode in range(N_EPISODES):
state = env.reset()
for step_num in range(max_episode_length):
action = env.action_space.sample()
state, reward, done, info = env.step(action)
if done:
break
The environment is typically defined by a JSON configuration file located on the gym_ai2thor/config_files
folder. You can find a full example at config_example.json to see how to customize it. Here there is
another one as well:
# gym_ai2thor/config_files/myconfig.json
{'pickup_put_interaction': True,
'open_close_interaction': true,
'pickup_objects': ['Mug', 'Apple', 'Book'],
'acceptable_receptacles': ['CounterTop', 'TableTop', 'Sink'],
'openable_objects': ['Microwave'],
'scene_id': 'FloorPlan28',
'gridSize': 0.1,
'continuous_movement': true,
'grayscale': True,
'resolution': (300, 300),
'task': {'task_name': 'PickUp',
'target_object': {"Mug": 1}}}
For experimentation it is important to be able to make slight modifications of the environment
without having to create a new config file each time. The class AI2ThorEnv includes the keyword
argument config_dict, that allows to input a python dictionary in addition to the config file
that overrides the parameters described in the config.
The tasks are defined in gym_ai2thor/tasks.py and allow for particular configurations regarding the
rewards given and termination conditions for an episode. You can use the tasks that we defined
there or create your own by adding it as a subclass of BaseTask.
Here an example of a new task definition:
# gym_ai2thor/tasks.py
class MoveAheadTask(BaseTask):
def __init__(self, *args, **kwargs):
super().__init__(kwargs)
self.rewards = []
def transition_reward(self, state):
reward = 1 if state.metadata['lastAction'] == 'MoveAhead' else -1
self.rewards.append(reward)
done = sum(rewards) > 100 or self.step_num > self.max_episode_length
if done:
self.rewards = []
return reward, done
def reset(self):
self.step_num = 0
We encourage you to explore the scripts on the examples folder to guide you on the wrapper
functionalities and explore how to create more customized versions of ai2thor environments and
tasks. It is possible for the agent to do continuous rotation with 10 degrees by setting
continuous_movement: True in the config as well, see task_on_ground_variation.py in examples.
In the config build_file_name can be set to a file/folder combination within gym_ai2thor/build_files.
We provide a preliminary unity build that you can download from Google Drive here
but of course you can create your own by following the instructions on the ai2thor repository. We will be adding more builds in the future.
Here is the desired result of an example task in which the goal of the agent is to place a cup in the sink.
Example of task "place cup in sink"
MTank is a non-partisan organisation that works solely to recognise the multifaceted nature of Artificial Intelligence research and to highlight key developments within all sectors affected by these advancements. Through the creation of unique resources, the combination of ideas and their provision to the public, this project hopes to encourage the dialogue which is beginning to take place globally.
To produce value for the individual, for researchers, for institutions and for the world.
This project is released under the MIT license.