HippoGym is a Python library designed for researchers and students focusing on human-AI interaction over the web. It simplifies the setup, execution, data collection, and management of experiments by providing an easy-to-use interface for OpenAI Gym and supports custom built environments.
The Websocket API by Amazon Web Services (AWS) provides the backbone for real-time, bi-directional communication between the client and the server in HippoGym. Specifically, it facilitates seamless interaction between the frontend, written in React, and the backend trial.py logic. The WebSocket class, part of the backend architecture, abstracts away the complexity of Websocket communications, allowing users to focus on human-AI interaction research.
In HippoGym, Websocket API enables:
- Real-time updates: For example, rendering frames in OpenAI Gym environments.
- Data Exchange: Messages containing experiment data are exchanged between the frontend and backend.
- Data saving: Enabling the real-time saving of step/trial data, which can then be fed into an external learning algorithm for further use.
With Websockets, you can easily send messages from the frontend to the backend and vice versa without worrying about the underlying communication protocols.
The architecture operates in the following sequence:
- Front-end Deployment: Projects are deployed using React, then it can be accessed on the web for a user to connect to.
- Backend Connection: Users run their custom
trial.pyafter reading through the step files and before you enter the game page. - AWS Lambda: The backendConnectionID is held in AWS Lambda server awaiting for the frontend to map to the backend.
- Front-end Launch: On game launch, the frontend connects to the WebSocket server, sends a unique userID and sends the request to map the user with the backend.
- UserID Storage: AWS Lambda stores the userID in a DynamoDB table for unique mapping.
This architecture ensures unique mapping and seamless communication between the frontend and backend. See below to get a brief overview of the current system design for connecting a client:
An example of how users appear on DynamoDB is shown:
Sometimes, clashes arise - please see The Concurrent Connection Clash Problem to learn how to avoid this problem.
Route keys serve as a routing mechanism in AWS Websocket API. A message must specify a route key to route it to the correct Lambda function.
Example data:
{
"routeKey": "save",
"data":
{
'timestep': 0,
'state': 1,
'action': 'left'
}
}Note routeKey is case sensitive.
Here are the current route keys we have available:
When sending a message from either frontend or backend, we must specify a route key from this list, along with any type of data we wish to send over. For more on sending and receiving messages, see Setting Up.
We use AWS Lambda as middleware for routing our Websocket messages to the relevant service. Lambda detects which user is sending a message, and where its intended destination is, that be the frontend or backend. The displayed route keys above route to our Lambda Websocket handler in which the Lambda reads the route key and the target endpoint.
Let’s dive into our React code to get a brief overview of how we setup the frontend. In this use case, we will be providing code snippets used for the Mountain Car environment from OpenAI Gym.
If we wish to make adjustments to the frontend, we can enter the HIPPO_Gym_FrontEnd_React repo and clone it as usual. Once we are in the repository, we run npm install to install the dependencies required for the frontend to work. We can navigate to the game.js file found in HIPPO_Gym_FrontEnd_React/src/components/Game/game.js. For those curious, it is possible to do a stack trace from main.js to explore all components of the frontend. In game.js, most if not all of the code is under the main class of Game. Here, we define our initial states, components, props and more of what React has to offer. Notice how one of the methods inside this Game class defines how we send messages to our Websocket. The exact code can be viewed here:
// Send data to websocket server in JSON format
sendMessage = (routeKey, data) => {
// action is added to indicate the routeKey for the backend.
if (this.state.isConnection){
this.websocket.send(JSON.stringify({
action: routeKey,
userId: USER_ID,
sendTo: "backend",
...data
}));
}
};For those unfamiliar with hashmaps, don’t worry about the inner workings; but rather understand that every time we reference this function inside our game.js, we reference it as this.sendMessage(“routeKey”, {data: wishToSend}. An example usage for this is shown bellow, where we send a message under the “start” routeKey to trigger the userID dynamoDB update process. We first define our routeKey = “start”, and then use a hashmap to send whatever form of data we want (we chose userID and projectID strings here). This part is standard start in all experiments and is already integrated in the boilerplate code. We typically run this code after we establish a valid connection to our Websocket server.
this.sendMessage("start",{
userId: USER_ID,
projectId: PROJECT_ID,
});After we run the code above, the expected output we receive is as follows: {'userId': <userID>, 'projectId': <projectID>} and the raw json gets parsed through our Lambda handler and notifies the backend of our userID and projectID for the given experiment.
Now that we have gotten an example of how to send messages to our Websocket, let's look at how we can receive expected messages as well. With every experiment, we want to render our UI for the frontend to visualize a dynamic page. Usually, our backend handles rendering the frames/displaying some sort of UI, whether that is passing in components for button data, or full scale environment npArray render frames provided from OpenAI Gym. See the example code below:
this.websocket.onmessage = (message) => {
if (message.data === "done") {
//"done" means the game has ended
this.setState({
isEnd: true,
gameEndVisible: true,
});
} else {
//parse the data from the websocket server
let parsedData = JSON.parse(message.data);
//Check if frame related information in ws message
if (parsedData.env.frame && parsedData.env.frameId) {
let frame = parsedData.env.frame;
let frameId = parsedData.env.frameId;
// **Optionally set a new border color
if ("borderColor" in parsedData) {
this.setState({
borderColor: parsedData.borderColor});
}
this.setState((prevState) => ({
// set new frame ID
frameSrc: "data:image/jpeg;base64, " + frame,
frameCount: prevState.frameCount + 1,
frameId: frameId,
}))
}
}
}In the code above, we listen to any incoming Websocket messages and parse the data from the message. If the Websocket message sent from our backend contains the data:
{'routeKey' = 'UI',
{'env':
{'frame': <frameData>,
'frameId': <frameId>
}
}
}then the code above will be executed. Notice how we use the routeKey here as UI, and the data we send can be anything, or in this case; a nested dictionary. Also, note that we check the Websocket to see if the Websocket message has the keyword done - that is the logic to end the game.
We have now fully gone over how the frontend sends and receives data from a backend client now. We will explore more on how the backend sends this env frame data to the frontend.
Now it's time to look into the backend; the heart of the experiment lies in the trial.py file.
The file defines a Trial class to run an experimental trial that interacts with a WebSocket server through websocket.py and an environment provided by an agent.py Agent class.
It also has functionalities such as configuring the trial tailored to a users needs, handling incoming messages, and extracting data.
- Clone the Repository
git clone https://github.com/IRLL/HIPPO_Gym.git
- Install Dependencies
pip install -r requirements.txt
- Navigate to Directory
cd App - Run the Program
python3 trial.py
trial.py is the most important aspect of HippoGym, as it allows any researcher/student to custom tailor their experiment to exactly how they desire.
Usually, in the __init__ method is where researchers specify their preferred experiment configuration:
class Trial():
def __init__(self):
print(f'{TAG} Initializing Trial...')
self.config = load_config() # Ability to load trial logic from external config
self.trialData = None
self.data = None
self.count = 1
self.websocket = Websocket() # If you wish to specify your own websocket server use it as param
self.episode = 0
self.done = False
self.play = False
self.nextEntry = {}
self.trialId = shortuuid.uuid()
self.outfile = None
self.userId = None
self.projectId = self.config.get('projectId')
self.show_demo = None
self.total_reward = 0
self.demo_idx = 1
self.action = 'noop'
self.modality = self.config.get('modality')
self.framerate = self.config.get('startingFrameRate', 30)
self.frameId = 0If we are using HippoGym for Mountain Car Environment specifically, the trial.py will be mostly configured for us.
Let's take a look at how we send messages using our abstracted Websocket class. Remember the frontend setup where we expected to receive some environment renders? Well, this is how we are sending it in the backend:
async def send_render(self, render:dict):
await self.websocket.sendData('UI', {'env':render})We utilize this websocket.sendData() method that is defined as follows from our websocket.py code, although once again we do not need to concern ourselves of the inner-workings:
async def sendData(self, routeKey, data):
if self.websocket is not None:
ws_data = {"action": routeKey, "userId": self.userID, "sendTo": "frontend"}
ws_data.update(data)
await self.websocket.send(json.dumps(action_data))so the data format is sent to the frontend as follows:
{'routeKey' = 'UI',
{'env':
{'frame': <frameData>,
'frameId': <frameId>
}
}
}exactly how we intend.
Now, to receive messages from the Websocket, we continuously listen for messages while the trial is not marked as done, and send messages to a function to parse the data accordingly.
async def run(self):
'''
This is the main event controlling function for a Trial.
It handles the render-step loop
'''
while not self.done:
message = await self.websocket.receiveData()
# Message is received from Websocket in JSON Format
await self.handle_message(message)In the handle_message function, we have the flexibility to define multiple approaches to processing the received messages. If curious about the backend configuration, particularly the setup of agent.py and websocket.py, the documented source code provides comprehensive insights for further exploration.
The Concurrent Connection Collision (CCC) Problem can can arise from the manner in which AWS Lambda handles connections. The CCC Problem emerges when users initiate backend and frontend connections in rapid succession. If a subsequent user's frontend connection occurs before the previous user's frontend connection, unintended session merging can happen.
In the case where we have 2 users wanting to connect to a trial at the same given time, if user 1 runs trial.py in their backend and user 2 connects to the frontend right after, then user 2's trial.py will be voided and connect to user 1's backend instead. This unintended merging can lead to disruptions in loading and communication.
This behavior is attributed to AWS Lambda, which holds backend connections until the corresponding frontend connects. If a newer backend connection arrives before the intended frontend, the Lambda prioritizes the most recent backend connection, causing unexpected results.
To prevent the CCC Problem, it's advised that users connect their backend and frontend components in quick succession, ideally right before the frontend loads. By ensuring the backend connection precedes the frontend connection, the likelihood of timing-related clashes is minimized.
Of course, we always have the option to refresh the page and retry again with no loss.
Let's get started on an example of using OpenAI's Mountain Car Environment with HippoGym provided by Callie Muslimani.
If we navigate to the trial.py file, we will see its already configured to run Mountain Car. Here is how we can run it:
-
Clone the Repository
git clone https://github.com/IRLL/HIPPO_Gym.git
-
Install Dependencies
pip install -r requirements.txt
-
Navigate to Directory
cd App -
Connect to site hosting trial Connect to the testing.irll.net site and get through the consent form.
Press the 'submit' button.
-
Run the Program While waiting for the game to load, run the trial.py to avoid the CCC problem.
python3 trial.py
The game will now be running the Mountain Car environment.
Example demo:
mountaincar-demo-github.mov
Example console output:
Other human trial demos:
Originally written by Nick Nissen and Yuan Wang
System upgrades and redesign by Mohamed Al-Nassirat and Vera Maoued
Supervised by Matt Taylor For the Intelligent Robot Learning Laboratory (IRLL) at the University of Alberta (UofA) Supported by the Alberta Machine Intelligence Institute (AMII)