Brains

It is software that is made to be run on a Laika the Dog — robodog for household chores. The robot listens for a wake phrase (e.g. "Hey Laika") and can understand natural commands through speech recognition and GPT-4, responding with a synthesized voice using ElevenLabs. Using an Intel RealSense camera and computer vision algorithms, it can locate objects in 3D space, approach them, and manipulate them using an Interbotix robotic arm.

laika-leaves-collection-small.mp4

Prerequisites

Prerequisites consist of hardware and packages that should be installed on this hardware.

Hardware

• Quadruped: Unitree Go2, cheapest version can work if you will jailbreak it
• Arm: any Trossen Robotics arm, we had WidowX 250 S
• Camera: Intel RealSense depth camera, in our case, it is D435i
• Computer (if your version of Go2 is not EDU then you need one): some NVIDIA device, we have Jetson Orin Nano
• Speaker: we used JBL lol
• Microphone
• DC/DC converter
• Optionally you can print the body which we designed to glue everything nicely

Software

You have two options: work within Docker or install dependencies on bare metal.

Docker

Build image:

sudo docker buildx build --tag brains .

Run it:

sudo docker run -it --tty --name brains --rm \
    --env OPENAI_API_KEY=${OPENAI_API_KEY} \
    --env ELEVEN_API_KEY=${ELEVEN_API_KEY} \
    brains

You will enter tmux environment with everything configured. You might have to forward devices inside the container so they can be seen from the inside.

Bare Metal

• ROS 2, we used Humble
• ROS 2 package for the arm
• ROS 2 package for the camera
• Optionally you can train your own wake word model using openWakeWord
• Poetry
• This package (poetry install)

How to Run

Launch arm and camera ROS nodes:

ros2 launch interbotix_xsarm_control xsarm_control.launch.py robot_model:=wx250
ros2 launch realsense2_camera rs_launch.py

Make sure you have OPENAI_API_KEY and ELEVEN_API_KEY environment variables, and run the service via:

poetry run python app.py

It is possible to control script parameters via environment variables listed in Args class, for example:

openai_temperature=0.7 poetry run python app.py