5_custom_gym_env.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "5.custom_gym_env.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/barrtender/euchre/blob/main/5_custom_gym_env.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AoxOjIlOImwx"
      },
      "source": [
        "# Following Stable Baselines Tutorial - Creating a custom Gym environment\n",
        "\n",
        "Originally at: https://stable-baselines.readthedocs.io/en/master/guide/custom_env.html\n",
        "\n",
        "But then I got errors and realized that stable-baselines3 was the thing to use so Frankenstein'd it together with https://stable-baselines3.readthedocs.io/en/master/guide/custom_env.html (note the 3 in the URL).\n",
        "\n",
        "The goal was to get a colab up that I can put a custom environment into and train easily.\n",
        "\n",
        "I wanted to validate with the simplest custom environment I could find, so \"Go Left\" seemed appropriate.\n",
        "\n",
        "The idea is that the game is to \"Go Left\". It's a 1-dimensional game that starts the agent off on the right side of a line and puts the goal on the left side of the line. The game gives two options \"go right\" and \"go left\". The agent has 50 moves to find the exit. If the agent makes it to the goal they get a positive reward. If they don't, they get a negative reward.\n",
        "\n",
        "I found that without the negative reward or higher reward for finishing early the models never figured out how to win."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Sp8rSS4DIhEV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ee9ddc63-16cc-4354-977b-848ecc1a3af4"
      },
      "source": [
        "## Install Dependencies and Stable Baselines Using Pip\n",
        "\n",
        "%tensorflow_version 2.x\n",
        "!pip install stable-baselines3>=2.0.0\n",
        "!pip install sb3-contrib\n",
        "!pip install 'shimmy>=0.2.1'"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Colab only includes TensorFlow 2.x; %tensorflow_version has no effect.\n",
            "Collecting sb3-contrib\n",
            "  Downloading sb3_contrib-2.1.0-py3-none-any.whl (80 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m80.3/80.3 kB\u001b[0m \u001b[31m2.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: stable-baselines3>=2.1.0 in /usr/local/lib/python3.10/dist-packages (from sb3-contrib) (2.1.0)\n",
            "Requirement already satisfied: gymnasium<0.30,>=0.28.1 in /usr/local/lib/python3.10/dist-packages (from stable-baselines3>=2.1.0->sb3-contrib) (0.29.1)\n",
            "Requirement already satisfied: numpy>=1.20 in /usr/local/lib/python3.10/dist-packages (from stable-baselines3>=2.1.0->sb3-contrib) (1.23.5)\n",
            "Requirement already satisfied: torch>=1.13 in /usr/local/lib/python3.10/dist-packages (from stable-baselines3>=2.1.0->sb3-contrib) (2.0.1+cu118)\n",
            "Requirement already satisfied: cloudpickle in /usr/local/lib/python3.10/dist-packages (from stable-baselines3>=2.1.0->sb3-contrib) (2.2.1)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from stable-baselines3>=2.1.0->sb3-contrib) (1.5.3)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from stable-baselines3>=2.1.0->sb3-contrib) (3.7.1)\n",
            "Requirement already satisfied: typing-extensions>=4.3.0 in /usr/local/lib/python3.10/dist-packages (from gymnasium<0.30,>=0.28.1->stable-baselines3>=2.1.0->sb3-contrib) (4.5.0)\n",
            "Requirement already satisfied: farama-notifications>=0.0.1 in /usr/local/lib/python3.10/dist-packages (from gymnasium<0.30,>=0.28.1->stable-baselines3>=2.1.0->sb3-contrib) (0.0.4)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=1.13->stable-baselines3>=2.1.0->sb3-contrib) (3.12.2)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.13->stable-baselines3>=2.1.0->sb3-contrib) (1.12)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.13->stable-baselines3>=2.1.0->sb3-contrib) (3.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13->stable-baselines3>=2.1.0->sb3-contrib) (3.1.2)\n",
            "Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13->stable-baselines3>=2.1.0->sb3-contrib) (2.0.0)\n",
            "Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.13->stable-baselines3>=2.1.0->sb3-contrib) (3.27.4.1)\n",
            "Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.13->stable-baselines3>=2.1.0->sb3-contrib) (16.0.6)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->stable-baselines3>=2.1.0->sb3-contrib) (1.1.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->stable-baselines3>=2.1.0->sb3-contrib) (0.11.0)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->stable-baselines3>=2.1.0->sb3-contrib) (4.42.1)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->stable-baselines3>=2.1.0->sb3-contrib) (1.4.5)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->stable-baselines3>=2.1.0->sb3-contrib) (23.1)\n",
            "Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->stable-baselines3>=2.1.0->sb3-contrib) (9.4.0)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->stable-baselines3>=2.1.0->sb3-contrib) (3.1.1)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib->stable-baselines3>=2.1.0->sb3-contrib) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->stable-baselines3>=2.1.0->sb3-contrib) (2023.3.post1)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib->stable-baselines3>=2.1.0->sb3-contrib) (1.16.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.13->stable-baselines3>=2.1.0->sb3-contrib) (2.1.3)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.13->stable-baselines3>=2.1.0->sb3-contrib) (1.3.0)\n",
            "Installing collected packages: sb3-contrib\n",
            "Successfully installed sb3-contrib-2.1.0\n",
            "Collecting shimmy>=0.2.1\n",
            "  Downloading Shimmy-1.2.1-py3-none-any.whl (37 kB)\n",
            "Requirement already satisfied: numpy>=1.18.0 in /usr/local/lib/python3.10/dist-packages (from shimmy>=0.2.1) (1.23.5)\n",
            "Requirement already satisfied: gymnasium>=0.27.0 in /usr/local/lib/python3.10/dist-packages (from shimmy>=0.2.1) (0.29.1)\n",
            "Requirement already satisfied: cloudpickle>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from gymnasium>=0.27.0->shimmy>=0.2.1) (2.2.1)\n",
            "Requirement already satisfied: typing-extensions>=4.3.0 in /usr/local/lib/python3.10/dist-packages (from gymnasium>=0.27.0->shimmy>=0.2.1) (4.5.0)\n",
            "Requirement already satisfied: farama-notifications>=0.0.1 in /usr/local/lib/python3.10/dist-packages (from gymnasium>=0.27.0->shimmy>=0.2.1) (0.0.4)\n",
            "Installing collected packages: shimmy\n",
            "Successfully installed shimmy-1.2.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rzevZcgmJmhi"
      },
      "source": [
        "## First steps with the gym interface (verbatim copy/pasted)\n",
        "\n",
        "As you have noticed in the previous notebooks, an environment that follows the gym interface is quite simple to use.\n",
        "It provides to this user mainly three methods:\n",
        "- `reset()` called at the beginning of an episode, it returns an observation\n",
        "- `step(action)` called to take an action with the environment, it returns the next observation, the immediate reward, whether the episode is over and additional information\n",
        "- (Optional) `render(method='human')` which allow to visualize the agent in action. Note that graphical interface does not work on google colab, so we cannot use it directly (we have to rely on `method='rbg_array'` to retrieve an image of the scene\n",
        "\n",
        "Under the hood, it also contains two useful properties:\n",
        "- `observation_space` which one of the gym spaces (`Discrete`, `Box`, ...) and describe the type and shape of the observation\n",
        "- `action_space` which is also a gym space object that describes the action space, so the type of action that can be taken\n",
        "\n",
        "The best way to learn about gym spaces is to look at the [source code](https://github.com/openai/gym/tree/master/gym/spaces), but you need to know at least the main ones:\n",
        "- `gym.spaces.Box`: A (possibly unbounded) box in $R^n$. Specifically, a Box represents the Cartesian product of n closed intervals. Each interval has the form of one of [a, b], (-oo, b], [a, oo), or (-oo, oo). Example: A 1D-Vector or an image observation can be described with the Box space.\n",
        "```python\n",
        "# Example for using image as input:\n",
        "observation_space = spaces.Box(low=0, high=255, shape=(HEIGHT, WIDTH, N_CHANNELS), dtype=np.uint8)\n",
        "```                                       \n",
        "\n",
        "- `gym.spaces.Discrete`: A discrete space in $\\{ 0, 1, \\dots, n-1 \\}$\n",
        "  Example: if you have two actions (\"left\" and \"right\") you can represent your action space using `Discrete(2)`, the first action will be 0 and the second 1.\n",
        "\n",
        "\n",
        "\n",
        "[Documentation on custom env](https://stable-baselines.readthedocs.io/en/master/guide/custom_env.html)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RqxatIwPOXe_"
      },
      "source": [
        "##  Gym env skeleton\n",
        "\n",
        "> In practice this is how a gym environment looks like.\n",
        "Here, we have implemented a simple grid world were the agent must learn to go always left."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rYzDXA9vJfz1"
      },
      "source": [
        "import numpy as np\n",
        "import gymnasium\n",
        "from gymnasium import spaces\n",
        "\n",
        "\n",
        "class GoLeftEnv(gymnasium.Env):\n",
        "  \"\"\"\n",
        "  Custom Environment that follows gym interface.\n",
        "  This is a simple env where the agent must learn to go always left.\n",
        "  \"\"\"\n",
        "  # Because of google colab, we cannot implement the GUI ('human' render mode)\n",
        "  metadata = {'render.modes': ['console']}\n",
        "  # Define constants for clearer code\n",
        "  LEFT = 0\n",
        "  RIGHT = 1\n",
        "  MAX_TIME = 50\n",
        "  time_left = MAX_TIME\n",
        "\n",
        "  def __init__(self, grid_size=10):\n",
        "    super(GoLeftEnv, self).__init__()\n",
        "\n",
        "    # Size of the 1D-grid\n",
        "    self.grid_size = grid_size\n",
        "    # Initialize the agent at the right of the grid\n",
        "    self.agent_pos = grid_size - 1\n",
        "\n",
        "    # Define action and observation space\n",
        "    # They must be gym.spaces objects\n",
        "    # Example when using discrete actions, we have two: left and right\n",
        "    n_actions = 2\n",
        "    self.action_space = spaces.Discrete(n_actions)\n",
        "    # The observation will be the coordinate of the agent\n",
        "    # this can be described both by Discrete and Box space\n",
        "    self.observation_space = spaces.Box(low=0, high=self.grid_size,\n",
        "                                        shape=(1,), dtype=np.float32)\n",
        "\n",
        "  def reset(self, seed=1):\n",
        "    \"\"\"\n",
        "    Important: the observation must be a numpy array\n",
        "    :return: (np.array)\n",
        "    \"\"\"\n",
        "    self.time_left = self.MAX_TIME\n",
        "    # Initialize the agent at the right of the grid\n",
        "    self.agent_pos = self.grid_size - 1\n",
        "    # here we convert to float32 to make it more general (in case we want to use continuous actions)\n",
        "    return np.array([self.agent_pos]).astype(np.float32), {}\n",
        "\n",
        "  def step(self, action):\n",
        "    self.time_left -= 1\n",
        "    if action == self.LEFT:\n",
        "      self.agent_pos -= 1\n",
        "    elif action == self.RIGHT:\n",
        "      self.agent_pos += 1\n",
        "    else:\n",
        "      raise ValueError(\"Received invalid action={} which is not part of the action space\".format(action))\n",
        "\n",
        "    # Account for the boundaries of the grid\n",
        "    self.agent_pos = np.clip(self.agent_pos, 0, self.grid_size)\n",
        "\n",
        "    # Are we at the left of the grid?\n",
        "    done = bool(self.agent_pos == 0)\n",
        "\n",
        "    # Null reward everywhere except when reaching the goal (left of the grid)\n",
        "    reward = self.time_left if self.agent_pos == 0 else 0\n",
        "\n",
        "    if self.time_left <= 0:\n",
        "      done = True\n",
        "      reward = -self.MAX_TIME\n",
        "\n",
        "    # Optionally we can pass additional info, we are not using that for now\n",
        "    info = {}\n",
        "\n",
        "    return np.array([self.agent_pos]).astype(np.float32), reward, done, done, info\n",
        "\n",
        "  def render(self, mode='console'):\n",
        "    if mode != 'console':\n",
        "      raise NotImplementedError()\n",
        "    # agent is represented as a cross, rest as a dot\n",
        "    print(\".\" * self.agent_pos, end=\"\")\n",
        "    print(\"x\", end=\"\")\n",
        "    print(\".\" * (self.grid_size - self.agent_pos))\n",
        "\n",
        "  def close(self):\n",
        "    pass\n",
        ""
      ],
      "execution_count": 98,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zy5mlho1-Ine"
      },
      "source": [
        "### Validate the environment\n",
        "\n",
        "> Stable Baselines provides a [helper](https://stable-baselines.readthedocs.io/en/master/common/env_checker.html) to check that your environment follows the Gym interface. It also optionally checks that the environment is compatible with Stable-Baselines (and emits warning if necessary)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9DOpP_B0-LXm"
      },
      "source": [
        "from stable_baselines3.common.env_checker import check_env"
      ],
      "execution_count": 130,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1CcUVatq-P0l"
      },
      "source": [
        "env = GoLeftEnv()\n",
        "# If the environment don't follow the interface, an error will be thrown\n",
        "check_env(env, warn=True)"
      ],
      "execution_count": 131,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eJ3khFtkSE0g"
      },
      "source": [
        "### Testing the environment"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i62yf2LvSAYY",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f0f082ce-7db3-4b55-87ed-e463d38578e5"
      },
      "source": [
        "env = GoLeftEnv(grid_size=10)\n",
        "\n",
        "obs = env.reset(1)\n",
        "env.render()\n",
        "\n",
        "print(env.observation_space)\n",
        "print(env.action_space)\n",
        "print(env.action_space.sample())\n",
        "print(env.action_space.sample())\n",
        "print(env.action_space.sample())\n",
        "print(env.action_space.sample())\n",
        "print(env.action_space.sample())\n",
        "\n",
        "GO_LEFT = 0\n",
        "# Hardcoded best agent: always go left!\n",
        "n_steps = 20\n",
        "for step in range(n_steps):\n",
        "  print(\"Step {}\".format(step + 1))\n",
        "  obs, reward, done, trun, info = env.step(GO_LEFT)\n",
        "  print('obs=', obs, 'reward=', reward, 'done=', done)\n",
        "  env.render()\n",
        "  if done or trun:\n",
        "    print(\"Goal reached!\", \"reward=\", reward)\n",
        "    break"
      ],
      "execution_count": 101,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            ".........x.\n",
            "Box(0.0, 10.0, (1,), float32)\n",
            "Discrete(2)\n",
            "1\n",
            "0\n",
            "1\n",
            "0\n",
            "1\n",
            "Step 1\n",
            "obs= [8.] reward= 0 done= False\n",
            "........x..\n",
            "Step 2\n",
            "obs= [7.] reward= 0 done= False\n",
            ".......x...\n",
            "Step 3\n",
            "obs= [6.] reward= 0 done= False\n",
            "......x....\n",
            "Step 4\n",
            "obs= [5.] reward= 0 done= False\n",
            ".....x.....\n",
            "Step 5\n",
            "obs= [4.] reward= 0 done= False\n",
            "....x......\n",
            "Step 6\n",
            "obs= [3.] reward= 0 done= False\n",
            "...x.......\n",
            "Step 7\n",
            "obs= [2.] reward= 0 done= False\n",
            "..x........\n",
            "Step 8\n",
            "obs= [1.] reward= 0 done= False\n",
            ".x.........\n",
            "Step 9\n",
            "obs= [0.] reward= 41 done= True\n",
            "x..........\n",
            "Goal reached! reward= 41\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pv1e1qJETfHU"
      },
      "source": [
        "### Try it with Stable-Baselines\n",
        "\n",
        "> Once your environment follow the gym interface, it is quite easy to plug in any algorithm from stable-baselines\n",
        "\n",
        "I trained two models here to sort of race them and to make sure the training was actually working.\n",
        "\n",
        "I don't know what the `make_vec_env` is. It's important though. It's mentioned in the [Getting Started](https://stable-baselines3.readthedocs.io/en/master/guide/quickstart.html) and is used in the colab they shared. Otherwise I couldn't get it to work"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from stable_baselines3 import PPO, A2C, DQN\n",
        "from stable_baselines3.common.env_util import make_vec_env"
      ],
      "metadata": {
        "id": "rMsz4tF19gCk"
      },
      "execution_count": 102,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Instantiate the env\n",
        "vec_env = make_vec_env(GoLeftEnv, n_envs=1, env_kwargs=dict(grid_size=10))\n",
        "\n",
        "env = GoLeftEnv(grid_size=10)\n",
        "env.reset()\n",
        "\n",
        "# Train the agent\n",
        "model = DQN(\"MlpPolicy\", env, verbose=1).learn(50000, log_interval=100)\n",
        "\n",
        "# Train the second agent. It's a little dumber but it can win\n",
        "model_two = DQN(\"MlpPolicy\", env, verbose=1)\n",
        "model_two.learn(total_timesteps=10000, log_interval=100)"
      ],
      "metadata": {
        "id": "PUk5K1GI9o8l",
        "outputId": "9dbbbb48-cf67-4993-aac9-abfef4aad497",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 121,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Using cuda device\n",
            "Wrapping the env with a `Monitor` wrapper\n",
            "Wrapping the env in a DummyVecEnv.\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 44.9     |\n",
            "|    ep_rew_mean      | -30.9    |\n",
            "|    exploration_rate | 0.147    |\n",
            "| time/               |          |\n",
            "|    episodes         | 100      |\n",
            "|    fps              | 11192    |\n",
            "|    time_elapsed     | 0        |\n",
            "|    total_timesteps  | 4487     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 43.5     |\n",
            "|    ep_rew_mean      | -28      |\n",
            "|    exploration_rate | 0.05     |\n",
            "| time/               |          |\n",
            "|    episodes         | 200      |\n",
            "|    fps              | 11222    |\n",
            "|    time_elapsed     | 0        |\n",
            "|    total_timesteps  | 8839     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 45       |\n",
            "|    ep_rew_mean      | -30      |\n",
            "|    exploration_rate | 0.05     |\n",
            "| time/               |          |\n",
            "|    episodes         | 300      |\n",
            "|    fps              | 11139    |\n",
            "|    time_elapsed     | 1        |\n",
            "|    total_timesteps  | 13335    |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 44.1     |\n",
            "|    ep_rew_mean      | -27.1    |\n",
            "|    exploration_rate | 0.05     |\n",
            "| time/               |          |\n",
            "|    episodes         | 400      |\n",
            "|    fps              | 11164    |\n",
            "|    time_elapsed     | 1        |\n",
            "|    total_timesteps  | 17746    |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 46.9     |\n",
            "|    ep_rew_mean      | -35.4    |\n",
            "|    exploration_rate | 0.05     |\n",
            "| time/               |          |\n",
            "|    episodes         | 500      |\n",
            "|    fps              | 11291    |\n",
            "|    time_elapsed     | 1        |\n",
            "|    total_timesteps  | 22431    |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 44.7     |\n",
            "|    ep_rew_mean      | -31.2    |\n",
            "|    exploration_rate | 0.05     |\n",
            "| time/               |          |\n",
            "|    episodes         | 600      |\n",
            "|    fps              | 11219    |\n",
            "|    time_elapsed     | 2        |\n",
            "|    total_timesteps  | 26898    |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 44.1     |\n",
            "|    ep_rew_mean      | -30.1    |\n",
            "|    exploration_rate | 0.05     |\n",
            "| time/               |          |\n",
            "|    episodes         | 700      |\n",
            "|    fps              | 11217    |\n",
            "|    time_elapsed     | 2        |\n",
            "|    total_timesteps  | 31313    |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 44.7     |\n",
            "|    ep_rew_mean      | -28.2    |\n",
            "|    exploration_rate | 0.05     |\n",
            "| time/               |          |\n",
            "|    episodes         | 800      |\n",
            "|    fps              | 11249    |\n",
            "|    time_elapsed     | 3        |\n",
            "|    total_timesteps  | 35779    |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 43.5     |\n",
            "|    ep_rew_mean      | -26      |\n",
            "|    exploration_rate | 0.05     |\n",
            "| time/               |          |\n",
            "|    episodes         | 900      |\n",
            "|    fps              | 11255    |\n",
            "|    time_elapsed     | 3        |\n",
            "|    total_timesteps  | 40132    |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 43.2     |\n",
            "|    ep_rew_mean      | -26.7    |\n",
            "|    exploration_rate | 0.05     |\n",
            "| time/               |          |\n",
            "|    episodes         | 1000     |\n",
            "|    fps              | 11297    |\n",
            "|    time_elapsed     | 3        |\n",
            "|    total_timesteps  | 44450    |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 44.4     |\n",
            "|    ep_rew_mean      | -31.4    |\n",
            "|    exploration_rate | 0.05     |\n",
            "| time/               |          |\n",
            "|    episodes         | 1100     |\n",
            "|    fps              | 11322    |\n",
            "|    time_elapsed     | 4        |\n",
            "|    total_timesteps  | 48888    |\n",
            "----------------------------------\n",
            "Using cuda device\n",
            "Wrapping the env with a `Monitor` wrapper\n",
            "Wrapping the env in a DummyVecEnv.\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 45       |\n",
            "|    ep_rew_mean      | -31.5    |\n",
            "|    exploration_rate | 0.05     |\n",
            "| time/               |          |\n",
            "|    episodes         | 100      |\n",
            "|    fps              | 11718    |\n",
            "|    time_elapsed     | 0        |\n",
            "|    total_timesteps  | 4502     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 45.7     |\n",
            "|    ep_rew_mean      | -32.7    |\n",
            "|    exploration_rate | 0.05     |\n",
            "| time/               |          |\n",
            "|    episodes         | 200      |\n",
            "|    fps              | 11732    |\n",
            "|    time_elapsed     | 0        |\n",
            "|    total_timesteps  | 9069     |\n",
            "----------------------------------\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<stable_baselines3.dqn.dqn.DQN at 0x7b8cf1e46f50>"
            ]
          },
          "metadata": {},
          "execution_count": 121
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Test the trained agent\n",
        "# using the vecenv\n",
        "obs = vec_env.reset()\n",
        "n_steps = 50\n",
        "path = []\n",
        "for step in range(n_steps):\n",
        "    action, _ = model.predict(obs, deterministic=False)\n",
        "    obs, reward, done, info = vec_env.step(action)\n",
        "    path.append(obs[0][0])\n",
        "    if done:\n",
        "        break\n",
        "print(path)\n",
        "print(f\"Reward={reward}\")\n",
        "\n",
        "print(\"Number two GO\")\n",
        "\n",
        "# Test the trained agent\n",
        "# using the vecenv\n",
        "obs = vec_env.reset()\n",
        "n_steps = 50\n",
        "path = []\n",
        "for step in range(n_steps):\n",
        "    action, _ = model_two.predict(obs, deterministic=False)\n",
        "    obs, reward, done, info = vec_env.step(action)\n",
        "    path.append(obs[0][0])\n",
        "    if done:\n",
        "        break\n",
        "print(path)\n",
        "print(f\"Reward={reward}\")"
      ],
      "metadata": {
        "id": "h-t6bJ3V3XKe",
        "outputId": "c40aeb90-5bf6-4e9f-b02b-7eb9a5da9fd0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 129,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, 9.0]\n",
            "Reward=[41.]\n",
            "Number two GO\n",
            "[10.0, 9.0, 10.0, 9.0, 10.0, 9.0, 10.0, 9.0, 10.0, 9.0, 10.0, 9.0, 10.0, 9.0, 10.0, 9.0, 10.0, 9.0, 10.0, 9.0, 10.0, 9.0, 10.0, 9.0, 10.0, 9.0, 10.0, 9.0, 10.0, 9.0, 10.0, 9.0, 10.0, 9.0, 10.0, 9.0, 10.0, 10.0, 9.0, 10.0, 9.0, 10.0, 9.0, 10.0, 9.0, 10.0, 9.0, 10.0, 9.0, 9.0]\n",
            "Reward=[-50.]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jOggIa9sU--b"
      },
      "source": [
        "## Future\n",
        "\n",
        "I'll finish my Euchre custom environment and put it in the place of this game and then train on that.\n"
      ]
    }
  ]
}