FinRL_stock_trading_NeurIPS_2018.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "FinRL_stock_trading_NeurIPS_2018.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "uijiWgkuh1jB",
        "MRiOtrywfAo1",
        "_gDkU-j-fCmZ",
        "3Zpv4S0-fDBv"
      ],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.12"
    },
    "pycharm": {
      "stem_cell": {
        "cell_type": "raw",
        "metadata": {
          "collapsed": false
        },
        "source": []
      }
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/AI4Finance-LLC/FinRL-Library/blob/master/FinRL_stock_trading_NeurIPS_2018.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gXaoZs2lh1hi"
      },
      "source": [
        "# Deep Reinforcement Learning for Stock Trading from Scratch: Multiple Stock Trading\n",
        "\n",
        "Tutorials to use OpenAI DRL to trade multiple stocks in one Jupyter Notebook | Presented at NeurIPS 2020: Deep RL Workshop\n",
        "\n",
        "* This blog uses FinRL to reproduce the paper: Practical Deep Reinforcement Learning Approach for Stock Trading, Workshop on Challenges and Opportunities for AI in Financial Services, NeurIPS 2018.\n",
        "* Check out medium blog for detailed explanations: https://towardsdatascience.com/finrl-for-quantitative-finance-tutorial-for-multiple-stock-trading-7b00763b7530\n",
        "* Please report any issues to our Github: https://github.com/AI4Finance-LLC/FinRL-Library/issues\n",
        "* **Pytorch Version** \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lGunVt8oLCVS"
      },
      "source": [
        "# Content"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HOzAKQ-SLGX6"
      },
      "source": [
        "* [1. Problem Definition](#0)\n",
        "* [2. Getting Started - Load Python packages](#1)\n",
        "    * [2.1. Install Packages](#1.1)    \n",
        "    * [2.2. Check Additional Packages](#1.2)\n",
        "    * [2.3. Import Packages](#1.3)\n",
        "    * [2.4. Create Folders](#1.4)\n",
        "* [3. Download Data](#2)\n",
        "* [4. Preprocess Data](#3)        \n",
        "    * [4.1. Technical Indicators](#3.1)\n",
        "    * [4.2. Perform Feature Engineering](#3.2)\n",
        "* [5.Build Environment](#4)  \n",
        "    * [5.1. Training & Trade Data Split](#4.1)\n",
        "    * [5.2. User-defined Environment](#4.2)   \n",
        "    * [5.3. Initialize Environment](#4.3)    \n",
        "* [6.Implement DRL Algorithms](#5)  \n",
        "* [7.Backtesting Performance](#6)  \n",
        "    * [7.1. BackTestStats](#6.1)\n",
        "    * [7.2. BackTestPlot](#6.2)   \n",
        "    * [7.3. Baseline Stats](#6.3)   \n",
        "    * [7.3. Compare to Stock Market Index](#6.4)             "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sApkDlD9LIZv"
      },
      "source": [
        "<a id='0'></a>\n",
        "# Part 1. Problem Definition"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HjLD2TZSLKZ-"
      },
      "source": [
        "This problem is to design an automated trading solution for single stock trading. We model the stock trading process as a Markov Decision Process (MDP). We then formulate our trading goal as a maximization problem.\n",
        "\n",
        "The algorithm is trained using Deep Reinforcement Learning (DRL) algorithms and the components of the reinforcement learning environment are:\n",
        "\n",
        "\n",
        "* Action: The action space describes the allowed actions that the agent interacts with the\n",
        "environment. Normally, a ∈ A includes three actions: a ∈ {−1, 0, 1}, where −1, 0, 1 represent\n",
        "selling, holding, and buying one stock. Also, an action can be carried upon multiple shares. We use\n",
        "an action space {−k, ..., −1, 0, 1, ..., k}, where k denotes the number of shares. For example, \"Buy\n",
        "10 shares of AAPL\" or \"Sell 10 shares of AAPL\" are 10 or −10, respectively\n",
        "\n",
        "* Reward function: r(s, a, s′) is the incentive mechanism for an agent to learn a better action. The change of the portfolio value when action a is taken at state s and arriving at new state s',  i.e., r(s, a, s′) = v′ − v, where v′ and v represent the portfolio\n",
        "values at state s′ and s, respectively\n",
        "\n",
        "* State: The state space describes the observations that the agent receives from the environment. Just as a human trader needs to analyze various information before executing a trade, so\n",
        "our trading agent observes many different features to better learn in an interactive environment.\n",
        "\n",
        "* Environment: Dow 30 consituents\n",
        "\n",
        "\n",
        "The data of the single stock that we will be using for this case study is obtained from Yahoo Finance API. The data contains Open-High-Low-Close price and volume.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ffsre789LY08"
      },
      "source": [
        "<a id='1'></a>\n",
        "# Part 2. Getting Started- Load Python Packages"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Uy5_PTmOh1hj"
      },
      "source": [
        "<a id='1.1'></a>\n",
        "## 2.1. Install all the packages through FinRL library\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mPT0ipYE28wL",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "f41b314e-3c47-47d4-bd0d-d297cae2d01f"
      },
      "source": [
        "## install finrl library\n",
        "!pip install git+https://github.com/AI4Finance-LLC/FinRL-Library.git"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting git+https://github.com/AI4Finance-LLC/FinRL-Library.git\n",
            "  Cloning https://github.com/AI4Finance-LLC/FinRL-Library.git to /tmp/pip-req-build-y67num83\n",
            "  Running command git clone -q https://github.com/AI4Finance-LLC/FinRL-Library.git /tmp/pip-req-build-y67num83\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from finrl==0.0.3) (1.19.5)\n",
            "Requirement already satisfied: pandas>=1.1.5 in /usr/local/lib/python3.6/dist-packages (from finrl==0.0.3) (1.1.5)\n",
            "Collecting stockstats\n",
            "  Downloading https://files.pythonhosted.org/packages/32/41/d3828c5bc0a262cb3112a4024108a3b019c183fa3b3078bff34bf25abf91/stockstats-0.3.2-py2.py3-none-any.whl\n",
            "Collecting yfinance\n",
            "  Downloading https://files.pythonhosted.org/packages/7a/e8/b9d7104d3a4bf39924799067592d9e59119fcfc900a425a12e80a3123ec8/yfinance-0.1.55.tar.gz\n",
            "Collecting pyfolio\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/28/b4/99799b743c4619752f88b70354924132a2e9b82f4656fe7c55eaa9101392/pyfolio-0.9.2.tar.gz (91kB)\n",
            "\u001b[K     |████████████████████████████████| 92kB 5.6MB/s \n",
            "\u001b[?25hRequirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from finrl==0.0.3) (3.2.2)\n",
            "Requirement already satisfied: scikit-learn>=0.21.0 in /usr/local/lib/python3.6/dist-packages (from finrl==0.0.3) (0.22.2.post1)\n",
            "Requirement already satisfied: gym>=0.17 in /usr/local/lib/python3.6/dist-packages (from finrl==0.0.3) (0.17.3)\n",
            "Collecting stable-baselines3[extra]\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/76/7c/ec89fd9a51c2ff640f150479069be817136c02f02349b5dd27a6e3bb8b3d/stable_baselines3-0.10.0-py3-none-any.whl (145kB)\n",
            "\u001b[K     |████████████████████████████████| 153kB 14.9MB/s \n",
            "\u001b[?25hRequirement already satisfied: pytest in /usr/local/lib/python3.6/dist-packages (from finrl==0.0.3) (3.6.4)\n",
            "Requirement already satisfied: setuptools>=41.4.0 in /usr/local/lib/python3.6/dist-packages (from finrl==0.0.3) (53.0.0)\n",
            "Requirement already satisfied: wheel>=0.33.6 in /usr/local/lib/python3.6/dist-packages (from finrl==0.0.3) (0.36.2)\n",
            "Collecting arrow\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ca/bc/ebc1afb3c54377e128a01024c006f983d03ee124bc52392b78ba98c421b8/arrow-0.17.0-py2.py3-none-any.whl (50kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 5.6MB/s \n",
            "\u001b[?25hCollecting python-rapidjson\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a0/8f/50488049033aba6943659d7906a45947ea5b510b74e7d44ab9d7457e1710/python_rapidjson-1.0-cp36-cp36m-manylinux2010_x86_64.whl (1.4MB)\n",
            "\u001b[K     |████████████████████████████████| 1.4MB 16.7MB/s \n",
            "\u001b[?25hCollecting questionary\n",
            "  Downloading https://files.pythonhosted.org/packages/0d/44/972fd92022c4dbc28fa4cbb2e161e60e2f116eb8c64b58c2c90a4b57c4c9/questionary-1.9.0-py3-none-any.whl\n",
            "Requirement already satisfied: sqlalchemy in /usr/local/lib/python3.6/dist-packages (from finrl==0.0.3) (1.3.23)\n",
            "Requirement already satisfied: tabulate in /usr/local/lib/python3.6/dist-packages (from finrl==0.0.3) (0.8.7)\n",
            "Collecting ccxt\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/7e/26/0c737254cc8f411765ae385a75292f3b68555a95232cba70e5b009824332/ccxt-1.41.82-py2.py3-none-any.whl (2.0MB)\n",
            "\u001b[K     |████████████████████████████████| 2.0MB 44.2MB/s \n",
            "\u001b[?25hCollecting colorama\n",
            "  Downloading https://files.pythonhosted.org/packages/44/98/5b86278fbbf250d239ae0ecb724f8572af1c91f4a11edf4d36a206189440/colorama-0.4.4-py2.py3-none-any.whl\n",
            "Requirement already satisfied: tensorflow in /usr/local/lib/python3.6/dist-packages (from finrl==0.0.3) (2.4.1)\n",
            "Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.6/dist-packages (from finrl==0.0.3) (1.5.1)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=1.1.5->finrl==0.0.3) (2018.9)\n",
            "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.6/dist-packages (from pandas>=1.1.5->finrl==0.0.3) (2.8.1)\n",
            "Collecting int-date>=0.1.7\n",
            "  Downloading https://files.pythonhosted.org/packages/43/27/31803df15173ab341fe7548c14154b54227dfd8f630daa09a1c6e7db52f7/int_date-0.1.8-py2.py3-none-any.whl\n",
            "Requirement already satisfied: requests>=2.20 in /usr/local/lib/python3.6/dist-packages (from yfinance->finrl==0.0.3) (2.23.0)\n",
            "Requirement already satisfied: multitasking>=0.0.7 in /usr/local/lib/python3.6/dist-packages (from yfinance->finrl==0.0.3) (0.0.9)\n",
            "Collecting lxml>=4.5.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/bd/78/56a7c88a57d0d14945472535d0df9fb4bbad7d34ede658ec7961635c790e/lxml-4.6.2-cp36-cp36m-manylinux1_x86_64.whl (5.5MB)\n",
            "\u001b[K     |████████████████████████████████| 5.5MB 52.9MB/s \n",
            "\u001b[?25hRequirement already satisfied: ipython>=3.2.3 in /usr/local/lib/python3.6/dist-packages (from pyfolio->finrl==0.0.3) (5.5.0)\n",
            "Requirement already satisfied: scipy>=0.14.0 in /usr/local/lib/python3.6/dist-packages (from pyfolio->finrl==0.0.3) (1.4.1)\n",
            "Requirement already satisfied: seaborn>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from pyfolio->finrl==0.0.3) (0.11.1)\n",
            "Collecting empyrical>=0.5.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/74/43/1b997c21411c6ab7c96dc034e160198272c7a785aeea7654c9bcf98bec83/empyrical-0.5.5.tar.gz (52kB)\n",
            "\u001b[K     |████████████████████████████████| 61kB 7.2MB/s \n",
            "\u001b[?25hRequirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->finrl==0.0.3) (2.4.7)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->finrl==0.0.3) (1.3.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->finrl==0.0.3) (0.10.0)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn>=0.21.0->finrl==0.0.3) (1.0.0)\n",
            "Requirement already satisfied: pyglet<=1.5.0,>=1.4.0 in /usr/local/lib/python3.6/dist-packages (from gym>=0.17->finrl==0.0.3) (1.5.0)\n",
            "Requirement already satisfied: cloudpickle<1.7.0,>=1.2.0 in /usr/local/lib/python3.6/dist-packages (from gym>=0.17->finrl==0.0.3) (1.3.0)\n",
            "Requirement already satisfied: torch>=1.4.0 in /usr/local/lib/python3.6/dist-packages (from stable-baselines3[extra]->finrl==0.0.3) (1.7.0+cu101)\n",
            "Requirement already satisfied: tensorboard; extra == \"extra\" in /usr/local/lib/python3.6/dist-packages (from stable-baselines3[extra]->finrl==0.0.3) (2.4.1)\n",
            "Requirement already satisfied: opencv-python; extra == \"extra\" in /usr/local/lib/python3.6/dist-packages (from stable-baselines3[extra]->finrl==0.0.3) (4.1.2.30)\n",
            "Requirement already satisfied: atari-py~=0.2.0; extra == \"extra\" in /usr/local/lib/python3.6/dist-packages (from stable-baselines3[extra]->finrl==0.0.3) (0.2.6)\n",
            "Requirement already satisfied: psutil; extra == \"extra\" in /usr/local/lib/python3.6/dist-packages (from stable-baselines3[extra]->finrl==0.0.3) (5.4.8)\n",
            "Requirement already satisfied: pillow; extra == \"extra\" in /usr/local/lib/python3.6/dist-packages (from stable-baselines3[extra]->finrl==0.0.3) (7.0.0)\n",
            "Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from pytest->finrl==0.0.3) (1.15.0)\n",
            "Requirement already satisfied: py>=1.5.0 in /usr/local/lib/python3.6/dist-packages (from pytest->finrl==0.0.3) (1.10.0)\n",
            "Requirement already satisfied: atomicwrites>=1.0 in /usr/local/lib/python3.6/dist-packages (from pytest->finrl==0.0.3) (1.4.0)\n",
            "Requirement already satisfied: pluggy<0.8,>=0.5 in /usr/local/lib/python3.6/dist-packages (from pytest->finrl==0.0.3) (0.7.1)\n",
            "Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.6/dist-packages (from pytest->finrl==0.0.3) (20.3.0)\n",
            "Requirement already satisfied: more-itertools>=4.0.0 in /usr/local/lib/python3.6/dist-packages (from pytest->finrl==0.0.3) (8.7.0)\n",
            "Collecting prompt_toolkit<4.0,>=2.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a6/0b/c6de29441b29f8b54d5bbe29a8b223de6e400714ff50e85541bd4c783421/prompt_toolkit-3.0.16-py3-none-any.whl (366kB)\n",
            "\u001b[K     |████████████████████████████████| 368kB 35.9MB/s \n",
            "\u001b[?25hCollecting cryptography>=2.6.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a6/a1/49543f8ae3165c598e6c1393c54f9af8eaf7111f86e769ab4b897cdcf096/cryptography-3.4.4-cp36-abi3-manylinux2014_x86_64.whl (3.2MB)\n",
            "\u001b[K     |████████████████████████████████| 3.2MB 36.0MB/s \n",
            "\u001b[?25hRequirement already satisfied: certifi>=2018.1.18 in /usr/local/lib/python3.6/dist-packages (from ccxt->finrl==0.0.3) (2020.12.5)\n",
            "Collecting aiodns<2.1,>=1.1.1; python_version >= \"3.5.2\"\n",
            "  Downloading https://files.pythonhosted.org/packages/da/01/8f2d49b441573fd2478833bdba91cf0b853b4c750a1fbb9e98de1b94bb22/aiodns-2.0.0-py2.py3-none-any.whl\n",
            "Collecting aiohttp<3.8,>=3.7.2; python_version >= \"3.5.2\"\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ad/e6/d4b6235d776c9b33f853e603efede5aac5a34f71ca9d3877adb30492eb4e/aiohttp-3.7.3-cp36-cp36m-manylinux2014_x86_64.whl (1.3MB)\n",
            "\u001b[K     |████████████████████████████████| 1.3MB 40.8MB/s \n",
            "\u001b[?25hCollecting yarl==1.1.0; python_version >= \"3.5.2\"\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/9a/fe/1fc0d9e277f1ff0d29ac44742f769c55f12cc7e66a89ab340f5321937179/yarl-1.1.0-cp36-cp36m-manylinux1_x86_64.whl (255kB)\n",
            "\u001b[K     |████████████████████████████████| 256kB 48.0MB/s \n",
            "\u001b[?25hRequirement already satisfied: protobuf>=3.9.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow->finrl==0.0.3) (3.12.4)\n",
            "Requirement already satisfied: typing-extensions~=3.7.4 in /usr/local/lib/python3.6/dist-packages (from tensorflow->finrl==0.0.3) (3.7.4.3)\n",
            "Requirement already satisfied: google-pasta~=0.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow->finrl==0.0.3) (0.2.0)\n",
            "Requirement already satisfied: absl-py~=0.10 in /usr/local/lib/python3.6/dist-packages (from tensorflow->finrl==0.0.3) (0.10.0)\n",
            "Requirement already satisfied: keras-preprocessing~=1.1.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow->finrl==0.0.3) (1.1.2)\n",
            "Requirement already satisfied: opt-einsum~=3.3.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->finrl==0.0.3) (3.3.0)\n",
            "Requirement already satisfied: wrapt~=1.12.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow->finrl==0.0.3) (1.12.1)\n",
            "Requirement already satisfied: h5py~=2.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->finrl==0.0.3) (2.10.0)\n",
            "Requirement already satisfied: gast==0.3.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow->finrl==0.0.3) (0.3.3)\n",
            "Requirement already satisfied: termcolor~=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->finrl==0.0.3) (1.1.0)\n",
            "Requirement already satisfied: astunparse~=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow->finrl==0.0.3) (1.6.3)\n",
            "Requirement already satisfied: flatbuffers~=1.12.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->finrl==0.0.3) (1.12)\n",
            "Requirement already satisfied: grpcio~=1.32.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->finrl==0.0.3) (1.32.0)\n",
            "Requirement already satisfied: tensorflow-estimator<2.5.0,>=2.4.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->finrl==0.0.3) (2.4.0)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.20->yfinance->finrl==0.0.3) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.20->yfinance->finrl==0.0.3) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.20->yfinance->finrl==0.0.3) (2.10)\n",
            "Requirement already satisfied: pygments in /usr/local/lib/python3.6/dist-packages (from ipython>=3.2.3->pyfolio->finrl==0.0.3) (2.6.1)\n",
            "Requirement already satisfied: pexpect; sys_platform != \"win32\" in /usr/local/lib/python3.6/dist-packages (from ipython>=3.2.3->pyfolio->finrl==0.0.3) (4.8.0)\n",
            "Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.6/dist-packages (from ipython>=3.2.3->pyfolio->finrl==0.0.3) (4.3.3)\n",
            "Requirement already satisfied: decorator in /usr/local/lib/python3.6/dist-packages (from ipython>=3.2.3->pyfolio->finrl==0.0.3) (4.4.2)\n",
            "Requirement already satisfied: simplegeneric>0.8 in /usr/local/lib/python3.6/dist-packages (from ipython>=3.2.3->pyfolio->finrl==0.0.3) (0.8.1)\n",
            "Requirement already satisfied: pickleshare in /usr/local/lib/python3.6/dist-packages (from ipython>=3.2.3->pyfolio->finrl==0.0.3) (0.7.5)\n",
            "Requirement already satisfied: pandas-datareader>=0.2 in /usr/local/lib/python3.6/dist-packages (from empyrical>=0.5.0->pyfolio->finrl==0.0.3) (0.9.0)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from pyglet<=1.5.0,>=1.4.0->gym>=0.17->finrl==0.0.3) (0.16.0)\n",
            "Requirement already satisfied: dataclasses in /usr/local/lib/python3.6/dist-packages (from torch>=1.4.0->stable-baselines3[extra]->finrl==0.0.3) (0.8)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from tensorboard; extra == \"extra\"->stable-baselines3[extra]->finrl==0.0.3) (0.4.2)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard; extra == \"extra\"->stable-baselines3[extra]->finrl==0.0.3) (1.0.1)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard; extra == \"extra\"->stable-baselines3[extra]->finrl==0.0.3) (3.3.3)\n",
            "Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard; extra == \"extra\"->stable-baselines3[extra]->finrl==0.0.3) (1.25.0)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard; extra == \"extra\"->stable-baselines3[extra]->finrl==0.0.3) (1.8.0)\n",
            "Requirement already satisfied: wcwidth in /usr/local/lib/python3.6/dist-packages (from prompt_toolkit<4.0,>=2.0->questionary->finrl==0.0.3) (0.2.5)\n",
            "Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.6/dist-packages (from cryptography>=2.6.1->ccxt->finrl==0.0.3) (1.14.4)\n",
            "Collecting pycares>=3.0.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/43/2d/7f4984a23f6e99cf6a8b20ddc59308efb209fe81e79c97af65e9b30eefae/pycares-3.1.1-cp36-cp36m-manylinux2010_x86_64.whl (228kB)\n",
            "\u001b[K     |████████████████████████████████| 235kB 48.9MB/s \n",
            "\u001b[?25hCollecting typing; python_version < \"3.7\"\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/05/d9/6eebe19d46bd05360c9a9aae822e67a80f9242aabbfc58b641b957546607/typing-3.7.4.3.tar.gz (78kB)\n",
            "\u001b[K     |████████████████████████████████| 81kB 8.2MB/s \n",
            "\u001b[?25hCollecting async-timeout<4.0,>=3.0\n",
            "  Downloading https://files.pythonhosted.org/packages/e1/1e/5a4441be21b0726c4464f3f23c8b19628372f606755a9d2e46c187e65ec4/async_timeout-3.0.1-py3-none-any.whl\n",
            "Collecting idna-ssl>=1.0; python_version < \"3.7\"\n",
            "  Downloading https://files.pythonhosted.org/packages/46/03/07c4894aae38b0de52b52586b24bf189bb83e4ddabfe2e2c8f2419eec6f4/idna-ssl-1.1.0.tar.gz\n",
            "Collecting multidict<7.0,>=4.5\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a1/35/b22524d6b9cacfb4c5eff413a069bbc17c6ea628e54da5c6c989998ced5f/multidict-5.1.0-cp36-cp36m-manylinux2014_x86_64.whl (141kB)\n",
            "\u001b[K     |████████████████████████████████| 143kB 43.3MB/s \n",
            "\u001b[?25hRequirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.6/dist-packages (from pexpect; sys_platform != \"win32\"->ipython>=3.2.3->pyfolio->finrl==0.0.3) (0.7.0)\n",
            "Requirement already satisfied: ipython-genutils in /usr/local/lib/python3.6/dist-packages (from traitlets>=4.2->ipython>=3.2.3->pyfolio->finrl==0.0.3) (0.2.0)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard; extra == \"extra\"->stable-baselines3[extra]->finrl==0.0.3) (1.3.0)\n",
            "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from markdown>=2.6.8->tensorboard; extra == \"extra\"->stable-baselines3[extra]->finrl==0.0.3) (3.4.0)\n",
            "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard; extra == \"extra\"->stable-baselines3[extra]->finrl==0.0.3) (4.2.1)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard; extra == \"extra\"->stable-baselines3[extra]->finrl==0.0.3) (0.2.8)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4; python_version >= \"3.6\" in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard; extra == \"extra\"->stable-baselines3[extra]->finrl==0.0.3) (4.7)\n",
            "Requirement already satisfied: pycparser in /usr/local/lib/python3.6/dist-packages (from cffi>=1.12->cryptography>=2.6.1->ccxt->finrl==0.0.3) (2.20)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard; extra == \"extra\"->stable-baselines3[extra]->finrl==0.0.3) (3.1.0)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < \"3.8\"->markdown>=2.6.8->tensorboard; extra == \"extra\"->stable-baselines3[extra]->finrl==0.0.3) (3.4.0)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.6/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard; extra == \"extra\"->stable-baselines3[extra]->finrl==0.0.3) (0.4.8)\n",
            "Building wheels for collected packages: finrl, yfinance, pyfolio, empyrical, typing, idna-ssl\n",
            "  Building wheel for finrl (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for finrl: filename=finrl-0.0.3-cp36-none-any.whl size=111367 sha256=73a88511f1e56ca7a69a94501ddb2c04820a190bdec539efd2394842bd635a9c\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-bydxrpfe/wheels/9c/19/bf/c644def96612df1ad42c94d5304966797eaa3221dffc5efe0b\n",
            "  Building wheel for yfinance (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for yfinance: filename=yfinance-0.1.55-py2.py3-none-any.whl size=22616 sha256=71cb3444ecaf84f910d0478dcf0a93f740a4d7d85cbd0ff253f1da733ed86ef3\n",
            "  Stored in directory: /root/.cache/pip/wheels/04/98/cc/2702a4242d60bdc14f48b4557c427ded1fe92aedf257d4565c\n",
            "  Building wheel for pyfolio (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyfolio: filename=pyfolio-0.9.2-cp36-none-any.whl size=88669 sha256=b730d00a0d65b85e5cfcf3923ed2bad8b56854f914001636b74479a03966e124\n",
            "  Stored in directory: /root/.cache/pip/wheels/0d/a8/7d/ac33fe6bf6f3db370291c9de303460d00b8f97cc40a1d80f72\n",
            "  Building wheel for empyrical (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for empyrical: filename=empyrical-0.5.5-cp36-none-any.whl size=39765 sha256=0acb8062868cdfcdaeef6e66f33e592ee7f91be52383d1f40a2657c5b11c8135\n",
            "  Stored in directory: /root/.cache/pip/wheels/ea/b2/c8/6769d8444d2f2e608fae2641833110668d0ffd1abeb2e9f3fc\n",
            "  Building wheel for typing (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for typing: filename=typing-3.7.4.3-cp36-none-any.whl size=26310 sha256=7cbe013a223bb71ee59aababb30065ac713d79f8043d4a9d84acac2add527b35\n",
            "  Stored in directory: /root/.cache/pip/wheels/2d/04/41/8e1836e79581989c22eebac3f4e70aaac9af07b0908da173be\n",
            "  Building wheel for idna-ssl (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for idna-ssl: filename=idna_ssl-1.1.0-cp36-none-any.whl size=3163 sha256=414f423584eada23761ebb55eca2ee8d47fbc29e6fa7579d321cec4f7fcddeae\n",
            "  Stored in directory: /root/.cache/pip/wheels/d3/00/b3/32d613e19e08a739751dd6bf998cfed277728f8b2127ad4eb7\n",
            "Successfully built finrl yfinance pyfolio empyrical typing idna-ssl\n",
            "\u001b[31mERROR: jupyter-console 5.2.0 has requirement prompt-toolkit<2.0.0,>=1.0.0, but you'll have prompt-toolkit 3.0.16 which is incompatible.\u001b[0m\n",
            "\u001b[31mERROR: ipython 5.5.0 has requirement prompt-toolkit<2.0.0,>=1.0.4, but you'll have prompt-toolkit 3.0.16 which is incompatible.\u001b[0m\n",
            "Installing collected packages: int-date, stockstats, lxml, yfinance, empyrical, pyfolio, stable-baselines3, arrow, python-rapidjson, prompt-toolkit, questionary, cryptography, pycares, typing, aiodns, async-timeout, multidict, yarl, idna-ssl, aiohttp, ccxt, colorama, finrl\n",
            "  Found existing installation: lxml 4.2.6\n",
            "    Uninstalling lxml-4.2.6:\n",
            "      Successfully uninstalled lxml-4.2.6\n",
            "  Found existing installation: prompt-toolkit 1.0.18\n",
            "    Uninstalling prompt-toolkit-1.0.18:\n",
            "      Successfully uninstalled prompt-toolkit-1.0.18\n",
            "Successfully installed aiodns-2.0.0 aiohttp-3.7.3 arrow-0.17.0 async-timeout-3.0.1 ccxt-1.41.82 colorama-0.4.4 cryptography-3.4.4 empyrical-0.5.5 finrl-0.0.3 idna-ssl-1.1.0 int-date-0.1.8 lxml-4.6.2 multidict-5.1.0 prompt-toolkit-3.0.16 pycares-3.1.1 pyfolio-0.9.2 python-rapidjson-1.0 questionary-1.9.0 stable-baselines3-0.10.0 stockstats-0.3.2 typing-3.7.4.3 yarl-1.1.0 yfinance-0.1.55\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "prompt_toolkit",
                  "typing"
                ]
              }
            }
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "osBHhVysOEzi"
      },
      "source": [
        "\n",
        "<a id='1.2'></a>\n",
        "## 2.2. Check if the additional packages needed are present, if not install them. \n",
        "* Yahoo Finance API\n",
        "* pandas\n",
        "* numpy\n",
        "* matplotlib\n",
        "* stockstats\n",
        "* OpenAI gym\n",
        "* stable-baselines\n",
        "* tensorflow\n",
        "* pyfolio"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nGv01K8Sh1hn"
      },
      "source": [
        "<a id='1.3'></a>\n",
        "## 2.3. Import Packages"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lPqeTTwoh1hn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d7cb9f1c-79a6-49fa-84b5-e23b01843426"
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib\n",
        "import matplotlib.pyplot as plt\n",
        "# matplotlib.use('Agg')\n",
        "import datetime\n",
        "\n",
        "%matplotlib inline\n",
        "from finrl.config import config\n",
        "from finrl.marketdata.yahoodownloader import YahooDownloader\n",
        "from finrl.preprocessing.preprocessors import FeatureEngineer\n",
        "from finrl.preprocessing.data import data_split\n",
        "from finrl.env.env_stocktrading import StockTradingEnv\n",
        "from finrl.model.models import DRLAgent\n",
        "from finrl.trade.backtest import backtest_stats, backtest_plot, get_daily_return, get_baseline\n",
        "\n",
        "from pprint import pprint\n",
        "\n",
        "import sys\n",
        "sys.path.append(\"../FinRL-Library\")\n",
        "\n",
        "import itertools"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/pyfolio/pos.py:28: UserWarning: Module \"zipline.assets\" not found; mutltipliers will not be applied to position notionals.\n",
            "  ' to position notionals.'\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T2owTj985RW4"
      },
      "source": [
        "<a id='1.4'></a>\n",
        "## 2.4. Create Folders"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w9A8CN5R5PuZ"
      },
      "source": [
        "import os\n",
        "if not os.path.exists(\"./\" + config.DATA_SAVE_DIR):\n",
        "    os.makedirs(\"./\" + config.DATA_SAVE_DIR)\n",
        "if not os.path.exists(\"./\" + config.TRAINED_MODEL_DIR):\n",
        "    os.makedirs(\"./\" + config.TRAINED_MODEL_DIR)\n",
        "if not os.path.exists(\"./\" + config.TENSORBOARD_LOG_DIR):\n",
        "    os.makedirs(\"./\" + config.TENSORBOARD_LOG_DIR)\n",
        "if not os.path.exists(\"./\" + config.RESULTS_DIR):\n",
        "    os.makedirs(\"./\" + config.RESULTS_DIR)"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A289rQWMh1hq"
      },
      "source": [
        "<a id='2'></a>\n",
        "# Part 3. Download Data\n",
        "Yahoo Finance is a website that provides stock data, financial news, financial reports, etc. All the data provided by Yahoo Finance is free.\n",
        "* FinRL uses a class **YahooDownloader** to fetch data from Yahoo Finance API\n",
        "* Call Limit: Using the Public API (without authentication), you are limited to 2,000 requests per hour per IP (or up to a total of 48,000 requests a day).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NPeQ7iS-LoMm"
      },
      "source": [
        "\n",
        "\n",
        "-----\n",
        "class YahooDownloader:\n",
        "    Provides methods for retrieving daily stock data from\n",
        "    Yahoo Finance API\n",
        "\n",
        "    Attributes\n",
        "    ----------\n",
        "        start_date : str\n",
        "            start date of the data (modified from config.py)\n",
        "        end_date : str\n",
        "            end date of the data (modified from config.py)\n",
        "        ticker_list : list\n",
        "            a list of stock tickers (modified from config.py)\n",
        "\n",
        "    Methods\n",
        "    -------\n",
        "    fetch_data()\n",
        "        Fetches data from yahoo API\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "h3XJnvrbLp-C",
        "outputId": "90ad1901-fda9-46f8-80f6-083cd8d6293a"
      },
      "source": [
        "# from config.py start_date is a string\n",
        "config.START_DATE"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'2000-01-01'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "FUnY8WEfLq3C",
        "outputId": "9bd12370-a5e9-4eb3-bc2a-d0d0610f14f7"
      },
      "source": [
        "# from config.py end_date is a string\n",
        "config.END_DATE"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'2021-01-01'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JzqRRTOX6aFu",
        "outputId": "8acb257f-171e-4069-8ed3-1e7311ffecc2"
      },
      "source": [
        "print(config.DOW_30_TICKER)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['AAPL', 'MSFT', 'JPM', 'V', 'RTX', 'PG', 'GS', 'NKE', 'DIS', 'AXP', 'HD', 'INTC', 'WMT', 'IBM', 'MRK', 'UNH', 'KO', 'CAT', 'TRV', 'JNJ', 'CVX', 'MCD', 'VZ', 'CSCO', 'XOM', 'BA', 'MMM', 'PFE', 'WBA', 'DD']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yCKm4om-s9kE",
        "outputId": "fa646200-c409-4eb2-ac44-a1a807b89334"
      },
      "source": [
        "df = YahooDownloader(start_date = '2009-01-01',\n",
        "                     end_date = '2021-01-01',\n",
        "                     ticker_list = config.DOW_30_TICKER).fetch_data()"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "[*********************100%***********************]  1 of 1 completed\n",
            "Shape of DataFrame:  (90630, 8)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CV3HrZHLh1hy",
        "outputId": "7e60ee96-57e8-46ba-dffe-c6e2ce80252e"
      },
      "source": [
        "df.shape"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(90630, 8)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "4hYkeaPiICHS",
        "outputId": "8d313640-84f0-41c1-c619-57a25c33bfb7"
      },
      "source": [
        "df.sort_values(['date','tic'],ignore_index=True).head()"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>date</th>\n",
              "      <th>open</th>\n",
              "      <th>high</th>\n",
              "      <th>low</th>\n",
              "      <th>close</th>\n",
              "      <th>volume</th>\n",
              "      <th>tic</th>\n",
              "      <th>day</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>3.067143</td>\n",
              "      <td>3.251429</td>\n",
              "      <td>3.041429</td>\n",
              "      <td>2.791740</td>\n",
              "      <td>746015200</td>\n",
              "      <td>AAPL</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>18.570000</td>\n",
              "      <td>19.520000</td>\n",
              "      <td>18.400000</td>\n",
              "      <td>15.745411</td>\n",
              "      <td>10955700</td>\n",
              "      <td>AXP</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>42.799999</td>\n",
              "      <td>45.560001</td>\n",
              "      <td>42.779999</td>\n",
              "      <td>33.941101</td>\n",
              "      <td>7010200</td>\n",
              "      <td>BA</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>44.910000</td>\n",
              "      <td>46.980000</td>\n",
              "      <td>44.709999</td>\n",
              "      <td>32.978760</td>\n",
              "      <td>7117200</td>\n",
              "      <td>CAT</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>16.410000</td>\n",
              "      <td>17.000000</td>\n",
              "      <td>16.250000</td>\n",
              "      <td>12.683227</td>\n",
              "      <td>40980600</td>\n",
              "      <td>CSCO</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         date       open       high        low      close     volume   tic  day\n",
              "0  2009-01-02   3.067143   3.251429   3.041429   2.791740  746015200  AAPL    4\n",
              "1  2009-01-02  18.570000  19.520000  18.400000  15.745411   10955700   AXP    4\n",
              "2  2009-01-02  42.799999  45.560001  42.779999  33.941101    7010200    BA    4\n",
              "3  2009-01-02  44.910000  46.980000  44.709999  32.978760    7117200   CAT    4\n",
              "4  2009-01-02  16.410000  17.000000  16.250000  12.683227   40980600  CSCO    4"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uqC6c40Zh1iH"
      },
      "source": [
        "# Part 4: Preprocess Data\n",
        "Data preprocessing is a crucial step for training a high quality machine learning model. We need to check for missing data and do feature engineering in order to convert the data into a model-ready state.\n",
        "* Add technical indicators. In practical trading, various information needs to be taken into account, for example the historical stock prices, current holding shares, technical indicators, etc. In this article, we demonstrate two trend-following technical indicators: MACD and RSI.\n",
        "* Add turbulence index. Risk-aversion reflects whether an investor will choose to preserve the capital. It also influences one's trading strategy when facing different market volatility level. To control the risk in a worst-case scenario, such as financial crisis of 2007–2008, FinRL employs the financial turbulence index that measures extreme asset price fluctuation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "id": "PmKP-1ii3RLS",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c9d75bf8-49e9-4227-c198-b2636a37b406"
      },
      "source": [
        "fe = FeatureEngineer(\n",
        "                    use_technical_indicator=True,\n",
        "                    tech_indicator_list = config.TECHNICAL_INDICATORS_LIST,\n",
        "                    use_turbulence=True,\n",
        "                    user_defined_feature = False)\n",
        "\n",
        "processed = fe.preprocess_data(df)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Successfully added technical indicators\n",
            "Successfully added turbulence index\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Kixon2tR3RLT"
      },
      "source": [
        "list_ticker = processed[\"tic\"].unique().tolist()\n",
        "list_date = list(pd.date_range(processed['date'].min(),processed['date'].max()).astype(str))\n",
        "combination = list(itertools.product(list_date,list_ticker))\n",
        "\n",
        "processed_full = pd.DataFrame(combination,columns=[\"date\",\"tic\"]).merge(processed,on=[\"date\",\"tic\"],how=\"left\")\n",
        "processed_full = processed_full[processed_full['date'].isin(processed['date'])]\n",
        "processed_full = processed_full.sort_values(['date','tic'])\n",
        "\n",
        "processed_full = processed_full.fillna(0)"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 549
        },
        "id": "grvhGJJII3Xn",
        "outputId": "9e7b48ae-a31a-4e35-a313-aa5ae9fa66ca"
      },
      "source": [
        "processed_full.sort_values(['date','tic'],ignore_index=True).head(10)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>date</th>\n",
              "      <th>tic</th>\n",
              "      <th>open</th>\n",
              "      <th>high</th>\n",
              "      <th>low</th>\n",
              "      <th>close</th>\n",
              "      <th>volume</th>\n",
              "      <th>day</th>\n",
              "      <th>macd</th>\n",
              "      <th>boll_ub</th>\n",
              "      <th>boll_lb</th>\n",
              "      <th>rsi_30</th>\n",
              "      <th>cci_30</th>\n",
              "      <th>dx_30</th>\n",
              "      <th>close_30_sma</th>\n",
              "      <th>close_60_sma</th>\n",
              "      <th>turbulence</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>AAPL</td>\n",
              "      <td>3.067143</td>\n",
              "      <td>3.251429</td>\n",
              "      <td>3.041429</td>\n",
              "      <td>2.791740</td>\n",
              "      <td>746015200.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>3.017278</td>\n",
              "      <td>2.684025</td>\n",
              "      <td>100.000000</td>\n",
              "      <td>66.666667</td>\n",
              "      <td>100.000000</td>\n",
              "      <td>2.791740</td>\n",
              "      <td>2.791740</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>AXP</td>\n",
              "      <td>18.570000</td>\n",
              "      <td>19.520000</td>\n",
              "      <td>18.400000</td>\n",
              "      <td>15.745411</td>\n",
              "      <td>10955700.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.002643</td>\n",
              "      <td>3.017278</td>\n",
              "      <td>2.684025</td>\n",
              "      <td>100.000000</td>\n",
              "      <td>66.666667</td>\n",
              "      <td>100.000000</td>\n",
              "      <td>2.850651</td>\n",
              "      <td>2.850651</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>BA</td>\n",
              "      <td>42.799999</td>\n",
              "      <td>45.560001</td>\n",
              "      <td>42.779999</td>\n",
              "      <td>33.941101</td>\n",
              "      <td>7010200.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.001880</td>\n",
              "      <td>2.972787</td>\n",
              "      <td>2.735796</td>\n",
              "      <td>70.355711</td>\n",
              "      <td>46.771878</td>\n",
              "      <td>100.000000</td>\n",
              "      <td>2.854292</td>\n",
              "      <td>2.854292</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>CAT</td>\n",
              "      <td>44.910000</td>\n",
              "      <td>46.980000</td>\n",
              "      <td>44.709999</td>\n",
              "      <td>32.978760</td>\n",
              "      <td>7117200.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>-0.000746</td>\n",
              "      <td>2.951725</td>\n",
              "      <td>2.729582</td>\n",
              "      <td>50.429389</td>\n",
              "      <td>-29.777993</td>\n",
              "      <td>43.607834</td>\n",
              "      <td>2.840654</td>\n",
              "      <td>2.840654</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>CSCO</td>\n",
              "      <td>16.410000</td>\n",
              "      <td>17.000000</td>\n",
              "      <td>16.250000</td>\n",
              "      <td>12.683227</td>\n",
              "      <td>40980600.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>-0.000088</td>\n",
              "      <td>2.939568</td>\n",
              "      <td>2.746169</td>\n",
              "      <td>60.227126</td>\n",
              "      <td>-9.019317</td>\n",
              "      <td>48.357918</td>\n",
              "      <td>2.842869</td>\n",
              "      <td>2.842869</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>CVX</td>\n",
              "      <td>74.230003</td>\n",
              "      <td>77.300003</td>\n",
              "      <td>73.580002</td>\n",
              "      <td>48.043262</td>\n",
              "      <td>13695900.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>-0.002480</td>\n",
              "      <td>2.931445</td>\n",
              "      <td>2.735506</td>\n",
              "      <td>47.932736</td>\n",
              "      <td>-27.237701</td>\n",
              "      <td>49.952380</td>\n",
              "      <td>2.833476</td>\n",
              "      <td>2.833476</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>DD</td>\n",
              "      <td>21.605234</td>\n",
              "      <td>22.060680</td>\n",
              "      <td>20.993229</td>\n",
              "      <td>15.402899</td>\n",
              "      <td>13251037.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>-0.006646</td>\n",
              "      <td>2.938422</td>\n",
              "      <td>2.698235</td>\n",
              "      <td>40.237370</td>\n",
              "      <td>-101.614424</td>\n",
              "      <td>9.320766</td>\n",
              "      <td>2.818329</td>\n",
              "      <td>2.818329</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>DIS</td>\n",
              "      <td>22.760000</td>\n",
              "      <td>24.030001</td>\n",
              "      <td>22.500000</td>\n",
              "      <td>20.597496</td>\n",
              "      <td>9796600.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>-0.010724</td>\n",
              "      <td>2.943226</td>\n",
              "      <td>2.663404</td>\n",
              "      <td>37.181826</td>\n",
              "      <td>-113.576183</td>\n",
              "      <td>3.614243</td>\n",
              "      <td>2.803315</td>\n",
              "      <td>2.803315</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>GS</td>\n",
              "      <td>84.019997</td>\n",
              "      <td>87.620003</td>\n",
              "      <td>82.190002</td>\n",
              "      <td>72.844467</td>\n",
              "      <td>14088500.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>-0.017016</td>\n",
              "      <td>2.960305</td>\n",
              "      <td>2.606701</td>\n",
              "      <td>31.067739</td>\n",
              "      <td>-136.004971</td>\n",
              "      <td>17.746098</td>\n",
              "      <td>2.783503</td>\n",
              "      <td>2.783503</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>HD</td>\n",
              "      <td>23.070000</td>\n",
              "      <td>24.190001</td>\n",
              "      <td>22.959999</td>\n",
              "      <td>17.909452</td>\n",
              "      <td>14902500.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>-0.024322</td>\n",
              "      <td>2.978173</td>\n",
              "      <td>2.545136</td>\n",
              "      <td>27.267345</td>\n",
              "      <td>-166.471239</td>\n",
              "      <td>43.235022</td>\n",
              "      <td>2.761654</td>\n",
              "      <td>2.761654</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         date   tic       open  ...  close_30_sma  close_60_sma  turbulence\n",
              "0  2009-01-02  AAPL   3.067143  ...      2.791740      2.791740         0.0\n",
              "1  2009-01-02   AXP  18.570000  ...      2.850651      2.850651         0.0\n",
              "2  2009-01-02    BA  42.799999  ...      2.854292      2.854292         0.0\n",
              "3  2009-01-02   CAT  44.910000  ...      2.840654      2.840654         0.0\n",
              "4  2009-01-02  CSCO  16.410000  ...      2.842869      2.842869         0.0\n",
              "5  2009-01-02   CVX  74.230003  ...      2.833476      2.833476         0.0\n",
              "6  2009-01-02    DD  21.605234  ...      2.818329      2.818329         0.0\n",
              "7  2009-01-02   DIS  22.760000  ...      2.803315      2.803315         0.0\n",
              "8  2009-01-02    GS  84.019997  ...      2.783503      2.783503         0.0\n",
              "9  2009-01-02    HD  23.070000  ...      2.761654      2.761654         0.0\n",
              "\n",
              "[10 rows x 17 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-QsYaY0Dh1iw"
      },
      "source": [
        "<a id='4'></a>\n",
        "# Part 5. Design Environment\n",
        "Considering the stochastic and interactive nature of the automated stock trading tasks, a financial task is modeled as a **Markov Decision Process (MDP)** problem. The training process involves observing stock price change, taking an action and reward's calculation to have the agent adjusting its strategy accordingly. By interacting with the environment, the trading agent will derive a trading strategy with the maximized rewards as time proceeds.\n",
        "\n",
        "Our trading environments, based on OpenAI Gym framework, simulate live stock markets with real market data according to the principle of time-driven simulation.\n",
        "\n",
        "The action space describes the allowed actions that the agent interacts with the environment. Normally, action a includes three actions: {-1, 0, 1}, where -1, 0, 1 represent selling, holding, and buying one share. Also, an action can be carried upon multiple shares. We use an action space {-k,…,-1, 0, 1, …, k}, where k denotes the number of shares to buy and -k denotes the number of shares to sell. For example, \"Buy 10 shares of AAPL\" or \"Sell 10 shares of AAPL\" are 10 or -10, respectively. The continuous action space needs to be normalized to [-1, 1], since the policy is defined on a Gaussian distribution, which needs to be normalized and symmetric."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5TOhcryx44bb"
      },
      "source": [
        "## Training data split: 2009-01-01 to 2018-12-31\n",
        "## Trade data split: 2019-01-01 to 2020-09-30"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W0qaVGjLtgbI",
        "outputId": "4041e74d-4e95-4f08-ea0d-993502037bc5"
      },
      "source": [
        "train = data_split(processed_full, '2009-01-01','2019-01-01')\n",
        "trade = data_split(processed_full, '2019-01-01','2021-01-01')\n",
        "print(len(train))\n",
        "print(len(trade))"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "75480\n",
            "15150\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 309
        },
        "id": "p52zNCOhTtLR",
        "outputId": "ef026856-f15e-49e3-d3d9-b45eaed9f3bb"
      },
      "source": [
        "train.head()"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>date</th>\n",
              "      <th>tic</th>\n",
              "      <th>open</th>\n",
              "      <th>high</th>\n",
              "      <th>low</th>\n",
              "      <th>close</th>\n",
              "      <th>volume</th>\n",
              "      <th>day</th>\n",
              "      <th>macd</th>\n",
              "      <th>boll_ub</th>\n",
              "      <th>boll_lb</th>\n",
              "      <th>rsi_30</th>\n",
              "      <th>cci_30</th>\n",
              "      <th>dx_30</th>\n",
              "      <th>close_30_sma</th>\n",
              "      <th>close_60_sma</th>\n",
              "      <th>turbulence</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>AAPL</td>\n",
              "      <td>3.067143</td>\n",
              "      <td>3.251429</td>\n",
              "      <td>3.041429</td>\n",
              "      <td>2.791740</td>\n",
              "      <td>746015200.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>3.017278</td>\n",
              "      <td>2.684025</td>\n",
              "      <td>100.000000</td>\n",
              "      <td>66.666667</td>\n",
              "      <td>100.000000</td>\n",
              "      <td>2.791740</td>\n",
              "      <td>2.791740</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>AXP</td>\n",
              "      <td>18.570000</td>\n",
              "      <td>19.520000</td>\n",
              "      <td>18.400000</td>\n",
              "      <td>15.745411</td>\n",
              "      <td>10955700.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.002643</td>\n",
              "      <td>3.017278</td>\n",
              "      <td>2.684025</td>\n",
              "      <td>100.000000</td>\n",
              "      <td>66.666667</td>\n",
              "      <td>100.000000</td>\n",
              "      <td>2.850651</td>\n",
              "      <td>2.850651</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>BA</td>\n",
              "      <td>42.799999</td>\n",
              "      <td>45.560001</td>\n",
              "      <td>42.779999</td>\n",
              "      <td>33.941101</td>\n",
              "      <td>7010200.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.001880</td>\n",
              "      <td>2.972787</td>\n",
              "      <td>2.735796</td>\n",
              "      <td>70.355711</td>\n",
              "      <td>46.771878</td>\n",
              "      <td>100.000000</td>\n",
              "      <td>2.854292</td>\n",
              "      <td>2.854292</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>CAT</td>\n",
              "      <td>44.910000</td>\n",
              "      <td>46.980000</td>\n",
              "      <td>44.709999</td>\n",
              "      <td>32.978760</td>\n",
              "      <td>7117200.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>-0.000746</td>\n",
              "      <td>2.951725</td>\n",
              "      <td>2.729582</td>\n",
              "      <td>50.429389</td>\n",
              "      <td>-29.777993</td>\n",
              "      <td>43.607834</td>\n",
              "      <td>2.840654</td>\n",
              "      <td>2.840654</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2009-01-02</td>\n",
              "      <td>CSCO</td>\n",
              "      <td>16.410000</td>\n",
              "      <td>17.000000</td>\n",
              "      <td>16.250000</td>\n",
              "      <td>12.683227</td>\n",
              "      <td>40980600.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>-0.000088</td>\n",
              "      <td>2.939568</td>\n",
              "      <td>2.746169</td>\n",
              "      <td>60.227126</td>\n",
              "      <td>-9.019317</td>\n",
              "      <td>48.357918</td>\n",
              "      <td>2.842869</td>\n",
              "      <td>2.842869</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         date   tic       open  ...  close_30_sma  close_60_sma  turbulence\n",
              "0  2009-01-02  AAPL   3.067143  ...      2.791740      2.791740         0.0\n",
              "0  2009-01-02   AXP  18.570000  ...      2.850651      2.850651         0.0\n",
              "0  2009-01-02    BA  42.799999  ...      2.854292      2.854292         0.0\n",
              "0  2009-01-02   CAT  44.910000  ...      2.840654      2.840654         0.0\n",
              "0  2009-01-02  CSCO  16.410000  ...      2.842869      2.842869         0.0\n",
              "\n",
              "[5 rows x 17 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 309
        },
        "id": "k9zU9YaTTvFq",
        "outputId": "a449c5b4-b244-44b8-b756-10c82fe9546b"
      },
      "source": [
        "trade.head()"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>date</th>\n",
              "      <th>tic</th>\n",
              "      <th>open</th>\n",
              "      <th>high</th>\n",
              "      <th>low</th>\n",
              "      <th>close</th>\n",
              "      <th>volume</th>\n",
              "      <th>day</th>\n",
              "      <th>macd</th>\n",
              "      <th>boll_ub</th>\n",
              "      <th>boll_lb</th>\n",
              "      <th>rsi_30</th>\n",
              "      <th>cci_30</th>\n",
              "      <th>dx_30</th>\n",
              "      <th>close_30_sma</th>\n",
              "      <th>close_60_sma</th>\n",
              "      <th>turbulence</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2019-01-02</td>\n",
              "      <td>AAPL</td>\n",
              "      <td>38.722500</td>\n",
              "      <td>39.712502</td>\n",
              "      <td>38.557499</td>\n",
              "      <td>38.505024</td>\n",
              "      <td>148158800.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2.335191</td>\n",
              "      <td>334.139821</td>\n",
              "      <td>308.258949</td>\n",
              "      <td>47.826590</td>\n",
              "      <td>-27.772146</td>\n",
              "      <td>14.525416</td>\n",
              "      <td>314.202890</td>\n",
              "      <td>312.704339</td>\n",
              "      <td>51.409032</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2019-01-02</td>\n",
              "      <td>AXP</td>\n",
              "      <td>93.910004</td>\n",
              "      <td>96.269997</td>\n",
              "      <td>93.769997</td>\n",
              "      <td>92.319603</td>\n",
              "      <td>4175400.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>0.775741</td>\n",
              "      <td>335.611436</td>\n",
              "      <td>305.851699</td>\n",
              "      <td>46.156688</td>\n",
              "      <td>-67.860390</td>\n",
              "      <td>22.313462</td>\n",
              "      <td>314.159711</td>\n",
              "      <td>312.579659</td>\n",
              "      <td>51.409032</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2019-01-02</td>\n",
              "      <td>BA</td>\n",
              "      <td>316.190002</td>\n",
              "      <td>323.950012</td>\n",
              "      <td>313.709991</td>\n",
              "      <td>314.645142</td>\n",
              "      <td>3292200.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>-0.338838</td>\n",
              "      <td>336.485023</td>\n",
              "      <td>304.296566</td>\n",
              "      <td>46.806048</td>\n",
              "      <td>-72.067253</td>\n",
              "      <td>24.597730</td>\n",
              "      <td>314.064055</td>\n",
              "      <td>312.452928</td>\n",
              "      <td>51.409032</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2019-01-02</td>\n",
              "      <td>CAT</td>\n",
              "      <td>124.029999</td>\n",
              "      <td>127.879997</td>\n",
              "      <td>123.000000</td>\n",
              "      <td>118.671188</td>\n",
              "      <td>4783200.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>-0.655001</td>\n",
              "      <td>336.668731</td>\n",
              "      <td>303.510025</td>\n",
              "      <td>49.792068</td>\n",
              "      <td>-33.729612</td>\n",
              "      <td>14.054906</td>\n",
              "      <td>314.492846</td>\n",
              "      <td>312.392662</td>\n",
              "      <td>51.409032</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2019-01-02</td>\n",
              "      <td>CSCO</td>\n",
              "      <td>42.279999</td>\n",
              "      <td>43.200001</td>\n",
              "      <td>42.209999</td>\n",
              "      <td>40.057236</td>\n",
              "      <td>23833500.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>-0.159995</td>\n",
              "      <td>336.703986</td>\n",
              "      <td>304.162298</td>\n",
              "      <td>53.389344</td>\n",
              "      <td>31.217425</td>\n",
              "      <td>7.135987</td>\n",
              "      <td>315.392609</td>\n",
              "      <td>312.451774</td>\n",
              "      <td>51.409032</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         date   tic        open  ...  close_30_sma  close_60_sma  turbulence\n",
              "0  2019-01-02  AAPL   38.722500  ...    314.202890    312.704339   51.409032\n",
              "0  2019-01-02   AXP   93.910004  ...    314.159711    312.579659   51.409032\n",
              "0  2019-01-02    BA  316.190002  ...    314.064055    312.452928   51.409032\n",
              "0  2019-01-02   CAT  124.029999  ...    314.492846    312.392662   51.409032\n",
              "0  2019-01-02  CSCO   42.279999  ...    315.392609    312.451774   51.409032\n",
              "\n",
              "[5 rows x 17 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zYN573SOHhxG",
        "outputId": "0c82387b-5820-4825-8a66-9d67f024f88e"
      },
      "source": [
        "config.TECHNICAL_INDICATORS_LIST"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['macd',\n",
              " 'boll_ub',\n",
              " 'boll_lb',\n",
              " 'rsi_30',\n",
              " 'cci_30',\n",
              " 'dx_30',\n",
              " 'close_30_sma',\n",
              " 'close_60_sma']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Q2zqII8rMIqn",
        "outputId": "77687bc0-f774-4fb1-8b2d-71de3d3e0154"
      },
      "source": [
        "stock_dimension = len(train.tic.unique())\n",
        "state_space = 1 + 2*stock_dimension + len(config.TECHNICAL_INDICATORS_LIST)*stock_dimension\n",
        "print(f\"Stock Dimension: {stock_dimension}, State Space: {state_space}\")\n"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Stock Dimension: 30, State Space: 301\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AWyp84Ltto19"
      },
      "source": [
        "env_kwargs = {\n",
        "    \"hmax\": 100, \n",
        "    \"initial_amount\": 1000000, \n",
        "    \"buy_cost_pct\": 0.001,\n",
        "    \"sell_cost_pct\": 0.001,\n",
        "    \"state_space\": state_space, \n",
        "    \"stock_dim\": stock_dimension, \n",
        "    \"tech_indicator_list\": config.TECHNICAL_INDICATORS_LIST, \n",
        "    \"action_space\": stock_dimension, \n",
        "    \"reward_scaling\": 1e-4\n",
        "    \n",
        "}\n",
        "\n",
        "e_train_gym = StockTradingEnv(df = train, **env_kwargs)"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "64EoqOrQjiVf"
      },
      "source": [
        "## Environment for Training\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xwSvvPjutpqS",
        "outputId": "acbfc87e-c552-44f4-abb4-55f47e0de284"
      },
      "source": [
        "env_train, _ = e_train_gym.get_sb_env()\n",
        "print(type(env_train))"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'stable_baselines3.common.vec_env.dummy_vec_env.DummyVecEnv'>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HMNR5nHjh1iz"
      },
      "source": [
        "<a id='5'></a>\n",
        "# Part 6: Implement DRL Algorithms\n",
        "* The implementation of the DRL algorithms are based on **OpenAI Baselines** and **Stable Baselines**. Stable Baselines is a fork of OpenAI Baselines, with a major structural refactoring, and code cleanups.\n",
        "* FinRL library includes fine-tuned standard DRL algorithms, such as DQN, DDPG,\n",
        "Multi-Agent DDPG, PPO, SAC, A2C and TD3. We also allow users to\n",
        "design their own DRL algorithms by adapting these DRL algorithms."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "364PsqckttcQ"
      },
      "source": [
        "agent = DRLAgent(env = env_train)"
      ],
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YDmqOyF9h1iz"
      },
      "source": [
        "### Model Training: 5 models, A2C DDPG, PPO, TD3, SAC\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uijiWgkuh1jB"
      },
      "source": [
        "### Model 1: A2C\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GUCnkn-HIbmj",
        "outputId": "2fdb297a-8d35-4c7e-806f-de859d70e19e"
      },
      "source": [
        "agent = DRLAgent(env = env_train)\n",
        "model_a2c = agent.get_model(\"a2c\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{'n_steps': 5, 'ent_coef': 0.01, 'learning_rate': 0.0007}\n",
            "Using cuda device\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/opt/conda/lib/python3.6/site-packages/torch/cuda/__init__.py:81: UserWarning: \n",
            "    Found GPU1 GeForce GTX 760 which is of cuda capability 3.0.\n",
            "    PyTorch no longer supports this GPU because it is too old.\n",
            "    The minimum cuda capability that we support is 3.5.\n",
            "    \n",
            "  warnings.warn(old_gpu_warn % (d, name, major, capability[1]))\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0GVpkWGqH4-D",
        "outputId": "9eb09ba2-fc4b-46a1-ea3d-bd9b3bfefffd"
      },
      "source": [
        "trained_a2c = agent.train_model(model=model_a2c, \n",
        "                             tb_log_name='a2c',\n",
        "                             total_timesteps=100000)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Logging to tensorboard_log/a2c/a2c_4\n",
            "------------------------------------\n",
            "| time/                 |          |\n",
            "|    fps                | 81       |\n",
            "|    iterations         | 100      |\n",
            "|    time_elapsed       | 6        |\n",
            "|    total_timesteps    | 500      |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -42.6    |\n",
            "|    explained_variance | 0.338    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 99       |\n",
            "|    policy_loss        | -41.5    |\n",
            "|    std                | 1        |\n",
            "|    value_loss         | 1.22     |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| time/                 |          |\n",
            "|    fps                | 84       |\n",
            "|    iterations         | 200      |\n",
            "|    time_elapsed       | 11       |\n",
            "|    total_timesteps    | 1000     |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -42.6    |\n",
            "|    explained_variance | 0.0336   |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 199      |\n",
            "|    policy_loss        | 10.6     |\n",
            "|    std                | 1        |\n",
            "|    value_loss         | 1.05     |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| time/                 |          |\n",
            "|    fps                | 86       |\n",
            "|    iterations         | 300      |\n",
            "|    time_elapsed       | 17       |\n",
            "|    total_timesteps    | 1500     |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -42.6    |\n",
            "|    explained_variance | 0.0456   |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 299      |\n",
            "|    policy_loss        | -416     |\n",
            "|    std                | 1        |\n",
            "|    value_loss         | 106      |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| time/                 |          |\n",
            "|    fps                | 86       |\n",
            "|    iterations         | 400      |\n",
            "|    time_elapsed       | 23       |\n",
            "|    total_timesteps    | 2000     |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -42.7    |\n",
            "|    explained_variance | -0.0152  |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 399      |\n",
            "|    policy_loss        | -19.6    |\n",
            "|    std                | 1        |\n",
            "|    value_loss         | 1.25     |\n",
            "------------------------------------\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-22-e000b3012843>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m trained_a2c = agent.train_model(model=model_a2c, \n\u001b[1;32m      2\u001b[0m                              \u001b[0mtb_log_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'a2c'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m                              total_timesteps=100000)\n\u001b[0m",
            "\u001b[0;32m/home/finrl/model/models.py\u001b[0m in \u001b[0;36mtrain_model\u001b[0;34m(self, model, tb_log_name, total_timesteps)\u001b[0m\n\u001b[1;32m    120\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    121\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mtrain_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtb_log_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtotal_timesteps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m5000\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 122\u001b[0;31m         \u001b[0mmodel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlearn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtotal_timesteps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtotal_timesteps\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtb_log_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtb_log_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    123\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    124\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/stable_baselines3/a2c/a2c.py\u001b[0m in \u001b[0;36mlearn\u001b[0;34m(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps)\u001b[0m\n\u001b[1;32m    197\u001b[0m             \u001b[0mtb_log_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtb_log_name\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    198\u001b[0m             \u001b[0meval_log_path\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0meval_log_path\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 199\u001b[0;31m             \u001b[0mreset_num_timesteps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mreset_num_timesteps\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    200\u001b[0m         )\n",
            "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/stable_baselines3/common/on_policy_algorithm.py\u001b[0m in \u001b[0;36mlearn\u001b[0;34m(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps)\u001b[0m\n\u001b[1;32m    225\u001b[0m         \u001b[0;32mwhile\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnum_timesteps\u001b[0m \u001b[0;34m<\u001b[0m \u001b[0mtotal_timesteps\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    226\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 227\u001b[0;31m             \u001b[0mcontinue_training\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcollect_rollouts\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrollout_buffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_rollout_steps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mn_steps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    228\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    229\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mcontinue_training\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/stable_baselines3/common/on_policy_algorithm.py\u001b[0m in \u001b[0;36mcollect_rollouts\u001b[0;34m(self, env, callback, rollout_buffer, n_rollout_steps)\u001b[0m\n\u001b[1;32m    166\u001b[0m                 \u001b[0mclipped_actions\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mactions\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maction_space\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlow\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maction_space\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhigh\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    167\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 168\u001b[0;31m             \u001b[0mnew_obs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrewards\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdones\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minfos\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mclipped_actions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    169\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    170\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnum_timesteps\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnum_envs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/stable_baselines3/common/vec_env/base_vec_env.py\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, actions)\u001b[0m\n\u001b[1;32m    161\u001b[0m         \"\"\"\n\u001b[1;32m    162\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep_async\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mactions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 163\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep_wait\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    164\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    165\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mget_images\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mSequence\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mndarray\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/stable_baselines3/common/vec_env/dummy_vec_env.py\u001b[0m in \u001b[0;36mstep_wait\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     42\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0menv_idx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnum_envs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     43\u001b[0m             obs, self.buf_rews[env_idx], self.buf_dones[env_idx], self.buf_infos[env_idx] = self.envs[env_idx].step(\n\u001b[0;32m---> 44\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mactions\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0menv_idx\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     45\u001b[0m             )\n\u001b[1;32m     46\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuf_dones\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0menv_idx\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/home/finrl/env/env_stocktrading.py\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, actions)\u001b[0m\n\u001b[1;32m    248\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mturbulence_threshold\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    249\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mturbulence\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'turbulence'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 250\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstate\u001b[0m \u001b[0;34m=\u001b[0m  \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_update_state\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    251\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    252\u001b[0m             \u001b[0mend_total_asset\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstate\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m\\\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/home/finrl/env/env_stocktrading.py\u001b[0m in \u001b[0;36m_update_state\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    321\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    322\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_update_state\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 323\u001b[0;31m         \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtic\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munique\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    324\u001b[0m             \u001b[0;31m# for multiple stock\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    325\u001b[0m             \u001b[0mstate\u001b[0m \u001b[0;34m=\u001b[0m  \u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstate\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m\\\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/pandas/core/series.py\u001b[0m in \u001b[0;36munique\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1879\u001b[0m         \u001b[0mCategories\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mobject\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'a'\u001b[0m \u001b[0;34m<\u001b[0m \u001b[0;34m'b'\u001b[0m \u001b[0;34m<\u001b[0m \u001b[0;34m'c'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1880\u001b[0m         \"\"\"\n\u001b[0;32m-> 1881\u001b[0;31m         \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munique\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1882\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1883\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/pandas/core/base.py\u001b[0m in \u001b[0;36munique\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1263\u001b[0m                     \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1264\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1265\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0munique1d\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1266\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1267\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/pandas/core/algorithms.py\u001b[0m in \u001b[0;36munique\u001b[0;34m(values)\u001b[0m\n\u001b[1;32m    397\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    398\u001b[0m     \u001b[0mtable\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhtable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 399\u001b[0;31m     \u001b[0muniques\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtable\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munique\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    400\u001b[0m     \u001b[0muniques\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_reconstruct_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0muniques\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moriginal\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moriginal\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    401\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0muniques\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MRiOtrywfAo1"
      },
      "source": [
        "### Model 2: DDPG"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "M2YadjfnLwgt",
        "outputId": "3b2a8f89-0561-4083-a015-fbee11693037"
      },
      "source": [
        "agent = DRLAgent(env = env_train)\n",
        "model_ddpg = agent.get_model(\"ddpg\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{'batch_size': 128, 'buffer_size': 50000, 'learning_rate': 0.001}\n",
            "Using cpu device\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "background_save": true,
          "base_uri": "https://localhost:8080/"
        },
        "collapsed": true,
        "id": "tCDa78rqfO_a",
        "jupyter": {
          "outputs_hidden": true
        },
        "outputId": "f651f8be-4c93-4b1e-c88a-7e3a09976693"
      },
      "source": [
        "trained_ddpg = agent.train_model(model=model_ddpg, \n",
        "                             tb_log_name='ddpg',\n",
        "                             total_timesteps=50000)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Logging to tensorboard_log/ddpg/ddpg_1\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:3761309.8057632465\n",
            "total_reward:2761309.8057632465\n",
            "total_cost:  6807.077776350557\n",
            "total_trades:  39070\n",
            "Sharpe:  1.0173492167488003\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4423657.61673363\n",
            "total_reward:3423657.61673363\n",
            "total_cost:  1277.392035166502\n",
            "total_trades:  32819\n",
            "Sharpe:  0.8726982452731067\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4423657.61673363\n",
            "total_reward:3423657.61673363\n",
            "total_cost:  1277.392035166502\n",
            "total_trades:  32819\n",
            "Sharpe:  0.8726982452731067\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4423657.61673363\n",
            "total_reward:3423657.61673363\n",
            "total_cost:  1277.392035166502\n",
            "total_trades:  32819\n",
            "Sharpe:  0.8726982452731067\n",
            "=================================\n",
            "---------------------------------\n",
            "| time/              |          |\n",
            "|    episodes        | 4        |\n",
            "|    fps             | 38       |\n",
            "|    time_elapsed    | 258      |\n",
            "|    total timesteps | 10064    |\n",
            "| train/             |          |\n",
            "|    actor_loss      | -2.81    |\n",
            "|    critic_loss     | 272      |\n",
            "|    learning_rate   | 0.001    |\n",
            "|    n_updates       | 7548     |\n",
            "---------------------------------\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4423657.61673363\n",
            "total_reward:3423657.61673363\n",
            "total_cost:  1277.392035166502\n",
            "total_trades:  32819\n",
            "Sharpe:  0.8726982452731067\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4423657.61673363\n",
            "total_reward:3423657.61673363\n",
            "total_cost:  1277.392035166502\n",
            "total_trades:  32819\n",
            "Sharpe:  0.8726982452731067\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4423657.61673363\n",
            "total_reward:3423657.61673363\n",
            "total_cost:  1277.392035166502\n",
            "total_trades:  32819\n",
            "Sharpe:  0.8726982452731067\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4423657.61673363\n",
            "total_reward:3423657.61673363\n",
            "total_cost:  1277.392035166502\n",
            "total_trades:  32819\n",
            "Sharpe:  0.8726982452731067\n",
            "=================================\n",
            "---------------------------------\n",
            "| time/              |          |\n",
            "|    episodes        | 8        |\n",
            "|    fps             | 33       |\n",
            "|    time_elapsed    | 604      |\n",
            "|    total timesteps | 20128    |\n",
            "| train/             |          |\n",
            "|    actor_loss      | -8.32    |\n",
            "|    critic_loss     | 12.8     |\n",
            "|    learning_rate   | 0.001    |\n",
            "|    n_updates       | 17612    |\n",
            "---------------------------------\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4423657.61673363\n",
            "total_reward:3423657.61673363\n",
            "total_cost:  1277.392035166502\n",
            "total_trades:  32819\n",
            "Sharpe:  0.8726982452731067\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4423657.61673363\n",
            "total_reward:3423657.61673363\n",
            "total_cost:  1277.392035166502\n",
            "total_trades:  32819\n",
            "Sharpe:  0.8726982452731067\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4423657.61673363\n",
            "total_reward:3423657.61673363\n",
            "total_cost:  1277.392035166502\n",
            "total_trades:  32819\n",
            "Sharpe:  0.8726982452731067\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4423657.61673363\n",
            "total_reward:3423657.61673363\n",
            "total_cost:  1277.392035166502\n",
            "total_trades:  32819\n",
            "Sharpe:  0.8726982452731067\n",
            "=================================\n",
            "---------------------------------\n",
            "| time/              |          |\n",
            "|    episodes        | 12       |\n",
            "|    fps             | 31       |\n",
            "|    time_elapsed    | 953      |\n",
            "|    total timesteps | 30192    |\n",
            "| train/             |          |\n",
            "|    actor_loss      | -9.46    |\n",
            "|    critic_loss     | 4.31     |\n",
            "|    learning_rate   | 0.001    |\n",
            "|    n_updates       | 27676    |\n",
            "---------------------------------\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4423657.61673363\n",
            "total_reward:3423657.61673363\n",
            "total_cost:  1277.392035166502\n",
            "total_trades:  32819\n",
            "Sharpe:  0.8726982452731067\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4423657.61673363\n",
            "total_reward:3423657.61673363\n",
            "total_cost:  1277.392035166502\n",
            "total_trades:  32819\n",
            "Sharpe:  0.8726982452731067\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4423657.61673363\n",
            "total_reward:3423657.61673363\n",
            "total_cost:  1277.392035166502\n",
            "total_trades:  32819\n",
            "Sharpe:  0.8726982452731067\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4423657.61673363\n",
            "total_reward:3423657.61673363\n",
            "total_cost:  1277.392035166502\n",
            "total_trades:  32819\n",
            "Sharpe:  0.8726982452731067\n",
            "=================================\n",
            "---------------------------------\n",
            "| time/              |          |\n",
            "|    episodes        | 16       |\n",
            "|    fps             | 30       |\n",
            "|    time_elapsed    | 1309     |\n",
            "|    total timesteps | 40256    |\n",
            "| train/             |          |\n",
            "|    actor_loss      | -10.2    |\n",
            "|    critic_loss     | 3.19     |\n",
            "|    learning_rate   | 0.001    |\n",
            "|    n_updates       | 37740    |\n",
            "---------------------------------\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4423657.61673363\n",
            "total_reward:3423657.61673363\n",
            "total_cost:  1277.392035166502\n",
            "total_trades:  32819\n",
            "Sharpe:  0.8726982452731067\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4423657.61673363\n",
            "total_reward:3423657.61673363\n",
            "total_cost:  1277.392035166502\n",
            "total_trades:  32819\n",
            "Sharpe:  0.8726982452731067\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4423657.61673363\n",
            "total_reward:3423657.61673363\n",
            "total_cost:  1277.392035166502\n",
            "total_trades:  32819\n",
            "Sharpe:  0.8726982452731067\n",
            "=================================\n",
            "begin_total_asset:1000000\n",
            "end_total_asset:4423657.61673363\n",
            "total_reward:3423657.61673363\n",
            "total_cost:  1277.392035166502\n",
            "total_trades:  32819\n",
            "Sharpe:  0.8726982452731067\n",
            "=================================\n",
            "---------------------------------\n",
            "| time/              |          |\n",
            "|    episodes        | 20       |\n",
            "|    fps             | 30       |\n",
            "|    time_elapsed    | 1675     |\n",
            "|    total timesteps | 50320    |\n",
            "| train/             |          |\n",
            "|    actor_loss      | -11.1    |\n",
            "|    critic_loss     | 2.24     |\n",
            "|    learning_rate   | 0.001    |\n",
            "|    n_updates       | 47804    |\n",
            "---------------------------------\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_gDkU-j-fCmZ"
      },
      "source": [
        "### Model 3: PPO"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "y5D5PFUhMzSV",
        "outputId": "2716af5e-06e5-4eab-b071-a506c60a0475"
      },
      "source": [
        "agent = DRLAgent(env = env_train)\n",
        "PPO_PARAMS = {\n",
        "    \"n_steps\": 2048,\n",
        "    \"ent_coef\": 0.01,\n",
        "    \"learning_rate\": 0.00025,\n",
        "    \"batch_size\": 128,\n",
        "}\n",
        "model_ppo = agent.get_model(\"ppo\",model_kwargs = PPO_PARAMS)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{'n_steps': 2048, 'ent_coef': 0.01, 'learning_rate': 0.00025, 'batch_size': 128}\n",
            "Using cpu device\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "collapsed": true,
        "id": "Gt8eIQKYM4G3",
        "jupyter": {
          "outputs_hidden": true
        },
        "outputId": "1016cc05-58b6-45dc-c871-a322f1c3dc89"
      },
      "source": [
        "trained_ppo = agent.train_model(model=model_ppo, \n",
        "                             tb_log_name='ppo',\n",
        "                             total_timesteps=100000)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Logging to tensorboard_log/ppo/ppo_2\n",
            "-----------------------------\n",
            "| time/              |      |\n",
            "|    fps             | 104  |\n",
            "|    iterations      | 1    |\n",
            "|    time_elapsed    | 19   |\n",
            "|    total_timesteps | 2048 |\n",
            "-----------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 102         |\n",
            "|    iterations           | 2           |\n",
            "|    time_elapsed         | 39          |\n",
            "|    total_timesteps      | 4096        |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.014151055 |\n",
            "|    clip_fraction        | 0.212       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -42.6       |\n",
            "|    explained_variance   | -28.1       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 5.76        |\n",
            "|    n_updates            | 10          |\n",
            "|    policy_gradient_loss | -0.0277     |\n",
            "|    std                  | 1           |\n",
            "|    value_loss           | 12          |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 101         |\n",
            "|    iterations           | 3           |\n",
            "|    time_elapsed         | 60          |\n",
            "|    total_timesteps      | 6144        |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.016467014 |\n",
            "|    clip_fraction        | 0.186       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -42.6       |\n",
            "|    explained_variance   | -176        |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 9.99        |\n",
            "|    n_updates            | 20          |\n",
            "|    policy_gradient_loss | -0.0275     |\n",
            "|    std                  | 1           |\n",
            "|    value_loss           | 18.9        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 101         |\n",
            "|    iterations           | 4           |\n",
            "|    time_elapsed         | 80          |\n",
            "|    total_timesteps      | 8192        |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.020772668 |\n",
            "|    clip_fraction        | 0.191       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -42.6       |\n",
            "|    explained_variance   | -87.8       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 16.7        |\n",
            "|    n_updates            | 30          |\n",
            "|    policy_gradient_loss | -0.028      |\n",
            "|    std                  | 1           |\n",
            "|    value_loss           | 32.2        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 101         |\n",
            "|    iterations           | 5           |\n",
            "|    time_elapsed         | 101         |\n",
            "|    total_timesteps      | 10240       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.019156657 |\n",
            "|    clip_fraction        | 0.225       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -42.7       |\n",
            "|    explained_variance   | -81.3       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 11          |\n",
            "|    n_updates            | 40          |\n",
            "|    policy_gradient_loss | -0.0184     |\n",
            "|    std                  | 1           |\n",
            "|    value_loss           | 26.6        |\n",
            "-----------------------------------------\n",
            "----------------------------------------\n",
            "| time/                   |            |\n",
            "|    fps                  | 100        |\n",
            "|    iterations           | 6          |\n",
            "|    time_elapsed         | 122        |\n",
            "|    total_timesteps      | 12288      |\n",
            "| train/                  |            |\n",
            "|    approx_kl            | 0.02388929 |\n",
            "|    clip_fraction        | 0.223      |\n",
            "|    clip_range           | 0.2        |\n",
            "|    entropy_loss         | -42.7      |\n",
            "|    explained_variance   | -67        |\n",
            "|    learning_rate        | 0.00025    |\n",
            "|    loss                 | 7.86       |\n",
            "|    n_updates            | 50         |\n",
            "|    policy_gradient_loss | -0.0269    |\n",
            "|    std                  | 1.01       |\n",
            "|    value_loss           | 23         |\n",
            "----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 7           |\n",
            "|    time_elapsed         | 142         |\n",
            "|    total_timesteps      | 14336       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.023960019 |\n",
            "|    clip_fraction        | 0.21        |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -42.8       |\n",
            "|    explained_variance   | -58.1       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 6.32        |\n",
            "|    n_updates            | 60          |\n",
            "|    policy_gradient_loss | -0.0234     |\n",
            "|    std                  | 1.01        |\n",
            "|    value_loss           | 12          |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 8           |\n",
            "|    time_elapsed         | 163         |\n",
            "|    total_timesteps      | 16384       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.021991765 |\n",
            "|    clip_fraction        | 0.212       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -42.8       |\n",
            "|    explained_variance   | -36.4       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 9.39        |\n",
            "|    n_updates            | 70          |\n",
            "|    policy_gradient_loss | -0.0243     |\n",
            "|    std                  | 1.01        |\n",
            "|    value_loss           | 19.9        |\n",
            "-----------------------------------------\n",
            "----------------------------------------\n",
            "| time/                   |            |\n",
            "|    fps                  | 100        |\n",
            "|    iterations           | 9          |\n",
            "|    time_elapsed         | 183        |\n",
            "|    total_timesteps      | 18432      |\n",
            "| train/                  |            |\n",
            "|    approx_kl            | 0.01857267 |\n",
            "|    clip_fraction        | 0.205      |\n",
            "|    clip_range           | 0.2        |\n",
            "|    entropy_loss         | -42.9      |\n",
            "|    explained_variance   | -59.3      |\n",
            "|    learning_rate        | 0.00025    |\n",
            "|    loss                 | 8.22       |\n",
            "|    n_updates            | 80         |\n",
            "|    policy_gradient_loss | -0.0235    |\n",
            "|    std                  | 1.01       |\n",
            "|    value_loss           | 20.5       |\n",
            "----------------------------------------\n",
            "day: 2515, episode: 130\n",
            "begin_total_asset:1000000.00\n",
            "end_total_asset:3383653.45\n",
            "total_reward:2383653.45\n",
            "total_cost: 255155.22\n",
            "total_trades: 72649\n",
            "Sharpe: 0.863\n",
            "=================================\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 10          |\n",
            "|    time_elapsed         | 203         |\n",
            "|    total_timesteps      | 20480       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.022291362 |\n",
            "|    clip_fraction        | 0.213       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -42.9       |\n",
            "|    explained_variance   | -70.1       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 12.4        |\n",
            "|    n_updates            | 90          |\n",
            "|    policy_gradient_loss | -0.019      |\n",
            "|    std                  | 1.01        |\n",
            "|    value_loss           | 34.1        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 11          |\n",
            "|    time_elapsed         | 224         |\n",
            "|    total_timesteps      | 22528       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.017316487 |\n",
            "|    clip_fraction        | 0.22        |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -42.9       |\n",
            "|    explained_variance   | -159        |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 21.4        |\n",
            "|    n_updates            | 100         |\n",
            "|    policy_gradient_loss | -0.0182     |\n",
            "|    std                  | 1.01        |\n",
            "|    value_loss           | 38.8        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 12          |\n",
            "|    time_elapsed         | 244         |\n",
            "|    total_timesteps      | 24576       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.018951386 |\n",
            "|    clip_fraction        | 0.179       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -43         |\n",
            "|    explained_variance   | -25.3       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 11.4        |\n",
            "|    n_updates            | 110         |\n",
            "|    policy_gradient_loss | -0.0135     |\n",
            "|    std                  | 1.02        |\n",
            "|    value_loss           | 29.9        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 13          |\n",
            "|    time_elapsed         | 265         |\n",
            "|    total_timesteps      | 26624       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.033302963 |\n",
            "|    clip_fraction        | 0.298       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -43.1       |\n",
            "|    explained_variance   | -58.1       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 9.17        |\n",
            "|    n_updates            | 120         |\n",
            "|    policy_gradient_loss | -0.0236     |\n",
            "|    std                  | 1.02        |\n",
            "|    value_loss           | 28.3        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 14          |\n",
            "|    time_elapsed         | 285         |\n",
            "|    total_timesteps      | 28672       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.027676268 |\n",
            "|    clip_fraction        | 0.278       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -43.1       |\n",
            "|    explained_variance   | -91.7       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 12.8        |\n",
            "|    n_updates            | 130         |\n",
            "|    policy_gradient_loss | -0.0192     |\n",
            "|    std                  | 1.02        |\n",
            "|    value_loss           | 32.7        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 15          |\n",
            "|    time_elapsed         | 306         |\n",
            "|    total_timesteps      | 30720       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.027800845 |\n",
            "|    clip_fraction        | 0.233       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -43.2       |\n",
            "|    explained_variance   | -85.9       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 26          |\n",
            "|    n_updates            | 140         |\n",
            "|    policy_gradient_loss | -0.0217     |\n",
            "|    std                  | 1.02        |\n",
            "|    value_loss           | 40.1        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 16          |\n",
            "|    time_elapsed         | 326         |\n",
            "|    total_timesteps      | 32768       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.016968882 |\n",
            "|    clip_fraction        | 0.219       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -43.3       |\n",
            "|    explained_variance   | -71.3       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 10.2        |\n",
            "|    n_updates            | 150         |\n",
            "|    policy_gradient_loss | -0.0209     |\n",
            "|    std                  | 1.02        |\n",
            "|    value_loss           | 26.9        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 17          |\n",
            "|    time_elapsed         | 347         |\n",
            "|    total_timesteps      | 34816       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.022131229 |\n",
            "|    clip_fraction        | 0.215       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -43.3       |\n",
            "|    explained_variance   | -15.7       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 20.6        |\n",
            "|    n_updates            | 160         |\n",
            "|    policy_gradient_loss | -0.0153     |\n",
            "|    std                  | 1.03        |\n",
            "|    value_loss           | 49.1        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 18          |\n",
            "|    time_elapsed         | 368         |\n",
            "|    total_timesteps      | 36864       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.029286291 |\n",
            "|    clip_fraction        | 0.266       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -43.4       |\n",
            "|    explained_variance   | -43.9       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 13.2        |\n",
            "|    n_updates            | 170         |\n",
            "|    policy_gradient_loss | -0.015      |\n",
            "|    std                  | 1.03        |\n",
            "|    value_loss           | 19.9        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 19          |\n",
            "|    time_elapsed         | 388         |\n",
            "|    total_timesteps      | 38912       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.027719798 |\n",
            "|    clip_fraction        | 0.24        |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -43.4       |\n",
            "|    explained_variance   | -131        |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 16.8        |\n",
            "|    n_updates            | 180         |\n",
            "|    policy_gradient_loss | -0.0183     |\n",
            "|    std                  | 1.03        |\n",
            "|    value_loss           | 34          |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 20          |\n",
            "|    time_elapsed         | 409         |\n",
            "|    total_timesteps      | 40960       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.022764063 |\n",
            "|    clip_fraction        | 0.217       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -43.5       |\n",
            "|    explained_variance   | -63.1       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 22.5        |\n",
            "|    n_updates            | 190         |\n",
            "|    policy_gradient_loss | -0.0186     |\n",
            "|    std                  | 1.03        |\n",
            "|    value_loss           | 37.9        |\n",
            "-----------------------------------------\n",
            "----------------------------------------\n",
            "| time/                   |            |\n",
            "|    fps                  | 100        |\n",
            "|    iterations           | 21         |\n",
            "|    time_elapsed         | 429        |\n",
            "|    total_timesteps      | 43008      |\n",
            "| train/                  |            |\n",
            "|    approx_kl            | 0.02734076 |\n",
            "|    clip_fraction        | 0.208      |\n",
            "|    clip_range           | 0.2        |\n",
            "|    entropy_loss         | -43.5      |\n",
            "|    explained_variance   | -113       |\n",
            "|    learning_rate        | 0.00025    |\n",
            "|    loss                 | 21         |\n",
            "|    n_updates            | 200        |\n",
            "|    policy_gradient_loss | -0.0201    |\n",
            "|    std                  | 1.03       |\n",
            "|    value_loss           | 60.7       |\n",
            "----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 22          |\n",
            "|    time_elapsed         | 450         |\n",
            "|    total_timesteps      | 45056       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.023378888 |\n",
            "|    clip_fraction        | 0.277       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -43.6       |\n",
            "|    explained_variance   | -57         |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 19.6        |\n",
            "|    n_updates            | 210         |\n",
            "|    policy_gradient_loss | -0.0153     |\n",
            "|    std                  | 1.03        |\n",
            "|    value_loss           | 38.9        |\n",
            "-----------------------------------------\n",
            "day: 2515, episode: 140\n",
            "begin_total_asset:1000000.00\n",
            "end_total_asset:5223199.40\n",
            "total_reward:4223199.40\n",
            "total_cost: 235269.98\n",
            "total_trades: 71552\n",
            "Sharpe: 1.128\n",
            "=================================\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 23          |\n",
            "|    time_elapsed         | 470         |\n",
            "|    total_timesteps      | 47104       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.025331508 |\n",
            "|    clip_fraction        | 0.29        |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -43.6       |\n",
            "|    explained_variance   | -61.4       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 19.7        |\n",
            "|    n_updates            | 220         |\n",
            "|    policy_gradient_loss | -0.0119     |\n",
            "|    std                  | 1.04        |\n",
            "|    value_loss           | 34.8        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 24          |\n",
            "|    time_elapsed         | 491         |\n",
            "|    total_timesteps      | 49152       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.025766762 |\n",
            "|    clip_fraction        | 0.231       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -43.7       |\n",
            "|    explained_variance   | -64.7       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 20.4        |\n",
            "|    n_updates            | 230         |\n",
            "|    policy_gradient_loss | -0.0187     |\n",
            "|    std                  | 1.04        |\n",
            "|    value_loss           | 47.4        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 25          |\n",
            "|    time_elapsed         | 511         |\n",
            "|    total_timesteps      | 51200       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.041917183 |\n",
            "|    clip_fraction        | 0.278       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -43.8       |\n",
            "|    explained_variance   | -34         |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 22.2        |\n",
            "|    n_updates            | 240         |\n",
            "|    policy_gradient_loss | -0.0164     |\n",
            "|    std                  | 1.04        |\n",
            "|    value_loss           | 48          |\n",
            "-----------------------------------------\n",
            "---------------------------------------\n",
            "| time/                   |           |\n",
            "|    fps                  | 100       |\n",
            "|    iterations           | 26        |\n",
            "|    time_elapsed         | 531       |\n",
            "|    total_timesteps      | 53248     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0367468 |\n",
            "|    clip_fraction        | 0.273     |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -43.8     |\n",
            "|    explained_variance   | -48.1     |\n",
            "|    learning_rate        | 0.00025   |\n",
            "|    loss                 | 21.5      |\n",
            "|    n_updates            | 250       |\n",
            "|    policy_gradient_loss | -0.00821  |\n",
            "|    std                  | 1.04      |\n",
            "|    value_loss           | 39.5      |\n",
            "---------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 27          |\n",
            "|    time_elapsed         | 552         |\n",
            "|    total_timesteps      | 55296       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.024581099 |\n",
            "|    clip_fraction        | 0.211       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -43.9       |\n",
            "|    explained_variance   | -198        |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 32.1        |\n",
            "|    n_updates            | 260         |\n",
            "|    policy_gradient_loss | -0.0106     |\n",
            "|    std                  | 1.05        |\n",
            "|    value_loss           | 58.2        |\n",
            "-----------------------------------------\n",
            "----------------------------------------\n",
            "| time/                   |            |\n",
            "|    fps                  | 100        |\n",
            "|    iterations           | 28         |\n",
            "|    time_elapsed         | 573        |\n",
            "|    total_timesteps      | 57344      |\n",
            "| train/                  |            |\n",
            "|    approx_kl            | 0.02569989 |\n",
            "|    clip_fraction        | 0.209      |\n",
            "|    clip_range           | 0.2        |\n",
            "|    entropy_loss         | -43.9      |\n",
            "|    explained_variance   | -161       |\n",
            "|    learning_rate        | 0.00025    |\n",
            "|    loss                 | 25.1       |\n",
            "|    n_updates            | 270        |\n",
            "|    policy_gradient_loss | -0.0137    |\n",
            "|    std                  | 1.05       |\n",
            "|    value_loss           | 55         |\n",
            "----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 29          |\n",
            "|    time_elapsed         | 593         |\n",
            "|    total_timesteps      | 59392       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.032340243 |\n",
            "|    clip_fraction        | 0.252       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -44         |\n",
            "|    explained_variance   | -24.1       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 9.23        |\n",
            "|    n_updates            | 280         |\n",
            "|    policy_gradient_loss | -0.0167     |\n",
            "|    std                  | 1.05        |\n",
            "|    value_loss           | 34          |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 30          |\n",
            "|    time_elapsed         | 613         |\n",
            "|    total_timesteps      | 61440       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.018233867 |\n",
            "|    clip_fraction        | 0.239       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -44         |\n",
            "|    explained_variance   | -34.1       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 28.1        |\n",
            "|    n_updates            | 290         |\n",
            "|    policy_gradient_loss | -0.0158     |\n",
            "|    std                  | 1.05        |\n",
            "|    value_loss           | 41.2        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 31          |\n",
            "|    time_elapsed         | 634         |\n",
            "|    total_timesteps      | 63488       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.030068567 |\n",
            "|    clip_fraction        | 0.152       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -44         |\n",
            "|    explained_variance   | -26.1       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 19.2        |\n",
            "|    n_updates            | 300         |\n",
            "|    policy_gradient_loss | -0.0121     |\n",
            "|    std                  | 1.05        |\n",
            "|    value_loss           | 64.9        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 32          |\n",
            "|    time_elapsed         | 654         |\n",
            "|    total_timesteps      | 65536       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.024889158 |\n",
            "|    clip_fraction        | 0.27        |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -44.1       |\n",
            "|    explained_variance   | -31.2       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 37.3        |\n",
            "|    n_updates            | 310         |\n",
            "|    policy_gradient_loss | -0.0148     |\n",
            "|    std                  | 1.05        |\n",
            "|    value_loss           | 58          |\n",
            "-----------------------------------------\n",
            "----------------------------------------\n",
            "| time/                   |            |\n",
            "|    fps                  | 100        |\n",
            "|    iterations           | 33         |\n",
            "|    time_elapsed         | 674        |\n",
            "|    total_timesteps      | 67584      |\n",
            "| train/                  |            |\n",
            "|    approx_kl            | 0.03883523 |\n",
            "|    clip_fraction        | 0.234      |\n",
            "|    clip_range           | 0.2        |\n",
            "|    entropy_loss         | -44.1      |\n",
            "|    explained_variance   | -39.9      |\n",
            "|    learning_rate        | 0.00025    |\n",
            "|    loss                 | 24.5       |\n",
            "|    n_updates            | 320        |\n",
            "|    policy_gradient_loss | -0.0121    |\n",
            "|    std                  | 1.05       |\n",
            "|    value_loss           | 84.4       |\n",
            "----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 34          |\n",
            "|    time_elapsed         | 695         |\n",
            "|    total_timesteps      | 69632       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.024309162 |\n",
            "|    clip_fraction        | 0.225       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -44.2       |\n",
            "|    explained_variance   | -12.9       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 8.79        |\n",
            "|    n_updates            | 330         |\n",
            "|    policy_gradient_loss | -0.015      |\n",
            "|    std                  | 1.06        |\n",
            "|    value_loss           | 23.8        |\n",
            "-----------------------------------------\n",
            "day: 2515, episode: 150\n",
            "begin_total_asset:1000000.00\n",
            "end_total_asset:6320097.75\n",
            "total_reward:5320097.75\n",
            "total_cost: 222029.44\n",
            "total_trades: 69973\n",
            "Sharpe: 1.250\n",
            "=================================\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 35          |\n",
            "|    time_elapsed         | 715         |\n",
            "|    total_timesteps      | 71680       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.024664927 |\n",
            "|    clip_fraction        | 0.183       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -44.2       |\n",
            "|    explained_variance   | -17.2       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 15.3        |\n",
            "|    n_updates            | 340         |\n",
            "|    policy_gradient_loss | -0.0141     |\n",
            "|    std                  | 1.06        |\n",
            "|    value_loss           | 48.7        |\n",
            "-----------------------------------------\n",
            "----------------------------------------\n",
            "| time/                   |            |\n",
            "|    fps                  | 100        |\n",
            "|    iterations           | 36         |\n",
            "|    time_elapsed         | 735        |\n",
            "|    total_timesteps      | 73728      |\n",
            "| train/                  |            |\n",
            "|    approx_kl            | 0.03882557 |\n",
            "|    clip_fraction        | 0.207      |\n",
            "|    clip_range           | 0.2        |\n",
            "|    entropy_loss         | -44.3      |\n",
            "|    explained_variance   | -27.1      |\n",
            "|    learning_rate        | 0.00025    |\n",
            "|    loss                 | 50.3       |\n",
            "|    n_updates            | 350        |\n",
            "|    policy_gradient_loss | -0.0141    |\n",
            "|    std                  | 1.06       |\n",
            "|    value_loss           | 93.7       |\n",
            "----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 37          |\n",
            "|    time_elapsed         | 756         |\n",
            "|    total_timesteps      | 75776       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.022156972 |\n",
            "|    clip_fraction        | 0.214       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -44.3       |\n",
            "|    explained_variance   | -23.9       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 26.5        |\n",
            "|    n_updates            | 360         |\n",
            "|    policy_gradient_loss | -0.0161     |\n",
            "|    std                  | 1.06        |\n",
            "|    value_loss           | 71.7        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 38          |\n",
            "|    time_elapsed         | 776         |\n",
            "|    total_timesteps      | 77824       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.022767432 |\n",
            "|    clip_fraction        | 0.223       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -44.4       |\n",
            "|    explained_variance   | -17.5       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 23.8        |\n",
            "|    n_updates            | 370         |\n",
            "|    policy_gradient_loss | -0.0154     |\n",
            "|    std                  | 1.06        |\n",
            "|    value_loss           | 38.7        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 39          |\n",
            "|    time_elapsed         | 797         |\n",
            "|    total_timesteps      | 79872       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.020827759 |\n",
            "|    clip_fraction        | 0.178       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -44.4       |\n",
            "|    explained_variance   | -56         |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 36.3        |\n",
            "|    n_updates            | 380         |\n",
            "|    policy_gradient_loss | -0.00964    |\n",
            "|    std                  | 1.07        |\n",
            "|    value_loss           | 82.1        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 40          |\n",
            "|    time_elapsed         | 817         |\n",
            "|    total_timesteps      | 81920       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.013000591 |\n",
            "|    clip_fraction        | 0.132       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -44.5       |\n",
            "|    explained_variance   | -23         |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 14          |\n",
            "|    n_updates            | 390         |\n",
            "|    policy_gradient_loss | -0.0162     |\n",
            "|    std                  | 1.07        |\n",
            "|    value_loss           | 63.1        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 41          |\n",
            "|    time_elapsed         | 837         |\n",
            "|    total_timesteps      | 83968       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.021172233 |\n",
            "|    clip_fraction        | 0.19        |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -44.5       |\n",
            "|    explained_variance   | -26.6       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 47.3        |\n",
            "|    n_updates            | 400         |\n",
            "|    policy_gradient_loss | -0.0191     |\n",
            "|    std                  | 1.07        |\n",
            "|    value_loss           | 98          |\n",
            "-----------------------------------------\n",
            "----------------------------------------\n",
            "| time/                   |            |\n",
            "|    fps                  | 100        |\n",
            "|    iterations           | 42         |\n",
            "|    time_elapsed         | 858        |\n",
            "|    total_timesteps      | 86016      |\n",
            "| train/                  |            |\n",
            "|    approx_kl            | 0.02925424 |\n",
            "|    clip_fraction        | 0.16       |\n",
            "|    clip_range           | 0.2        |\n",
            "|    entropy_loss         | -44.6      |\n",
            "|    explained_variance   | -33.8      |\n",
            "|    learning_rate        | 0.00025    |\n",
            "|    loss                 | 59.2       |\n",
            "|    n_updates            | 410        |\n",
            "|    policy_gradient_loss | -0.0117    |\n",
            "|    std                  | 1.07       |\n",
            "|    value_loss           | 163        |\n",
            "----------------------------------------\n",
            "----------------------------------------\n",
            "| time/                   |            |\n",
            "|    fps                  | 100        |\n",
            "|    iterations           | 43         |\n",
            "|    time_elapsed         | 878        |\n",
            "|    total_timesteps      | 88064      |\n",
            "| train/                  |            |\n",
            "|    approx_kl            | 0.01635669 |\n",
            "|    clip_fraction        | 0.138      |\n",
            "|    clip_range           | 0.2        |\n",
            "|    entropy_loss         | -44.6      |\n",
            "|    explained_variance   | -28.9      |\n",
            "|    learning_rate        | 0.00025    |\n",
            "|    loss                 | 57.9       |\n",
            "|    n_updates            | 420        |\n",
            "|    policy_gradient_loss | -0.0135    |\n",
            "|    std                  | 1.07       |\n",
            "|    value_loss           | 122        |\n",
            "----------------------------------------\n",
            "----------------------------------------\n",
            "| time/                   |            |\n",
            "|    fps                  | 100        |\n",
            "|    iterations           | 44         |\n",
            "|    time_elapsed         | 898        |\n",
            "|    total_timesteps      | 90112      |\n",
            "| train/                  |            |\n",
            "|    approx_kl            | 0.03150232 |\n",
            "|    clip_fraction        | 0.188      |\n",
            "|    clip_range           | 0.2        |\n",
            "|    entropy_loss         | -44.6      |\n",
            "|    explained_variance   | -20.9      |\n",
            "|    learning_rate        | 0.00025    |\n",
            "|    loss                 | 45.1       |\n",
            "|    n_updates            | 430        |\n",
            "|    policy_gradient_loss | -0.0222    |\n",
            "|    std                  | 1.07       |\n",
            "|    value_loss           | 84.9       |\n",
            "----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 45          |\n",
            "|    time_elapsed         | 919         |\n",
            "|    total_timesteps      | 92160       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.035686597 |\n",
            "|    clip_fraction        | 0.335       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -44.6       |\n",
            "|    explained_variance   | -5.37       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 4.34        |\n",
            "|    n_updates            | 440         |\n",
            "|    policy_gradient_loss | -0.0119     |\n",
            "|    std                  | 1.07        |\n",
            "|    value_loss           | 14.2        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 46          |\n",
            "|    time_elapsed         | 940         |\n",
            "|    total_timesteps      | 94208       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.028425248 |\n",
            "|    clip_fraction        | 0.293       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -44.7       |\n",
            "|    explained_variance   | -4.65       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 22.9        |\n",
            "|    n_updates            | 450         |\n",
            "|    policy_gradient_loss | -0.0184     |\n",
            "|    std                  | 1.07        |\n",
            "|    value_loss           | 26.4        |\n",
            "-----------------------------------------\n",
            "day: 2515, episode: 160\n",
            "begin_total_asset:1000000.00\n",
            "end_total_asset:5044806.56\n",
            "total_reward:4044806.56\n",
            "total_cost: 237117.70\n",
            "total_trades: 70270\n",
            "Sharpe: 1.271\n",
            "=================================\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 47          |\n",
            "|    time_elapsed         | 960         |\n",
            "|    total_timesteps      | 96256       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.034343738 |\n",
            "|    clip_fraction        | 0.299       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -44.7       |\n",
            "|    explained_variance   | -3.42       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 17.6        |\n",
            "|    n_updates            | 460         |\n",
            "|    policy_gradient_loss | -0.0185     |\n",
            "|    std                  | 1.08        |\n",
            "|    value_loss           | 56.9        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 48          |\n",
            "|    time_elapsed         | 981         |\n",
            "|    total_timesteps      | 98304       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.017608875 |\n",
            "|    clip_fraction        | 0.231       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -44.8       |\n",
            "|    explained_variance   | -17.7       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 17.6        |\n",
            "|    n_updates            | 470         |\n",
            "|    policy_gradient_loss | -0.0054     |\n",
            "|    std                  | 1.08        |\n",
            "|    value_loss           | 35.9        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 49          |\n",
            "|    time_elapsed         | 1001        |\n",
            "|    total_timesteps      | 100352      |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.024408635 |\n",
            "|    clip_fraction        | 0.168       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -44.8       |\n",
            "|    explained_variance   | -4.44       |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 16.2        |\n",
            "|    n_updates            | 480         |\n",
            "|    policy_gradient_loss | -0.00922    |\n",
            "|    std                  | 1.08        |\n",
            "|    value_loss           | 50.8        |\n",
            "-----------------------------------------\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Zpv4S0-fDBv"
      },
      "source": [
        "### Model 4: TD3"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JSAHhV4Xc-bh",
        "outputId": "e531db14-aab4-47d1-cc15-02c893ec66c9"
      },
      "source": [
        "agent = DRLAgent(env = env_train)\n",
        "TD3_PARAMS = {\"batch_size\": 100, \n",
        "              \"buffer_size\": 1000000, \n",
        "              \"learning_rate\": 0.001}\n",
        "\n",
        "model_td3 = agent.get_model(\"td3\",model_kwargs = TD3_PARAMS)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{'batch_size': 100, 'buffer_size': 1000000, 'learning_rate': 0.001}\n",
            "Using cpu device\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OSRxNYAxdKpU",
        "outputId": "ddc4193c-884b-4a2c-9e49-31397e2cfbec"
      },
      "source": [
        "trained_td3 = agent.train_model(model=model_td3, \n",
        "                             tb_log_name='td3',\n",
        "                             total_timesteps=30000)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Logging to tensorboard_log/td3/td3_2\n",
            "---------------------------------\n",
            "| time/              |          |\n",
            "|    episodes        | 4        |\n",
            "|    fps             | 33       |\n",
            "|    time_elapsed    | 296      |\n",
            "|    total timesteps | 10064    |\n",
            "| train/             |          |\n",
            "|    actor_loss      | 67.9     |\n",
            "|    critic_loss     | 979      |\n",
            "|    learning_rate   | 0.001    |\n",
            "|    n_updates       | 7548     |\n",
            "---------------------------------\n",
            "day: 2515, episode: 10\n",
            "begin_total_asset:1000000.00\n",
            "end_total_asset:4438572.29\n",
            "total_reward:3438572.29\n",
            "total_cost: 1038.05\n",
            "total_trades: 40290\n",
            "Sharpe: 1.049\n",
            "=================================\n",
            "---------------------------------\n",
            "| time/              |          |\n",
            "|    episodes        | 8        |\n",
            "|    fps             | 30       |\n",
            "|    time_elapsed    | 669      |\n",
            "|    total timesteps | 20128    |\n",
            "| train/             |          |\n",
            "|    actor_loss      | 54       |\n",
            "|    critic_loss     | 199      |\n",
            "|    learning_rate   | 0.001    |\n",
            "|    n_updates       | 17612    |\n",
            "---------------------------------\n",
            "---------------------------------\n",
            "| time/              |          |\n",
            "|    episodes        | 12       |\n",
            "|    fps             | 28       |\n",
            "|    time_elapsed    | 1052     |\n",
            "|    total timesteps | 30192    |\n",
            "| train/             |          |\n",
            "|    actor_loss      | 41.4     |\n",
            "|    critic_loss     | 25.2     |\n",
            "|    learning_rate   | 0.001    |\n",
            "|    n_updates       | 27676    |\n",
            "---------------------------------\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Dr49PotrfG01"
      },
      "source": [
        "### Model 5: SAC"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xwOhVjqRkCdM",
        "outputId": "74adb7a3-6392-4a0e-e720-24cac0391cf1"
      },
      "source": [
        "agent = DRLAgent(env = env_train)\n",
        "SAC_PARAMS = {\n",
        "    \"batch_size\": 128,\n",
        "    \"buffer_size\": 1000000,\n",
        "    \"learning_rate\": 0.0001,\n",
        "    \"learning_starts\": 100,\n",
        "    \"ent_coef\": \"auto_0.1\",\n",
        "}\n",
        "\n",
        "model_sac = agent.get_model(\"sac\",model_kwargs = SAC_PARAMS)"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{'batch_size': 128, 'buffer_size': 1000000, 'learning_rate': 0.0001, 'learning_starts': 100, 'ent_coef': 'auto_0.1'}\n",
            "Using cpu device\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "K8RSdKCckJyH",
        "outputId": "9d647c28-72ed-4b13-b3b4-cf3c4bf28bf8"
      },
      "source": [
        "trained_sac = agent.train_model(model=model_sac, \n",
        "                             tb_log_name='sac',\n",
        "                             total_timesteps=80000)"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Logging to tensorboard_log/sac/sac_1\n",
            "----------------------------------\n",
            "| environment/        |          |\n",
            "|    portfolio_value  | 4.5e+06  |\n",
            "|    total_cost       | 1.25e+05 |\n",
            "|    total_reward     | 3.5e+06  |\n",
            "|    total_reward_pct | 350      |\n",
            "|    total_trades     | 64851    |\n",
            "| time/               |          |\n",
            "|    episodes         | 4        |\n",
            "|    fps              | 26       |\n",
            "|    time_elapsed     | 380      |\n",
            "|    total timesteps  | 10064    |\n",
            "| train/              |          |\n",
            "|    actor_loss       | 558      |\n",
            "|    critic_loss      | 1.74e+03 |\n",
            "|    ent_coef         | 0.108    |\n",
            "|    ent_coef_loss    | -104     |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    n_updates        | 9963     |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| environment/        |          |\n",
            "|    portfolio_value  | 3e+06    |\n",
            "|    total_cost       | 9.44e+03 |\n",
            "|    total_reward     | 2e+06    |\n",
            "|    total_reward_pct | 200      |\n",
            "|    total_trades     | 46290    |\n",
            "| time/               |          |\n",
            "|    episodes         | 8        |\n",
            "|    fps              | 26       |\n",
            "|    time_elapsed     | 768      |\n",
            "|    total timesteps  | 20128    |\n",
            "| train/              |          |\n",
            "|    actor_loss       | 230      |\n",
            "|    critic_loss      | 120      |\n",
            "|    ent_coef         | 0.0402   |\n",
            "|    ent_coef_loss    | -120     |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    n_updates        | 20027    |\n",
            "----------------------------------\n",
            "day: 2515, episode: 10\n",
            "begin_total_asset: 1000000.00\n",
            "end_total_asset: 2547174.32\n",
            "total_reward: 1547174.32\n",
            "total_cost: 6037.44\n",
            "total_trades: 45126\n",
            "Sharpe: 0.630\n",
            "=================================\n",
            "----------------------------------\n",
            "| environment/        |          |\n",
            "|    portfolio_value  | 2.89e+06 |\n",
            "|    total_cost       | 5.35e+03 |\n",
            "|    total_reward     | 1.89e+06 |\n",
            "|    total_reward_pct | 189      |\n",
            "|    total_trades     | 44813    |\n",
            "| time/               |          |\n",
            "|    episodes         | 12       |\n",
            "|    fps              | 26       |\n",
            "|    time_elapsed     | 1156     |\n",
            "|    total timesteps  | 30192    |\n",
            "| train/              |          |\n",
            "|    actor_loss       | 118      |\n",
            "|    critic_loss      | 10.7     |\n",
            "|    ent_coef         | 0.0149   |\n",
            "|    ent_coef_loss    | -137     |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    n_updates        | 30091    |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| environment/        |          |\n",
            "|    portfolio_value  | 3.61e+06 |\n",
            "|    total_cost       | 3.87e+03 |\n",
            "|    total_reward     | 2.61e+06 |\n",
            "|    total_reward_pct | 261      |\n",
            "|    total_trades     | 46192    |\n",
            "| time/               |          |\n",
            "|    episodes         | 16       |\n",
            "|    fps              | 26       |\n",
            "|    time_elapsed     | 1546     |\n",
            "|    total timesteps  | 40256    |\n",
            "| train/              |          |\n",
            "|    actor_loss       | 69.9     |\n",
            "|    critic_loss      | 6        |\n",
            "|    ent_coef         | 0.0056   |\n",
            "|    ent_coef_loss    | -135     |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    n_updates        | 40155    |\n",
            "----------------------------------\n",
            "day: 2515, episode: 20\n",
            "begin_total_asset: 1000000.00\n",
            "end_total_asset: 3289629.43\n",
            "total_reward: 2289629.43\n",
            "total_cost: 2535.14\n",
            "total_trades: 46913\n",
            "Sharpe: 0.759\n",
            "=================================\n",
            "----------------------------------\n",
            "| environment/        |          |\n",
            "|    portfolio_value  | 3.36e+06 |\n",
            "|    total_cost       | 2.57e+03 |\n",
            "|    total_reward     | 2.36e+06 |\n",
            "|    total_reward_pct | 236      |\n",
            "|    total_trades     | 46859    |\n",
            "| time/               |          |\n",
            "|    episodes         | 20       |\n",
            "|    fps              | 25       |\n",
            "|    time_elapsed     | 1936     |\n",
            "|    total timesteps  | 50320    |\n",
            "| train/              |          |\n",
            "|    actor_loss       | 41.8     |\n",
            "|    critic_loss      | 9.2      |\n",
            "|    ent_coef         | 0.00218  |\n",
            "|    ent_coef_loss    | -70.5    |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    n_updates        | 50219    |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| environment/        |          |\n",
            "|    portfolio_value  | 3.5e+06  |\n",
            "|    total_cost       | 3.47e+03 |\n",
            "|    total_reward     | 2.5e+06  |\n",
            "|    total_reward_pct | 250      |\n",
            "|    total_trades     | 46581    |\n",
            "| time/               |          |\n",
            "|    episodes         | 24       |\n",
            "|    fps              | 25       |\n",
            "|    time_elapsed     | 2331     |\n",
            "|    total timesteps  | 60384    |\n",
            "| train/              |          |\n",
            "|    actor_loss       | 29.6     |\n",
            "|    critic_loss      | 4.64     |\n",
            "|    ent_coef         | 0.00147  |\n",
            "|    ent_coef_loss    | -7.26    |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    n_updates        | 60283    |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| environment/        |          |\n",
            "|    portfolio_value  | 3.58e+06 |\n",
            "|    total_cost       | 2.91e+03 |\n",
            "|    total_reward     | 2.58e+06 |\n",
            "|    total_reward_pct | 258      |\n",
            "|    total_trades     | 43527    |\n",
            "| time/               |          |\n",
            "|    episodes         | 28       |\n",
            "|    fps              | 25       |\n",
            "|    time_elapsed     | 2737     |\n",
            "|    total timesteps  | 70448    |\n",
            "| train/              |          |\n",
            "|    actor_loss       | 21.7     |\n",
            "|    critic_loss      | 7.59     |\n",
            "|    ent_coef         | 0.00158  |\n",
            "|    ent_coef_loss    | 5.83     |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    n_updates        | 70347    |\n",
            "----------------------------------\n",
            "day: 2515, episode: 30\n",
            "begin_total_asset: 1000000.00\n",
            "end_total_asset: 3603070.58\n",
            "total_reward: 2603070.58\n",
            "total_cost: 3036.97\n",
            "total_trades: 43502\n",
            "Sharpe: 0.810\n",
            "=================================\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f2wZgkQXh1jE"
      },
      "source": [
        "## Trading\n",
        "Assume that we have $1,000,000 initial capital at 2019-01-01. We use the DDPG model to trade Dow jones 30 stocks."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bEv5KGC8h1jE"
      },
      "source": [
        "### Set turbulence threshold\n",
        "Set the turbulence threshold to be greater than the maximum of insample turbulence data, if current turbulence index is greater than the threshold, then we assume that the current market is volatile"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "efwBi84ch1jE"
      },
      "source": [
        "data_turbulence = processed_full[(processed_full.date<'2019-01-01') & (processed_full.date>='2009-01-01')]\n",
        "insample_turbulence = data_turbulence.drop_duplicates(subset=['date'])"
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VHZMBpSqh1jG",
        "outputId": "ff6fc21f-0a54-4cad-8c70-5a617206959f"
      },
      "source": [
        "insample_turbulence.turbulence.describe()"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "count    2516.000000\n",
              "mean       33.278198\n",
              "std        33.999895\n",
              "min         0.000000\n",
              "25%        15.233636\n",
              "50%        25.166759\n",
              "75%        39.289961\n",
              "max       332.851269\n",
              "Name: turbulence, dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yuwDPkV9h1jL"
      },
      "source": [
        "turbulence_threshold = np.quantile(insample_turbulence.turbulence.values,1)"
      ],
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wwoz_7VSh1jO",
        "outputId": "cb1dc2ad-a6ef-439b-f33d-358e516656ca"
      },
      "source": [
        "turbulence_threshold"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "332.85126857983164"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U5mmgQF_h1jQ"
      },
      "source": [
        "### Trade\n",
        "\n",
        "DRL model needs to update periodically in order to take full advantage of the data, ideally we need to retrain our model yearly, quarterly, or monthly. We also need to tune the parameters along the way, in this notebook I only use the in-sample data from 2009-01 to 2018-12 to tune the parameters once, so there is some alpha decay here as the length of trade date extends. \n",
        "\n",
        "Numerous hyperparameters – e.g. the learning rate, the total number of samples to train on – influence the learning process and are usually determined by testing some variations."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eLOnL5eYh1jR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ab0fa9e7-787e-4f71-d51a-1e6da0538970"
      },
      "source": [
        "trade = data_split(processed_full, '2019-01-01','2021-01-01')\n",
        "e_trade_gym = StockTradingEnv(df = trade, turbulence_threshold = 380, **env_kwargs)\n",
        "# env_trade, obs_trade = e_trade_gym.get_sb_env()\n",
        "\n",
        "df_account_value, df_actions = DRLAgent.DRL_prediction(\n",
        "    model=model_sac, \n",
        "    environment = e_trade_gym)"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "hit end!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ERxw3KqLkcP4",
        "outputId": "1de56e59-ce4e-494e-f542-e5b5cd2fb699"
      },
      "source": [
        "df_account_value.shape"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(505, 2)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "2yRkNguY5yvp",
        "outputId": "92b404d6-30ca-4ef2-9264-07a336ce7e8d"
      },
      "source": [
        "df_account_value.head()"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>date</th>\n",
              "      <th>account_value</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2019-01-02</td>\n",
              "      <td>1.000000e+06</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2019-01-03</td>\n",
              "      <td>9.963386e+05</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2019-01-04</td>\n",
              "      <td>1.006066e+06</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2019-01-07</td>\n",
              "      <td>1.008263e+06</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2019-01-08</td>\n",
              "      <td>1.017674e+06</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         date  account_value\n",
              "0  2019-01-02   1.000000e+06\n",
              "1  2019-01-03   9.963386e+05\n",
              "2  2019-01-04   1.006066e+06\n",
              "3  2019-01-07   1.008263e+06\n",
              "4  2019-01-08   1.017674e+06"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "nFlK5hNbWVFk",
        "outputId": "89c3d5a6-c6eb-4f61-d829-60f1b6d27f94"
      },
      "source": [
        "df_actions.head()"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>AAPL</th>\n",
              "      <th>AXP</th>\n",
              "      <th>BA</th>\n",
              "      <th>CAT</th>\n",
              "      <th>CSCO</th>\n",
              "      <th>CVX</th>\n",
              "      <th>DD</th>\n",
              "      <th>DIS</th>\n",
              "      <th>GS</th>\n",
              "      <th>HD</th>\n",
              "      <th>IBM</th>\n",
              "      <th>INTC</th>\n",
              "      <th>JNJ</th>\n",
              "      <th>JPM</th>\n",
              "      <th>KO</th>\n",
              "      <th>MCD</th>\n",
              "      <th>MMM</th>\n",
              "      <th>MRK</th>\n",
              "      <th>MSFT</th>\n",
              "      <th>NKE</th>\n",
              "      <th>PFE</th>\n",
              "      <th>PG</th>\n",
              "      <th>RTX</th>\n",
              "      <th>TRV</th>\n",
              "      <th>UNH</th>\n",
              "      <th>V</th>\n",
              "      <th>VZ</th>\n",
              "      <th>WBA</th>\n",
              "      <th>WMT</th>\n",
              "      <th>XOM</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>97</td>\n",
              "      <td>99</td>\n",
              "      <td>99</td>\n",
              "      <td>0</td>\n",
              "      <td>48</td>\n",
              "      <td>0</td>\n",
              "      <td>98</td>\n",
              "      <td>86</td>\n",
              "      <td>13</td>\n",
              "      <td>98</td>\n",
              "      <td>0</td>\n",
              "      <td>82</td>\n",
              "      <td>0</td>\n",
              "      <td>93</td>\n",
              "      <td>29</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>88</td>\n",
              "      <td>0</td>\n",
              "      <td>76</td>\n",
              "      <td>16</td>\n",
              "      <td>0</td>\n",
              "      <td>50</td>\n",
              "      <td>0</td>\n",
              "      <td>58</td>\n",
              "      <td>0</td>\n",
              "      <td>97</td>\n",
              "      <td>63</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "      <td>97</td>\n",
              "      <td>94</td>\n",
              "      <td>95</td>\n",
              "      <td>68</td>\n",
              "      <td>94</td>\n",
              "      <td>0</td>\n",
              "      <td>98</td>\n",
              "      <td>76</td>\n",
              "      <td>-13</td>\n",
              "      <td>88</td>\n",
              "      <td>0</td>\n",
              "      <td>82</td>\n",
              "      <td>0</td>\n",
              "      <td>67</td>\n",
              "      <td>96</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>80</td>\n",
              "      <td>0</td>\n",
              "      <td>-76</td>\n",
              "      <td>91</td>\n",
              "      <td>0</td>\n",
              "      <td>80</td>\n",
              "      <td>0</td>\n",
              "      <td>73</td>\n",
              "      <td>0</td>\n",
              "      <td>97</td>\n",
              "      <td>96</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>96</td>\n",
              "      <td>97</td>\n",
              "      <td>75</td>\n",
              "      <td>33</td>\n",
              "      <td>64</td>\n",
              "      <td>0</td>\n",
              "      <td>88</td>\n",
              "      <td>93</td>\n",
              "      <td>40</td>\n",
              "      <td>98</td>\n",
              "      <td>0</td>\n",
              "      <td>99</td>\n",
              "      <td>0</td>\n",
              "      <td>90</td>\n",
              "      <td>79</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>88</td>\n",
              "      <td>0</td>\n",
              "      <td>98</td>\n",
              "      <td>93</td>\n",
              "      <td>0</td>\n",
              "      <td>75</td>\n",
              "      <td>0</td>\n",
              "      <td>89</td>\n",
              "      <td>0</td>\n",
              "      <td>93</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0</td>\n",
              "      <td>84</td>\n",
              "      <td>96</td>\n",
              "      <td>90</td>\n",
              "      <td>91</td>\n",
              "      <td>85</td>\n",
              "      <td>0</td>\n",
              "      <td>99</td>\n",
              "      <td>99</td>\n",
              "      <td>-9</td>\n",
              "      <td>97</td>\n",
              "      <td>0</td>\n",
              "      <td>90</td>\n",
              "      <td>0</td>\n",
              "      <td>38</td>\n",
              "      <td>93</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>70</td>\n",
              "      <td>0</td>\n",
              "      <td>-83</td>\n",
              "      <td>50</td>\n",
              "      <td>0</td>\n",
              "      <td>92</td>\n",
              "      <td>0</td>\n",
              "      <td>96</td>\n",
              "      <td>0</td>\n",
              "      <td>93</td>\n",
              "      <td>90</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>99</td>\n",
              "      <td>95</td>\n",
              "      <td>84</td>\n",
              "      <td>17</td>\n",
              "      <td>16</td>\n",
              "      <td>0</td>\n",
              "      <td>99</td>\n",
              "      <td>93</td>\n",
              "      <td>18</td>\n",
              "      <td>97</td>\n",
              "      <td>0</td>\n",
              "      <td>72</td>\n",
              "      <td>0</td>\n",
              "      <td>-50</td>\n",
              "      <td>90</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>26</td>\n",
              "      <td>0</td>\n",
              "      <td>-15</td>\n",
              "      <td>-23</td>\n",
              "      <td>0</td>\n",
              "      <td>79</td>\n",
              "      <td>0</td>\n",
              "      <td>62</td>\n",
              "      <td>0</td>\n",
              "      <td>83</td>\n",
              "      <td>22</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   AAPL  AXP  BA  CAT  CSCO  CVX  DD  DIS  ...  RTX  TRV  UNH  V  VZ  WBA  WMT  XOM\n",
              "0     0   97  99   99     0   48   0   98  ...   16    0   50  0  58    0   97   63\n",
              "1     0   97  94   95    68   94   0   98  ...   91    0   80  0  73    0   97   96\n",
              "2     0   96  97   75    33   64   0   88  ...   93    0   75  0  89    0   93    4\n",
              "3     0   84  96   90    91   85   0   99  ...   50    0   92  0  96    0   93   90\n",
              "4     0   99  95   84    17   16   0   99  ...  -23    0   79  0  62    0   83   22\n",
              "\n",
              "[5 rows x 30 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W6vvNSC6h1jZ"
      },
      "source": [
        "<a id='6'></a>\n",
        "# Part 7: Backtest Our Strategy\n",
        "Backtesting plays a key role in evaluating the performance of a trading strategy. Automated backtesting tool is preferred because it reduces the human error. We usually use the Quantopian pyfolio package to backtest our trading strategies. It is easy to use and consists of various individual plots that provide a comprehensive image of the performance of a trading strategy."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lr2zX7ZxNyFQ"
      },
      "source": [
        "<a id='6.1'></a>\n",
        "## 7.1 BackTestStats\n",
        "pass in df_account_value, this information is stored in env class\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Nzkr9yv-AdV_",
        "outputId": "3d4406ac-3d5c-44fd-a55b-711559104014"
      },
      "source": [
        "print(\"==============Get Backtest Results===========\")\n",
        "now = datetime.datetime.now().strftime('%Y%m%d-%Hh%M')\n",
        "\n",
        "perf_stats_all = backtest_stats(account_value=df_account_value)\n",
        "perf_stats_all = pd.DataFrame(perf_stats_all)\n",
        "perf_stats_all.to_csv(\"./\"+config.RESULTS_DIR+\"/perf_stats_all_\"+now+'.csv')"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "==============Get Backtest Results===========\n",
            "Annual return          0.186750\n",
            "Cumulative returns     0.409333\n",
            "Annual volatility      0.225053\n",
            "Sharpe ratio           0.875060\n",
            "Calmar ratio           0.748224\n",
            "Stability              0.324021\n",
            "Max drawdown          -0.249591\n",
            "Omega ratio            1.185554\n",
            "Sortino ratio          1.270542\n",
            "Skew                        NaN\n",
            "Kurtosis                    NaN\n",
            "Tail ratio             0.981014\n",
            "Daily value at risk   -0.027573\n",
            "dtype: float64\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9U6Suru3h1jc"
      },
      "source": [
        "<a id='6.2'></a>\n",
        "## 7.2 BackTestPlot"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 982
        },
        "id": "lKRGftSS7pNM",
        "outputId": "3b1199bf-fa09-47ca-a215-0d25100076de"
      },
      "source": [
        "print(\"==============Compare to DJIA===========\")\n",
        "%matplotlib inline\n",
        "# S&P 500: ^GSPC\n",
        "# Dow Jones Index: ^DJI\n",
        "# NASDAQ 100: ^NDX\n",
        "backtest_plot(df_account_value, \n",
        "             baseline_ticker = '^DJI', \n",
        "             baseline_start = '2019-01-01',\n",
        "             baseline_end = '2021-01-01')"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "==============Compare to DJIA===========\n",
            "\r[*********************100%***********************]  1 of 1 completed\n",
            "Shape of DataFrame:  (505, 8)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\"><th>Start date</th><td colspan=2>2019-01-02</td></tr>\n",
              "    <tr style=\"text-align: right;\"><th>End date</th><td colspan=2>2020-12-31</td></tr>\n",
              "    <tr style=\"text-align: right;\"><th>Total months</th><td colspan=2>24</td></tr>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Backtest</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Annual return</th>\n",
              "      <td>18.7%</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Cumulative returns</th>\n",
              "      <td>40.9%</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Annual volatility</th>\n",
              "      <td>22.5%</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Sharpe ratio</th>\n",
              "      <td>0.88</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Calmar ratio</th>\n",
              "      <td>0.75</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Stability</th>\n",
              "      <td>0.32</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Max drawdown</th>\n",
              "      <td>-25.0%</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Omega ratio</th>\n",
              "      <td>1.19</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Sortino ratio</th>\n",
              "      <td>1.27</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Skew</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Kurtosis</th>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Tail ratio</th>\n",
              "      <td>0.98</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Daily value at risk</th>\n",
              "      <td>-2.8%</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Alpha</th>\n",
              "      <td>0.10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Beta</th>\n",
              "      <td>0.61</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "error",
          "ename": "AttributeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-32-00ee700acb02>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      7\u001b[0m              \u001b[0mbaseline_ticker\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'^DJI'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m              \u001b[0mbaseline_start\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'2019-01-01'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 9\u001b[0;31m              baseline_end = '2021-01-01')\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/finrl/trade/backtest.py\u001b[0m in \u001b[0;36mbacktest_plot\u001b[0;34m(account_value, baseline_start, baseline_end, baseline_ticker, value_col_name)\u001b[0m\n\u001b[1;32m     51\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0mpyfolio\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mplotting\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mplotting_context\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfont_scale\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1.1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     52\u001b[0m         pyfolio.create_full_tear_sheet(\n\u001b[0;32m---> 53\u001b[0;31m             \u001b[0mreturns\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtest_returns\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbenchmark_rets\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbaseline_returns\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mset_context\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     54\u001b[0m         )\n\u001b[1;32m     55\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pyfolio/tears.py\u001b[0m in \u001b[0;36mcreate_full_tear_sheet\u001b[0;34m(returns, positions, transactions, market_data, benchmark_rets, slippage, live_start_date, sector_mappings, bayesian, round_trips, estimate_intraday, hide_positions, cone_std, bootstrap, unadjusted_returns, style_factor_panel, sectors, caps, shares_held, volumes, percentile, turnover_denom, set_context, factor_returns, factor_loadings, pos_in_dollars, header_rows, factor_partitions)\u001b[0m\n\u001b[1;32m    209\u001b[0m         \u001b[0mturnover_denom\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mturnover_denom\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    210\u001b[0m         \u001b[0mheader_rows\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mheader_rows\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 211\u001b[0;31m         set_context=set_context)\n\u001b[0m\u001b[1;32m    212\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    213\u001b[0m     create_interesting_times_tear_sheet(returns,\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pyfolio/plotting.py\u001b[0m in \u001b[0;36mcall_w_context\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     52\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     53\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 54\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     55\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mcall_w_context\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     56\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pyfolio/tears.py\u001b[0m in \u001b[0;36mcreate_returns_tear_sheet\u001b[0;34m(returns, positions, transactions, live_start_date, cone_std, benchmark_rets, bootstrap, turnover_denom, header_rows, return_fig)\u001b[0m\n\u001b[1;32m    502\u001b[0m                              header_rows=header_rows)\n\u001b[1;32m    503\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 504\u001b[0;31m     \u001b[0mplotting\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshow_worst_drawdown_periods\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mreturns\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    505\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    506\u001b[0m     \u001b[0mvertical_sections\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m11\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pyfolio/plotting.py\u001b[0m in \u001b[0;36mshow_worst_drawdown_periods\u001b[0;34m(returns, top)\u001b[0m\n\u001b[1;32m   1662\u001b[0m     \"\"\"\n\u001b[1;32m   1663\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1664\u001b[0;31m     \u001b[0mdrawdown_df\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtimeseries\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgen_drawdown_table\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mreturns\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtop\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtop\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1665\u001b[0m     utils.print_table(\n\u001b[1;32m   1666\u001b[0m         \u001b[0mdrawdown_df\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msort_values\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Net drawdown in %'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mascending\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pyfolio/timeseries.py\u001b[0m in \u001b[0;36mgen_drawdown_table\u001b[0;34m(returns, top)\u001b[0m\n\u001b[1;32m   1006\u001b[0m         df_drawdowns.loc[i, 'Peak date'] = (peak.to_pydatetime()\n\u001b[1;32m   1007\u001b[0m                                             .strftime('%Y-%m-%d'))\n\u001b[0;32m-> 1008\u001b[0;31m         df_drawdowns.loc[i, 'Valley date'] = (valley.to_pydatetime()\n\u001b[0m\u001b[1;32m   1009\u001b[0m                                               .strftime('%Y-%m-%d'))\n\u001b[1;32m   1010\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrecovery\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfloat\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mAttributeError\u001b[0m: 'numpy.int64' object has no attribute 'to_pydatetime'"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SlLT9_5WN478"
      },
      "source": [
        "<a id='6.3'></a>\n",
        "## 7.3 Baseline Stats"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YktexHcqh1jc",
        "outputId": "b5cf94e6-1dee-44ed-9c4a-c096cada5ff5"
      },
      "source": [
        "print(\"==============Get Baseline Stats===========\")\n",
        "baseline_df = get_baseline(\n",
        "        ticker=\"^DJI\", start = '2019-01-01',\n",
        "                                  end = '2021-01-01')\n",
        "\n",
        "stats = backtest_stats(baseline_df, value_col_name = 'close')\n"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "==============Get Baseline Stats===========\n",
            "\r[*********************100%***********************]  1 of 1 completed\n",
            "Shape of DataFrame:  (505, 8)\n",
            "Annual return          0.144674\n",
            "Cumulative returns     0.310981\n",
            "Annual volatility      0.274619\n",
            "Sharpe ratio           0.631418\n",
            "Calmar ratio           0.390102\n",
            "Stability              0.116677\n",
            "Max drawdown          -0.370862\n",
            "Omega ratio            1.149365\n",
            "Sortino ratio          0.870084\n",
            "Skew                        NaN\n",
            "Kurtosis                    NaN\n",
            "Tail ratio             0.860710\n",
            "Daily value at risk   -0.033911\n",
            "dtype: float64\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BzBaE63H3RLc"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}