intermediate_notebooks/examples/Dask_with_cuDF_and_XGBoost.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dask with cuDF and XGBoost\n",
    "#### By Paul Hendricks\n",
    "-------\n",
    "\n",
    "While the world’s data doubles each year, CPU computing has hit a brick wall with the end of Moore’s law. For the same reasons, scientific computing and deep learning has turned to NVIDIA GPU acceleration, data analytics and machine learning where GPU acceleration is ideal. \n",
    "\n",
    "NVIDIA created RAPIDS – an open-source data analytics and machine learning acceleration platform that leverages GPUs to accelerate computations. RAPIDS is based on Python, has pandas-like and Scikit-Learn-like interfaces, is built on Apache Arrow in-memory data format, and can scale from 1 to multi-GPU to multi-nodes. RAPIDS integrates easily into the world’s most popular data science Python-based workflows. RAPIDS accelerates data science end-to-end – from data prep, to machine learning, to deep learning. And through Arrow, Spark users can easily move data into the RAPIDS platform for acceleration.\n",
    "\n",
    "In this notebook, we will show how to quickly setup Dask and train an XGBoost model using cuDF.\n",
    "\n",
    "**Table of Contents**\n",
    "\n",
    "* Setup\n",
    "* Load Libraries\n",
    "* Setup Dask\n",
    "* Hello World!\n",
    "* Sleeping in Parallel\n",
    "* Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "This notebook was tested using the `nvcr.io/nvidia/rapidsai/rapidsai:0.5-cuda10.0-runtime-ubuntu18.04-gcc7-py3.7` Docker container from [NVIDIA GPU Cloud](https://ngc.nvidia.com) and run on the NVIDIA Tesla V100 GPU. Please be aware that your system may be different and you may need to modify the code or install packages to run the below examples. \n",
    "\n",
    "If you think you have found a bug or an error, please file an issue here: https://github.com/rapidsai/notebooks/issues\n",
    "\n",
    "Before we begin, let's check out our hardware setup by running the `nvidia-smi` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's see what CUDA version we have:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvcc --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cudf\n",
    "import dask.dataframe as dd\n",
    "from dask.delayed import delayed\n",
    "from dask.distributed import Client, wait\n",
    "import dask_xgboost as dxgb_gpu\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "from pandas.util.testing import assert_frame_equal\n",
    "import subprocess\n",
    "import time\n",
    "import xgboost as xgb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup Dask\n",
    "\n",
    "Dask is a library the allows for parallelized computing. Written in Python, it allows one to schedule tasks and do so dynamically as well handle large data structures - similar to those found in NumPy and Pandas. In the subsequent tutorials, we'll show how to use Dask with Pandas and cuDF and how we can use both to accelerate common ETL tasks as well as build ML models like XGBoost.\n",
    "\n",
    "To learn more about Dask, check out the documentation here: http://docs.dask.org/en/latest/\n",
    "\n",
    "Dask operates using a concept of a \"Client\" and \"workers\". The client tells the workers what tasks to perform and when to perform. Typically, we set the number of works to be equal to the number of computing resources we have available to us. For example, wer might set `n_workers = 8` if we have 8 CPU cores on our machine that can each operate in parallel. This allows us to take advantage of all of our computing resources and enjoy the most benefits from parallelization.\n",
    "\n",
    "Dask is a first class citizen in the world of General Purpose GPU computing and the RAPIDS ecosystem makes it very easy to use Dask with cuDF and XGBoost. As we see below, we can inititate a Cluster and Client using only few lines of code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dask_cuda import LocalCUDACluster\n",
    "\n",
    "cluster = LocalCUDACluster()\n",
    "client = Client(cluster)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's show our current Dask status. We should see the IP Address for our Scheduler as well the the number of workers in our Cluster. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show current Dask status\n",
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also see the status and more information at the Dashboard, found at `http://<scheduler_uri>/status`. You can ignore this for now, we'll dive into this in subsequent tutorials."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create GPU DataFrames using Dask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def initialize_rmm_pool():\n",
    "    from librmm_cffi import librmm_config as rmm_cfg\n",
    "\n",
    "    rmm_cfg.use_pool_allocator = True\n",
    "    #rmm_cfg.initial_pool_size = 2<<30 # set to 2GiB. Default is 1/2 total GPU memory\n",
    "    import cudf\n",
    "    return cudf._gdf.rmm_initialize()\n",
    "\n",
    "def initialize_rmm_no_pool():\n",
    "    from librmm_cffi import librmm_config as rmm_cfg\n",
    "    \n",
    "    rmm_cfg.use_pool_allocator = False\n",
    "    import cudf\n",
    "    return cudf._gdf.rmm_initialize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.run(initialize_rmm_pool)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_pandas():\n",
    "    X = np.random.randint(2, size=(10000, 101)).astype(np.float32)\n",
    "    df = pd.DataFrame({'fea%d'%i:X[:,i] for i in range(X.shape[1])})\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_cudf(df):\n",
    "    gdf = cudf.DataFrame.from_pandas(df)\n",
    "    return gdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_features(gdf):\n",
    "    column_names = gdf.columns\n",
    "    return gdf.loc[:, column_names[1:]]\n",
    "\n",
    "\n",
    "def make_labels(gdf):\n",
    "    column_names = gdf.columns\n",
    "    return gdf.loc[:, column_names[:1]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_workers = 4\n",
    "dfs = [delayed(make_pandas)() for _ in range(n_workers)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gdfs = [delayed(make_cudf)(df) for df in dfs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.run(cudf._gdf.rmm_finalize)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.run(initialize_rmm_no_pool)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gpu_dfs = [[delayed(make_features)(gdf), delayed(make_labels)(gdf)] for gdf in gdfs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gpu_dfs = [delayed(xgb.DMatrix)(gpu_df[0], gpu_df[1]) for gpu_df in gpu_dfs]\n",
    "gpu_dfs = [gpu_df.persist() for gpu_df in gpu_dfs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wait(gpu_dfs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dxgb_gpu_params = {\n",
    "    'nround':            1000,\n",
    "    'max_depth':         8,\n",
    "    'max_leaves':        2**8,\n",
    "    'alpha':             0.9,\n",
    "    'eta':               0.1,\n",
    "    'gamma':             0.1,\n",
    "    'learning_rate':     0.1,\n",
    "    'subsample':         1,\n",
    "    'reg_lambda':        1,\n",
    "    'scale_pos_weight':  2,\n",
    "    'min_child_weight':  30,\n",
    "    'tree_method':       'gpu_hist',\n",
    "    'n_gpus':            1,\n",
    "    'distributed_dask':  True,\n",
    "    'loss':              'ls',\n",
    "    'objective':         'gpu:reg:linear',\n",
    "    'max_features':      'auto',\n",
    "    'criterion':         'friedman_mse',\n",
    "    'grow_policy':       'lossguide',\n",
    "    'verbose':           True\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "labels = None\n",
    "bst = dxgb_gpu.train(client, dxgb_gpu_params, gpu_dfs, labels, num_boost_round=dxgb_gpu_params['nround'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "To learn more about RAPIDS, be sure to check out: \n",
    "\n",
    "* [Open Source Website](http://rapids.ai)\n",
    "* [GitHub](https://github.com/rapidsai/)\n",
    "* [Press Release](https://nvidianews.nvidia.com/news/nvidia-introduces-rapids-open-source-gpu-acceleration-platform-for-large-scale-data-analytics-and-machine-learning)\n",
    "* [NVIDIA Blog](https://blogs.nvidia.com/blog/2018/10/10/rapids-data-science-open-source-community/)\n",
    "* [Developer Blog](https://devblogs.nvidia.com/gpu-accelerated-analytics-rapids/)\n",
    "* [NVIDIA Data Science Webpage](https://www.nvidia.com/en-us/deep-learning-ai/solutions/data-science/)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}