From 844124c68ce3a46434bdb1df3b3684ec4fd2d46d Mon Sep 17 00:00:00 2001 From: Khedia Date: Wed, 18 Jul 2018 14:18:07 -0700 Subject: [PATCH 1/6] fix R windows install docs --- docs/install/windows_setup.md | 139 ++++++++++++++++++++++++---------- 1 file changed, 98 insertions(+), 41 deletions(-) diff --git a/docs/install/windows_setup.md b/docs/install/windows_setup.md index 8a2b1c141700..e10425ba30dd 100755 --- a/docs/install/windows_setup.md +++ b/docs/install/windows_setup.md @@ -62,7 +62,7 @@ Next, we install ```graphviz``` library that we use for visualizing network grap We have installed MXNet core library. Next, we will install MXNet interface package for programming language of your choice: - [Python](#install-the-mxnet-package-for-python) -- [R](#install-mxnet-for-r) +- [R](#install-mxnet-package-for-r) - [Julia](#install-the-mxnet-package-for-julia) - **Scala** is not yet available for Windows @@ -91,7 +91,7 @@ Done! We have installed MXNet with Python interface. Run below commands to verif ``` We actually did a small tensor computation using MXNet! You are all set with MXNet on your Windows machine. -## Install MXNet for R +## Install MXNet Package for R MXNet for R is available for both CPUs and GPUs. ### Installing MXNet on a Computer with a CPU Processor @@ -101,7 +101,7 @@ To install MXNet on a computer with a CPU processor, choose from two options: * Use the prebuilt binary package * Build the library from source code -#### Installing MXNet with the Prebuilt Binary Package +#### Installing MXNet with the Prebuilt Binary Package(CPU) For Windows users, MXNet provides prebuilt binary packages. You can install the package directly in the R console. @@ -114,81 +114,138 @@ For CPU-only package: install.packages("mxnet") ``` -For GPU-enabled package: +#### Building MXNet from Source Code(CPU) +1. Clone the MXNet github repo. -```r - cran <- getOption("repos") - cran["dmlc"] <- "https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/R/CRAN/GPU" - options(repos = cran) - install.packages("mxnet") +```sh +git clone --recursive https://github.com/dmlc/mxnet ``` -#### Building MXNet from Source Code +The `--recursive` is to clone all the submodules used by MXNet. You will be editing the ```"/mxnet/R-package"``` folder. +2. Download prebuilt GPU-enabled MXNet libraries for Windows from [Windows release](https://github.com/yajiedesign/mxnet/releases). You will need `mxnet_x64_vc14_cpu.7z` and `prebuildbase_win10_x64_vc14.7z` where X stands for your CUDA toolkit version +3. Create a folder called ```R-package/inst/libs/x64```. MXNet supports only 64-bit operating systems, so you need the x64 folder. +4. Copy the following shared libraries (.dll files) into the ```R-package/inst/libs/x64``` folder: +``` +libgcc_s_seh-1.dll +libgfortran-3.dll +libmxnet.dll +libmxnet.lib +libopenblas.dll +libquadmath-0.dll +mxnet.dll +unzip.exe +unzip32.dll +vcomp140.dll +wget.exe +``` +These dlls can be found in `prebuildbase_win10_x64_vc14/3rdparty`, `mxnet_x64_vc14_cpu/build`, `mxnet_x64_vc14_cpu/lib`. -Run the following commands to install the MXNet dependencies and build the MXNet R package. +5. Copy the header files from `dmlc`, `mxnet`, `mxshadow` and `nnvm` from mxnet_x64_vc14_cpu/include and mxnet_x64_vc14_cpu/nvnm/include into `./R-package/inst/include`. It should look like: -```r - Rscript -e "install.packages('devtools', repo = 'https://cloud.r-project.org/')" ``` +./R-package/inst +└── include + ├── dmlc + ├── mxnet + ├── mshadow + └── nnvm + +``` +6. Make sure that R is added to your ```PATH``` in the environment variables. Running the ```where R``` command at the command prompt should return the location. +7. Also make sure that Rtools in installed and added to your ```PATH``` in the environment variables. +8. Temporary patch - im2rec currently results in crashes during the build. Remove the im2rec.h and im2rec.cc files in R-package/src/ from cloned repository and comment out the two im2rec lines in R-package/src/mxnet.cc. +9. Now open the Windows CMD with admin rights and change the directory to the `mxnet` folder(cloned repository). Then use the following commands +to build R package: -```bash - cd R-package - Rscript -e "library(devtools); library(methods); options(repos=c(CRAN='https://cloud.r-project.org/')); install_deps(dependencies = TRUE)" - cd .. - make rpkg +```bat +echo import(Rcpp) > R-package\NAMESPACE +echo import(methods) >> R-package\NAMESPACE +Rscript -e "install.packages('devtools', repos = 'https://cloud.r-project.org')" +cd R-package +Rscript -e "library(devtools); library(methods); options(repos=c(CRAN='https://cloud.r-project.org')); install_deps(dependencies = TRUE)" +cd .. + +R CMD INSTALL --no-multiarch R-package + +Rscript -e "require(mxnet); mxnet:::mxnet.export('R-package')" +rm R-package/NAMESPACE +Rscript -e "require(devtools); install_version('roxygen2', version = '5.0.1', repos = 'https://cloud.r-project.org/', quiet = TRUE)" +Rscript -e "require(roxygen2); roxygen2::roxygenise('R-package')" + +R CMD INSTALL --build --no-multiarch R-package ``` + ### Installing MXNet on a Computer with a GPU Processor +To install MXNet on a computer with a GPU processor, choose from two options: + +* Use the prebuilt binary package +* Build the library from source code + +However, few dependencies remains same for both options. You will need the following: -To install MXNet R package on a computer with a GPU processor, you need the following: +* Install Microsoft Visual Studio 2017 (required by CUDA) -* Microsoft Visual Studio 2013 +* Install [NVidia CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit) -* The NVidia CUDA Toolkit +* Download and install [CuDNN 7](https://developer.nvidia.com/cudnn) (to provide a Deep Neural Network library) -* The MXNet package +Note: A pre-requisite to above softwares is [Nvidia-drivers](http://www.nvidia.com/Download/index.aspx?lang=en-us) which we assume is installed. -* CuDNN (to provide a Deep Neural Network library) +#### Installing MXNet with the Prebuilt Binary Package(GPU) +For Windows users, MXNet provides prebuilt binary packages. +You can install the package directly in the R console after you have the above softwares installed -To install the required dependencies and install MXNet for R: +For GPU package: -1. Install the [CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit). The CUDA Toolkit depends on Visual Studio. To check whether your GPU is compatible with the CUDA Toolkit and for information on installing it, see NVidia's [CUDA Installation Guide](http://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/). -3. Clone the MXNet github repo. +```r + cran <- getOption("repos") + cran["dmlc"] <- "https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/R/CRAN/GPU/cuX" + options(repos = cran) + install.packages("mxnet") +``` +Change X to 80,90,91 or 92 based on your CUDA toolkit version. Currently, MXNet supports these versions of CUDA. +#### Building MXNet from Source Code(GPU) +After you have installed above softwares +1. Clone the MXNet github repo. ```sh git clone --recursive https://github.com/dmlc/mxnet ``` The `--recursive` is to clone all the submodules used by MXNet. You will be editing the ```"/mxnet/R-package"``` folder. -4. Download prebuilt GPU-enabled MXNet libraries for Windows from https://github.com/yajiedesign/mxnet/releases. You will need `mxnet_x64_vc14_gpu.7z` and `prebuildbase_win10_x64_vc14.7z`. -5. Download and install [CuDNN](https://developer.nvidia.com/cudnn). -6. Create a folder called ```R-package/inst/libs/x64```. MXNet supports only 64-bit operating systems, so you need the x64 folder. -7. Copy the following shared libraries (.dll files) into the ```R-package/inst/libs/x64``` folder: -``` -cublas64_80.dll -cudart64_80.dll -cudnn64_5.dll -curand64_80.dll +2. Download prebuilt GPU-enabled MXNet libraries for Windows from https://github.com/yajiedesign/mxnet/releases. You will need `mxnet_x64_vc14_gpu_cuX.7z` and `prebuildbase_win10_x64_vc14.7z` where X stands for your CUDA toolkit version +3. Create a folder called ```R-package/inst/libs/x64```. MXNet supports only 64-bit operating systems, so you need the x64 folder. +4. Copy the following shared libraries (.dll files) into the ```R-package/inst/libs/x64``` folder: +``` libgcc_s_seh-1.dll libgfortran-3.dll libmxnet.dll libmxnet.lib libopenblas.dll libquadmath-0.dll -nvrtc64_80.dll +mxnet.dll +unzip.exe +unzip32.dll +vcomp140.dll +wget.exe ``` -These dlls can be found in `prebuildbase_win10_x64_vc14/3rdparty/cudart`, `prebuildbase_win10_x64_vc14/3rdparty/openblas/bin`, `mxnet_x64_vc14_gpu/build`, `mxnet_x64_vc14_gpu/lib` and the `cuDNN` downloaded from NVIDIA. -8. Copy the header files from `dmlc`, `mxnet` and `nnvm` into `./R-package/inst/include`. It should look like: +These dlls can be found in `prebuildbase_win10_x64_vc14/3rdparty`, `mxnet_x64_vc14_gpu_cuX/build`, `mxnet_x64_vc14_gpu_cuX/lib`. +5. Copy the header files from `dmlc`, `mxnet`, `mxshadow` and `nnvm` from mxnet_x64_vc14_gpuX/include and mxnet_x64_vc14_gpuX/nvnm/include into `./R-package/inst/include`. It should look like: ``` ./R-package/inst └── include ├── dmlc ├── mxnet - └── nnvm + ├── mshadow + └── nnvm + ``` -9. Make sure that R is added to your ```PATH``` in the environment variables. Running the ```where R``` command at the command prompt should return the location. -10. Now open the Windows CMD and change the directory to the `mxnet` folder. Then use the following commands +6. Make sure that R is added to your ```PATH``` in the environment variables. Running the ```where R``` command at the command prompt should return the location. +7. Also make sure that Rtools in installed and added to your ```PATH``` in the environment variables. +8. Temporary patch - im2rec currently results in crashes during the build. Remove the im2rec.h and im2rec.cc files in R-package/src/ from cloned repository and comment out the two im2rec lines in R-package/src/mxnet.cc. +9. Now open the Windows CMD with admin rights and change the directory to the `mxnet` folder(cloned repository). Then use the following commands to build R package: ```bat From 5f66a10f91c1ddfb1cb6a6f19e89d535ed398767 Mon Sep 17 00:00:00 2001 From: Khedia Date: Mon, 23 Jul 2018 15:50:21 -0700 Subject: [PATCH 2/6] addressed PR comments --- docs/install/windows_setup.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/install/windows_setup.md b/docs/install/windows_setup.md index e10425ba30dd..e3e85d09b369 100755 --- a/docs/install/windows_setup.md +++ b/docs/install/windows_setup.md @@ -118,7 +118,7 @@ For CPU-only package: 1. Clone the MXNet github repo. ```sh -git clone --recursive https://github.com/dmlc/mxnet +git clone --recursive https://github.com/apache/incubator-mxnet ``` The `--recursive` is to clone all the submodules used by MXNet. You will be editing the ```"/mxnet/R-package"``` folder. @@ -184,7 +184,7 @@ To install MXNet on a computer with a GPU processor, choose from two options: However, few dependencies remains same for both options. You will need the following: -* Install Microsoft Visual Studio 2017 (required by CUDA) +* Install [Microsoft Visual Studio 2017](https://visualstudio.microsoft.com/downloads/) (required by CUDA) * Install [NVidia CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit) @@ -210,7 +210,7 @@ After you have installed above softwares 1. Clone the MXNet github repo. ```sh -git clone --recursive https://github.com/dmlc/mxnet +git clone --recursive https://github.com/apache/incubator-mxnet ``` The `--recursive` is to clone all the submodules used by MXNet. You will be editing the ```"/mxnet/R-package"``` folder. From 956a4e4e8a8d410999a79de75f8ce3f87dc68e3a Mon Sep 17 00:00:00 2001 From: Khedia Date: Tue, 24 Jul 2018 13:16:35 -0700 Subject: [PATCH 3/6] PR comments --- docs/install/windows_setup.md | 46 ++++++++++++++++++++++++++++------- 1 file changed, 37 insertions(+), 9 deletions(-) diff --git a/docs/install/windows_setup.md b/docs/install/windows_setup.md index e3e85d09b369..1ea45b0c22dc 100755 --- a/docs/install/windows_setup.md +++ b/docs/install/windows_setup.md @@ -152,8 +152,22 @@ These dlls can be found in `prebuildbase_win10_x64_vc14/3rdparty`, `mxnet_x64_vc ``` 6. Make sure that R is added to your ```PATH``` in the environment variables. Running the ```where R``` command at the command prompt should return the location. -7. Also make sure that Rtools in installed and added to your ```PATH``` in the environment variables. -8. Temporary patch - im2rec currently results in crashes during the build. Remove the im2rec.h and im2rec.cc files in R-package/src/ from cloned repository and comment out the two im2rec lines in R-package/src/mxnet.cc. +7. Also make sure that Rtools is installed and added to your ```PATH``` in the environment variables. +8. Temporary patch - im2rec currently results in crashes during the build. Remove the im2rec.h and im2rec.cc files in R-package/src/ from cloned repository and comment out the two im2rec lines in [R-package/src/mxnet.cc](https://github.com/apache/incubator-mxnet/blob/master/R-package/src/mxnet.cc) as shown below. +```bat +#include "./kvstore.h" +#include "./export.h" +//#include "./im2rec.h" +...... +...... + DataIterCreateFunction::InitRcppModule(); + KVStore::InitRcppModule(); + Exporter::InitRcppModule(); +// IM2REC::InitRcppModule(); +} + +``` + 9. Now open the Windows CMD with admin rights and change the directory to the `mxnet` folder(cloned repository). Then use the following commands to build R package: @@ -183,18 +197,19 @@ To install MXNet on a computer with a GPU processor, choose from two options: * Build the library from source code However, few dependencies remains same for both options. You will need the following: +* Install [Nvidia-drivers](http://www.nvidia.com/Download/index.aspx?lang=en-us) if not installed. Latest driver based on your system configuration is recommended. -* Install [Microsoft Visual Studio 2017](https://visualstudio.microsoft.com/downloads/) (required by CUDA) +* Install [Microsoft Visual Studio](https://visualstudio.microsoft.com/downloads/) (VS2015 or VS2017 is required by CUDA) -* Install [NVidia CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit) +* Install [NVidia CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit)(cu92 is recommended though we support cu80, cu90, cu91 and cu92) -* Download and install [CuDNN 7](https://developer.nvidia.com/cudnn) (to provide a Deep Neural Network library) +* Download and install [CuDNN](https://developer.nvidia.com/cudnn) (to provide a Deep Neural Network library). Latest version recommended. Note: A pre-requisite to above softwares is [Nvidia-drivers](http://www.nvidia.com/Download/index.aspx?lang=en-us) which we assume is installed. #### Installing MXNet with the Prebuilt Binary Package(GPU) For Windows users, MXNet provides prebuilt binary packages. -You can install the package directly in the R console after you have the above softwares installed +You can install the package directly in the R console after you have the above software installed. For GPU package: @@ -206,7 +221,7 @@ For GPU package: ``` Change X to 80,90,91 or 92 based on your CUDA toolkit version. Currently, MXNet supports these versions of CUDA. #### Building MXNet from Source Code(GPU) -After you have installed above softwares +After you have installed above software, continue with the following steps to build MXNet-R: 1. Clone the MXNet github repo. ```sh @@ -243,8 +258,21 @@ These dlls can be found in `prebuildbase_win10_x64_vc14/3rdparty`, `mxnet_x64_vc ``` 6. Make sure that R is added to your ```PATH``` in the environment variables. Running the ```where R``` command at the command prompt should return the location. -7. Also make sure that Rtools in installed and added to your ```PATH``` in the environment variables. -8. Temporary patch - im2rec currently results in crashes during the build. Remove the im2rec.h and im2rec.cc files in R-package/src/ from cloned repository and comment out the two im2rec lines in R-package/src/mxnet.cc. +7. Also make sure that Rtools is installed and added to your ```PATH``` in the environment variables. +8. Temporary patch - im2rec currently results in crashes during the build. Remove the im2rec.h and im2rec.cc files in R-package/src/ from cloned repository and comment out the two im2rec lines in [R-package/src/mxnet.cc](https://github.com/apache/incubator-mxnet/blob/master/R-package/src/mxnet.cc) as shown below. +```bat +#include "./kvstore.h" +#include "./export.h" +//#include "./im2rec.h" +...... +...... + DataIterCreateFunction::InitRcppModule(); + KVStore::InitRcppModule(); + Exporter::InitRcppModule(); +// IM2REC::InitRcppModule(); +} + +``` 9. Now open the Windows CMD with admin rights and change the directory to the `mxnet` folder(cloned repository). Then use the following commands to build R package: From 31f7d5eea4e797aefc416f09e98695a65ef1ae60 Mon Sep 17 00:00:00 2001 From: Khedia Date: Tue, 24 Jul 2018 13:54:53 -0700 Subject: [PATCH 4/6] PR comments --- docs/install/windows_setup.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/install/windows_setup.md b/docs/install/windows_setup.md index 1ea45b0c22dc..e058915dca17 100755 --- a/docs/install/windows_setup.md +++ b/docs/install/windows_setup.md @@ -151,8 +151,8 @@ These dlls can be found in `prebuildbase_win10_x64_vc14/3rdparty`, `mxnet_x64_vc └── nnvm ``` -6. Make sure that R is added to your ```PATH``` in the environment variables. Running the ```where R``` command at the command prompt should return the location. -7. Also make sure that Rtools is installed and added to your ```PATH``` in the environment variables. +6. Make sure that R executable is added to your ```PATH``` in the environment variables. Running the ```where R``` command at the command prompt should return the location. +7. Also make sure that Rtools is installed and the executable is added to your ```PATH``` in the environment variables. 8. Temporary patch - im2rec currently results in crashes during the build. Remove the im2rec.h and im2rec.cc files in R-package/src/ from cloned repository and comment out the two im2rec lines in [R-package/src/mxnet.cc](https://github.com/apache/incubator-mxnet/blob/master/R-package/src/mxnet.cc) as shown below. ```bat #include "./kvstore.h" @@ -257,8 +257,8 @@ These dlls can be found in `prebuildbase_win10_x64_vc14/3rdparty`, `mxnet_x64_vc └── nnvm ``` -6. Make sure that R is added to your ```PATH``` in the environment variables. Running the ```where R``` command at the command prompt should return the location. -7. Also make sure that Rtools is installed and added to your ```PATH``` in the environment variables. +6. Make sure that R executable is added to your ```PATH``` in the environment variables. Running the ```where R``` command at the command prompt should return the location. +7. Also make sure that Rtools is installed and the executable is added to your ```PATH``` in the environment variables. 8. Temporary patch - im2rec currently results in crashes during the build. Remove the im2rec.h and im2rec.cc files in R-package/src/ from cloned repository and comment out the two im2rec lines in [R-package/src/mxnet.cc](https://github.com/apache/incubator-mxnet/blob/master/R-package/src/mxnet.cc) as shown below. ```bat #include "./kvstore.h" From dad0712c91a4a3c73498748d9350c209dd7351bb Mon Sep 17 00:00:00 2001 From: Khedia Date: Tue, 24 Jul 2018 14:39:35 -0700 Subject: [PATCH 5/6] fixed line wrappings --- docs/install/windows_setup.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/install/windows_setup.md b/docs/install/windows_setup.md index e058915dca17..d07789189568 100755 --- a/docs/install/windows_setup.md +++ b/docs/install/windows_setup.md @@ -122,8 +122,11 @@ git clone --recursive https://github.com/apache/incubator-mxnet ``` The `--recursive` is to clone all the submodules used by MXNet. You will be editing the ```"/mxnet/R-package"``` folder. + 2. Download prebuilt GPU-enabled MXNet libraries for Windows from [Windows release](https://github.com/yajiedesign/mxnet/releases). You will need `mxnet_x64_vc14_cpu.7z` and `prebuildbase_win10_x64_vc14.7z` where X stands for your CUDA toolkit version + 3. Create a folder called ```R-package/inst/libs/x64```. MXNet supports only 64-bit operating systems, so you need the x64 folder. + 4. Copy the following shared libraries (.dll files) into the ```R-package/inst/libs/x64``` folder: ``` libgcc_s_seh-1.dll @@ -196,7 +199,7 @@ To install MXNet on a computer with a GPU processor, choose from two options: * Use the prebuilt binary package * Build the library from source code -However, few dependencies remains same for both options. You will need the following: +However, a few dependencies remain for both options. You will need the following: * Install [Nvidia-drivers](http://www.nvidia.com/Download/index.aspx?lang=en-us) if not installed. Latest driver based on your system configuration is recommended. * Install [Microsoft Visual Studio](https://visualstudio.microsoft.com/downloads/) (VS2015 or VS2017 is required by CUDA) @@ -229,8 +232,11 @@ git clone --recursive https://github.com/apache/incubator-mxnet ``` The `--recursive` is to clone all the submodules used by MXNet. You will be editing the ```"/mxnet/R-package"``` folder. + 2. Download prebuilt GPU-enabled MXNet libraries for Windows from https://github.com/yajiedesign/mxnet/releases. You will need `mxnet_x64_vc14_gpu_cuX.7z` and `prebuildbase_win10_x64_vc14.7z` where X stands for your CUDA toolkit version + 3. Create a folder called ```R-package/inst/libs/x64```. MXNet supports only 64-bit operating systems, so you need the x64 folder. + 4. Copy the following shared libraries (.dll files) into the ```R-package/inst/libs/x64``` folder: ``` libgcc_s_seh-1.dll From 40a10989e63e47df00165a4cde5483709787ef4d Mon Sep 17 00:00:00 2001 From: Khedia Date: Tue, 24 Jul 2018 14:41:35 -0700 Subject: [PATCH 6/6] fixed line wrappings --- docs/install/windows_setup.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/install/windows_setup.md b/docs/install/windows_setup.md index d07789189568..9d03474b5949 100755 --- a/docs/install/windows_setup.md +++ b/docs/install/windows_setup.md @@ -252,6 +252,7 @@ vcomp140.dll wget.exe ``` These dlls can be found in `prebuildbase_win10_x64_vc14/3rdparty`, `mxnet_x64_vc14_gpu_cuX/build`, `mxnet_x64_vc14_gpu_cuX/lib`. + 5. Copy the header files from `dmlc`, `mxnet`, `mxshadow` and `nnvm` from mxnet_x64_vc14_gpuX/include and mxnet_x64_vc14_gpuX/nvnm/include into `./R-package/inst/include`. It should look like: ```