[INTERIM-DEV] Remote Kernel Provider Development

This "release" exists to provide instructions and pre-built dependency files that are required for the initial development of remote kernel providers. Once dependencies are officially available, this release will be removed. For the most part, because most dependencies are now available via PyPi, the contents of this release are waning.

On-prem (non-container) Setup for YARN

To setup an on-prem environment for YARN, follow these steps. Note: Use of a virtual env is highly recommended.

1. Download the Notebook whl file attached to this release and pip install it into a development environment or build/install this fork and branch. This should also install jupyter_kernel_mgmt >= 0.4.0 and tornado == 5.1.1.
2. Install YarnKernelProvider: pip install yarn_kernel_provider . This should also install remote_kernel_provider >= 0.1.0 (and jupyter_kernel_mgmt >= 0.4.0 if performed before previous step).
3. Replace any wheel files with your own dev versions and install those using pip install --upgrade.
4. Using juptyer yarn-kernelspec install, create desired kernel specifications.
5. Make any necessary edits to the kernelspec files relative to your environment.
6. Download on-prem-start-notebook.sh. Make any necessary edits relative to your environment.
7. Run on-prem-start-notebook.sh. You might want some notebooks in the same directory or use the notebook dir option to point to a different location.

These steps worked for me:

conda create -n py3-rkp python=3.6
conda activate py3-rkp
pip install notebook-6.0.0.dev0-py2.py3-none-any.whl 
pip install yarn_kernel_provider

jupyter yarn-kernelspec install 
jupyter yarn-kernelspec install --language=R
jupyter yarn-kernelspec install --language=Scala

on-prem-start-notebook.sh

After which you should have three kernelspecs located in /usr/local/share/juptyer/kernels/yarnkp_spark_{python,r,scala}. You may need to use various options to fine-tune to your environment. Please open an issue at https://github.com/gateway-experiments/yarn_kernel_provider should there be warranted changes or fixes.

Container-based Setup

These steps were used to create the Kubernetes environment. Setup for Docker{,Swarm} will vary:

1. An image (elyra/nb-kernel-mgmt:dev) containing the Notebook, juptyer_kernel_mgmt, remote_kernel_provider, kubernetes_kernel_provider and docker_kernel_provider whl and kernelspec files can be found here. Note: This has not been tested in Docker or Docker Swarm but does appear to work for Kubernetes.
2. Should you need to modify the image, download the Dockerfile attached to this release.
3. I deployed the image on Kubernetes using the attached notebook.yaml file. This file is derived directly from the one used by Enterprise Gateway. As a result, there are a number of items in this file that do not apply to this configuration or require more work (e.g., whitelists are not applied, and kernel-image-puller isn't able to access specs from Notebook, etc.)