diff --git a/enterprise_gateway/proposal.md b/enterprise_gateway/proposal.md new file mode 100644 index 0000000..a7ee8e2 --- /dev/null +++ b/enterprise_gateway/proposal.md @@ -0,0 +1,98 @@ +# Proposal for Incubation + +## Subproject name + +Jupyter Enterprise Gateway ('jupyter-incubator/enterprise_gateway') + +## Development team and Advocate + +* Luciano Resende, IBM ('@lresende') +* Kevin Bates, IBM ('@kevin-bates') +* Christian Kadner, IBM ('@ckadner') +* Kun Liu, IBM ('@liukun1016') +* Alan Chin, IBM ('@akchinSTC') +* Sherry Guo, IBM ('@sherryxg') +* Fred Reiss, IBM ('@frreiss') +* Sanjay Saxena ('@sanjay-saxena') + +Who is the Steering Council Advocate for this Subproject? + +* Kyle Kelley ('@rgbkrk') +* Peter Parente ('@parente') + +## Subproject goals, scope and functionality + +### Problem +Founded in academia, the Jupyter projects provide a rich and highly popular set of applications for +interacting with and iterating on large and complex applications. It has been truly ground-breaking. +However, when we first attempted to build a Notebook service that could enable a large number of data +scientists to run frequent and large workloads against a large Apache Spark cluster, we identified +several requirements that were not currently available in the Jupyter open source ecosystem. We tried +to use the Jupyter Kernel Gateway project, but we quickly realized that the JKG server became the +bottleneck because the co-located Spark driver application for these kinds of workloads (in this case, +the kernel process running on behalf of notebook cells) were extremely resource intensive. Toss in a team +or organization of data scientists, and you quickly saturate the compute resources of the Kernel Gateway +server. + +### Goals +The primary goal behind Jupyter Enterprise Gateway is to address the gaps identified during our experience +described above and provide a lightweight, multi-tenant gateway that enables many notebook users to share +a single Spark Cluster and run kernels as managed resources within the cluster (i.e. Yarn Cluster mode). +By providing tight integration with the resource manager used by the Spark cluster, we avoid resource starvation +caused by multiple processes that are unaware of each other and compete for the same resources. We can also provide +complex scheduling capabilities such as [capacity scheduling and priority queues](https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html) +with a very low development cost, thus satisfying Enterprise Administrators manageability and security +requirements. Although the Apache Yarn resource manager is immediately supported, we have plans to support +others such as Spark on Kubernetes in the near future utilizing the existing resource manager pluggable framework. + +We strive to not alter any kernel code itself. As a result, we've developed kernel launchers that essentially +house the target kernel and serve as life-cycle managers for functionality like interrupt or, in some cases, +kernel termination. + +![System Diagram](system-diagram.jpg) + +### Scope +In general, Enterprise Gateway is intended to the address the needs of enterprise and cloud administrators +such that cluster resource utilization is optimized across all applications, including those of data scientists +performing analysis via Notebooks residing on their desktops. + +### Use Cases + +*As a data scientist, I want to attach a local Jupyter Notebook server to the enterprise/cloud shared compute cluster +to run compute-intensive analytical workload (e.g. multi-tenant interactive gateway to Apache Spark cluster). + +* As an Spark Cluster administrator, I want to enable notebook users to utilize all the computing resources available +in the enterprise/cloud compute cluster to run analytical workloads in a managed and secure way. + +* As an administrator, I want enforce user isolation such that notebook kernel processes are protected against each +other allowing users to preserve and leverage their own environment, i.e. libraries and/or packages, data, etc. + +## Audience + +Platform Engineers and Apache Spark Administrators that want to enable Data Scientists and other Jupyter +Notebook users to connect and share existing Spark cluster resources when executing their notebooks. + + +## Other options + +Enterprise Gateway has very close affinity with Kernel Gateway and could be considered the next generation of JKG. +As JKG is planning to move its functionality directly to Jupyter Notebook (see [JKG-259 ](https://github.com/jupyter/kernel_gateway/issues/259) +and [JUPYTER-2644 ](https://github.com/jupyter/notebook/issues/2644)), and based on recommendation from some +members from the JKG community, we have decided to boostrap a new project (“Enterprise Gateway”). We expect that in +the future, once functionality has been moved out of JKG, Enterprise Gateway will have a direct dependency on the +Notebook server. + +Other projects have tried to solve the 'remote kernel' problem (e.g. [remotekernel](https://github.com/danielballan/remotekernel) +or [remote_ikernel](https://bitbucket.org/tdaff/remote_ikernel)) but these seem to be providing a wrapper/proxy kernel +that will introduce a level of indirection that can cause side-effects. In any case, they do not provide any integration +with Spark and its resource manager. + +As for JupyterHub, we believe its a full solution, providing capabilities to spawn and manage Notebook servers per +user. In this scenario, the JupyterHub will be independently consuming and managing cluster compute resources that +compete directly with managed resources. This may lead to operational and management issues for cluster administrators. + +## Integration with Project Jupyter + +Enterprise Gateway is based on the Jupyter Notebook stack, and directly extends Kernel Gateway, Notebook Server, +and Jupyter Client. These components, including it's transient dependencies to other projects are all used and +required by Jupyter Enterprise Gateway. diff --git a/enterprise_gateway/system-diagram.jpg b/enterprise_gateway/system-diagram.jpg new file mode 100644 index 0000000..83dee1b Binary files /dev/null and b/enterprise_gateway/system-diagram.jpg differ