Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deploy Zeppelin on Kubernetes #107

Closed
stvoutsin opened this issue May 25, 2020 · 3 comments
Closed

Deploy Zeppelin on Kubernetes #107

stvoutsin opened this issue May 25, 2020 · 3 comments

Comments

@stvoutsin
Copy link
Collaborator

Experiment with the deployment of Zeppelin on Kuberenetes. This follows the deployment of Hadoop & Spark on Kubernetes, and will allow us to initially experiment and compare with the existing prototype and also will allow us to end up with a more automated build with fault tolerance and scalability. (User session gets its own Kubernetes Pod)

@Zarquan
Copy link
Collaborator

Zarquan commented May 26, 2020

As with #19 there are already 100's of blogs and online tutorials showing how to do this. So getting the service to run is useful, but not the most important part of the work. Our own notes showing what we learned about how to deploy and how to administer the service are the important part.

@stvoutsin
Copy link
Collaborator Author

I've tried some experiments trying to get Zeppelin working on Kuberenetes. Unfortunately it looks like there aren't as many tutorials as there are for JHub for example, as I was only able to find two which I could follow and get something to work.

Some links that I found and followed are:
https://hub.kubeapps.com/charts/stable/zeppelin - Deploy Zeppelin as Helm Chart
https://zeppelin.apache.org/docs/0.9.0-SNAPSHOT/quickstart/kubernetes.html - Deploy Zeppelin from yaml file using kubectl

Some notes on these experiments are here:
https://github.com/stvoutsin/aglais/blob/stv-issue-107/notes/stv/20200525-zeppelin-k8s.txt
https://github.com/stvoutsin/aglais/blob/stv-issue-107/notes/stv/20200526-zeppelin-k8s.txt

In both examples I was able to get Zeppelin deployed and able to access the Zeppelin UI.
However in one case I could not get any of the interpreters to work successfully, and in the second case the example notebooks I ran worked, but were not deployed as individual pods, which is not ideal in terms of scalability.

I'll leave this task for now, but something to keep in mind is that this wasn't trivial, and I wasn't able to successfully deploy what I expected. (A Zeppelin deployment where user sessions spawn a separate container/pod)

@Zarquan
Copy link
Collaborator

Zarquan commented Sep 8, 2020

Done, merged in PR #152 and #148.

@Zarquan Zarquan closed this as completed Sep 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants