-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
R package hexbin for r-notebook image #532
Labels
type:Enhancement
A proposed enhancement to the docker images
Comments
@terhorst Thanks for reporting this issue. If you'd like to add |
terhorst
added a commit
to terhorst/docker-stacks
that referenced
this issue
Jan 14, 2018
Merged
jsuwala
added a commit
to HatfieldConsultants/docker-stacks
that referenced
this issue
May 10, 2018
…merge * base image on ubuntu:16.04 * make jupyterhub env->cli args optional DockerSpawner will stop passing these args, so make them optional. * Remove Python 2 Only Python 3 from now on * quote NOTEBOOK_DIR * fix condition-testing syntax bugs * [ci skip] Note in README about Python 2.x * Update jupyterlab to 0.26.* * [ci skip] Add a jupyter lab example with ports * [ci skip] Doc host mount permission requirements * Update jupyterlab to 0.27.* * [ci skip] Copy/paste is evil * use groups for managing write-access to files - any files the user should be able to write should have group `user-permissions` with `g+rwX` - remove `chown` from start.sh because it is no longer needed - add `fix-permissions` script for setting the user-writable permissions on a path - user-permissions group as GID 10000 (is there a reason for it to have a different value?) - containers can set group with `--group-add user-writable` if they want to run with a different uid/gid (without -u root -e NB_UID -e NB_GID, which make this unnecessary) * set setuid/setgid bits in fix-permissions ensures files have the right owner:group unfortunately, not enough to get group-writable permissions (need acl or umask for that), so we still need to run it after each install * leftover mentions of `user-writable` group use `users` instead * remove user-facing start-singleuser.sh docs it’s handled internally * [ci skip] Remove py2 reference * Update ipywidgets to 7.0.x * Added support for facets visualization * keras stack * Add keras to the tensorflow stack * Fix working directory problems with custom NB_USER Improve start.sh logging along the way * Fix inherit-diagram: debian@SHA -> ubuntu@SHA Based on jupyter@0351f0c * homedir fixes - don't relocate home if home exists (e.g. mounted volume) - respect workdir, only cd to new home if we were in /home/jovyan/ * further emphasize the use of tags in README * changed the sys-prefix * changed the sys-prefix * Change starting directory of scipy-notebook * Fix facets install * Don't change WORKDIR * Fix permissions after install * Cleanup source tree * Upgrade to notebook 5.1 * jupyterhub to version 0.8 * Upgrade to notebook 5.2.0 * Update jupyterlab to 0.28 Update jupyterlab from 0.27.* to 0.28.*. This allows the current version of ipywidgets (7.0.2) to be installed. * MAINTAINER is deprecated, using LABEL now * bump miniconda version * Patch R package install user issues -The following patch addresses issue jupyter#489 (jupyter#489) * bump jupyterlab and install jupyter-labhub extension * add r channel at lower priority than conda-forge avoids broken R due to mismatched icu version (probably a bad or missing pin somewhere) * (fix) Add tzdata as R pre-requisites. jupyter#502 * Activate ipywidgets for JupyterLab. jupyter#505 * Remove legacy make targets * Switch to pytest for container testing * Add placeholder tests for base-notebook * Add requests to travis requirements * Refactor to share fixtures with option tests * Tests for NB_UID, NB_GID, GRANT_SUDO options * Improve test container teardown * Fix needless groupmod when NB_GID is unchanged * MNT: Clean npm packages and JupyterLab artifacts Spotted while writing tests * Fix bogus assert, add mount test * julia 0.6.0 * Fix mount test on CI * Add sha256 check and use JULIA_VERSION env var * Remove mount permissions test Not really testing a docker-stacks feature, only that docker host permissions work * Add make test-reqs and separate requirements file Make local test setup easier * Fix make test-reqs naming * Updating jupyterlab to 0.30.* * Update jupyterlab hub ext to 0.7.0 * julia 0.6.0 * Add sha256 check and use JULIA_VERSION env var * Julia v0.6.2 * Fix URL domain * Update Julia version in README to match script * Improved README, with explaining 2 methods to start the Jupyter stacks * Adding info how to mount local directory into 'work' directory * Added list to quickly find README of each image, when reading the main README * Fix for granting SUDO to jovyan user and run bash commands. Test image builded with: > docker build -t $USER/base-notebook -f Dockerfile . > docker run -ti --rm -p 8888:8888 -e GRANT_SUDO=yes --user root jupyter/base-notebook:033056e6d164 bash root@84db8819258d:~# docker run -ti --rm -p 8888:8888 -e GRANT_SUDO=yes --user root $USER/base-notebook bash > root@c23f1fd7d279:~# > docker run -ti --rm -p 8888:8888 -e GRANT_SUDO=yes --user root jupyter/base-notebook:033056e6d164 start.sh Set username to: jovyan usermod: no changes Set jovyan GID to: 100 Granting jovyan sudo access Execute the command: HOSTNAME=a231202e6d1a TERM=xterm SHELL=/bin/bash NB_USER=jovyan LC_ALL=en_US.UTF-8 USER=jovyan MAIL=/var/mail/jovyan PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin MINICONDA_VERSION=4.3.30 PWD=/home/jovyan LANG=en_US.UTF-8 GRANT_SUDO=yes HOME=/home/jovyan SHLVL=1 LANGUAGE=en_US.UTF-8 no_proxy=*.local, 169.254/16 LOGNAME=jovyan DEBIAN_FRONTEND=noninteractive CONDA_DIR=/opt/conda NB_GID=100 NB_UID=1000 _=/usr/bin/env tlinnet@linmac:base-notebook$ > docker run -ti --rm -p 8888:8888 -e GRANT_SUDO=yes --user root $USER/base-notebook start.sh Set username to: jovyan usermod: no changes Set jovyan GID to: 100 Granting jovyan sudo access Execute the command: No arguments supplied HOSTNAME=364e5fc77224 SHELL=/bin/bash TERM=xterm LC_ALL=en_US.UTF-8 USER=jovyan SUDO_USER=root SUDO_UID=0 USERNAME=jovyan MAIL=/var/mail/jovyan PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin PWD=/home/jovyan LANG=en_US.UTF-8 SHLVL=1 SUDO_COMMAND=/bin/bash -c env; PATH=$PATH; bash HOME=/home/jovyan LANGUAGE=en_US.UTF-8 LOGNAME=jovyan SUDO_GID=0 _=/usr/bin/env jovyan@364e5fc77224:~$ > docker run -ti --rm -p 8888:8888 -e GRANT_SUDO=yes --user root jupyter/base-notebook:033056e6d164 start.sh echo hello tlinnet@linmac:base-notebook$ docker run -ti --rm -p 8888:8888 -e GRANT_SUDO=yes --user root jupyter/base-notebook:033056e6d164 start.sh echo hello Set username to: jovyan usermod: no changes Set jovyan GID to: 100 Granting jovyan sudo access Execute the command: echo hello hello tlinnet@linmac:base-notebook$ > docker run -ti --rm -p 8888:8888 -e GRANT_SUDO=yes --user root $USER/base-notebook start.sh echo hello Set username to: jovyan usermod: no changes Set jovyan GID to: 100 Granting jovyan sudo access Execute the command: echo hello hello tlinnet@linmac:base-notebook$ * This is to fix running SUDO as eihter the jovyan user or root. Please refer to alias, in bottom of commit. > dbn Execute the command: jupyter notebook > dbn bash jovyan@eedcc93a837d:~$ > dbn start.sh Execute the command: exit > dbnr Set username to: jovyan usermod: no changes Set jovyan GID to: 100 Execute the command: jupyter notebook Running as root is not recommended. Use --allow-root to bypass. exit > dbnr bash root@893cb78b8c9c:~# > dbnr start.sh Set username to: jovyan usermod: no changes Set jovyan GID to: 100 Execute the command: No arguments supplied HOSTNAME=d45c52e788b7 TERM=xterm SHELL=/bin/bash NB_USER=jovyan LC_ALL=en_US.UTF-8 PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin _=/usr/bin/env MINICONDA_VERSION=4.3.30 PWD=/home/jovyan LANG=en_US.UTF-8 HOME=/home/jovyan SHLVL=2 LANGUAGE=en_US.UTF-8 no_proxy=*.local, 169.254/16 DEBIAN_FRONTEND=noninteractive CONDA_DIR=/opt/conda NB_GID=100 NB_UID=1000 root@d45c52e788b7:~# > dbnrs Set username to: jovyan usermod: no changes Set jovyan GID to: 100 Granting jovyan sudo access Execute the command: jupyter notebook Copy/paste this URL into your browser when you connect for the first time, .... > dbnrs bash root@f293dce949db:~# > dbnrs start.sh Set username to: jovyan usermod: no changes Set jovyan GID to: 100 Granting jovyan sudo access Execute the command: No arguments supplied HOSTNAME=d0cd57ea32e2 SHELL=/bin/bash TERM=xterm LC_ALL=en_US.UTF-8 USER=jovyan SUDO_USER=root SUDO_UID=0 USERNAME=jovyan MAIL=/var/mail/jovyan PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin PWD=/home/jovyan LANG=en_US.UTF-8 SHLVL=1 SUDO_COMMAND=/bin/bash -c env; PATH=$PATH; bash HOME=/home/jovyan LANGUAGE=en_US.UTF-8 LOGNAME=jovyan SUDO_GID=0 _=/usr/bin/env jovyan@d0cd57ea32e2:~$ > dbns Container must be run as root to grant sudo permissions Execute the command: jupyter notebook Copy/paste this URL in.. > dbns bash jovyan@ce5c2491fa32:~$ > dbns start.sh Container must be run as root to grant sudo permissions Execute the command: exit ###### Build user setup docker build -t $USER/base-notebook -f Dockerfile . # Normal, dbn: docker-base-notebook alias dbn='docker run -ti --rm -p 8888:8888 -v "$PWD":/home/jovyan/work --name base-notebook $USER/base-notebook' # Root, dbnr: dbn with root alias dbnr='docker run -ti --rm -p 8888:8888 --user root -v "$PWD":/home/jovyan/work --name base-notebook $USER/base-notebook' # Jovyan SUDO, dbnr: dbn with SUDO for jovyan alias dbnrs='docker run -ti --rm -p 8888:8888 --user root -e GRANT_SUDO=yes -v "$PWD":/home/jovyan/work --name base-notebook $USER/base-notebook' # Root, fail to sudo for jovyan. alias dbns='docker run -ti --rm -p 8888:8888 -e GRANT_SUDO=yes -v "$PWD":/home/jovyan/work --name base-notebook $USER/base-notebook' * Minor README touchups * Fix missing env vars (inc. PATH) in some cases * Refactor to use sudo to become jovyna when the container starts as root * Retain all environment variables when becoming jovyan * Handle start.sh without additional commands when launching a container as NB_USER too * Fix test to use bash for multiple commands * Fix missing jupyterlab-manager version * Fix missing @ in version * Append $CONDA_DIR/bin to sudo secure_path Retain behavior when su was used instead * Verify path prefix in test * Update Dockerfile to install r-hexbin Fixes jupyter#532 * Add correct command :) * Add /bin to sudo secure_path * Enable jupyterhub-lab support. * rework to use JUPYTER_ENABLE_LAB for both notebook and hub * Support provisioned storage with incorrect permissions I ran into an issue when trying to get this to work with a NFS server which I did not have direct control over (EFS). As part of the PersistentVolumeClaim, there is no easy way to set the UID and GID of the created directory.on the networked FS. My only concern with this chown is that some user out there might be running jupyterhub in an odd configuration where $NB_USER is not supposed to have these exact permissions on the storage. I think this is quite unlikely, but it is worth mentioning. I chronicled my experiences with working around this issue and setting up z2jh on EFS in jupyterhub/zero-to-jupyterhub-k8s#421 with @yuvipanda. * Gate chown /home/$NB_USER behind $CHOWN_HOME * Added PYTHONPATH to sudo * Upgrade to the first JupyterLab beta v0.31 * Fix to check dict status code response in test_container_options * Upgrade ipywidgets in scipy for compatibility with 0.31 jupyterlab version * Add example templates for deploying notebooks to OpenShift. * Generate password from OpenShift template if none is supplied. * Add README for OpenShift example. * Use NB_UID for USER statement in Dockerfile so deployment platform can verify that image doesn't run as root. * Add missing label to config map in OpenShift template. * Add notes on changing password when using OpenShift templates. * Ensure anything under $HOME is always writable to group. * improve secure_path modification by storing in sudoers.d * Add example Source-to-Image scripts for building custom notebook images. * Add save-artifacts script to Source-to-Image to avoid warnings/errors. * Ensure assemble script of Source-to-Image script logs progress. * Add template for using Source-to-Image builder scripts with OpenShift. * Add OpenShift template for building, but not deploying an image using Source-to-Image builder. * Remove packages file from image when using Source-to-Image before doing a build, rather than after. * Add README for Source-to-Image examples directory. * Fix URLs for location of Source-to-Image example directory in master. * Avoid placing anything under home directory in OpenShift deployment example so can use it to mount volumes. * Avoid relying on existence of work subdirectory in Source-to-Image examples. * Correct URL in Source-to-Image template to reference master branch. * Add entries to passwd/group if running as user not in passwd file. * Upgrading Spark to 2.3 * Changed Spark mirror server and upgraded py4j library to 0.10.6 * Fixing checksum error * Add instructions for enabling Jupyter Lab interface with OpenShift deployment. * Update to Toree 0.2.0 RC3 * add r-htmltools pkg * Chown of home dir needs to be recursive * Disable datascience / tensorflow builds on travis temporarily * Restore all builds, disable julia precompilation * Add jupyterlab_bokeh extension Without the extension it is not possible to use all Bokeh functionalities in JupyterLab * Add pyarrow installation This way we can profit from pandas UDF * sparklyr add to r-notebook sparklyr alone without any other spark package added to the install image. * Add netcat Netcat (nc) is often used as a tool to test socket server/client, and very useful to quickly get started with, f. e., Spark Streaming. See jupyter#583. * Fix netcat install * net -> netcat * Fix r-notebook notebook start * Fix other IRKernel images * Notebook 5.4, force tornado 4.* at the outset * Make extracted files root id * Add TEST_ONLY_BUILD arg * Fix arg name mismatch * Update tensorflow readme with distributed mode instructions * Unify docs, test dev env * Sphinx quickstart * Outline, include some info from READMEs * Add image descriptions and content summaries * Notes about community stacks * Fill-in and refine instructions to run * Fix travis env creation command * Stick with pip for travis testing * Finish single container examples * First draft configuration page * Split pages, use RTD theme for now I can't seem to configure the Jupyter Albaster theme correctly and will reach out for help later. * Fix path to readthedocs.yaml * Try the RTD path fix again * base image on ubuntu:xenial-20180228 * Fix link to stack contrib page * Fix TOC heading level * fixed tar --owner root to be correct * Address initial review comments * Fix typo too * Run additional scripts in /usr/local/bin/start-notebook.d/ * Keyserver pool hkp://pool.sks-keyservers.net over keyserver.ubuntu.com see https://sks-keyservers.net/overview-of-pools.php * Drop the `r` channel The `r` channel has been considered part of `defaults` since `conda` version `4.3.0`. So it should already be taken into consideration by `conda` installs without having to explicitly add the channel. Further the `conda-forge` channel has incorporated an ever growing, healthy stack of R packages. As such it appears all of the R packages used in this stack now come from `conda-forge` and not `defaults`. Given all of this, it seems safe to drop the `r` channel from explicit addition to the channel list. * Use conda-forge's blas package w/openblas As conda-forge does not use the `nomkl` package, but instead uses the `blas` package to select BLAS versions, install the `blas` package and select the `openblas` implementation. * Keyserver used in Hashicorps Nomad files Keyserver used in Hashicorps Nomad files * update all-spark to toree 0.2.0rc4 * Remove tornado<5 pin * Fix pyarrow install Need to install as NB_UID, not root, to keep all /opt/conda folders writeable by the default user * in start.sh ensure usename jovyan exists before attempting usermod * Remove facets directory The `cd` command switches to the user's home directory (away from the directory we want to delete) Then we remove the facets directory using the full path * Force npm cache clean * Force all npm cache clean commands * Update jupyterlab and extensions Update jupyterlab from 0.31 to 0.32. Update the following extensions to matching versions: - jupyterlab/hub-extension - jupyter-widget/jupyterlab-manager - jupyterlab_bokeh * Update toree to rc5 * Update scikit-learn to 0.19* Release notes can be found here http://scikit-learn.org/stable/whats_new.html#version-0-19-1 * Update scikit-image to 0.13* Release notes can be found here https://github.com/scikit-image/scikit-image/blob/master/doc/release/release_0.13.rst * Update miniconda installer to 4.4 * Added conda clean. Addresses jupyter#627 * Update tensorflow to 1.5, keras to 2.1 * Update scipy-notebook packages
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The R package
hexbin
(https://cran.r-project.org/web/packages/hexbin) is required to usegeom_hex
in ggplot, however it does not come with the r-notebook image.The text was updated successfully, but these errors were encountered: