[ENH] Updating resen-base and resen-core

[asreimer]

Accomplishes all checklist items in #59

• Updated resen-base to ubuntu 20.04 and updated start scripts to track the latest jupyter-stacks versions.
• upgrade to python 3.8
• deprecate python 2.7
• install seaborn, sunpy, pyephem, and skyfield,
• replace custom npm install with apt-get install

Testing

Build the resen-base image:

$ cd resen-base
$ docker build -t resenbasetest

Bulid the resen-core image:

$ cd resen-core
$ docker build -t resencoretest

Create a container with resencoretest and test things (e.g.):

$ docker run -it --name test <sha256:imageid> /bin/bash

[asreimer]

This needs to be changed to resen-base:2020.2.0 once we are happy with resen-base!

[asreimer]

Checked all files. Only one comment.

[asreimer]

There's a small issue with the terminal in jupyterlab not starting up with bash. Should be able to be solved with a jupyterlab_config.py file.

[ljlamarche]

I'm having issues building the resen-core image - looks like jupyterlab is failing.

Step 33/37 : RUN /bin/bash -cl 'source /home/$NB_USER/envs/py38/bin/activate &&                    jupyter labextension install @jupyter-widgets/jupyterlab-manager &&                    jupyter labextension install jupyter-matplotlib'
 ---> Running in 3803aedb71cc
Building jupyterlab assets (build:prod:minimize)
An error occured.
RuntimeError: JupyterLab failed to build
See the log file for details:  /tmp/jupyterlab-debug-egv3ho2l.log
The command '/bin/sh -c /bin/bash -cl 'source /home/$NB_USER/envs/py38/bin/activate &&                    jupyter labextension install @jupyter-widgets/jupyterlab-manager &&                    jupyter labextension install jupyter-matplotlib'' returned a non-zero code: 1

This issue persists if I try running the commands from within the docker container.

1. Start a docker container from the last image constructed before the docker build failed:

$ docker run -it --name resen-test <image id> /bin/bash

2. Attempt to run the two jupyterlab commands:

$ jupyter labextension install @jupyter-widgets/jupyterlab-manager

$ jupyter labextension install jupyter-matplotlib

Both these commands produce the error

Building jupyterlab assets (build:prod:minimize)
An error occured.
RuntimeError: JupyterLab failed to build

End of log file:

Error: Command failed: /home/jovyan/envs/py38/share/jupyter/lab/staging/node_modules/.bin/webpack --config webpack.prod.minimize.config.js
    at checkExecSyncError (child_process.js:629:11)
    at Object.execFileSync (child_process.js:647:13)
    at Object.<anonymous> (/home/jovyan/envs/py38/share/jupyter/lab/staging/node_modules/@jupyterlab/buildutils/lib/ensure-max-old-space.js:38:17)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
    at startup (internal/bootstrap/node.js:283:19)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

JupyterLab failed to build
Traceback (most recent call last):

  File "/home/jovyan/envs/py38/lib/python3.8/site-packages/jupyterlab/debuglog.py", line 47, in debug_logging
    yield

  File "/home/jovyan/envs/py38/lib/python3.8/site-packages/jupyterlab/labextensions.py", line 104, in start
    build(clean_staging=self.should_clean,

  File "/home/jovyan/envs/py38/lib/python3.8/site-packages/jupyterlab/commands.py", line 459, in build
    return handler.build(name=name, version=version, static_url=static_url,

  File "/home/jovyan/envs/py38/lib/python3.8/site-packages/jupyterlab/commands.py", line 661, in build
    raise RuntimeError(msg)

RuntimeError: JupyterLab failed to build

Exiting application: jupyter

edit by @asreimer: here's the full log file:
jupyterlab-debug-bwy28z9x.log

[asreimer]

I cannot reproduce on my fedora VM (has 9GB of RAM available).

I think the issue might be related to memory usage during minification: facebook/create-react-app#8320 (comment) (edit: Yes, 100% this must be our problem, see here: webpack-contrib/terser-webpack-plugin#202 (comment))

If you increase the amount of memory that the container is allowed to use while building, does this problem go away?

[pmreyes2]

I cannot reproduce the error either. My system is similar to @ljlamarche : MacBook Pro 16 GB RAM, and my docker is allowed to use up to 6 CPUs, 4 GB memory, 1GB swap, and disk image size is 60 GB (34 GB used)

[ljlamarche]

Increasing the CPUs, memory, and swap to @pmreyes2 configuration fixed the problem. Thanks!

[valentic]

It's scary to think about the amount of system resources that we can through at things nowadays...

…

On 9/15/20 11:30 PM, Pablo M. Reyes wrote: I cannot reproduce the error either. My system is similar to @ljlamarche <https://github.com/ljlamarche> : MacBook Pro 16 GB RAM, and my docker is allowed to use up to 6 CPUs, 4 GB memory, 1GB swap, and disk image size is 60 GB (34 GB used) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#61 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMC5ZVKD3UTCNODNDPTPUTSGBLRFANCNFSM4RGVDDVQ>.

[ljlamarche]

Tried this out, everything looks good to me! We'll have to update some of the tutorials, but I can do that on a separate branch.

[ljlamarche]

We're running into this issue: timduly4/pyglow#128
Basically, pyglow will crash the kernel if you give IRI 2016 a date after 2010, but IRI 2012 still works fine. It looks like it's a documented pyglow bug though so I'm not sure there's much we can/should do about it.

[asreimer]

I bumped the version of basemap and have fixed the bug with the jupyterlab terminal not using bash.

I think we're good to push the new resen-base to dockerhub and merge this into develop. Agree?

[ljlamarche]

I agree that we're ready to push the new resen-base to dockerhub. Do we want to include sciunit2 in this version of resen-core? (Can discuss this Thursday if anyone has strong opinions.)

[pmreyes2]

After two very small changes: adding pyglow latest commit and recently released sciunit2 for python3 I agree to merge this branch into develop. And yes, resen-base should be pushed to dockerhub.

[asreimer]

@pmreyes2 You accidentally added a bunch of python2.7 scripts back into this PR, which specifically removed them.

Please check your commit a81f824 and clean things up before merging.

[ljlamarche]

In order for bokeh to work, we need to install the following jupyterlab extensions:

jupyter labextension install @jupyter-widgets/jupyterlab-manager
jupyter labextension install @bokeh/jupyter_bokeh

https://docs.bokeh.org/en/latest/docs/user_guide/jupyter.html

[pmreyes2]

@ljlamarche I just added widgets interaction for bokeh. The jupyterlab-manager was already installed for interactions with matplotlib, so just the bokeh interactions was added. I think we can merge this to develop branch to keep fine tuning there.

[ljlamarche]

Just tested the bokeh widget. I have no further concerns about merging this.