Logging errors when running jupyter lab in latest release

[tmadlener]

When running jupyter lab from the latest release in a container there are quite a few error messages in the logs. However, at least for python notebooks they seem to be inconsequential. On the other hand given that it is complaining about a non-writable directory on cvmfs, I think that this is not really related to trying to run things in a container.

To reproduce run the following:

• Start a docker container and bind-mount /cvmfs

docker run -it -v /cvmfs:/cvmfs:shared -p 8888:8888 ghcr.io/aidasoft/centos7:latest /bin/bash

• Inside the container source the setup script and launch jupyter lab

source /cvmfs/sw.hsf.org/key4hep/setup.sh
jupyter lab --ip=0.0.0.0 --port=8888 --allow-root --no-browser

• Use the displayed URL to open jupyter lab in a local browser. This will open jupyter lab interface there and it will be possible to open and use python3 notebooks. However, at the same time, the following error will be logged over and over again:

[...]
[W 2022-11-04 17:37:42.900 ServerApp] 500 PUT /lab/api/workspaces/default?1667579862895 (172.17.0.1): [Errno 30] Read-only file system: '/cvmfs/sw.hsf.org/spackages6/root/6.26.06/x86_64-centos7-gcc11.2.0-opt/bc7bv/etc/notebook/lab'
[W 2022-11-04 17:37:42.901 LabApp] [Errno 30] Read-only file system: '/cvmfs/sw.hsf.org/spackages6/root/6.26.06/x86_64-centos7-gcc11.2.0-opt/bc7bv/etc/notebook/lab'
[E 2022-11-04 17:37:42.901 LabApp] {
      "Host": "127.0.0.1:8888",
      "Connection": "keep-alive",
      "Content-Length": "405",
      "Sec-Ch-Ua": "\"Google Chrome\";v=\"107\", \"Chromium\";v=\"107\", \"Not=A?Brand\";v=\"24\"",
      "Dnt": "1",
      "Sec-Ch-Ua-Mobile": "?0",
      "Authorization": "token a05b48bf30b3d091b1fff095d1ecd6cf5892fa4fbe7552f2",
      "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36",
      "Content-Type": "text/plain;charset=UTF-8",
      "X-Xsrftoken": "2|e0de04b6|799b90cf20f21fe5f8d36ab3aa162fe6|1666626861",
      "Sec-Ch-Ua-Platform": "\"Linux\"",
      "Accept": "*/*",
      "Origin": "http://127.0.0.1:8888",
      "Sec-Fetch-Site": "same-origin",
      "Sec-Fetch-Mode": "cors",
      "Sec-Fetch-Dest": "empty",
      "Referer": "http://127.0.0.1:8888/lab",
      "Accept-Encoding": "gzip, deflate, br",
      "Accept-Language": "en-US,en-AT;q=0.9,en;q=0.8,de-DE;q=0.7,de;q=0.6",
      "Cookie": "_xsrf=2|e0de04b6|799b90cf20f21fe5f8d36ab3aa162fe6|1666626861; username-127-0-0-1-8888=\"2|1:0|10:1667579862|23:username-127-0-0-1-8888|44:N2EyNzcxMTM0NGNiNDFkYmJjZDIzNWY3NTBmOWY1OWI=|8f142c6896e4644140eebda14a997ecb5858f59e650f80df120ecdf619fcade6\""
    }
[E 2022-11-04 17:37:42.902 LabApp] 500 PUT /lab/api/workspaces/default?1667579862895 (172.17.0.1) 2.22ms referer=http://127.0.0.1:8888/lab
[...]

• C++ (ROOT) notebooks are not working (they can be opened, but the Kernel will not launch properly).

From the error message it seems that jupyter is trying to write to some directory within the root install tree, which is obviously failing due to cvmfs being read-only.

Interestingly trying the above with /cvmfs/ilc.desy.de/key4hep/setup.sh instead does not exhibit the same error. However, in that case there are also no C++ notebooks available in the jupyter lab interface. Given that these releases in principle use the same package configuration it is not entirely clear to me where this difference comes from.

[vvolkl]

I also don't fully understand the problem - it seems like this is coming from ROOT.
The following is a workaround for the error:

export JUPYTER_CONFIG_DIR=$HOME/.jupytercfg

although I'm not sure the C++ kernel will still work. @tmadlener Can you confirm this?

[tmadlener]

(Re)setting JUPYTER_CONFIG_DIR to a writeable directory resolves the error logs caused by the attempts to write to a cvmfs directory.

C++ kernels still do not work. However, that seems to be related to some missing python package(s), as far as I can tell. From the output (after opening a c++ notebook):

[I 2022-11-10 16:36:13.978 ServerApp] Creating new notebook in 
[I 2022-11-10 16:36:13.997 ServerApp] Writing notebook-signing key to /root/.local/share/jupyter/notebook_secret
[I 2022-11-10 16:36:14.286 ServerApp] Kernel started: 648c5b73-47d8-4423-b6aa-22362aa4015b
Traceback (most recent call last):
  File "/cvmfs/sw.hsf.org/spackages6/root/6.26.06/x86_64-centos7-gcc11.2.0-opt/bc7bv/lib/JupyROOT/kernel/rootkernel.py", line 22, in <module>
    from metakernel import MetaKernel
  File "/cvmfs/sw.hsf.org/spackages6/root/6.26.06/x86_64-centos7-gcc11.2.0-opt/bc7bv/lib/ROOT/_facade.py", line 153, in _importhook
    return _orig_ihook(name, *args, **kwds)
ModuleNotFoundError: No module named 'metakernel'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/cvmfs/sw.hsf.org/spackages6/python/3.9.12/x86_64-centos7-gcc11.2.0-opt/cqr5o/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/cvmfs/sw.hsf.org/spackages6/python/3.9.12/x86_64-centos7-gcc11.2.0-opt/cqr5o/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/cvmfs/sw.hsf.org/spackages6/root/6.26.06/x86_64-centos7-gcc11.2.0-opt/bc7bv/lib/JupyROOT/kernel/rootkernel.py", line 25, in <module>
    raise Exception("Error: package metakernel not found.(install it running 'pip install metakernel')")
Exception: Error: package metakernel not found.(install it running 'pip install metakernel')
[I 2022-11-10 16:36:20.267 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (1/5), new random ports
Traceback (most recent call last):
  File "/cvmfs/sw.hsf.org/spackages6/root/6.26.06/x86_64-centos7-gcc11.2.0-opt/bc7bv/lib/JupyROOT/kernel/rootkernel.py", line 22, in <module>
    from metakernel import MetaKernel
  File "/cvmfs/sw.hsf.org/spackages6/root/6.26.06/x86_64-centos7-gcc11.2.0-opt/bc7bv/lib/ROOT/_facade.py", line 153, in _importhook
    return _orig_ihook(name, *args, **kwds)
ModuleNotFoundError: No module named 'metakernel'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/cvmfs/sw.hsf.org/spackages6/python/3.9.12/x86_64-centos7-gcc11.2.0-opt/cqr5o/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/cvmfs/sw.hsf.org/spackages6/python/3.9.12/x86_64-centos7-gcc11.2.0-opt/cqr5o/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/cvmfs/sw.hsf.org/spackages6/root/6.26.06/x86_64-centos7-gcc11.2.0-opt/bc7bv/lib/JupyROOT/kernel/rootkernel.py", line 25, in <module>
    raise Exception("Error: package metakernel not found.(install it running 'pip install metakernel')")
Exception: Error: package metakernel not found.(install it running 'pip install metakernel')
[...]

It then tries to restart the kernel another 4 times before giving up.

It looks to me, like either root or jupyter is missing a dependency here, resp. the combination of the two results in an additional dependency.

[tmadlener]

For completeness: Actually installing metakernel via pip makes the c++ notebooks work.

[tmadlener]

I think I got this figured out. In order for additional kernels to show up in the jupyter lab interface they have to be on JUPYTER_PATH. By default the spack recipes only put py-nbconvert on there (as it is explicitly set there here).

On the other hand sourcing thisroot.sh from a root installation will additionally put $ROOTSYS/etc/notebook onto it. In the usual setup script generated by spack thisroot.sh is not sourced. However, the setup.sh script that is used in sw.hsf.org explicitly does so at the very end:

#
# figuring out the setup_actual script
#
source ${setup_actual}
# for swan (temp fix, should be in root recipe)
source ${ROOTSYS}/bin/thisroot.sh
export CPPYY_BACKEND_LIBRARY=${ROOTSYS}/lib/libcppyy_backend3_9

So in order to fix this properly the following things would need to happen (assuming root always builds with jupyter/notebook support):

• The root spack recipe would either
  • need to detect whether jupyter is available and in that case populate JUPYTER_PATH accordingly
  • or actually depend on jupyter and also populate JUPYTER_PATH accordingly
• In either case also depend on the metakernel package (no spack recipe yet)