Julia fails when using a singularity container

[rbeucher]

Hi All,

We are using singularity containers (.sif file generated from the official ESMValTool docker images) to run ESMValTool on our system.
Singularity containers have limited write access which causes issues when pre-compiling Julia packages.

I was wondering if anyone had success running the ESMValTool docker images as singularity containers.
My knowledge of Julia is limited but maybe it is just a matter of binding the right folder.
Now the fact that we are dealing with a conda environment might make things harder? maybe?

Here is a dump of the log for the recipe_julia.yml:

ERROR: LoadError: IOError: mkdir("/opt/conda/envs/esmvaltool/share/julia/compil
Stacktrace:
  [1] uv_error
    @ ./libuv.jl:97 [inlined]
  [2] mkdir(path::String; mode::UInt16)
    @ Base.Filesystem ./file.jl:185
  [3] mkpath(path::String; mode::UInt16)
    @ Base.Filesystem ./file.jl:241
  [4] mkpath
    @ ./file.jl:235 [inlined]
  [5] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal
    @ Base ./loading.jl:1668
  [6] compilecache
    @ ./loading.jl:1651 [inlined]
  [7] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1337
  [8] _require_prelocked(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1200
  [9] macro expansion
    @ ./loading.jl:1180 [inlined]
 [10] macro expansion

Thanks for your help!

Romain

[valeriupredoi]

@rbeucher is that the full stack trace? What Julia is it installing, it could be a permissions error, see JuliaLang/julia#38411

[bouweandela]

Are you actually planning to use ESMValTool with Julia diagnostic scripts? That would be interesting for us to know because I have the impression that maintaining support for Julia as a diagnostic language is a drain on our time with little benefit to our users. So far the only scientific recipe in Julia ever contributed is recipe_rainfarm.yml.

See #2575 for some previous discussion on singularity containers.

[rbeucher]

@bouweandela It is more of a nice to have really. We have a few Julia hardcore people...
I agree it is very niche and not a high priority.

It's definitely a permission issue. Singularity containers don't have root access. I think it's possible but my knowledge of Julia is limited.
I was hoping someone would have a quick solution.

[bouweandela]

If anyone is actually using it we're happy to support it of course! Maybe you could ask one of those hardcore Julia people to come and have a look here at how we're installing Julia dependencies? It is probably outdated again.

The docker container is built from this recipe: https://github.com/ESMValGroup/ESMValTool/blob/main/docker/Dockerfile

the Julia installation code lives here:
https://github.com/ESMValGroup/ESMValTool/tree/main/esmvaltool/install

and the code we use for running a Julia diagnostic in the correct Julia environment lives here:
https://github.com/ESMValGroup/ESMValCore/blob/6e61590e3f58bf9e66ba7f6716e8fca8fbb1e168/esmvalcore/_task.py#L350-L411