How to update the conda base environment?

[rbeucher]

Hi,

I am struggling a bit, trying to get a new environment deployed. For some reasons, it works on my local machine. Mamba can solve the environment and install it.

I was gonna try to update the conda base environment but I am not sure how to do that without breaking things.

@dsroberts can you help?

R

[rbeucher]

Also, what about using micromamba instead of miniconda?

[dsroberts]

There is no automated procedure for updating the base environment. Its outside of the containerisation anyway, so updating it would just follow the standard procedure for updating a conda env. Micromamba looks interesting, I had no idea it existed. I'd gladly replace a base conda environment with micromamba, but I'd a) alias the 'conda' command to it and b) make sure that it does everything users need the 'conda' command to do. That way, users can follow external conda docs and everything should work as expected.

[rbeucher]

I am really struggling with the deployment of a conda environment that includes esmvaltool, ilamb etc...
I am reconsidering things a bit and was thinking that having a separate environment for ESMValtool (which has a lot of dependencies) would be a better approach going forward.
I tried to deploy a new environment with a different name, separate versioning, admin folder etc. but with the same base conda environment but I get an error with the build script. It can't find the base.sif file. It does not build the container as it considers that the container.def as not change.
It would be good to have your input. Are you planning to come to the NRI in the next few days?

[dsroberts]

Hmmm... The idea behind this setup was that the base conda installation would happen once per system, and any environments would be built on top of that. I can see the issue around versioning though, I guess it doesn't cope when you change it to build an 'ilamb-0.1' or 'esmvaltool-0.1' environment. I think that'd be a good feature to implement, maybe separating out different environment classes under the 'scripts' directory and separating out install_config.sh into site_config.sh and env_config.sh files would be the way to do that. Then the github action can detect which environment has changed, tell the scripts to source the appropriate env_config.sh and go from there.

I probably won't be in this week, but I'm happy to tee up something for next week. I'll ping you on slack and we can discuss further.

[dsroberts]

Hi @rbeucher. Reviving this issue as I took advantage of the January quiet time to revisit ways to handle creation and maintenance of different environments. This is the result of that. environment.yml files are now split into subdirectories of environments, and there are various hooks available for different behaviour required by different environments. My test repo has an example of an esmvaltool env and a simple xarray environment that can all be maintained and updated separately (or all at once if necessary). Github actions automatically detects which environment(s) have changed and runs the appropriate build/test/deploy steps.

I've also switched it to micromamba, which worked great apart from compatibility with nb_conda_kernels. It wasn't strictly necessary, but I've made some small modifications to micromamba around symlink handling and mamba info output just so that all of the available ipython kernels show up neatly when creating a new notebook in jupyterlab.

I'm planning to do the switchover on launch of the analysis-24.01 environment, which will be towards the end of next week. With any luck, users won't notice the change and I can quietly delete the entire conda base environment after a little while. Thanks for the discussion, and for raising this issue in the first place!

[rbeucher]

Hi @dsroberts,

Thanks! That looks great. I am going to have a look and will let you know how it goes. Thanks for sharing!

[dsroberts]

Completed in #64