Revert conda environment handling

[ewels]

Previously, we created a conda environment with a specific name and then manually added this directory to the PATH. This worked fine, but felt a little hacky. After some discussion (@apeltzer @sven1103 @ewels @pditommaso and others), we removed it. Instead, we now install the environment to the base conda environment.

This all seemed fine, until we just came across a strange bug. After some investigation, it seems that the host filesystem conda installation was taking priority over the base conda environment (due to configuration files in the home directory). Combined with conflicting path mounts we ended up with a steaming mess and non-functional software. This issue creates two problems:

• Software breaks (as above)
• Software works, but is actually the host installation and not from the container (not reproducible)

Either we need a fix, or we can revert to the previous method. Manually prepending to the PATH env skips all of these problems, as we're essentially not using conda any more.

[pditommaso]

it seems that the host filesystem conda installation was taking priority over the base conda environment (due to configuration files in the home directory)

How config files (of what) in the home can affect the PATH ?

Installing the deps in a separate conda env and "activating" it by adding it by adding the env path to the PATH variable, may not be safe when the source activate command adds custom variables in the target environment.

[HadrienG]

Closed with 7152f8d

[ewels]

Sorry @pditommaso - inbox is overflowing and only just saw your comment..

How config files (of what) in the home can affect the PATH ?

The conda config file: ~/.condarc was showing up when we ran conda info inside the singularity image (singularity shell pipeline.simg). This was adding the paths for the local conda environments and somehow the local environments were being activated instead of using the root conda env. Not sure if it was the .condarc file that was doing this activation or something else..?

..may not be safe when the source activate command adds custom variables in the target environment.

When would source activate add custom variables? As we have complete control of the environment.yml in the pipeline, we should be able to ensure that this isn't a problem right? I'm still inclined to revert as the previous method has been working well across a range of systems, and we've had problems as soon as we switch to the root conda env.

[pditommaso]

The conda config file: ~/.condarc was showing up when we ran conda info inside

the singularity image (singularity shell pipeline.simg). You are saying the `~/.condarc` file in host environment affects the container execution, right? But still I'm not understanding how this can happen. Once the conda root environment is updated in the container, what is trigger the usage of the `~/.condarc` file affecting the task environment?

When would source activate add custom variables?

My understanding is that `source activate` not just modify the PATH env, but it *may* update other env variables such `LD_LIBRARY_PATH`, etc. So, even manually updating the `PATH` prepending the env directory works in most cases, it could not work for some specific recipes. p p

…

On Fri, Aug 10, 2018 at 10:55 AM, Phil Ewels ***@***.***> wrote: Sorry @pditommaso <https://github.com/pditommaso> - inbox is overflowing and only just saw your comment.. How config files (of what) in the home can affect the PATH ? The conda config file: ~/.condarc was showing up when we ran conda info inside the singularity image (singularity shell pipeline.simg). This was adding the paths for the local conda environments and somehow the local environments were being activated instead of using the root conda env. Not sure if it was the .condarc file that was doing this activation or something else..? When would source activate add custom variables? As we have complete control of the environment.yml in the pipeline, we should be able to ensure that this isn't a problem right? I'm still inclined to revert as the previous method has been working well across a range of systems, and we've had problems as soon as we switch to the root conda env. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#114 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAx3SEW7rM2izkGwHPKA4SnTnmasG30Zks5uPUp4gaJpZM4V1WOQ> .