Add Gitpod environment to nf-core tools

[mahesh-panchal]

Adds a gitpod environment with

• Nextflow
• nf-core tools
• pytest-workflow
• Docker
• Conda
• Mamba

Singularity might be possible through the docker image: https://github.com/kaczmarj/singularity-in-docker

My fork has prebuilds enabled.
Check if it meets your needs @drpatelh . We can then enable prebuilds on the nf-core repo.

I've not added extensions, but check the extensions tab if there's something you think might be useful.

[drpatelh]

Let's add this functionality to the pipeline template too here. Would it need to be any different?

As mentioned on Slack my main concern is re-building images unecessarily. I don't know how all of this works on the back-end but be great if we can re-use these images as much as possible, especially since they will now potentially be used across all nf-core pipelines too after we have added it to the template.

[mahesh-panchal]

We should build it differently then if we want to propagate this to the other repos.

.gitpod.yml: This goes into the root of the pipeline template, and the root of this workflow.

# Pull prebuilt image from ghcr - Built using GH actions on nf-core/tools
image: ghcr.io/nf-core/gitpod:1.0

And then put the Dockerfile somewhere in this repo, and use a GH action to publish it to ghcr.io on modification. Where is a good folder to put the Dockerfile? Also if not ghcr.io, then where?

This means image is built when the Dockerfile is updated here, and then the other repo's just pull the image ready to work.

[drpatelh]

Where is a good folder to put the Dockerfile?

I would be tempted to put it in .github/ somewhere? Maybe .github/.gitpod/Dockerfile / .gitpod/Dockerfile ? Don't know if that would clash with any standard Github Actions or the likes though.

Also if not ghcr.io, then where?

We have an nf-core Dockerhub account we can push to but I guess we would only want an Actions to update the image on the tools repo to keep things centralised (as opposed to having an Actions to update the container in the pipeline repos too).

Another consideration is how we perform the tagging and how important that is overall i.e. do we always just pull the latest within pipelines / other nf-core repos or do we pull specific tagged versions?

[mahesh-panchal]

Where is a good folder to put the Dockerfile?

I would be tempted to put it in .github/ somewhere? Maybe .github/.gitpod/Dockerfile / .gitpod/Dockerfile ? Don't know if that would clash with any standard Github Actions or the likes though.

Why put it in a hidden folder? While it's unlikely to need lots of updating, I think it's better not to "hide" it.
What about nf-core/gitpod/Dockerfile ?

Also if not ghcr.io, then where?

We have an nf-core Dockerhub account we can push to but I guess we would only want an Actions to update the image on the tools repo to keep things centralised (as opposed to having an Actions to update the container in the pipeline repos too).

Ya, exactly. Keep it centralised. But the built image needs to be hosted somewhere. Are there still costs to using Dockerhub?
We can set the GH actions git context in the build to something like:

on:
  push:
      branches:
        - 'master'
      paths:
        - 'nf-core/gitpod/Dockerfile'

Another consideration is how we perform the tagging and how important that is overall i.e. do we always just pull the latest within pipelines / other nf-core repos or do we pull specific tagged versions?

I would say always pull the latest. There's really no need to pull an older environment that I could think of.

[mahesh-panchal]

Kept the original .gitpod.yml file so @mashehu or someone else can test the environment.
Added the correct .gitpod.yml to the pipeline-templates folder, and my attempt at a GH action for it

[mashehu]

I created a test pipeline from this PR (using my docker-hub image for now). Everything seems to work as intended 🤩 (initial start up time was the normal 30-60 sec). You can play around with it if you want: https://github.com/mashehu/nf-core_test (I included also the nf-core extensions (except for liveshare, because that functionality is already included on gitpod)

[mahesh-panchal]

Anyone know how to test the GH actions workflow?

[mashehu]

Anyone know how to test the GH actions workflow?

You could try to have it run in your repo by removing the if-clause (and adding the dockerhub secrets).

[mahesh-panchal]

Can someone who has permissions build and upload the image to the nf core dockerhub?

[mashehu]

Why do you need it preloaded? It will be pushed with this PR, no?

[mahesh-panchal]

True.
Where do the credentials come from here though?

[mashehu]

They are part of the github org secrets for nf-core.

[mahesh-panchal]

    command: |
      NFCORE_BUILD_PATH=$HOME/nf-core_build
      pip install --upgrade -r requirements-dev.txt -e . -t $NFCORE_BUILD_PATH
      PATH="$NFCORE_BUILD_PATH:$PATH"

That should allow one to develop with the current nf-core build.
Using tasks.env seems to disable using the docker image for some reason ( so I dropped it )
Also added using the latest tag.

[mahesh-panchal]

OK. Next attempt. My prebuild is now using a cached version of the Docker image so pip is not installed via conda.
If you install pip manually with conda install pip and then rerun the python -m pip install ... command, it should hopefully work now, overwriting the nf-core installed via conda.

[mashehu]

hmm, I don't think this works. python -m pip install --upgrade -r requirements-dev.txt -e . throws /opt/conda/bin/python: No module named pip. Do we maybe need to run this in a conda environment?

[mahesh-panchal]

hmm, I don't think this works. python -m pip install --upgrade -r requirements-dev.txt -e . throws /opt/conda/bin/python: No module named pip. Do we maybe need to run this in a conda environment?

Sorry, I guess I wasn't clear enough with my previous message. The current docker image used for the Gitpod environment is the cached one, so it doesn't have pip installed in the conda environment. The current Dockerfile though does have it. We have a bit of a chicken-egg scenario here. The Dockerfile needs to be rebuilt to make a new image but since I'm not building from my Docker Hub any more, the new image hasn't been built.

One way to test locally is to do what Gitpod is doing in the background. Build the Docker image, clone the tools repo into workspace, and then do the python -m pip install .... If that works for you, then it should work in the Gitpod.

[mahesh-panchal]

Local test instructions:

# locally build docker image
docker build -f nf_core/gitpod/gitpod.Dockerfile -t gitpod:test .
docker run -it --rm gitpod:test bash
# Inside docker image
cd /workspace/
sudo git clone https://github.com/nf-core/tools.git
sudo chown -R gitpod:gitpod .
cd tools/
python -m pip install --upgrade -r requirements-dev.txt -e .
nf-core
vim nf_core/__main__.py
nf-core

[mashehu]

Yes, I tested it also now locally and it works great! Awesome!
I am currently trying to squash the image to make it hopefully a bit smaller (it is currently 7.7 GB uncompressed).
I just want to test the template gitpod file once more and then we are good to go, I think.

[mashehu]

Tested also the template and everything works as expected. 🚀

[mahesh-panchal]

Thanks for all the testing!

[mashehu]

We would save around 1.5 GB by squashing the image, but the gh action can't do that at the moment (it is an experimental feature), but would be great to get the size still down somehow. Something for a feature release 🙂

[mahesh-panchal]

Well the image doesn't need a lot of what's in there and could likely be made a whole lot lighter, but for now I'm not going to mess too much with it. It shouldn't really be a problem as I see it so far.

[mashehu]

Well, a smaller image means shorter start up time, plus lower resource consumption. Both things are nice to have. 🙂

[mahesh-panchal]

@drpatelh Anything else that needs to be done here before merging?

[drpatelh]

Sorry, missed this. This is amaaaazing! Thanks guys :)