Base AiiDAlab image on jupyter/base-notebook image and support AiiDA 2.0 #254

yakutovicha · 2022-06-08T15:11:05Z

Do not run reentry scan after app installation aiidalab#301 (https://github.com/aiidalab/aiidalab/milestone/2)
Aiida 2.0 aiidalab-home#102
Revise tests (the wait-for-services executable is no longer present)
~~Mount PostgreSQL and RabbitMQ data folders into home directory (to keep AiiDA data self-contained)~~ (not possible)
Install aiidalab package from conda-forge

Edited by @csadorf

csadorf · 2022-06-21T13:14:18Z

Open questions:

Previously, we were populating the user's folder at the startup time. If we want to base the image on official Jupyter ones, we might rely on the existing user (jovyan). The question is: what to do with the user's data (.ssh, .local, apps, etc). A possible solution would be to create volumes for all of them.

If mounting volumes to store all the persistent user's information is the way to go, which one should mount:

.ssh

.local

apps

@yakutovicha In aiidalab-launch we already have a dedicated volume for the conda environment, because some of the conda libraries cannot be installed in all host file systems, but what would be the motivation to having dedicated volumes for each of the other directories?

csadorf · 2022-07-27T14:25:48Z

This PR is almost ready to go, we have the following outstanding issues:

Do not run reentry scan after app installation aiidalab#301 (https://github.com/aiidalab/aiidalab/milestone/2)
Revise tests (the wait-for-services executable is no longer present)

Dockerfile

csadorf · 2022-08-05T14:49:05Z

Blocked by: conda-forge/staged-recipes#19940

- The PGSQL and RabbitMQ services are expected to started in separate containers. - Do not separate the Jupyter and AiiDA Python kernels; use shared requirements file. - Use requirements file for environment-wide pinning (conda and pip). - Supports AiiDA 2.0. Co-authored-by: Aliaksandr Yakutovich <yakutovicha@gmail.com>

ltalirz · 2022-08-15T09:10:04Z

@csadorf I guess this is no longer blocked?

csadorf · 2022-08-16T08:20:04Z

@csadorf I guess this is no longer blocked?

Kind of. We'll have to make a release of https://github.com/aiidalab/aiidalab and then we can switch to installing from conda. I'm also trying to merge #276, but I'm having some trouble getting GitHub actions to actually build the aiidalab image. Feel free to have a look, maybe you can spot the problem?

The refactoring into an actual hierarchical stack will enable me to implement the one-off migration image needed for the switch to AiiDA 2.0 and the slightly changed user environment which is the last task on #240.

ltalirz · 2022-08-16T08:30:52Z

I'm also trying to merge #276, but I'm having some trouble getting GitHub actions to actually build the aiidalab image. Feel free to have a look, maybe you can spot the problem?

The Dockerfile is starting from an aiidalab/aiida image. I don't see this on https://hub.docker.com/u/aiidalab

csadorf · 2022-08-16T09:20:44Z

The Dockerfile is starting from an aiidalab/aiida image. I don't see this on https://hub.docker.com/u/aiidalab

Thanks for checking! The image is built locally in the step prior ("Build aiida image"), however I think I finally identified the underlying problem: the buildx context is probably not using the docker driver (compare docker/buildx#301). I'd suggest to move any further discussion to #276 .

csadorf · 2022-08-18T18:11:29Z

Mounting everything into the same volume is likely blocked by moby/moby#32582 . See also https://stackoverflow.com/questions/38164939/can-we-mount-sub-directories-of-a-named-volume-in-docker .

Port is configurable via a command line argument ('-p' / '--port).

Using an older jupyter stack version we cannot use the ENV_NOTEBOOK_ARGS environment variable to configure the startup command.

csadorf · 2022-08-19T10:27:02Z

@yakutovicha This is ready to go, do you want to have another look at it before I merge?

yakutovicha

Hi @csadorf, there are a couple of issues/questions. I would also like to carefully test this before merging it. IMO, we need to have at least one working Kubernetes instance using those images to confidently go ahead.

Another thing we should not forget is the ssh-agent to be able to connect to machines that require ssh keys with password

environment.yml

stack/base/requirements.txt

Co-authored-by: Carl Simon Adorf <simon.adorf@epfl.ch>

csadorf · 2022-08-19T14:19:28Z

@yakutovicha I'll merge this, ok?

yakutovicha · 2022-08-19T14:27:51Z

@yakutovicha I'll merge this, ok?

sure, got ahead.

[EDIT] please either add the ssh-agent as I mentioned above or make an issue to not forget about this. People are actually using it.

Example: The lab image should be rebuilt when the base image is changed. No need to further declare depencies since the process is already optimized through docker layer caching.

And install the ssh-singleagent.sh script.

csadorf · 2022-08-22T15:08:38Z

[EDIT] please either add the ssh-agent as I mentioned above or make an issue to not forget about this. People are actually using it.

@yakutovicha Implemented in 323bb51, I'm not sure how to test it.

csadorf · 2022-08-24T12:01:58Z

@yakutovicha I'll merge this, ok?

yakutovicha · 2022-08-25T14:10:47Z

@yakutovicha I'll merge this, ok?

yes, please go ahead

) * Major revision that refactors the previous single image into a base and a lab image, where the former contains only AiiDA and the latter adds the AiiDAlab related software. * The PGSQL and RabbitMQ services are expected to be started in separate containers, e.g., with docker-compose. * There is only a single Python environment with shared dependencies, unlike the previous separation with one environment for the Jupyter server and the other one for the AiiDA Python kernels. * Some requirements, e.g., the installed AiiDA version are pinned for both conda/mamba and pip to prevent the user from inadvertently breaking the environment; the user can override this or alternatively create a dedicated environment. * Add tests that are implemented with pytest. * Common repository tasks such as building the stack, running tests, and starting a lab server can be run with pydoit. * Supports Python 3.9.2 and AiiDA 2.0.0. Co-authored-by: Aliaksandr Yakutovich <yakutovicha@gmail.com>

* Major revision that refactors the previous single image into a base and a lab image, where the former contains only AiiDA and the latter adds the AiiDAlab related software. * The PGSQL and RabbitMQ services are expected to be started in separate containers, e.g., with docker-compose. * There is only a single Python environment with shared dependencies, unlike the previous separation with one environment for the Jupyter server and the other one for the AiiDA Python kernels. * Some requirements, e.g., the installed AiiDA version are pinned for both conda/mamba and pip to prevent the user from inadvertently breaking the environment; the user can override this or alternatively create a dedicated environment. * Add tests that are implemented with pytest. * Common repository tasks such as building the stack, running tests, and starting a lab server can be run with pydoit. * Supports Python 3.9.2 and AiiDA 2.0.0. Co-authored-by: Aliaksandr Yakutovich <yakutovicha@gmail.com>

yakutovicha mentioned this pull request Jun 8, 2022

Launch additional services in separate container aiidalab/aiidalab-launch#119

Open

4 tasks

yakutovicha mentioned this pull request Jun 21, 2022

Build Docker image using Ansible. #242

Closed

csadorf linked an issue Jun 21, 2022 that may be closed by this pull request

Implement AiiDAlab based on an official Jupyter image #259

Closed

yakutovicha mentioned this pull request Jul 5, 2022

Add an explicit check of the notebook kernel. aiidalab/aiidalab#293

Closed

3 tasks

csadorf changed the title ~~Prepare AiiDAlab image to be used with docker-compose~~ Base AiiDAlab image on jupyter/base-notebook image Jul 19, 2022

csadorf self-assigned this Jul 19, 2022

csadorf mentioned this pull request Jul 19, 2022

Switch to docker-compose approach. aiidalab/aiidalab-launch#127

Draft

csadorf force-pushed the feature/minimal-image-for-docker-compose branch from c98ea05 to 457f31b Compare July 27, 2022 14:17

csadorf linked an issue Jul 27, 2022 that may be closed by this pull request

Support AiiDA core 2.0 #240

Closed

8 tasks

csadorf reviewed Jul 27, 2022

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

csadorf changed the title ~~Base AiiDAlab image on jupyter/base-notebook image~~ Base AiiDAlab image on jupyter/base-notebook image and support AiiDA 2.0 Jul 27, 2022

This was referenced Jul 27, 2022

Support AiiDA core 2.0 #240

Closed

Support AiiDA 2.0 #241

Closed

Improve documentation on how to customize the user environment #250

Open

csadorf force-pushed the feature/minimal-image-for-docker-compose branch 2 times, most recently from 74ae759 to 6894d22 Compare August 5, 2022 14:45

csadorf added the blocked This issue/PR is blocked by another issue/PR. label Aug 5, 2022

csadorf force-pushed the feature/minimal-image-for-docker-compose branch 2 times, most recently from 29a6647 to 3012766 Compare August 5, 2022 15:01

csadorf and others added 3 commits August 8, 2022 10:30

Add black to pre-commit hooks.

fd1fc00

Implement tests.

667fda5

csadorf force-pushed the feature/minimal-image-for-docker-compose branch from 3012766 to 667fda5 Compare August 8, 2022 08:30

csadorf added 7 commits August 19, 2022 07:21

Expose AiiDAlab on host for doit up task.

68770bb

Port is configurable via a command line argument ('-p' / '--port).

Pin the installed aiidalab version.

716dbcf

Start home app by default.

adec776

Using an older jupyter stack version we cannot use the ENV_NOTEBOOK_ARGS environment variable to configure the startup command.

Use consistent casing for Docker args.

256e7a3

Add tests for package pinning and constraints.

f941537

Increase verbosity for 'doit tests'.

c0b626e

Trigger CI.

38f0e3e

csadorf mentioned this pull request Aug 19, 2022

Refactor into stack #276

Closed

csadorf approved these changes Aug 19, 2022

View reviewed changes

csadorf marked this pull request as ready for review August 19, 2022 10:26

yakutovicha commented Aug 19, 2022

View reviewed changes

environment.yml Outdated Show resolved Hide resolved

stack/base/requirements.txt Outdated Show resolved Hide resolved

yakutovicha mentioned this pull request Aug 19, 2022

Update dependencies for aiida-2.x #277

Merged

csadorf changed the base branch from develop to stack August 19, 2022 14:01

csadorf and others added 2 commits August 19, 2022 16:02

Remove prefix from conda environment file.

8267f68

Update stack/base/requirements.txt

9ed064e

Co-authored-by: Carl Simon Adorf <simon.adorf@epfl.ch>

csadorf added 2 commits August 22, 2022 16:59

Trigger build of entire stack on change to any of the images.

342b227

Example: The lab image should be rebuilt when the base image is changed. No need to further declare depencies since the process is already optimized through docker layer caching.

Move ssh configuration into base image.

323bb51

And install the ssh-singleagent.sh script.

csadorf merged commit 6a3c63e into stack Aug 25, 2022

csadorf deleted the feature/minimal-image-for-docker-compose branch August 25, 2022 16:04

danielhollas mentioned this pull request Oct 2, 2023

Vendor load-singlesshagent.sh script #409

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Base AiiDAlab image on jupyter/base-notebook image and support AiiDA 2.0 #254

Base AiiDAlab image on jupyter/base-notebook image and support AiiDA 2.0 #254

yakutovicha commented Jun 8, 2022 •

edited by csadorf

Loading

csadorf commented Jun 21, 2022

csadorf commented Jul 27, 2022 •

edited

Loading

csadorf commented Aug 5, 2022

ltalirz commented Aug 15, 2022

csadorf commented Aug 16, 2022

ltalirz commented Aug 16, 2022

csadorf commented Aug 16, 2022

csadorf commented Aug 18, 2022

csadorf commented Aug 19, 2022

yakutovicha left a comment

csadorf commented Aug 19, 2022

yakutovicha commented Aug 19, 2022 •

edited

Loading

csadorf commented Aug 22, 2022

csadorf commented Aug 24, 2022

yakutovicha commented Aug 25, 2022

Base AiiDAlab image on jupyter/base-notebook image and support AiiDA 2.0 #254

Base AiiDAlab image on jupyter/base-notebook image and support AiiDA 2.0 #254

Conversation

yakutovicha commented Jun 8, 2022 • edited by csadorf Loading

csadorf commented Jun 21, 2022

csadorf commented Jul 27, 2022 • edited Loading

csadorf commented Aug 5, 2022

ltalirz commented Aug 15, 2022

csadorf commented Aug 16, 2022

ltalirz commented Aug 16, 2022

csadorf commented Aug 16, 2022

csadorf commented Aug 18, 2022

csadorf commented Aug 19, 2022

yakutovicha left a comment

Choose a reason for hiding this comment

csadorf commented Aug 19, 2022

yakutovicha commented Aug 19, 2022 • edited Loading

csadorf commented Aug 22, 2022

csadorf commented Aug 24, 2022

yakutovicha commented Aug 25, 2022

yakutovicha commented Jun 8, 2022 •

edited by csadorf

Loading

csadorf commented Jul 27, 2022 •

edited

Loading

yakutovicha commented Aug 19, 2022 •

edited

Loading