Base AiiDAlab image on jupyter/base-notebook image and support AiiDA 2.0 #254
Conversation
@yakutovicha In aiidalab-launch we already have a dedicated volume for the conda environment, because some of the conda libraries cannot be installed in all host file systems, but what would be the motivation to having dedicated volumes for each of the other directories? |
csadorf
changed the title
csadorf
force-pushed
the
feature/minimal-image-for-docker-compose
branch
from
c98ea05
to
457f31b
Compare
|
This PR is almost ready to go, we have the following outstanding issues:
|
csadorf
changed the title
csadorf
force-pushed
the
feature/minimal-image-for-docker-compose
branch
2 times, most recently
from
74ae759
to
6894d22
Compare
|
Blocked by: conda-forge/staged-recipes#19940 |
csadorf
force-pushed
the
feature/minimal-image-for-docker-compose
branch
2 times, most recently
from
29a6647
to
3012766
Compare
- The PGSQL and RabbitMQ services are expected to started in separate containers. - Do not separate the Jupyter and AiiDA Python kernels; use shared requirements file. - Use requirements file for environment-wide pinning (conda and pip). - Supports AiiDA 2.0. Co-authored-by: Aliaksandr Yakutovich <yakutovicha@gmail.com>
csadorf
force-pushed
the
feature/minimal-image-for-docker-compose
branch
from
3012766
to
667fda5
Compare
|
@csadorf I guess this is no longer blocked? |
Kind of. We'll have to make a release of https://github.com/aiidalab/aiidalab and then we can switch to installing from conda. I'm also trying to merge #276, but I'm having some trouble getting GitHub actions to actually build the aiidalab image. Feel free to have a look, maybe you can spot the problem? The refactoring into an actual hierarchical stack will enable me to implement the one-off migration image needed for the switch to AiiDA 2.0 and the slightly changed user environment which is the last task on #240. |
The Dockerfile is starting from an |
Thanks for checking! The image is built locally in the step prior ("Build aiida image"), however I think I finally identified the underlying problem: the buildx context is probably not using the docker driver (compare docker/buildx#301). I'd suggest to move any further discussion to #276 . |
|
Mounting everything into the same volume is likely blocked by moby/moby#32582 . See also https://stackoverflow.com/questions/38164939/can-we-mount-sub-directories-of-a-named-volume-in-docker . |
Port is configurable via a command line argument ('-p' / '--port).
Using an older jupyter stack version we cannot use the ENV_NOTEBOOK_ARGS environment variable to configure the startup command.
|
@yakutovicha This is ready to go, do you want to have another look at it before I merge? |
There was a problem hiding this comment.
Hi @csadorf, there are a couple of issues/questions. I would also like to carefully test this before merging it. IMO, we need to have at least one working Kubernetes instance using those images to confidently go ahead.
Another thing we should not forget is the ssh-agent to be able to connect to machines that require ssh keys with password
Co-authored-by: Carl Simon Adorf <simon.adorf@epfl.ch>
|
@yakutovicha I'll merge this, ok? |
sure, got ahead. [EDIT] please either add the ssh-agent as I mentioned above or make an issue to not forget about this. People are actually using it. |
Example: The lab image should be rebuilt when the base image is changed. No need to further declare depencies since the process is already optimized through docker layer caching.
And install the ssh-singleagent.sh script.
@yakutovicha Implemented in 323bb51, I'm not sure how to test it. |
|
@yakutovicha I'll merge this, ok? |
yes, please go ahead |
) * Major revision that refactors the previous single image into a base and a lab image, where the former contains only AiiDA and the latter adds the AiiDAlab related software. * The PGSQL and RabbitMQ services are expected to be started in separate containers, e.g., with docker-compose. * There is only a single Python environment with shared dependencies, unlike the previous separation with one environment for the Jupyter server and the other one for the AiiDA Python kernels. * Some requirements, e.g., the installed AiiDA version are pinned for both conda/mamba and pip to prevent the user from inadvertently breaking the environment; the user can override this or alternatively create a dedicated environment. * Add tests that are implemented with pytest. * Common repository tasks such as building the stack, running tests, and starting a lab server can be run with pydoit. * Supports Python 3.9.2 and AiiDA 2.0.0. Co-authored-by: Aliaksandr Yakutovich <yakutovicha@gmail.com>
* Major revision that refactors the previous single image into a base and a lab image, where the former contains only AiiDA and the latter adds the AiiDAlab related software. * The PGSQL and RabbitMQ services are expected to be started in separate containers, e.g., with docker-compose. * There is only a single Python environment with shared dependencies, unlike the previous separation with one environment for the Jupyter server and the other one for the AiiDA Python kernels. * Some requirements, e.g., the installed AiiDA version are pinned for both conda/mamba and pip to prevent the user from inadvertently breaking the environment; the user can override this or alternatively create a dedicated environment. * Add tests that are implemented with pytest. * Common repository tasks such as building the stack, running tests, and starting a lab server can be run with pydoit. * Supports Python 3.9.2 and AiiDA 2.0.0. Co-authored-by: Aliaksandr Yakutovich <yakutovicha@gmail.com>
* Major revision that refactors the previous single image into a base and a lab image, where the former contains only AiiDA and the latter adds the AiiDAlab related software. * The PGSQL and RabbitMQ services are expected to be started in separate containers, e.g., with docker-compose. * There is only a single Python environment with shared dependencies, unlike the previous separation with one environment for the Jupyter server and the other one for the AiiDA Python kernels. * Some requirements, e.g., the installed AiiDA version are pinned for both conda/mamba and pip to prevent the user from inadvertently breaking the environment; the user can override this or alternatively create a dedicated environment. * Add tests that are implemented with pytest. * Common repository tasks such as building the stack, running tests, and starting a lab server can be run with pydoit. * Supports Python 3.9.2 and AiiDA 2.0.0. Co-authored-by: Aliaksandr Yakutovich <yakutovicha@gmail.com>
wait-for-servicesexecutable is no longer present)Mount PostgreSQL and RabbitMQ data folders into home directory (to keep AiiDA data self-contained)(not possible)Edited by @csadorf