conda distribution

[chrisjsewell]

It would be good to have an aiida distribution on conda-forge (a lot easier than messing around with homebrew and ports etc).
I see you've created an environment.yml previously #2081

Here an attached environment.yml that (at least today) works with the provenance redesign branch, for a python 3.6 environment, that allows aiida_core to be installed (and work) with no dependencies:

conda env create --file=conda_env_py36_aiida1b1.yaml
conda activate aiida_1b1
pip install --no-deps -e .

conda_env_py36_aiida1b1.txt (I had to change the extension from yml, for GitHub to accept it)

There was some minor tweaks to versions (see comments), but they don't seem to break anything

[chrisjsewell]

On a related note, I use this script to spin-up different 'aiida environments',
defined by a yaml config file

[ltalirz]

@chrisjsewell you may want to try conda install -c conda-forge aiida-core ;-)

Currently, we only have 0.12.2 there and so far I think it is not very well tested (=haven't heard of many people using it, but I now see there are >1000 downloads already...).
But the plan is definitely to promote it more - the beauty of conda is that it even allows to install postgresql and rabbitmq (currently not yet added as a dependency but planning to do this).

[ltalirz]

Now started looking through your script - thanks for your work, much appreciated!
As soon as the beta1 is out I will prepare the recipe on conda-forge.
Can I count on you for testing / assistance?

[chrisjsewell]

@ltalirz perfect I didn't notice that :) yes if you look at my yaml, I have
postgresql and rabbitmq in there already

Can I count on you for testing / assistance?

absolutely

[chrisjsewell]

aiida-crystal17 is now on conda-forge :)

This environment, runs and passes all my tests:

name: aiida_cry17_0_4_0_py27
channels:
  - conda-forge
  - bioconda # for chainmap
  - cjs14 # for pgtest
  - defaults
dependencies:
  - aiida-core==0.12.2
  - chainmap==1.0.2
  - aiida-crystal17==0.4.0
  - pytest
  - pytest-timeout
  - pgtest

[ltalirz]

Very cool!
I'll be updating the docs for aiida 0.12.3 (working on the release now), streamlining the installation instruction and also pointing out the conda-forge alternative.

There are essentially two reasons I didn't yet include postgres as a dependency in the 0.12.2 conda package:

• conda didn't have an "extras" mechanism, so you are forcing everyone to download it, even if they don't need it (not sure whether this changed in the meanwhile?)
• I wasn't quite sure what to tell people on how to run it - of course they can always start it by hand (or we could include scripts to do that) but it seems like you rather want it to be a system service

If you have any thoughts on these issues, I'd be very happy to hear them

[chrisjsewell]

conda "extras" mechanism

Had a little look into it for another package, here's some relevant issues:

• Optional groups of dependencies conda/conda#7502
• Optional dependencies for conda packages conda/conda#1696
• Mirroring the pip sub-package (extras) dependency semantics conda/conda#2984

rather want it to be a system service

Yeh I guess like travis's mechanism, would be cool. I'll leave that to you though!

[ltalirz]

Thanks for the hints concerning "extras" - Yes, I could imagine including e.g. an aiida-core and aiida-core.services package as outputs.

Yeh I guess like travis's mechanism, would be cool. I'll leave that to you though!

So, just that I understand: you currently use postgres and rabbitmq installed via conda and always start it manually? Would you be happy with this solution to start with?

[chrisjsewell]

Ok so I've redone my activate_aiida script into a conda package, which effectively spits the process into two stages:

1. Conda Environment
   1. (first time only) create it from environment.yml
   2. activate environment
   3. (optional, first time only) pip install local development packages (pip install --no-deps -e .) # TODO
   4. (optional) set branches of local development packages
2. AiiDA Environment
   1. start SQL
   2. (first time only) create SQL User & Database
   3. rabbitmq-server -detached (aiida_core >= v1)
   4. reentry scan -r aiida
   5. (first time only) create AiiDA Database
   6. (first time only) setup verdi tab completion
   7. start aiida daemon
   8. (optional) import common nodes (created by verdi export created)

The following example creates and activates a full aiida environment from scratch,
without having to go through all the steps manually:

conda_environment.yaml:

name: aiida_0_12_2_190305
channels:
    - conda-forge
    - cjs14
    - bioconda
    - defaults
dependencies:
    - aiida-core ==0.12.2
    - chainmap ==1.0.2
    - postgresql >=9.4
    - aiida-crystal17 =0.4
    - aiida-quantumespresso ==2.1.1a2
    #- jupyter =1.0
    - activate-aiida

aiida_environment.yaml:

conda_env: aiida_0_12_2_activate
aiida_version: 0.12
aiida_path: /Users/cjs14/GitHub/aiida-cjs-working/databases/aiida/
db_pgsql:
  path: /Users/cjs14/GitHub/aiida-cjs-working/databases/pgsql/aiida_0_12_2_190305
  user: testuser
  user-password: xxxx  # should handle this more securely
  name: testdb
  port: 5432  
db_aiida:
  path: /Users/cjs14/GitHub/aiida-cjs-working/databases/aiida/aiida_0_12_2_190305
  profile: testprofile
  email: chrisj_sewell@hotmail.com
  first-name: chris
  last-name: sewell
  institution: imperial
import_nodes:
  - common_nodes.zip
git_branches:
  - path: /Users/cjs14/GitHub/aiida-lammps
    branch: master

Then run:

>> conda env create --file=conda_environment.yaml
>> conda activate aiida_0_12_2_activate
>> source activate-aiida -c -i aiida_environment.yaml

and to stop the daemon and pgsql: >> deactivate-aiida

and you can run source activate-aiida aiida_environment.yaml,
whenever you want to restart the database & daemon.

Some notes:

• the main script is still written in bash, this could probably be ported into python.
  the main sticking point I found though, is parsing environmental variables back to the parent process
• verdi daemon configureuser (in v0.12) doesn't allow the new daemon user to be parsed via an argument, so you have to do it manually. This is basically the only non-automated bit.
• could also add an aiida-backup script
• in your potential aiida-core.services it would be nice to include jupyter and load the aiida notebook extension during the conda install.

[ltalirz]

@chrisjsewell thanks a lot for your efforts, this looks very interesting!

I've already added aiida-core.services on conda-forge - once we've gotten some feedback on the activate_aiida python package, we can get it on conda-forge as well and include it as a dependency in aiida-core.services.

I'll come back to your "notes" later.

@giovannipizzi @sphuber In summary:

• It is now possible to install aiida-core 0.12, rabbitmq and postgres using conda install -c conda-forge aiida-core aiida-core.services (on linux, macos & windows; untested on windows though). The great advantage is that this makes the install simple and consistent across the three OS's. It does not provide integration with the system service infrastructures of the different OS's, however.
• @chrisjsewell has written a python package activate_aiida that fills this gap (and does a bit more). It provides two scripts activate-aiida and deactivate-aiida (currently still written in bash) that work together with the environment from the conda package to

1. start SQL
2. (first time only) create SQL User & Database
3. rabbitmq-server -detached (aiida_core >= v1)
4. reentry scan -r aiida
5. (first time only) create AiiDA Database
6. (first time only) setup verdi tab completion
7. start aiida daemon
8. (optional) import common nodes (created by verdi export created)

There are a few points we should decide:

1. activate_aiida is currently configured via a custom yaml format, see previous post. The information stored in there is very similar to what is stored in the config.json for an AiiDA profile, so I think we should merge the two in the sense that activate_aiida should be able to read an AiiDA configuration file, and we should consider adding information that is currently missing (e.g. we anyhow wanted to store the information of a "default user" in the config.json).
2. After working quite extensively with yaml for the deployment of materials cloud, I have to say that I find yaml a significantly better format for human-readable configuration files than JSON (you can add comments, it is more flexible and I find it as easy or easier to read as json). I would propose to move to yaml for the AiiDA config file - but happy to discuss.
3. I think it is not a bad idea to start with a separate package for the scripts - we can decide later whether they should be included in the core. This means, we should agree on the name. Things to think about:
   • should conda be in the name?
   • where should it be hosted?

Happy to hear your comments!

[giovannipizzi]

Thanks to both!

1. I agree to unify. You mean a 'default user' independent of the profile?

2. In general I agree, we should double check that the security issue in pyyaml is now fixed (probably I see, I see a few new releases in the past few days and no security warning from GitHub anymore)

3. no clear idea, let's discuss next week

[sphuber]

@giovannipizzi I muted the warning about pyyaml because I am pretty sure I found two different pyyaml packages two weeks ago, one of which had been abandoned and contained the problem that Github was flagging and the version we were using was another distribution that didn't have the problem. Now, strangely I can no longer find the two different packages and it seems pyyaml==5.1 has been released. We should see that that fixes the safe_dump problem and add it as dependency

[chrisjsewell]

FYI, I generally use the ruamel.yaml package as a replacement for pyyaml, since it fixed all the security issues years ago, and supports a newer version of the yaml format.

[ltalirz]

@giovannipizzi

You mean a 'default user' independent of the profile?

Indeed, namely the one used for autofilling verdi quicksetup, see here.

Regarding ruamel vs pyyaml

• see differences
• pyyaml: 93 open issues, 78 closed https://github.com/yaml/pyyaml/issues
• ruamel: 10 open issues, 270 closed https://bitbucket.org/ruamel/yaml/issues?status=new&status=open

Happy to switch to ruamel

[sphuber]

ruamel seems to be owned by a Dutch guy: i.e. 👍

[giovannipizzi]

Happy to switch as well. Also, when we go to yaml, we should 1. see if we can still support the automatic format detection and migration, and 2. also migrate in the same file the caching configuration (I imagine it should be possible using the "options" implemented by @sphuber: do we already have an issue for this?)

[ltalirz]

some updates:

• like the 0.12 series, develop now includes a working environment.yml
• creating a conda environment from environment.yml is tested on travis
• I've just added aiida-core 1.0b to conda-forge. Install it using conda install -c conda-forge aiida-core==1.0.0b1 aiida-core.services==1.0.0b1

@chrisjsewell could you check whether your activate_aiida package works with this?

[chrisjsewell]

Hey @ltalirz just checked;

For the current conda-forge package, the verdi console_script entry point path is wrong (this has been fixed in the develop branch).

Once that was the fixed, the only other change was to the verdi quicksetup interface.
After that change everything still worked.

v0.12.3

Usage: verdi quicksetup [OPTIONS]
Options:
  --profile TEXT
  --email TEXT           This email address will be associated with your data...
  --first-name TEXT
  --last-name TEXT
  --institution TEXT
  --backend [django|sqlalchemy]
  --db-name TEXT
  --db-user TEXT
  --db-user-pw TEXT
  --db-port INTEGER
  --repo TEXT
  --set-default / --no-set-default Whether to set new profile as default fo shell and daemon.
  --non-interactive     never prompt the user for input, read values from options

v1.0.0b1

Usage: verdi quicksetup [OPTIONS] [PROFILE_NAME]
Options:
  --email TEXT                   Email address that will be associated with your data...
  --first-name TEXT              First name of the user.
  --last-name TEXT               Last name of the user.
  --institution TEXT             Institution name of the user.
  --backend [django|sqlalchemy]  Database backend to use.
  --db-host TEXT                 Database server host.
  --db-name TEXT                 Database name.
  --db-username TEXT             Database user name.
  --db-password TEXT             Database user password.
  --db-port INTEGER              Database server port.
  --repository DIRECTORY         Absolute path for the file system repository.
  --set-default                  Set the profile as the new default.
  -n, --non-interactive          Non-interactive mode: never prompt for input.
  --only-config                  Only configure the user and skip creating the database.

[chrisjsewell]

A note on moving entirely away from a bash script:

The main issue was setting the $PGDATA and $AIIDA_PATH environmental variables in the parent terminal.
As explained here, you can only do that directly by having the python script open its own terminal. I imagine conda did something similar when they recently moved from source activate env to conda activate env. I had a quick look at their code, but its quite complex.

[ltalirz]

Closing this, since the main purpose of this issue (a conda distribution for aiida) was addressed long ago.

Opened issues chrisjsewell/activate_aiida#3 and #3310 for points raised in here.