Config: Check current working directory for existing config dir (#6322)

Currently, the location of the configuration directory could only be
controlled through the `AIIDA_PATH` environment variable. Here, the
heuristic is updated to also consider the current working directory or
any of its parent directories in case `AIIDA_PATH` is not set. This
provides another easy way for users to maintain multiple separate AiiDA
instances.

.gitignore

@@ -19,6 +19,8 @@
 .tox
 Pipfile
 
+.aiida
+
 # files created by coverage
 .cache
 .pytest_cache

docs/source/howto/installation.rst

@@ -293,68 +293,57 @@ This can be useful to temporarily enable deprecation warnings for a single comma
 
 Isolating multiple instances
 ----------------------------
-An AiiDA instance is defined as the installed source code plus the configuration folder that stores the configuration files with all the configured profiles.
-It is possible to run multiple AiiDA instances on a single machine, simply by isolating the code and configuration in a virtual environment.
 
-To isolate the code, make sure to install AiiDA into a virtual environment, e.g., with conda or venv, as described :ref:`here <intro:get_started:setup>`.
-Whenever you activate this particular environment, you will be running the particular version of AiiDA (and all the plugins) that you installed specifically for it.
-
-This is separate from the configuration of AiiDA, which is stored in the configuration directory which is always named ``.aiida`` and by default is stored in the home directory.
-Therefore, the default path of the configuration directory is ``~/.aiida``.
-By default, each AiiDA instance (each installation) will store associated profiles in this folder.
-A best practice is to always separate the profiles together with the code to which they belong.
-The typical approach is to place the configuration folder in the virtual environment itself and have it automatically selected whenever the environment is activated.
-
-The location of the AiiDA configuration folder can be controlled with the ``AIIDA_PATH`` environment variable.
-This allows us to change the configuration folder automatically, by adding the following lines to the activation script of a virtual environment.
-For example, if the path of your virtual environment is ``/home/user/.virtualenvs/aiida``, add the following line:
-
-.. code:: bash
-
-    $ export AIIDA_PATH='/home/user/.virtualenvs/aiida'
-
-Make sure to reactivate the virtual environment, if it was already active, for the changes to take effect.
+An AiiDA instance is defined by its configuration directory, which is always named ``.aiida``.
+It contains the configuration file, which holds all the profile information, daemon log files, and `PID files <https://en.wikipedia.org/wiki/Process_identifier>`_.
 
 .. note::
 
-   For ``conda``, create a directory structure ``etc/conda/activate.d`` in the root folder of your conda environment (e.g. ``/home/user/miniconda/envs/aiida``), and place a file ``aiida-init.sh`` in that folder which exports the ``AIIDA_PATH``.
+    Depending on the storage backend, a profile's data is typically also stored in this directory.
+    However, this is not necessarily always the case, as often the data location can be configured, and some storage backends use a database service that stores the data elsewhere on disk.
 
-You can test that everything works by first echoing the environment variable with ``echo $AIIDA_PATH`` to confirm it prints the correct path.
-Finally, you can check that AiiDA know also properly realizes the new location for the configuration folder by calling ``verdi profile list``.
-This should display the current location of the configuration directory:
+The location of the configuration directory is determined as follows:
 
-.. code:: bash
+1. First, the ``AIIDA_PATH`` environment variables is checked, which can be a colon-separated list of directories.
+The first directory that points to an existing configuration directory is selected.
+If no existing directories are found, the last directory defined in the variable is used and the configuration directory is created there if it did not already exist.
+2. If the ``AIIDA_PATH`` is not defined, the current working directory is checked, going up the hierarchy until the first existing configuration directory is encountered.
+3. If no existing configuration directory is found yet, the ``.aiida`` directory in the user's home folder is used, and is created automatically if it does not already exist.
 
-    Info: configuration folder: /home/user/.virtualenvs/aiida/.aiida
-    Critical: configuration file /home/user/.virtualenvs/aiida/.aiida/config.json does not exist
-
-The second line you will only see if you haven't yet setup a profile for this AiiDA instance.
-For information on setting up a profile, refer to :ref:`creating profiles<how-to:installation:profile>`.
+Examples
+........
 
-Besides a single path, the value of ``AIIDA_PATH`` can also be a colon-separated list of paths.
-AiiDA will go through each of the paths and check whether they contain a configuration directory, i.e., a folder with the name ``.aiida``.
-The first configuration directory that is encountered will be used as the configuration directory.
-If no configuration directory is found, one will be created in the last path that was considered.
-For example, the directory structure in your home folder ``~/`` might look like this::
+Consider the following directory structure::
 
-    .
+    ~
     ├── .aiida
-    └── project_a
-        ├── .aiida
+    ├── project_a
+    │   ├── .aiida
+    │   └── subfolder
+    │       └── .aiida
+    ├── project_b
+    │   ├── .aiida
+    │   └── subfolder
+    └── project_c
         └── subfolder
 
-If you leave the ``AIIDA_PATH`` variable unset, the default location ``~/.aiida`` will be used.
-However, if you set:
-
-.. code:: bash
-
-    $ export AIIDA_PATH='~/project_a:'
+The following table shows the configuration directory that is selected given a certain ``AIIDA_PATH`` variable and the current working directory:
 
-the configuration directory ``~/project_a/.aiida`` will be used.
++--------------------------------------------------------------+----------------------------------+
+| Variables                                                    | Configuration directory          |
++==============================================================+==================================+
+| ``AIIDA_PATH = '~/project_b/'``                              | ``~/project_b/.aiida``           |
+| ``AIIDA_PATH = '~/project_b/.aiida'``                        | ``~/project_b/.aiida``           |
+| ``AIIDA_PATH = '~/project_a/.aiida:~/project_b/.aiida'``     | ``~/project_a/.aiida``           |
+| ``AIIDA_PATH = '~/project_a/subfolder:~/project_a/.aiida:'`` | ``~/project_a/subfolder/.aiida`` |
+| ``CWD = '~/project_a/subfolder``                             | ``~/project_a/subfolder/.aiida`` |
+| ``CWD = '~/project_b/subfolder``                             | ``~/project_b/.aiida``           |
+| ``CWD = '~/project_c/subfolder``                             | ``~/.aiida``                     |
++--------------------------------------------------------------+----------------------------------+
 
-.. warning::
+.. tip::
 
-    If there was no ``.aiida`` directory in ``~/project_a``, AiiDA would have created it for you, so make sure to set the ``AIIDA_PATH`` correctly.
+    The output of ``verdi status`` contains the location of the matched configuration directory.
 
 
 .. _how-to:installation:configure:daemon-as-service:

src/aiida/manage/configuration/settings.py

@@ -66,45 +66,94 @@ def create_instance_directories() -> None:
         os.umask(umask)
 
 
-def set_configuration_directory(aiida_config_folder: pathlib.Path | None = None) -> None:
-    """Determine location of configuration directory, set related global variables and create instance directories.
+def get_configuration_directory():
+    """Return the path of the configuration directory.
 
-    The location of the configuration folder will be determined and optionally created following these heuristics:
+    The location of the configuration directory is defined following these heuristics in order:
 
-        * If an explicit path is provided by `aiida_config_folder`, that will be set as the configuration folder.
-        * Otherwise, if the `AIIDA_PATH` variable is set, all the paths will be checked to see if they contain a
-          configuration folder. The first one to be encountered will be set as `AIIDA_CONFIG_FOLDER`. If none of them
-          contain one, a configuration folder will be created in the last path considered.
-        * If the `AIIDA_PATH` variable is not set the `DEFAULT_AIIDA_PATH` value will be used as base path and if it
-          does not yet contain a configuration folder, one will be created.
+        * If the ``AIIDA_PATH`` variable is set, all the paths will be checked to see if they contain a
+          configuration folder. The first one to be encountered will be set as ``AIIDA_CONFIG_FOLDER``. If none of them
+          contain one, the last path defined in the environment variable considered is used.
+        * If ``AIIDA_PATH`` is not set, the current working directory is checked for an existing configuration directory
+          moving up the file hierarchy until the first is encountered or the root directory is hit.
+        * If an existing directory is still not found, the ``DEFAULT_AIIDA_PATH`` is used.
 
-    In principle then, a configuration folder should always be found or automatically created.
+    :returns: The path of the configuration directory.
     """
-    global AIIDA_CONFIG_FOLDER  # noqa: PLW0603
-    global DAEMON_DIR  # noqa: PLW0603
-    global DAEMON_LOG_DIR  # noqa: PLW0603
-    global ACCESS_CONTROL_DIR  # noqa: PLW0603
+    dirpath_config: pathlib.Path | None = None
+
+    if environment_variable := os.environ.get(DEFAULT_AIIDA_PATH_VARIABLE):
+        dirpath_config = get_configuration_directory_from_envvar(environment_variable)
+    else:
+        dirpath_config = get_configuration_directory_from_cwd()
 
-    if aiida_config_folder is not None:
-        AIIDA_CONFIG_FOLDER = aiida_config_folder
+    # If no existing configuration directory is found, fall back to the default
+    if dirpath_config is None:
+        dirpath_config = pathlib.Path(DEFAULT_AIIDA_PATH).expanduser() / DEFAULT_CONFIG_DIR_NAME
 
-    elif environment_variable := os.environ.get(DEFAULT_AIIDA_PATH_VARIABLE):
-        # Loop over all the paths in the `AIIDA_PATH` variable to see if any of them contain a configuration folder
-        for base_dir_path in [path for path in environment_variable.split(':') if path]:
-            AIIDA_CONFIG_FOLDER = pathlib.Path(base_dir_path).expanduser()
+    return dirpath_config
 
-            # Only add the base config directory name to the base path if it does not already do so
-            # Someone might already include it in the environment variable. e.g.: AIIDA_PATH=/home/some/path/.aiida
-            if AIIDA_CONFIG_FOLDER.name != DEFAULT_CONFIG_DIR_NAME:
-                AIIDA_CONFIG_FOLDER = AIIDA_CONFIG_FOLDER / DEFAULT_CONFIG_DIR_NAME
 
-            # If the directory exists, we leave it set and break the loop
-            if AIIDA_CONFIG_FOLDER.is_dir():
-                break
-    else:
-        # The `AIIDA_PATH` variable is not set so use the default path and try to create it if it does not exist
-        AIIDA_CONFIG_FOLDER = pathlib.Path(DEFAULT_AIIDA_PATH).expanduser() / DEFAULT_CONFIG_DIR_NAME
+def get_configuration_directory_from_envvar(environment_variable: str) -> pathlib.Path:
+    """Return the path of a config directory from the ``AIIDA_PATH`` environment variable.
+
+    The environment variable should be a colon separated string of filepaths that either point directly to a config
+    directory or a path that contains a config directory. The first match is returned. If no existing config directory
+    is found, the last path in the environment variable is used.
+
+    :returns: The path of the configuration directory.
+    """
+    # Loop over all the paths in the ``AIIDA_PATH`` variable to see if any of them contain a configuration folder
+    for base_dir_path in [path for path in environment_variable.split(':') if path]:
+        dirpath_config = pathlib.Path(base_dir_path).expanduser()
+
+        # Only add the base config directory name to the base path if it does not already do so
+        # Someone might already include it in the environment variable. e.g.: ``AIIDA_PATH=/home/some/path/.aiida``
+        if dirpath_config.name != DEFAULT_CONFIG_DIR_NAME:
+            dirpath_config = dirpath_config / DEFAULT_CONFIG_DIR_NAME
+
+        # If the directory exists, we leave it set and break the loop
+        if dirpath_config.is_dir():
+            break
+
+    return dirpath_config
+
+
+def get_configuration_directory_from_cwd() -> pathlib.Path | None:
+    """Return the path of the first occurrence of a config directory in the hierarchy of the current working directory.
+
+    :returns: The path of an existing config directory in the hierarchy of the current working directory or ``None`` if
+        no such directory exists.
+    """
+    dirpath = pathlib.Path.cwd()
+
+    while dirpath.is_dir():
+        if (dirpath / DEFAULT_CONFIG_DIR_NAME).is_dir():
+            return dirpath / DEFAULT_CONFIG_DIR_NAME
+
+        if dirpath.parent == dirpath:
+            # End of the line, no more parent directories to check
+            break
+
+        # Check the parent directory next
+        dirpath = dirpath.parent
+
+    return None
+
+
+def set_configuration_directory(aiida_config_folder: pathlib.Path | None = None) -> None:
+    """Set the configuration directory, related global variables and create instance directories.
+
+    The location of the configuration directory is defined by ``aiida_config_folder`` or if not defined, the path that
+    is returned by ``get_configuration_directory``. If the directory does not exist yet, it is created, together with
+    all its subdirectories.
+    """
+    global AIIDA_CONFIG_FOLDER  # noqa: PLW0603
+    global DAEMON_DIR  # noqa: PLW0603
+    global DAEMON_LOG_DIR  # noqa: PLW0603
+    global ACCESS_CONTROL_DIR  # noqa: PLW0603
 
+    AIIDA_CONFIG_FOLDER = aiida_config_folder or get_configuration_directory()
     DAEMON_DIR = AIIDA_CONFIG_FOLDER / DEFAULT_DAEMON_DIR_NAME
     DAEMON_LOG_DIR = DAEMON_DIR / DEFAULT_DAEMON_LOG_DIR_NAME
     ACCESS_CONTROL_DIR = AIIDA_CONFIG_FOLDER / DEFAULT_ACCESS_CONTROL_DIR_NAME

tests/conftest.py

@@ -419,7 +419,7 @@ def suppress_internal_deprecations():
 def chdir_tmp_path(request, tmp_path):
     """Change to a temporary directory before running the test and reverting to original working directory."""
     os.chdir(tmp_path)
-    yield
+    yield tmp_path
     os.chdir(request.config.invocation_dir)
 
 

tests/manage/configuration/test_config.py

@@ -43,14 +43,18 @@ def cache_aiida_path_variable():
 
 @pytest.mark.filterwarnings('ignore:Creating AiiDA configuration folder')
 @pytest.mark.usefixtures('cache_aiida_path_variable')
-def test_environment_variable_not_set(tmp_path, monkeypatch):
+def test_environment_variable_not_set(chdir_tmp_path, monkeypatch):
     """Check that if the environment variable is not set, config folder will be created in `DEFAULT_AIIDA_PATH`.
 
     To make sure we do not mess with the actual default `.aiida` folder, which often lives in the home folder
     we create a temporary directory and set the `DEFAULT_AIIDA_PATH` to it.
+
+    Since if the environment variable is not set, the code will check for a config folder in the current working dir
+    or any of its parents, we switch the working directory to the temporary path, which is unlikely to have a config
+    directory in its hierarchy.
     """
     # Change the default configuration folder path to temp folder instead of probably `~`.
-    monkeypatch.setattr(settings, 'DEFAULT_AIIDA_PATH', tmp_path)
+    monkeypatch.setattr(settings, 'DEFAULT_AIIDA_PATH', chdir_tmp_path)
 
     # Make sure that the environment variable is not set
     try:
@@ -59,7 +63,7 @@ def test_environment_variable_not_set(tmp_path, monkeypatch):
         pass
     settings.set_configuration_directory()
 
-    config_folder = os.path.join(tmp_path, settings.DEFAULT_CONFIG_DIR_NAME)
+    config_folder = chdir_tmp_path / settings.DEFAULT_CONFIG_DIR_NAME
     assert os.path.isdir(config_folder)
     assert settings.AIIDA_CONFIG_FOLDER == pathlib.Path(config_folder)
 
@@ -137,6 +141,45 @@ def test_environment_variable_set_multiple_path(tmp_path):
     assert settings.AIIDA_CONFIG_FOLDER == config_folder
 
 
+@pytest.mark.filterwarnings('ignore:Creating AiiDA configuration folder')
+@pytest.mark.usefixtures('cache_aiida_path_variable')
+@pytest.mark.parametrize('environment_variable', (True, False))
+def test_cwd(environment_variable, chdir_tmp_path):
+    """Test that the current working directory is checked as long as ``AIIDA_PATH`` environment variable is not set."""
+    if environment_variable:
+        dirpath_env = chdir_tmp_path / 'env' / settings.DEFAULT_CONFIG_DIR_NAME
+        dirpath_env.mkdir(parents=True)
+        os.environ[settings.DEFAULT_AIIDA_PATH_VARIABLE] = str(dirpath_env.absolute())
+    else:
+        os.environ.pop(settings.DEFAULT_AIIDA_PATH_VARIABLE, None)
+
+    dirpath_cwd = chdir_tmp_path / settings.DEFAULT_CONFIG_DIR_NAME
+    dirpath_cwd.mkdir()
+
+    settings.set_configuration_directory()
+    if environment_variable:
+        assert settings.AIIDA_CONFIG_FOLDER == dirpath_env
+    else:
+        assert settings.AIIDA_CONFIG_FOLDER == dirpath_cwd
+
+
+@pytest.mark.filterwarnings('ignore:Creating AiiDA configuration folder')
+@pytest.mark.usefixtures('cache_aiida_path_variable')
+def test_cwd_parent(chdir_tmp_path):
+    """Test that if ``AIIDA_PATH`` is not set, the current working directory is checked, moving up all parents."""
+    os.environ.pop(settings.DEFAULT_AIIDA_PATH_VARIABLE, None)
+
+    dirpath = chdir_tmp_path / settings.DEFAULT_CONFIG_DIR_NAME
+    dirpath.mkdir()
+    subdirpath = dirpath / 'subdirectory'
+    subdirpath.mkdir()
+    os.chdir(subdirpath)
+    assert pathlib.Path.cwd() == subdirpath
+
+    settings.set_configuration_directory()
+    assert settings.AIIDA_CONFIG_FOLDER == dirpath
+
+
 def compare_config_in_memory_and_on_disk(config, filepath):
     """Verify that the contents of `config` are identical to the contents of the file with path `filepath`.