Add verdi group delete --delete-nodes
#4578
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## develop #4578 +/- ##
===========================================
- Coverage 79.45% 79.44% -0.00%
===========================================
Files 483 484 +1
Lines 35487 35540 +53
===========================================
+ Hits 28191 28232 +41
- Misses 7296 7308 +12
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
|
CLI: $ verdi group delete --help
Usage: verdi group delete [OPTIONS] GROUP
Delete a group and (optionally) the nodes it contains.
Options:
-f, --force Do not ask for confirmation.
--delete-nodes Delete all nodes in the group along with the
group itself.
--call-calc-forward / --no-call-calc-forward
Whether to expand the node set by following
CALL links to calculations forwards.
[default: True]
--call-work-forward / --no-call-work-forward
Whether to expand the node set by following
CALL links to workflows forwards. [default:
True]
--create-forward / --no-create-forward
Whether to expand the node set by following
CREATE links forwards. [default: True]
-n, --dry-run Perform a dry run.
-v, --verbose Be more verbose in printing output.
-c, --clear Remove all nodes before deleting the group
itself. [deprecated: No longer has any
effect. Will be removed in 2.0.0]
-h, --help Show this message and exit.$ verdi group delete 4 --delete-nodes
Are you sure to delete Group<abcd>? [y/N]: y
100 Node(s) marked for deletion
Warning: YOU ARE ABOUT TO DELETE 100 NODES! THIS CANNOT BE UNDONE!
Shall I continue? [y/N]: y
Starting node deletion...
Nodes deleted from database, deleting files from the repository now...
Deletion of nodes completed.
Success: Group<abcd> deleted.$ verdi group delete 6 --delete-nodes --force
100 Node(s) marked for deletion
Starting node deletion...
Nodes deleted from database, deleting files from the repository now...
Deletion of nodes completed.
Success: Group<abcd> deleted.$ verdi group delete 5 --delete-nodes --dry-run
Info: Would have deleted Group<abcd>.
100 Node(s) marked for deletion
This was a dry run, exiting without deleting anything$ verdi group delete 5 --delete-nodes --dry-run -v
Info: Would have deleted Group<abcd>.
100 Node(s) marked for deletion
Node(s) to delete:
46736bac-d196-4b2e-b9ce-114b3d4ad718 1001 Data
fd1e087f-99c8-4fd7-8403-27f99ceaaa78 1002 Data
deae9279-da5c-4a41-92d7-f127fa4ca5fc 1003 Data
a968ad32-dad3-4434-bb03-5440604164f9 1004 Data
c8176306-1f0c-483e-b56b-2edc2a58fa70 1005 Data
32804abd-05ec-4653-9c6b-b0f843418bcf 1006 Data
1dbfee0c-5a1f-4edb-a9aa-f782bebf356c 1007 Data
9febd0e7-afe7-4842-802a-8c66900364e3 1008 Data
0e173df5-55a7-48ef-ab43-048fc6b79a24 1009 Data
d21a0782-fce1-4b0e-96bf-54818ed2be5d 1010 Data
d0bc9127-c466-4d1e-9935-5fea82848df1 1011 Data
00524cbf-dcea-44fa-978f-b1e42a5e4ff0 1012 Data
673ba9a1-6ee3-439c-b56f-db464220bb43 1013 Data
51b4c742-6cc2-441c-9921-35026564bf9e 1014 Data
de6ab329-4469-4267-901b-e7767b1ec26b 1015 Data
f8824b49-8188-4c60-b4b4-fab8d6b11f01 1016 Data
b52e595c-616a-4577-a32f-60eced44fc0c 1017 Data
b46e789d-7bbc-44d7-8153-74aa5f0d5c41 1018 Data
8a0756e3-8a7f-45b5-8339-73c4e0ff97cc 1019 Data
f7215169-c178-4f11-9279-c29dbbdbd516 1020 Data
9941a9cf-d24d-487f-9f6b-01dde6617cb1 1021 Data
b69d1428-b80f-49fe-a337-e17cc44d0d33 1022 Data
769ef923-edee-43cf-a6d3-05709d9ee0fb 1023 Data
4fa132d2-aee5-485c-bab1-3457aa03af1b 1024 Data
21186e6c-1d9c-4ce7-9d26-6154cd18858d 1025 Data
094c4e82-340b-40fd-81d3-61275f667c93 1026 Data
fe60f640-675d-426c-80d9-4bdab48f5751 1027 Data
348fb868-70bc-4cc3-a56c-777711025770 1028 Data
4cf3a3c6-bb74-4df2-961b-4afa0b692a68 1029 Data
f2e4d8c2-eeda-47d9-bfbf-ee48474f5bf7 1030 Data
15756ee6-7116-4043-a376-caa2e21e6065 1031 Data
bd96385e-be87-4847-9b4c-33caaa0b6930 1032 Data
41af5642-a3f1-47e5-8378-166ff6344e71 1033 Data
3dea1455-3b6d-4dd2-8439-31bbd5642707 1034 Data
5675d4f0-ec86-48c9-b224-047877cd88f5 1035 Data
651034aa-03d0-468f-a428-0fd608d86825 1036 Data
61147dd4-c713-466c-9953-9659bb7a16a3 1037 Data
7bbc0ee8-c3bf-49bd-afa2-7b174912d47f 1038 Data
f6c156a1-9fd6-4626-80f7-dc259101097b 1039 Data
d9c2a1fe-fd19-4952-8980-71a87b6ef18e 1040 Data
4dae48d8-1667-4079-a2a9-17ee139edbf0 1041 Data
a2804cf4-245e-4f03-9e3c-ce0927c336dd 1042 Data
ab0079e3-b33d-40dc-90cd-318be855eff7 1043 Data
f65981a9-3d63-4e5d-b931-45b4690489c6 1044 Data
a7bb0735-7322-4683-8299-7c21a4d54c9b 1045 Data
83b21285-3068-41bf-8240-033343b80f7c 1046 Data
cae093a7-2592-4e68-8125-161b75e811af 1047 Data
fb16e01d-9db5-4670-920b-08b7569067e6 1048 Data
b337fc9a-f999-4181-b93b-3b83d2c88c36 1049 Data
4086a3fb-e1d6-4bc0-9fdc-625c35515a28 1050 Data
abc31cce-596c-4741-8012-558bdb4dcdb4 1051 Data
7794146b-ad19-4f75-a39f-25e454046d1d 1052 Data
24ab376c-e4bb-4306-bdb5-7abae086d6bb 1053 Data
9240af8f-82d7-4f81-b6ef-d2fcf43ed519 1054 Data
2f656aea-9c01-4ac2-a686-fe09671b7b7d 1055 Data
52f5537c-6cca-49b1-a11e-700cb6a12801 1056 Data
c2523b5b-1718-45a5-8c30-d44b713921aa 1057 Data
96ee0470-ce0f-4381-9558-3729b497fe9f 1058 Data
ee395b14-f1f3-44e5-bbe8-328c4b90cdeb 1059 Data
29506ecf-70e2-4a7b-b717-b4438315d5ea 1060 Data
7595b38c-54d7-49f4-af18-9ce5c4ea1ef2 1061 Data
932e0c0f-aa7b-4430-a5f5-350bdfbf4192 1062 Data
2007347a-0fad-4568-a100-e3e278b54190 1063 Data
dc4a8e3e-bdfb-4728-b9c7-fedeb0c44908 1064 Data
e7e6cd4c-bc34-424e-ae47-d543f98036db 1065 Data
c550550d-4883-431f-9c18-9869eb73a4b7 1066 Data
7228c8ef-62bc-4d2c-8a54-d8f55659d299 1067 Data
5654888a-aaf7-4d3b-b6c1-7d64588d2df9 1068 Data
b4f4a806-9275-4f89-a893-9393ece05129 1069 Data
4b9bbb43-396c-4394-a357-c6cbdfe5a8d5 1070 Data
90fc6871-bc5c-4b75-8fe3-f825682c4a43 1071 Data
410cc91c-2310-4ecb-8bef-faa28d5a9d68 1072 Data
7fc21854-63c3-45b1-9950-44e35c3bfcc7 1073 Data
cebe23e3-3321-4862-b1b9-cc9771d5e81b 1074 Data
4b1e3c88-cab8-455d-8d69-7731e9744a85 1075 Data
8a6d7a5a-074d-47ea-bc6b-50112e37e758 1076 Data
bb8d61d4-3169-4635-bb1d-9358a013ef67 1077 Data
63498c20-7567-4a44-9b9a-c0354b69635e 1078 Data
9fe8ff39-fb46-4634-b5fc-01eb01f48654 1079 Data
6f8267a7-15ee-4ea5-bd6b-42e5ca58db79 1080 Data
caf49935-b6ac-4314-8a95-2133e9e8a8b4 1081 Data
f68d110e-2abd-4266-9a66-06207aa4f098 1082 Data
f6a22d66-98d1-4529-8688-28c7e106415f 1083 Data
3c49e535-85f9-4c55-b4d9-75ddb4ef5192 1084 Data
c75be649-4ad0-4581-9f23-e5f90da2e919 1085 Data
f921d381-80ca-459f-bbc7-45b686e367e8 1086 Data
c28d5f61-562f-4b03-a142-f656909cb3d0 1087 Data
6bce9205-c003-491f-9a3b-d0ea4176e049 1088 Data
14cef50f-9bb5-4e19-bc65-c3140eb17b1b 1089 Data
261151fd-87cc-4acf-8a45-57c10e284604 1090 Data
e599b65c-3704-4a08-8a0e-17be074a0583 1091 Data
c3169bd2-21cb-4fa2-8679-01d5d7bfc34b 1092 Data
c160b5c5-23f2-4d50-90f2-a9bb8f5d2e8b 1093 Data
9aae4eeb-444d-4399-8113-5f8165f088c7 1094 Data
562619df-21be-41b1-af0f-5dcdd6482a56 1095 Data
8fd8fb96-789e-452b-ae6d-738e03bad2cc 1096 Data
a4643674-7981-4e88-a1d0-062cd101e574 1097 Data
28ef5fb2-bc8f-446b-9a7c-4f61cb5d309c 1098 Data
9a47601f-90bd-4883-a0e4-57e41de4a9ed 1099 Data
658400f6-9c11-454e-8989-6d787cbc91f6 1100 Data
This was a dry run, exiting without deleting anything |
|
Performance: $ time verdi group delete 7 --delete-nodes --force
100 Node(s) marked for deletion
Starting node deletion...
Nodes deleted from database, deleting files from the repository now...
Deletion of nodes completed.
Success: Group<abcd> deleted.
real 0m4.619s
user 0m2.862s
sys 0m0.629s |
|
my other thought on this is perhaps there should be a separate |
😭 I'll close that one then. 😉
I have definitely already encountered this use case (I tend to use a lot of |
3 tasks
|
@mbercx can you give a once over then I'll merge |
|
@mbercx I will be merging this by tomorrow, so speak now or forever hold your peace 😉 |
Sorry, got distracted by other work! I'll have a look at it later today, pinky swear! 😅 |
mbercx
previously approved these changes
Jan 7, 2021
There was a problem hiding this comment.
Great stuff @chrisjsewell! Very holistic 👌
I left a few small comments/questions, feel free to merge at your discretion.
aiida/cmdline/commands/cmd_code.py
Outdated
| echo.echo_warning(f'YOU ARE ABOUT TO DELETE {len(pks)} NODES! THIS CANNOT BE UNDONE!') | ||
| return not click.confirm('Shall I continue?') | ||
|
|
||
| DELETE_LOGGER.setLevel(verbosity) |
There was a problem hiding this comment.
Didn't you already set the logger verbosity on line 200?
| from aiida.cmdline.utils import echo | ||
| """Functions to delete nodes from the database, preserving provenance integrity.""" | ||
| from typing import Callable, Iterable, Optional, Set, Tuple, Union | ||
| import warnings | ||
|
|
||
|
|
||
| def delete_nodes( |
There was a problem hiding this comment.
Do we need to deprecate this function if it wasn't part of the public API?
There was a problem hiding this comment.
well no, but I know that I've certainly used it before (since there's no other way with the API to delete nodes), so better safe than sorry
| pks_to_be_deleted = delete_nodes( | ||
| [1, 2, 3], dry_run=True, create_forward=True, call_calc_forward=True, call_work_forward=True | ||
| ) | ||
|
|
There was a problem hiding this comment.
Did you mean to add this code block after line 773? That would explain the colon there. 😅
There was a problem hiding this comment.
oh yeh I added it there first, but then decided to move it here
|
|
||
| __all__ = (calculations.__all__ + data.array.kpoints.__all__ + data.structure.__all__ + dbimporters.__all__) | ||
| __all__ = ( |
There was a problem hiding this comment.
Be sure to also update the public API in the Reference section:
https://aiida-core.readthedocs.io/en/stable/reference/api/public.html
There was a problem hiding this comment.
👍 (I've got plans anyhow to automate the creation of this: #4558 (comment))
| inputs, and so on. | ||
|
|
||
| .. deprecated:: 1.6.0 | ||
| The `verbosity` keyword will be removed in `v2.0.0`, set the level of `DELETE_LOGGER` instead. |
There was a problem hiding this comment.
If the delete_nodes and delete_group_nodes are part of the public API, perhaps it would be more user friendly to keep the verbosity argument instead of asking them to use the DELETE_LOGGER?
There was a problem hiding this comment.
no, no, no we should be / are moving to the use of loggers for all logging across the whole code base. Plenty of reasons why, see e.g. https://stackoverflow.com/a/6918596/5033292, essentially it is an inversion of control whereby it should not be the function that decides how logging is implemented
chrisjsewell
commented
Jan 7, 2021
Co-authored-by: Marnik Bercx <mbercx@gmail.com>
|
thanks @mbercx all sorted! |
chrisjsewell
changed the title
verdi group delete --delete-nodes
chrisjsewell
added a commit
that referenced
this pull request
Mar 18, 2021
* Dependencies: bump cryptography to 3.2 in `requirements` (#4520)
Bumps `cryptography` from 2.8 to 3.2.
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Sebastiaan Huber <mail@sphuber.net>
* CI: remove `run-on-comment` job in benchmark workflow (#4569)
This job is failing due to this change:
https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/
It's not really used, so lets just remove it
* Docs: update citations with AiiDA workflows paper (#4568)
Citation for the latest paper on the engine is added to the README and
the documentation index page. The paper in `aiida/__init__.py` is also
updated which was still referencing the original publication of 2016.
* Enforce verdi quicksetup --non-interactive (#4573)
When in non-interactive mode, do not ask whether to use existing user/database
* `SinglefileData`: add support for `pathlib.Path` for `file` argument (#3614)
* DOCS: Reverse daemon start and profile setup sections in intro. (#4574)
The profile must be setup prior to starting the daemons to avoid an error.
* Fix `verdi --version` in editable mode (#4576)
This commit fixes a bug,
whereby click was using a version statically stored on install of the package.
This meant changes to `__version__` were not dynamically reflected.
* Improve `verdi node delete` performance (#4575)
The `verdi node delete` process fully loaded all ORM objects at multiple stages
during the process, which is highly inefficient.
This commit ensures the process now only loads the PKs when possible.
As an example, the time to delete 100 "empty" nodes (no attributes/objects)
is now reduced from ~32 seconds to ~5 seconds.
* `CalcJob`: add the `additional_retrieve_list` metadata option (#4437)
This new option allows one to specify additional files to be retrieved
on a per-instance basis, in addition to the files that are already
defined by the plugin to be retrieved. This was often implemented by
plugin packages itself through a `settings` node that supported a key
that would allow a user to specify these additional files.
Since this is a common use case, we implement this functionality on
`aiida-core` instead to guarantee a consistent interface across plugins.
* Add options for transport tasks (#4583)
* Add options for transport tasks
When encountering failures during the execution of transport tasks, a runner
will wait for a time interval between transport task attempts. This time
interval between attempts is increased using an exponential backoff
mechanism, i.e. the time interval is equal to:
(TRANSPORT_TASK_RETRY_INITIAL_INTERVAL) * 2 ** (N_ATTEMPT - 1)
where N_ATTEMPT is the number of failed attempts. This mechanism is
interrupted once the TRANSPORT_TASK_MAXIMUM_ATTEMPTS is reached.
The initial interval and maximum attempts are currently fixed to 20
seconds and 5, respectively. This commit adds two configuration options
that use these defaults, but allow the user to adjust them using `verdi
config`.
* Fix command for getting EBM config options (#4587)
Currently the transport options for the EBM are obtained by using the
get_config function, e.g.:
`initial_interval = get_config_option(RETRY_INTERVAL_OPTION)`
However, it seems that `get_config()` does not get you the current
configuration (see #4586).
Replacing `get_config().get_option()` with `get_config_option()` fixes this
issue for the EBM options.
* CI: revert apt source list removal
This work around was added some time ago because this source for the
`apt` package manager was causing the install of system dependencies to
fail.
* CI: Add workflow to run tests against various RabbitMQ versions
The main test workflow runs against a single version of RabbitMQ but
experience has shown that the code can break for different versions of
the RabbitMQ server. Here we add a new CI workflow that runs various
unit tests through pytest that simulate the typical interaction with the
RabbitMQ server in normal AiiDA operation. The difference is that these
are tested against the currently available versions of RabbitMQ.
The current setup, still only tests part of the functionality that AiiDA
uses, for example, the default credentials and virtual host are used.
Connections over TLS are also not tested. These options would require
the RabbitMQ service that is running in a docker container to be
configured differently. It is not clear how these various options can be
parametrized in concert with the actual unit tests.
* Engine: replace `tornado` with `asyncio`
The `plumpy` and `kiwipy` dependencies have already been migrated from
using `tornado` to the Python built-in module `asyncio` in the versions
`0.16.0` and `0.6.0`, respectively. This allows us to also rid AiiDA of
the `tornado` dependency, which has been giving requirement clashes with
other tools, specifically from the Jupyter and iPython world. The final
limitation was the `circus` library that is used to daemonize the daemon
workers, which as of `v0.17.1` also supports `tornado~=5`.
A summary of the changes:
* Replace `tornado.ioloop` with `asyncio` event loop.
* Coroutines are marked with `async` instead of decorated with the
`tornado.gen.coroutine` decorator.
* Replace `yield` with `await` when calling a coroutine.
* Replace `raise tornado.gen.Return` with `return` when returning from
a coroutine.
* Replace `add_callback` call on event loop with `call_soon` when
scheduling a callback.
* Replace `add_callback` call on event loop with `create_task` when
scheduling `process.step_until_terminated()`.
* Replace `run_sync` call on event loop with `run_until_complete`.
* Replace `pika` uses with `aio-pika` which is now used by the `plumpy`
and `kiwipy` libraries.
* Replace `concurrent.Future` with `asyncio.Future`.
* Replace `yield tornado.gen.sleep` with `await asyncio.sleep`.
Additional changes:
* Remove the `tornado` logger from the logging configuration.
* Remove the `logging.tornado_loglevel` configuration option.
* Turn the `TransportQueue.loop` attribute from method into property.
* Call `Communicator.close()` instead of `Communicator.stop()` in the
`Manager.close()` method. The `stop` method has been deprecated in
`kiwipy==0.6.0`.
* `Process.kill`: properly resolve the killing futures
The result returned by `ProcessController.kill_process` that is called
in `Process.kill` for each of its children, if it has any, can itself be
a future, since the killing cannot always be performed directly, but
instead will be scheduled in the event loop. To resolve the future of
the main process, it will have to wait for the futures of all its
children to be resolved as well. Therefore an intermediate future needs
to be added that will be done once all child futures are resolved.
* Unwrap the futures returned by `ProcessController` in `verdi process`
The commands of `verdi process` that perform an RPC on a live process
will do so through the `ProcessController`, which returns a future.
Currently, the process controller uses the `LoopCommunicator` as its
communicator which adds an additional layer of wrapping. Ideally, the
return type of the communicator should not change depending on the
specific implementation that is used, however, for now that is the case
and so the future needs to be unwrapped explicitly one additional time.
Once the `LoopCommunicator` is fixed to return the same future type as
the base `Communicator` class, this workaround can and should be
removed.
* `Runner`: use global event loop and global runner for process functions
With the migration to `asyncio`, there is now only a single event loop
that is made reentrant through the `nest-asyncio` library, that monkey
patches `asyncio`'s built-in mechanism to prevent this. This means that
in the `Runner` constructor, we should simply get the global event loop
instead of creating a new one, if no explicit loop is passed into the
constructor. This also implies that the runner should never take charge
in closing the loop, because it no longer owns the global loop.
In addition, process functions now simply use the global runner instead
of creating a new runner. This used to be necessary because running in
the same runner, would mean running in the same loop and so the child
process would block the parent. However, with the new design on
`asyncio`, everything runs in a single reentrant loop and so child
processes no longer need to spawn their own independent nested runner.
* Engine: cancel active tasks when a daemon runner is shutdown
When a daemon runner is started, the `SIGINT` and `SIGTERM` signals are
captured to shutdown the runner before exiting the interpreter. However,
the async tasks associated with the interpreter should be properly
canceled first.
* Engine: enable `plumpy`'s reentrant event loop policy
The event loop implementation of `asyncio` does not allow to make the
event loop to be reentrant, which essentially means that event loops
cannot be nested. One event loop cannot be run within another event
loop. However, this concept is crucial for `plumpy`'s design to work and
was perfectly allowed by the previous event loop provider `tornado`.
To work around this, `plumpy` uses the library `nest_asyncio` to patch
the `asyncio` event loop and make it reentrant. The trick is that this
should be applied at the correct time. Here we update the `Runner` to
enable `plumpy`'s event loop policy, which will patch the default event
loop policy. This location is chosen since any process in `aiida-core`
*has* to be run by a `Runner` and only one runner instance will ever be
created in a Python interpreter. When the runner shuts down, the event
policy is reset to undo the patch.
* Tests: do not create or destroy event loop in test setup/teardown
* Engine: explicitly enable compatibility for RabbitMQ 3.5
RabbitMQ 3.6 changed the way integer values are interpreted for
connection parameters. This would cause certain integer values that used
to be perfectly acceptable, to all of suddent cause the declaration of
resources, such as channels and queues, to fail.
The library `pamqp`, that is used by `aiormq`, which in turn is used
ultimately by `kiwipy` to communicate with the RabbitMQ server, adapted
to these changes, but this would break code with RabbitMQ 3.5 that used
to work just fine. For example, the message TTL when declaring a queue
would now fail when `32767 < TTL < 655636` due to incorrect
interpretation of the integer type.
The library `pamqp` provides a way to enable compatibility with these
older versions. One should merely call the method:
pamqp.encode.support_deprecated_rabbitmq()
This will enable the legacy integer conversion table and will restore
functionality for RabbitMQ 3.5.
* Dependencies: update minimum version for `notebook>=6.1.5` (#4593)
Lower versions suffer from vulnerability `GHSA-c7vm-f5p4-8fqh`.
Also update the requirement files to only use explicit pinned versions.
The compatibility operator was erroneously used for the `aio-pika`,
`pamqp` and `pytest-asyncio` dependencies.
For `pamqp` the minimum required version is upped to `2.3` since that
was the version that introduced the `support_deprecated_rabbitmq`
function that is required from that library.
* Daemon: replace deprecated classmethods of `asyncio.Task` in shutdown (#4608)
The `shutdown` function, that was attached to the loop of the daemon
runner in `aiida.engine.daemon.runner.start_daemon`, was calling the
classmethods `current_task` and `all_tasks` of `asyncio.Task` which have
been deprecated in Python 3.7 and are removed in Python 3.9. This would
prevent the daemon runners from being shutdown in Python 3.9. The
methods have been replaced with top level functions that can be imported
directl from `asyncio`.
This was not noticed in the tests because in the tests the daemon is
stopped but it is not checked whether this happens successfully. Anyway,
the error would only show up in the daemon log. To test the shutdown
method, it has been made into a standalone coroutine and renamed to
`shutdown_runner`.
Since the `shutdown_runner` is a coroutine, the unit test that calls it
also has to be one and therefore we need `pytest-asyncio` as a
dependency. The `event_loop` fixture, that is provided by this library,
is overrided such that it provides the event loop of the `Manager`,
since in AiiDA only ever this single reentrant loop should be used.
Note that the current CI tests run against Python 3.6 and Python 3.9 and
so will still not catch this problem, however, the `test-install`
workflow _does_ run against Python 3.9. I have opted not to change the
continuous integrations to run against Python 3.9 instead of 3.8, since
they take more than twice the time. Supposedly this is because certain
dependencies have to be built and compiled from scratch when the
testing environment is started.
* CLI: add the `verdi database version` command (#4613)
This shows the schema generation and version of the database of the
given profile, useful mostly for developers when debugging.
In addition to the new command, the code in `aiida.manage.manager` had
to be updated for the new functionality to work. The `get_backend_manager`
was so far _not_ loading the backend, although that really doesn't make
any sense. It is providing access to data from the database, but to do
so the backend should be loaded, otherwise a connection isn't possible.
This problem went unnoticed, because the `BackendManager` was so far only
used in `aiida.engine.utils.set_process_state_change_timestamp`. By the
time this gets used, the database backend will already have been loaded
through another code path.
For the change `verdi database version` command, however, the call to
get the backend manager needed to make sure that the database backend
itself was also loaded. It was not possible to have `get_backend_manager`
simply call `_load_backend()` because this would lead to infinite
recursion as `_load_backend()` also calls `get_backend_manager`.
Therefore `_load_backend` is refactored to not call the former but rather
to directly fetch it through `aiida.backends`.
* Add the `TransferCalcJob` plugin (#4194)
This calcjob allows the user to copy files between a remote machine and
the local machine running AiiDA. More specifically, it can do any of the
following:
* Take any number of files from any number of `RemoteData` folders in
a remote machine and copy them in the local repository of a single
newly created `FolderData` node.
* Take any number of files from any number of `FolderData` nodes in the
local machine and copy them in a single newly created `RemoteData` folder
in a given remote machine.
These are the main two use cases, but there are also other more complex
combinations allowed by the current implementation.
Co-authored-by: Sebastiaan Huber <mail@sphuber.net>
* Dependencies: update requirement `kiwipy~=0.7.1` and `plumpy~=0.18.0` (#4629)
A breaking change was released with `kiwipy==0.5.4` where the default
value for the task message TTL was changed. This caused connections to
existing RabbitMQ queues to fail. Since process task queues are
permanent in AiiDA, this would break all existing installations.
This problem was fixed by reverting the change which was released with
`kiwipy==0.5.5`, however, this was a support patch at the time and the
revert never made it into the main line, leaving all versions up from
`v0.6.0` still affected. Since these versions of `kiwipy` were never
required by a released version of `aiida-core`, but only the current
`develop`, which will become `v1.6.0`, we can simply update the
requirement to the latest patch `kiwipy==0.7.1` that addressed the
problem.
The dependency requirement for `plumpy` also had to be updated because
the old pinned minor version was pinned to `kiwipy~=0.6.0` which is not
compatible with our new requirements.
* Docs: add content from old documentation on caching/hashing (#4546)
Move the content of "Controlling hashing" and "Design guidelines" inside
of `developer_guide/core/caching.rst` to `topics/provenance/caching`.
* Engine: remote `with_persistence=False` from process function runner (#4633)
In principle the runner for a process function does not need a persister
since it runs in one go and does not have intermediate steps at which
the progress needs to be persisted. However, since the process function
implementation calls `Manager.get_runner`, if a runner has not yet been
created in the interpreter, one will be created and set to be the global
one. This is where the problem occurs because the process function
specifies `with_persistence=False` for the runner. This will cause any
subsequent process submissions to fail since the `submit` function will
call `runner.persister.save_checkpoint` which will fail since the
`persister` of the runner is `None`.
* `CalcJob`: improve testing and documentation of `retrieve_list` (#4611)
The documentation on the `retrieve_list` syntax and its functioning was
incorrect. The inaccuracies are corrected and extensive examples are
provided that give an example file hierarchy for the remote working
directory and then for a variety of definitions of the `retrieve_list`
the resulting file structure in the retrieved folder is depicted.
* CI: remote the `numpy` install workaround for `pymatgen`
The problem occurred due to an outdated version of `setuptools` which
would be invoked when `pymatgen` gets installed from a tarball, in which
case the wheel has to be built. In this scenario, the build requirements
get installed by `setuptools`, which at outdated versions did not
respect the Python requirements of the dependencies which would cause
incompatible version of `numpy` to be installed, calling the build to
fail. By updating `setuptools` the workaround of manually installing a
compatible `numpy` version beforehand is no longer necessary.
* CI: skip `restapi.test_threaded_restapi:test_run_without_close_session`
This test has been consistently failing on Python 3.8 and 3.9 despite
the two reruns using flaky. For now we skip it entirely instead.
* Dependencies: update requirement `plumpy~=0.18.1` (#4642)
This patch release of `plumpy` fixes a critical bug that makes the new
`asyncio` based implementation of the engine compatible with Jupyter
notebooks.
* CLI: ensure `verdi database version` works even if schema outdated (#4641)
The command was failing if the database schema was out of sync because
the backend was loaded, through `get_manager`, with the default schema
check on. Since the database does not actually have to be used, other
than to retrieve the current schema version and generation, we can load
the backend without the check.
* Add `verdi group delete --delete-nodes` (#4578)
This commit makes a number of improvements to the deletion of nodes API/CLI:
1. Makes `delete_nodes` usable outside of `click`;
adding a callback for the confirmation step,
rather than calling `click.confirm` directly,
and using logging instead of `click.echo`
2. Moves the function from `aiida/manage/database/delete/nodes.py`
to `aiida/tools/graph/deletions.py`,
leaving a deprecation warning at the old location.
This is a more intuitive place since the function is directly build on the graph traversal functionality.
3. Exposes API functions *via* `from aiida.tools import delete_nodes` and adds their use to the documentation.
4. Adds `delete_group_nodes` mainly as a wrapper around `delete_nodes`;
querying for all the node pks in the groups, then passing these to `delete_nodes`
5. Adds the ability to delete nodes to `verdi group delete --delete-nodes`,
with the same flags and logic as `verdi node delete`
6. Fixes a bug in `verdi node delete`, introduced by #4575, if a node does not exist
* 🧪 FIX: engine benchmark tests (#4652)
The `test_workchain_daemon` test group required updating to using asyncio (rather than tornado)
* Docs: Minor documentation fixes (#4643)
Small changes and fixes in the documentation.
* Docs: clarify docstrings of `get_last_job_info` and `get_detailed_job_info` (#4657)
`CalcJobNode`s contain two differente job infos, the `detailed_job_info` and
the `last_job_info`. The distinction between the two was not obvious,
and not documented. The docstrings are improved to clarify the difference.
* docs: simplify proxycommand (#4662)
The 'netcat mode' `-W` was added in OpenSSH 5.4, released March 2010.
Given that this simplifies the setup and and delegates handling of netcat
to ssh, this is what we should recommend.
For example, MacOS ships with OpenSSH 5.6 since MacOS 10.7, released October 2010.
* Docs: Add redirect for database backup page (#4675)
* Type checking: `aiida/engine` (+bug fixes) (#4669)
Added type checking for the modules
* `aiida.engine`
* `aiida.manage.manager`
Move `aiida.orm` imports to top of file in `aiida.engine` module. This should be
fine as `aiida.orm` should not import anything from `aiida.engine` and this way
we don't need import guards specifically for type checking.
* Fix `run_get_node`/`run_get_pk` namedtuples (#4677)
Fix a regression made in #4669, whereby the namedtuple's were incorrectly named
* REST API fixes
- Use node_type in construct_full_type().
- Don't use try/except for determining full_type.
- Remove unnecessary try/except in App for catch_internal_server.
- Use proper API_CONFIG for configure_api.
* New /querybuilder-endpoint with POST for REST API
The POST endpoint returns what the QueryBuilder would return, when
providing it with a proper queryhelp dictionary.
Furthermore, it returns the entities/results in the "standard" REST API
format - with the exception of `link_type` and `link_label` keys for
links. However, these particular keys are still present as `type` and
`label`, respectively.
The special Node property `full_type` will be removed from any entity,
if its value is `None`. There are two cases where this will be True:
- If the entity is not a `Node`; and
- If neither `node_type` or `process_type` are among the projected
properties for any given `Node`.
Concerning security:
The /querybuilder-endpoint can be toggled on/off with the configuration
parameter `CLI_DEFAULTS['POSTING']`.
Added this to `verdi restapi` as `--posting/--no-posting` option.
The option is hidden by default, as the naming may be changed in the
future.
Reviewed by @ltalirz.
* Use importlib in .ci folder
* Fix: pre-store hash for -0. and 0. is now the same
* ci: update paramiko version (#4686)
Now that the Github Action runners switched to Ubuntu 20.04, the default SSH
key format of OpenSSH changed and is no longer supported by paramiko
<=2.7.1.
* Fix: release signal handlers after run execution (#4682)
After a process has executed (when running rather than submitting),
return the signal handlers to their original state.
This fixes an issue whereby using `CTRL-C` after a process has run still calls the `process.kill`.
It also releases the `kill_process` function's reference to the process,
a step towards allowing the finished process to be garbage collected.
* Fix: `PluginVersionProvider` should cache process class (#4683)
Currently, the `PluginVersionProvider` is caching process instance, rather than class.
This commit fixes the bug, meaning the cache will now work correctly.
Removing the reference to the process instance also is a step towards allowing it to be garbage collected.
* remove leftover use of Computer.name (#4681)
Remove leftover use of deprecated Computer.name attribute in `verdi
computer list`.
Also update minimum version of click dependency to 7.1, since click 7.1
introduces additional whitespace in the verdi autodocs (running with
click 7.0 locally resulted in pre-commit check failing on CI).
Co-authored-by: Chris Sewell <chrisj_sewell@hotmail.com>
* Add `to_aiida_type` to the public API (#4672)
Since `to_aiida_type` is intended for public use,
this commit makes it part of the public API,
via `from aiida.orm import to_aiida_type`.
* Add .dockerignore (#4564)
This commit adds a `.dockerignore` file to inhibit any unecessary/unwanted files being copied into the Docker container,
during the `COPY . aiida-core` command,
and also reduces the build time.
* CI: Remove `--use-feature=2020-resolver` pip feature flag tests. (#4689)
The feature is now on by default in the latest stable release.
* CI: Notify slack on failure of the test-install workflow. (#4690)
* Improve namedtuples in aiida/engine (#4688)
This commit replaces old-style namedtuples with `typing.NamedTuple` sub-classes.
This allows for typing of fields and better default value assignment.
Note this feature requires python>=3.6.1,
but it is anyhow intended that python 3.6 be dropped for the next release.
* test AiiDA ipython magics and remove copy-paste in docs (#4548)
Adds tests for the AiiDA IPython extension.
Also:
* move some additional lines from the registration snippet to
aiida-core (where we can adapt it if the IPython API ever changes)
* rename and deprecate misnomer `load_ipython_extension` to
`register_ipython_extension` (to be removed in aiida 3)
* include the snippet to register the AiiDA ipython magics from the
aiida-core codebase instead of the (already outdated) copy-pasted
version.
* revisit the corresponding section of the documentation, starting
with the setup, and removing some generic information about jupyter.
* 🐛 FIX: typing failure (#4700)
As of numpy v1.20, `numpy.inf` is no longer recognised as an integer type
* 📚 DOCS: fix typo (#4711)
* BUILD: drop support for python 3.6 (#4701)
Following our support table, we drop python 3.6 support.
* BUILD: bump jenkins dockerimage to 20.04 (#4714)
Despite python3.7 being installed on the Jenkins dockerimage, pip
install failed after dropping python 3.6 support (likely because pip
from python 3.6 was being used).
We update ubuntu to 20.04, which comes with python 3.8.2 by default.
* Switch matrix order in continuous-integration tests job. (#4713)
To harmonize with test-install workflow.
* ♻️ REFACTOR: verdi export/import -> verdi archive (#4710)
This commit deprecates `verdi export` and `verdi import` and combines them into `verdi archive`.
* Dependencies: Require `ipython~=7.20` (#4715)
* Dependencies: Require `ipython~=7.20`
Package jedi version 0.18 introduces backwards incompatible changes that
break compatibility with ipython<7.20.
Fixes issue #4668.
* Automated update of requirements/ files. (#4716)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* ♻️ REFACTOR: `ci/` folder (#4565)
This commit looks to address two issues:
1. The `ci/` folder has become cluttered;
it contains configuration and scripts for both the GitHub Actions and Jenkins CI
and it is not easily clear which is for which.
2. The Jenkins tests are somewhat of a black-box to most,
since it is certainly not trivial to set up and run them locally.
This has lead to them essentially not being touched since they were first written.
The changes are as follows:
1. Moved the GH actions specific scripts to `.github/system_tests`
2. Refactored the Jenkins setup/tests to use [molecule](https://molecule.readthedocs.io) in the `.molecule/` folder
(note we use molecule for testing all the quantum mobile code).
You can read about this setup in `.molecule/README.md`,
but essentially if you just run `tox -e molecule-django` locally it will create/launch a docker container,
setup and run the tests within that container, then destroy the container.
Locally, it additionally records and prints an analysis of queries made to the database during the workchain runs.
3. Moved the Jenkins configuration to `.jenkins/`, which is now mainly a thin wrapper around (2).
This makes these tests more portable and easier to understand, modify or add to.
* 🔧 MAINTAIN: drop setuptools upper pinning (#4725)
* CI: Improve polish workchain failure debugging (#4729)
* fix: don't pass process stack via context (#4699)
This PR fixes a memory leak: when running `CalcJob`s over an SSH connection,
the first CalcJob that was run remained in memory indefinitely.
`plumpy` uses the `contextvars` module to provide a reference to the
`current_process` anywhere in a task launched by a process. When using any of
`asyncio`'s `call_soon`, `call_later` or `call_at` methods, each individual
function execution gets their own copy of this context. This means that as
long as a handle to these scheduled executions remains in memory, the copy of
the `'process stack'` context var (and thus the process itself) remain in
memory,
In this particular case, a handle to such a task (`do_open` a `transport`)
remained in memory and caused the whole process to remain in memory as well via
the 'process stack' context variable. This is fixed by explicitly passing an
empty context to the execution of `do_open` (which anyhow does not need access
to the `current_process`). An explicit test is added to make sure that no
references to processes are leaked after running process via the interpreter
as well as in the daemon tests.
This PR adds the empty context in two other invocations of `call_later`, but
there are more places in the code where these methods are used. As such it is a
bit of a workaround. Eventually, this problem should likely be addressed by
converting any functions that use `call_soon`, `call_later` or `call_at` and
all their parents in the call stack to coroutines.
Co-authored-by: Chris Sewell <chrisj_sewell@hotmail.com>
* CI: Add retry for polish workchains (#4733)
To mitigate failures on Jenkins
* 🐛 FIX: Standardise transport task interrupt handling (#4692)
For all transport tasks (upload, submit, update, retrieve),
both `plumpy.futures.CancelledError` and `plumpy.process_states.Interruption` exceptions
should be ignored by the exponential backoff mechanism (i.e. the task should not be retried)
and raised directly (as opposed to as a `TransportTaskException`),
so that they can be correctly caught by the `Waiting.execute` method.
As an example, this fixes a known bug, whereby the upload task could not be
cancelled via `CTRL-C` in an ipython shell.
* Update use of various deprecated APIs (#4719)
This replaces the use of various deprecated APIs pointed out by warnings
thrown during runs of the test suite.
It also introduces one new feature and a bug fix.
Features:
* Add non-zero exit code for failure to most `verdi daemon` commands,
so tests will catch possible errors.
Bug fixes:
* A couple of files were opened but not closed
Updates of deprecated APIs:
* np.int is deprecated alias of int
* np.float is deprecated alias of float
* put_object_from_filelike: force is deprecated
* archive import/export: `silent` keyword is deprecated in favor of logger
* computer name => label
* Fix tests writing to the repository of nodes after they had been stored
by replacing all times we use `.open` with `'w'` or `'wb'` mode
with a correct call to `put_object_from_filelike` *before* the node is stored.
In one case, the data comes from a small archive file. In this case,
I recreated the (zipped) .aiida file adding two additional (binary) files
obtained by gzipping a short string.
This was used to ensure that `inputcat` and `outputcat` work also
when binary data was requested. Actually, this is better than before,
where the actual input or output of the calculation were overwritten
and then replaced back.
* communicator: replace deprecated stop() by close()
* silence some deprecation warnings in tests of APIs that will be removed in 2.0
Note that while unmuting the `ResourceWarning` was good to spot
some issues (bug fix above), the warning is raised in a couple more
places where it's less obvious to fix (typically related to the daemon
starting some process in the background - or being started itself -
and not being stopped before the test actually finished).
I think this is an acceptable compromise - maybe we'll figure out
how to selectively silence those, and keeping warnings visible will
help us figure out possible leaks in the future.
Co-authored-by: Giovanni Pizzi <giovanni.pizzi@epfl.ch>
* ✨ NEW: Add `verdi database summary` (#4737)
Prints a summary of the count of each entity and,
with `-v` flag, additional summary of the unique identifiers for some entities.
* Upgrading dependency of sqlalchemy-utils (#4724)
* Upgrading dependency of sqlalchemy-utils
In sqlalchemy-utils 0.35, imports from collections where correctly
fixed to import from collections.abc (where this is needed).
This removes a few deprecation warnings (claiming that this will not
work in py 3.9, even if in reality this will stop working in py 3.10).
This partially addresses #4723.
We are actually pinning to >=0.36 since in 0.36 a feature was dropped
that we were planning to use (see #3845). In this way, we avoid relying
on a feature that is removed in later versions (risking to implement
something that then we have to remove, or even worse remain "pinned"
to an old version of sqlalchemy-utils because nobody has the time
to fix it with a different implementation [which is tricky, requires
some knowledge of how SqlAlchemy and PosgreSQL work]).
* Automated update of requirements/ files. (#4734)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Carl Simon Adorf <simon.adorf@epfl.ch>
* Bump aiida-prerequisites base image to 0.3.0 (#4738)
Changes in the new image:
- Updated conda (4.9.2)
- Start ssh-agent at user's startup
Co-authored-by: Chris Sewell <chrisj_sewell@hotmail.com>
* Add CalcJob test over SSH (#4732)
Adds a configuration for a remote computer (slurm docker container) and uses it
to run a CalcJob test over SSH.
This is a follow-up on the memory leak tests, since the leak of the process
instance was discovered to occur only when running CalcJobs on a remote
computer via an SSH connection.
Co-authored-by: Chris Sewell <chrisj_sewell@hotmail.com>
* 🧪 TESTS: Add pytest `requires_rmq` marker (#4739)
* Work on `verdi group remove-nodes` command (#4728).
* `verdi group remove-nodes`: Add warning when nodes are not in the group
Currently, the `verdi group remove-nodes` command does not raise any
warning when the nodes that the user wants to remove are not in the
group. It also says it removed the number of requested nodes from the
group, even when none of them is in the group specified.
Here we:
* Have the command fail with a `Critical` message when none of the
requested nodes are in the group.
* Raise a warning when any of the nodes requested are not in the
specified group, and list the PK's of the nodes that are missing.
Note that the Group.remove_nodes() command still does not raise any
warning when the requested nodes are not in the group.
* Fix bug and improve API
Fixes a bug when the user actually doesn't provide any nodes. In case
the `--clear` flag is also not provided, the command will fail since
there is nothing to remove. In case it is provided, the command will ask
for confirmation to remove all nodes unless the force flag is also set.
* Fail if both the `--clear` flag and nodes are provided
In the current API, it doesn't make sense to provide *both* the
`--clear` flag and a list of node identifiers. Here we check if both are
provided and abort the command in this case.
* Add tests.
* CI: Increase output verbosity of tests suite. (#4740)
* Skip test 'TestVerdiProcessDaemon::test_pause_play_kill'. (#4747)
The test randomly fails to complete within a reasonable amount of time
leading to a significant disruption of our CI pipeline.
Investigated in issue #4731.
* 🧪 TESTS: Fix pre-commit (pin astroid) (#4757)
Temporary fix for https://github.com/PyCQA/astroid/issues/895
* 🧪 TESTS: Fix plumpy incompatibility (#4751)
As of https://github.com/aiidateam/plumpy/commit/7004bd96bbaa678b5486a62677e139216877deef,
a paused workchain will hang if it is closed then played.
This test violated that rule and also was faulty,
in that it should test that the reloaded workchain can be played,
not the original workchain.
* 🔧 MAINTAIN: Reduce test warnings (#4742)
This commit reduces the number of pytest warnings of the test suite,
from 719 to 122:
- Replace `collections` with `collections.abc`
- pytest-asyncio does not work with `unittest.TestCase` derived tests (https://github.com/pytest-dev/pytest-asyncio/issues/77).
- `ProcessFuture` already closed via polling should not set a result via a broadcast event.
- Upgrade kiwipy and plumpy to fix:
- https://github.com/aiidateam/kiwipy/pull/98
- https://github.com/aiidateam/plumpy/pull/204
- https://github.com/aiidateam/plumpy/pull/206
* 📚 DOCS: Add `BaseRestartWorkchain` how-to (#4709)
This section is adapted from:
https://github.com/aiidateam/aiida-tutorials/blob/master/docs/pages/2020_Intro_Week/sections/workflows_adv.rst
* CI: Bump reentry to v1.3.2 (#4746)
* 🐛 FIX: Node comments API (#4760)
* Fix hanging direct scheduler+ssh (#4735)
* Fix hanging direct scheduler+ssh
The fix is very simple: in the ssh transport, to emulate 'chdir',
we keep the current directory in memory, and we prepend every command
with a `cd FOLDER_NAME && ACTUALCOMMAND`.
One could put `;` instead of `&&`, but then if the folder does not
exist the ACTUALCOMMAND would still be run in the wrong folder, which is
very bad (imagine you are removing files...).
Now, in general this is not a problem. However, the direct scheduler
inserts a complex-syntax bash command to run the command in the background
and immediately get the PID of that process without waiting.
When combined with SSH, this hangs until the whole process is completed, unless
the actual command is wrapped in brackets.
A simple way to check this is running these two commands, that reproduce
the issue with plain ssh, without paramiko:
This hangs for 5 seconds:
```
ssh localhost 'cd tmp && sleep 5 > /dev/null 2>&1 & echo $!'
```
This returns immediately, as we want:
```
ssh localhost 'cd tmp && ( sleep 5 > /dev/null 2>&1 & echo $! )'
```
Also, adding a regression test for the hanging direct+ssh combination
This test checks that submitting a long job over the direct scheduler
does not "hang" with any plugin.
Co-authored-by: Leopold Talirz <leopold.talirz@gmail.com>
* ♻️ REFACTOR: configuration management API and CLI (#4712)
This commit primarily refactors the `verdi config` command
and merges the `cache_config.yml` into the `config.json`.
`config.json` changes:
- A jsonschema is added to validate the `config.json`,
and also provide the options/defaults previously in `aiida/manage/configuration/options.py`.
- Rename option keys (with migration),
for consistency with the internal representation
(also rename `user.` fields to `autofill.user.`)
- Allow the `config.json` to contain a `$schema` key,
that is preserved when storing new data
- Deprecated `cache_config.yml`: auto-merged into `config.json`,
with deprecation warning, then renamed
- An `rmq.task_timeout` option has also been added
(with default increased from 5 to 10 seconds),
to fix timeout errors at high process loads.
`verdi config` changes:
- Refactor `verdi config` into separate commands: list/get/set/show/unset
- Include deprecation for current `verdi config <KEY>`
- `verdi caching` lists all process entry points that are enabled/disabled for caching
Also, code in `aiida/manage/caching.py` now utilises
the `get_config_option` function to retrieve caching configuration.
* 🧪 TESTS: add `config_with_profile` fixture (#4764)
This allows for the removal of
`temporary_config_instance` and `with_temporary_config_instance`
from `tests/utils/configuration.py`
* 👌 IMPROVE: `verdi config list/show` (#4762)
Ensure these commands still work before a profile has been configured.
* 👌 IMPROVE: Add config `logging.aiopika_loglevel` (#4768)
* 📚 DOCS: Add process submit diagram (#4766)
* 📚 DOCS: Add process submit diagram
* Create submit_sysml.pptx
* 👌 IMPROVE: CTRL-C on running process (#4771)
Do not call `kill` on a process that is already being killed.
Also log a different message,
so that the user can see that the original CTRL-C was actioned.
* 🐛 FIX: kill_calculation before job submitted (#4770)
`job_id` will not yet have been set,
so we should not ask the scheduler to kill it.
* 🐛 FIX: `ModificationNotAllowed` on workchain kill (#4773)
In `Process.kill` the parent is killed first, then the children.
However, for workchains when entering the `Wait` state, awaitables (e.g. children)
are each assigned to `WorkChain.on_process_finished` as a callback on termination.
When the child is killed, this callback then calls `resolve_awaitable`,
which tries to update the status of the parent.
The parent is already terminated though and the node sealed -> `ModificationNotAllowed`.
In this commit we therefore check if the parent is already in a terminal state,
before attempting to update its status.
* 👌 IMPROVE: capture of node hashing errors (#4778)
Currently all exceptions are caught and ignored.
This commit adds a specific `HashingError` exception,
for known failure modes.
Only this exception is caught, if `ignore_errors=True`,
and the exception logged.
Also an `aiida_caplog` pytest fixture is added,
to enable logs from `AiiDA_LOGGER` to be captured.
* ⬆️ UPDATE: kiwipy/plumpy (#4776)
Update to new patch versions:
kiwipy v0.7.3:
- 👌 IMPROVE: Add debug logging for sending task/rpc/broadcast to RMQ.
- 👌 IMPROVE: Close created asyncio loop on RmqThreadCommunicator.close
plumpy v0.18.6:
- 👌 IMPROVE: Catch state change broadcast timeout
When using an RMQ communicator, the broadcast can timeout on heavy loads to RMQ.
This broadcast is not critical to the running of the process,
and so a timeout should not except it.
In aiida-core, the broadcast is subscribed to by `verdi process watch` (not critical),
in `aiida/engine/processes/futures.py:ProcessFuture` (unused),
and in `aiida/engine/runners.py:Runner.call_on_process_finish`
which has a backup polling mechanism on the node.
Also ensure the process PID is included in all log messages.
* Add fallback equality relationship based on uuid (#4753)
Add fallback equality relationship based on node uuid .
* Simplify AiidaTestCase implementation (#4779)
This simplifies the `AiidaTestCase` implementation - not yet replacing it with pytest fixtures,
but hopefully getting one step closer to doing so eventually.
In particular
* only truly backend-specific code is left in the backend-specific test classes
* introduces `refurbish_db()` which includes the combination of cleaning the db and repopulating it with a user (which is a common combination)
* move creation of default computer from `setUpClass` to "on demand" (not needed by many tests)
* merges `reset_database` and `clean_db` function that basically did the same
* factors out the `get_default_user` function so that it can be reused outside the AiidaTestCase (`verdi setup`, pytest fixtures, ...) in a follow-up PR
* add `orm.Computer.objects.get_or_create` (in analogy to similar methods for user, group, ...)
Note: While this change gets rid of unnecessary complexity, it does *not* switch to a mode where the database is cleaned between *every* test.
While some subclasses of `AiidaTestCase` do this, the `AiidaTestCase` itself only cleans the database in `setupClass`.
Some subclasses do significant test setup at the class level, which might slow things down if they had to be done for every test.
* 👌 IMPROVE: add broker_parameters to config schema (#4785)
* 👌 IMPROVE: Add 'exception' to projection mapping (#4786)
This commit adds `exception` to the list of allowed projections,
and also standardises the way the exception is set on the node
(capturing both the type and message).
* docs: reorder/simplify caching howto (#4787)
The howto on enabling caching has been reordered to move the concepts to
the beginning and technical details to where they fit better.
The figure has been simplified (complexity introduced by second input
node unnecessary).
Added explicit mention of the fact that hashing is enabled by default
(which may not be obvious).
Co-authored-by: Chris Sewell <chrisj_sewell@hotmail.com>
* docs: reference caching howto in workflow section (#4789)
It might be helpful to people learning about AiiDA workflows to know that caching exists and point them in that direction.
Co-authored-by: Leopold Talirz <leopold.talirz@gmail.com>
* setup: move away from legacy build backend (#4790)
The `pyproject.toml` was originally added in ca75832afb002b344b5854f2f049c74e80cad36b
without specifying a backend, which implicitly defaults to the legacy `setuptools.build_meta:__legacy__` one.
This choice was made explicit in a2bebb422f4a7b75e8ef65fd797f128abf12c6cc
This can lead to issues when using a *system* version of setuptools
< 40.8.0, see [1].
We believe there is no good reason for sticking with the legacy build system.
I've tested that the `reentry_register` hook still works with the new
build backend.
[1] https://github.com/pypa/setuptools/issues/1694#issuecomment-466010982
* fix pymatgen imports (#4794)
pymatgen made a breaking change in v2021.3.4 that removed many classes
from the top level of the package.
The alternative imports were already available in previous versions,
i.e. we don't need to upgrade the pymatgen dependency.
* 🐛 FIX: `get_pymatgen_version` (#4796)
In version 2022.0.3 it was moved
* 👌 IMPROVE: add type checking for aiida/orm/nodes/process (#4772)
This commit adds type definitions to all code in `aiida/orm/nodes/process`,
and enables mypy type checking of the files.
Additionally, to fix mypy failures, two changes to the code were made:
1. Change `CalcJobNode.get_description` to return a string
2. In `aiida/engine/processes/calcjobs/tasks.py`,
change `node.computer.get_authinfo(node.user)` to `node.get_authinfo()`,
to use `CalcJobNode.get_authinfo` which checks if the computer is set.
* 🐛 FIX: `WorkChain.resolve_awaitable` (#4795)
An alteration to a recent fix (#4773);
`Process.has_terminated` is a method, not a property.
* 🐛 FIX: `Task.cancel` should not set state as EXCEPTED (#4792)
Currently, stopping the daemon in python 3.7 excepts all processes.
This is due to the code in `shutdown_runner`,
which cancels all asyncio tasks running on the loop,
including process continue and transport tasks.
Cancelling a task raises an `asyncio.CancellErrror`.
In python 3.8+ this exception only inherits from `BaseException`,
and so is not caught by any `except Exception` "checkpoints" in plumpy/aiida-core.
In python <= 3.7 however, the exception is equal to `concurrent.futures.CancelledError`,
and so it was caught by one of:
`Process.step`, `Running.execute` or `ProcessLauncher.handle_continue_exception`
and the process was set to an excepted state.
Ideally in the long-term, we will alter `shutdown_runner`,
to not use such a "brute-force" mechanism.
But in the short-term term this commit directly fixes the issue,
by re-raising the `asyncio.CancelledError` exception.
* Docs: fix the citation links on the index page (#4800)
The links were still using markdown syntax instead of restructured text.
* `CalcJob`: add the option to stash files after job completion (#4424)
A new namespace `stash` is added to the `metadata.options` input
namespace of the `CalcJob` process. This option namespace allows a user
to specify certain files that are created by the calculation job to be
stashed somewhere on the remote. This can be useful if those files need
to be stored for a longer time than the scratch space where the job was
run is typically not cleaned for, but need to be kept on the remote
machine and not retrieved. Examples are files that are necessary to
restart a calculation but are too big to be retrieved and stored
permanently in the local file repository.
The files that are to be stashed are specified through their relative
filepaths within the working directory in the `stash.source_list`
option. For now, the only supported option is to have AiiDA's engine
copy the files to another location on the same filesystem as the working
directory of the calculation job. The base path is defined through the
`stash.target_base` option. In the future, other methods may be
implemented, such as placing all files in a (compressed) tarball or even
stash files on tape. Which mode is to be used is communicated through
the enum `aiida.common.datastructures.StashMode` which for now therefore
only has the `COPY` value.
If the `stash` option namespace is defined for a calculation job, the
daemon will perform the stashing operations before the files are
retrieved. This also means that the stashing also happens before the
parsing of the output files (which occurs after the retrieving step)
which means that the files will be stashed independent of the final
exit status that the parser will assign to the calculation job. This
may cause files to be stashed of calculations that will later be
considered to have failed. However, the stashed files can always be
deleted manually by the user afterwards if needed.
Finally, the stashed files are represented by an output node that is
attached to the calculation node through the label `remote_stash`. Just
like the `remote_folder` node, this represents a location or files on a
remote machine and so is merely a "symbolic link" of sorts. AiiDA does
not actually own the files and the contents may disappear at some point.
To be able to distinguish the stashed folder from the remote folder, a
new data plugin is used, the `RemoteStashFolderData`. The base class is
`RemoteStashData` which is not instantiable, but will merely serve as a
base class for future subclasses, one for each `StashMode` value. The
reason is that the way files need to be accessed depend on the way they
were stashed and so it is good to have separate classes for this.
It was considered to give `RemoteFolderData` and `RemoteData` the same
base class (changing the type of the `remote_folder` to a new subclass
`RemoteFolderData`) but this would introduce breaking changes and so this
was relegated to a potential future major release.
* `verdi process play`: only query for active processes with `--all` flag (#4671)
The query used to target all process nodes with the `paused` attribute, so even
those in a terminal state. Here an additional filter is added to only query for nodes
in an active process state, because terminal nodes should not be affected. This
should speed up the query in principle.
* Dependencies: update pymatgen version specification (#4805)
Addresses #4797
* Dependencies: Pin sqlalchemy to minor release (#4809)
Version 1.4 currently breaks `verdi setup` and indeed,
according to https://www.sqlalchemy.org/download.html,
minor releases of SqlAlchemy may have breaking changes.
* 📚 DOCS: Add documentation on stashing (#4812)
Some additional minor changes
* Add link for `TransferCalcjob` feedback
* Add `versionadded` to `TransferCalcjob` docs
* 🔧 MAINTAIN: Add PyPI release workflow (#4807)
This is workflow is intended to reduce the potential for manual errors and faulty releases.
When you create the release, and hence git tag, this workflow is triggered;
checks the tag created matches the aiida package version,
runs pre-commit and (some) pytests and, if they all pass, deploys to PyPI.
* 🚀 RELEASE: v1.6.0 (#4816)
Co-authored-by: ramirezfranciscof <ramirezfranciscof@users.noreply.github.com>
Co-authored-by: Sebastiaan Huber <mail@sphuber.net>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dominik Gresch <greschd@users.noreply.github.com>
Co-authored-by: flavianojs <flavianojs@live.com>
Co-authored-by: Marnik Bercx <mbercx@gmail.com>
Co-authored-by: Jason Eu <morty.yu@yahoo.com>
Co-authored-by: ramirezfranciscof <ramirezfranciscof@users.noreply.github.com>
Co-authored-by: Pranjal Mishra <39010495+pranjalmish1@users.noreply.github.com>
Co-authored-by: Leopold Talirz <leopold.talirz@gmail.com>
Co-authored-by: Casper Welzel Andersen <casper.andersen@epfl.ch>
Co-authored-by: Carl Simon Adorf <carl.simon.adorf@gmail.com>
Co-authored-by: Carl Simon Adorf <simon.adorf@epfl.ch>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Giovanni Pizzi <giovanni.pizzi@epfl.ch>
Co-authored-by: Aliaksandr Yakutovich <yakutovicha@gmail.com>
Co-authored-by: DanielMarchand <Daniel.marchand@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR builds on #4575
delete_nodesusable outside ofclick; adding a callback for the confirmation step, rather than callingclick.confirmdirectly, and using logging instead ofclick.echoaiida/manage/database/delete/nodes.pytoaiida/tools/graph/deletions.py, leaving a deprecation warning at the old location. This is a more intuitive place since it is directly build on the graph traversal functionality.from aiida.tools import delete_nodesand adds their use to the documentation.delete_group_nodesmainly as a wrapper arounddelete_nodes; querying for all the node pks in the groups, then passing these todelete_nodesverdi group delete --delete-nodes, with the same flags and logic asverdi node deleteverdi node delete, introduced by Improveverdi node deleteperformance #4575 😬, if a node does not exists (I needed to convert the pks to int 🤦)Sorry @mbercx I've superseded your PR #4425 with a more holistic PR 😬
I also want to make(done)delete_nodesanddelete_group_nodesavailable in the public API.I'm not sure if
aiida.manage.database.delete.nodesis really the right place (then bubble up toaiida.manage), or this module should be moved elsewhere