-
Notifications
You must be signed in to change notification settings - Fork 192
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
♻️ REFACTOR: Move archive backend to aiida/storage
#5375
Conversation
cebcf86
to
6814965
Compare
64dd454
to
4474123
Compare
Ok this is just about done, so ready for review. some potential TODOS:
|
6a06448
to
bd1e3eb
Compare
0288589
to
5ee3590
Compare
Replaced by `verdi archive version` and `verdi archive info`, to bring it inline with `verdi storage`
90103c4
to
f0991b3
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @chrisjsewell . I have given it a bit of a look, but there is too much to really do it in detail and since this needs to go in for the beta release, I am approving. We should test ourselves the new archive functionality soon such that any bugs can be fixed a.s.a.p.
Way ahead of you 😉 I already added #5407 on Friday Just gotta update the PR description, then I'll merge |
This PR moves the archive storage backend implementation (
sqlite_zip
) intoaiida/storage
alongside the "main"
psql_dos
backend.It includes both the storage reading and migration functionality.
Inline with this synchronisation of the archive and the storage backend:
UnreadableArchiveError
/CorruptArchive
->CorruptStorage
IncompatibleArchiveVersionError
->IncompatibleStorageSchema
ArchiveMigrationError
->StorageMigrationError
ArchiveClosedError
->ClosedStorage
MIGRATE_LOGGER
has been moved toaiida/storage/log.py
verdi archive inspect
has been deprecatedand replaced by
verdi archive version
andverdi archive info
,to bring it inline with
verdi storage
.alembic migrations have been added to the
sqlite_zip
migration logic,to support future migrations of the sqlite database.
Alongside this, the non-null restrictions for legacy archive fields have been relaxed,
to improve robustness.
Subsequent migrations have been added;
first, to replace any existing (unwanted) null values with default values,
then to modify the schema to re-add the non-null constraints.
Finally, additional tests have been put in place (in
tests/tools/archive/test_schema.py
),to ensure that the sqlite schema is always synchronised with the postgresql schema.
Notes
Writing of archive files is still specific to the archive functionality, due to the design constraint of an archive being contained within a single file.
Because of this, one cannot efficiently modify an existing archive, without
This should also eventually allow for the
aiida/tools/archive/create.py::create_archive
andaiida/tools/archive/imports.py::import_archive
functions to be "re-conceptualised" as simply transfers of data between storage backends (see also aiidateam/AEP#31)