Future: make migration of data types defined by plugin packages possible

[mbercx]

Motivation

Currently, database migrations are only possible for data types that are defined in aiida-core. This means that e.g. data plugins that are a part of a plugin package like aiida-quantumespresso can not change the way the data is stored in the SQL database and repository.

Desired Outcome

This item will have been resolved once it is possible to migrate data types that come with a plugin instead of being defined in aiida-core.

Impact

A couple of example use cases are:

1. There are already plenty of data types defined in plugin packages at the moment, whose storage can now not be changed (e.g. the UpfData in aiida-pseudo, ForceConstantsData in aiida-quantumespresso). In case we want to for example add some information to the database to make it easier to query for, this is currently not possible.

2. New data plugins are also being implemented (see e.g. Hubbard data structure for handling newest QuantumESPRESSO versions (>=7.1) aiida-quantumespresso#849), where the best solution on how to store the data in the database/repository isn't always clear. Any new data type that is now implemented outside aiida-core needs to be very careful in making this choice, and it's always possible to overlook issues with the design, complicating the development of new data types in plugin packages.

3. There are several data types currently defined in aiida-core that we may want to move to a more appropriate plugin package (see Background). One barrier to doing so is the fact that once we do this it will be impossible to change their storage.

Complexity

Implementing this feature has already been discussed at several previous coding events, and so far no straightforward solution has been found. So it's fair to say this is a quite complex issue that will take at least a few weeks of dedicated time to solve.

Background

One context in which this feature was discussed is the move of the materials science-related data types outside of aiida-core to make it more field agnostic:

aiidateam/aiida-core#2686

Progress

There have been some brain storming sessions on how to proceed with this in previous coding weeks, but as far as I know no real progress has been made towards implementing this feature.

[mbercx]

@sphuber if I remember correctly, you already had a look at this? Do we have some notes on the topic somewhere?