Usability: Make it possible to interact with an AiiDA instance over a web API

[sphuber]

Motivation

AiiDA provides an extensive Python API, which is the main way to interact with the system. In addition, there is the verdi CLI for quick access over the command line. While both APIs provide in most required use-cases, they require direct access to the machine where the AiiDA instance is running. However, there are use-cases where direct access is either not possible or not desirable. In these cases, it would be useful to provide a web API to allow interaction with an AiiDA instance over the HTTP protocol.

There are a number of use-cases known that would benefit from a web API:

• Expose data from an AiiDA database to users. Prime example for this is the Materials Cloud which already uses a REST API to expose data through a browser.
• Allow users to submit processes to AiiDA. Prime example is AiiDAlab. Currently the HTTP server serving the web interface runs on the same server as the AiiDA instance. Providing a web API would allow to decouple these two services.
• Institutions or organizations could more easily provide a multi-user setup for a single AiiDA instance on a dedicated server, with users interacting it on their own client machines through the web API.

Desired Outcome

Ideally, there should be a single stable well-supported (official) web API that allows users to interact with an AiiDA instance over HTTP.

Impact

There are already a number of known use-cases of a web API that each have a considerable number of (in)direct users. Consolidating existing solutions into a single stable solution would benefit everyone and would reduce development and maintenance cost in the long-term.

Complexity

The main challenges in realizing this project consist of:

• Provide a secure method of authentication and authorization. By allowing the web API to mutate the state of the AiiDA instance it is important that the user can be properly authenticated and verified to have the required authorization
• The web API won't be able to use the Python API. All data will need to be serialized to probably JSON to be able to be sent over the wire. This presents a challenge of how to serialize/deserialize various AiiDA ORM entities. The ones provided by aiida-core can of course be hard-coded in the server, however, any classes provided by plugins (such as Data plugins) have to be handled dynamically. It is currently not yet clear if a general serialization/deserialization scheme can be implemented without breaking the current Python ORM implementation.
• Deciding which existing web API to keep: the aiida-core REST API or the web API from aiida-restapi.
• If aiida-restapi is kept, we should probably decide if we keep its REST API or GraphQL based implementation. Currently there is a mix of the two with a non-zero overlap in functionality.

Progress

The aiida-core package already ships with a REST API implementation. However, this is read-only and does not support requests that mutate the state of a profile storage. The aiida-restapi has implemented a web API that does allow mutations, but it currently mixes two approaches: a REST API implemented using fastapi and a web API using GraphQL.

Concrete action points:

• Check whether all functionality required by Materials Cloud is provided by aiida-restapi. Once this attains feature parity, Materials Cloud can switch to aiida-restapi as a drop in replacement of the REST API that ships with aiida-core, which can then be deprecated and phased out (Assigned to @eimrek )
• Check whether all functionality required by AiiDAlab QeApp is provided by aiida-restapi. This should serve as a good benchmark to see whether the API can be used for process management. (Assigned to @unkcpz and @superstar54 )