Downloading only user-created metadata

[qubixes]

Related to #221, but different question.

I'm currently working on trying to implementing an experimental feature to allow users to download and upload the metadata. We're using the python-irodsclient API to get all available metadata for collections and data objects. I have a prototype that is working, however, it also downloads non-user created metadata such as org_replication_scheduled and org_revision_scheduled.

I could blacklist those two specific keys within the iBridges API, but I'm worried that there will be other entries that I haven't seen yet that also should be blacklisted. What is the best way to skip these metadata entries?

[lwesterhof]

You can use the meta_form API (https://petstore.swagger.io/?url=https://utrechtuniversity.github.io/yoda-ruleset/api_core.json#/meta_form) to download and upload the metadata for collections using the python-irodsclient. This API also validates the metadata against the configured metadata schema.

[qubixes]

Thanks @lwesterhof for the suggestion. I have a few questions:

• How would I use the python-irodsclient with the linked API? Wouldn't I use something like the requests library in Python?
• If I understand correctly, the meta_form API is YoDa specific?
• What about metadata linked to data objects, how would I retrieve/set those?

[lwesterhof]

• You can call that specific API directly as a rule with the PRC or indeed use request to go through the http API. For an example you can look at our test code for the API: https://github.com/UtrechtUniversity/yoda-ruleset/blob/62d697d343db7c75be50d879e72cd622bae0e3e0/tests/conftest.py#L225-L249
• Yes, meta_form is working with the metadata schemas configured in Yoda
• Yoda does not (yet) support metadata on data objects, so there are no API available. You can use GenQuery through the PRC to gather all the metadata you are looking for.

[qubixes]

• You can call that specific API directly as a rule with the PRC or indeed use request to go through the http API. For an example you can look at our test code for the API: https://github.com/UtrechtUniversity/yoda-ruleset/blob/62d697d343db7c75be50d879e72cd622bae0e3e0/tests/conftest.py#L225-L249

  • Yes, meta_form is working with the metadata schemas configured in Yoda

  • Yoda does not (yet) support metadata on data objects, so there are no API available. You can use GenQuery through the PRC to gather all the metadata you are looking for.

For the feature I'm working on it would be necessary to also get the metadata on data objects, so for now, if I understand correctly, I can't use the meta_form API (using the API as a rule sounds the most promising, since we can use the authenticated PRC session).

In the current implementation I'm using the PRC API with item.metadata.items(), where item can be a collection or a data object. If I understand correctly, the PRC uses GenQuery to get this information. The problem is that it will get too much metadata (org_replication_scheduled and org_revision_scheduled are not user metadata).

The problem I foresee is not necessarily with the download of the metadata (users can ignore the items they don't understand). However, if these items get uploaded back to the YoDa server, that might have some unintended consequences?

Would it be a good workaround to get/put all the metadata I need through PRC/GenQuery and ignore all entries with the keywords org_replication_scheduled and org_revision_scheduled? Or do I need to add more keywords that should be ignored in this black list?

[lwesterhof]

That would be a good workaround, just ignore all metadata with org_ prefix.