IQSS/6497 migrate dataset api #7504

qqmyers · 2021-01-08T17:16:28Z

What this PR does / why we need it: This PR supports migration of datasets from other systems, keeping their PID, version, and publicationdate, and allowing file upload via the existing API. It leverages leveraging the json-ld metadata api from #7414.
(Note that #7414 in turn requires resolution of updating our Java version or having a work-around for Java 8 added there - see that PR for details).

Which issue(s) this PR closes:

Closes #6497

Special notes for your reviewer: This is similar to the :import API but is different (hopefully usefully) in a few ways:

that API call requires files to be prepositioned. The api calls here allow you to create a dataset to be migrated, then add files/filemetadata using the existingAPIs (inlcuding direct upload or using the HttpOverlay store in Simple support for remote web stores #7325), and then conclude by marking the dataset as published.
this call uses the json-ld metadata format from the ORE export, Bag archiving, and Iqss/6497 semantic api #7414 which is flatter and supports incremental metadata updates (one doesn't have to download the whole metadata tree, make an edit and reupload everything),
the last point makes it highly compatible with re-importing an archived Bag which includes the same ORE/json-ld formatted metadata
since dataset creation and version update/publish capabilities are separated into two calls, it may be simpler to support migration of multiple versions via this API (it may work already - I still need to test that capability).

FYI: The last step is slightly different than the normal publish step since version and publication dates are preserved. The PR does update the PID/DOI (e.g. the landing URL specifically) assuming it has the authority/shoulder configured in Dataverse.

Suggestions on how to test this: The DVUploader uses this API to re-import Bags. Nominally running an upload on Bags of the sample datasets would test this fairly well (current versions at https://drive.google.com/drive/folders/1YXAmJLziFs7XhsvRfqfZ6o9AvLXywyBT?usp=sharing , note that work to improve the Bag/ORE contents is ongoing, so these may get out of date w.r.t. code, so check dates/remind me if an update looks like its needed).

Assuming #7414 is tested separately, the testing here could/should focus on just the added migrate calls, so looking for datasets with a broad range of metadata probably isn't necessary (since testing #7414 covers that.) and the things to look at in a migrated dataset are the DOI, creation/publication dates, whether the DOI is updated at DataCite, etc.

Does this PR introduce a user interface change? If mockups are available, please link/include them here: Not directly - there are questions in the community (remote store/registry metadata working group topic, DANS migration, etc.) as to how one should indicate a migrated dataset, what metadata about the source should be captured, etc. This PR doesn't add any metadata or make an display changes, but this PR may increase interest in resolving these other questions.

Is there a release notes update needed for this change?:
Probably worthwhile to note it exists. As a v~1 implementation, I'd phrase this as an ~experimental API (I don't think we've decided exactly how to phrase this but there have been other recent API adds in this category).
Additional documentation:

FWIW: We have an error handler for the edu.harvard.iq.dataverse.util.json.JsonParseException class but not for javax.json.stream.JsonParsingException which was getting caught by the Throwable handler and returned as a 500 error with json message {}

and avoid having contexts with specific entries for terms that are in a namespace already

Conflicts: src/main/java/edu/harvard/iq/dataverse/util/bagit/OREMap.java

which can be used with existing AbstractApiBean.ok()

…taset_api

Co-authored-by: Philip Durbin <philipdurbin@gmail.com>

IQSS/6497-migrate_dataset_api Conflicts: src/main/java/edu/harvard/iq/dataverse/util/json/JSONLDUtil.java

…obalDataverseCommunityConsortium/dataverse.git into IQSS/6497-migrate_dataset_api

qqmyers · 2021-07-13T21:01:54Z

doc/sphinx-guides/source/admin/dashboard.rst

@@ -22,7 +22,7 @@ This dashboard tool allows you to define sets of local datasets to make availabl
 Metadata Export
 ---------------

-This part of the Dashboard is simply a reminder message that metadata export happens through the Dataverse Software API. See the :doc:`metadataexport` section and the :doc:`/api/native-api` section of the API Guide for more details.
+This part of the Dashboard is simply a reminder message that metadata export happens through the Dataverse Software API. See the :doc:`/admin/metadataexport` section and the :doc:`/api/native-api` section of the API Guide for more details.


FWIW: unrelated - just found it when fixing another link

qqmyers · 2021-07-13T21:06:24Z

src/main/java/edu/harvard/iq/dataverse/util/json/JSONLDUtil.java

-            if(migrating) {
-              // unparsable PID passed. Terminate.
-              throw new BadRequestException("Cannot parse the @id. Make sure it is in valid form - see Dataverse Native API documentation.");
+        if (migrating) {


FWIW: Submitting an '@id' value for the dataset PID, and a modificationdate (next diff) are only allowed for migrate (not valid in the create dataset case), so these are cleaner fixes to just skip running into the null values in the create case.

qqmyers · 2021-07-13T21:08:01Z

OK - with the semantic api PR merged, this is ready for review - just minor changes beyond the new migrate and releasemigrated methods.

pdurbin

Code-wise, I think this is ready for QA but there are two things I think we should fix:

The new integrations tests are yielding a 500 error.
We should update the list of APIs to include this one and others that have been recently added. Otherwise they're hard to find. @qqmyers let me know if you'd like me to make this change.

pdurbin · 2021-07-16T18:53:07Z

src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java

+        // Now use the migrate API to recreate the dataset
+        response = UtilIT.recreateDatasetJsonLD(apiToken, dataverseAlias, expectedString);
+        String body = response.getBody().asString();
+        response.then().assertThat().statusCode(CREATED.getStatusCode());


This is showing a 500 error:

pdurbin · 2021-07-16T18:58:57Z

doc/sphinx-guides/source/developers/dataset-migration-api.rst

@@ -0,0 +1,51 @@
+Dataset Migration API


Now that we're seeing a number of APIs show up under the Dev Guide, it's probably time to update the "List of Dataverse APIs" I started when I did a light re-write of the API Guide. As of https://guides.dataverse.org/en/5.5/api/intro.html#lists-of-dataverse-apis this is how it looks:

@qqmyers can you please update the "Please note that some APIs are only documented in other guides that are more suited to their audience" section and add "Developer Guide" and under that recent APIs that have been added? (Please include aux files in this.) Thanks.

can you please update the "Please note that some APIs are only documented in other guides that are more suited to their audience" section and add "Developer Guide" and under that recent APIs that have been added?

@qqmyers I just opened this pull request for this: GlobalDataverseCommunityConsortium#13

add dev guide links to list of APIs IQSS#6497

…taset_api

qqmyers · 2021-07-23T20:38:27Z

OK - @pdurbin made the doc change he asked for and I fixed the test - moving to QA (feel free to drop it back if there's something else that needs to happen first).

kcondon · 2021-07-27T13:47:56Z

Some questions/issues encountered:

Update release notes, remove solr step since it is no longer necessary, per Jim
Check and mention super user only in doc for api
In (re)create example in doc, shows PERSISTENT_IDENTIFIER in env variable but does not reference it on command line
When updating sample jsonld file with a real, published PID from demo, fails saying cannot recreate dataset that had yet to be created (sample file missing publication date)
Recreate Endpoint doesn't seem to read doi jvm-options, verifies against prod namespace, doesn't work with test namespace: curl -H X-Dataverse-key:2642aa13-5e8b-4ff4-884d-4841545c06ce -X POST "http://localhost:8080/api/dataverses/kc072621/datasets/:startmigration" --upload-file dataset-migrate.jsonld
{"status":"ERROR","message":"Provided PID does not exist. Status code for GET 'https://doi.org/10.70122/FK2/T6ESOU' is 404."

qqmyers · 2021-07-27T16:56:38Z

@kcondon - I addressed 1-4 in the latest commit. For 5), I added #8028 and referenced that in the docs.

qqmyers added 29 commits December 1, 2020 16:58

add replace param

b45336c

remove OREMap parameter

83c8794

fix error handling

6d537ce

FWIW: We have an error handler for the edu.harvard.iq.dataverse.util.json.JsonParseException class but not for javax.json.stream.JsonParsingException which was getting caught by the Throwable handler and returned as a 500 error with json message {}

append to current terms

6ee7499

handle append on terms - fix cut/paste errors

8827a0c

fix logic

92328cb

specify default

2a83a70

make replace still append for multiple val fields

d75fb78

add migrating switch

782d5ec

expose uri in datasetField api

be978e9

track defined namespaces

faa92f6

and avoid having contexts with specific entries for terms that are in a namespace already

define equals, avoid duplicates in list

fb08a38

replace string with const

ece3bff

constant for CC0_URI

8b737dd

GET/DELETE endpoints

63362d9

Conflicts: src/main/java/edu/harvard/iq/dataverse/util/bagit/OREMap.java

7130-handle missing contact name

849f6f8

Fix multiple description logic for info file

8321f62

put is always for :draft version

5050e93

don't cast to String[]

11292c0

add more logging

cbbb084

add method that can return JsonObjectBuilder

fd14226

which can be used with existing AbstractApiBean.ok()

log details on failure

7fe0249

multiple updates/fixes, added logging

24cfcfa

Merge remote-tracking branch 'IQSS/develop' into IQSS/6497-migrate_da…

8d889c1

…taset_api

merge issues

a01c421

fix terms retrieval

6ce3a1c

merge issue

0bde4e8

add note

e6366e4

Merge remote-tracking branch 'IQSS/develop' into IQSS/6497-migrate_da…

043420a

…taset_api

qqmyers marked this pull request as draft January 8, 2021 17:16

qqmyers and others added 3 commits July 8, 2021 13:12

Update doc/sphinx-guides/source/developers/dataset-migration-api.rst

64ed2b0

Co-authored-by: Philip Durbin <philipdurbin@gmail.com>

Merge remote-tracking branch 'IQSS/develop' into

e96c24a

IQSS/6497-migrate_dataset_api Conflicts: src/main/java/edu/harvard/iq/dataverse/util/json/JSONLDUtil.java

Merge branch 'IQSS/6497-migrate_dataset_api' of https://github.com/Gl…

68f33a6

…obalDataverseCommunityConsortium/dataverse.git into IQSS/6497-migrate_dataset_api

qqmyers commented Jul 13, 2021

View reviewed changes

pdurbin self-assigned this Jul 16, 2021

pdurbin requested changes Jul 16, 2021

View reviewed changes

pdurbin assigned qqmyers Jul 16, 2021

add dev guide links to list of APIs IQSS#6497

b1acb2b

pdurbin mentioned this pull request Jul 21, 2021

add dev guide links to list of APIs #6497 GlobalDataverseCommunityConsortium/dataverse#13

Merged

qqmyers added 4 commits July 22, 2021 15:09

Merge pull request #13 from IQSS/6497-api-list

e074c1e

add dev guide links to list of APIs IQSS#6497

Merge remote-tracking branch 'IQSS/develop' into IQSS/6497-migrate_da…

30d131c

…taset_api

lost test update

52fbc93

Merge remote-tracking branch 'IQSS/develop' into IQSS/6497-migrate_da…

e544a9a

…taset_api

pdurbin unassigned pdurbin and qqmyers Jul 23, 2021

kcondon self-assigned this Jul 26, 2021

qqmyers mentioned this pull request Jul 27, 2021

Import APIs don't work with DataCite test DOIs #8028

Closed

respond to QA comments

8f749e7

qqmyers added 3 commits July 27, 2021 15:23

handle version 1.0 info already set

c833eb9

enable DOI updates

d549f72

cut/pasted to the wrong place

93481bd

kcondon merged commit 1c3dbe8 into IQSS:develop Jul 27, 2021

djbrooke added this to the 5.6 milestone Jul 28, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IQSS/6497 migrate dataset api #7504

IQSS/6497 migrate dataset api #7504

qqmyers commented Jan 8, 2021 •

edited

Loading

qqmyers Jul 13, 2021

qqmyers Jul 13, 2021

qqmyers commented Jul 13, 2021

pdurbin left a comment

pdurbin Jul 16, 2021

pdurbin Jul 16, 2021

pdurbin Jul 21, 2021

qqmyers commented Jul 23, 2021

kcondon commented Jul 27, 2021 •

edited

Loading

qqmyers commented Jul 27, 2021

IQSS/6497 migrate dataset api #7504

IQSS/6497 migrate dataset api #7504

Conversation

qqmyers commented Jan 8, 2021 • edited Loading

qqmyers Jul 13, 2021

Choose a reason for hiding this comment

qqmyers Jul 13, 2021

Choose a reason for hiding this comment

qqmyers commented Jul 13, 2021

pdurbin left a comment

Choose a reason for hiding this comment

pdurbin Jul 16, 2021

Choose a reason for hiding this comment

pdurbin Jul 16, 2021

Choose a reason for hiding this comment

pdurbin Jul 21, 2021

Choose a reason for hiding this comment

qqmyers commented Jul 23, 2021

kcondon commented Jul 27, 2021 • edited Loading

qqmyers commented Jul 27, 2021

qqmyers commented Jan 8, 2021 •

edited

Loading

kcondon commented Jul 27, 2021 •

edited

Loading